Podcasts about SQL

About ThomasThomas Hazel is Founder, CTO, and Chief Scientist of ChaosSearch. He is a serial entrepreneur at the forefront of communication, virtualization, and database technology and the inventor of ChaosSearch's patented IP. Thomas has also patented several other technologies in the areas of distributed algorithms, virtualization and database science. He holds a Bachelor of Science in Computer Science from University of New Hampshire, Hall of Fame Alumni Inductee, and founded both student & professional chapters of the Association for Computing Machinery (ACM).Links:ChaosSearch: https://www.chaossearch.io TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by my friends at ThinkstCanary. Most companies find out way too late that they've been breached. ThinksCanary changes this and I love how they do it. Deploy canaries and canary tokens in minutes and then forget about them. What's great is the attackers tip their hand by touching them, giving you one alert, when it matters. I use it myself and I only remember this when I get the weekly update with a “we're still here, so you're aware” from them. It's glorious! There is zero admin overhead  to this, there are effectively no false positives unless I do something foolish. Canaries are deployed and loved on all seven continents. You can check out what people are saying at canary.love. And, their Kub config canary token is new and completely free as well. You can do an awful lot without paying them a dime, which is one of the things I love about them. It is useful stuff and not an, “ohh, I wish I had money.” It is speculator! Take a look; that's canary.love because it's genuinely rare to find a security product that people talk about in terms of love. It really is a unique thing to see. Canary.love. Thank you to ThinkstCanary for their support of my ridiculous, ridiculous non-sense.   Corey: This episode is sponsored in part by our friends at Vultr. Spelled V-U-L-T-R because they're all about helping save money, including on things like, you know, vowels. So, what they do is they are a cloud provider that provides surprisingly high performance cloud compute at a price that—while sure they claim its better than AWS pricing—and when they say that they mean it is less money. Sure, I don't dispute that but what I find interesting is that it's predictable. They tell you in advance on a monthly basis what it's going to going to cost. They have a bunch of advanced networking features. They have nineteen global locations and scale things elastically. Not to be confused with openly, because apparently elastic and open can mean the same thing sometimes. They have had over a million users. Deployments take less that sixty seconds across twelve pre-selected operating systems. Or, if you're one of those nutters like me, you can bring your own ISO and install basically any operating system you want. Starting with pricing as low as $2.50 a month for Vultr cloud compute they have plans for developers and businesses of all sizes, except maybe Amazon, who stubbornly insists on having something to scale all on their own. Try Vultr today for free by visiting: vultr.com/screaming, and you'll receive a $100 in credit. Thats v-u-l-t-r.com slash screaming.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. This promoted episode is brought to us by our friends at ChaosSearch.We've been working with them for a long time; they've sponsored a bunch of our nonsense, and it turns out that we've been talking about them to our clients since long before they were a sponsor because it actually does what it says on the tin. Here to talk to us about that in a few minutes is Thomas Hazel, ChaosSearch's CTO and founder. First, Thomas, nice to talk to you again, and as always, thanks for humoring me.Thomas: [laugh]. Hi, Corey. Always great to talk to you. And I enjoy these conversations that sometimes go up and down, left and right, but I look forward to all the fun we're going to have.Corey: So, my understanding of ChaosSearch is probably a few years old because it turns out, I don't spend a whole lot of time meticulously studying your company's roadmap in the same way that you presumably do. When last we checked in with what the service did-slash-does, you are effectively solving the problem of data movement and querying that data. The idea behind data warehouses is generally something that's shoved onto us by cloud providers where, “Hey, this data is going to be valuable to you someday.” Data science teams are big proponents of this because when you're storing that much data, their salaries look relatively reasonable by comparison. And the ChaosSearch vision was, instead of copying all this data out of an object store and storing it on expensive disks, and replicating it, et cetera, what if we queried it in place in a somewhat intelligent manner?So, you take the data and you store it, in this case, in S3 or equivalent, and then just query it there, rather than having to move it around all over the place, which of course, then incurs data transfer fees, you're storing it multiple times, and it's never in quite the format that you want it. That was the breakthrough revelation, you were Elasticsearch—now OpenSearch—API compatible, which was great. And that was, sort of, a state of the art a year or two ago. Is that generally correct?Thomas: No, you nailed our mission statement. No, you're exactly right. You know, the value of cloud object stores, S3, the elasticity, the durability, all these wonderful things, the problem was you couldn't get any value out of it, and you had to move it out to these siloed solutions, as you indicated. So, you know, our mission was exactly that, transformed customers' cloud storage into an analytical database, a multi-model analytical database, where our first use case was search and log analytics, replacing the ELK stack and also replacing the data pipeline, the schema management, et cetera. We automate the entire step, raw data to insights.Corey: It's funny we're having this conversation today. Earlier, today, I was trying to get rid of a relatively paltry 200 gigs or so of small files on an EFS volume—you know, Amazon's version of NFS; it's like an NFS volume except you're paying Amazon for the privilege—great. And it turns out that it's a whole bunch of operations across a network on a whole bunch of tiny files, so I had to spin up other instances that were not getting backed by spot terminations, and just firing up a whole bunch of threads. So, now the load average on that box is approaching 300, but it's plowing through, getting rid of that data finally.And I'm looking at this saying this is a quarter of a terabyte. Data warehouses are in the petabyte range. Oh, I begin to see aspects of the problem. Even searching that kind of data using traditional tooling starts to break down, which is sort of the revelation that Google had 20-some-odd years ago, and other folks have since solved for, but this is the first time I've had significant data that wasn't just easily searched with a grep. For those of you in the Unix world who understand what that means, condolences. We're having a support group meeting at the bar.Thomas: Yeah. And you know, I always thought, what if you could make cloud object storage like S3 high performance and really transform it into a database? And so that warehouse capability, that's great. We like that. However to manage it, to scale it, to configure it, to get the data into that, was the problem.That was the promise of a data lake, right? This simple in, and then this arbitrary schema on read generic out. The problem next came, it became swampy, it was really hard, and that promise was not delivered. And so what we're trying to do is get all the benefits of the data lake: simple in, so many services naturally stream to cloud storage. Shoot, I would say every one of our customers are putting their data in cloud storage because their data pipeline to their warehousing solution or Elasticsearch may go down and they're worried they'll lose the data.So, what we say is what if you just said activate that data lake and get that ELK use case, get that BI use case without that data movement, as you indicated, without that ETL-ing, without that data pipeline that you're worried is going to fall over. So, that vision has been Chaos. Now, we haven't talked in, you know, a few years, but this idea that we're growing beyond what we are just going after logs, we're going into new use cases, new opportunities, and I'm looking forward to discussing with you.Corey: It's a great answer that—though I have to call out that I am right there with you as far as inappropriately using things as databases. I know that someone is going to come back and say, “Oh, S3 is a database. You're dancing around it. Isn't that what Athena is?” Which is named, of course, after the Greek Goddess of spending money on AWS? And that is a fair question, but to my understanding, there's a schema story behind that does not apply to what you're doing.Thomas: Yeah, and that is so crucial is that we like the relational access. The time-cost complexity to get it into that, as you mentioned, scaled access, I mean, it could take weeks, months to test it, to configure it, to provision it, and imagine if you got it wrong; you got to redo it again. And so our unique service removes all that data pipeline schema management. And because of our innovation because of our service, you do all schema definition, on the fly, virtually, what we call views on your index data, that you can publish an elastic index pattern for that consumption, or a relational table for that consumption. And that's kind of leading the witness into things that we're coming out with this quarter into 2022.Corey: I have to deal with a little bit of, I guess, a shame here because yeah, I'm doing exactly what you just described. I'm using Athena to wind up querying our customers' Cost and Usage Reports, and we spend a couple hundred bucks a month on AWS Glue to wind up massaging those into the way that they expect it to be. And it's great. Ish. We hook it up to Tableau and can make those queries from it, and all right, it's great.It just, burrr goes the money printer, and we somehow get access and insight to a lot of valuable data. But even that is knowing exactly what the format is going to look like. Ish. I mean, Cost and Usage Reports from Amazon are sort of aspirational when it comes to schema sometimes, but here we are. And that's been all well and good.But now the idea of log files, even looking at the base case of sending logs from an application, great. Nginx, or Apache, or [unintelligible 00:07:24], or any of the various web servers out there all tend to use different logging formats just to describe the same exact things, start spreading that across custom in-house applications and getting signal from that is almost impossible. “Oh,” people say, “So, we'll use a structured data format.” Now, you're putting log and structuring requirements on application developers who don't care in the first place, and now you have a mess on your hands.Thomas: And it really is a mess. And that challenge is, it's so problematic. And schemas changing. You know, we have customers and one reasons why they go with us is their log data is changing; they didn't expect it. Well, in your data pipeline, and your Athena database, that breaks. That brings the system down.And so our system uniquely detects that and manages that for you and then you can pick and choose how you want to export in these views dynamically. So, you know, it's really not rocket science, but the problem is, a lot of the technology that we're using is designed for static, fixed thinking. And then to scale it is problematic and time-consuming. So, you know, Glue is a great idea, but it has a lot of sharp [pebbles 00:08:26]. Athena is a great idea but also has a lot of problems.And so that data pipeline, you know, it's not for digitally native, active, new use cases, new workloads coming up hourly, daily. You think about this long-term; so a lot of that data prep pipelining is something we address so uniquely, but really where the customer cares is the value of that data, right? And so if you're spending toils trying to get the data into a database, you're not answering the questions, whether it's for security, for performance, for your business needs. That's the problem. And you know, that agility, that time-to-value is where we're very uniquely coming in because we start where your data is raw and we automate the process all the way through.Corey: So, when I look at the things that I have stuffed into S3, they generally fall into a couple of categories. There are a bunch of logs for things I never asked for nor particularly wanted, but AWS is aggressive about that, first routing through CloudTrail so you can get charged 50-cent per gigabyte ingested. Awesome. And of course, large static assets, images I have done something to enter colloquially now known as shitposts, which is great. Other than logs, what could you possibly be storing in S3 that lends itself to, effectively, the type of analysis that you built around this?Thomas: Well, our first use case was the classic log use cases, app logs, web service logs. I mean, CloudTrail, it's famous; we had customers that gave up on elastic, and definitely gave up on relational where you can do a couple changes and your permutation of attributes for CloudTrail is going to put you to your knees. And people just say, “I give up.” Same thing with Kubernetes logs. And so it's the classic—whether it's CSV, where it's JSON, where it's log types, we auto-discover all that.We also allow you, if you want to override that and change the parsing capabilities through a UI wizard, we do discover what's in your buckets. That term data swamp, and not knowing what's in your bucket, we do a facility that will index that data, actually create a report for you for knowing what's in. Now, if you have text data, if you have log data, if you have BI data, we can bring it all together, but the real pain is at the scale. So classically, app logs, system logs, many devices sending IoT-type streams is where we really come in—Kubernetes—where they're dealing with terabytes of data per day, and managing an ELK cluster at that scale. Particularly on a Black Friday.Shoot, some of our customers like—Klarna is one of them; credit card payment—they're ramping up for Black Friday, and one of the reasons why they chose us is our ability to scale when maybe you're doing a terabyte or two a day and then it goes up to twenty, twenty-five. How do you test that scale? How do you manage that scale? And so for us, the data streams are, traditionally with our customers, the well-known log types, at least in the log use cases. And the challenge is scaling it, is getting access to it, and that's where we come in.Corey: I will say the last time you were on the show a couple of years ago, you were talking about the initial logging use case and you were speaking, in many cases aspirationally, about where things were going. What a difference a couple years is made. Instead of talking about what hypothetical customers might want, or what—might be able to do, you're just able to name-drop them off the top of your head, you have scaled to approximately ten times the number of employees you had back then. You've—Thomas: Yep. Yep.Corey: —raised, I think, a total of—what, 50 million?—since then.Thomas: Uh, 60 now. Yeah.Corey: Oh, 60? Fantastic.Thomas: Yeah, yeah.Corey: Congrats. And of course, how do you do it? By sponsoring Last Week in AWS, as everyone should. I'm taking clear credit for that every time someone announces around, that's the game. But no, there is validity to it because telling fun stories and sponsoring exciting things like this only carry you so far. At some point, customers have to say, yeah, this is solving a pain that I have; I'm willing to pay you money to solve it.And you've clearly gotten to a point where you are addressing the needs of those customers at a pretty fascinating clip. It's bittersweet from my perspective because it seems like the majority of your customers have not come from my nonsense anymore. They're finding you through word of mouth, they're finding through more traditional—read as boring—ad campaigns, et cetera, et cetera. But you've built a brand that extends beyond just me. I'm no longer viewed as the de facto ombudsperson for any issue someone might have with ChaosSearch on Twitters. It's kind of, “Aww, the company grew up. What happened there?”Thomas: No, [laugh] listen, this you were great. We reached out to you to tell our story, and I got to be honest. A lot of people came by, said, “I heard something on Corey Quinn's podcasts,” or et cetera. And it came a long way now. Now, we have, you know, companies like Equifax, multi-cloud—Amazon and Google.They love the data lake philosophy, the centralized, where use cases are now available within days, not weeks and months. Whether it's logs and BI. Correlating across all those data streams, it's huge. We mentioned Klarna, [APM Performance 00:13:19], and, you know, we have Armor for SIEM, and Blackboard for [Observers 00:13:24].So, it's funny—yeah, it's funny, when I first was talking to you, I was like, “What if? What if we had this customer, that customer?” And we were building the capabilities, but now that we have it, now that we have customers, yeah, I guess, maybe we've grown up a little bit. But hey, listen to you're always near and dear to our heart because we remember, you know, when you stop[ed by our booth at re:Invent several times. And we're coming to re:Invent this year, and I believe you are as well.Corey: Oh, yeah. But people listening to this, it's if they're listening the day it's released, this will be during re:Invent. So, by all means, come by the ChaosSearch booth, and see what they have to say. For once they have people who aren't me who are going to be telling stories about these things. And it's fun. Like, I joke, it's nothing but positive here.It's interesting from where I sit seeing the parallels here. For example, we have both had—how we say—adult supervision come in. You have a CEO, Ed, who came over from IBM Storage. I have Mike Julian, whose first love language is of course spreadsheets. And it's great, on some level, realizing that, wow, this company has eclipsed my ability to manage these things myself and put my hands-on everything. And eventually, you have to start letting go. It's a weird growth stage, and it's a heck of a transition. But—Thomas: No, I love it. You know, I mean, I think when we were talking, we were maybe 15 employees. Now, we're pushing 100. We brought on Ed Walsh, who's an amazing CEO. It's funny, I told him about this idea, I invented this technology roughly eight years ago, and he's like, “I love it. Let's do it.” And I wasn't ready to do it.So, you know, five, six years ago, I started the company always knowing that, you know, I'd give him a call once we got the plane up in the air. And it's been great to have him here because the next level up, right, of execution and growth and business development and sales and marketing. So, you're exactly right. I mean, we were a young pup several years ago, when we were talking to you and, you know, we're a little bit older, a little bit wiser. But no, it's great to have Ed here. And just the leadership in general; we've grown immensely.Corey: Now, we are recording this in advance of re:Invent, so there's always the question of, “Wow, are we going to look really silly based upon what is being announced when this airs?” Because it's very hard to predict some things that AWS does. And let's be clear, I always stay away from predictions, just because first, I have a bit of a knack for being right. But also, when I'm right, people will think, “Oh, Corey must have known about that and is leaking,” whereas if I get it wrong, I just look like a fool. There's no win for me if I start doing the predictive dance on stuff like that.But I have to level with you, I have been somewhat surprised that, at least as of this recording, AWS has not moved more in your direction because storing data in S3 is kind of their whole thing, and querying that data through something that isn't Athena has been a bit of a reach for them that they're slowly starting to wrap their heads around. But their UltraWarm nonsense—which is just, okay, great naming there—what is the point of continually having a model where oh, yeah, we're going to just age it out, the stuff that isn't actively being used into S3, rather than coming up with a way to query it there. Because you've done exactly that, and please don't take this as anything other than a statement of fact, they have better access to what S3 is doing than you do. You're forced to deal with this thing entirely from a public API standpoint, which is fine. They can theoretically change the behavior of aspects of S3 to unlock these use cases if they chose to do so. And they haven't. Why is it that you're the only folks that are doing this?Thomas: No, it's a great question, and I'll give them props for continuing to push the data lake [unintelligible 00:17:09] to the cloud providers' S3 because it was really where I saw the world. Lakes, I believe in. I love them. They love them. However, they promote the move the data out to get access, and it seems so counterintuitive on why wouldn't you leave it in and put these services, make them more intelligent? So, it's funny, I've trademark ‘Smart Object Storage,' I actually trademarked—I think you [laugh] were a part of this—‘UltraHot,' right? Because why would you want UltraWarm when you can have UltraHot?And the reason, I feel, is that if you're using Parquet for Athena [unintelligible 00:17:40] store, or Lucene for Elasticsearch, these two index technologies were not designed for cloud storage, for real-time streaming off of cloud storage. So, the trick is, you have to build UltraWarm, get it off of what they consider cold S3 into a more warmer memory or SSD type access. What we did, what the invention I created was, that first read is hot. That first read is fast.Snowflake is a good example. They give you a ten terabyte demo example, and if you have a big instance and you do that first query, maybe several orders or groups, it could take an hour to warm up. The second query is fast. Well, what if the first query is in seconds as well? And that's where we really spent the last five, six years building out the tech and the vision behind this because I like to say you go to a doctor and say, “Hey, Doc, every single time I move my arm, it hurts.” And the doctor says, “Well, don't move your arm.”It's things like that, to your point, it's like, why wouldn't they? I would argue, one, you have to believe it's possible—we're proving that it is—and two, you have to have the technology to do it. Not just the index, but the architecture. So, I believe they will go this direction. You know, little birdies always say that all these companies understand this need.Shoot, Snowflake is trying to be lake-y; Databricks is trying to really bring this warehouse lake concept. But you still do all the pipelining; you still have to do all the data management the way that you don't want to do. It's not a lake. And so my argument is that it's innovation on why. Now, they have money; they have time, but, you know, we have a big head start.Corey: I remembered last year at re:Invent they released a, shall we say, significant change to S3 that it enabled read after write consistency, which is awesome, for again, those of us in the business of misusing things as databases. But for some folks, the majority of folks I would say, it was a, “I don't know what that means and therefore I don't care.” And that's fine. I have no issue with that. There are other folks, some of my customers for example, who are suddenly, “Wait a minute. This means I can sunset this entire janky sidecar metadata system that is designed to make sure that we are consistent in our use of S3 because it now does it automatically under the hood?” And that's awesome. Does that change mean anything for ChaosSearch?Thomas: It doesn't because of our architecture. We're append-only, write-once scenario, so a lot of update-in-place viewpoints. My viewpoint is that if you're seeing S3 as the database and you need that type of consistency, it make sense of why you'd want it, but because of our distributive fabric, our stateless architecture, our append-only nature, it really doesn't affect us.Now, I talked to the S3 team, I said, “Please if you're coming up with this feature, it better not be slower.” I want S3 to be fast, right? And they said, “No, no. It won't affect performance.” I'm like, “Okay. Let's keep that up.”And so to us, any type of S3 capability, we'll take advantage of it if benefits us, whether it's consistency as you indicated, performance, functionality. But we really keep the constructs of S3 access to really limited features: list, put, get. [roll-on 00:20:49] policies to give us read-only access to your data, and a location to write our indices into your account, and then are distributed fabric, our service, acts as those indices and query them or searches them to resolve whatever analytics you need. So, we made it pretty simple, and that is allowed us to make it high performance.Corey: I'll take it a step further because you want to talk about changes since the last time we spoke, it used to be that this was on top of S3, you can store your data anywhere you want, as long as it's S3 in the customer's account. Now, you're also supporting one-click integration with Google Cloud's object storage, which, great. That does mean though, that you're not dependent upon provider-specific implementations of things like a consistency model for how you've built things. It really does use the lowest common denominator—to my understanding—of object stores. Is that something that you're seeing broad adoption of, or is this one of those areas where, well, you have one customer on a different provider, but almost everything lives on the primary? I'm curious what you're seeing for adoption models across multiple providers?Thomas: It's a great question. We built an architecture purposely to be cloud-agnostic. I mean, we use compute in a containerized way, we use object storage in a very simple construct—put, get, list—and we went over to Google because that made sense, right? We have customers on both sides. I would say Amazon is the gorilla, but Google's trying to get there and growing.We had a big customer, Equifax, that's on both Amazon and Google, but we offer the same service. To be frank, it looks like the exact same product. And it should, right? Whether it's Amazon Cloud, or Google Cloud, multi-select and I want to choose either one and get the other one. I would say that different business types are using each one, but our bulk of the business isn't Amazon, but we just this summer released our SaaS offerings, so it's growing.And you know, it's funny, you never know where it comes from. So, we have one customer—actually DigitalRiver—as one of our customers on Amazon for logs, but we're growing in working together to do a BI on GCP or on Google. And so it's kind of funny; they have two departments on two different clouds with two different use cases. And so do they want unification? I'm not sure, but they definitely have their BI on Google and their operations in Amazon. It's interesting.Corey: You know its important to me that people learn how to use the cloud effectively. Thats why I'm so glad that Cloud Academy is sponsoring my ridiculous non-sense. They're a great way to build in demand tech skills the way that, well personally, I learn best which I learn by doing not by reading. They have live cloud labs that you can run in real environments that aren't going to blow up your own bill—I can't stress how important that is. Visit cloudacademy.com/corey. Thats C-O-R-E-Y, don't drop the “E.” Use Corey as a promo-code as well. You're going to get a bunch of discounts on it with a lifetime deal—the price will not go up. It is limited time, they assured me this is not one of those things that is going to wind up being a rug pull scenario, oh no no. Talk to them, tell me what you think. Visit: cloudacademy.com/corey,  C-O-R-E-Y and tell them that I sent you!Corey: I know that I'm going to get letters for this. So, let me just call it out right now. Because I've been a big advocate of pick a provider—I care not which one—and go all-in on it. And I'm sitting here congratulating you on extending to another provider, and people are going to say, “Ah, you're being inconsistent.”No. I'm suggesting that you as a provider have to meet your customers where they are because if someone is sitting in GCP and your entire approach is, “Step one, migrate those four petabytes of data right on over here to AWS,” they're going to call you that jackhole that you would be by making that suggestion and go immediately for option B, which is literally anything that is not ChaosSearch, just based upon that core misunderstanding of their business constraints. That is the way to think about these things. For a vendor position that you are in as an ISV—Independent Software Vendor for those not up on the lingo of this ridiculous industry—you have to meet customers where they are. And it's the right move.Thomas: Well, you just said it. Imagine moving terabytes and petabytes of data.Corey: It sounds terrific if I'm a salesperson for one of these companies working on commission, but for the rest of us, it sounds awful.Thomas: We really are a data fabric across clouds, within clouds. We're going to go where the data is and we're going to provide access to where that data lives. Our whole philosophy is the no-movement movement, right? Don't move your data. Leave it where it is and provide access at scale.And so you may have services in Google that naturally stream to GCS; let's do it there. Imagine moving that amount of data over to Amazon to analyze it, and vice versa. 2020, we're going to be in Azure. They're a totally different type of business, users, and personas, but you're getting asked, “Can you support Azure?” And the answer is, “Yes,” and, “We will in 2022.”So, to us, if you have cloud storage, if you have compute, and it's a big enough business opportunity in the market, we're there. We're going there. When we first started, we were talking to MinIO—remember that open-source, object storage platform?—We've run on our laptops, we run—this [unintelligible 00:25:04] Dr. Seuss thing—“We run over here; we run over there; we run everywhere.”But the honest truth is, you're going to go with the big cloud providers where the business opportunity is, and offer the same solution because the same solution is valued everywhere: simple in; value out; cost-effective; long retention; flexibility. That sounds so basic, but you mentioned this all the time with our Rube Goldberg, Amazon diagrams we see time and time again. It's like, if you looked at that and you were from an alien planet, you'd be like, “These people don't know what they're doing. Why is it so complicated?” And the simple answer is, I don't know why people think it's complicated.To your point about Amazon, why won't they do it? I don't know, but if they did, things would be different. And being honest, I think people are catching on. We do talk to Amazon and others. They see the need, but they also have to build it; they have to invent technology to address it. And using Parquet and Lucene are not the answer.Corey: Yeah, it's too much of a demand on the producers of that data rather than the consumer. And yeah, I would love to be able to go upstream to application developers and demand they do things in certain ways. It turns out as a consultant, you have zero authority to do that. As a DevOps team member, you have limited ability to influence it, but it turns out that being the ‘department of no' quickly turns into being the ‘department of unemployment insurance' because no one wants to work with you. And collaboration—contrary to what people wish to believe—is a key part of working in a modern workplace.Thomas: Absolutely. And it's funny, the demands of IT are getting harder; the actual getting the employees to build out the solutions are getting harder. And so a lot of that time is in the pipeline, is the prep, is the schema, the sharding, and et cetera, et cetera, et cetera. My viewpoint is that should be automated away. More and more databases are being autotune, right?This whole knobs and this and that, to me, Glue is a means to an end. I mean, let's get rid of it. Why can't Athena know what to do? Why can't object storage be Athena and vice versa? I mean, to me, it seems like all this moving through all these services, the classic Amazon viewpoint, even their diagrams of having this centralized repository of S3, move it all out to your services, get results, put it back in, then take it back out again, move it around, it just doesn't make much sense. And so to us, I love S3, love the service. I think it's brilliant—Amazon's first service, right?—but from there get a little smarter. That's where ChaosSearch comes in.Corey: I would argue that S3 is in fact, a modern miracle. And one of those companies saying, “Oh, we have an object store; it's S3 compatible.” It's like, “Yeah. We have S3 at home.” Look at S3 at home, and it's just basically a series of failing Raspberry Pis.But you have this whole ecosystem of things that have built up and sprung up around S3. It is wildly understated just how scalable and massive it is. There was an academic paper recently that won an award on how they use automated reasoning to validate what is going on in the S3 environment, and they talked about hundreds of petabytes in some cases. And folks are saying, ah, S3 is hundreds of petabytes. Yeah, I have clients storing hundreds of petabytes.There are larger companies out there. Steve Schmidt, Amazon's CISO, was recently at a Splunk keynote where he mentioned that in security info alone, AWS itself generates 500 petabytes a day that then gets reduced down to a bunch of stuff, and some of it gets loaded into Splunk. I think. I couldn't really hear the second half of that sentence because of the sound of all of the Splunk salespeople in that room becoming excited so quickly you could hear it.Thomas: [laugh]. I love it. If I could be so bold, those S3 team, they're gods. They are amazing. They created such an amazing service, and when I started playing with S3 now, I guess, 2006 or 7, I mean, we were using for a repository, URL access to get images, I was doing a virtualization [unintelligible 00:29:05] at the time—Corey: Oh, the first time I played with it, “This seems ridiculous and kind of dumb. Why would anyone use this?” Yeah, yeah. It turns out I'm really bad at predicting the future. Another reason I don't do the prediction thing.Thomas: Yeah. And when I started this company officially, five, six years ago, I was thinking about S3 and I was thinking about HDFS not being a good answer. And I said, “I think S3 will actually achieve the goals and performance we need.” It's a distributed file system. You can run parallel puts and parallel gets. And the performance that I was seeing when the data was a certain way, certain size, “Wait, you can get high performance.”And you know, when I first turned on the engine, now four or five years ago, I was like, “Wow. This is going to work. We're off to the races.” And now obviously, we're more than just an idea when we first talked to you. We're a service.We deliver benefits to our customers both in logs. And shoot, this quarter alone we're coming out with new features not just in the logs, which I'll talk about second, but in a direct SQL access. But you know, one thing that you hear time and time again, we talked about it—JSON, CloudTrail, and Kubernetes; this is a real nightmare, and so one thing that we've come out with this quarter is the ability to virtually flatten. Now, you heard time and time again, where, “Okay. I'm going to pick and choose my data because my database can't handle whether it's elastic, or say, relational.” And all of a sudden, “Shoot, I don't have that. I got to reindex that.”And so what we've done is we've created a index technology that we're always planning to come out with that indexes the JSON raw blob, but in the data refinery have, post-index you can select how to unflatten it. Why is that important? Because all that tooling, whether it's elastic or SQL, is now available. You don't have to change anything. Why is Snowflake and BigQuery has these proprietary JSON APIs that none of these tools know how to use to get access to the data?Or you pick and choose. And so when you have a CloudTrail, and you need to know what's going on, if you picked wrong, you're in trouble. So, this new feature we're calling ‘Virtual Flattening'—or I don't know what we're—we have to work with the marketing team on it. And we're also bringing—this is where I get kind of excited where the elastic world, the ELK world, we're bringing correlations into Elasticsearch. And like, how do you do that? They don't have the APIs?Well, our data refinery, again, has the ability to correlate index patterns into one view. A view is an index pattern, so all those same constructs that you had in Kibana, or Grafana, or Elastic API still work. And so, no more denormalizing, no more trying to hodgepodge query over here, query over there. You're actually going to have correlations in Elastic, natively. And we're excited about that.And one more push on the future, Q4 into 2022; we have been given early access to S3 SQL access. And, you know, as I mentioned, correlations in Elastic, but we're going full in on publishing our [TPCH 00:31:56] report, we're excited about publishing those numbers, as well as not just giving early access, but going GA in the first of the year, next year.Corey: I look forward to it. This is also, I guess, it's impossible to have a conversation with you, even now, where you're not still forward-looking about what comes next. Which is natural; that is how we get excited about the things that we're building. But so much less of what you're doing now in our conversations have focused around what's coming, as opposed to the neat stuff you're already doing. I had to double-check when we were talking just now about oh, yeah, is that Google cloud object store support still something that is roadmapped, or is that out in the real world?No, it's very much here in the real world, available today. You can use it. Go click the button, have fun. It's neat to see at least some evidence that not all roadmaps are wishes and pixie dust. The things that you were talking to me about years ago are established parts of ChaosSearch now. It hasn't been just, sort of, frozen in amber for years, or months, or these giant periods of time. Because, again, there's—yeah, don't sell me vaporware; I know how this works. The things you have promised have come to fruition. It's nice to see that.Thomas: No, I appreciate it. We talked a little while ago, now a few years ago, and it was a bit of aspirational, right? We had a lot to do, we had more to do. But now when we have big customers using our product, solving their problems, whether it's security, performance, operation, again—at scale, right? The real pain is, sure you have a small ELK cluster or small Athena use case, but when you're dealing with terabytes to petabytes, trillions of rows, right—billions—when you were dealing trillions, billions are now small. Millions don't even exist, right?And you're graduating from computer science in college and you say the word, “Trillion,” they're like, “Nah. No one does that.” And like you were saying, people do petabytes and exabytes. That's the world we're living in, and that's something that we really went hard at because these are challenging data problems and this is where we feel we uniquely sit. And again, we don't have to break the bank while doing it.Corey: Oh, yeah. Or at least as of this recording, there's a meme going around, again, from an old internal Google Video, of, “I just want to serve five terabytes of traffic,” and it's an internal Google discussion of, “I don't know how to count that low.” And, yeah.Thomas: [laugh].Corey: But there's also value in being able to address things at much larger volume. I would love to see better responsiveness options around things like Deep Archive because the idea of being able to query that—even if you can wait a day or two—becomes really interesting just from the perspective of, at that point, current cost for one petabyte of data in Glacier Deep Archive is 1000 bucks a month. That is ‘why would I ever delete data again?' Pricing.Thomas: Yeah. You said it. And what's interesting about our technology is unlike, let's say Lucene, when you index it, it could be 3, 4, or 5x the raw size, our representation is smaller than gzip. So, it is a full representation, so why don't you store it efficiently long-term in S3? Oh, by the way, with the Glacier; we support Glacier too.And so, I mean, it's amazing the cost of data with cloud storage is dramatic, and if you can make it hot and activated, that's the real promise of a data lake. And, you know, it's funny, we use our own service to run our SaaS—we log our own data, we monitor, we alert, have dashboards—and I can't tell you how cheap our service is to ourselves, right? Because it's so cost-effective for long-tail, not just, oh, a few weeks; we store a whole year's worth of our operational data so we can go back in time to debug something or figure something out. And a lot of that's savings. Actually, huge savings is cloud storage with a distributed elastic compute fabric that is serverless. These are things that seem so obvious now, but if you have SSDs, and you're moving things around, you know, a team of IT professionals trying to manage it, it's not cheap.Corey: Oh, yeah, that's the story. It's like, “Step one, start paying for using things in cloud.” “Okay, great. When do I stop paying?” “That's the neat part. You don't.” And it continues to grow and build.And again, this is the thing I learned running a business that focuses on this, the people working on this, in almost every case, are more expensive than the infrastructure they're working on. And that's fine. I'd rather pay people than technologies. And it does help reaffirm, on some level, that—people don't like this reminder—but you have to generate more value than you cost. So, when you're sitting there spending all your time trying to avoid saving money on, “Oh, I've listened to ChaosSearch talk about what they do a few times. I can probably build my own and roll it at home.”It's, I've seen the kind of work that you folks have put into this—again, you have something like 100 employees now; it is not just you building this—my belief has always been that if you can buy something that gets you 90, 95% of where you are, great. Buy it, and then yell at whoever selling it to you for the rest of it, and that'll get you a lot further than, “We're going to do this ourselves from first principles.” Which is great for a weekend project for just something that you have a passion for, but in production mistakes show. I've always been a big proponent of buying wherever you can. It's cheaper, which sounds weird, but it's true.Thomas: And we do the same thing. We have single-sign-on support; we didn't build that ourselves, we use a service now. Auth0 is one of our providers now that owns that [crosstalk 00:37:12]—Corey: Oh, you didn't roll your own authentication layer? Why ever not? Next, you're going to tell me that you didn't roll your own payment gateway when you wound up charging people on your website to sign up?Thomas: You got it. And so, I mean, do what you do well. Focus on what you do well. If you're repeating what everyone seems to do over and over again, time, costs, complexity, and… service, it makes sense. You know, I'm not trying to build storage; I'm using storage. I'm using a great, wonderful service, cloud object storage.Use whats works, whats works well, and do what you do well. And what we do well is make cloud object storage analytical and fast. So, call us up and we'll take away that 2 a.m. call you have when your cluster falls down, or you have a new workload that you are going to go to the—I don't know, the beach house, and now the weekend shot, right? Spin it up, stream it in. We'll take over.Corey: Yeah. So, if you're listening to this and you happen to be at re:Invent, which is sort of an open question: why would you be at re:Invent while listening to a podcast? And then I remember how long the shuttle lines are likely to be, and yeah. So, if you're at re:Invent, make it on down to the show floor, visit the ChaosSearch booth, tell them I sent you, watch for the wince, that's always worth doing. Thomas, if people have better decision-making capability than the two of us do, where can they find you if they're not in Las Vegas this week?Thomas: So, you find us online chaossearch.io. We have so much material, videos, use cases, testimonials. You can reach out to us, get a free trial. We have a self-service experience where connect to your S3 bucket and you're up and running within five minutes.So, definitely chaossearch.io. Reach out if you want a hand-held, white-glove experience POV. If you have those type of needs, we can do that with you as well. But we booth on re:Invent and I don't know the booth number, but I'm sure either we've assigned it or we'll find it out.Corey: Don't worry. This year, it is a low enough attendance rate that I'm projecting that you will not be as hard to find in recent years. For example, there's only one expo hall this year. What a concept. If only it hadn't taken a deadly pandemic to get us here.Thomas: Yeah. But you know, we'll have the ability to demonstrate Chaos at the booth, and really, within a few minutes, you'll say, “Wow. How come I never heard of doing it this way?” Because it just makes so much sense on why you do it this way versus the merry-go-round of data movement, and transformation, and schema management, let alone all the sharding that I know is a nightmare, more often than not.Corey: And we'll, of course, put links to that in the [show notes 00:39:40]. Thomas, thank you so much for taking the time to speak with me today. As always, it's appreciated.Thomas: Corey, thank you. Let's do this again.Corey: We absolutely will. Thomas Hazel, CTO and Founder of ChaosSearch. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast episode, please leave a five-star review on your podcast platform of choice, whereas if you've hated this episode, please leave a five-star review on your podcast platform of choice along with an angry comment because I have dared to besmirch the honor of your homebrewed object store, running on top of some trusty and reliable Raspberries Pie.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

This week on the podcast, Kyle and Dan talk about the new Rundeck Cloud product, working with the DDDAudit report, preventing App Designer from stalling at 30%, and updating the ACM with SQL. Show Notes Rundeck Cloud @ 4:15 Rundeck and OCI MySQL Service DDDAudit Table Output @ 15:00 App Designer Stalling at 30% @ 18:45 Working with ACM via SQL @ 25:30

This is episode 213, part 2 of 2, recorded on November 16th, 2021 where John & Jason go over the November 2021 Power BI updates including the new connectors for Azure Synapse Analytics and Google Sheets, Datasets hub improvements, and Support embedding a Power BI report that contains a paginated report visual. For show notes please visit www.bifocal.show

In this episode Martin and one of the Grumpies (as in Grumpy Old Coders) battle it out: SQL or NoSQL - which technology is better? If you ever wondered why the Structured Query Language was invented in the first place and why the hipster abandoned ship for the latest (?) rage of the likes of the NoSQL variety, this is for you. Plus: A whole family of never-heard-of sound effects make their debut on this bumper of an episode. Links: SQL: https://www.iso.org/standard/63555.html NoSQL: https://en.wikipedia.org/wiki/NoSQL NoSQL Geek: http://www.nosqlgeek.org ACID compliance: https://en.wikipedia.org/wiki/ACID redis: https://github.com/redis/redis CAP theorem: https://en.wikipedia.org/wiki/CAP_theorem TorroDB: https://github.com/gordol/torrodb-server Grumpy Old Coders episode on the Dark Side: https://soundcloud.com/user-498377588/grumpy-old-coders-ep-11-the-dark-side

A first look at SQL Server 2022—the latest Azure-enabled database and data integration innovations. See what it means for your hybrid workloads, including first-time bi-directional high availability and disaster recovery between Azure SQL Managed Instance and SQL Server, Azure Synapse Link integration with SQL for ETL free near real-time reporting and analytics over your operational data, and new next-generation built-in query intelligence with parameter sensitive plan optimization. Bob Ward, SQL engineering leader, joins Jeremy Chapman to share the focus on this round of updates. ► QUICK LINKS: 00:00 - Introduction 00:38 - Overview of updates 02:19 - Disaster recovery 04:26 - Failover and restore example 06:16 - Azure Synapse integration 09:04 - Built-in query intelligence 10:19 - See it in action 12:52 - Wrap up ► Link References: Learn more about SQL Server 2022 at https://aka.ms/SQLServer2022 Apply to join our private preview, and try it out at https://aka.ms/EAPSignUp ► Unfamiliar with Microsoft Mechanics? We are Microsoft's official video series for IT. You can watch and share valuable content and demos of current and upcoming tech from the people who build it at Microsoft. Subscribe to our YouTube: https://www.youtube.com/c/MicrosoftMechanicsSeries?sub_confirmation=1 Join us on the Microsoft Tech Community: https://techcommunity.microsoft.com/t5/microsoft-mechanics-blog/bg-p/MicrosoftMechanicsBlog Watch or listen via podcast here: https://microsoftmechanics.libsyn.com/website ► Keep getting this insider knowledge, join us on social: Follow us on Twitter: https://twitter.com/MSFTMechanics Follow us on LinkedIn: https://www.linkedin.com/company/microsoft-mechanics/ 

Vlado Cingel recounts his story where he needed common table expressions within SQL for a project he was working on and wrote a patch to AREL and ActiveRecord which he submitted to the Rails Core. Since it hasn't been accepted, he's supporting it as a gem. Vlado explains what Common Table Expressions (CTEs) are, how they work, and where they're used. Panel John EppersonLuke StuttersValentino Stoll Guest Vlado Cingel Sponsors Top End DevsRaygun | Click here to get started on your free 14-day trialCoaching | Top End Devs Links GitHub | vlado/activerecord-cteOrganising complex SQL queries in RailsGitHub: Vlado Cingel ( vlado ) Picks John- Digital Storm: Custom Gaming Computers & Gaming PCsLuke- Pitch PerfectValentino- Ruby TogetherValentino- Ruby CentralValentino- The Ruby VM a speedrun - Penelope PhippenVlado- The Wood Whisperer Guild - Online Woodworking SchoolVlado- Polished Ruby Programming Contact John: Rock Agile ConsultingGitHub: John Epperson ( kirillian )LinkedIn: John Epperson Contact Luke: GitHub: Luke Stutters ( lukestuts ) Contact Valentino: Doximity Technology BlogWork @ DoximityGitHub: Valentino Stoll ( codenamev )Twitter: V ( @thecodenamev ) Special Guest: Vlado Cingel.

Watch the live stream: Watch on YouTube About the show Sponsored by Shortcut - Get started at shortcut.com/pythonbytes Special guest: Renee Teate Michael #1: pypi-changes via Brian Skinn, created by Bernát Gábor Visually show you which dependencies in an environment are out of date. See the age of everything you depend upon. Also, shoutout again to pipdeptree Brian #2: Late-bound argument defaults for Python Default values for arguments to functions are evaluated at function definition time. If a value is a short expression that uses a variable, that variable is in the scope of the function definition. The expression cannot use other arguments. Example of what you cannot do: def foo(a, b = None, c = len(a)): ... There's a proposal by Chris Angelico to add a =: operator for late default evaluation. syntax still up in the air. => and ?= also discussed However, it's non-trivial to add syntax to an established language, and this article notes: At first blush, Angelico's idea to fix this "wart" in Python seems fairly straightforward, but the discussion has shown that there are multiple facets to consider. It is not quite as simple as "let's add a way to evaluate default arguments when the function is called"—likely how it was seen at the outset. That is often the case when looking at new features for an established language like Python; there is a huge body of code that needs to stay working, but there are also, sometimes conflicting, aspirations for features that could be added. It is a tricky balancing act. Renee #3: pandas.read_sql Since I wrote my book SQL for Data Scientists, I've gotten several questions about how I use SQL in my python scripts. It's really simple: You can save your SQL as a text file and then import the dataset into a pandas dataframe to do the rest of my data cleaning, feature engineering, etc. Pandas has a built-in way to use SQL as a data source. You set up a connection to your database using another package like SQL Alchemy, then send the SQL string and the connection to the pandas.read_sql function. It returns a dataframe with the results of your query. Michael #4: pyjion by Anthony Shaw Pyjion is a JIT for Python based upon CoreCLR Check out live.trypyjion.com *to see it in action.* Requires Python 3.10, .NET Core 6 Enable with just a couple of lines: >>> import pyjion >>> pyjion.enable() Brian #5: Tips for debugging with print() Adam Johnson 7 tips altogether, but I'll highlight a few I loved reading about Debug variables with f-strings and = print(f``"``{myvar=}``") Saves typing over print(f``"``myvar={myvar}") with the same result Make output “pop” with emoji (Brilliant!) print("

This is episode 212 recorded on November 16th, 2021 where John & Jason go over the November 2021 Power BI updates including the New Format Pane, Page & Bookmark Navigators, Sort Legend, and the new Visual Scorecard from Power BI Goals. For show notes please visit www.bifocal.show

2021-11-16 Weekly News - Episode 126Watch the video version on YouTube at https://youtu.be/83taKaR58xs Hosts: Eric Peterson - Senior Developer for Ortus SolutionsThanks to our Sponsor - Ortus SolutionsThe makers of ColdBox, CommandBox, ForgeBox, TestBox and almost every other Box out there. A few ways  to say thanks back to Ortus Solutions: Like and subscribe to our videos on YouTube.  Subscribe to our Podcast on your Podcast Apps and leave us a review Sign up for a free or paid account on CFCasts, which is releasing new content every week Buy Ortus's new Book - 102 ColdBox HMVC Quick Tips and Tricks on GumRoad (http://gum.co/coldbox-tips) Patreon SupportWe have 38 patreons providing 98% of the funding for our Modernize or Die Podcasts via our Patreon site: https://www.patreon.com/ortussolutions. News and EventsOrtus Webinar for November - Javier Quintero - FORGEBOX Business Plan: Introducing Organizations and TeamsNovember 19th at 11:00 AM Central Time (US and Canada)In this webinar, Javier Quintero, lead developer of FORGEBOX, will present the new features and the improved UI that is now available on FORGEBOX 6. Moreover, he'll explore in depth the Business Plan that is directed towards organizations and teams so they can collaborate and support their software building needs. He will show us how to create a new organization, how you can add members to it with specific roles, and how you can control teams, members, packages and publish access.with Javier Quinterohttps://us02web.zoom.us/meeting/register/tZclfuGopjkiG9TIMoC93YbKIcLM1ok_KKlw ICYMI - Mid Michigan CFUG Meeting - Using AI and machine learning along with ColdFusion to build a smarter call center with Nick KwiatkowskiTuesday 11/9/21 at 7 pm easternUsing AI and machine learning along with ColdFusion to build a smarter call center at the next Mid-Michigan CFUG meeting Tuesday 11/9/21 at 7 pm eastern.  Michigan State University's, Nick Kwiatkowski, will be showing how to create voice and text-based chat bots that you can deploy to your contact centers (and help desks!) to help automate frequently asked questions.Recording - check Facebook groupICYMI - Online CF Meetup - "Avoiding Server-Side Request Forgery (SSRF) Vulns in CFML", with Brian ReillyThursday, November 11, 2021 - 9:00 AM to 10:00 AM PSTServer-Side Request Forgery (SSRF) vulnerabilities allow an attacker to make arbitrary web requests (and in some cases, other protocols too) from the application environment. Exploiting these flaws can lead to leaking sensitive data, accessing internal resources, and under certain circumstances, remote command execution.Several ColdFusion/CFML tags and functions can process URLs as file path arguments -- including some tags and and functions that you might not expect. If these tags and functions process unvalidated user-controlled input, this can lead to SSRF vulnerabilities in your applications. In addition to providing a list of affected tags and functions, I'll cover some approaches for identifying and remediating vulnerable code. My goal for this talk is to raise awareness about what may be a security blindspot for some ColdFusion/CFML developers.https://www.meetup.com/coldfusionmeetup/events/281850930/ Recording: https://www.youtube.com/watch?v=-wu6cRZcRx0 CFCasts Content Updateshttps://www.cfcasts.com Just ReleasedSoapBox - ColdBox Anniversary Edition with Brad WoodComing this weekYouth Trainings - Universidad Don BoscoA new series of ForgeBox coming very soonSend your suggestions at https://cfcasts.com/supportConferences and TrainingDeploy by Digital Ocean - THIS WEEKTHE VIRTUAL CONFERENCE FOR GLOBAL DEVELOPMENT TEAMSNovember 16-17, 2021 https://deploy.digitalocean.com/homeAWS re:InventNOV. 29 – DEC. 3, 2021 | LAS VEGAS, NVCELEBRATING 10 YEARS OF RE:INVENTVirtual: FreeIn Person: $1799https://reinvent.awsevents.com/ Postgres BuildOnline - FreeNov 30-Dec 1 2021https://www.postgresbuild.com/ ITB Latam 2021December 2-3, 2021Into the Box LATAM is back and better than ever! Our virtual conference will include speakers from El Salvador and all over the world, who'll present on the latest web and mobile technologies in Latin America.Registration is completely free so don't miss out!ITB Latam Schedule Postedhttps://latam.intothebox.org/ Adobe ColdFusion Summit 2021December 7th and 8th - VirtualAgenda is out!!!@Adobe @coldfusion #CFSummit2021 keynote we will be featuring @ashleymcnamara! Her talk will focus on the history & future of DevRel how we got here & where we're going.2 tracks - 1 all CFML - the other a mix of CFML and semi-related topicsRegister for Free - https://cfsummit.vconfex.com/site/adobe-cold-fusion-summit-2021/1290Blog - https://coldfusion.adobe.com/2021/09/adobe-coldfusion-summit-2021-registrations-open/ jConf.devNow a free virtual eventDecember 9th starting at 8:30 am CDT/2:30 pm UTC.https://2021.jconf.dev/?mc_cid=b62adc151d&mc_eid=8293d6fdb0 VueJS Nation ConferenceOnline Live EventJanuary 26th & 27th 2022Register for FreeCall for Speakers is open until Dec 31 2021https://vuejsnation.com/ More conferencesNeed more conferences, this site has a huge list of conferences for almost any language/community.https://confs.tech/Blogs, Tweets and Videos of the WeekBlog - Charlie Arehart - Should you “bother” to file bug reports at tracker.adobe.com? Yes you shouldI just wanted to offer a quick plug to get folks to please consider filing bugs (and feature requests) at the Adobe site for tracking them, https://tracker.adobe.com. I've blogged before about how it can be used for more than most may realize. What I want to share here is that it's not a “waste of time to bother”.Some may wonder first, “why is is worth pointing out Tracker? Doesn't everyone know about it?” The answer to the second question is “no”: many do NOT know about it. But the more important question may be the first, and it's the real reason I'm writing this post.https://coldfusion.adobe.com/2021/11/should-you-bother-to-file-bug-reports/ Blog - Ben Nadel - Phill Nacelli's SQL Tip Is Making My CFQuery Upgrades In Adobe ColdFusion 2021 EasyAs I've started to modernize my blogging platform for Adobe ColdFusion 2021, one of the things that I was dreading was the lack of Lucee CFML's Tag Islands. Tag Islands have really been a game changer for me, allowing me to seamlessly execute the CFQuery tag inside CFScript. I was afraid that I was going to have to keep using Tag-based syntax for my Gateway / Data Access components. But then, I remembered a hot tip from Phill Nacelli on giving dynamic SQL statements a consistent structure. It turns out, Phill's technique is making it bearable for me to use the queryExecute() Function in lieu of the CFQuery inside a Tag Island.https://www.bennadel.com/blog/4153-phill-nacellis-sql-tip-is-making-my-cfquery-upgrades-in-adobe-coldfusion-2021-easy.htmBlog - Ben Nadel - A Query Object Maintains Its CurrentRow When Passed Out-Of-Context In Adobe ColdFusion 2021As I'm attempting to modernize my blogging platform for Adobe ColdFusion 2021, I'm moving a lot of my old-school, inline CFQuery tags into various "Service" and "Data Access" ColdFusion components where they can be reused across multiple templates. And, as much as I love the ColdFusion Query object, my "service boundaries" deals with Arrays and Structs, not queries. As such, I have code that deals with mapping queries onto other normalized data structures. While writing this code, I was tickled by the fact that the Query object maintains its .currentRow property even when passed out-of-context. This .currentRow can then be used a default argument value in Function signatures. This is a really old behavior of ColdFusion; but, I thought it would be fun to demonstrate since it may not be a feature people consider very often.https://www.bennadel.com/blog/4152-a-query-object-maintains-its-currentrow-when-passed-out-of-context-in-adobe-coldfusion-2021.htm CFML JobsSeveral positions available on https://www.getcfmljobs.com/Listing over 233 ColdFusion positions from 103 companies across 123 locations in 5 Countries.6 new jobs listedFull-Time - Senior Coldfusion Developer |LATAM| at Colon, PA - United States Posted Nov 15https://www.getcfmljobs.com/jobs/index.cfm/united-states/Senior-Coldfusion-Developer-LATAM-at-Colon-PA/11381Full-Time - ColdFusion Developer | 4 to 6 years | Pune at Pune, Maharash.. - India Posted Nov 12https://www.getcfmljobs.com/jobs/index.cfm/india/ColdFusion-Developer-4-to-6-years-Pune-at-Pune-Maharashtra/11380Full-Time - Senior Coldfusion Developer (RQ02208) at Toronto, ON - Canada Posted Nov 11https://www.getcfmljobs.com/jobs/index.cfm/canada/Senior-Coldfusion-Developer-RQ02208-at-Toronto-ON/11379Full-Time - Programmer (Coldfusion Java - Remote) at United States - United States Posted Nov 11https://www.getcfmljobs.com/jobs/index.cfm/united-states/Programmer-Coldfusion-Java-Remote-at-United-States/11378Full-Time - Front End / Coldfusion Developer - Salford Quays + WFH at Sa.. - United Kingdom Posted Nov 10https://www.getcfmljobs.com/jobs/index.cfm/united-kingdom/Front-End-Coldfusion-Developer-Salford-Quays-WFH-at-Salford/11377Full-Time - ColdFusion Jr. Web Developer at Pune, Maharashtra - India Posted Nov 09https://www.getcfmljobs.com/jobs/index.cfm/india/ColdFusion-Jr-Web-Developer-at-Pune-Maharashtra/11376ForgeBox Module of the WeekGlobberBy Brad Wood and Ortus SolutionsA utility module to match file system path patterns (globbing) in a similar manner as Unix file systems or .gitignore syntax.box install globberLast Update: August 10, 2021 - 3.0.7https://forgebox.io/view/globberVS Code Hint Tips and Tricks of the WeekEncode DecodeThe Encode/Decode (ecdc) extension allows you to quickly convert one or more selections of text to and from various formatsThe extension provides a single command to the command palette. To active the command simply launch the command palette (Shift-CMD-P on OSX or Shift-Ctrl-P on Windows and Linux), then just type Encode/Decode: Convert Selection, then a menu of possible conversions will be displayed. Alternatively you can use the keyboard bindings CMD-ALT-C and CTRL-ALT-C for Mac & PC respectively.https://marketplace.visualstudio.com/items?itemName=mitchdenny.ecdc Thank you to all of our Patreon SupportersThese individuals are personally supporting our open source initiatives to ensure the great toolings like CommandBox, ForgeBox, ColdBox,  ContentBox, TestBox and all the other boxes keep getting the continuous development they need, and funds the cloud infrastructure at our community relies on like ForgeBox for our Package Management with CommandBox. You can support us on Patreon here https://www.patreon.com/ortussolutionsNow offering Annual Memberships, pay for the year and save 10% - great for businesses. Bronze Packages and up, now get a ForgeBox Pro and CFCasts subscriptions as a perk for their Patreon Subscription. All Patreon supporters have a Profile badge on the Community Website All Patreon supporters have their own Private Forum access on the Community Website Patreons John Wilson - Synaptrix  Eric Hoffman Gary Knight Mario Rodrigues Giancarlo Gomez David Belanger Jonathan Perret Jeffry McGee - Sunstar Media Dean Maunder Joseph Lamoree Don Bellamy Jan Jannek Laksma Tirtohadi Carl Von Stetten Dan Card Jeremy Adams Jordan Clark Matthew Clemente Daniel Garcia Scott Steinbeck - Agri Tracking Systems Ben Nadel Mingo Hagen Brett DeLine Kai Koenig Charlie Arehart Jonas Eriksson Jason Daiger Jeff McClain Shawn Oden Matthew Darby Ross Phillips Edgardo Cabezas Patrick Flynn Stephany Monge Kevin Wright Steven Klotz You can see an up to date list of all sponsors on Ortus Solutions' Websitehttps://ortussolutions.com/about-us/sponsors ★ Support this podcast on Patreon ★

This is episode 211 recorded on November 10th, 2021 where John & Jason talk Shane Young, Principal Consultant at PowerApps911, Power Apps MVP, and YouTube Sensation, about Microsoft Power Apps, Microsoft Dataverse, and some of the announcements made at Microsoft Ignite around the Power Platform stack. For show notes please visit www.bifocal.show

Which type of lead is the winner? Sam shares what MQLs and SQLs mean when it comes to sales leads and which you should focus on.

About RichardHe's also an instructor at Pluralsight, a frequent public speaker, and the author of multiple books on software design and development. Richard maintains a regularly updated blog (seroter.com) on topics of architecture and solution design and can be found on Twitter as @rseroter. Links: Twitter: https://twitter.com/rseroter LinkedIn: https://www.linkedin.com/in/seroter Seroter.com: https://seroter.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friends at Vultr. Spelled V-U-L-T-R because they're all about helping save money, including on things like, you know, vowels. So, what they do is they are a cloud provider that provides surprisingly high performance cloud compute at a price that—while sure they claim its better than AWS pricing—and when they say that they mean it is less money. Sure, I don't dispute that but what I find interesting is that it's predictable. They tell you in advance on a monthly basis what it's going to going to cost. They have a bunch of advanced networking features. They have nineteen global locations and scale things elastically. Not to be confused with openly, because apparently elastic and open can mean the same thing sometimes. They have had over a million users. Deployments take less that sixty seconds across twelve pre-selected operating systems. Or, if you're one of those nutters like me, you can bring your own ISO and install basically any operating system you want. Starting with pricing as low as $2.50 a month for Vultr cloud compute they have plans for developers and businesses of all sizes, except maybe Amazon, who stubbornly insists on having something to scale all on their own. Try Vultr today for free by visiting: vultr.com/screaming, and you'll receive a $100 in credit. Thats v-u-l-t-r.com slash screaming.Corey: You know how git works right?Announcer: Sorta, kinda, not really Please ask someone else!Corey: Thats all of us. Git is how we build things, and Netlify is one of the best way I've found to build those things quickly for the web. Netlify's git based workflows mean you don't have to play slap and tickle with integrating arcane non-sense and web hooks, which are themselves about as well understood as git. Give them a try and see what folks ranging from my fake Twitter for pets startup, to global fortune 2000 companies are raving about. If you end up talking to them, because you don't have to, they get why self service is important—but if you do, be sure to tell them that I sent you and watch all of the blood drain from their faces instantly. You can find them in the AWS marketplace or at www.netlify.com. N-E-T-L-I-F-Y.comCorey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Once upon a time back in the days of VH1, which was like MTV except it played music videos, would have a show that was, “Where are they now?” Looking at former celebrities. I will not use the term washed up because that's going to be insulting to my guest.Richard Seroter is a returning guest here on Screaming in the Cloud. We spoke to him a year ago when he was brand new in his role at Google as director of outbound product management. At that point, he basically had stars in his eyes and was aspirational around everything he wanted to achieve. And now it's a year later and he has clearly failed because it's Google. So, outbound products are clearly the things that they are going to be deprecating, and in the past year, I am unaware of a single Google Cloud product that has been outright deprecated. Richard, thank you for joining me, and what do you have to say for yourself?Richard: Yeah, “Where are they now?” I feel like I'm the Leif Garrett of cloud here, joining you. So yes, I'm still here, I'm still alive. A little grayer after twelve months in, but happy to be here chatting cloud, chatting whatever else with you.Corey: I joke a little bit about, “Oh, Google winds up killing things.” And let's be clear, your consumer division which, you know, Google is prone to that. And understanding a company's org chart is a challenge. A year or two ago, I was of the opinion that I didn't need to know anything about Google Cloud because it would probably be deprecated before I really had to know about it. My opinion has evolved considerably based upon a number of things I'm seeing from Google.Let's be clear here, I'm not saying this to shine you on or anything like that; it's instead that I've seen some interesting things coming out of Google that I consider to be the right moves. One example of that is publicly signing multiple ten-year deals with very large, serious institutions like Deutsche Bank, and others. Okay, you don't generally sign contracts with companies of that scale and intend not to live up to them. You're hiring Forrest Brazeal as your head of content for Google Cloud, which is not something you should do lightly, and not something that is a short-term play in any respect. And the customer experience has continued to improve; Google Cloud products have not gotten worse, and I'm seeing in my own customer conversations that discussions about Google Cloud have become significantly less dismissive than they were over the past year. Please go ahead and claim credit for all of that.Richard: Yeah. I mean, the changes a year ago when I joined. So, Thomas Kurian has made a huge impact on some of that. You saw us launch the enterprise APIs thing a while back, which was, “Hey, here's, for the most part, every one of our products that has a fixed API. We're not going to deprecate it without a year's notice, whatever it is. We're not going to make certain types of changes.” Maybe that feels like, “Well, you should have had that before.” All right, all we can do is improve things moving forward. So, I think that was a good change.Corey: Oh, I agree. I think that was a great thing to do. You had something like 80-some-odd percent coverage of Google Cloud services, and great, that's going to only increase with time, I can imagine. But I got a little pushback from a few Googlers for not being more congratulatory towards them for doing this, and look, it's a great thing. Don't get me wrong, but you don't exactly get a whole lot of bonus points and kudos and positive press coverage—not that I'm press—for doing the thing you should have been doing [laugh] all along.It's, “This is great. This is necessary.” And it demonstrates a clear awareness that there was—rightly or wrongly—a perception issue around the platform's longevity and that you've gone significantly out of your way to wind up addressing that in ways that go far beyond just yelling at people on Twitter they don't understand the true philosophy of Google Cloud, which is the right thing to do.Richard: Yeah, I mean, as you mentioned, look, the consumer side is very experimental in a lot of cases. I still mourn Google Reader. Like, those things don't matter—Corey: As do we all.Richard: Of course. So, I get that. Google Cloud—and of course we have the same cultural thing, but at the same time, there's a lifecycle management that's different in Google Cloud. We do not deprecate products that much. You know, enterprises make decade-long bets. I can't be swap—changing databases or just turning off messaging things. Instead, we're building a core set of things and making them better.So, I like the fact that we have a pretty stable portfolio that keeps getting a little bit bigger. Not crazy bigger; I like that we're not just throwing everything out there saying, “Rock on.” We have some opinions. But I think that's been a positive trend, customers seem to like that we're making these long-term bets. We're not going anywhere for a long time and our earnings quarter after quarter shows it—boy, this will actually be a profitable business pretty soon.Corey: Oh, yeah. People love to make hay, and by people, I stretch the term slightly and talk about, “Investment analysts say that Google Cloud is terrible because at your last annual report you're losing something like $5 billion a year on Google Cloud.” And everyone looked at me strangely, when I said, “No, this is terrific. What that means is that they're investing in the platform.” Because let's be clear, folks at Google tend to be intelligent, by and large, or at least intelligent enough that they're not going to start selling cloud services for less than it costs to run them.So yeah, it is clearly an investment in the platform and growth of it. The only way it should be turning a profit at this point is if there's no more room to invest that money back into growing the platform, given your market position. I think that's a terrific thing, and I'm not worried at all about it losing money. I don't think anyone should be.Richard: Yeah, I mean, strategically, look, this doesn't have to be the same type of moneymaker that even some other clouds have to be to their portfolio. Look, this is an important part, but you look at those ten-year deals that we've been signing: when you look at Univision, that's a YouTube partnership; you look at Ford that had to do with Android Auto; you look at these others, this is where us being also a consumer and enterprise SaaS company is interesting because this isn't just who's cranking out the best IaaS. I mean, that can be boring stuff over time. It's like, who's actually doing the stuff that maybe makes a traditional company more interesting because they partner on some of those SaaS services. So, those are the sorts of deals and those sorts of arrangements where cloud needs to be awesome, and successful, and make money, doesn't need to be the biggest revenue generator for Google.Corey: So, when we first started talking, you were newly minted as a director of outbound product management. And now, you are not the only one, there are apparently 60 of you there, and I'm no closer to understanding what the role encompasses. What is your remit? Where do you start? Where do you stop?Richard: Yeah, that's a good question. So, there's outbound product management teams, mostly associated with the portfolio area. So network, storage, AI, analytics, database, compute, application modernization-y sort of stuff—which is what I cover—containers, dev tools, serverless. Basically, I am helping make sure the market understands the product and the product understands the market. And not to be totally glib, but a lot of that is, we are amplification.I'm amplifying product out to market, analysts, field people, partners: “Do you understand this thing? Can I help you put this in context?” But then really importantly, I'm trying to help make sure we're also amplifying the market back to our product teams. You're getting real customer feedback: “Do you know what that analyst thinks? Have you heard what happened in the competitive space?”And so sometimes companies seem to miss that, and PMs poke their head up when I'm about to plan a product or I'm about to launch a product because I need some feedback. But keeping that constant pulse on the market, on customers, on what's going on, I think that can be a secret weapon. I'm not sure everybody does that.Corey: Spending as much time as I do on bills, admittedly AWS bills, but this is a pattern that tends to unfold across every provider I've seen. The keynotes are chock-full of awesome managed service announcements, things that are effectively turnkey at further up the stack levels, but the bills invariably look a lot more like, yeah, we spend a bit of money on that and then we run 10,000 virtual instances in a particular environment and we just treat it like it's an extension of our data center. And that's not exciting; that's not fun, quote-unquote, but it's absolutely what customers are doing and I'm not going to sit here and tell them that they're wrong for doing it. That is the hallmark of a terrible consultant of, “I don't understand why you're doing what you're doing, so it must be foolish.” How about you stop and gain some context into why customers do the things that they do?Richard: No, I send around a goofy newsletter every week to a thousand or two people, just on things I'm learning from the field, from customers, trying to make sure we're just thinking bigger. A couple of weeks ago, I wrote an idea about modernization is awesome, and I love when people upgrade their software. By the way, most people migration is a heck of a lot easier than if I can just get this into your cloud, yeah love that; that's not the most interesting thing, to move VMs around, but most people in their budget, don't have time to rewrite every Java app to go. Everybody's not changing .NET framework to .NET core.Like, who do I think everybody is? No, I just need to try to get some incremental value first. Yes, then hopefully I'll swap out my self-managed SQL database for a Spanner or a managed service. Of course, I want all of that, but this idea that I can turn my line of business loan processing app into a thousand functions overnight is goofy. So, how are we instead thinking more pragmatically about migration, and then modernizing some of it? But even that sort of mindset, look, Google thinks about innovation modernization first. So, also just trying to help us take a step back and go, “Gosh, what is the normal path? Well, it's a lot of migration first, some modernization, and then there's some steady-state work there.”Corey: One of the things that surprised me the most about Google Cloud in the market, across the board, has been the enthusiastic uptake for enterprise workloads. And by enterprise workloads, I'm talking about things like SAP HANA is doing a whole bunch of deployments there; we're talking Big Iron-style enterprise-y things that, let's be honest, countervene most of the philosophy that Google has always held and espoused publicly, at least on conference stages, about how software should be built. And I thought that would cut against them and make it very difficult for you folks to gain headway in that market and I could not have been more wrong. I'm talking to large enterprises who are enthusiastically talking about Google Cloud. I've got a level with you, compared to a year or two ago, I don't recognize the place.Richard: Mmm. I mean, some of that, honestly, in the conversations I have, and whatever I do a handful of customer calls every week, I think folks still want something familiar, but you're looking for maybe a further step on some of it. And that means, like, yes, is everybody going to offer VMs? Yeah, of course. Is everyone going to have MySQL? Obviously.But if I'm an enterprise and I'm doing these generational bets, can I cheat a little bit, and maybe if I partner with a more of an innovation partner versus maybe just the easy next step, am I buying some more relevance for the long-term? So, am I getting into environment that has some really cool native zero-trust stuff? Am I getting into environment with global backend services and I'm not just stitching together a bunch of regional stuff? How can I cheat by using a more innovation vendor versus just lifting and shifting to what feels like hosted software in another cloud? I'm seeing more of that because these migrations are tough; nobody should be just randomly switching clouds. That's insane.So, can I make, maybe, one of these big bets with somebody who feels like they might actually even improve my business as a whole because I can work with Google Pay and improve how I do mobile payments, or I could do something here with Android? Or, heck, all my developers are using Angular and Flutter; aren't I going to get some benefit from working with Google? So, we're seeing that, kind of, add-on effect of, “Maybe this is a place not just to host my VMs, but to take a generational leap.”Corey: And I think that you're positioning yourselves in a way to do it. Again, talk about things that you wouldn't have expected to come out of Google of all places, but your console experience has been first-rate and has been for a while. The developer experience is awesome; I don't need to learn the intricacies of 12 different services for what I'm trying to do just in order to get something basic up and running. I can stop all the random little billing things in my experimental project with a single click, which that admittedly has a confirm, which you kind of want. But it lets you reason about these things.It lets you get started building something, and there's a consistency and cohesiveness to the console that, again, I am not a graphic designer, by any stretch of the imagination. My most commonly used user interface is a green-screen shell prompt, and then I'm using Vim to wind up writing something horrifying, ideally in Python, but more often in YAML. And that has been my experience, but just clicking around the console, it's clear that there was significant thought put into the design, the user experience, and the way of approaching folks who are starting to look very different, from a user persona perspective.Richard: I can—I mean, I love our user research team; they're actually fun to hang out with and watch what they do, but you have to remember, Google as a company, I don't know, cloud is the first thing we had to sell. Did have to sell Gmail. I remember 15 years ago, people were waiting for invites. And who buys Maps or who buys YouTube? For the most part, we've had to build things that were naturally interesting and easy-to-use because otherwise, you would just switch to anything else because everything was free.So, some of that does infuse Google Cloud, “Let's just make this really easy to use. And let's just make sure that, maybe, you don't hate yourself when you're done jumping into a shell from the middle of the console.” It's like, that should be really easy to do—or upgrade a database, or make changes to things. So, I think some of the things we've learned from the consumer good side, have made their way to how we think of UX and design because maybe this stuff shouldn't be terrible.Corey: There's a trope going around, where I wound up talking about the next million cloud customers. And I'm going to have to write a sequel to it because it turns out that I've made a fundamental error, in that I've accepted the narrative that all of the large cloud vendors are pushing, to the point where I heard from so many folks I just accepted it unthinkingly and uncritically, and that's not what I should be doing. And we'll get to what I was wrong about in a minute, but the thinking goes that the next big growth area is large enterprises, specifically around corporate IT. And those are folks who are used to managing things in a GUI environment—which is fine—and clicking around in web apps. Now, it's easy to sit here on our high horse and say, “Oh, you should learn to write code,” or YAML, which is basically code. Cool.As an individual, I agree, someone should because as soon as they do that, they are now able to go out and take that skill to a more lucrative role. The company then has to backfill someone into the role that they just got promoted out of, and the company still has that dependency. And you cannot succeed in that market with a philosophy of, “Oh, you built something in the console. Now, throw it away and do it right.” Because that is maddening to that user persona. Rightfully so.I'm not that user persona and I find it maddening when I have to keep tripping over that particular thing. How did that come to be, from your perspective? First, do you think that is where the next million cloud customers come from? And have I adequately captured that user persona, or am I completely often the weeds somewhere?Richard: I mean, I shared your post internally when that one came out because that resonated with me of how we were thinking about it. Again, it's easy to think about the cloud-native operators, it's Spotify doing something amazing, or this team at Twitter doing something, or whatever. And it's not even to be disparaging. Like, look, I spent five years in enterprise IT and I was surrounded by operators who had to run dozen different systems; they weren't dedicated to just this thing or that. So, what are the tools that make my life easy?A lot of software just comes with UIs for quick install and upgrades, and how does that logic translate to this cloud world? I think that stuff does matter. How are you meeting these people a little better where they are? I think the hard part that we will always have in every cloud provider is—I think you've said this in different forums, but how do I not sometimes rub the data center on my cloud or vice versa? I also don't want to change the experience so much where I degrade it over the long term, I've actually somehow done something worse.So, can I meet those people where they are? Can we pull some of those experiences in, but not accidentally do something that kind of messes up the cloud experience? I mean, that's a fine line to walk. Does that make sense to you? Do you see where there's a… I don't know, you could accidentally cater to a certain audience too much, and change the experience for the worse?Corey: Yes, and no. My philosophy on it is that you have to meet customers where they are, but only to a point. At some point, what they're asking for becomes actively harmful or disadvantageous to wind up providing for them. “I want you to run my data center for me,” is on some level what some cloud environments look like, and I'm not going to sit here and tell people they're inherently wrong for that. Their big reason for moving to the cloud was because they keep screwing up replacing failed hard drives in their data center, so we're going to put it in the cloud.Is it more expensive that way? Well, sure in terms of actual cash outlay, it almost certainly is, but they're also not going down every month when a drive fails, so once the value of that? It's a capability story. That becomes interesting to me, and I think that trying to sit here in isolation, and say that, “Oh, this application is not how we would build it at Google.” And it's, “Yeah, you're Google. They are insert an entire universe of different industries that look nothing whatsoever like Google.” The constraints are different, the resources are different, and—Richard: Sure.Corey: —their approach to problem-solving are different. When you built out Google, and even when you're building out Google Cloud, look at some of the oldest craftiest stuff you have in your entire all of Google environment, and then remember that there are companies out there that are hundreds of years old. It's a different order of magnitude as far as era, as far as understanding of what's in the environment, and that's okay. It's a very broad and very diverse world.Richard: Yeah. I mean, that's, again, why I've been thinking more about migration than even some of the modernization piece. Should you bring your network architecture from on-prem to the cloud? I mean, I think most cases, no. But I understand sometimes that edge firewall, internal trust model you had on-prem, okay, trying to replicate that.So, yeah, like you say, I want to meet people where they are. Can we at least find some strategic leverage points to upgrade aspects of things as you get to a cloud, to save you from yourself in some places because all of a sudden, you have ten regions and you only had one data center before. So, many more rooms for mistakes. Where are the right guardrails? We're probably more opinionated than others at Google Cloud.I don't really apologize for that completely, but I understand. I mean, I think we've loosened up a lot more than maybe people [laugh] would have thought a few years ago, from being hyper-opinionated on how you run software.Corey: I will actually push back a bit on the idea that you should not replicate your on-premises data center in your cloud environment. Sure, are there more optimal ways to do it that are arguably more secure? Absolutely. But a common failure mode in moving from data center to cloud is, “All right, we're going to start embracing this entirely new cloud networking paradigm.” And it is confusing, and your team that knows how the data center network works really well are suddenly in way over their heads, and they're inadvertently exposing things they don't intend to or causing issues.The hard part is always people, not technology. So, when I glance at an environment and see things like that, perfect example, are there more optimal ways to do it? Oh, from a technology perspective, absolutely. How many engineers are working on that? What's their skill set? What's their position on all this? What else are they working on? Because you're never going to find a team of folks who are world-class experts in every cloud? It doesn't work that way.Richard: No doubt. No doubt, you're right. There's areas where we have to at least have something that's going to look similar, let you replicate aspects of it. I think it's—it'll just be interesting to watch, and I have enough conversations with customers who do ask, “Hey, where are the places we should make certain changes as we evolve?” And maybe they are tactical, and they're not going to be the big strategic redesign their entire thing. But it is good to see people not just trying to shovel everything from one place to the next.Corey: This episode is sponsored in part by something new. Cloud Academy is a training platform built on two primary goals. Having the highest quality content in tech and cloud skills, and building a good community the is rich and full of IT and engineering professionals. You wouldn't think those things go together, but sometimes they do. Its both useful for individuals and large enterprises, but here's what makes it new. I don't use that term lightly. Cloud Academy invites you to showcase just how good your AWS skills are. For the next four weeks you'll have a chance to prove yourself. Compete in four unique lab challenges, where they'll be awarding more than $2000 in cash and prizes. I'm not kidding, first place is a thousand bucks. Pre-register for the first challenge now, one that I picked out myself on Amazon SNS image resizing, by visiting cloudacademy.com/corey. C-O-R-E-Y. That's cloudacademy.com/corey. We're gonna have some fun with this one!Corey: Now, to follow up on what I was saying earlier, what I think I've gotten wrong by accepting the industry talking points on is that the next million cloud customers are big enterprises moving from data centers into the cloud. There's money there, don't get me wrong, but there is a larger opportunity in empowering the creation of companies in your environment. And this is what certain large competitors of yours get very wrong, where it's we're going to launch a whole bunch of different services that you get to build yourself from popsicle sticks. Great. That is not useful.But companies that are trying to do interesting things, or people who want to found companies to do interesting things, want something that looks a lot more turnkey. If you are going to be building cloud offerings, that for example, are terrific building blocks for SaaS companies, then it behooves you to do actual investments, rather than just a generic credit offer, into spurring the creation of those types of companies. If you want to build a company that does payroll systems, in a SaaS, cloud way, “Partner with us. Do it here. We will give you a bunch of credits. We will introduce you to your first ten prospective customers.”And effectively actually invest in a company success, as opposed to pitch-deck invest, which is, “Yeah, we'll give you some discounting and some credits, and that's our quote-unquote, ‘investment.'” actually be there with them as a partner. And that's going to take years for folks to wrap their heads around, but I feel like that is the opportunity that is significantly larger, even than the embedded existing IT space because rather than fighting each other for slices of the pie, I'm much more interested in expanding that pie overall. One of my favorite questions to get asked because I think it is so profoundly missing the point is, “Do you think it's possible for Google to go from number three to number two,” or whatever the number happens to be at some point, and my honest, considered answer is, “Who gives a shit?” Because number three, or number five, or number twelve—it doesn't matter to me—is still how many hundreds of billions of dollars in the fullness of time. Let's be real for a minute here; the total addressable market is expanding faster than any cloud or clouds are going to be able to capture all of.Richard: Yeah. Hey, look, whoever who'll be more profitable solving user problems, I really don't care about the final revenue number. I can be the number one cloud tomorrow by making Google Cloud free. What's the point? That's not a sustainable business. So, if you're just going for who can deploy the most VCPUs or who can deploy the most whatever, there's ways to game that. I want to make sure we are just uniquely solving problems better than anybody else.Corey: Sorry, forgive me. I just sort of zoned out for a second there because I'm just so taken aback and shocked by the idea of someone working at a large cloud provider who expresses a philosophy that isn't lying awake at night fretting over the possibility of someone who isn't them as making money somewhere.Richard: [laugh]. I mean, your idea there, it'll be interesting to watch, kind of, the maker's approach of are you enabling that next round of startups, the next round of people who want to take—I mean, honestly, I like the things we're doing building block-wise, even with our AI: we're not just handing you a vision API, we're giving you a loan processing AI that can process certain types of docs, that more packaged version of AI. Same with healthcare, same with whatever. I can imagine certain startups or a company idea going, “Hey, maybe I could disrupt or serve a new market.”I always love what Square did. They've disrupted emerging markets, small merchants here in North America, wherever, where I didn't need a big expensive point of sale system. You just gave me the nice, right building blocks to disrupt and run my business. Maybe Google Cloud can continue to provide better building blocks, but I do like your idea of actually investment zones, getting part of this. Maybe the next million users are founders and it's not just getting into some of these companies with, frankly, 10, 20, 30,000 people in IT.I think there's still plenty of room in these big enterprises to unlock many more of those companies, much more of their business. But to your point, there's a giant market here that we're not all grabbing yet. For crying out loud, there's tons of opportunity out here. This is not zero-sum.Corey: Take it a step further beyond that, and today, if you have someone who's enterprising, early on in their career, maybe they just got out of school, maybe they have just left their job and are ready to snap, or they have some severance money that they want to throw into something. Great. What do they want to do if they have an idea for a company? Well today, that answer looks a lot like, well, time to go to a boot camp and learn to code for six months so you can build a badly done MVP well enough to get off the ground and get some outside investment, and then go from there. Well, what if we cut that part out entirely?What if there were building blocks of I don't need to know or care that there's a database behind it, or what a database looks like. Picture Visual Basic in a web browser for building apps, and just take this bit of information I give you and store it and give it back to me later. Sure, you're going to have some significant challenges in the architecture or something like that as it goes from this thing that I'm talking about as an MVP to something planet-scale—like a Spotify for example—but that's not most businesses, and that's okay. Get out of the way and let people innovate and iterate on what it is they're doing more rapidly, and make it more accessible to teach people. That becomes huge; that gets the infrastructure bits that cloud providers excel at out of the way, and all it really takes is packaging those things into a golden path of what a given company of a particular profile should be doing, if—unless they have reason to deviate from it—and instead of having this giant paradox of choice issue, it's, “Oh, okay, I'll drag-drop, build things accordingly.”And under the hood, it's doing all the configuration of services and that's great. But suddenly, you've made being a founder of a software company—fundamentally—accessible to people who are not themselves software engineers. And I know that's anathema to some people, and I don't even slightly care because I am done with gatekeeping.Richard: Yeah. No, it's exciting if that can pull off. I mean, it's not the years ago where, how much capital was required to find the rack and do all sorts of things with tech, and hire some developers. And it's an amazing time to be software creators, now. The more we can enable that—yeah, I'm along for that journey, sign me up.Corey: I'm looking forward to seeing how it winds up shaking out. So, I want to talk a little bit about the paradox of choice problem that I just mentioned. If you take a look at the various compute services that every cloud provider offers, there are an awful lot of different choices as far as what you can run. There's the VM model, there's containers—if you're in AWS, you have 17 ways to run those—and you wind up—any of the serverless function story, and other things here and there, and managed services, I mean and honestly, Google has a lot of them, nowhere near as many as you do failed messaging products, but still, an awful lot of compute options. How do customers decide?What is the decision criteria that you see? Because the worst answer you can give someone who doesn't really know what they're doing is, “It depends,” because people don't know how to make that decision. It's, “What factors should I consider then, while making that decision?” And the answer has to be something somewhat authoritative because otherwise, they're going to go on the internet and get yelled at by everyone because no one is ever going to agree on this, except that everyone else is wrong.Richard: Mm-hm. Yeah, I mean, on one hand, look, I like that we intentionally have fewer choices than others because I don't think you need 17 ways to run a container. I think that's excessive. I think more than five is probably excessive because as a customer, what is the trade-off? Now, I would argue first off, I don't care if you have a lot of options as a vendor, but boy, the backends of those better be consistent.Meaning if I have a CI/CD tool in my portfolio and it only writes to two of them, shame on me. Then I should make sure that at least CI/CD, identity management, log management, monitoring, arguably your compute runtime should be a late-binding choice. And maybe that's blasphemous because somebody says, “I want to start up front knowing it's a function,” or, “I want to start it's a VM.” How about, as a developer, I couldn't care less. How about I just build cool software and maybe even at deploy time, I say, “This better fits in running in Kubernetes.” “This is better in a virtual machine.”And my cost of changing that later is meaningless because, hey, if it is in the container, I can switch it between three or four different runtimes, the identity management the same, it logs the exact same way, I can deploy CI/CD the same way. So, first off, if those things aren't the same, then the vendor is messing up. So, the customer shouldn't have to pay the cost of that. And then there gets to be other actual criteria. Look, I think you are looking at the workload itself, the team who makes it, and the strategy to figure out the runtime.It's easy for us. Google Compute Engine for VMs, containers go in GKE, managed services that need some containers, there are some apps around them, are Cloud Functions and Cloud Run. Like, it's fairly straightforward and it's going to be an OR situation—or an AND situation not an OR, which is great. But we're at least saying the premium way to run containers in Google Cloud for systems is GKE. There you go. If you do have a bunch of managed services in your architecture and you're stitching them together, then you want more serverless things like Cloud Run and Cloud Functions. And if you want to just really move some existing workload, GCE is your best choice. I like that that's fairly straightforward. There's still going to be some it depends, but it feels better than nine ways to run Kubernetes engines.Corey: I'm sure we'll see them in the fullness of time.Richard: [laugh].Corey: So, talk about Anthos a bit. That was a thing that was announced a while back and it was extraordinarily unclear what it was. And then I looked at the pricing and it was $10,000 a month with a one-year minimum commitment, and is like, “Oh, it's not for me. That's why I don't get it.” And I haven't really looked back at it since. But it is something else now. It almost feels like a wrapper brand, in some respects. How's it going? [unintelligible 00:29:26]?Richard: Yeah. Consumption, we'll talk more upcoming months on some of the adoption, but we're finally getting the hockey stick, which always comes delayed with platforms because nobody adopts platforms quickly. They buy the platform and a year later they start to actually build new development, migrate the things they have. So, we're starting to see the sort of growth. But back to your first point. And I even think I poorly tried to explain it a year ago with you. Basically, look, Anthos is the ability to manage fleets of GKE clusters, wherever they are. I don't care if they're on-prem, I don't care if they're in Google Cloud, I don't care if they're Amazon. We have one customer who only uses Anthos on AWS. Awesome, rock on.So, how do I put GKE clusters everywhere, but then do fleet management because look, some people are doing an app per cluster. They don't want to jam 50 apps in the cluster from different teams because they don't like the idea that this app requires root access; now you can screw around with mine. Or, you didn't update; that broke the cluster. I don't want any of that. So, you're going to see companies more, doing even app per cluster, app per developer per cluster.So, now I have a fleet problem. How do I keep it in sync? How do I make sure policy is consistent? Those sorts of things. So, Anthos is kind of solving the fleet management challenge and replacing people's first-gen app platform.Seeing a lot of those use cases, “Hey, we're retiring our first version of Docker Enterprise, Mesos, Cloud Foundry, even OpenShift,” saying, “All right, now's the time for our next version of our app platform. How about GKE, plus Cloud Run on top of it, plus other stuff?” Sounds good. So, going well is a, sort of—as you mentioned, there's a brand story here, mainly because we've also done two things that probably matter to you. A, we changed the price a lot.No minimum commit, remarkably at 20% of the cost it was when we launched, on purpose because we've gotten better at this. So, much cheaper, no minimum commit, pay as you go. Be on-premises, on bare metal with GKE. Pay by the hour, I don't care; sounds great. So, you can do that sort of stuff.But then more importantly, if you're a GKE customer and you just want config management, service mesh, things like that, now you can buy all of those independently as well. And Anthos is really the brand for fleet management of GKE. And if you're on Google Cloud only, it adds value. If you're off Google Cloud, if you're multi-cloud, I don't care. But I want to manage fleets of compute clusters and create them. We're going to keep doubling down on that.Corey: The big problem historically for understanding a lot of the adoption paradigm of Kubernetes has been that it was, to some extent, a reimagining of how Google ran and built software internally. And I thought at the time, the idea was—from a cynical perspective—that, “All right, well, your crappy apps don't run well on Google-style infrastructure so we're going to teach the entire world how to write software the way that we do.” And then you end up with people running their blog on top of Kubernetes, where it's one of those, like, the first blog post is, like, “How I spent the last 18 months building Kubernetes.” And, okay, that is certainly a philosophy and an approach, but it's almost approaching Windows 95 launch level of hype, where people who didn't own computers were buying copies of it, on some level. And I see the term come up in conversations in places where it absolutely has no place being brought up. “How do I run a Kubernetes cluster inside of my laptop?” And, “It's what you got going on in there, buddy?”Richard: [laugh].Corey: “What do you think you're trying to do here because you just said something that means something that I think is radically different to me than it is to you.” And again, I'm not here to judge other people's workflows; they're all terrible, except for mine, which is an opinion held by everyone about their own workflow. But understanding where people are, figuring out how to get there, how to meet customers where they are and empower them. And despite how heavily Google has been into the Kubernetes universe since its inception, you're very welcoming to companies—and loud-mouth individuals on Twitter—who have no use for Kubernetes. And working through various products you offer, I don't ever feel like a second-class citizen. There's really something impressive about that, of not letting the hype dictate the product and marketing decisions of it.Richard: Yeah, look, I think I tweeted it recently, I think the future of software is managed services with containers in the gap, for the most part. Whereas—if you can use managed services, please do. Use them wherever you can. And if you have to sling some code, maybe put it in a really portable thing that's really easy to run in lots of places. So, I think that's smart.But for us, look, I think we have the best container workflow from dev tools, and build tools, and artifact registries, and runtimes, but plenty of people are running containers, and you shouldn't be running Kubernetes all over the place. That makes sense for the workload, I think it's better than a VM at the retail edge. Can I run a small cluster, instead of a weird point-of-sale Windows app? Maybe. Maybe it makes sense to have a lightweight Kubernetes cluster there for consistency purposes.So, for me, I think it's a great medium for a subset of software. Google Cloud is going to take whatever you got, which is great. I think containers are great, but at the same time, I'm happily going to let you deploy a function that responds to you adding a storage item to a bucket, where at the same time give you a SaaS service that replaces the need for any code. All of those are terrific. So yeah, we love Kubernetes. We think it's great. We're going to be the best version to run it. But that's not going to be your whole universe.Corey: No, and I would argue it absolutely shouldn't be.Richard: [laugh]. Right. Agreed. Now again, for some companies, it's a great replacement for this giant fleet of VMs that all runs at eight percent utilization. Can I stick this into a bunch of high-density clusters? Absolutely you should. You're going to save an absolute fortune doing that and probably pick up some resilience and functionality benefits.But to your point, “Do I want to run a WordPress site in there?” I don't know, probably not. “Do I need to run my own MySQL?” I'd prefer you not do that. So, in a lot of cases, don't use it unless you have to. That should go for all compute nowadays. Use managed services.Corey: I'm a big believer in going down that approach just because it is so much easier than trying to build it yourself from popsicle sticks because you theoretically might have to move it someday in the future, even though you're not.Richard: [laugh]. Right.Corey: And it lets me feel better about a thing that isn't going to be used by anything that I'm doing in the near future. I just don't pretend to get it.Richard: No, I don't install a general purpose electric charger in my garage for any electric car I may get in the future; I charge for the one I have now. I just want it to work for my car; I don't want to plan for some mythical future. So yeah, premature optimization over architecture, or death in IT, especially nowadays where speed matters, don't waste your time building something that can run in nine clouds.Corey: Richard, I want to thank you for coming on again a year later to suffer my slings, arrows, and other various implements of misfortune. If people want to learn more about what you're doing, how you're doing it, possibly to pull a Forrest Brazeal and go work with you, where can they find you?Richard: Yeah, we're a fun place to work. So, you can find me on Twitter at @rseroter—R-S-E-R-O-T-E-R—hang out on LinkedIn, annoy me on my blog seroter.com as I try to at least explore our tech from time to time and mess around with it. But this is a fun place to work. There's a lot of good stuff going on here, and if you work somewhere else, too, we can still be friends.Corey: Thank you so much for your time today. Richard Seroter, director of outbound product management at Google. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment into which you have somehow managed to shove a running container.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Recorded Date 10/23/2021 Title: In this episode, Kito, Ian, Danno and Josh welcome special guest Richard Fitchner, Founder JUG Oberpfalz, CEO of XDev, and one of the organizers of JCON. They discuss #lowcode, running a conference, being CEO, #microstream, #jooq, the future of the #playframework, #rapidclipse, XDEV IDE, #tabnine AI IDE plugin, #jreleaser, #testcontainers, and much more. Richard / XDEV XDEV Software https://xdev.software/en/ XAPI - XDEV IDE Framework https://github.com/xdev-software/xapi RapidClipse https://rapidclipse.com/en Server Side Java Payara Cloud now Available https://www.payara.fish/products/payara-cloud/ Microstream https://microstream.one/ Open source announcement https://www.infoq.com/news/2021/09/microstream-5-is-open-source/ jOOQ: The easiest way to write SQL in Java https://www.jooq.org/ Web On the future of Play Framework https://www.lightbend.com/blog/on-the-future-of-play-framework IE11 Countdown Clock https://death-to-ie11.com/ Java Web Start is dead. Long live OpenWebStart! - openwebstart.com https://openwebstart.com/ Developer Tools Github Copilot https://copilot.github.com/ Tabnine - AI plugin https://www.tabnine.com/ Picks JReleaser https://jreleaser.org/ Free Guy (Movie) https://www.imdb.com/title/tt6264654/ TestContainers http://testcontainers.org Book: Procrastinate on Purpose https://www.amazon.com/Procrastinate-on-Purpose-audiobook/dp/B00RV58ZFO/ref=sr_1_1?dchild=1&gclid=CjwKCAjwwsmLBhACEiwANq-tXOE3sB0aAh-2reeOQLzwXxJ52sDFhz9DlcXwJG_ADcWJvXrKCRjmWRoChJcQAvD_BwE&hvadid=241897656186&hvdev=c&hvlocphy=1027142&hvnetw=g&hvqmt=e&hvrand=12176312028861189251&hvtargid=kwd-76000619146&hydadcr=22564_10346437&keywords=procrastinate+on+purpose&qid=1634933460&sr=8-1 AdoptOpenJDK is now Eclipse Tamarin https://blog.adoptopenjdk.net/2021/08/goodbye-adoptopenjdk-hello-adoptium/ Events W-JAX Nov 8 – 12, 2021, Munich, Germany or virtual https://jax.de/munich/ JakartaOne LiveStream 2021 - Dec 7th, 2021 in US https://jakartaone.org/ Progressive Web Experience Dec 5-8, 2021, Clearwater, FL, USA or virtual https://progressivewebexperience.io/ Jconf.dev Dec 8-10, 2021, Chicago, IL, USA https://2021.jconf.dev/ Archconf Dec 13-16, Clearwater, FL, USA or virtual https://archconf.com/ CodeMash Jan 11-14, 2022 - Sandusky, OH https://www.codemash.org/ Software Design and Development - May 16-20, 2022 - London, UK https://sddconf.com/    

This week, we are joined by Rob Petrone, Vice President of IT, TRC Companies. Moving to a cloud-first IT model, TRC Companies needed simpler, more reliable and higher-performing data protection for their SQL databases and other critical data. They chose Dell EMC Data Protection Suite, Dell EMC PowerProtect DD Virtual Edition/Azure, and Dell EMC PowerProtect Appliances to reduces backup and restore times and lowers storage consumption and costs. Tune in to discover how Dell Technologies gave them peace of mind with fast, uncomplicated cloud data protection

Howard Chu, CTO of Symas Corp and chief architect of the OpenLDAP project, discusses the key features of B+Tree Data Structures which make it the default selection for efficient and predictable storage of sorted data.

In this episode of Building The Backend we hear from Dipti Borkar cofounder @ Ahana  a managed service for Presto on AWS, where we talk all about the data lake, how it should be structured and where the industry is going. Below are top 3 value bombs: Presto is an open source distributed SQL query engine originally created by Facebook, mainly used to run SQL queries on data lakes but can be connected to relational data stores as well. Ahana is a managed Presto service on AWS with 3x price/performance. When optimizing your data lake, it's normally best to store the data in Parquet or ORC format vs JSON or CSV as they are columnar formats that can have indexes built in. Data Lake Houses are continuing to gain popularity by bringing the benefits of your data lake and data warehouse together with the help of tools like  Databricks DeltaLake and Apache HUDI.

In this episode of Scaling Postgres, we discuss the open sourcing of Babelfish, deconstructing the Postgres planner, when to avoid exist & not exist and having fun with SQL. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://babelfishpg.org/blog/releases/2021/10/babelfish-launch/ https://aws.amazon.com/blogs/aws/goodbye-microsoft-sql-server-hello-babelfish/ https://pganalyze.com/blog/deconstructing-the-postgres-planner https://postgres.ai/blog/20211103-three-cases-against-if-not-exists-and-if-exists-in-postgresql-ddl https://blog.crunchydata.com/blog/fun-with-sql-in-postgres-finding-revenue-accrued-per-day https://notes.eatonphil.com/exploring-plpgsql-forth-like.html https://www.highgo.ca/2021/11/01/the-postgresql-timeline-concept/ https://blog.crunchydata.com/blog/patroni-etcd-in-high-availability-environments https://blog.crunchydata.com/blog/resize-postgres-kubernetes-volume-instance-sets https://aws.amazon.com/blogs/database/vehicle-routing-optimization-with-amazon-aurora-postgresql-compatible-edition/ https://postgresql.life/post/tatsuro_yamada/ https://www.rubberduckdevshow.com/episodes/19-how-much-time-should-you-spend-planning/

This is episode 210 recorded on November 3rd, 2021 where John & Jason talk about the latest news announced at and around Microsoft Ignite Fall 2021 including Azure Chaos Studio, Power BI Teams App goes GA, and Hybrid tables - coming "this year". For show notes please visit www.bifocal.show

On The Cloud Pod this week, half the team misses Rob and Ben. Also, AWS Gaudi Accelerators speed up deep learning, GCP announces that its Tau VMs are an independently verified delight, and Azure gets the chance to be Number One for once (with industrial IoT platforms.) A big thanks to this week's sponsors: Foghorn Consulting, which provides full-stack cloud solutions with a focus on strategy, planning and execution for enterprises seeking to take advantage of the transformative capabilities of AWS, Google Cloud and Azure. JumpCloud, which offers a complete platform for identity, access, and device management — no matter where your users and devices are located.  This week's highlights

Kafka Streams is a native streaming library for Apache Kafka® that consumes messages from Kafka to perform operations like filtering a topic's message and producing output back into Kafka. After working as a developer in stream processing, Bill Bejeck (Apache Kafka Committer and Integration Architect, Confluent) has found his calling in sharing knowledge and authoring his book, “Kafka Streams in Action.” As a Kafka Streams expert, Bill is also the author of the Kafka Streams 101 course on Confluent Developer, where he delves into what Kafka Streams is, how to use it, and how it works. Kafka Streams provides the abstraction over Kafka consumers and producers by minimizing administrative details like the need to code and manage frameworks required when using plain Kafka consumers and producers to process streams. Kafka Streams is declarative—you can state what you want to do, rather than how to do it. Kafka Streams leverages the KafkaConsumer protocol internally; it inherits its dynamic scaling properties and the consumer group protocol to dynamically redistribute the workload. When Kafka Streams applications are deployed separately but have the same application.id, they are logically still one application. Kafka Streams has two processing APIs, the declarative API or domain-specific language (DSL)  is a high-level language that enables you to build anything needed with a processor topology, whereas the Processor API lets you specify a processor typology node by node, providing the ultimate flexibility. To underline the differences between the two APIs, Bill says it's almost like using the object-relational mapping framework (ORM) versus SQL. The Kafka Streams 101 course is designed to get you started with Kafka Streams and to help you learn the fundamentals of: How streams and tables work How stateless and stateful operations work How to handle time windows and out of order dataHow to deploy Kafka StreamsEPISODE LINKSKafka Streams 101 courseA Guide to Kafka Streams and Its UsesKafka Streams 101 meetupWatch the video version of this podcastJoin the Confluent CommunityLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperLive demo: Intro to Event-Driven Microservices with ConfluentUse podcon19 to get 40% off "Kafka Streams in Action"Use podcon19 to get 40% off "Event Streaming with Kafka Streams and ksqlDB"Use PODCAST100 to get $100 of free Confluent Cloud usage (details)

In this episode of the next CMO podcast we speak to Erin Hutchinson the global CMO of Merkle, a leading provider of customer experience technologies and solutions.Erin helps us understand how global brands are approaching their customer experience transformation projects, and how those projects have been accelerated because of changing consumer expectations during the pandemic.Merkle is also going through their own brand transformation as Erin shepherds them through Merkle 5.0, the fifth iteration of the company's brand.We also learn how Erin, who trained to be a teacher, went from an SQL developer to the global CMO of a leading technology company during her successful 20 year career.Learn more about Erin hereLearn more about Merkle hereLearn more about Plannuh here

On this week's show Patrick Gray and Adam Boileau discuss the week's security news, including: Someone took down Iranian fuel stations Papua New Guinea ransomware attack is pretty grim stuff Russia's SVR still going berserk in cloudtown China Telecom America gets the boot Much, much more We'll be hearing from Senetas CEO Andrew Wilson in this week's sponsor interview. He's joining us to talk about how the global semiconductor shortage is making him a very, very sad panda. Links to everything that we discussed are below and you can follow Patrick or Adam on Twitter if that's your thing. Show notes Iran says sweeping cyberattack took down gas stations across country Cyber ​​group 'Adalat Ali' published documents related to the November 1998 protests - BBC News Farsi Papua New Guinea Hit by Ransomware Hackers With Millions in Aid Frozen - Bloomberg (1) Cloudpng on Twitter: "This is the setup for all agencies must be on-site at Vulupindi Haus, Finance dept POM to process claims for IFMS after the system was hacked in October 2021. It's pretty full so bookings must be made to secure a PC. #ifms #systems #png https://t.co/VCiUYE9hFL" / Twitter (1) Hon Sasindran Muthuvel MP on Twitter: "Statement on the financial system failure and the challenges it now creates for all provinces. This issue must be addressed holistically and the Finance Dept must work in conjunction with the provinces. Sasi https://t.co/OLMAHxgDel" / Twitter 'Destructive' cyberattack hits National Bank of Pakistan - The Record by Recorded Future Microsoft says Russia hacked at least 14 IT service providers this year - The Record by Recorded Future Industry group warns of coordinated DDoS extortion campaign against VoIP providers - The Record by Recorded Future Bandwidth.com expects to lose up to $12M following DDoS extortion attempt - The Record by Recorded Future DDoS attacks hit multiple email providers - The Record by Recorded Future FCC revokes license for China Telecom Americas amid national security concerns - The Record by Recorded Future LinkedIn to Shutter Service in China - The Record by Recorded Future A Roaming Threat to Telecommunications Companies | CrowdStrike NSA warns of threat actors compromising entire 5G networks via cloud systems - The Record by Recorded Future Commerce Department announces new rule aimed at stemming sale of hacking tools to Russia and China - The Washington Post Windows 10, iOS 15, Ubuntu, Chrome fall at China's Tianfu hacking contest - The Record by Recorded Future FBI Raids Chinese Point-of-Sale Giant PAX Technology – Krebs on Security Malware found in npm package with millions of weekly downloads - The Record by Recorded Future Polygon pays out record $2 million bug bounty reward for critical vulnerability | The Daily Swig Hacker steals government ID database for Argentina's entire population - The Record by Recorded Future Fraudsters Cloned Company Director's Voice In $35 Million Bank Heist, Police Find How Hackers Hijacked Thousands of High-Profile YouTube Accounts | WIRED Instagram Hacker Forces Victim to Make Hostage-Style Video Missouri governor calls for prosecution of journalist who flagged website flaw Israeli hospital cancels non-urgent procedures following ransomware attack | The Daily Swig Ransomware Has Disrupted Almost 1,000 Schools in the US This Year Ransomware attack disrupts Toronto's public transportation system - The Record by Recorded Future Workers sent home after ransomware attack on major automotive parts manufacturer - The Record by Recorded Future Largest candy corn maker in US gets hacked ahead of Halloween Sinclair Workers Say TV Channels Are in ‘Pandemonium' After Ransomware Attack Cybercriminals claim to have hacked the NRA 'Cyber event' knocks dairy giant Schreiber Foods offline amid industry ransomware outbreak - CyberScoop Cyberattack hits Meliá, one of the largest hotel chains in the world - The Record by Recorded Future Olympus US hack tied to sanctioned Russian ransomware group | TechCrunch Europol detains suspects behind LockerGoga, MegaCortex, and Dharma ransomware attacks - The Record by Recorded Future Hitting the BlackMatter gang where it hurts: In the wallet - Emsisoft | Security Blog Ransomware hackers nervous, allege harassment from U.S. DarkSide ransomware gang moves some of its Bitcoin after REvil got hit by law enforcement - The Record by Recorded Future Hackers use SQL injection bug in BillQuick billing app to deploy ransomware - The Record by Recorded Future Ransomware gangs are abusing a zero-day in EntroLink VPN appliances - The Record by Recorded Future Conti Ransom Gang Starts Selling Access to Victims – Krebs on Security Cybercrime gang sets up fake company to hire security experts to aid in ransomware attacks - The Record by Recorded Future FBI PIN on ransomware crew targeting trend EXCLUSIVE Governments turn tables on ransomware gang REvil by pushing it offline | Reuters REvil gang shuts down for the second time after its Tor servers were hacked - The Record by Recorded Future Countries agree to fight ransomware together after White House meetings - The Record by Recorded Future CISA, FBI, and NSA warn of BlackMatter attacks on agriculture and other critical infrastructure - The Record by Recorded Future International community joins forces as ransomware attacks create major disruptions | PBS NewsHour US Treasury said it tied $5.2 billion in BTC transactions to ransomware payments - The Record by Recorded Future Stream when do we get on the beers cause i'm losing it by Candy Moore | Listen online for free on SoundCloud

Visit Solciety.co now! Welcome to the Solarpreneur podcast, where we teach you to take your solar business to the next level. My name is Taylor Armstrong and I went from $50 in my bank account and struggling for groceries to closing 150 deals in a year and cracking the code on why sales reps fail. I teach you to avoid the mistakes I made and bringing the top solar dogs, the industry to let you in on the secrets of generating more leads, falling up like a pro and closing more deals. What is a Solarpreneur you might ask a Solarpreneur is a new breed of solar pro that is willing to do whatever it takes to achieve mastery and you are about to become one.Speaker 1 (00:01):What's going on Solarpreneurs. I am stoked for this episode today because we have someone that, um, has been doing something pretty unreal. We've got a guy that's I got a text a few weeks ago, heard about chance here, that's with us today and you're going to hear all about them. But he said to my knowledge and industry record close 68 freaking deals in a month. Let's go appreciate it, bro. So we're going to hear, um, what was working for him and yeah, just kind of his whole story and what he's been doing to achieve that level of success. Cause I've never heard anything like it before, so a chance welcome to the show. Thanks for comingSpeaker 2 (00:40):On. Yeah, appreciate it. Thanks for having me.Speaker 1 (00:42):Yeah. So I'm stoked to hear just kind of what led up to this. And I I'll be honest. I didn't believe it. At first when I heard I got a text from my buddy Marshall and um, he told me about this guy and I'm like, no, no way, Marshall. I got a guy that I work with that close. I think, uh, yeah, he's closed like 43 in a month. I was just telling you chance. I, I thought that was the record. Like no ways anyone closing more than, than 43, that Marshall texts me. He's like, no chance got 68. I did. I was like, all right, I got to get this show on the guy on the show.Speaker 2 (01:15):Yeah. So a little background about me. Um, again, my name is chance [inaudible] but so ever since like I grew up, I've always had like an entrepreneurial mind, of course. But yeah, when I was 13, I, I mean, I've been doing e-commerce and everything. Solar I've been in the industry for about 15 months. Um, but yeah, literally I came out here, I saw an ad. Um, I actually filed Zane and that's the CEO of better earth. So that's who I work for. But I saw an ad with Zane and I was like, all right, I'm going to apply. So I think that they had like 800 applicants. It was all housing. Um, all expenses paid for, for your first three months. And I dropped out of school about a year and a half in. I never really wanted a piece of paper that millions of people get every single year, but I've flew out here.Speaker 2 (02:02):Uh, I got the job flew out here and I dropped everything to my parents, whether you guys like it or not, I'm going to California. So I'm originally from Wisconsin. But when I was about 13 years old, I had like $150,000. So she was my room. I used to buy and resell tickets, shoes, whatever it may be. You name it? I did it. So any type of get rich quick, and I realize, of course you just got to keep putting the work in. You said drop ship, like crazy. But when I was 17 or 16 years old, I created a company called smile, big clothing. So you guys can all check that out if you want. But yeah. Um, I was going through a little different time with my parents and they actually like got a divorce and everything was hard on me just cause my brothers I'm the youngest of, I have three older brothers, but, um, yeah, so I was going through it a little different time and I've made smile, be clothing, just cause I knew a lot of people are going through bigger struggles than myself.Speaker 2 (02:54):And that's why I dropped out of school. I used to sit in a dorm room, wake up at four 30, work on websites, inventory, skip all my classes, just to go package a bunch of merch. And it was, it was ridiculous. Yeah. So I dropped everything came out to California and when I came out to California, I didn't have a car or anything like that. I put all my money into the inventory. You still order way more than I could afford just to push my paradigm and make sure that I have to, I have one thing, one option. So, soSpeaker 1 (03:21):That's awesome. Always been hustling.Speaker 2 (03:23):Absolutely. So after that I came out to California, didn't have a car used to Uber, to turf used everything, my bank account to do it and you're out there. There's no other excuse to it. Um, you're either going to sit on the sidewalk, you're going to go bang doors and get after some people. So I did that. And my first month I think I had like 16 deals in my first month in solar. Um, and yeah, we were pushing a lot of paradigms in the company. I've kind of set like that company record right away. And ever since that, like I'm always just been holding myself accountable to my goals of course. And I don't really compare myself to others. I really just keep improving. And that's the most important part and staying consistent and making sure that you're outworking every single person. Yeah.Speaker 1 (04:03):A hundred percent. And yeah, I think that's probably, we'll talk more about this, but that's probably a huge key to success is most guys they'd hit, you know, 30, 40 in a month and be like, like I was thinking, oh, this is probably a record. I don't need to do more than this. Right. Go take a LondonSpeaker 2 (04:17):Person.Speaker 1 (04:18):Yeah. So I think that's probably a lot of your success. You're was like, oh no, I'm going to beat Mike previous record. So big. And you're probably not even looking at this score, but you're just comparing it to what you hit in the other months, right? Yeah,Speaker 2 (04:28):Absolutely. Yeah. I always have a big whiteboard in my room and that's literally all I compare myself to and it's just my last day or my last month or whatever may be. But staying in the present, taking every single day, one by one every day is a sprint always just control what you can of course can control and yeah, you definitely see a lot of results and a lot of improvement as well. Yeah.Speaker 1 (04:49):So we'll get more into the solar stuff, but yeah, I wanted to ask you chance, like with your previous businesses, was that always pretty successful when you started out selling shoes and everything? Tell me about that. Was that always a big success or was that kinda up and down? How did that go?Speaker 2 (05:02):You know, it was up and down just because I was in middle school and high school and of course, like I was playing sports. My dad used to be a coach and had a lot of practices after I used to get my phone taken away and all my classes. Cause I was always messaging people back on eBay. So that was always like a struggle and staying up super late and trying to balance work like schoolwork or whatever it may be. But yeah, the shoe is, was like actually a pretty big success, especially just being young. And honestly like, even if I broke even, I didn't like, I, I did pretty well for myself being young, but yeah. I mean, honestly just like the knowledge I learned and seeing how like businesses operate and seeing like how to actually sell definitely benefits in a lot of other ways are than just business. It develops yourself as a person too.Speaker 1 (05:45):Yeah. And so I love hearing about people's background because I mean, it's like we were talking about a lot of people in solar. It's kind of the paradigm that reps get is they're lazy. They close a few deals were paid at like we're talking about before we did the show and you literally close a couple of deals a month and be making more than most average Americans make. And so like, do you think, I don't know your upbringing, your previous as previous businesses, things like that. Do you think that contributed to just kind of like the hustle mindset you have now? Or do you think that was more developed or were you born in this way or how do you, how do you develop that stuff? MaybeSpeaker 2 (06:19):That's a good question. Um, honestly I've kind of just been born that way, but yeah, I've realized, so when I first came out here, it was called the 90 day blitz and I was like, dang, like, we're going to have to go like 13, 14, 15 hours a day or whatever it is, you wake up at six, you come back at until it's, I mean, pitch black. I used to literally knock my own neighborhood until it was literally pitch black. But yeah, I think that was huge for me. And I realized that I've kind of been blitzing for the last seven years. So yeah, I kind of just like trained myself to that and I didn't realize like, of course this is like more like physical or you're walking around, but I used to be on my computer for 12, 13 hours in college and high school or whatever it may be. So it definitely kind of like trained me just to kind of, of course bring it to the doors. Yeah.Speaker 1 (07:04):Yeah. I love that. And so yeah, at better, if, um, do you guys have like a, I dunno, a set schedule for your reps? Are you, you're talking about blitzes, do you guys do mostly Brit blitzes that better? Or how does the schedule work that you guys get your reps success over there?Speaker 2 (07:19):Yeah. Good question. So honestly like you you're on your own hours. Like no one really can like tell you exactly what to do. So we don't really have like a schedule, um, a lot of like team leaders or whatever may be, might make of course, like a blitz, like a two week blitz, three week blitz for like, of course their team and have like a cool goal or whatever may be like a competition. But a lot of it is just kinda like all self jammed, like work-based performance, like of course there's you get what you put out of it? Yeah.Speaker 1 (07:44):Okay. Yeah. Cause I'm noticing that a lot. Um, a lot of companies I've been with, they just have the typical, like, all right, guys, we're going to try to go out and work from, I dunno, three to seven today. It's kind of like the hours, but then what I've also seen with that is sometimes I get like mediocre results. Cause you're not doing like go hustle blitz. Everyone's just kind of trained. Okay. Let's try to get her, you know, one to two deals in the week, a hundred percent. If we work 20 hours on the doors or something, then we can probably get like one or two deals. Yeah. But especially recently effort from wat I guess, coming on the podcast is that they're really trying to do more. This blitz. You can get people to hit, you know, high numbers. So I think it's really cool is because if you do that, then a lot of times what I've seen on teams as reps limit themselves, like, okay, I'm going to work my 15 hours, whatever I'm the doors get like, I don't know, maybe four or five deals this month, which is great money.Speaker 1 (08:37):But what I think is really cool from guests, I'm hearing, okay, let's do like a three week blitz and then guys are hitting like, I don't know, maybe 15 deals. Whoa. That's crazy. I didn't know. That was possible. Yeah. So it's kind of less also breaking the mindset too, from what I've seen during the blitz and really like working more hours and they thought they could work and getting them out of the shell because yeah. I mean, I've kind of been in that, unfortunately I think it's time for me to do a blitz, but I've kind of been in the same thing, work three, four hours a day or at least knocked doors for four hours a day. I'm in San Diego. So I'm just working kind of local areas too. Yep. And so it's yeah, it's an easy trap to fall into is I'm just going to work like, you know, kind of, not the minimum, but minimum to get a couple of deals in a week and then not push myself further. So would you say that's a big thing for you guys to just um, or in anything you guys do to help reps just, I don't know, increase their potential and break past the limits, things like that. Yeah.Speaker 2 (09:29):100%. One thing that I do for like any type of my reps, like, um, so when I first started my team just cause I've only been in the industry for about say 15 months. So I brought four guys out, uh, probably my third month in and yeah, they were up at six o'clock I'd bang on their doors. Make sure that they're up, whether you're going to be reading meditating, of course, doing whatever that's going to get your mind. Right. Nine o'clock we'd have a morning meeting right after the morning meeting you're at the doors and you're not coming back until it's pitch black. There's no other way around it. Or is it going to be locked if you come back, you're not coming in. So sit in your car, do follow up whatever it is. But definitely like that was really, really beneficial. So I came in, his name was Andrew Zimmern, Betty.Speaker 2 (10:05):I came in, I lived with [inaudible] and that was big for me just to have like my routine of course down. And after that, like, yeah, like you're not going to waste your time. There's nothing worse than going to hit 200, 300 whatever. How many doors in a 10 hour period and just get nothing. So yeah, definitely just making sure like your health, like holding yourself accountable and everyone else, like being a leader, you got to lead from the front. If you're not doing results, why would I would really reps ever get results? So always pushing, like it's addictive. It's very, very addictive to push everyone else's paradigm, see like the results that you can bring to them and see, of course like when you do 60 ideals, anyone that does tens thinking, oh my gosh, like I got to get going. So it's the same thing with my team. And last, last month we did like 200 something like 200 deals as a, as a team, which is pretty coolSpeaker 1 (10:54):And credible. And yet you're what, you're 21, right chance 21 crazy say I'm here 28. And then like, man, these young hustlers coming out, it's funny, I'm on a team right now where at like my previous company as with a lot of us were like married guys and yeah. All right guys, let's close our deals, get back to their kids, whatever. But now I'm seeing these guys coming in like your age 21 and they're just like straight hustle.Speaker 2 (11:18):Right? Not as many distractions. Right. You don't have a full-time job at home with the kidsSpeaker 1 (11:22):Or whatever it may be. Yeah. Obviously you don't got, you know, wife, kid to give back to you, anything like that. Right. So I wish, I wish I would've been more like that when I was your age too though, is because like, even though I'm married now, I'm still probably working about the same as when I was like single and all that. Yeah. But if I get a go, if I could go back, I would have done like more blitz style when I had more time and I have like wife, whatever, to go back to your kids. Um, so for all like you young people listening, you young folks, not like me, 28, married and all that. I think that's a huge key push as hard as you right now, because trust me when you're married, when you have a kid and stuff, I mean, you're still gonna push, but it's not the same.Speaker 1 (11:58):Like you're not going to be able to push as hard. You still, still going to your bullets and stuff like that. But uh, I mean I'm already sleeping on them. Not like literally, but I'm already like in the doghouse sometimes with my wife, like I hate you putting too many hours this week, stuff like that. Right. So I think that's a big thing. Um, but yeah. So for you chance, like people you're training, um, do you ever get any pushback or is it like people you bring in, are you immediately saying, all right guys, we're blitzing like crazy or do you get pushback from guys? Like, cause I know obviously you're a straight killer out there working all these hours and like hitting massive things and all that. But what about for guys that aren't like you that come in, maybe they've never worked this hard in their lives. Um, anything else that you guys do to kind of like condition them to really hit big things and break those limits? Is it just kind of doing bootcamp style, like you said, or anything else that you guys?Speaker 2 (12:46):Yeah, I think like the biggest part is like setting the expectations straight right away. So, um, if they're getting pushed back or like, I don't see like them actually doing it, then I'm just not going to bring them out. So like I have a bunch of people that want to come out, but yeah, just making sure we're scaling, like of course, like a, a good pace and making sure where I can put a lot of time into these guys, but no there's really no pushback or anything like that, just because I tell them exactly how it's going to be. I tell them they have to get up at six and they're not gonna be able to come back. So they know that they're gonna be working 80 to 110 hour weeks. And that's literally the truth that month in August. Like literally I didn't have time to eat. So just being super busy and making sure that they are, and again, like that's my job. Like if they put the work in, I want to make sure that they like see a ton of results last month. Um, one of my top guys, Ryan testing her, he had like 28 deals. So it's definitely just shifting paradigms and yeah, everyone works really, really hard and yeah, we're just a big family. Yeah.Speaker 1 (13:40):That's awesome. And so let's get into a little bit like this huge month you had tell me, yeah. You talked about, you barely have time to eat in, obviously in the, you know, super busy, like 68 deals. I don't know how you do anything besides close deals. Um, literally, but yeah. Do you want to tell it, like, what was your schedule this month and did you kind of map it out? Yeah, yeah. Tell us about that. Oh, this came to be, yeah.Speaker 2 (14:02):So I mean, of course my goal is actually 70 for that month and I literally saw myself accountable. So I took a whiteboard and I literally broke down every single day and I just kept chasing myself if I'm behind, I gotta like catch up of course. But my schedule was of course I get up super early and since I do have guys, like, I seriously would not go to bed until like 12, um, going over like pitches wins. And of course, like voice memos, like you'reSpeaker 1 (14:25):Still managing a team while you're trying toSpeaker 2 (14:27):Live. Yeah. I live with 15 guys, so I brought in a whole entire squad. Um, we're actually in Orinda, California, but yeah, literally I was managing everyone and I went to Arizona. I was brand new market for me, which was definitely different. Um, but kinda crushed it there. I had 11 deals in four days there and then I ended up having to go to LA, um, which is a newer market for me as well. And I think I had like 14 deals in like five there. So yeah, just like adapting to whatever, like is put in front of you. And I did travel a lot, but it was pretty sick.Speaker 1 (14:58):Wow. That's awesome. So like 6:00 AM and then like you're talking about before is kind of like 6:00 AM then meaning at nine and then just like knocking her in Dale's still dark, dark,Speaker 2 (15:08):Basically a hundred percent. Yeah. And then I think like the most beneficial, like the, something that was really, really beneficial for me is a lot of times when people like get an appointment sad or like your time is so valuable, as I said before, but like, don't sit people out. Of course aren't worth your time. So I don't like force it anyone, like if I get, say I go get eight or nine calls in a day, like I'm probably saying only three or four of them. So I like that was super beneficial or don't like set appointments way out, same day or next day at all times. And yeah, your time is super, super valuable. Like I don't think I've ever sent an email with a proposal in my whole entire life. It's either you're going to sit with me or are you just not going to go solar with me?Speaker 1 (15:46):That's awesome. Yeah. No, I think that's a big thing. Especially new reps. I see they get kind of this, I dunno, we're rushed. They're having success when they get a lead and maybe, maybe they'll let them in the house and they're like chatting and stuff. But I know for me as a new rep, I would like get in the house with some old lady or whatever that wasn't even qualified for roofs, like wrecked. She has like zero credits, um, you know, on social security and all this stuff. And I like sweet. I just booked a solid appointment. She's letting me in. She's like telling me your whole life story. And then two hours later she's like, oh, by the way, a non-interest in solar have zero credit. I'm like all this stuff.Speaker 2 (16:21):Yeah. Just getting kind of all that stuff out of the way to start. And that, that is definitely like the most important part. A lot of people want to like set appointments that they just force it and they kind of waste an hour and a half, like an hour and a half on the doors. That's that's real are leads. So super, super important just to balance your time. And again, like, know that your time is that valuable in this industry? It's crazy. Like I always like to look at it as like an Easter egg hunt, right? Like literally you're just looking for golden eggs. So gain like the hardest people, like the hardest people look you out are the easiest to close. So yeah. And then of course like simple is better. So I don't really like talk about the product as much. Like I attach myself to the deal and from there, like it's really, really simple.Speaker 2 (17:01):You got dumbed down the deal, I'm big into analogies and just really putting into perspective on, I only can help your situation. This is something you've already been paying for. Like, all you do is you just pay a lower bill and that's kind of where it comes down to. And after that, like once it clicks or you get like that one objection or they like spilled the beans on why they didn't do it in the past and why it didn't make sense in the past. That's where you just actually tee that thing up and hit a home run.Speaker 1 (17:25):Yeah. I love that. And now I think those are super key to, um, just helping people dumb it down as in as simple as possible. I think the most successful guys I've seen in the industry are breaking it down. So basically like a third grader could understand it so simple. Like how could you not do it? Absolutely. Oh yeah. Tell me, do you have any, uh, specific analogies you use that you feel like you help a ton as you're sitting in homes? Yeah, justSpeaker 2 (17:49):Like a lot of the times, like even it's on a door most of the time like that, I really like dumb it down to them. But one thing that I always like to say is like, so of course, like they're already renting their power and now they get to own it for nothing on their pocket. But then I just like put it into perspective on like things that they actually own. Like you probably own this house. Right. And they go, yup. Probably on your car. Yup. I'll commute on your power. Pretty contradicting. You get to own your power for less than you already renting it for, for nothing out of your pocket. You literally could have $0, your name to do this. My mom always says when it sounds too good to be true, it usually is right. What's the catch there. Isn't one. And that's why I moved all the way from Antarctic, Wisconsin for this. Do you get your bill online or in the mail?Speaker 1 (18:30):The assumption love it. And that's awesome. Um, ask you. Yeah. And so like for guys, uh, another thing I was going to ask you a chance, um, especially for newer reps, like I've seen just recognize someone that's super qual qualified versus someone that could be possibly a waste of time. And when you close these 60 deals, I'm sure you had to be like, you know, super, super good with your time. So how did you recognize, like maybe someone's going to be waste of time or like filter all these people they're going to be felt credits or they weren't good leads. How do you filter out the holes and just spend your time with as many good prospectsSpeaker 2 (19:04):That's possible? Yeah. Good question. Honestly, I usually like kind of qualify them on the door. Um, I asked them questions after like get the bill and everything, like start talking about like them talk about myself for a little bit. And then after that, like if I see like something that may be a concern, like a roof or whatever it may be, I kind of get it out of the way, but what I usually do, and I dunno if this is like the best strategy I, it works for me eyes. I always just double book appointments. So ones that are like iffy or like, um, maybe they won't sit cause it was like a one-legged or whatever. It may be. Alice have two appointments. So I get to pick two, one doesn't show the other one's going to show. So that was something that I always like, felt like was really, really beneficial, but all our part two is just mapping out your schedule. So before I go out to Tara for out to of course, knock, I always have like my schedule plan. So like there on my time, I'm not on theirs. So making sure that I'm not putting an appointment right in the middle of the day, all my appointments would be at like 4 30, 6, 7 30 and nine. So that's where I always would do I have four appointments, I got to fill them. And that's what of course leaves you open for some same days. But yeah, that, that definitely works for me, um, at almost always. So yeah.Speaker 1 (20:10):Yeah. I like that. Yeah. I don't feel this in the grant cartoon much, but it talks about like new problems. It's good to have new problems. Right. Like double booking appointments. That's a problem. Right. But it's like, like, especially in solar, how many appointments fall through or stuff happens. Right. So I think, especially if you're trying to do volume like that, you gotta have like some deployments just stacked all day. Absolutely laying them up. And so for our doubters out there chance tells like how many of these were self Jan's? Um, like, I don't know. It's almost like a little bit of the stats forSpeaker 2 (20:40):Yeah. They were like almost all self gen. So, I mean like in Arizona, like that 11, um, was also of gen I one the days in California at six same days and our dad five same days and those are all door knocks. Um, and so Cal I had those 14, those were self gen, but yeah, like almost all of them, herself, Jen, of course I have to help some of my teammates like out, um, if they ever need like help on a deal, but a lot of times, like I'm always rescheduled in my appointments for it. So it's kind of like a sacrifice, but of course, like their production is more important to me than mine. So yeah, for sure.Speaker 1 (21:12):Yeah. That's a good mindset to have. And then how many, uh, at a little 60 for Lozar Blanco, bro, he probably had like 40 of those cancers come to St. John'sSpeaker 2 (21:21):When you think you can look at my commission summary. Um, now I've probably had like transparently. I probably had like six to eight, cancel, maybe nine. Um, and I mean, sometimes you just can't control it. Two of them were after the sites for it came out, the roof didn't qualify and they didn't want to do a reroof, but that's just the honest, transparent truth, but people can think whatever they want, I guess. Right. Yeah. You can shadow me on the door thenSpeaker 1 (21:43):That's seriously. We all go out the mouth. It's more video footage. Yeah. But no, that's still incredible. I mean 10% was at 10% or something maybe. Yeah. Thank you. Yeah. Appreciate that. Super incredible. Um, so yeah, but no, I like how you're still rep focused. Like most guys they're hitting huge numbers. I see like, all right guys, I'm not going to take any deals. I'm not like focused on anything and they just have to go all in. But the fact that you did this all while still like leading a team and training guys and still going to other people's deals and that makes it, uh, you know, even more incredible. It was crazy. Yeah. Um, what did your guys think when you hit this really? Like, just like, like blown away orSpeaker 2 (22:24):Not really? Um, or they saw,Speaker 1 (22:26):You said you hit been hitting some big numbers.Speaker 2 (22:28):Yeah. I was hitting like a lot of big numbers before and like I was really like rep focused then like a lot of times when you have new reps and they don't know exactly like what's a really good appointment or wherever it may be like there I'm sitting with people that would know show or I'm driving an hour just to go to an appointment that doesn't sit or whatever it may be. And I would still do like 40 some or whatever it may be. But yeah, a lot of my guys weren't really surprised. Like I set that goal in our morning meeting and I kind of like spoke it into existence. I like just telling someone or the team what I'm going to do. And then I already put it out to the world, so I have to do it otherwise I make a fool out of myself. SoSpeaker 1 (23:03):Yeah. That's awesome. And yeah. So like leading up to that, you had a few months, you were saying before we started this a few months at like 40, right. And yeah. And then you just like anything that led to you wanting to like hit 70 or just like, I'm just want to have a massive month and just go all out. I think the,Speaker 2 (23:18):I guess part is I didn't have like a brand new like waiver recruits. So a lot of these guys have been out here for like two months beforehand or two and a half months I would say. So like they were kind of on their own doing all like solo production where then I really got to like blitz on my own. Okay. So that was nice. And of course, like I had some people, like they I'd knock with a couple of people just for fun. Um, just cause they always like to knock with me and I always like to get some deals from, so yeah, again, like very unselfish. Like I love, literally change other people's lives. That's the most important part to me. And like, that's my why. And so the more people I can bring out, the more differences in life, second change, like that's a big, big win.Speaker 2 (23:56):It's not even about the dollar science, not about the deals, not about the numbers, nothing like that. It's about really just like of course, like switching their paradigm and changing their lives and their family's lives. And that's why my why's like to have a billion dollars to make a bit like a billion differences. So it kind of goes back to like smile, be clothing for every order we get, we donate a t-shirt to a child in the hospital. So yeah. So I've always just been a big giver. My mom's always taught me that and we used to sacrifice our Christmases for other people's Christmases. So it really is just like, that's the coolest part about this industry is like, we only can help someone out, literally every single house that you see. It makes sense for them unless they sit in the dark with like a flashlight. Right. They're going to have a high enough electric bill. Yeah,Speaker 1 (24:33):Yeah, yeah, no, I think it's yeah. All the successful people. I know they have sort of that same mindset and Hey, you probably read some of those books, you know, like the Zig Ziglar, Zig Ziglar and you know, greatest salesman in the world. It's not their book. And that's something that key points of it is like the more you can care about other people, the more you're going to get back, the more deals you're going to get back in your life. Absolutely. So big key. I think Fran, when Jane have success, don't like, look at it. How many people can you serve this this month? How many people can you help? How many lives can you change? And uh, I think that's, there's a big thing speaking to in existence. Do you like your saying? Absolutely. Um, and so yeah, another question I just thought of too, like you mentioned, you're double booking appointments, things like that. So let's say you have you show up to one, they sit then the other one. What do you do with the other one? I just show them what time.Speaker 2 (25:20):Yeah. It's true. My texts really quick and say like, Hey, my meeting's going a little bit longer than already. Like of course it plans to say, I didn't have an appointment after the seven o'clock work or seven o'clock probably will work for you. Right. And then after like a lot of times, or I can just reschedule it for like the next morning, like 8:00 AM or at late night, again, like nine o'clock, eight o'clock. So I always like to set up appointments early, early morning or late at night. So I always want to keep that window where I can go hit some doors. Yeah. Okay. Pipelines always got before.Speaker 1 (25:46):Yeah. That's awesome. And so yeah, for, uh, same days, how, yeah. Um, to go into a little bit of like your clothes, how long would you say your typical clothes works? And what's kind of like your closing process, if you don't mind sharing?Speaker 2 (25:59):Um, honestly I would say my typical clothes is like 30 to 35 minutes. Okay. So I get to the numbers, um, probably in like five minutes and after the numbers, I'm probably getting to the forms in 20 minutes tops. So keeping it really simple, like we were already talking about, but yeah, when you keep it simple, like once you're like a lot of people like to overexplain the deal and you're just shooting yourself in the foot and you're over analytical then of course like you keep telling him all this like nonsense then of course they're going to say, yeah, I might need to do some more research instead. You're just paying a lower bill. Hopefully don't have a problem with building equity in your property, owning your power instead of renting it. And that will be good. So yeah. That's like how long my closing takes, but yeah, I've talked about like a couple of main points right away.Speaker 2 (26:40):Um, probably like for five minutes, just like information, um, just to give me like a better feel. What's your biggest concern? What's your biggest goal? Have you ever looked in going solar? And a lot of people, what they do is like they, oh, my biggest concern is I don't want to lease my power. I don't want to pay anything out of pocket. Like, yeah. I wouldn't want to either. And I keep going and I want to use all of that as ammo. Like once I get to like the numbers, after that, I can just tee up whatever, like of course there, my biggest concern was, and that's how I go right into my clothes. Really like harping on the things that they're concerned about. And then from there keeping it simple. It's just like, this is an absolute, no brainer. Nice.Speaker 1 (27:13):So, so you don't go, like, I don't know. What about for people that are like have checked out solar are more analytical, things like that. Do you change your presentation much? Or is it pretty much the same with what those guys do? I go chance we've gotten like four quotes now. Like just show me the numbers, whatever.Speaker 2 (27:29):Yeah. So I'll never go right to the numbers no matter what. Um, I always just tell them like, we'll get to your number. So like, if they say like they have three or four quotes or whatever, it may be right away. What I'll always say is like, okay, like I'm going to keep away from the whole entire, like sales spiel. But I do want to go over like, like a couple of things. We'll take two minutes, we'll get right to your numbers. Absolute like that. They're actually open-minded. But a lot of times, even when people get quotes, they don't even know what like net energy metering is or whatever, how everything works. And then they ask that and literally just like kills like your momentum and the deal. Yeah. So that's something where I get everything on the way they have to have full understanding before I show them their numbers. After of course the numbers are going to make sense. I'll tell them why we're different. And then from there I'm going to reach the close.Speaker 1 (28:08):Nice. Nice. So it sounds like even before you even get an appointment, it sounds like they already kind of pretty good, pretty good idea of like how the process works, how the solar works and like, like with your analogies, things like that. Cause you don't, you don't get into a ton of like descriptive stuff in the presentation. A lot of that stuff they know before.Speaker 2 (28:25):Yeah. Um, they know like a lot of things, like, I don't really like get into like equipment as much. Of course I'll tell them like how many like modules I have and I'll tell them like the brand of the inverter, but from there, just like, honestly I could put a stuffed animal on your roof. And since he ever preferred like a production or performance guarantee, like either he doesn't matter, like the system's going to produce. So that's where like I kind of stay away from that kind of stuff. But yeah, I kind of get right to the numbers right away. And then it's just going for the clothes. But honestly their clothes, every, almost every single time I sit down with.Speaker 1 (28:56):Yeah. The big part is just your mindset. I can tell you, like I can tell just from the way you're talking, like every deal you sit in, it's like, you're, you're telling yourself they're close. As soon as a slam dunkSpeaker 2 (29:06):Every, every time. Like if you have any doubt or like any doubt in your mind that you're going to a deal and they're not going to close, like yeah, you just spoke that into existence. Every single time I go to a door or every single time I go to like a meeting. Yeah. Like my number one, like my mindset slurry, just like you're closed. Yeah. Like yesterday, like for instance, both of my people were like, yeah, we're just looking around. Um, we're not gonna sign any papers. And after him, I was just like, you know, tell me that in about 45 minutes, buddy. Exactly.Speaker 1 (29:34):Yeah. It's a Champion's mindset and every high producer has exact same mindset that you have, um, take taking their McCarthy. Um, yeah. He came on the show and he's seen a lot of the same stuff that you said, it's like this one's going to close a hundred percent, a hundred percent. This one's closed. He'd snapped him as he's walking up to the doorstep. Absolutely. Just based on stuff. So it's like, he's speaking in existence. You have that. And then yeah. Another thing that, uh, I can grant Cardone talks about too, you're just looking at, um, stuff that they've already bought. It's like they were bought this nice car. They already bought a house like white, white shouldn't they buy their electricity. Why shouldn't they own their lectures direct. Yeah. They're not going to do it. Yeah.Speaker 2 (30:08):Yeah. Literally rather like you're already paying for it, you know, literally just makes no sense not to do it. Yeah. So,Speaker 1 (30:16):And so if you're, that's another key for our solar printers listening, if you're not like a hundred percent sold on your product and a hundred percent sold that you're helping people out a ton and just like, don't have that complete confidence, just figure out a way to get that. Cause that's going to be the difference maker. Like probably for you. I'm going to guess that like when you first started, you closed, you said you closed 16 deals, right? Yeah. You probably know new, barely anything. It's the lawyer at that point, right.Speaker 2 (30:40):At you, 70% of the company better earth. And that was more about sorta to me stale. So yeah, literally you don't have to get like super descriptive and it makes sense. So that's all it matters.Speaker 1 (30:52):Yeah. Yeah. So that's the mindset you got to have for our listeners. Figure out a way to develop that and that's going to help you have this success theater trying to hit for sure. And so at chance, tell us, um, like for people that are having a high cancellation, I know that's another thing we deal with. Sometimes that a lot of reps, especially in California, it's been hit so hard. There's like so many comparisons, anything you do that helps to have like have so many of these push through and actually go to install. Yeah.Speaker 2 (31:18):I mean, there's always like a couple of things it's always different. Like for every like situation, but one thing maybe like take a picture of the homeowner Senate right after, just to like, of course, like when they go to your text messages, like they see them smiling. But another thing too is like, stay at the house, like build value with them. Like they can't like think you're actually selling something. So if I put a girl Pearl on my body, every single time that I do in FC, after I leave it, literally we're friends. So it's not like, oh, thank you for helping with the solar. It's just like, thanks brother. You're the man. You know? Like that's just kind of how it is. And it's just like, I always like put it into perspective, like talk to people like they're your best friend? Like your parents' best friend.Speaker 2 (31:52):Yeah. So doing that, it's just like, your tonality is not sales. Like my tonality and the FC is just like this. I sit back, relax, whatever it may be. But yeah, I think the biggest part is of course like do follow up, but don't do too much. And with it, take a picture with them at the very end. So they always have that. And then in our thing I was to do is just of course, like we still got to make sure the site is going to qualify too. So if the site doesn't qualify, like literally you're Sol you know, then you're stuck with PGNE. But as long as you don't have a problem with paying a lower bill, bill and equity, and of course using more electricity instead of chucking it down the gutter every month, like, man, this makes sense. Right. Nice. Yeah, absolutely. And then from there, yeah. Then I'll maybe bring up a testimonial, provide them with a reference if they really need it, but you always have like that feel. But a lot of times, like the people that I have, like QL a lot of times are people like no one else really like will QL. So it's clearly new to them. Qualified lead. Yeah.Speaker 1 (32:45):Yeah. Okay. No, that's helpful. And so you're saying you're taking a picture with them and then texting it to the homeowner right after and just seeing like congrats, something like that.Speaker 2 (32:53):Yeah, absolutely. You can't wait to start saving your wallet in the planet. Nice. Let's get to it, you know, and then with it, other than that too, sounded like a $5 Venmo. Here's a taste of your solar savings, whatever it may be. Yeah.Speaker 1 (33:03):That's cool. That's big. And yeah, a lot of people, uh, you know, do the videos. I mean video picture, that's a big thing. That's been a game changer too is because you can get like a video or a picture. Um, I mean, especially if they're explaining back to you, like why solar is a good thing. Yup. Like sums in hell with me. Hey, let's grab a quick video. Just tell me like, if you like the experience, whatever, it's be like 30 seconds. Yeah, yeah. Absolutely. Like, sure. And then they're having to sell me again on why they even wanted to go like solar in the first place. Yep, absolutely. So, yeah, that's huge. And then like, um, another big thing you're seeing are same day appointments. So I get a lot of questions asked about that from people I go, how do you, I think that's kind of a newer thing. I didn't like when I first started in the floor, no one that I knew of was doing same day appointments. Yeah.Speaker 2 (33:47):That was the same thing with better earth. My first starting to, we were definitely a lot smaller and I remember people were posting like same-day Q or something like that. And when I first started, but um, same days and everything, like, it's not too difficult. Like of course, like make sure like both decision makers are there, even if they're not like a one-liner and everything, like, they're not hard, but with it, like, I always just like say, Hey, I'm going to send this over to our engineers. We'll be back in 30 minutes. You're free. Right. Yep. Okay, cool. And I just go right back. Um, every single time I don't really like same, like knock it and go right in. I do that sometimes, but a lot of times, like, I want to be like, Hey, I'm actually going to take about 30 to 45 minutes to drop your custom plan.Speaker 2 (34:22):So it doesn't look like I just like put these panels on their roof. Where the heck did you just get your numbers? It like makes it look like I put a lot more time into their plan. So that's why I do that. Um, I realized like, and then of course, like I'll send them like give them like the website. They can look at like this little training course, just so they have a better feel for it. And then I come back and like, they're somewhat knowledgeable about it. Or they looked at our reviews or testimonials or whatever it may be. Okay. But I don't get business cards or anything like that.Speaker 1 (34:47):Yeah. That's I think a big camp scene dues, just like there's something Missy did you're Hey, you're going to be, you'll be here 30 minutes. Right. Someone will take me. Cause like, especially reps that I see newer reps, I'm training and stuff like that. I go, what am I getting? Not getting same days. They're going. They never free. But the way they're asking me is like, Hey, are you guys going to be here? Like here? But like notice the difference. You're just like, you're, you'll be here 30 minutes. Right. We'll be right back with it. Absolutely. That most people are going to be here if that's the only option in their mind. Right. Cause you didn't saySpeaker 2 (35:16):Right. And then also like of course, like put it like bill and scarcely behind it. Like we're only picking out two more homes than the zip code. So like, this is a big opportunity. Like again, I don't want to waste your time. I don't want to waste my time. But like hopefully you guys are on the right rate schedule. If you're not. Or if it's like this doesn't make sense. I'll just shoot you a text and say, Hey, this is not going to work for you. But I'm actually meeting with the Benson's like four homes down. You probably even know them. They always walk the German shepherd at seven. But with that, I'm actually going to be back at seven or eight, which one works best. And just like a multiple choice answer. Like they have to pick one of them. It's not your, or you forget seven. Now they can make any excuse. You know, like now I have to go to my kid's game or whatever it may be. Yeah.Speaker 1 (35:54):Yeah. That's awesome. And then you get the takeaway in there too. Only picking two more homes in the zip code on the street, um, are big, so chance, some awesome stuff. You've been, uh, Sharon Yan. I can see why you're having success. It's just like all these little things, nothing like crazy think people think they're going to hear some crazy lanes you're using her. Um, some of the same, but really I think it just comes down to you're working more hours than I think probably anyone has in a month. And so it was like, I don't know. Do you, do you have an idea of how many doors you knocked or how many total hours you put in that month? If you had to guess?Speaker 2 (36:28):I mean like doors are knocked like a lot hours, a ridiculous amount. Like seriously. Like I didn't do anything fun, you know? Like I just worked and like, I love doing that. And I wanted, like I told myself, I want 70, I fall short by two, but yeah, it was a ridiculous month, but honestly I'd say doors wise, I don't really have to hit like a crazy amount of doors. It sounds like kind of dumb, whatever it may be. But yeah, I really don't have to hit that main doors. Like with Ryan, like testing her, we went and knocked and I think we knocked for probably 45 minutes and we had four same days and like, um, but yeah, like when you, when you said that too, it's just like, yeah, everything's super simple. The biggest part about anything is just attaching yourself to the deal.Speaker 2 (37:09):Like if it was the company there, there's a reason why reps have four and one has 40 or one has 10. Someone has 60, whatever it may be. It literally just comes down to your conviction and how much you believe in the product and how much you believe in yourself. Once those two things add up and they're in line, there is no reason you can't like double or triple your production that you're doing right now. And so that's the biggest part. And that's the thing that I knew I could control. It was like how hard I'm going to work. And every single day, like there's not one person like out there that will outwork me. And I work when others don't I knock when others don't and that's why you get those results. But yeah, it's super simple, but yeah, just dumbing down the deal, keeping it, like you literally could tell, like tell us like a third grade and they would do it. You know? So honestly that is something like I found myself likeSpeaker 1 (37:56):Pretty good at it. Yeah. No, no doubt about that. And I mean, in your you're working like zero days off in the month, right? Third 30 days, same schedule 6:00 AM till dark every day for that month. So usually every daySpeaker 2 (38:09):And that's, that's the truth. I mean, sometimes like of course, like around the holidays or maybe there's a day. Um, but like when it comes to a blitz, like my first 90 days I did not take one day off. Yeah.Speaker 1 (38:20):Yeah. And so like with your guys' blitzes, um, cause I mean that's crapped on hours that guys are working, so I'm sure, you know, guys, when you do your blitzes, you're working straight hard, but yet for you, do you have anything like to re kind of rejuvenate yourself or you get yourself back in the flow? Cause um, I mean, I don't think you're gonna work that day. That schedule, I would see like 365 for sure. In a year, but like what do you do on your off days? Let's say your months, you're not trying to hit 70 anything you do to kind of like refresh yourself and yeah. Get yourself back back to new, to do another blitz or anything. Yeah. That'sSpeaker 2 (38:54):Um, on my off days and everything, like, I'm still like doing a lot of personal development, like helping everyone else, but maybe I'll go hit the links. Maybe I'll go get a round of golf. And um, I really don't golf much anymore. I used to work at a country club back home. So I used to golf every day when I was like in high school. But, or like in the summer days, like I used to play a little bit of golf, just free golf, but yeah, maybe like going out to dinner with some of the guys, um, like top producers hanging out with some friends like here and with that, like honestly, like it doesn't really feel like a job just because my house is like a college house. It seems like, well, we just all have like that same mindset where it's not like drinking and stuff. Like no one drinks in her house. Like of course, like we'll have one night out or whatever it may be in a month. Yeah. But yeah, it's just like everyone just surround yourself with people that have the same mindset and like gets fun.Speaker 1 (39:43):15 dudes you got now it's yeah. It's aSpeaker 2 (39:45):Pretty big house, but it definitely got a little more crammed with, so we're actually splitting back up. So like two Airbnb's. Okay.Speaker 1 (39:51):That's awesome. Well, good stuff, man. Will a chance. Yeah, you're crushing it. Dude ends. We appreciate your having on the podcast before you kind of start wrapping up here. Do you want to tell guys where anything can kind of connect with you more or find you on social media and all that good stuff? Yeah.Speaker 2 (40:04):Yeah. My Instagram is chance Pran. So C H a N C E P R O N. And that's kinda like my main thing. So yeah. Find me on Instagram or you can file smile, be clothing on Instagram, wherever it works for me. But yeah. Anyone on this podcast appreciate you guys listening. And from there, like if you guys ever have any questions or interested in having like mentorship, wherever may be like, you guys can always reach me there. Yeah.Speaker 1 (40:27):Yeah. He'll probably be on the doors when you hit him up. Probably. Huh? Maybe it'll take a couple minutes. It's funny. That's incredible, dude. And um, yeah. For your clothing, do you guys, uh, do you want to tell us a little bit about your clothing stuff? Is that a like, well door knockers, like this clothing or you got any ed door knocking gear or anything like that yet?Speaker 2 (40:44):Yeah. I mean like smile big, like your solar savings. Get it like for like, of course, like a family, um, on install day. I mean literally and your smile is contagious, so yeah, like, honestly it's it's playful brand. It gets pretty sick. Like it's all comfy clothes, all the best like materials I used, I went to Las Vegas to make sure I got all the best materials. It's all American made. But yeah, I honestly, I think like our clothing is literally good for anyone, any age, any gender, whatever it may be.Speaker 1 (41:12):And it's your customers or anything like that? She saw the sun smiling or something. Yeah.Speaker 2 (41:17):Or shirt on smile big. But yeah, a lot of that stuff, like I had designed all of it and definitely being able to put a lot more capital into it and we got a lot of big things coming, so hopefully everyone will follow along. Yeah.Speaker 1 (41:27):Oh cool man. So just before we wrap up here, what's next for you, man? You, you trying to hit past that goal or surpass your goal or any big thingsSpeaker 2 (41:34):We've got coming. So I took like the new position with better earth. So now I'm like the VP of sales. And um, honestly, like right now I'm going to be kind of going back and forth to Arizona. So Cal Norco with my team and of course like training a lot more people in the company. Definitely like my number one goal is to have every single person on my team, in the company to one beat the record, but to everyone needs to get into double digits. So that's kind of what I'm going to be focusing on. Yeah. Like I'll probably end up like having a lot of deals, but I'm going to do a lot of stuff where I'm not going with them trying to like, of course, like shift paradigms and make sure that they're going to be financially free as well. I mean, that's literally the most important part for me. So yeah, after that. Yeah. Maybe one, one of those months, I'll try to break it, but from there, I'm going to make sure that I can keep focusing on my guys. Yeah.Speaker 1 (42:17):That's awesome. So, yeah, I don't, I think that record or stand for a little bit for our listeners, I guess, let us know if you hear of anyone that's creeping up close and then we'll tell chance, say, man, you gotta, you gotta up there.Speaker 2 (42:28):If someone is someone broke it though, then, then yes, next month, the month that I'm going to go get it. Okay. Yeah.Speaker 1 (42:33):We'll be in touch, but I cool. It's a chance. Appreciate you coming on the show. And then last question. Do you have any, like, I dunno, final words or anything like for a newer rep that you, uh, would want to leave with them before we end here? Yeah.Speaker 2 (42:47):At work every single person, like do the personal development, listen to yourself on like voice memos. Like sometimes like you're always gonna be saying like a word that you don't even notice. It's just like your unconscious competence. So that's one thing. Another thing that I always just like to say to you is just like, stay level-headed like, again, control what you can like control you. Can't control who's behind that door, but you can control your mindset. So every like the doors are just a game. So honestly, my number one thing was keep the door open as long as possible that any other rep couldn't. So that's one thing for me that was really, really beneficial. But the other part too is just having that conviction. You gotta be able to find like that passion behind the product and you have to believe in yourself and literally attach yourself to the deal. If you see like your FCS or like your closings going about like an hour, hour and 15 hour and 30. Yeah. Shorten that up. Literally shorten it up. I promise you, you will see a lot more results. Yeah.Speaker 1 (43:37):Love that whole cool chance. So yeah, Solarpreneurs make sure you're keeping it super simple and make sure you're outworking anyone. Cause that's really what it takes to succeed at the level of that. I'm sure you want to hit in the industry. It's what Coby Bryan was dealing. So Michael Jordan, all the greats they're putting in the hours and they're combining an app with their SQL and then that's how we're going to have success. So thanks again, chance for coming on. We'll keep in touch and make sure you hit them up. Let them know. You're grateful for him coming on the show today and we will talk with everyone soon. Appreciate it, man. ThanksSpeaker 2 (44:07):So much, Taylor. Appreciate you bro.Hey, Solarpreneurs quick question. What if you could surround yourself with the industry's top performing sales pros, marketers, and CEOs, and learn from their experience and wisdom in less than 20 minutes a day. For the last three years, I've been placed in the fortunate position to interview dozens of elite level solar professionals and learn exactly what they do behind closed doors to build their solar careers to an all-star level. That's why I want to make a truly special announcement about the new learning community, exclusively for solar professionals to learn, compete, and win with top performers in the industry. And it's called the Solciety, this learning community with designed from the ground up to level the playing field to give solar pros access to proven members who want to give back to this community and help you or your team to be held accountable by the industry. Brightest minds four, are you ready for it? Less than $3 and 45 cents a day currently Solciety is open, launched, and ready to be enrolled. So go to Solciety.co To learn more and join the learning experience. Now this is exclusively for Solarpreneur listeners. So be sure to go to solciety.co and join. We'll see you on the inside.

About EdEd Boyajian, President and CEO of EDB, drives the development and execution of EDB's strategic vision and growth strategy in the database industry, steering the company through 47 consecutive quarters of recurring revenue growth. He also led EDB's acquisition of 2ndQuadrant, a deal that brought together the world's top PostgreSQL experts and positioned EDB as the largest dedicated provider of PostgreSQL products and solutions worldwide. A 15+ year veteran of the open source software movement, Ed is a seasoned enterprise software executive who emphasizes that EDB must be a technology-first business in order to lead the open source data management ecosystem. Ed joined EDB in 2008 after serving at Red Hat, where he rose to Vice President and General Manager of North America. While there, he played a central leadership role in the development of the modern business model for bringing open source to enterprises.Links:EDB: https://enterprisedb.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate: is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards, while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other, which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at Honeycomb.io/screaminginthecloud. Observability, it's more than just hipster monitoring. Corey: This episode is sponsored in part by our friends at Jellyfish. So, you're sitting in front of your office chair, bleary eyed, parked in front of a powerpoint and—oh my sweet feathery Jesus its the night before the board meeting, because of course it is! As you slot that crappy screenshot of traffic light colored excel tables into your deck, or sift through endless spreadsheets looking for just the right data set, have you ever wondered, why is it that sales and marketing get all this shiny, awesome analytics and inside tools? Whereas, engineering basically gets left with the dregs. Well, the founders of Jellyfish certainly did. That's why they created the Jellyfish Engineering Management Platform, but don't you dare call it JEMP! Designed to make it simple to analyze your engineering organization, Jellyfish ingests signals from your tech stack. Including JIRA, Git, and collaborative tools. Yes, depressing to think of those things as your tech stack but this is 2021. They use that to create a model that accurately reflects just how the breakdown of engineering work aligns with your wider business objectives. In other words, it translates from code into spreadsheet. When you have to explain what you're doing from an engineering perspective to people whose primary IDE is Microsoft Powerpoint, consider Jellyfish. Thats Jellyfish.co and tell them Corey sent you! Watch for the wince, thats my favorite part. Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Today's promoted episode is a treasure and a delight. Longtime listeners of this show know that it's not really a database—unless of course, it's Route 53—and of course, I don't solve pronunciation problems with answers that make absolutely everyone hate me. Longtime listeners of the show know that if there's one thing I adore when it comes to databases—you know, other than Route 53—it is solving pronunciation holy wars in such a way that absolutely everyone is furious with me as a result, and today is no exception. My guest is Ed Boyajian, the CEO of EDB, a company that effectively is the driving force behind the Postgres-squeal database. Ed, thank you for joining me.Ed: Hey, Corey.Corey: So, I know that other people pronounce it ‘post-gree,' ‘Postgresql,' ‘Postgres-Q-L,' all kinds of other things. We know it's decidedly not ‘Postgres-squeal,' which is how I go for it. How do you pronounce it?Ed: We say ‘Postgres,' and this is one of the great branding challenges this fantastic open-source project has endured over many years.Corey: So, I want to start at the very beginning because when I say that you folks are the driving force behind Postgres—or Postgres-squeal—I mean it. I've encountered folks from EDB—formerly EnterpriseDB—in the wild in consulting engagements before, and it's great because whenever we found an intractable database problem, back at my hands-on keyboard engineering implementation days, very quickly after you folks got involved, it stopped being a problem, which is kind of the entire point. A lot of companies will get up there and say, “Oh, it's an open-source project,” with an asterisk next to it and 15 other things that follow from it, or, “Now, we're changing our license so the big companies can't compete with us.” Your company's not named after Postgres-squeal and you're also—when you say you have people working on it, we're not talking just one or two folks; your fingerprints are all over the codebase. How do you engage with an open-source project in that sense?Ed: First and foremost, Postgres itself is, as you know, an independent open-source project, a lot like Linux. And that means it's not controlled by a company. I think that's inherently one of Postgres's greatest strengths and assets. With that in mind, it means that a company like EDB—and this started when I came to the company; I came from Red Hat, so I've been in open-source for 20 years—when I came to the company back in 2008, it starts with a commitment and investment in bringing technology leaders in and around Postgres into a business like EDB, to help enterprises and customers. And that dynamic intersection between building the core database in the community and addressing customer needs in a business, at that intersection is where the magic happens. And we've been doing that since I joined EDB in 2008; it was really an explicit focus for the company.Corey: I'd like to explore a little bit, well first and foremost, this story of is there a future for running databases in cloud environments yourself? And I have my own angry, loud opinion on this that I'm sure we'll get to momentarily, but I want to start with yours. Who is writing their own databases in the Year of our Lord 2021, rather than just using whatever managed thing is their cloud provider of choice today is offering for them?Ed: Well, let me give you context, Corey, because I think it matters. We've been bringing enterprise Postgres solutions to companies now, since the inception of the company, which dates back to 2004, and over that trajectory, we've been helping companies as they've done really two things: migrate away, in particular from Oracle, and land on Postgres, and then write new apps. Probably the first ten of the last 13 years since I've been in the company, the focus was in traditional on-prem database transformations that companies were going through. In the last three years, we've really seen an acceleration of that intersection of their traditional deployments and their cloud deployments. Our customers now, who are represented mostly in the Fortune 500 and Global 2000, 40% of our customers report they're deploying EDB's Postgres in the cloud, not in a managed context, but in a traditional EC2 or GCP self-managed cloud deployment.Corey: And that aligns with what I've seen, a fair bit. Years ago, I wound up getting the AWS Cloud Practitioner Certification—did a whole blog post on it—not because it was opening any doors for me, but because it let me get into the certified lounge at re:Invent, and ideally charge a battery and have some mostly crappy coffee. The one question I got wrong was I was honest when I answered, “How long does it take to restore an RDS database from snapshot backup?” Rather than giving the by-the-book answer, which is way shorter than I found in practice a fair bit of the time. And that's the problem I always ran into is that when you're starting out and building something that needs a database, and it needs a relational database that runs in that model so all the no SQL options are not viable for whatever reason, great, RDS is great for getting you started, but there's only so much that you can tune and tweak before you start to run into issues were, for particular workloads as they scale-out, it's no longer a fit for a variety of reasons.And most of the large companies that I work with that are heavily relational-database-driven have either started off or migrated to the idea of, “Oh, we're going to run our own databases on top of EC2 instances,” for a variety of reasons that, again, the cloud providers will say, “Oh, that's not accurate, and they're doing the wrong thing.” But, you know, it takes a certain courage to tell a large-scale customer, “You're doing it wrong.” “Well, why is that?” “Because I have things to sell you,” is kind of a terrible answer. How do you see it? Let's not pick on RDS, necessarily, because all of the cloud providers offered managed database offerings. Where do those make sense and where do they fall down?Ed: Yeah, I think many of our customers who made their first step into cloud picked a single vendor to do it, and we often hear AWS is been that early, early—Corey: Yeah, a five-year head start makes a pretty compelling story.Ed: That's right. And let's remember what these vendors are mostly. They are mostly infrastructure companies, they build massive data centers and set those up, and they do that beautifully well. And they lean on software, but they're not software companies themselves. And I think the early implementation of many of our customers in cloud relied on what I'll call relatively lightweight software offerings from their cloud vendor, including database.They traded convenience, ease of use, an easy on-ramp, and they traded some capability in some depth for that. And it was a good trade, in fact. And for a large number of workloads it may still be a good trade. But our more sophisticated customers, enterprise customers who are running Postgres or databases at scale in their traditional environments have long depended on a very intimate relationship with their database technology vendor. And that relationship is the intersection of their evolving and emerging needs and the actual development of the database capabilities in support of that.And that's the heart of who we are at EDB and what we do with Postgres and the many people we have committed to doing that. And we don't see our customers changing that appetite. So, I think for those customers, they've emerged more aware of the need to have a primary relationship with a database vendor and still be in cloud. And so I think that's how this evolves to see two different kinds of services side-by-side, what they really want is a Database as a Service from the database vendor, which is what we just announced here at Microsoft Ignite event.Corey: So, talk to me a little bit more about that, where it's interesting in 2021 to see a company launching a managed service offering, especially in the database space, when there's been so much pushback in different ways against the large cloud providers—[cough] Amazon—who tend to effectively lose sleep at night over the haunting fear that someone who isn't them is making money, somehow. And they will take whatever is available to them and turn it into a managed service offering. That's always been the fear, so people play games with licenses and the rest. Well, they've been running Postgres offerings for a long time. It is an independent open-source project.I don't think you can wind up forcing a license change through that says everyone except big companies can run this themselves and don't do a managed service with it because that cat is very much out of the bag. How is it that you're taking something to market now and expecting that to fare competitively?Ed: So, I think there's a few things that our customers are clearly telling us they want, and I think this is the most important thing: they want control of their data. And if you step back, Corey, look at it historically, they made a huge trade to big proprietary database companies, companies like Oracle, and they made that trade actually for convenience. They traded data to that database vendor. And we all know the successes Oracle's had, and the sheer extraordinary expense of those technologies. So, it felt like a walled garden.And that's where EDB and Postgres entered to really change that equation. What's interesting is the re-platforming that happened and the transformation to cloud actually had the same, kind of, binding effect; we now moved all that data over to the public cloud vendors, arguably in an even stickier context, and now I think customers are realizing that's created a dimension of inflexibility. It's also created some—as you rightly pointed out—some deficiencies in technical depth, in database, and in software. So, our customers have sorted that out and are kind of coming back to middle. And what they're saying is, “Well, we want all the advantages of an open-source database like a Postgres, but we want control of the data.”And so what control looks like is more the ability to take one version of that software—in our case, we're worrying about Postgres—and deploy the same thing everywhere they go. And that opens the door up for EDB to be their partner as a traditional on-prem partner, in the cloud where they run our Postgres and they manage it themselves, and as their managed service, Postgres Database as a Service Provider, which is what we're doing.Corey: I've been something of a bear on the idea of, “I'm going to build a workload to run everywhere in every cloud provider,” which I get. I think that's generally foolish, and people chasing that, with remarkably few exceptions, are often going after the wrong thing. That said, I'm also a fan of having a path to strategic Exodus, where Google's Cloud Spanner is fascinating, DynamoDB is revelatory, Cosmos DB is a security nightmare, which is neither here nor there, but the idea that I can take a provider's offering that even if it solves a bunch of problems for me, well, if I ever need to move this somewhere else for any reason, I'm re-architecting, my data model and re-architecting the built-in assumptions around how the database acts and behaves, and that is a very heavy lift. We have proof of that from Amazon, who got up on stage and told a story about how much they hate Oracle, and they're migrating everything off of Oracle to Aurora, which they had to build in order to get off of Oracle, and it took them three years to migrate things. And Oracle loves telling that story, too.And it's, you realize you both sound terrible when you tell that story? It's, “This is a massive undertaking that even we struggle with, so you should probably not attempt it.” Well, what I hear from that is good God, don't wind up getting locked into a particular database that is only available from one source. So, if you're all-in on a cloud provider, which I'm a fan of, personally—I don't care which one but pick a cloud provider—having a database that is not only going to work in that environment is just a reasonable step as far as how I view things. Trading up that optionality has got to pay serious dividends, and in many database use cases, I've just don't see it.Ed: Yeah, I think you're bringing up a really important point. So, let's unpack it for a minute.Corey: Please.Ed: Because I think you brought up some really prominent specialty database technologies, and I'm not sure there's ever a way out of that intersection and commitment to a single vendor if you pick their specialty database. But underneath this is exactly one of the things that we've worried about here at EDB, which is to make Postgres a more capable, robust database in its entirety. A Postgres superpower is its ability to run a vast array of workloads. Guess what, it's not sexy. It's not sexy not to be that specialty database, but it's incredibly powerful in the hands of an enterprise who can do more.And that really creates an opportunity, so we're trying to make Postgres apply to a much broader set of workloads, from traditional systems of record, like your ERP systems; systems of analysis, where people are doing lightweight analytic workloads or reporting, you can think in the world of data warehouse; and then systems of engagement, where customers are interacting with a website and have a database on the backend. All areas Postgres has done incredibly well in and we have customer experience with. So, when you separate out that core capability and then you look at it on a broader scale like Postgres, you realize that customers who want to make Postgres strategic, by definition need to be able to deploy it wherever they want to deploy it, and not be gated or bound by one cloud vendor. And all the cloud vendors picked up Postgres offerings, and that's been great for Postgres and great for enterprises. But that corresponding lock-in is what people want to get away from, at this point.Corey: There's something to be said for acknowledging that there is a form of lock-in as far as technology selection goes. If you have a team of folks who are terrific at one database engine and suddenly you're switching over to an entirely different database, well, folks who spent their entire career working on one particular database that's still in widespread use are probably not super thrilled to stick around for that. Having something that can migrate from environment to environment is valuable and important. When you say you're launching this as a database as a service offering, how does that actually work? Is that going to be running in your own cloud environment somewhere and people just make queries across the wire through standard connections to the database like they would something locally? Are you running inside of their account or environment? Is it something else?Ed: So, this is a fully-managed database as a service, just like you'd get from any cloud vendor or DBAAS vendor that you've worked with in the past, just being managed and run by EDB. And with that, you get lot of the goodies that we bring, including our compatibility, and all our deep Postgres expertise, but I think one of the other important attributes is we're going to run that service in our clients' account, which gives them a level of isolation and a level of independence that we think is really important. And as different as that is, it's not heroic; it's exactly what our customers told us they wanted.Corey: There's something to be said for building the thing that your customers have said that they want and make sense for you to build as opposed to, “We're going to build this ridiculous thing and we're sure folks are going to love it.” It's nice to see that shaping up in the proper order. And I've fallen victim to that myself; I think most technologists have to some extent. How big is EDB these days?Ed: So, we have over 650 employees. Now, around the world, we have 6000 customers. And of the 650 employees, about 300 of those are focused on Postgres. A subset of that are 30-odd core team members in the Postgres community, committers in the Postgres community, major contributors, and contributors in the Postgres community. So, we have a density of technical depth that is really unparalleled in Postgres.Corey: You're not, for lack of a better term, pulling an Amazon, insofar as you're, “Well, we have three people working on open-source projects, so we're going to go ahead and claim we're an open-source company,” in other words. Conversely, you're also not going down the path of this is a project that you folks have launched, and it claims to be open-source because we love it when people volunteer for for-profit entities, but we exercise total control over the project. You have a lot of contributors, but you're also still a minority, I think the largest minority, but still a minority of people contributing to Postgres.Ed: That's right. And, look, we're all-in on Postgres, and it's been that way since I got here. As I mentioned earlier, I came from Red Hat where I was—I was at Red Hat for a little over six years, so I've been an open-source now for 20 years. So, my orientation is towards really powerful, independent open-source projects. And I think we'll see Postgres really be the most transformative open-source technology since Linux.I think we'll see that as we look forward. And you're right, though, I think what's powerful about Postgres is it's an independent project, which means it's supported by thousands of contributors who aren't tied to single companies, around the world. And it just makes the software—we develop innovation faster, and I think it makes the software better. Now, EDB plays a big part in there. Roughly, a little less than a third of the last res—actually, the 13 release—were contributions that came from contributors who came from EDB.So, that's not a majority, and that's healthy. But it's a big part of what helps move Postgres along and there aren't—you know, the next set of companies are much, much—next set of combined contributors add up to quite small numbers. But the cloud vendors are virtually non-existent in that contribution.Corey: This episode is sponsored in part by something new. Cloud Academy is a training platform built on two primary goals. Having the highest quality content in tech and cloud skills, and building a good community the is rich and full of IT and engineering professionals. You wouldn't think those things go together, but sometimes they do. Its both useful for individuals and large enterprises, but here's what makes it new. I don't use that term lightly. Cloud Academy invites you to showcase just how good your AWS skills are. For the next four weeks you'll have a chance to prove yourself. Compete in four unique lab challenges, where they'll be awarding more than $2000 in cash and prizes. I'm not kidding, first place is a thousand bucks. Pre-register for the first challenge now, one that I picked out myself on Amazon SNS image resizing, by visiting cloudacademy.com/corey. C-O-R-E-Y. That's cloudacademy.com/corey. We're gonna have some fun with this one!Corey: Something else that does strike me as, I guess, strange, just because I've seen so many companies try to navigate this in different ways with varying levels of success. I always encountered EDB—even back when it was EnterpriseDB, which was, given their love of acronyms, I'm still somewhat partial to. I get it; branding, it's a thing—but the folks that I engaged with were always there in a consulting service's capacity, and they were great at this. Is EDB a services company or a product company?Ed: Yeah, we are unashamedly a product technology company. Our business is over 90% of our revenue is annually recurring subscription revenue that comes from technical products, database server, mostly, but then various adjacent capabilities in replication and other areas that we add around the database server itself. So no, we're a database technology company selling a subscription. Now, we help our customers, so we do have a really talented team of consultants who help our customers with their business strategy for Postgres, but also with migrations and all the things they need to do to get Postgres up and running.Corey: And the screaming, “Help, help, help, fix it, fix it, fix it now,” emergencies as well.Ed: I think we have the best Postgres support operation in the world. It is a global 24/7 organization, and I think a lot of what you likely experienced, Corey, came out of our support organization. So, our support guys, these guys aren't just handling lightweight issues. I mean, they wade into the gnarly questions and challenges that customers face. But that's a support business for us. So, that's part and parcel. You get that, it's included with the subscription.Corey: I would not be remembering this for 11 years later, if it hadn't been an absolutely stellar experience—or a horrible experience, for that matter; one or the other. You remember the superlatives, not the middle of the road ones—and if it hadn't been important. And it was. It also noteworthy; with many vendors that are product-focused, their services may have an asterisk next to it because it's either a, “Buy our product and then we'll support it,” or it's, “Ohh, we're going to sell you a whole thing just to get us on the phone.” And as I recall, there wasn't a single aspect of upsell involved in this.It was, “Let's get you back up and running and solve the problem.” Sure, later in time, there were other conversations, as all good businesses will have, but there was no point during those crisis moments where it felt like, “Oh, if you had gone ahead and bought this thing that we sell, this wouldn't happen,” or, “You need to buy this or we won't help you.” I guess that's why I've contextualized you folks as a services company, first and foremost.Ed: Well, I'm glad you have that [laugh] experience because that's our goal. And I think—look, this is an interesting point where customers want us to bring that capability to their managed DBAAS world. Step back again, go back to what I said about the big cloud vendors; they are, at their core, infrastructure companies. I mean, they're really good at that. They're not particularly well-positioned to take your Postgres call, and I don't think they want that call.We're the other guys; we want to help you run your Postgres, at scale, on-prem, in the cloud, fully managed in the cloud, by EDB, and solve those problems at the same time. And I think that's missing in the market today. And we can step back and look at this overall cloud evolution, and I think some might think, “Gee, we're into the mature phase of cloud adoption.” I would tell you, since the Red Sox have done well this year, I think in a nine-inning baseball game—for those of your listeners who follow American baseball—we're in, like, the top of the second inning, maybe. Maybe the bottom of the second inning. So, we've been able to listen and learn from the experiences our customers have had. I think that's an incredible advantage as we now firmly plant ourselves in the cloud DBAAS market alongside our robust Postgres capabilities that you experienced.Corey: The world isn't generating less data, and it's important that we're able to access that in a bunch of different ways. And the last time I really was playing with relational databases, you can view my understanding of it as Excel with a weirder interface, and you're mostly there. One thing that really struck me since the last time I went deep into database-land over in the Postgres-squeal world has been just the sheer variety of native data types that it winds up supporting. The idea of, “Here's some JSON. Take this and store it that way,” or it's GIS data that it can represent, or the idea of having data types that are beyond just string or var or whatever other somewhat limited boolean values or whatnot. Without having just that traditional list, which is of course all there as well. It also seems to have extensively improved its coverage that just can only hint to my small mind about these things and what sort of use cases people are really putting these things into.Ed: Yeah, I think this is one of Postgres' superpowers. And it started with Mike Stonebraker's original development of Postgres as an object-relational database. Mike is an adviser to EDB, which has been incredibly helpful as we've continued to evolve our thinking about what's possible in Postgres. But I think because of that core technology, or that core—because of that core technical capability within Postgres, we have been able to build a whole host of data types. And so now you see Postgres being used not just as the context of a traditional relational database, but we see it used as a time-series database. You pointed out a geospatial database, more and more is a document-oriented database with JSON and JSONB.These are all the things that make Postgres have much more universal appeal, universal appeal to developers—which is worth talking about in the recent StackOverflow developer survey, but we can come back to that—and I think universal applicability for new applications. This is what's bringing Postgres forward faster, unlike many of the specialty database companies that you mentioned earlier.Corey: Now, this is something that you can use for your traditional CRUD app, the my first hello world app that returns something from a database, yeah, that stuff works. But it also, for example, has [cyter 00:25:09] data types, where you can say, give me the results where the IP range contains this address, and it'll do that. Before that, you're trying to solve a whole bunch of very messy things in application logic that's generally awful. The database now does that for you automatically, and there's something—well, it would if I were smart and used it instead of storing it as strings because I make terrible life choices, but for sensible people, it solves a lot of those problems super well. And it's taken the idea of where logic should live in application versus database, and sort of turn a lot of those assumptions I was starting my career with on their head.Ed: Yeah, I think if you look now at the appeal of Postgres to developers, which we've paid a lot of attention to—one of our stated strategies at EDB is to make Postgres easier. That's been true for many years, so a drive for engineering and development here has been that call to action. And if you measure that, over time, we've been contributing—not alone, but contributing to making Postgres more approachable, easier to use, easier to engage with. Some of those things we do just through edb.com, and the way we handle EDB docs is a great example of that, and our developer advocacy and outreach into adjacent communities that care about Postgres. But here's where that's landed us. If you looked at the last Stack Overflow developer survey—the 2021 Stack Overflow developer survey, which I love because I think it's very independent-oriented—and they surveyed, I think this past year was 80,000 developers.Corey: Oh yeah, if Stack Overflow is captured by any particular constituency, it's got to be ‘Big Copy and Paste' that is really behind them. But yeah, other than the cabal of keyboard manufacturers for those copy-and-paste stories, yeah, they're fairly objective when it comes to stuff like this.Ed: And if you look at that survey, Corey, if you just took and summed it because it's helpful to sum it, most used, most loved, and most wanted database: Postgres wins. And I find it fascinating that if you—having been here, in this company for 13 years and watch the evolution from—you know, 13 years ago, Postgres needed help, both in terms of its awareness in the market and some technical capabilities it just lacked, we've come so far. For that to be the new standard for developers, I think, is a remarkable achievement. And I think it's a representation of why Postgres is doing so well in the market that we've long served, in the cloud market that we are now serving, and I think it speaks to what's ahead as a transformational database for the future.Corey: There really is something to be said for a technology as—please don't take this term the wrong way—old. As a relational database, Postgres has been around for a very long time, but it's also not your grandparents' Postgres. It is continuing to evolve. It continues to be there in a bunch of really interesting ways for developers in a variety of different capacities, and it's not the sort of thing that you're only using in, “Legacy environments,” quote-unquote. Instead, it's something that you'll see all over the place. It is rare that I see an environment that doesn't have Postgres in it somewhere these days.Ed: Yeah, I think quite the contrary to the old-school database, which I love that; I love that shade because when you step away from it, you realize, the Postgres community represents the very best of what's possible with open-source. And that's why Postgres continues to accelerate and move forward at the rate that it does. And obviously, we're proud to be a contributor to that, so we don't just watch that outcome happen; we're actually part of creating it. But I also think that when you see all that Postgres has become and where it's going, you really start to understand why the market is adopting open-source.Corey: It's one of those areas where even if some company comes out with something that is amazing and transformatively better, and you should jump into it with both feet and never look back, yeah, it turns out that it takes a long time to move databases, even when they're terrible. And you can lobby an awful lot of accusations at Postgres—or Postgres-squeal—but you can't call it terrible. It's used in enough interesting applications by enough large-scale companies out there—and small as well—that it's very hard to find a reason not to explore it. It's my default relational database when Route 53 loses steam. It just makes sense in a bunch of ways that other things really didn't for me before.Ed: Yeah, and I think we'll continue to see that. And we're just going to keep making Postgres better. And it gets better because of that intersection, as I mentioned, that intimate intersection between enterprise users, and the project, and the community, and the bridge that a company like EDB provides for that. That's why it'll get better faster; the breadth of use of Postgres will keep it accelerating. And I think it's different than many of the specialty databases.Look, I've been in open-source now for 20 years and it's intriguing to me how many new specialty open-source databases have come to market. We tend to forget the amount of roadkill we've had over the course of the past ten years of some of those open-source projects and companies. We certainly are tuned into some of the more prolific ones, even today. And I think again, here again, this is where Postgres shines, and where I think Postgres is a better call for a long-term. Just like Linux was.Corey: I want to thank you for taking so much time out of your day to talk to me about databases, which given my proclivities, is probably like pulling teeth for you. If people want to learn more, where can they find you?Ed: So, come to enterprisedb.com. You still get EnterpriseDB, Corey. Just come to enterprise—Corey: There we go. It's hidden in the URL, right in plain sight.Ed: Come to enterprisedb.com. You can learn all the things you need about the technology, and certainly more that we can do to help you.Corey: And we will, of course, put links to that in the [show notes 00:31:10]. Thank you once again for your time. I really do appreciate it.Ed: Thanks, Corey. My pleasure.Corey: Ed Boyajian, CEO of EDB. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with a long angry comment because you are one of the two Amazonian developers working on open-source databases.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

This week on the podcast, Dan talks about some changes to the DDDAudit report to generate SQL, and some changes in refresh scripts for 8.58. Kyle discusses ways to test and validate Integration Broker messages after a refresh. Show Notes Facebook Outage @ 5:00 DDDAudit Modification @ 12:00 Refresh SQL Updates @ 17:00 Testing Async IB Messages @ 26:30 Put a bird on it

Watch the live stream: Watch on YouTube About the show Sponsored by Shortcut - Get started at shortcut.com/pythonbytes Special guest: The Anthony Shaw Michael #0: It's episode 2^8 (nearly 5 years of podcasting) Brian #1: Where does all the effort go?: Looking at Python core developer activity Łukasz Langa A look into CPython repository history and PR data Also, nice example of datasette in action and lots of SQL queries. The data, as well as the process, is open for anyone to look at. Cool that the process was listed in the article, including helper scripts used. Timeframe for data is since Feb 10, 2017, when source moved to GitHub, through Oct 9, 2021. However, some queries in the article are tighter than that. Queries Files involved in PRs since 1/1/20 top is ceval.c with 259 merged PRs Contributors by number of merged PRs lots of familiar names in the top 50, along with some bots it'd be fun to talk with someone about the bots used to help the Python project nice note: “Clearly, it pays to be a bot … or a release manager since this naturally causes you to make a lot of commits. But Victor Stinner and Serhiy Storchaka are neither of these things and still generate amazing amounts of activity. Kudos! In any case, this is no competition but it was still interesting to see who makes all these recent changes.” Who contributed where? Neat. There's a self reported Experts Index in the very nice Python Developer's Guide. But some libraries don't have anyone listed. The data does though. Łukasz generated a top-5 list for each file. Contributing to some file and have a question. These folks may be able to help. Averages for PR activity core developer authoring and merging their own PR takes on average ~7 days (std dev ±41.96 days); core developer authoring a PR which was merged by somebody else takes on average 20.12 days (std dev ±77.36 days); community member-authored PRs get merged on average after 19.51 days (std dev ±81.74 days). Interesting note on those std deviations: “Well, if we were a company selling code review services, this standard deviation value would be an alarmingly large result. But in our situation which is almost entirely volunteer-driven, the goal of my analysis is to just observe and record data. The large standard deviation reflects the large amount of variation but isn't necessarily something to worry about. We could do better with more funding but fundamentally our biggest priority is keeping CPython stable. Certain care with integrating changes is required. Erring on the side of caution seems like a wise thing to do.” More questions to be asked, especially from the issue tracker Which libraries require most maintenance? Michael #2: Why you shouldn't invoke setup.py directly By Paul Ganssle (from Talk Python #271: Unlock the mysteries of time, Python's datetime that is!) In response to conversation in Talk Python's cibuildwheel episode? For a long time, setuptools and distutils were the only game in town when it came to creating Python packages You write a setup.py file that invokes the setup() method, you get a Makefile-like interface exposed by invoking python setup.py [HTML_REMOVED] The last few years all direct invocations of setup.py are effectively deprecated in favor of invocations via purpose-built and/or standards-based CLI tools like pip, build and tox. In Python 2.0, the distutils module was introduced as a standard way to convert Python source code into *nix distro packages One major problem with this approach, though, is that every Python package must use distutils and only distutils — there was no standard way for a package author to make it clear that you need other packages in order to build or test your package. => Setuptools Works, but sometimes you need requirements before the install (see cython example) A build backend is something like setuptools or flit, which is a library that knows how to take a source tree and turn it into a distributable artifact — a source distribution or a wheel. A build frontend is something like pip or build, which is a program (usually a CLI tool) that orchestrates the build environment and invokes the build backend In this taxonomy, setuptools has historically been both a backend and a frontend - that said, setuptools is a terrible frontend. It does not implement PEP 517 or PEP 518's requirements for build frontends Why am I not seeing deprecation warnings? Use build package. Also can be replaced by tox, nox or even a Makefile Probably should just check out the summary table. Anthony #3: OpenTelemetry is going stable soon Cloud Native Computing Foundation project for cross-language event tracing, performance tracing, logging and sampling for distributed applications. Engineers from Microsoft, Amazon, Splunk, Google, Elastic, New Relic and others working on standards and specification. Formed through a merger of the OpenTracing and OpenCensus projects. Python SDK supports instrumentation of lots of frameworks, like Flask, Django, FastAPI (ASGI), and ORMs like SQLalchemy, or templating engines. All data can then be exported onto various platforms : NewRelic, Prometheus, Jaeger, DataDog, Azure Monitor, Google Cloud Monitoring. If you want to get started and play around, checkout the rich console exporter I submitted recently. Brian #4: Understanding all of Python, through its builtins Tushar Sadhwani I really enjoyed the discussion before he actually got to the builtins. LEGB rule defines the order of scopes in which variables are looked up in Python. Local, Enclosing (nonlocal), Global, Builtin Understanding LEGB is a good thing to do for Python beginners or advanced beginners. Takes a lot of the mystery away. Also that all the builtins are in one The rest is a quick scan through the entire list. It's not detailed everywhere, but pulls over scenic viewpoints at regular intervals to discuss interesting parts of builtins. Grouped reasonably. Not alphabetical Constants: There's exactly 5 constants: True, False, None, Ellipsis, and NotImplemented. globals and locals: Where everything is stored bytearray and memoryview: Better byte interfaces bin, hex, oct, ord, chr and ascii: Basic conversions … Well, it's a really long article, so I suggest jumping around and reading a section or two, or three. Luckily there's a nice TOC at the top. Michael #5: FastAPI, Dask, and more Python goodies win best open source titles Things that stood out to me FastAPI Dask Windows Terminal minikube - Kubernetes cluster on your PC OBS Studio Anthony #6: Notes From the Meeting On Python GIL Removal Between Python Core and Sam Gross Following on from last week's share on the “nogil” branch by Sam Gross, the Core Dev sprint included an interview. Targeted to 3.9 (alpha 3!), needs to at least be updated to 3.9.7. Nogil: Replaces pymalloc with mimalloc for thread safety Ties objects to the thread that created them witha. non-atomic local reference count within the owner thread Allows for (slower) reference counting from other threads. Immortalized some objects so that references never get inc/dec'ed like True, False, None, etc. Deferred reference counting Adjusts the GC to wait for all threads to pause at a safe point, doesn't wait for I/O blocked threads and constructs a list of objects to deallocate using mimalloc Relocates the MRO to a thread local (instead of process-local) to avoid contention on ref counting Modifies the builtin collections to be thread-safe (lists, dictionaries, etc,) since they could be shared across threads. IMHO, biggest thing to happen to Python in 5 years. Encouragingly, Sam was invited to be a Core Dev and Lukasz will mentor him! Extras Michael Python Developers Survey 2021 is open More PyPI CLI updates bump2version via Bahram Aghaei (youtube comment) Was there a bee stuck in Brian's mic last time? Brian PyCon US 2022 CFP is open until Dec 20 Python Testing with pytest, 2nd edition, Beta 7.0 All chapters now there. (Final chapter was “Advanced Parametrization”) It's in technical review phase now. If reading, please skip ahead to the chapter you really care about and submit errata if you find anything confusing. Joke:

Streaming data systems have been growing more capable and flexible over the past few years. Despite this, it is still challenging to build reliable pipelines for stream processing. In this episode Eric Sammer discusses the shortcomings of the current set of streaming engines and how they force engineers to work at an extremely low level of abstraction. He also explains why he started Decodable to address that limitation and the work that he and his team have done to let data engineers build streaming pipelines entirely in SQL.

Links: 1Password University: https://blog.1password.com/introducing-1password-university/ Penetration testing: https://www.darkreading.com/cloud/pentesting-in-the-cloud-demands-a-different-approach New AWS workbook for New Zealand financial services customers: https://aws.amazon.com/blogs/security/new-aws-workbook-for-new-zealand-financial-services-customers/ Secretive: https://github.com/maxgoedjen/secretive TranscriptCorey: This is the AWS Morning Brief: Security Edition. AWS is fond of saying security is job zero. That means it's nobody in particular's job, which means it falls to the rest of us. Just the news you need to know, none of the fluff.Corey: This episode is sponsored in part by Liquibase. If you're anything like me, you've screwed up the database part of a deployment so severely that you've been banned from ever touching anything that remotely sounds like SQL at least three different companies. We've mostly got code deployment solved for, but when it comes to databases, we basically rely on desperate hope, with a rollback plan of keeping our resumes up to date. It doesn't have to be that way. Meet Liquibase. It's both an open-source project and a commercial offering. Liquibase lets you track, modify, and automate database schema changes across almost any database, with guardrails that ensure you'll still have a company left after you deploy the change. No matter where your database lives, Liquibase can help you solve your database deployment issues. Check them out today at liquibase.com. Offer does not apply to Route 53.Corey: So, it's been an interesting week in the world of AWS security, and a light one. And that's okay. 1Password introduced 1Password University, and I'm interested in it, not because I expect to learn a whole lot that I didn't know before about security, but because this might be able to replace my current, fairly awful Security Awareness Training.See, a lot of companies have contractual requirements to provide SAT to their staff and contractors. Most of them are terrible courses that actively push crap advice like, “Rotate your password every 60 days.” This has the potential, just based on my experiences with 1Password, to be way better than that. But we'll see.“Things are different in the cloud,” is something of a truism, and that applies as much to penetration testing as anything else. Understanding that your provider may have no sense of humor whatsoever around this, and thus require you to communicate with them in advance, for example. There was a great interview with Josh Stella, who I've had on Screaming in the Cloud. He's CEO of Fugue—that he will say is pronounced ‘Fugue', but it's ‘Fwage'—and he opined on this in an article I discovered, and interview, with quite some eloquence. I should really track him down and see if I can get him back on the podcast one of these days. It has been far too long.now, from the mouth of AWS Horse. There's a New AWS workbook for New Zealand financial services customers, and that honestly kind of harkens back to school: unnecessary work that you're paying for the privilege of completing. But it is good to be able to sit down and work through the things you're going to need to be able to answer in a world of cloud when you're in a regulated industry like that, and those regulations vary from country to country. You can tell where the regulations around data residency are getting increasingly tight because that's where AWS is announcing regions.Corey: This episode is sponsored in part by something new. Cloud Academy is a training platform built on two primary goals: having the highest quality content in tech and cloud skills, and building a good community that is rich and full of IT and engineering professionals. You wouldn't think those things go together, but sometimes they do. It's both useful for individuals and large enterprises, but here's what makes this something new—I don't use that term lightly—Cloud Academy invites you to showcase just how good your AWS skills are. For the next four weeks, you'll have a chance to prove yourself. Compete in four unique lab challenges where they'll be awarding more than $2,000 in cash and prizes. I'm not kidding: first place is a thousand bucks. Pre-register for the first challenge now, one that I picked out myself on Amazon SNS image resizing, by visiting cloudacademy.com/corey—C-O-R-E-Y. That's cloudacademy.com/corey. We're going to have some fun with this one.Corey: And of course, a tool for the week. I'll be playing around with Secretive in the next week or two. It's an open-source project that stores SSH keys in a Mac's Secure Enclave instead of on disk. I don't love the idea of having my key material on disk wherever possible, even though I do passphrase-protect it.This stores it in the Mac Secure Enclave and presents it well. I've had a couple of problems on a couple of machines so far, and I'm talking to the developer in a GitHub issue, but it is important to think about these things. I, of course, turn on full-disk encryption, but if something winds up subverting my machine, I don't want it to just be able to look at what's on disk and get access to things that matter. That feels like it could blow up in my face.Corey: And that's really what happened last week in AWS security. It's been a light week; I hope you enjoy it, there is much more to come next week, now that I'm back from vacation.Corey: I have been your host, Corey Quinn, and if you remember nothing else, it's that when you don't get what you want, you get experience instead. Let my experience guide you with the things you need to know in the AWS security world, so you can get back to doing your actual job. Thank you for listening to the AWS Morning Brief: Security Editionwith the latest in AWS security that actually matters. Please follow AWS Morning Brief on Apple Podcast, Spotify, Overcast—or wherever the hell it is you find the dulcet tones of my voice—and be sure to sign up for the Last Week in AWS newsletter at lastweekinaws.com.Announcer: This has been a HumblePod production. Stay humble.

Alexander Pugh is a software engineer at Albertsons. He has worked in Robotic Process Automation and the cognitive services industry for over five years.This episode originally aired on Software Engineering Radio.Related LinksAlexander Pugh's personal siteEnterprise RPA Solutions Automation Anywhere UiPath blueprism Enterprise "Low Code/No Code" API Solutions appian mulesoft Power Automate RPA and the OS Office primary interop assemblies Office Add-ins documentation Task Scheduler for developers The Component Object Model The Document Object Model TranscriptYou can help edit this transcript on GitHub.[00:00:00] Jeremy: Today, I'm talking to Alexander Pugh. He's a solutions architect with over five years of experience working on robotic process automation and cognitive services. Today, we're going to focus on robotic process automation. Alexander welcome to software engineering radio. [00:00:17] Alex: Thank you, Jeremy. It's really good to be here. [00:00:18] Jeremy: So what does robotic process automation actually mean? [00:00:23] Alex: Right. It's a, it's a very broad nebulous term. when we talk about robotic process automation, as a concept, we're talking about automating things that humans do in the way that they do them. So that's the robotic, an automation that is, um, done in the way a human does a thing.Um, and then process is that thing, um, that we're automating. And then automation is just saying, we're turning this into an automation where we're orchestrating this and automating this. and the best way to think about that in any other way is to think of a factory or a car assembly line. So initially when we went in and we, automated a car or factory, automation line, what they did is essentially they replicated the process as a human did it. So one day you had a human that would pick up a door and then put it on the car and bolt it on with their arms. And so the initial automations that we had on those factory lines were a robot arm that would pick up that door from the same place and put it on the car and bolt it on there.Um, so the same can be said for robotic process automation. We're essentially looking at these, processes that humans do, and we're replicating them, with an automation that does it in the same way. Um, and where we're doing that is the operating system. So robotic process automation is essentially going in and automating the operating system to perform tasks the same way a human would do them in an operating system.So that's, that's RPA in a nutshell, Jeremy: So when you say you're replicating something that a human would do, does it mean it has to go through some kind of GUI or some kind of user interface?[00:02:23] Alex: That's exactly right, actually. when we're talking about RPA and we look at a process that we want to automate with RPA, we say, okay. let's watch the human do it. Let's record that. Let's document the human process. And then let's use the RPA tool to replicate that exactly in that way.So go double click on Chrome, launch that click in the URL line and send key in www.cnn.com or what have you, or servicenow hit enter, wait for it to load and then click, you know, where you want to, you know, fill out your ticket for service. Now send key in. So that's exactly how an RPA solution at the most basic can be achieved.Now and any software engineer knows if you sit there and look over someone's shoulder and watch them use an operating system. Uh, you'll say, well, there's a lot of ways we can do this more efficiently without going over here, clicking that, you know, we can, use a lot of services that the operating system provides in a programmatic way to achieve the same ends and RPA solutions can also do that.The real key is making sure that it is still achieving something that the human does and that if the RPA solution goes away, a human can still achieve it. So if you're, trying to replace or replicate a process with RPA, you don't want to change that process so much so that a human can no longer achieve it as well.that's something where if you get a very technical, and very fluent software engineer, they lose sight of that because they say, oh, you know what? There's no reason why we need to go open a browser and go to you know, the service now portal and type this in when I can just directly send information to their backend.which a human could not replicate. Right? So that's kind of where the line gets fuzzy. How efficiently can we make this RPA solution? [00:04:32] Jeremy: I, I think a question that a lot of people are probably having is a lot of applications have APIs now. but what you're saying is that for it to, to be, I suppose, true RPA, it needs to be something that a user can do on their own and not something that the user can do by opening up dev tools or making a post to an end point.[00:04:57] Alex: Yeah. And so this, this is probably really important right now to talk about why RPA, right? Why would you do this when you could put on a server, a a really good, API ingestion point or trigger or a web hook that can do this stuff. So why would we, why would we ever pursue our RPA?There there's a lot of good reasons for it. RPA is very, very enticing to the business. RPA solutions and tools are marketed as a low code, no code solution for the business to utilize, to solve their processes that may not be solved by an enterprise solution and the in-between processes in a way.You have, uh, a big enterprise, finance solution that everyone uses for the finance needs of your business, but there are some things that it doesn't provide for that you have a person that's doing a lot of, and the business says, Okay. well, this thing, this human is doing this is really beneath their capability. We need to get a software solution for it, but our enterprise solution just can't account for it. So let's get a RPA capability in here. We can build it ourselves, and then there we go. So there, there are many reasons to do that. financial, IT might not have, um, the capability or the funding to actually build and solve the solution. Or it it's at a scale that is too small to open up, uh, an IT project to solve for. Um, so, you know, a team of five is just doing this and they're doing it for, you know, 20 hours a week, which is uh large, but in a big enterprise, that's not really. Maybe um, worth building an enterprise solution for it. or, and this is a big one. There are regulatory constraints and security constraints around being able to access this or communicate some data or information in a way that is non-human or programmatic. So that's really where, um, RPA is correctly and best applied and you'll see it most often.So what we're talking about there is in finance, in healthcare or in big companies where they're dealing with a lot of user data or customer data in a way. So when we talk about finance and healthcare, there are a lot of regulatory constraints and security reasons why you would not enable a programmatic solution to operate on your systems. You know, it's just too hard. We we're not going to expose our databases or our data to any other thing. It would, it would take a huge enterprise project to build out that capability, secure that capability and ensure it's going correctly. We just don't have the money the time or the strength honestly, to afford for it.So they say, well, we already have. a user pattern. We already allow users to, to talk to this information and communicate this information. Let's get an RPA tool, which for all intents and purposes will be acting as a user. And then it can just automate that process without us exposing to queries or any other thing, an enterprise solution or programmatic, um, solution.So that's really why RPA, where and why you, you would apply it is there's, there's just no capability at enterprise for one reason or another to solve for it. [00:08:47] Jeremy: as software engineers, when we see this kind of problem, our first thought is, okay, let's, let's build this custom application or workflow. That's going to talk to all these API APIs. And, and what it sounds like is. In a lot of cases there just isn't the time there just isn't the money, to put in the effort to do that.And, it also sounds like this is a way of being able to automate that. and maybe introducing less risk because you're going through the same, security, the same workflow that people are doing currently. So, you know, you're not going to get into things that they're not supposed to be able to get into because all of that's already put in place.[00:09:36] Alex: Correct. And it's an already accepted pattern and it's kind of odd to apply that kind of very IT software engineer term to a human user, but a human user is a pattern in software engineering. We have patterns that do this and that, and, you know, databases and not, and then the user journey or the user permissions and security and all that is a pattern.And that is accepted by default when you're building these enterprise applications okay.What's the user pattern. And so since that's already established and well-known, and all the hopefully, you know, walls are built around that to enable it to correctly do what it needs to do. It's saying, Okay. we've already established that. Let's just use that instead of. You know, building a programmatic solution where we have to go and find, do we already have an appropriate pattern to apply to it? Can we build it in safe way? And then can we support it? You know, all of a sudden we, you know, we have the support teams that, you know, watch our Splunk dashboards and make sure nothing's going down with our big enterprise application.And then you're going to build a, another capability. Okay. WHere's that support going to come from? And now we got to talk about change access boards, user acceptance testing and, uh, you know, UAT dev production environments and all that. So it becomes, untenable, depending on your, your organization to, to do that for things that might fall into a place that is, it doesn't justify the scale that needs to be thrown upon it.But when we talk about something like APIs and API exist, um, for a lot of things, they don't exist for everything. And, a lot of times that's for legacy databases, that's for mainframe capability. And this is really where RPA shines and is correctly applied. And especially in big businesses are highly regulated businesses where they can't upgrade to the newest thing, or they can't throw something to the cloud.They have a, you know, their mainframe systems or they have their database systems that have to exist for one reason or the other until there is the motivation and the money and the time to correctly migrate and, and solve for them. So until that day, and again, there's no, API to, to do anything on a, on a mainframe, in this bank or whatnot, it's like, well, Okay. let's just throw RPA on it.Let's, you know, let's have a RPA do this thing, uh, in the way that a human does it, but it can do it 24 7. and an example, or use cases, you work at a bank and, uh, there's no way that InfoSec is going to let you query against this database with, your users that have this account or your customers that have this no way in any organization at a bank.Is InfoSec going to say, oh yeah. sure. Let me give you an Odata query, you know, driver on, you know, and you can just set up your own SQL queries and do whatever they're gonna say no way. In fact, how did you find out about this database in the first place and who are you.How do we solve it? We, we go and say, Okay. how does the user get in here well they open up a mainframe emulator on their desktop, which shows them the mainframe. And then they go in, they click here and they put this number in there, and then they look up this customer and then they switch this value to that value and they say, save.And it's like, okay. cool. That's that RPA can do. And we can do that quite easily. And we don't need to talk about APIs and we don't need to talk about special access or doing queries that makes, you know, Infosec very scared. you know, a great use case for that is, you know, a bank say they, they acquire, uh, a regional bank and they say, cool, you're now part of our bank, but in your systems that are now going to be a part of our systems, you have users that have this value, whereas in our bank, that value is this value here. So now we have to go and change for 30,000 customers this one field to make it line up with our systems. Traditionally you would get a, you know, extract, transform load tool an ETL tool to kind of do that. But for 30,000 customers that might be below the threshold, and this is banking. So it's very regulated and you have to be very, very. Intentional about how you manipulate and move data around.So what do we have to do? okay. We have to hire 10 contractors for six months, and literally what they're going to do eight hours a day is go into the mainframe through the simulator and customer by customer. They're going to go change this value and hit save. And they're looking at an Excel spreadsheet that tells them what customer to go into.And that's going to cost X amount of money and X, you know, for six months, or what we could do is just build a RPA solution, a bot, essentially that goes, and for each line of that Excel spreadsheet, it repeats this one process, open up mainframe emulator, navigate into the customer profile and then changes value, and then shut down and repeat.And It can do that in one week and, and can be built in two, that's the, the dream use case for RPA and that's really kind of, uh, where it would shine.[00:15:20] Jeremy: It sounds like the. best use case for it is an old system, a mainframe system, in COBOL maybe, uh, doesn't have an API. And so, uh, it makes sense to rather than go, okay, how can we get directly into the database?[00:15:38] Alex: How can we build on top of it? Yeah,[00:15:40] Jeremy: we build on top of it? Let's just go through the, user interface that exists, but just automate that process. And, the, you know, the example you gave, it sounds very, very well-defined you're gonna log in and you're going to put in maybe this ID, here's the fields you want to get back.and you're going to save those and you didn't have to make any real decisions, I suppose, in, in terms of, do I need to click this thing or this thing it's always going to be the same path to, to get there.[00:16:12] Alex: exactly. And that's really, you need to be disciplined about your use cases and what those look like. And you can broadly say a use case that I am going to accept has these features, and one of the best ways to do that is say it has to be a binary decision process, which means there is no, dynamic or interpreted decision that needs to, or information that needs to be made.Exactly like that use case it's very binary either is, or it isn't you go in you journey into there. and you change that one thing and that's it there's no oh, well this information says this, which means, and then I have to go do this. Once you start getting in those if else, uh, processes you're, you're going down a rabbit hole and it could get very shaky and that introduces extreme instability in what you're trying to do.And also really expands your development time cause you have to capture these processes and you have to say, okay. tell me exactly what we need to build this bot to do. And for, binary decision processes, that's easy go in here, do this, but nine times out of 10, as you're trying to address this and solution for it, you'll find those uncertainties.You'll find these things where the business says, oh, well, yeah. that happens, you know, one times out of 10 and this is what we need to do. And it's like, well, that's going to break the bot. It, you know, nine times out of 10, this, this spot is going to fall over. this is now where we start getting into, the machine learning and AI, realm.And why RPA, is classified. Uh, sometimes as a subset of the AI or machine learning field, or is a, a pattern within that field is because now that you have this bot or this software that enables you to do a human process, let's enable that bot to now do decision-making processes where it can interpret something and then do something else.Because while we can just do a big tree to kind of address every capability, you're never going to be able to do that. And also it's, it's just a really heavy, bad way to build things. So instead let's throw in some machine learning capability where it just can understand what to do and that's, you know, that's the next level of RPA application is Okay. we've got it. We've, we've gone throughout our organization. We found every kind of binary thing, that can be replaced with an RPA bot. Okay.Now what are the ones that we said we couldn't do? Because it had some of that decision-making that, required too much of a dynamic, uh, intelligence behind it. And let's see if we can address those now that we have this. And so that's, that's the 2.0, in RPA is addressing those non-binary, paths. I would argue that especially in organizations that are big enough to justify bringing in an RPA solution to solve for their processes. They have enough binary processes, binary decision processes to keep them busy.Some people, kind of get caught up in trying to right out the gate, say, we need to throw some machine learning. We need to make these bots really capable instead of just saying, well, we we've got plenty of work, just changing the binary processes or addressing those. Let's just be disciplined and take that, approach.Uh, I will say towards RPA and bots, the best solution or the only solution. When you talk about building a bot is the one that you eventually turn off. So you can say, I built a bot that will go into our mainframe system and update this value. And, uh, that's successful.I would argue that's not successful. When that bot is successful is when you can turn it off because there's an enterprise solution that addresses it. and, and you don't have to have this RPA bot that lives over here and does it instead, you're enterprise, capability now affords for it. And so that's really, I think a successful bot or a successful RPA solution is you've been able to take away the pain point or that human process until it can be correctly addressed by your systems that everyone uses. [00:21:01] Jeremy: from, the business perspective, you know, what are some of the limitations or long-term problems with, with leaving an RPA solution in place?[00:21:12] Alex: that's a, that's a good question. Uh, from the business there, isn't, it's solved for. leaving it in place is other than just servicing it and supporting it. There's no real issue there, especially if it's an internal system, like a mainframe, you guys own that. If it changes, you'll know it, if it changes it's probably being fixed or addressed.So there's no, problem. However, That's not the only application for RPA. let's talk about another use case here, your organization, uses, a bank and you don't have an internal way to communicate it. Your user literally has to go to the bank's website, log in and see information that the bank is saying, Hey, this is your stuff, right?The bank doesn't have an API for their, that service. because that would be scary for the bank. They say, we don't want to expose this to another service. So the human has to go in there, log in, look at maybe a PDF and download it and say, oh, Okay.So that is happens in a browser. So it's a newer technology.This isn't our mainframe built in 1980. You know, browser based it's in the internet and all that, but that's still a valid RPA application, right? It's a human process. There's no API, there's no easy programmatic way to, to solution for it. It would require the bank and your it team to get together and, you know, hate each other. Think about why this, this is so hard. So let's just throw a bot on it. That's going to go and log in, download this thing from the bank's website and then send it over to someone else. And it's going to do that all day. Every day. That's a valid application. And then tomorrow the bank changes its logo. And now my bot is it's confused.Stuff has shifted on the page. It doesn't know where to click anymore. So you have to go in and update that bot because sure enough, that bank's not going to send out an email to you and saying, Hey, by the way, we're upgrading our website in two weeks. Not going to happen, you'll know after it's happened.So that's where you're going to have to upgrade the bot. and that's the indefinite use of RPA is going to have to keep until someone else decides to upgrade their systems and provide for a programmatic solution that is completely outside the, uh, capability of the organization to change. And so that's where the business would say, we need this indefinitely.It's not up to us. And so that is an indefinite solution that would be valid. Right? You can keep that going for 10 years as long, I would say you probably need to get a bank that maybe meets your business needs a little easier, but it's valid. And that would be a good way for the business to say yes, this needs to keep running forever until it doesn't.[00:24:01] Jeremy: you, you brought up the case of where the webpage changes and the bot doesn't work anymore. specifically, you're, you're giving the example of finance and I feel like it would be basically catastrophic if the bot is moving money to somewhere, it shouldn't be moving because the UI has moved around or the buttons not where it expects it to be.And I'm kind of curious what your experience has been with that sort of thing.[00:24:27] Alex: you need to set organizational thresholds and say, this is this something this impacting or something that could go this wrong. It is not acceptable for us to solve with RPA, even though we could do it, it's just not worth it. Some organizations say that's anything that touches customer data healthcare and banking specialists say, yeah, we have a human process where the human will go and issue refunds to a customer, uh, and that could easily be done via RPA solution, but it's fraught with, what, if it does something wrong, it's literally going to impact.Uh, someone somewhere they're their moneys or their, their security or something like that. So that, that definitely should be part of your evaluation. And, um, as an organization, you should set that up early and stick to it and say, Nope, this is outside our purview. Even we can do it. It has these things.So I guess the answer to that is you should never get to that process, but now we're going to talk about, I guess, the actual nuts and bolts of how RPA solutions work and how they can be made to not action upon stuff when it changes or if it does so RPA software, by and large operates by exposing the operating system or the browsers underlying models and interpreting them.Right. So when we talk about something like a, mainframe emulator, you have your RPA software on Microsoft windows. It's going to use the COM the component operating model, to see what is on the screen, what is on that emulator, and it's gonna expose those objects. to the software and say, you can pick these things and click on that and do that.when we're talking about browser, what the RPA software is looking at is not only the COM the, the component object model there, which is the browser, itself. But then it's also looking at the DOM the document object model that is the webpage that is being served through the browser. And it's exposing that and saying, these are the things that you can touch or, operate on.And so when you're building your bots, what you want to make sure is that the uniqueness of the thing that you're trying to access is something that is truly unique. And if it changes that one thing that the bot is looking for will not change. So we let's, let's go back to the, the banking website, right?We go in and we launch the browser and the bot is sitting there waiting for the operating system to say, this process is running, which is what you wanted to launch. And it is in this state, you know, the bot says, okay. I'm expecting this kind of COM to exist. I see it does exist. It's this process, and it has this kind of name and cool Chrome is running. Okay. Let's go to this website. And after I've typed this in, I'm going to wait and look at the DOM and wait for it to return this expected a webpage name, but they could change their webpage name, the title of it, right. They can say, one day can say, hello, welcome to this bank. And the next day it says bank website, all of a sudden your bot breaks it no longer is finding what it was told to expect.So you want to find something unique that will never change with that conceivably. And so you find that one thing on the DOM on the banking website, it's, you know, this element or this tag said, okay, there's no way they're changing that. And so it says cool the page is loaded. Now click on this field, which is log in.Okay. You want to find something unique on that field that won't change when they upgrade, you know, from bootstrap to this kind of, you know, UI framework. that's all well, and good. That's what we call the happy path. It's doing this perfectly. Now you need to define what it should do when it doesn't find these things, which is not keep going or find similar it's it needs to fail fast and gracefully and pass on that failure to someone and not keep going. And that's kind of how we prevent that scary use case where it's like. okay. it's gone in, it's logged into the bank website now it's transactioning, bad things to bad places that we didn't program it for it, Well you unfortunately did not specify in a detailed enough way what it needs to look for.And if it doesn't find that it needs to break, instead of saying that this is close enough. And so, in all things, software engineering, it's that specificity, it's that detail, that you need to hook onto. And that's also where, when we talk about this being a low-code no-code solutions that sometimes RPA is marketed to the business.It's just so often not the case, because yes. It might provide a very user, business, friendly interface for you to build bots. But the knowledge you need to be able to ensure stability and accuracy, um, to build the bots is, is a familiarity that's probably not going to be had in the business. It's going to be had by a developer who knows what the DOM and COM are and how the operating system exposes services and processes and how.JavaScript, especially when we're talking about single page apps and react where you do have this very reactive DOM, that's going to change. You need to be fluent with that and know, not only how HTML tags work and how CSS will change stuff on you in classes, but also how clicking on something on a single page app is as simple as a username input field will dynamically change that whole DOM and you need to account for it. so, it is it's, traditionally not as easy as saying, oh, the business person can just click, click, click, click, and then we have a bot. You'll have a bot, but it's probably going to be break breaking quite often. It's going to be inaccurate in its execution.this is a business friendly user-friendly non-technical tool. And I launch it and it says, what do you want to do? And it says, let me record what you're going to do. And you say, cool.And then you go about you open up Chrome and you type in the browser, and then you click here, click there, hit send, and then you stop recording. The tool says, cool, this is what you've done. Well, I have yet to see a, a solution that is that isn't able to not need further direction or, or defining on that process, You still should need to go in there and say, okay, yeah.you recorded this correctly, but you know, you're not interpreting correctly or as accurate as you need to that field that I clicked on.And if you know, anybody hits, you know, F12 on their keyboard while they have Chrome open and they see how the DOM is built, especially if this is using kind of any kind of template, Webpage software. It's going to have a lot of cruft in that HTML. So while yes, the recording did correctly see that you clicked on the input box.What it's actually seen is that you actually clicked on the div. That is four levels scoped above it, whereas the parent, and there are other things within that as well. And so the software could be correctly clicking on that later, but other things could be in there and you're going to get some instability.So the human or the business, um, bot builder, the roboticist, I guess, would need to say, okay, listen, we need to pare this down, but it's, it's even beyond that. There are concepts that you can't get around when building bots that are unique to software engineering as a concept. And even though they're very basic, it's still sometimes hard for the business user to, they felt to learn that.And I I'm talking concepts as simple as for loops or loops in general where the business of course has, has knowledge of what we would call a loop, but they wouldn't call it a loop and it's not as accurately defined. So they have to learn that. And it's not as easy as just saying, oh Yeah.do a loop. And the business will say, well, what's a loop.Like I know, you know, conceptually what a loop could be like a loop in my, when I'm tying my shoe. But when you're talking about loop, that's a very specific thing in software and what you can do. And when you shouldn't do it, and that's something that these, no matter how good your low code, no code solution might be, it's going to have to afford for that concept.And so a business user is still going to have to have some lower level capability to apply those concepts. And, and I I've yet to see anybody be able to get around that in their RPA solutions.[00:33:42] Jeremy: So in your experience, even though these vendors may sell it as being a tool that anybody can just sit down and use but then you would want a developer to, to sit with them or, or see the result and, and try and figure out, okay, what do you, what do you really want this, this code to do?Um, not just sort of these broad strokes that you were hoping the tool was gonna take care of for you? Yeah.[00:34:06] Alex: that that's exactly right. And that's how every organization will come to that realization pretty quickly. the head of the game ones have said, okay, we need to have a really good, um, COE structure to this robotic operating model where we can have, a software engineering, developer capability that sits with the business, capability.And they can, marry with each other, other businesses who may take, um, these vendors at their word and say, it's a low code meant for business. It just needs to make sure it's on and accessible. And then our business people are just gonna, uh, go in there and do this. They find out pretty quickly that they need some technical, um, guidance to go in because they're building unstable or inaccurate bots.and whether they come to that sooner or later, they, they always come to that. Um, and they realize that, okay, there there's a technical capability And, this is not just RPA. This is the story of all low-code no-code solutions that have ever existed. It always comes around that, while this is a great interface for doing that, and it's very easy and it makes concepts easy.Every single time, there is a technical capability that needs to be afforded. [00:35:26] Jeremy: For the. The web browser, you mentioned the DOM, which is how we typically interact with applications there. But for native applications, you, you briefly mentioned, COM. And I was wondering when someone is writing, um, you know, a bot, uh, what are the sorts of things they see, or what are the primitives they're working with?Like, is there a name attached to each button, each text, field, [00:35:54] Alex: wouldn't that be a great world to live in, so there's not. And, and, as we build things in the DOM. People get a lot better. We've seen people are getting much better about using uniqueness when they build those things so that they can latch onto when things were built or built for the COM or, you know, a .NET for OS that might, that was not no one no one was like oh yeah, we're going to automate this.Or, you know, we need to make this so that this button here is going to be unique from that button over there on the COM they didn't care, you know, different name. Um, so yeah, that is, that is sometimes a big issue when you're using, uh, an RPA solution, you say, okay. cool. Look at this, your calculator app. And Okay. it's showing me the component object model that this was built. It that's describing what is looking at, but none of these nodes have, have a name. They're all, you know, node one node, 1.1 node two, or, or whatnot, or button is just button and there's no uniqueness around it. And that is, you see a lot of that in legacy older software, um, E legacy is things built in 2005, 2010.Um, you do see that, and that's the difficulty at that point. You can still solve for this, but what you're doing is you're using send keys. So instead of saying, Okay.RPA software, open up this, uh, application and then look for. You know, thing, this object in the COM and click on it, it's going to, you know, it can't, there is no uniqueness.So what you say is just open up the software and then just hit tab three times, and that should get you to this one place that was not unique, but we know if you hit tab three times, it's going to get there now. That's all well and good, but there's so many things that could interfere with that and break it.And the there's no context for the bot to grab onto, to verify, Okay. I am there. So any one thing, you could have a pop-up which essentially hijacks your send key, right? And so the bot yes, absolutely hit tab three times and it should be in that one place. It thinks it is, and it hits in enter, but in between the first and second tab, a pop-up happened and now it's latched onto this other process, hits enter. And all of a sudden outlook's opening bot doesn't know that, but it's still going on and it's going to enter in some financial information into oops, an email that it launched because it thought hitting enter again would do so. Yeah.That's, that's where you get that instability. Um, there are other ways around it or other solutions.and this is where we get into the you're using, um, lower level software engineering solutioning instead of doing it exactly how the user does it. When we're talking about the operating system and windows, there are a ton of interop services and assemblies that a, uh, RPA solution can access.So instead of cracking open Excel, double-clicking on Excel workbook waiting for it to load, and then reading the information and putting information in, you can use the, you know, the office 365 or whatnot that, um, interop service assembly and say, Hey, launch this workbook without the UI, showing it, attach to that process that, you know, it is.And then just send to it, using that assembly send information into it. And the human user can't do that. It can't manipulate stuff like that, but the bot can, and it it's the same end as the human users trying. And it's much more efficient and stable because the UI couldn't afford for that kind of stability.So that would be a valid solution. But at that point, you're really migrating into a software engineering, it developer solution of something that you were trying not to do that for. So when is that? Why, you know, why not just go and solve for it with an enterprise or programmatic solution in the first place?So that's the balance. [00:40:18] Jeremy: earlier you're talking about the RPA needs to be something that, uh, that the person is able to do. And it sounds like in this case, I guess there still is a way for the person to do it. They can open up the, the Excel sheet and right it's just that the way the, the RPA tool is doing it is different. Yeah.[00:40:38] Alex: Right. And more efficient and more stable. Certainly. Uh, especially when we're talking about Excel, you have an Excel with, you know, 200,000 lines, just opening that that's, that's your day, that's going to Excel it, just going to take its time opening and visualizing that information for you. Whereas you, you know, an RPA solution doesn't even need to crack that open.Uh, it can just send data right directly to that workbook and it that's a valid solution. And again, some of these processes, it might be just two people at your organization that are essentially doing it. So it's, you know, you don't really, it's not at a threshold where you need an enterprise solution for it, but they're spending 30 minutes of their day just waiting for that Excel workbook to open and then manipulating the data and saving it.And then, oh, their computer crashed. So you can do an RPA solution. It's going to be, um, to essentially build for a more efficient way of doing it. And that would be using the programmatic solution, but you're right. It is doing it in a way that a human could not achieve it. Um, and that again is. The where the discipline and the organizational, aspect of this comes in where it's saying, is that acceptable?Is it okay to have it do things in this way, that are not human, but achieving the same ends. And if you're not disciplined that creeps, and all of a sudden you have a RPA solution that is doing things in a way that where the whole reason to bring that RPA solution is to not have something that did something like that. And that's usually where the stuff falls apart. IT all of a sudden perks their head up and says, wait, I have a lot of connections coming in from this one computer doing stuff very quickly with a, you know, a SQL query. It's like, what is going on? And so all of a sudden, someone built a bot to essentially do a programmatic connection.And it is like, you should not be who gave you this permissions who did this shut down everything that is RPA here until we figure out what you guys went and did. So that's, that's the dance. [00:42:55] Jeremy: it's almost like there's this hidden API or there's this API that you're not intended to use. but in the process of trying to automate this thing, you, you use it and then if your, IT is not aware of it, then things just kind of spiral out of control.[00:43:10] Alex: Exactly. Right. So let's, you know, a use case of that would be, um, we need to get California tax information on alcohol sales. We need to see what each county taxes for alcohol to apply to something. And so today the human users, they go into the California, you know, tobacco, wildlife, whatever website, and they go look up stuff and okay, let's, that's, that's very arduous.Let's throw a bot on that. Let's have a bot do that. Well, the bot developers, smart person knows their way around Google and they find out, well, California has an API for that. instead of the bot cracking open Chrome, it's just going to send this rest API call and it's going to get information back and that's awesome and accurate and way better than anything. but now all of a sudden IT sees connections going in and out. all of a sudden it's doing very quickly and it's getting information coming into your systems in a way that you did not know was going to be, uh, happening. And so while it was all well and good, it's, it's a good way for, the people whose job it is to protect yourself or know about these things, to get very, um, angry, rightly so that this is happening.that's an organizational challenge, uh, and it's an oversight challenge and it's a, it's a developer challenge because, what you're getting into is the problems with having too technical people build these RPA bots, right? So on one hand we have business people who are told, Hey, just crack this thing open and build it.And it's like, well, they don't have enough technical fluency to actually build a stable bot because they're just taking it at face value. Um, on the other hand, you have software engineers or developers that are very technical that say, oh, this process. Yeah. Okay. I can build a bot for that. But what if I used, you know, these interop services, assemblies that Microsoft gives me and I can access it like that.And then I can send an API call over here. And while I'm at it, I'm going to, you know, I'm just going to spin up a server just on this one computer that can do this. When the bot talks to it. And so you have the opposite problem. Now you have something that is just not at all RPA, it's just using the tool to, uh, you know, manipulate stuff, programmatically.[00:45:35] Jeremy: so, as a part of all this, is using the same credentials as a real user, right. You're you're logging in with a username and password. if the form requires something like two factor authentication or, you know, or something like that, like, how does that work since it's not an actual person?[00:45:55] Alex: Right. So in a perfect world, you're correct. Um, a bot is a user. I know a lot of times you'll hear, say, people will be like, oh, hi, I have 20 RPA bots. What they're usually saying is I have 20 automations that are being run for separate processes, with one user's credentials, uh, on a VDI. So you're right.They, they are using a user's credentials with the same permissions that any user that does that process has, that's why it's easy. but now we have these concepts, like two factor authentication, which every organization is using that should require something that exists outside of that bot users environment. And so how do you afford for that in a perfect world? It would be a service account, not a user account and service accounts are governed a little differently. A lot of times service accounts, um, have much more stringent rules, but also allow for things like password resets, not a thing, um, or two factor authentication is not a thing for those.So that would be the perfect solution, but now you're dragging in IT. Um, so, you know, if you're not structurally set up for that, that's going to be a long slog. Uh, so what would want to do some people actually literally have a, we'll have a business person that has their two factor auth for that bot user on their phone.And then just, you know, they'll just go in and say, yeah.that's me. that's untenable. So, um, sometimes what a lot of these, like Microsoft, for instance, allow you to do is to install a two factor authentication, application, um, on your desktop so that when you go to log in a website and says, Hey, type in your password.Cool. Okay. Give me that code. That's on your two factor auth app. The bot can actually launch that. Copy the code and paste it in there and be on its way. But you're right now, you're having to afford for things that aren't really part of the process you're trying to automate. They are the incidentals that also happen.And so you have to build your bot to afford for those things and interpret, oh, I need to do two factor authentication. And a lot of times, especially if you have an entirely business focused PA um, robotic operating model, they will forget about those things or find ways around them that the bot isn't addressing, like having that authenticator app on their phone.that's, um, stuff that definitely needs to be addressed. And sometimes is only, found at runtime like, oh, it's asking for login. And when I developed it, I didn't need to do that because I had, you know, the cookie that said you're good for 30 days, but now, oh, no. [00:48:47] Jeremy: yeah. You could have two factor. Um, you could have, it asking you to check your email for a code. There could be a fraud warning. There's like all sorts of, you know, failure cases that can happen. [00:48:58] Alex: exactly. And those things are when we talk about, uh, third-party vendors, um, third-party provider vendors, like going back to the banking website, if you don't tell them that you're going to be using a bot to get their information or to interface with that website, you're setting yourself up for a bad time because they're going to see that kind of at runtime behavior that is not possible at scale by user.And so you run into that issue at runtime, but then you're correct. There are other things that you might run into at runtime that are not again, part of the process, the business didn't think that that was part of the process. It's just something they do that actually the bot has to afford for. that's part of the journey, uh, in building these. [00:49:57] Jeremy: when you're, when you're building these, these bots, what are the types of tools that, that you've used in the past? Are these commercial, packages, are these open source? Like what, what does that ecosystem look like?[00:50:11] Alex: Yeah, in this space, we have three big ones, which is, uh, automation anywhere UI path and, blue prism. Those are the RPA juggernauts providing this software to the companies that need it. And then you have smaller ones that are, trying to get in there, or provide stuff in a little different way. and you even have now, big juggernauts that are trying to provide for it, like Microsoft with something like power automate desktop.So all of these, say three years ago, all of these softwares existed or all of these RPA solution softwares existed or operated in the same kind of way, where you would install it on your desktop. And it would provide you a studio to either record or define, uh, originally the process that was going to be automated on that desktop when you pushed play and they all kind of expose or operate in the same way they would interpret the COM or the DOM that the operating system provided. Things like task scheduler have traditionally, uh, exposed, uh, and they all kind of did that in the same way. Their value proposition in their software was the orchestration capability and the management of that.So I build a bot to do this, Jim over there built a bot to do that. Okay. This RPA software, not only enabled you to define those processes, But what their real value was is they enable a place where I can say this needs to run at this time on this computer.And it needs to, you know, I need to be able to monitor it and it needs to return information and all that kind of orchestration capability. Now all of these RPA solutions actually exist in that, like everything else in the browser. So instead of installing, you know, the application and launching it and, and whatnot, and the orchestration capability being installed on another computer that looked at these computers and ran stuff on them.Now it's, it's all in the cloud as it were, and they are in the browser. So I go to. Wherever my RPA solution is in my browser. And then it says, okay, cool. You, you still need to install something on the desktop where you want the spot to run and it deploys it there. But I define and build my process in the provided browser studio.And then we're going to give you a capability to orchestrate, monitor, and, uh, receive information on those things that you have, those bots that you have running, and then what they're now providing as well is the ability to tie in other services to your bot so that it has expanded capability. So I'm using automation anywhere and I built my bot and it's going, and it's doing this or that.And automation anywhere says, Hey, that's cool. Wouldn't you like your bot to be able to do OCR? Well, we don't have our own OCR engine, but you probably as an enterprise do. Just use, you know, use your Kofax OCR engine or Hey, if you're really a high speed, why don't you use your Azure cognitive services capability?We'll tie it right into our software. And so when you're building your bot, instead of just cracking open a PDF and send key control C and key control V to do stuff instead, we'll use your OCR engine that you've already paid for to, to understand stuff. And so that's, how they expand, what they're offering, um, into addressing more and more capabilities.[00:53:57] Alex: But now we're, we're migrating into a territory where it's like, well, things have APIs why even build a bot for them. You know, you can just build a program that uses the API and the user can drive this. And so that's where people kind of get stuck. It's they they're using RPA on a, something that just as easily provides for a programmatic solution as opposed to an RPA solution.but because they're in their RPA mode and they say, we can use a bot for everything, they don't even stop and investigate and say, Hey, wouldn't this be just as easy to generate a react app and let a user use this because it has an API and IT can just as easily monitor and support that because it's in an Azure resource bucket.that's where an organization needs to be. Clear-eyed and say, Okay. at this point RPA is not the actual solution. We can do this just as easy over here and let's pursue that. [00:54:57] Jeremy: the experience of making these RPAs. It sounds like you have this browser-based IDE, there's probably some kind of drag and drop set up, and then you, you, you mentioned JavaScript. So I suppose, does that mean you can kind of dive a little bit deeper and if you want to set up specific rules or loops, you're actually writing that in JavaScript.[00:55:18] Alex: Yeah. So not, not necessarily. So, again, the business does not know what an IDE is. It's a studio. Um,so that's, but you're correct. It's, it's an IDE. Um, each, whether we're talking about blue prism or UiPath or automation anywhere, they all have a different flavor of what that looks like and what they enable.Um, traditionally blue prism gave you, uh, a studio that was more shape based where you are using UML shapes to define or describe your process. And then there you are, whereas automation anywhere traditionally used, uh, essentially lines or descriptors. So I say, Hey, I want to open this file. And your studio would just show a line that said open file.You know, um, although they do now, all of them have a shape based way to define your process. Go here, here. You know, here's a circle which represents this. Uh, let's do that. Um, or a way for you to kind of more, um, creatively define it in a, like a text-based way. When we talk about Java script, um, or anything like that, they provide predefined actions, all of them saying, I want to open a file or execute this that you can do, but all of them as well, at least last time I checked also allow you for a way to say, I want to programmatically run something I want to define.And since they're all in the browser, it is, uh, you know, Javascript that you're going to be saying, Hey, run this, this JavaScript, run this function. Um, previously, uh, things like automation anywhere would, uh, let you write stuff in, in .NET essentially to do that capability. But again, now everything's in the browser.So yeah, they do, They do provide for a capability to introduce more low level capability to your automation. That can get dangerous. Uh, it can be powerful and it can be stabilizing, but it can be a very slippery slope where you have an RPA solution bot that does the thing. But really all it does is it starts up and then executes code that you built.[00:57:39] Alex: Like what, what was the, the point in the first place? [00:57:43] Jeremy: Yeah. And I suppose at that point, then anybody who knows how to use the RPA tool, but isn't familiar with that code you wrote, they're just, they can't maintain it [00:57:54] Alex: you have business continuity and this goes back to our, it has to be replicable or close as close to the human process, as you can make. Because that's going to be the easiest to inherit and support. That's one of the great things about it. Whereas if you're a low level programmer, a dev who says, I can easily do this with a couple of lines of, you know, dot net or, you know, TypeScript or whatever.And so the bot just starts up in executes. Well, unless someone that is just as proficient comes along later and says, this is why it's breaking you now have an unsupportable business, solution. that's bad Juju. [00:58:38] Jeremy: you have the software engineers who they want to write code. then you have the people who are either in business or in IT that go, I don't want to look at your code.I don't want to have to maintain it. Yeah. So it's like you almost, if you're a software engineer coming in, you almost have to fight that urge to, to write anything yourself and figure out, okay, what can I do with the tool set and only go to code if I can't do it any other way.[00:59:07] Alex: That's correct. And that's the, it takes discipline. more often than not, not as fun as writing the code where you're like, I can do this. And this is really where the wheels come off is. You went to the business that is that I have this process, very simple. I need to do this and you say, cool, I can do that.And then you're sitting there writing code and you're like, but you know what? I know what they really want to do. And I can write that now. And so you've changed the process and while it is, and nine times out of 10, the business will be like, oh, that's actually what we wanted. The human process was just as close as we could get nothing else, but you're right.That's, that's exactly what we needed. Thank you nine times out of 10. They'll love you for that. But now you own their process. Now you're the one that defined it. You have to do the business continuity. You have to document it. And when it falls over, you have to pick it back up and you have to retrain.And unless you have an organizational capacity to say, okay, I've gone in and changed your process. I didn't automate it. I changed it. Now I have to go in and tell you how I changed it and how you can do it. And so that, unless you have built your robotic operating model and your, your team to afford for that, your developer could be writing checks bigger than they can cash.Even though this is a better capability. [01:00:30] Jeremy: you, you sort of touched on this before, and I think this is probably the, the last topic we'll cover, but you've been saying how the end goal should be to not have to use the RPAs anymore And I wonder if you have any advice for how to approach that process and, and what are some of the mistakes you've seen people make[01:00:54] Alex: Mm Hmm. I mean the biggest mistake I've seen organizations make, I think is throwing the RPA solution out there, building bots, and they're great bots, and they are creating that value. They're enabling you to save money and also, enabling your employees to go on and do better, more gratifying work. but then they say, that's, it that's as far as we're going to think, instead of taking those savings and saying, this is for replacing this pain point that we had to get a bot in the first place to do so.That's a huge common mistake. Absolutely understandable if I'm a CEO or even, you know, the person in charge of, you know, um, enterprise transformation. Um, it's very easy for me to say, ha victory, here's our money, here's our savings. I justified what we've done. Go have fun. Um, and instead of saying, we need to squirrel this money away and give it to the people that are going to change the system. So that, that's definitely one of the biggest things.The problem with that is that's not realized until years later when they're like, oh, we're still supporting these bots. So it is upfront having a turnoff strategy. When can we turn this bot off? What is that going to look like? Does it have a roadmap that will eventually do that?And that I think is the best way. And that will define what kind of processes you do indeed build bots for is you go to it and say, listen, we've got a lot of these user processes, human processes that are doing this stuff. Is there anything on your roadmap that is going to replace that and they say, oh yeah you know, in three years we're actually going to be standing up our new thing.We're going to be converting. And part of our, uh, analysis of the solution that we will eventually stand up will be, does it do these things? And so yes, in three years, you're good. And you say, cool, those are the processes I'm going to automate and we can shut those off. That's your point of entry for these things not doing that leads to bots running and doing things even after there is a enterprise solution for that. And more often than not, I would say greater than five times out of 10, when we are evaluating a process to build a bot for easily five times out of 10, we say, whoa, no, actually there's, you don't even need to do this.Our enterprise application can do this. you just need retraining, because your process is just old and no one knew you were doing this. And so they didn't come in and tell you, Hey, you need to use this.So that's really a lot of times what, what the issue is. And then after that, we go in and say, Okay.no, there's, there's no solution for this. This is definitely a bot needs to do this. Let's make sure number one, that there isn't a solution on the horizon six months to a year, because otherwise we're just going to waste time, but let's make sure there is, or at least IT, or the people in charge are aware that this is something that needs to be replaced bot or no bot.And so let's have an exit strategy. Let's have a turn-off strategy. When you have applications that are relatively modern, like you have a JIRA, a ServiceNow, you know, they must have some sort of API and it may just be that nobody has come in and told them, you just need to plug these applications together.[01:04:27] Alex: And so kind of what you're hitting on and surfacing is the future of RPA. Whereas everything we're talking about is using a bot to essentially bridge a gap, moving data from here to there that can't be done, programmatically. Accessing something from here to there that can't be done programmatically.So we use a bot to do it. That's only going to exist for so long. Legacy can only be legacy for so long, although you can conceivably because we had that big COBOL thing, um, maybe longer than we we'd all like, but eventually these things will be. upgraded. and so either the RPA market will get smaller because there's less legacy out there.And so RPA as a tool and a solution will become much more targeted towards specific systems or we expand what RPA is and what it can afford for. And so that I think is more likely the case. And that's the future where bots or automations aren't necessary interpreting the COM and the DOM and saying, okay, click here do that.But rather you're able to quickly build bots that utilize APIs that are built in and friendly. And so what we're talking about there is things like Appian or MuleSoft, which are these kind of API integrators are eventually going to be classified as RPA. They're going to be within this realm.And I think, where, where you're seeing that at least surfaced or moving towards is really what Microsoft's offering in that, where they, uh, they have something called power automate, which essentially is it just a very user-friendly way to access API. that they built or other people have built.So I want to go and I need to get information to service now, service now has an API. Yeah. Your, IT can go in and build you a nice little app that does a little restful call to it, or a rest API call to it gets information back, or you can go in and, you know, use Microsoft power automate and say, okay, I want to access service now.And it says, cool. These are the things you can do. And I say, okay, I just want to put information in this ticket and we're not talking about get or patch or put, uh, or anything like that. We're just saying, ah, that's what it's going to do. And that's kind of what Microsoft is, is offering. I think that is the new state of RPA is being able to interface in a user-friendly way with APIs. Cause everything's in the browser to the point. where, you know, Microsoft's enabling add ins for Excel to be written in JavaScript, which is just the new frontier. Um, so that's, that's kind of going to be the future state of this. I believe. [01:07:28] Jeremy: so, so moving from RPAs being this thing, that's gonna click through website, click through, um, a desktop application instead it's maybe more of this high, higher level tool where the user will still get this, I forget the term you used, but this tool to build a workflow, right. A studio. Okay. Um, and instead of saying, oh, I want this to click this button or fill in this form.It'll be, um, I want to get this information from service now. And I want to send a message using that information to slack or to Twilio, or, um, you're basically, talking directly to these different services and just telling it what you want and where it should go.[01:08:14] Alex: That's correct. So, as you said, everything's going to have an API, right? Seemingly everything has an API. And so instead of us, our RPA bots or solutions being UI focused, they're going to be API focused, um, where it doesn't have to use the user interface. It's going to use the other service. And again, the cool thing about APIs in that way is that it's not, directly connecting to your data source.It's the same as your UI for a user. It sits on top of it. It gets the request and it correctly interprets that. And does it the same thing with your UI where I say I click here and you know, wherever it says. okay. yeah. You're allowed to do that. Go ahead. So that's kind of that the benefit to that.Um, but to your point, the, the user experience for whether you're using a UI or API to build up RPA bot, it's going to be the same experience for the user. And then at this point, what we're talking about, well, where's the value offering or what is the value proposition of RPA and that's orchestration and monitoring and data essentially.we'll take care of hosting these for you. we'll take care of where they're going to run, uh, giving you a dashboard, things like that.[01:09:37] Alex: That's a hundred percent correct. It's it's providing a view into that thing and letting the business say, I want to no code this. And I want to be able to just go in and understand and say, oh, I do want to do that. I'm going to put these things together and it's going to automate this business process that I hate, but is vital, and I'm going to save it, the RPA software enables you to say, oh, I saw they did that. And I see it's running and everything's okay in the world and I want to turn it on or off. And so it's that seamless kind of capability that that's what that will provide. And I think that's really where it isn't, but really where it's going. Uh, it'll be interesting to see when the RPA providers switch to that kind of language because currently and traditionally they've gone to business and said, we can build you bots or no, no, your, your users can build bots and that's the value proposition they can go in.And instead of writing an Excel where you had one very, very advanced user. Building macros into Excel with VBA and they're unknown to the, the IT or anybody else instead, you know, build a bot for it. And so that's their business proposition today. Instead, it's going to shift, and I'd be interested to see when it shifts where they say, listen, we can provide you a view into those solutions and you can orchestrate them in, oh, here's the studio that enables people to build them.But really what you want to do is give that to your IT and just say, Hey, we're going to go over here and address business needs and build them. But don't worry. You'll be able to monitor them and at least say, yeah okay. this is, this is going.[01:11:16] Jeremy: Yeah. And that's a, a shift. It sounds like where RPA is currently, you were talking about how, when you're configuring them to click on websites and GUIs, you really do still need someone with the software expertise to know what's going on. but maybe when you move over to communicating with API, Um, maybe that won't be as important maybe, somebody who just knows the business process really can just use that studio and get what they need.[01:11:48] Alex: that's correct. Right. Cause the API only enables you to do what it defined right. So service now, which does have a robust API, it says you can do these things the same as a user can only click a button that's there that you've built and said they can click. And so that is you can't go off the reservation as easy with that stuff, really what's going to become prime or important is as no longer do I actually have an Oracle server physically in my location with a database.Instead I'm using Oracle's cloud capability, which exists on their own thing. That's where I'm getting data from. What becomes important about being able to monitor these is not necessarily like, oh, is it falling over? Is it breaking? It's saying, what information are you sending or getting from these things that are not within our walled garden.And that's really where, it or the P InfoSec is, is going to be maybe the main orchestrator owner of RPA, because they're, they're going to be the ones to say you can't, you can't get that. You're not allowed to get that information. It's not necessarily that you can't do it. Um, and you can't do it in a dangerous way, but it's rather, I don't want you

SummaryMads Torgersen answers questions from listeners about the upcoming release of C# 10.DetailsDeprecated features. Extension everything, some background, some possible features, starting over, an extension interface. Roles and shapes, maybe preview in C# 11, maybe release in C# 12 - "the edge of programming languages". Is the work in the design or the implementation of a feature; keeping the spirit of the language, harmony, and philosophy. Hot reload and impact on language. Performance improvements. C# and Linux; .NET is a cross-platform framework, not tied to Windows, Bryan has written a lot of .NET that runs on Linux, even MS SQL apps. Mads is not making C# into F#.Support this podcastFull show notes@madstorgersenMads' blogWhat's new in C# 10.0GitHub dotnet/csharplangBryan's blog posts on running .NET in Linux containers

2021-10-27 Weekly News - Episode 123Watch the video version on YouTube at https://www.youtube.com/watch?v=dLQhiLcHpH0 Hosts: Brad Wood - Senior Developer for Ortus SolutionsGavin Pickin - Senior Developer for Ortus SolutionsThanks to our Sponsor - Ortus SolutionsThe makers of ColdBox, CommandBox, ForgeBox, TestBox and almost every other Box out there. A few ways  to say thanks back to Ortus Solutions: Like and subscribe to our videos on YouTube.  Sign up for a free or paid account on CFCasts, which is releasing new content every week Buy Ortus's new Book - 102 ColdBox HMVC Quick Tips and Tricks on GumRoad (http://gum.co/coldbox-tips) Patreon SupportWe have 37 patreons providing 93% of the funding for our Modernize or Die Podcasts via our Patreon site: https://www.patreon.com/ortussolutions. Now offering Annual Memberships, pay for the year and save 10% - great for businesses.News and EventsPreside Version 10.16.0 is outSee our release and upgrade notes/video:Video: https://t.co/OZo8qRURWe Release Notes: https://t.co/bSt8vA9OT3 Documentation: https://t.co/k3P3rHff6k Online CF Meetup - Using LaunchDarkly for feature flag management in CF applications, w/ Brad WoodThursday, October 28, 2021 at 9:00 AM to 10:00 AM PDTFeature flags are a system of enabling certain functionality in your app based on test groups, cross-cutting segments of users, and your internal release processes. Feature flags can be updated on the fly at any time by any user and don't require deploying new code to your servers. LaunchDarkly is a system that helps you manage your feature flags and how they respond to the users of your site. It offers detailed tracking of each user, each flag, and a robust set of rules for determining which users see which features. In this session, we'll see an overview of how to use the new LaunchDarkly SDK which can be used in ColdFusion applications. Demos will include both ColdBox apps and non-ColdBox legacy apps.https://www.meetup.com/coldfusionmeetup/events/281577538/ Adobe 1 Day Workshop - Adobe ColdFusion Workshop with Damien BruyndonckxWed, November 10, 202109:00 - 17:00 CEST EUROPEANJoin the Adobe ColdFusion Workshop to learn how you and your agency can leverage ColdFusion to create amazing web content. This one-day training will cover all facets of Adobe ColdFusion that developers need to build applications that can run across multiple cloud providers or on-premise.https://coldfusion-workshop.meetus.adobeevents.com/ ICYMI - Into the Box 2021 - Videos are now availableVideos are now available on CFCasts!https://cfcasts.com/series/into-the-box-2021Free for subscribers; Free for ITB 2021 attendees; available as a one-time purchase for $199.If you bought a ticket to Into the Box 2021 and have not received a coupon for access to the videos on CFCasts, please contact us from the CFCasts support page. https://cfcasts.com/supportICYMI - Ortus Webinar for October - Gavin Pickin - Building Quick APIs - the extended versionIn this session we will use ColdBox's built in REST BaseHandler, and with CBSecurity and Quick ORM we will set up a secure API using fluent query language - and you'll see how quick Quick development can be!https://www.ortussolutions.com/events/webinarsRecording will be posted to CFCasts soonHacktoberfest 2021Support open source throughout October!Hacktoberfest encourages participation in the open source community, which grows bigger every year. Complete the 2021 challenge and earn a limited edition T-shirt.GIVING TO OPEN SOURCEOpen-source projects keep the internet humming—but they can't do it without resources. Donate and support their awesome work.TREES NOT TEESRather than receive t-shirts as swag, you can choose to have a tree planted in your name and help make Hacktoberfest 2021 more carbon neutral.To win a reward, you must sign up on the Hacktoberfest site and make four pull requests on any repositories classified with the 'hacktoberfest 'topic on GitHub or GitLab by October 31. If an Ortus Solutions repo that you want to contribute to is not marked with the `hacktoberfest` topic, please let us know so we can fix it.https://hacktoberfest.digitalocean.com/ CFCasts Content Updateshttps://www.cfcasts.com Just ReleasedUp and Running with Quick Testing with Quick Step 11 Exercise Coming this week Up and Running with Quick Building Quick APIs Send your suggestions at https://cfcasts.com/supportConferences and TrainingMicrosoft IgniteNovember 2–4, 2021 Opportunity awaits, with dedicated content spotlighting Microsoft Business Applications and Microsoft Security.https://myignite.microsoft.com/homeDeploy by Digital OceanTHE VIRTUAL CONFERENCE FOR GLOBAL DEVELOPMENT TEAMSNovember 16-17, 2021 https://deploy.digitalocean.com/homeAWS re:InventNOV. 29 – DEC. 3, 2021 | LAS VEGAS, NVCELEBRATING 10 YEARS OF RE:INVENTVirtual: FreeIn Person: $1799https://reinvent.awsevents.com/ Postgres BuildOnline - FreeNov 30-Dev 1 2021https://www.postgresbuild.com/ ITB Latam 2021December 2-3, 2021Into the Box LATAM is back and better than ever! Our virtual conference will include speakers from El Salvador and all over the world, who'll present on the latest web and mobile technologies in Latin America.Registration is completely free so don't miss out!https://latam.intothebox.org/ Adobe ColdFusion Summit 2021December 7th and 8th - VirtualSpeakers are finalized and some Speakers and some session descriptions are now on the siteRegister for Free - https://cfsummit.vconfex.com/site/adobe-cold-fusion-summit-2021/1290Blog - https://coldfusion.adobe.com/2021/09/adobe-coldfusion-summit-2021-registrations-open/ Tweet from Mark Takata OK! I can finally let you all know that for the @Adobe @coldfusion #CFSummit2021 keynote we will be featuring @ashleymcnamara! Her talk will focus on the history & future of DevRel how we got here & where we're going.cfsummit.vconfex.com to register!#CFML #DevRel #conferencehttps://twitter.com/MarkTakata/status/1449063259072438277 https://twitter.com/MarkTakata jConf.devNow a free virtual eventDecember 9th starting at 8:30 am CDT/2:30 pm UTC.https://2021.jconf.dev/?mc_cid=b62adc151d&mc_eid=8293d6fdb0 More conferencesNeed more conferences, this site has a huge list of conferences for almost any language/community.https://confs.tech/Blogs, Tweets and Videos of the WeekBlog - Ben Nadel - Reading Environment (ENV) Variables From The Server Scope In Lucee CFML 5.3.7.47This is a pro-tip that I originally picked up from Julian Halliwell a few years ago. However, I sometimes talk to people who don't realize that this is possible. So, I wanted to try and amplify Julian's post. In Lucee CFML, you can read environment (ENV) variables directly out of the server scope. They are just automatically there - no dipping into the Java layer or dealing with the java.lang.System class. Lucee CFML brings these values to the surface for easy consumption.https://www.bennadel.com/blog/4140-reading-environment-env-variables-from-the-server-scope-in-lucee-cfml-5-3-7-47.htm Blog - Ben Nadel - Making SQL Queries More Flexible With LIKE In MySQL 5.7.32 And Lucee CFML 5.3.7.47While you might stand-up something like Elasticsearch, Lucene, or Solr in order to provide robust and flexible text-based searches in your ColdFusion application, your relational database is more than capable of performing (surprisingly fast) pattern matching on TEXT and VARCHAR fields using the LIKE operator. This is especially true if the SQL query in question is already being limited based on an indexed value. At InVision, I often use the LIKE operator to allow for light-weight text-based searches. And, as of late, I've been massaging the inputs in order to make the matches even more flexible, allowing for some slightly fuzzy matching in Lucee CFML 5.3.7.47.https://www.bennadel.com/blog/4137-making-sql-queries-more-flexible-with-like-in-mysql-5-7-32-and-lucee-cfml-5-3-7-47.htm Blog - Ben Nadel - Creating A Group-Based Incrementing Value In MySQL 5.7.32 And Lucee CFML 5.3.7.47In the past few weeks, I've been learning a lot about how I can leverage SERIALIZABLE transactions in MySQL, the scope of said transactions, and some hidden gotchas around locking empty rows. As a means to lock (no pun intended) some of that information in my head-meat, I thought it would be a fun code kata to create a Jira-inspired ticketing system in Lucee CFML 5.3.7.47 that uses an application-defined, group-based incrementing value in MySQL 5.7.32.https://www.bennadel.com/blog/4135-creating-a-group-based-incrementing-value-in-mysql-5-7-32-and-lucee-cfml-5-3-7-47.htm Blog - Ben Nadel - Creating A Group-Based Incrementing Value Using LAST_INSERT_ID() In MySQL 5.7.32 And Lucee CFML 5.3.7.47Yesterday, I took inspiration from Jira's ticketing system and explored the idea of creating a group-based incrementing value in MySQL. In my approach, I used a SERIALIZABLE transaction to safely "update and read" a shared sequence value across parallel threads. In response to that post, my InVision co-worker - Michael Dropps - suggested that I look at using LAST_INSERT_ID(expr) to achieve the same outcome with less transaction isolation. I had never seen the LAST_INSERT_ID() function used with an expression argument before. So, I wanted to revisit yesterday's post using this technique.https://www.bennadel.com/blog/4136-creating-a-group-based-incrementing-value-using-last-insert-id-in-mysql-5-7-32-and-lucee-cfml-5-3-7-47.htm Blog / Documentation - Zac Spitszer - Building and testing Lucee extensions documentationI have written up a detailed guide on how to Build and Test Lucee Extensions, using Lucee Script Runner and Apache Ant.It's a little bit complicated to setup, but I have developed a toolchain, which once set up, makes the entire process really dead simple.https://dev.lucee.org/t/building-and-testing-lucee-extensions-documentation/9053 Tweet - Mark Takata - Adobe - The CF Summit 2021 Keynote announcementOK! I can finally let you all know that for the @Adobe @coldfusion #CFSummit2021 keynote we will be featuring @ashleymcnamara! Her talk will focus on the history & future of DevRel how we got here & where we're going.cfsummit.vconfex.com to register!#CFML #DevRel #conferencehttps://twitter.com/MarkTakata/status/1449063259072438277 https://twitter.com/MarkTakata Tweet - Ben Nadel - Monolith DeploysIt's 10:50 AM.I work in a monolithic #Lucee #CFML codebase.And, I just started my 3rd deployment of the day.It's amazing how much work you can get done when you stop worrying about what other people think of your technology choices.

FULL SHOW NOTES https://podcast.nz365guy.com/330A brief introduction about Mark Carrington's life Mark's view regarding the post COVID world and the concept of the way he works. The Pros and Cons of Working from Home Discussion about Mark's thoughts about employees returning to work A conversation with Mark about his journey and passion for technology. Talks about Mark's educational and career background. A story about how Mark landed his first job. Find out more about the interesting projects and reporting Mark worked on Check out Marks journey into becoming a Microsoft MVP Mark's first contribution tool for the community Check out Mark's ways to develop his skills. Discover more about the XrmToolBox and SQL 4 with Mark. Support the show (https://www.buymeacoffee.com/nz365guy)

For episode 400, Scott and Wes talk about web dev horror stories - 2021 edition! LogRocket - Sponsor LogRocket lets you replay what users do on your site, helping you reproduce bugs and fix issues faster. It's an exception tracker, a session re-player and a performance monitor. Get 14 days free at logrocket.com/syntax. Mux - Sponsor Mux Video is an API-first platform that makes it easy for any developer to build beautiful video. Powered by data and designed by video experts, your video will work perfectly on every device, every time. Mux Video handles storage, encoding, and delivery so you can focus on building your product. Live streaming is just as easy and Mux will scale with you as you grow, whether you're serving a few dozen streams or a few million. Visit mux.com/syntax. Linode - Sponsor Whether you're working on a personal project or managing enterprise infrastructure, you deserve simple, affordable, and accessible cloud computing solutions that allow you to take your project to the next level. Simplify your cloud infrastructure with Linode's Linux virtual machines and develop, deploy, and scale your modern applications faster and easier. Get started on Linode today with a $100 in free credit for listeners of Syntax. You can find all the details at linode.com/syntax. Linode has 11 global data centers and provides 24/7/365 human support with no tiers or hand-offs regardless of your plan size. In addition to shared and dedicated compute instances, you can use your $100 in credit on S3-compatible object storage, Managed Kubernetes, and more. Visit linode.com/syntax and click on the “Create Free Account” button to get started. Show Notes 02:54 - Hi guys, love the show. I wanted to share with you something that happened just the other day (Oct 4th), I was starting my new job today at a large tech company. They use React for everything (even DNS!, don't ask me how, it's complicated). I figured I'd celebrate my first day and push some code to prod, (how hard could useEffect be right?) Next thing you know, they ended up bringing in a guy with an angle grinder to get access to the server cage. 04:15 - No one from Denver can buy 06:38 - Bug accidentally gives $90 million to users https://www.cnbc.com/2021/10/01/defi-protocol-compound-mistakenly-gives-away-millions-to-users.html 08:34 - Share Pointy Knives Hi! I'm a developer at a consulting firm in Sweden, writing C# on the backend and using React with either JavaScript or TypeScript and hosting things in Azure 99% of the time (and 1% in SharePoint). I was in my last week at my last job before I was due to start my new job. Worked 12 h/day to keep up with all the handovers etc. to colleagues so they would have a chance to continue working on the solutions I have taken care of. One project was a process tool hosted in SharePoint Online. The guy who would oversee it had -1% experience with SharePoint (which I pointed out to my bosses). But to make things a bit easier, I wrote a deploy script to ease things a bit. Starts the terminal and runs the script towards the acceptance environment. Umpteen million errors appear… Which is strange, because there would only be about 20 commands (which can cause errors like these). I log into the environment to double check if I now accidentally entered the wrong values in the script (which looked okay according to me). But I get a 404 error when I try to reach the environment… I log into the admin interface; I discover that the site is gone… Also checking the trash can, there are no things there. Very strange. I find that I'm in a different folder than the one where I saved my script… In that folder there is an old deploy script that was used when the project was started a thousand years ago (which was not used after the project was “finished”). The first thing the script does is force delete the site and then try to create a new empty site… The site is gone with lists and everything (lists are a SharePoint thing, think of it as sql-lite), there are no backups of the acceptance environment (although it is very important). I just feel a little panicked about how I'm going to solve this. However, I remember testing a tool six months ago to copy entire environments. Where the first attempt was made on the acceptance environment. Finds the cloned environment and can use the same tool to clone it back. It took only 8-12 hours of work to create all the new things done in the environment in the last 6 months instead of X number of hours to build everything from scratch. Once I updated a feature that saves accessories on orders (same solution). However, I failed to add all the new fields to the production environment. Which meant that accessories were not saved at all… Which was discovered after a week… I fixed the error in 5 minutes and the sellers had to contact x number of customers to double check what kind of accessories they would have for their orders… 11:22 - External HD One time I needed to format a server. It was an outdated Windows server. I selected all the files and copied and pasted to an external hard drive. My drive was pretty fast and it took like a minute. I was like: “Wow! That's a great external hd”. Formatted the server and, as soon as I realized it didn't copy 10% of the files, I had that face. We all know that face. Anyways. Tried to restore the files using some HD recovery tools but they were all corrupted, not by the formatting itself but for the installation of the new OS. My boss was pissed! I was very young so I blame it on the server. I'm not proud of it. But why the heck they would ask a developer to format a server in the first place? By the way, my birthday is on Halloween. Spoooky. 13:07 - Hey Loser I was testing new code to automate mass-mailings to our customers. Who knows what demon drove me but I wrote the “test” mailings like ransom notes: “Dear loser! Fork over all your $$$ or else!” Well, all was looking great and I wa s feeling pretty pleased with myself. Progress bars were sliding and counters were spinning. But I could hear a rising commotion from the marketing guys behind me. Phones ringing, voices raised. Turns out I had moronically wired myself to the production database! Even worse for me, I'd only been at the company a month or two. I thought my goose was cooked and the Big Boss was plenty mad, but I owned up right away and apologized. We put out a cover story that we'd been hacked and all was forgiven. 15:01 - HE HATE ME I was part of the developer team that accidentally leaked the 8 cities the XFL, an alternate football league, a week before their press conference. ewrestling.com/article/wwe-ac… We were using Contentful and Gatsby. A junior dev entered the information into the prod space instead of the UAT space and when we released some bug fixes, it picked up the contact us content update. I found out after seeing stories pop up in Google News when I was about to go to sleep. Was taking the content down when we started getting calls from the CIO of the WWE. The league went bust because of COVID. 19:23 - I Don't Have Memory of This I had two pretty bad code changes that only showed their problems when they went live in production. Around 6 years ago, I was running into a large performance issue with some of our queries running slowly against this giant DB. We were using JPA/Hibernate and we had a bunch of joins that were done lazily. I switched a few of them to eager so that they would create a single SQL statement instead of a bunch (or thousands). The change worked fine on my dev environment, QA, and staging. Staging was supposed to be representative of production. So we went live and within minutes the entire system went down because of out of memory errors. We quickly switched back to the lazy joins. We found out that staging had more memory and fewer DB records than production though they were supposed to be exactly the same. 21:05 - Your Performance is Slowing us down Back when VMWare was becoming a thing, like 2010 or so. I was working at an ecomm site and we were seeing slow performance between the app server and some data services. I decided to build a little multithreaded logger that could track when a query to Oracle Financials was running too slow and generate a warning. Oracle Financials was doing the credit card transactions, orders, and all the rest of the sites DB work. The code had no impact on my dev, QA, and staging environments. We were hitting well over our minimum number of concurrent users. We deployed it to production and then the system got slower and slower, but never crashed. Again, production and staging were set up differently. Staging was a bare-metal server. Production was running on an ESXi server on a host that was split 4 ways. The multi-threaded code meant to detect performance degradations was slowing the whole system down when it tried to synchronize data across threads. I was pretty embarrassed by both these two issues. It went to show that production is its own special thing and that you really don't know if your server-side code is really going to work until it starts running there. 23:15 - Dead Button Way back when mainframes were king, a guy I worked with pushed a button in, that if released, would immediately take down the entire company. He stood there for 4 hours, holding the button in, until we could let it crash after business hours. We gave him a chair after 2 hours. 25:12 - No Deploys on Fridays I was a junior dev working on our company's website. They were HTML + nunjucks templates that were later being integrated with the backend using some Python witchcraft. There was also a metric ton of JS libraries added (like Babra for page transitions, threejs for a cool interactive animation on the landing page etc.). Didn't really get much of all this package.json stuff at that seniority level. So after running yarn or npm or whatever, and seeing some warnings about a couple packages being outdated, I decided to update some of them. It ran great locally, but I didn't build the prod version, as I didn't know there could be any differences. I was working on some minor feature (or maybe even some minor bug) and the PM decided there's no time for code review. So I pushed it to the repo, the backend guy did his integration, and launched it on prod. As it turned out, there were some breaking changes in one of the libraries I decided to update. It crashed the entire site. On Friday. At 4:30PM. And that, kids, is why you don't deploy on Fridays. 27:33 - Stupid Selfie Horror story for you Wes. I work for one of the biggest retailers in the UK and we were working on an app that would go on a ‘media wall' in their flagship store in London. Basically a giant 200-inch screen in the middle of the store that social content can go on. Turns out that I left my local Dev version connected to the production API when I uploaded a couple of stupid selfies of my big head in the office. Get a call the next day to ask why my face is on the medial wall. 28:37 - Soda I was a computer operator back in the late 1960's, operating a Honeywell mainframe. The consoles were huge, about the size of a dishwashing machine, with the console typewriter and printer inset in the middle, on top. I had a soft drink on the console, next to the typewriter mechanism. We were told never to bring a drink into the room but we all did it, especially on third shift. Long story short, someone called my name, I turned around and knocked the glass of soda into the console. Had to be completely replaced – machine was down for two days. My boss was not happy. 31:22 - Oof A bigger horror story. I had my own software company in the 90's and was in Singapore, customizing my software package for Johnson & Higgins Insurance Brokers – I had their Asian contract for my Insurance Broker/Accounting package. I spent a good 40 hours on Saturday and Sunday, making all the changes they asked for, getting ready for a demo on Monday morning. I finished up about 4am on Monday morning and was cleaning up my files. All this work was done on a Novell server. Print files had an extension of .prt and I had a ton of them in the main directory from all of the testing I had done. I was cleaning out old files, getting ready to back everything up and I thought I would delete all of the print files. I mistakenly keyed in erase *.prg, instead of erase *.prt (or whatever the delete command was – can't remember it now). Programming files have a .prg extension – I had deleted all of my updated files from the weekend. In desperation I called Novell in Utah, hoping they could help me recover the files, but no-go. The demo Monday morning was not fun. 33:24 - Young Dev I was a young dev right out of college. My first job was at a child support company where we had desktop apps that would handle case information more efficiently than using Excel. My first project was to write a POC that would later be implemented into a new, bigger app that consolidated all the “POCs” for various parts of the child support process. For some odd reason, I still don't know why to this day, my boss wanted me to write this “new” app on top of an old app with a bunch of legacy code. I never understood why but as a young dev fresh out of school, you tend to just do what you're told. In school, I mainly used PHP/HTML/CSS for learning how to work with a database; this job however used C#/.NET for their desktop apps so I was doing a lot of learning as I went. I remember finally learning how to connect to the database and run some SQL after fighting with this old pile of legacy code. In early versions, I chose to handle creates/updates for these records in the same function. My young, dumb self wrote a try catch statement that would attempt to create the record and if it failed, it would try to update the record. Before the first production release, I updated the flow to handle creates/updates in separate functions - but never removed the update in the catch block of the original function now used for creates only. Somehow I, or any PM/QA, never failed on a create and hit this catch block while testing. Fast-forward probably 9-12 months later, I got a ticket to investigate why every case's data looked the same in Production. I login to the app, search a few case numbers and sure enough, every case's data is the same. I began freaking out as I had no clue how this could've happened. I mean it had never happened in all the dev work, testing, and months of live Production use. After I investigated with a senior dev, we realized the try block had failed and the update query in the catch block ran for that record - we also realized that I left off the where clause in the related SQL query to specify which record needs updating - so ALL records got updated with this data. Thankfully, we kept regular back-ups and were able to restore the data to a recent timeframe without users losing a ton of work. We commented out that database update call and redeployed the code ASAP. Also the senior dev was cool about it and was like “hey, it happens to all of us at some point”. Let's just say I've learned a ton since then and definitely steer clear from writing code like that. You live and you learn I suppose. 38:40 - Where Wolf Here's my development tale of terror: One night I was burning the midnight oil trying to get caught up on a never-ending workload. At the time I was working for an online travel booking site. It was after 11, and the last thing I had to do for the night was to rename one of the hotels in our production database. So I wrote my query: UPDATE hotels SET name=‘Some Hotel Chain'; One problem, I FORGOT THE WHERE CLAUSE. Suddenly, over 5,000 hotels in our production database all had the same name. This was around 2003, so well before the time of point-in-time restores, and we were only backing up the database every week at that point. I was panicking. Fortunately, I had a dump of the production database that I had created only a couple of hours earlier sitting on my local hard drive. So thankfully, I was able to restore almost all of the hotel names, save for a couple that signed up after that data dump, and my boss was none the wiser. That's when I learned that working late hours is not worth it, because at some point you are so tired that you can no longer make good decisions. 41:19 - I Want Your Job When I first started out I worked for a consultancy and they trained us in sales meetings to help managers get promoted because we were coming in to make them “look good”. This was okay b/c obviously, we were coming in as a contractor; however, after being laid off due to 9/11 (yes, this was about 20 years ago), I was looking for a new job and during an interview when asked where I'd like to be in X years, I mentioned to the hiring manager that I wanted to eventually do what he was doing. Well, I guess he didn't take it that I wanted to make him get promoted to then take his spot. Safe to say I didn't get hired.

This is episode 209 recorded on October 25th, 2021 where John & Jason share their views on the upcoming Microsoft Ignite 2021 Conference and the state of remote conferences in General. For show notes please visit www.bifocal.show

In today's podcast we cover four crucial cyber and technology topics, including:  1. Scammers use SMS scam via fake android apps to steal money  2. Discourse installs vulnerable to remote code execution  3. BillQuick Web Suite exploited to compromise users  4. Kansas man used mobile phone to tamper with water plant remotely  I'd love feedback, feel free to send your comments and feedback to  | cyberandtechwithmike@gmail.com

Sadie St. Lawrence talks in-depth about her extensive work as a data science educator through both online and collegiate courses as well as her organization for diversifying data science careers. In this episode you will learn: • Sadie's education work in SQL [4:13] • The popularity of Sadie's course [13:32] • Sadie's forthcoming machine learning certificate course [16:29] • Women in Data [25:32] • Sadie's non-technical background [36:17] • NFTs and VR [46:41] Additional materials: www.superdatascience.com/517

About ScottScott first typed ‘docker run' in 2013 and hasn't looked back. He's been with Docker since 2014 in a variety of leadership roles and currently serves as CEO. His experience previous to Docker includes Sun Microsystems, Puppet, Netscape, Cisco, and Loudcloud (parent of Opsware). When not fussing with computers he spends time with his three kids fussing with computers.Links: Docker: https://www.docker.com Twitter: https://twitter.com/scottcjohnston TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Liquibase. If you're anything like me, you've screwed up the database part of a deployment so severely that you've been banned from touching every anything that remotely sounds like SQL, at at least three different companies. We've mostly got code deployments solved for, but when it comes to databases we basically rely on desperate hope, with a roll back plan of keeping our resumes up to date. It doesn't have to be that way. Meet Liquibase. It is both an open source project and a commercial offering. Liquibase lets you track, modify, and automate database schema changes across almost any database, with guardrails to ensure you'll still have a company left after you deploy the change. No matter where your database lives, Liquibase can help you solve your database deployment issues. Check them out today at liquibase.com. Offer does not apply to Route 53.Corey: This episode is sponsored in part by something new. Cloud Academy is a training platform built on two primary goals. Having the highest quality content in tech and cloud skills, and building a good community the is rich and full of IT and engineering professionals. You wouldn't think those things go together, but sometimes they do. Its both useful for individuals and large enterprises, but here's what makes it new. I don't use that term lightly. Cloud Academy invites you to showcase just how good your AWS skills are. For the next four weeks you'll have a chance to prove yourself. Compete in four unique lab challenges, where they'll be awarding more than $2000 in cash and prizes. I'm not kidding, first place is a thousand bucks. Pre-register for the first challenge now, one that I picked out myself on Amazon SNS image resizing, by visiting cloudacademy.com/corey. C-O-R-E-Y. That's cloudacademy.com/corey. We're gonna have some fun with this one!Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Once upon a time, I started my public speaking career as a traveling contract trainer for Puppet; I've talked about this before. And during that time, I encountered someone who worked there as an exec, Scott Johnston, who sat down, talked to me about how I viewed things, and then almost immediately went to go work at Docker instead. Today's promoted episode brings Scott on to the show. Scott, you fled to get away from me, became the CEO of Docker over the past, oh what is it, seven years now. You're still standing there, and I'm not making fun of Docker quite the way that I used to. First, thanks for joining me.Scott: Great to be here, Corey. Thanks for the invitation. I'm not sure I was fleeing you, but we can recover that one at another time.Corey: Oh, absolutely. In that era, one of my first talks that I started giving that anyone really paid any attention to was called, “Heresy in the Church of Docker,” where I listed about 10 to 13 different things that Docker didn't seem to have answers for, like network separation, security, audit logging, et cetera, et cetera. And it was a fun talk that I used to basically learn how to speak publicly without crying before and after the talk. And in time, it wound up aging out as these problems got addressed, but what surprised me at the time was how receptive the Docker community was to the idea of a talk that wound up effectively criticizing something that for, well, a number of them it felt a lot of the time like it wasn't that far from a religion; it was very hype-driven: “Docker, Docker, Docker” was a recurring joke. Docker has changed a lot. The burning question that I think I want to start this off with is that it's 2021; what is Docker? Is it a technology? Is it a company? Is it a religion? Is it a community? What is Docker?Scott: Yes. I mean that sincerely. Often, the first awareness or the first introduction that newcomers have is in fact the community, before they get their hands on the product, before they learn that there's a company behind the product is they have a colleague who is, either through a Zoom or sitting next to them in some places, or in a coffee shop, and says, “Hey, you got to try this thing called Docker.” And they lean over—either virtually or physically—and look at the laptop of their friend who's promoting Docker, and they see a magical experience. And that is the introduction of so many of our community members, having spoken with them and heard their own kind of journeys.And so that leads to like, “Okay, so why the excitement? Why did the friend lean over to the other friend and introduce?” It's because the tools that Docker provides just helps devs get their app built and shipping faster, more securely, with choice, without being tied into any particular runtime, any particular infrastructure. And that combination has proven to be a breakthrough dopamine hit to developers since the very beginning, since 2013, when Docker is open-source.Corey: It feels like originally, the breakthrough of Docker, that people will say, “Oh, containers aren't new. We've had that going back to LPARs on mainframes.” Yes, I'm aware, but suddenly, it became easy to work with and didn't take tremendous effort to get unified environments. It was cynically observed at the time by lots of folks smarter than I am, that the big breakthrough Docker had was how to make my MacBook look a lot more like a Linux server in production. And we talk about breaking down silos between ops and dev, but in many ways, this just meant that the silo became increasingly irrelevant because, “Works on my machine” was no longer a problem.“Well, you better back up your email because your laptop's about to go into production in that case.” Containers made it easier and that was a big deal. It seems, on some level, like there was a foray where Docker the company was moving into the world of, “Okay, now we're going to run a lot of these containers in production for you, et cetera.” It really feels like recently, the company as a whole and the strategy has turned towards getting back to its roots of solving developer problems and positioning itself as a developer tool. Is that a fair characterization?Scott: A hundred percent. That's very intentional, as well. We certainly had good products, and great customers, and we're solving problems for customers on the ops side, I'll call it, but when we stood back—this is around 2019—and said, “Where's the real… joy?” For lack of a better word, “Where's the real joy from a community standpoint, from a product experience standpoint, from a what do we do different and better and more capable than anyone else in the ecosystem?” It was that developer experience. And so the reset that you're referring to in November 2019, was to give us the freedom to go back and just focus the entire company's efforts on the needs of developers without any other distractions from a revenue, customer, channel, so on and so forth.Corey: So, we knew this was going to come up in the conversation, but as of a couple of weeks ago—as of the time of this recording—you announced a somewhat, well, let's say controversial change in how the pricing and licensing works. Now, as of—taking effect at the end of this year—the end of January, rather, of next year—Docker Desktop is free for folks to use for individual use, and that's fine, and for corporate use, Docker Desktop also remains free until you are a large company defined by ten million in revenue a year and/or 250 employees or more. And that was interesting and I don't think I'd seen that type of requirement placed before on what was largely an open-source project that's now a developer tool. I believe there are closed-source aspects of it as well for the desktop experience, but please don't quote me on that; I'm not here to play internet lawyer engineer. But at that point, the internet was predictably upset about this because it is easy to yell about any change that is coming, regardless.I was less interested in that than I am in what the reception has been from your corporate customers because, let's be clear, users are important, community is important, but goodwill will not put food on the table past a certain point. There has to be a way to make a company sustainable, there has to be a recurring revenue model. I realize that you know this, but I'm sure there are people listening to this who are working in development somewhere who are, “Wait, you mean I need to add more value than I cost?” It was a hard revelation for [laugh] me back when I had been in the industry a few years—Scott: [laugh]. Sure.Corey: —and I'm still struggling with that—Scott: Sure.Corey: Some days.Scott: You and me both. [laugh].Corey: So, what has the reaction been from folks who have better channels of communicating with you folks than angry Twitter threads?Scott: Yeah. Create surface area for a discussion, Corey. Let's back up and talk on a couple points that you hit along the way there. One is, “What is Docker Desktop?” Docker Desktop is not just Docker Engine.Docker Desktop is a way in which we take Docker Engine, Compose, Kubernetes, all important tools for developers building modern apps—Docker Build, so on and so forth—and we provide an integrated engineered product that is engineered for the native environments of Mac and Windows, and soon Linux. And so we make it super easy to get the container runtime, Kubernetes stack, the networking, the CLI, Compose, we make it super easy just to get that up and running and configured with smart defaults, secured, hardened, and importantly updated. So, any vulnerabilities patched and so on and so forth. The point is, it's a product that is based on—to your comments—upstream open-source technologies, but it is an engineered commercial product—Docker Desktop is.Corey: Docker Desktop is a fantastic tool; I use it myself. I could make a bunch of snide comments that on Mac, it's basically there to make sure the fans are still working on the laptop, but again, computers are hard. I get that. It's incredibly handy to have a graphical control panel. It turns out that I don't pretend to understand those people, but some folks apparently believe that there are better user interfaces than text and an 80-character-wide terminal window. I don't pretend to get those people, but not everyone has the joy of being a Linux admin for far too long. So, I get it, making it more accessible, making it easy, is absolutely worth using.Scott: That's right.Corey: It's not a hard requirement to run it on a laptop-style environment or developer workstation, but it makes it really convenient.Scott: Before Docker desktop, one had to install a hypervisor, install a Linux VM, install Docker Engine on that Linux VM, bridge between the VM and the local CLI on the native desktop—like, lots of setup and maintenance and tricky stuff that can go wrong. Trust me how many times I stubbed my own toes on putting that together. And so Docker Desktop is designed to take all of that setup nonsense overhead away and just let the developer focus on the app. That's what the product is, and just talking about where it came from, and how it uses these other upstream technologies. Yes, and so we made a move on August 31, as you noted, and the motivation was the following: one is, we started seeing large organizations using Docker Desktop at scale.When I say ‘at scale,' not one or two or ten developers; like, hundreds and thousands of developers. And they were clamoring for capabilities to help them manage those developer environments at scale. Second is, we saw them getting a lot of benefit in terms of productivity, and choice, and security from using Docker Desktop, and so we stood back and said, “Look, for us to scale our business, we're at 10-plus million monthly active developers today. We know there's 45 million developers coming in this decade; how do we keep scaling while giving a free experience, but still making sure we can fund our engineers and deliver features and additional value?” We looked at other projects, Corey.The first thing we did is we looked outside our four walls, said, “How have other projects with free and open-source components navigated these waters?” And so the thresholds that you just mentioned, the 250 employees and the ten million revenue, were actually thresholds that we saw others put in place to draw lines between what is available completely for free and what is available for those users that now need to purchase subscription if they're using it to create value for their organizations. And we're very explicit about that. You could be using Docker for training, you could be using Docker for eval in those large organizations; we're not going to chase you or be looking to you to step up to a subscription. However, if you're using Docker Desktop in those environments, to build applications that run your business or that are creating value for your customers, then purchasing a subscription is a way for us to continue to invest in a product that the ecosystem clearly loves and is getting a lot of value out of. And so, that was again, the premise of this change. So, now to the root of your question is, so what's the reaction? We're very, very pleased. First off, yes, there were some angry voices out there.Corey: Yeah. And I want to be clear, I'm not trivializing people who feel upset.Scott: No.Corey: When you're suddenly using a thing that is free and discovering that, well, now you have to pay money for it, people are not generally going to be happy about that.Scott: No.Corey: When people are viewed themselves as part of the community, of contributing to what they saw as a technical revolution or a scrappy underdog and suddenly they find themselves not being included in some way, shape or form, it's natural to be upset, I don't want to trivialize—Scott: Not at all.Corey: People's warm feelings toward Docker. It was a big part of a lot of folks' personality, for better or worse, [laugh] for a few years in there. But the company needs to be sustainable, so what I'm really interested in is what has that reaction been from folks who are, for better or worse, “Yes, yes, we love Docker, but I don't get to sign $100,000 deals because I just really like the company I'm paying the money to. There has to be business value attached to that.”Scott: That's right. That's right. And to your point, we're not trivializing either the reaction by the community, it was encouraging to see many community members got right away what we're doing, they saw that still, a majority of them can continue using Docker for free under the Docker Personal subscription, and that was also intentional. And you saw on the internet and on Twitter and other social media, you saw them come and support the company's moves. And despite some angry voices in there, there was overwhelmingly positive.So, to your question, though, since August 31, we've been overwhelmed, actually, by the positive response from businesses that use Docker Desktop to build applications and run their businesses. And when I say overwhelmed, we were tracking—because Docker Desktop has a phone-home capability—we had a rough idea of what the baseline usage of Docker Desktops were out there. Well, it turns out, in some cases, there are ten times as many Docker Desktops inside organizations. And the average seems to be settling in around three times to four times as many. And we are already closing business, Corey.In 12 business days, we have companies come through, say, “Yes, our developers use this product. Yes, it's a valuable product. We're happy to talk to a salesperson and give you over to procurement, and here we go.” So, you and I both been around long enough to know, like 12 working days to have a signed agreement with an enterprise agreement is unheard of.Corey: Yeah, but let's be very clear here, on The Duckbill Group's side of things where I do consulting projects, I sell projects to companies that are, “Great, this project will take, I don't know, four to six weeks, whatever it happens to be, and, yeah, you're going to turn a profit on this project in about the first four hours of the engagement.” It is basically push button and you will receive more money in your budget than you had when you started, and that is probably the easiest possible enterprise sale, and it still takes 60 to 90 days most of the time to close deals.Scott: That's right.Corey: Trying to get a procurement deal for software through enterprise procurement processes is one of those things when people say, “Okay, we're going to have a signature in Q3,” you have to clarify what year they're talking about. So, 12 days is unheard of.Scott: [laugh]. Yep. So, we've been very encouraged by that. And I'll just give you a rough numbers: the overall response is ten times our baseline expectations, which is why—maybe unanticipated question, or you going to ask it soon—we came back within two weeks—because we could see this curve hit right away on the 31st of August—we came back and said, “Great.” Now, that we have the confidence that the community and businesses are willing to support us and invest in our sustainability, invest in the sustainable, scalable Docker, we came and we accelerated—pulled forward—items in our roadmap for developers using Docker Desktop, both for Docker Personal, for free in the community, as well as the subscribers.So, things like Docker Desktop for Linux, right? Docker Desktop for Mac, Docker Desktop for Windows has been out there about five years, as I said. We have heard Docker Desktop for Linux rise in demand over those years because if you're managing a large number of developers, you want a consistent environment across all the developers, whether they're using Linux, Mac, or Windows desktops. So, Docker Desktop for Linux will give them that consistency across their entire development environment. That was the number two most requested feature on our public roadmap in the last year, and again, with the positive response, we're now able to confidently invest in that. We're hiring more engineers than planned, we're pulling that forward in the roadmap to show that yes, we are about growing and growing sustainably, and now that the environment and businesses are supporting us, we're happy to double down and create more value.Corey: My big fear when the change was announced was the uncertainty inherent to it. Because if there's one thing that big companies don't like, it's uncertainty because uncertainty equates to risk in their mind. And a lot of other software out there—and yes, Oracle Databases I am looking at you—have a historical track record of, “Okay, great. We have audit rights to inspect your environment, and then when we wind up coming in, we always find that there have been licensing shortfalls,” because people don't know how far things spread internally, as well as, honestly, it's accounting for this stuff in large, complex organizations is a difficult thing. And then there are massive fines at stake, and then there's this whole debate back and forth.Companies view contracts as if every company behaves like that when it comes down to per-seat licensing and the rest. My fear was that that risk avoidance in large companies would have potentially made installing Docker Desktop in their environment suddenly a non-starter across the board, almost to the point of being something that you would discipline employees for, which is not great. And it seems from your response, that has not been a widespread reaction. Yes of course, there's always going to be some weird company somewhere that does draconian things that we don't see, but the fact that you're not sitting here, telling me that you've been taking a beating from this from your enterprise buyers, tells me you're onto something.Scott: I think that's right, Corey. And as you might expect, the folks that don't reach out are silent, and so we don't see folks who don't reach out to us. But because so many have reached out to us so positively, and basically quickly gone right to a conversation with procurement versus any sort of back-and-forth or questions and such, tells us we are on the right track. The other thing, just to be really clear is, we did work on this before the August 31 announcement as well—this being how do we approach licensing and compliance and such—and we found that 80% of organizations, 80% of businesses want to be in compliance, they have a—not just want to be in compliance, but they have a history of being in compliance, regardless of the enforcement mechanism and whatnot. And so that gave us confidence to say, “Hey, we're going to trust our users. We're going to say, ‘grace period ends on January 31.'”But we're not shutting down functionality, we're not sending in legal [laugh] activity, we're not putting any sort of strictures on the product functionality because we have found most people love the product, love what it does for them, and want to see the company continue to innovate and deliver great features. And so okay, you might say, “Well, doesn't that 20% represent opportunity?” Yeah. You know, it does, but it's a big ecosystem. The 80% is giving us a great boost and we're already starting to plow that into new investment. And let's just start there; let's start there and grow from there.This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of "Hello, World" demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking databases, observability, management, and security.And - let me be clear here - it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself all while gaining the networking load, balancing and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build.With Always Free you can do things like run small scale applications, or do proof of concept testing without spending a dime. You know that I always like to put asterisks next to the word free. This is actually free. No asterisk. Start now. Visit https://snark.cloud/oci-free that's https://snark.cloud/oci-free.Corey: I also have a hard time imagining that you and your leadership team would be short-sighted enough to say, “Okay, that”—even 20% of companies that are willing to act dishonestly around stuff like that seems awfully high to me, but assuming it's accurate, would tracking down that missing 20% be worth setting fire to the tremendous amount of goodwill that Docker still very much enjoys? I have a hard time picturing any analysis where that's even a question other than something you set up to make fun of.Scott: [laugh]. No, that's exactly right Corey, it wouldn't be worth it which is why again, we came out of the gate with like, we're going to trust our users. They love the community, they love the product, they want to support us—most of them want to support us—and, you know, when you have most, you're never going to get a hundred percent. So, we got most and we're off to a good start, by all accounts. And look, a lot of folks too sometimes will be right in that gray middle where you let them know that they're getting away with something they're like, “All right, you caught me.”We've seen that behavior before. And so, we can see all this activity out there and we can see if folks have a license or compliance or not, and sometimes just a little tap on the shoulder said, “Hey, did you know that you might be paying for that?” We've seen most folks at the time say, “Ah, okay. You caught me. Happy to talk to procurement.”So, this does not have to be heavy-handed as you said, it does not have to put at risk the goodwill of the 80%. And we don't have to get a hundred percent to have a great successful business and continuing successful community.Corey: Yeah. I'll also point out that, by my reading of your terms and conditions and how you've specified this—I mean, this is not something I've asked you about, so this could turn into a really awkward conversation but I'm going to roll with it anyway, it explicitly states that it is and will remain free for personal development.Scott: That is correct.Corey: When you're looking at employees who work at giant companies and have sloppy ‘bring your own device' controls around these things, all right, they have it installed on their work machine because in their spare time, they're building an app somewhere, they're not going to get a nasty gram, and they're not exposing their company to liability by doing that?Scott: That is exactly correct. And moreover, just keep looking at those use cases, if the company is using it for internal training or if the company is using it to evaluate someone else's technology, someone else's software, all those cases are outside the pay-for subscription. And so we believe it's quite generous in allowing of trials and tests and use cases that make it accessible and easy to try, easy to use, and it's just in the case where if you're a large organization and your developers are using it to build applications for your business and for your customers, thus you're getting a lot of value using the product, we're asking you to share that value with us so we can continue to invest in the product.Corey: And I think that's a reasonable expectation. The challenge that Docker seems to have had for a while has been that the interesting breakthrough, revelatory stuff that you folks did was all open-source. It was a technology that was incredibly inspired in a bunch of different ways. I am, I guess, mature enough to admit that my take that, “Oh, Docker is terrible”—which was never actually my take—was a little short-sighted. I'm very good at getting things wrong across the board, and that is no exception.I also said virtualization was a flash in the pan and look how that worked out. I was very anti-cloud, et cetera, et cetera. Times change, people change, and doubling down on being wrong gains you nothing. But the question that was always afterwards what is the monetization strategy? Because it's not something you can give away for free and make it up in volume?Even VC money doesn't quite work like that forever, so there's a—the question is, what is the monetization strategy that doesn't leave people either resenting you because, “Remember that thing that used to be free isn't anymore? Doesn't it suck to be you?” And is still accessible as broadly as you are, given the sheer breadth and diversity of your community? Like I can make bones about the fact that ten million in revenue and 250 employees are either worlds apart, or the wrong numbers, or whatever it is, but it's not going to be some student somewhere sitting someplace where their ramen budget is at risk because they have to spend $5 a month or whatever it is to have this thing. It doesn't apply to them.And this feels like, unorthodox though it certainly is, it's not something to be upset about in any meaningful sense. The people that I think would actually be upset and have standing to be upset about this are the enterprise buyers, and you're hearing from them in what is certainly—because I will hear it if not—that this is something they're happy about. They are thrilled to work with you going forward. And I think it makes sense. Even when I was doing stuff as an independent consultant, before I formalized the creation of The Duckbill Group and started hiring people, my policy was always to not use the free tier of things, even if I fit into them because I would much rather personally be a paying customer, which elevates the, I guess, how well my complaints are received.Because I'm a free user, I'm just another voice on Twitter; albeit a loud one and incredibly sarcastic one at times. But if I'm a paying customer, suddenly the entire tenor of that conversation changes, and I think there's value to that. I've always had the philosophy of you pay for the things you use to make money. And that—again, that is something that's easy for me to say now. Back when I was in crippling debt in my 20s, I assure you, it was not, but I still made the effort for things that I use to make a living.Scott: Yeah.Corey: And I think that philosophy is directionally correct.Scott: No, I appreciate that. There's a lot of good threads in there. Maybe just going way back, Docker stands on the shoulders of giants. There was a lot of work with container tech in the Linux kernel, and you and I were talking before about it goes back to LPAR on IBMs, and you know, BS—Berkeley's—Corey: BSD jails and chroots on Linux. Yeah.Scott: Chroot, right? I mean, Bill Joy, putting chroot in—Corey: And Tupperware parties, I'm sure. Yeah.Scott: Right. And all credit to Solomon Hykes, Docker's founder, who took a lot of good up and coming tech—largely on the ops side and in Linux kernel—took the primitives from Git and combined that with immutable copy-on-write file system and put those three together into a really magical combination that simplified all this complexity of dependency management and portability of images across different systems. And so in some sense, that was the magic of standing on these giant shoulders but seeing how these three different waves of innovation or three different flows of innovation could come together to a great user experience. So, also then moving forward, I wouldn't say they're happy, just to make sure you don't get inbound, angry emails—the enterprise buyers—but they do recognize the value of the product, they think the economics are fair and straight ahead, and to your point about having a commercial relationship versus free or non-existing relationship, they're seeing that, “Oh, okay, now I have insight into the roadmap. Now, I can prioritize my requirements that my devs have been asking for. Now, I can double-down on the secure supply chain issues, which I've been trying to get in front of for years.”So, it gives them an avenue that now, much different than a free user as you observed, it's a commercial relationship where it's two way street versus, “Okay, we're just going to use this free stuff and we don't have much of a say because it's free, and so on and so forth.” So, I think it's been an eye-opener for both the company but also for the businesses. There is a lot of value in a commercial relationship beyond just okay, we're going to invest in new features and new value for developers.Corey: The challenge has always been how do you turn something that is widely beloved, that is effectively an open-source company, into money? There have been a whole bunch of questions about this, and it seems that the consensus that has emerged is that a number of people for a long time mistook open-source for a business model instead of a strategy, and it's very much not. And a lot of companies are attempting to rectify that with weird license changes where, “Oh, you're not allowed to take our code and build a service out of it if you're a cloud provider.” Amazon's product strategy is, of course, “Yes,” so of course, there's always going to be something coming out of AWS that is poorly documented, has a ridiculous name, and purports to do the same thing for way less money, except magically you pay them by the hour. I digress.Scott: No, it's a great surface area, and you're right I completely didn't answer that question. [laugh]. So—Corey: No, it's fair. It's—Scott: Glad you brought it back up.Corey: —a hard problem. It's easy to sit here and say, “Well, what I think they should do”—but all of those solutions fall apart under ten seconds of scrutiny.Scott: Super, super hard problem which, to be fair, we as a team and a community wrestled with for years. But here's where we landed, Corey. The short version is that you can still have lots of great upstream open-source technologies, and you'll have an early adopter community that loves those, use those, gets a lot of progress running fast and far with those, but we've found that the vast majority of the market doesn't want to spend its time cobbling together bits and bytes of open-source tech, and maintaining it, and patching it, and, and, and. And so what we're offering is an engineered product that takes the upstream but then adds a lot of value—we would say—to make it an engineered, easy to use, easy to configure, upgraded, secure, so on and so forth. And the convenience of that versus having to cobble together your own environment from upstream has proved to be what folks are willing to pay for. So, it's the classic kind of paying for time and convenience versus not.And so that is one dimension. And the other dimension, which you already referenced a little bit with AWS is that we have SaaS; we have a SaaS product in Docker Hub, which is providing a hosted registry with quality content that users know is updated not less than every 30 days, that is patched and maintained by us. And so those are examples of, in some sense, consumption [unintelligible 00:27:53]. So, we're using open-source to build this SaaS service, but the service that users receive, they're willing to pay for because they're not having to patch the Mongo upstream, they're not having to roll the image themselves, they're not having to watch the CVEs and scramble when everything comes out. When there's a CVE out in our upstream, our official images are patched no less than 24 hours later and typically within hours.That's an example of a service, but all based on upstream open-source tech that for the vast majority of uses are free. If you're consuming a lot of that, then there's a subscription that kicks in there as well. But we're giving you value in exchange for you having to spend your time, your engineers, managing all that that I just walked through. So, those are the two avenues that we found that are working well, that seem to be a fair trade and fair balance with the community and the rest of the ecosystem.Corey: I think the hardest part for a lot of folks is embracing change. And I have encountered this my entire career where I started off doing large-scale email systems administration, and hey, turns out that's not really a thing anymore. And I used to be deep in the bowels of Postfix, for example. I'm referenced in the SVN history of Postfix, once upon a time, just for helping with documentation and finding weird corner cases because I'm really good at breaking things by accident. And I viewed it as part of my identity.And times have changed and moved on; I don't run Postfix myself for anything anymore. I haven't touched it in years. Docker is still there and it's still something that people are actively using basically everywhere. And there's a sense of ownership and identity for especially early adopters who glom on to it because it is such a better way of doing some things that it is almost incomprehensible that we used to do it any other way. That's transformation.That's something awesome. But people want to pretend that we're still living in that era where technology has not advanced. The miraculous breakthrough in 2013 is today's de rigueur type of environment where this is just, “Oh, yeah. Of course you're using Docker.” If you're not, people look at you somewhat strangely.It's like, “Oh, I'm using serverless.” “Okay, but you can still build that in Docker containers. Why aren't you doing that?” It's like, “Oh, I don't believe in running anything that doesn't make me pay AWS by the second.” So okay, great. People are going to have opinions on this stuff. But time marches on and whatever we wish the industry would do, it's going to make its own decisions and march forward. There's very little any of us can do to change that.Scott: That's right. Look, it was a single container back in 2013, 2014, right? And now what we're seeing—and you kind of went there—is we're separating the implementation of service from the service. So, the service could be implemented with a container, could be a serverless function, could be a hosted XYZ as a service on some cloud, but what developers want to do is—what they're moving towards is, assemble your application based on services regardless of the how. You know, is that how a local container? To your point, you can roll a local serverless function now in an OCI image, and push it to Amazon.Corey: Oh, yeah. It's one of that now 34 ways I found to run containers on AWS.Scott: [laugh]. You can also, in Compose, abstract all that complexity away. Compose could have three services in it. One of those services is a local container, one of those services might be a local serverless function that you're running to test, and one of those services could be a mock to a Database as a Service on a cloud. And so that's where we are.We've gone beyond the single-container Docker run, which is still incredibly powerful but now we're starting to uplevel to applications that consist of multiple services. And where do those services run? Increasingly, developers do not need to care. And we see that as our mission is continue to give that type of power to developers to abstract out the how, extract out the infrastructure so they can just focus on building their app.Corey: Scott, I want to thank you so much for taking the time to speak with me. If people want to learn more—and that could mean finding out your opinions on things, potentially yelling at you about pricing changes, more interestingly, buying licenses for their large companies to run this stuff, and even theoretically, since you alluded to it a few minutes ago, look into working at Docker—where can they find you?Scott: No, thanks, Corey. And thank you for the time to discuss and look back over both years, but also zoom in on the present day. So, www.docker.com; you can find any and all what we just walked through. They're more than happy to yell at me on Twitters at @scottcjohnston, and we have a public roadmap that is in GitHub. I'm not going to put the URL here, but you can find it very easily. So, we love hearing from our community, we love engaging with them, we love going back and forth. And it's a big community; jump in, the waters warm, very welcoming, love to have you.Corey: And we'll of course, but links to that in the [show notes. 00:32:28] Thank you so much for your time. I really do appreciate it.Scott: Thank you, Corey. Right back at you.Corey: Scott Johnston, CEO of Docker. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with a comment telling me that Docker isn't interested in at all because here's how to do exactly what Docker does in LPARs on your mainframe until the AWS/400 comes to [unintelligible 00:33:02].Scott: [laugh].Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Yesterday Ethan and I recorded a fascinating episode of Day Two Cloud with Ben Weissman. He brought the knowledge about Azure Arc and its Data Services. That led to me thinking deeply about the plans Microsoft laid back in 2017 with SQL on Linux and investments in Kubernetes, and how its paying off now in Azure Arc. Don't be surprised if there's some big stuff coming down the pike for Arc, especially with Microsoft Ignite around the corner. ----------------------------------------------------------------------------------------------------- Website: https://nedinthecloud.com Pluralsight: https://app.pluralsight.com/profile/author/edward-bellavance GitHub: https://github.com/ned1313

Your usual Chris duo becomes a trio, as they welcome Friend of Redgate and autism spectrum self-advocate, Chris Voss. Only our 2nd ever guest Chris on the show, he joins us to demystify the myths of neurodiversity, its meaning, influence, and prevalence within the IT sector. Things get a little hazy as our Chris trinity start on the beer, but they keep IT relevant talking cloud, widgets, and acronyms. So, grab yourself a beer and join the Chris trifecta – cheers.

Links: Entirely optional for attackers: https://osamaelnaggar.com/blog/aws_waf_dangerous_defaults/ Worst Case: https://www.tbray.org/ongoing/When/202x/2021/10/08/The-WOrst-Case Are looking to change that: https://www.theregister.com/2021/10/11/cyan_zero_day_legislative_project/ Introducing Security at the Edge: https://aws.amazon.com/blogs/security/introducing-the-security-at-the-edge-core-principles-whitepaper/ Password reuse: https://www.hypr.com/password-reuse/ TranscriptCorey: This is the AWS Morning Brief: Security Edition. AWS is fond of saying security is job zero. That means it's nobody in particular's job, which means it falls to the rest of us. Just the news you need to know, none of the fluff.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate. Is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter. Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other; which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at honeycomb.io/screaminginthecloud observability; it's more than just hipster monitoring.Corey: I must confess, I didn't expect to see an unpatched AWS vulnerability being fodder for this podcast so early in the security lifespan here, but okay. Yes, yes, before I get letters, it's not a vulnerability as AWS would define it, but it's a pretty crappy default that charges customers money while giving them a false sense of security.Past that, it's going to be a short podcast this week, and that's just fine by me because the point of it is, “The things you should know as someone who has to care about security.” On slow news weeks like last week that means I'm not here to give you pointless filler. Onward.Now, AWS WAF is expensive and apparently, as configured by default, entirely optional for attackers. Only the first 8KB of a request are inspected by default. That means that any malicious payload that starts after the 8KB limit in a POST request will completely bypass AWS WAF unless you've explicitly added a rule to block any POST request greater than 8KB in size, which you almost assuredly have not done. Even their managed rule that addresses size limits only kicks in at 10KB. This is—as the kids say—less than ideal.I had a tweet recently that talked about the horror of us-east-1 being globally unavailable for ages. Tim Bray took this and ran with the horrifying concept in a post he called, “Worst Case.” It's really worth considering things like this when it comes to disaster and continuity planning. How resilient are our apps and infrastructure really when all is said and done? What dependencies do we take on third parties who in turn rely on the same infrastructure that we're trying to guard against failure from?An unfortunate reality is that many cybersecurity researchers don't have much in the way of legal protections; some folks are looking to change that through legislation. Here's some good advice: if a security researcher reports a vulnerability to you or your company in good faith, perhaps not acting like a raging jackhole is an option that's on the table. Bug bounties are hilariously small; they could make many times as much money by selling vulnerabilities to the highest bidder. Instead they're reporting bugs to you in good faith. Word spreads. If you're a hassle to deal with, other researchers won't report things to you in the future. “Be a nice person,” is surprisingly undervalued when it comes to keeping yourself and your company out of trouble.Now, only one interesting thing came out of the mouth of AWS horse last week in a security context, and it's a Core Principles whitepaper: “Introducing Security at the Edge.” Setting aside entirely the fact that neither contributor to this has the job title of “EdgeLord,” I like it. Rather than focusing on specific services—although of course there's some of that because vendors are going to vendor—it emphasizes how to think about the various considerations of edge locations that aren't deep within hardened data centers. “How should I think about this problem,” is the kind of question that really deserves to be asked a lot more than it is.and lastly, let's end up with a tip of the week. If you have a multi-cloud anything, ensure that credentials are not shared between two cloud providers. I'm talking about passwords, keys, et cetera. This is a step beyond the standard password reuse warning of not using the same password for multiple accounts. Think it through; if one of your providers happens to be Azure, and they Azure up the security yet again, you really don't want that to grant an attacker or other random Azure customers access to your AWS account as well, do you? I thought not.Corey: This episode is sponsored in part by Liquibase. If you're anything like me, you've screwed up the database part of a deployment so severely that you've been banned from ever touching anything that remotely sounds like SQL at least three different companies. We've mostly got code deployment solved for, but when it comes to databases, we basically rely on desperate hope, with a rollback plan of keeping our resumes up to date. It doesn't have to be that way. Meet Liquibase. It's both an open-source project and a commercial offering. Liquibase lets you track, modify, and automate database schema changes across almost any database, with guardrails that ensure you'll still have a company left after you deploy the change. No matter where your database lives, Liquibase can help you solve your database deployment issues. Check them out today at liquibase.com. Offer does not apply to Route 53.Corey: And that is what happened last week in AWS security. I have been your host, Corey Quinn, and if you remember nothing else, it's that when you don't get what you want, you get experience instead. Let my experience guide you with the things you need to know in the AWS security world, so you can get back to doing your actual job. Thank you for listening to the AWS Morning Brief: Security Edition with the latest in AWS security that actually matters. Please follow AWS Morning Brief on Apple Podcast, Spotify, Overcast—or wherever the hell it is you find the dulcet tones of my voice—and be sure to sign up for the Last Week in AWS newsletter at lastweekinaws.com.Announcer: This has been a HumblePod production. Stay humble.

About NickNick Heudecker leads market strategy and competitive intelligence at Cribl, the observability pipeline company. Prior to Cribl, Nick spent eight years as an industry analyst at Gartner, covering data and analytics. Before that, he led engineering and product teams at multiple startups, with a bias towards open source software and adoption, and served as a cryptologist in the US Navy. Join Corey and Nick as they discuss the differences between observability and monitoring, why organizations struggle to get value from observability data, why observability requires new data management approaches, how observability pipelines are creating opportunities for SRE and SecOps teams, the balance between budgets and insight, why goats are the world's best mammal, and more.Links: Cribl: https://cribl.io/ Cribl Community: https://cribl.io/community Twitter: https://twitter.com/nheudecker Try Cribl hosted solution: https://cribl.cloud TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Thinkst. This is going to take a minute to explain, so bear with me. I linked against an early version of their tool, canarytokens.org in the very early days of my newsletter, and what it does is relatively simple and straightforward. It winds up embedding credentials, files, that sort of thing in various parts of your environment, wherever you want to; it gives you fake AWS API credentials, for example. And the only thing that these things do is alert you whenever someone attempts to use those things. It's an awesome approach. I've used something similar for years. Check them out. But wait, there's more. They also have an enterprise option that you should be very much aware of canary.tools. You can take a look at this, but what it does is it provides an enterprise approach to drive these things throughout your entire environment. You can get a physical device that hangs out on your network and impersonates whatever you want to. When it gets Nmap scanned, or someone attempts to log into it, or access files on it, you get instant alerts. It's awesome. If you don't do something like this, you're likely to find out that you've gotten breached, the hard way. Take a look at this. It's one of those few things that I look at and say, “Wow, that is an amazing idea. I love it.” That's canarytokens.org and canary.tools. The first one is free. The second one is enterprise-y. Take a look. I'm a big fan of this. More from them in the coming weeks.Corey: This episode is sponsored in part by our friends at Jellyfish. So, you're sitting in front of your office chair, bleary eyed, parked in front of a powerpoint and—oh my sweet feathery Jesus its the night before the board meeting, because of course it is! As you slot that crappy screenshot of traffic light colored excel tables into your deck, or sift through endless spreadsheets looking for just the right data set, have you ever wondered, why is it that sales and marketing get all this shiny, awesome analytics and inside tools? Whereas, engineering basically gets left with the dregs. Well, the founders of Jellyfish certainly did. That's why they created the Jellyfish Engineering Management Platform, but don't you dare call it JEMP! Designed to make it simple to analyze your engineering organization, Jellyfish ingests signals from your tech stack. Including JIRA, Git, and collaborative tools. Yes, depressing to think of those things as your tech stack but this is 2021. They use that to create a model that accurately reflects just how the breakdown of engineering work aligns with your wider business objectives. In other words, it translates from code into spreadsheet. When you have to explain what you're doing from an engineering perspective to people whose primary IDE is Microsoft Powerpoint, consider Jellyfish. Thats Jellyfish.co and tell them Corey sent you! Watch for the wince, thats my favorite part.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. This promoted episode is a bit fun because I'm joined by someone that I have a fair bit in common with. Sure, I moonlight sometimes as an analyst because I don't really seem to know what that means, and he spent significant amounts of time as a VP analyst at Gartner. But more importantly than that, a lot of the reason that I am the way that I am is that I spent almost a decade growing up in Maine, and in Maine, there's not a lot to do other than sit inside for the nine months of winter every year and develop personality problems.You've already seen what that looks like with me. Please welcome Nick Heudecker, who presumably will disprove that, but maybe not. He is currently a senior director of market strategy and competitive intelligence at Cribl. Nick, thanks for joining me.Nick: Thanks for having me. Excited to be here.Corey: So, let's start at the very beginning. I like playing with people's titles, and you certainly have a lofty one. ‘competitive intelligence' feels an awful lot like jeopardy. What am I missing?Nick: Well, I'm basically an internal analyst at the company. So, I spend a lot of time looking at the broader market, seeing what trends are happening out there; looking at what kind of thought leadership content that I can create to help people discover Cribl, get interested in the products and services that we offer. So, I'm mostly—you mentioned my time in Maine. I was a cryptologist in the Navy and I spent almost all of my time focused on what the bad guys do. And in this job, I focus on what our potential competitors do in the market. So, I'm very externally focused. Does that help? Does that explain it?Corey: No, it absolutely does. I mean, you folks have been sponsoring our nonsense for which we thank you, but the biggest problem that I have with telling the story of Cribl was that originally—initially it was, from my perspective, “What is this hokey nonsense?” And then I learned and got an answer and then finish the sentence with, “And where can I buy it?” Because it seems that the big competitive threat that you have is something crappy that some rando sysadmin has cobbled together. And I say that as the rando sysadmin, who has cobbled a lot of things like that together. And it's awful. I wasn't aware you folks had direct competitors.Nick: Today we don't. There's a couple that it might be emerging a little bit, but in general, no, it's mostly us, and that's what I analyze every day. Are there other emerging companies in the space? Are there open-source projects? But you're right, most of the things that we compete against are DIY today. Absolutely.Corey: In your previous role, which you were at for a very long time in tech terms—which in a lot of other cases is, “Okay, that doesn't seem that long,” but seven and a half years is a respectable stint at a company. And you were at Gartner doing a number of analyst-like activities. Let's start at the beginning because I assure you, I'm asking this purely for the audience and not because I don't know the answer myself, but what exactly is the purpose of an analyst firm, of which Gartner is the most broadly known and, follow up, why do companies care what Gartner thinks?Nick: Yeah. It's a good question, one that I answer a lot. So, what is the purpose of an analyst firm? The purpose of an analyst firm is to get impartial information about something, whether that is supply chain technology, big data tech, human resource management technologies. And it's often difficult if you're an end-user and you're interested in say, acquiring a new piece of technology, what really works well, what doesn't.And so the analyst firm because in the course of a given year, I would talk to nearly a thousand companies and both end-users and vendors as well as investors about what they're doing, what challenges they're having, and I would distill that down into 30-minute conversations with everyone else. And so we provided impartial information in aggregate to people who just wanted to help. And that's the purpose of an analyst firm. Your second question, why do people care? Well, I didn't get paid by vendors.I got paid by the company that I worked for, and so I got to be Tron; I fought for the users. And because I talk to so many different companies in different geographies, in different industries, and I share that information with my colleagues, they shared with me, we had a very robust understanding of what's actually happening in any technology market. And that's uncommon kind of insight to really have in any kind of industry. So, that's the purpose and that's why people care.Corey: It's easy from the engineering perspective that I used to inhabit to make fun of it. It's oh, it's purely justification when you're making a big decision, so if it goes sideways—because find me a technology project that doesn't eventually go sideways—I want to be able to make sure that I'm not the one that catches heat for it because Gartner said it was good. They have an amazing credibility story going on there, and I used to have that very dismissive perspective. But the more I started talking to folks who are Gartner customers themselves and some of the analyst-style things that I do with a variety of different companies, it's turned into, “No, no. They're after insight.”Because it turns out, from my perspective at least, the more that you are focused on building a product that solves a problem, you sort of lose touch with the broader market because the only people you're really talking to are either in your space or have already acknowledged and been right there and become your customer and have been jaded to see things from your point of view. Getting a more objective viewpoint from an impartial third party does have value.Nick: Absolutely. And I want you to succeed, I want you to be successful, I want to carry on a relationship with all the clients that I would speak with, and so one of the fun things I would always ask is, “Why are you asking me this question now?” Sometimes it would come in, they'd be very innocuous;, “Compare these databases,” or, “Compare these cloud services.” “Well, why are you asking?” And that's when you get to, kind of like, the psychology of it.“Oh, we just hired a new CIO and he or she hates vendor X, so we have to get rid of it.” “Well, all right. Let's figure out how we solve this problem for you.” And so it wasn't always just technology comparisons. Technology is easy, you write a check and you hope for the best.But when you're dealing with large teams and maybe a globally distributed company, it really comes down to culture, and personality, and all the harder factors. And so it was always—those were always the most fun and certainly the most challenging conversations to have.Corey: One challenge that I find in this space is—in my narrow niche of the world where I focus on AWS bills, where things are extraordinarily yes or no, black or white, binary choices—that I talked to companies, like during the pandemic, and they were super happy that, “Oh, yeah. Our infrastructure has auto-scaling and it works super well.” And I look at the bill and the spend graph over time is so flat you could basically play a game of pool on top of it. And I don't believe that I'm talking to people who are lying to me. I truly don't believe that people make that decision, but what they believe versus what is evidenced in reality are not necessarily congruent. How do you disambiguate from the stories that people want to tell about themselves? And what they're actually doing?Nick: You have to unpack it. I think you have to ask a series of questions to figure out what their motivation is. Who else is on the call, as well? I would sometimes drop into a phone call and there would be a dozen people on the line. Those inquiry calls would go the worst because everyone wants to stake a claim, everyone wants to be heard, no one's going to be honest with you or with anyone else on the call.So, you typically need to have a pretty personal conversation about what does this person want to accomplish, what does the company want to accomplish, and what are the factors that are pushing against what those things are? It's like a novel, right? You have a character, the character wants to achieve something, and there are multiple obstacles in that person's way. And so by act five, ideally everything wraps up and it's perfect. And so my job is to get the character out of the tree that is on fire and onto the beach where the person can relax.So, you have to unpack a lot of different questions and answers to figure out, well, are they telling me what their boss wants to hear or are they really looking for help? Sometimes you're successful, sometimes you're not. Not everyone does want to be open and honest. In other cases, you would have a team show up to a call with maybe a junior engineer and they really just want you to tell them that the junior engineer's architecture is not a good idea. And so you do a lot of couples therapy as well. I don't know if this is really answering the question for you, but there are no easy answers. And people are defensive, they have biases, companies overall are risk-averse. I think you know this.Corey: Oh, yeah.Nick: And so it can be difficult to get to the bottom of what their real motivation is.Corey: My approach has always been that if you want serious data, you go talk to Gartner. If you want [anec-data 00:09:48] and some understanding, well, maybe we can have that conversation, but they're empowering different decisions at different levels, and that's fine. To be clear, I do not consider Gartner to be a competitor to what I do in any respect. It turns out that I am not very good at drawing charts in varying shades of blue and positioning things just so with repeatable methodology, and they're not particularly good at having cartoon animals as their mascot that they put into ridiculous situations. We each have our portion of the universe, and that's working out reasonably well.Nick: Well, and there's also something to unpack there as well because I would say that people look at Gartner and they think they have a lot of data. To a certain degree they do, but a lot of it is not quantifiable data. If you look at a firm like IDC, they specialize in—like, they are a data house; that is what they do. And so their view of the world and how they advise their clients is different. So, even within analyst firms, there is differentiation in what approach they take, how consultative they might be with their clients, one versus another. So, there certainly are differences that you could find the more exposure you get into the industry.Corey: For a while, I've been making a recurring joke that Route 53—Amazon's managed DNS service—is in fact a database. And then at some point, I saw a post on Reddit where someone said, “Yeah, I see the joke and it's great, but why should I actually not do this?” At which point I had to jump in and say, “Okay, look. Jokes are all well and good, but as soon as people start taking me seriously, it's very much time to come clean.” Because I think that's the only ethical and responsible thing to do in this ecosystem.Similarly, there was another great joke once upon a time. It was an April Fool's Day prank, and Google put out a paper about this thing they called MapReduce. Hilarious prank that Yahoo fell for hook, line, and sinker, and wound up building Hadoop out of it and we're still paying the price for that, years later. You have a bit of a reputation from your time at Gartner as being—and I quote—“The man who killed Hadoop.” What happened there? What's the story? And I appreciate your finally making clear to the rest of us that it was, in fact, a joke. What happened there?Nick: Well, one of the pieces of research that Gartner puts out every year is this thing called a Hype Cycle. And we've all seen it, it looks like a roller coaster in profile; big mountain goes up really high and then comes down steeply, drops into a valley, and then—Corey: ‘the trough of disillusionment,' as I recall.Nick: Yes, my favorite. And then plateaus out. And one of the profiles on that curve was Hadoop distributions. And after years of taking inquiry calls, and writing documents, and speaking with everybody about what they were doing, we realized that this really isn't taking off like everyone thinks it is. Cluster sizes weren't getting bigger, people were having a lot of challenges with the complexity, people couldn't find skills to run it themselves if they wanted to.And then the cloud providers came in and said, “Well, we'll make a lot of this really simple for you, and we'll get rid of HDFS,” which is—was a good idea, but it didn't really scale well. I think that the challenge of having to acquire computers with compute storage and memory again, and again, and again, and again, just was not sustainable for the majority of enterprises. And so we flagged it as this will be obsolete before plateau. And at that point, we got a lot of hate mail, but it just seemed like the right decision to make, right? Once again, we're Tron; we fight for the users.And that seemed like the right advice and direction to provide to the end-users. And so didn't make a lot of friends, but I think I was long-term right about what happened in the Hadoop space. Certainly, some fragments of it are left over and we're still seeing—you know, Spark is going strong, there's a lot of Hive still around, but Hadoop as this amalgamation of open-source projects, I think is effectively dead.Corey: I sure hope you're right. I think it has a long tail like most things that are there. Legacy is the condescending engineering term for ‘it makes money.' You were at Gartner for almost eight years and then you left to go work at Cribl. What triggered that? What was it that made you decide, “This is great. I've been here a long time. I've obviously made it work for me. I'm going to go work at a startup that apparently, even though it recently raised a $200 million funding round”—congratulations on that, by the way—“It still apparently can't afford to buy a vowel in its name.” That's C-R-I-B-L because, of course, it is. Maybe another consonant, while you're shopping. But okay, great. It's oddly spelled, it is hard to explain in some cases, to folks who are not already feeling pain in that space. What was it that made you decide to sit up and, “All right, this is where I want to be?”Nick: Well, I met the co-founders when I was an analyst. They were working at Splunk and oddly enough—this is going to be an interesting transition compared to the previous thing we talked about—they were working on Hunk, which was, let's use HDFS to store Splunk data. Made a lot of sense, right? It could be much more cost-effective than high-cost infrastructure for Splunk. And so they told me about this; I was interested.And so I met the co-founders and then I reconnected with them after they left and formed Cribl. And I thought the story was really cool because where they're sitting is between sources and destinations of observability data. And they were solving a problem that all of my customers had, but they couldn't resolve. They would try and build it themselves. They would look at—Kafka was a popular choice, but that had some challenges for observability data—works fantastically well for application data.And they were just—had a very pragmatic view of the world that they were inhabiting and the problem that they were looking to solve. And it looked kind of like a no-brainer of a problem to solve. But when you double-click on it, when you really look down and say, “All right, what are the challenges with doing this?” They're really insurmountable for a lot of organizations. So, even though they may try and take a DIY approach, they often run into trouble after just a few weeks because of all the protocols you have to support, all the different data formats, and all the destinations, and role-based access control, and everything else that goes along with it.And so I really liked the team. I thought the product inhabited a unique space in the market—we've already talked about the lack of competitors in the space—and I just felt like the company was on a rocket ship—or is a rocket ship—that basically had unbounded success potential. And so when the opportunity arose to join the team and do a lot of the things I like doing as an analyst—examining the market, talking to people looking at competitive aspects—I jumped at it.Corey: It's nice when you see those opportunities that show up in front of you, and the stars sort of align. It's like, this is not just something that I'm excited about and enthused about, but hey, they can use me. I can add something to where they're going and help them get there better, faster, sooner, et cetera, et cetera.Nick: When you're an analyst, you look at dozens of companies a month and I'd never seen an opportunity that looked like that. Everything kind of looked the same. There's a bunch of data integration companies, there's a bunch of companies with Spark and things like that, but this company was unique; the product was unique, and no one was really recognizing the opportunity. So, it was just a great set of things that all happen at the same time.Corey: It's always fun to see stars align like that. So—Nick: Yeah.Corey: —help me understand in a way that can be articulated to folks who don't have 15 years of grumpy sysadmin experience under their belts, what does Cribl do?Nick: So, Cribl does a couple of things. Our flagship product is called LogStream, and the easiest way to describe that is as an abstraction between sources and destinations of data. And that doesn't sound very interesting, but if you, from your sysadmin background, you're always dealing with events, logs, now there's traces, metrics are also hanging around—Corey: Oh, and of course, the time is never synchronized with anything either, so it's sort of a giant whodunit, mystery, where half the eyewitnesses lie.Nick: Well, there's that. There's a lot of data silos. If you got an agent deployed on a system, it's only going to talk to one destination platform. And you repeat this, maybe a dozen times per server, and you might have 100,000 or 200,000 servers, with all of these different agents running on it, each one locked into one destination. So, you might want to be able to mix and match that data; you can't. You're locked in.One of the things LogStream does is it lets you do that exact mixing and matching. Another thing that this product does, that LogStream does, is it gives you ability to manage that data. And then what I mean by that is, you may want to reduce how much stuff you're sending into a given platform because maybe that platform charges you by your daily ingest rates or some other kind of event-based charges. And so not all that data is valuable, so why pay to store it if it's not going to be valuable? Just dump it or reduce the amount of volume that you've got in that payload, like a Windows XML log.And so that's another aspect that it allows you to do, better management of that stuff. You can redact sensitive fields, you can enrich the data with maybe, say, GeoIPs so you know what kind of data privacy laws you fall under and so on. And so, the story has always been, land the data in your destination platform first, then do all those things. Well, of course, because that's how they charge you; they charge you based on daily ingest. And so now the story is, make those decisions upfront in one place without having to spread this logic all over, and then send the data where you want it to go.So, that's really, that's the core product today, LogStream. We call ourselves an observability pipeline for observability data. The other thing we've got going on is this project called AppScope, and I think this is pretty cool. AppScope is a black box instrumentation tool that basically resides between the application runtime and the kernel and any shared libraries. And so it provides—without you having to go back and instrument code—it instruments the application for you based on every call that it makes and then can send that data through something like LogStream or to another destination.So, you don't have to go back and say, “Well, I'm going to try and find the source code for this 30-year old c++ application.” I can simply run AppScope against the process, and find out exactly what that application is doing for me, and then relay that information to some other destination.Corey: This episode is sponsored in part by Liquibase. If you're anything like me, you've screwed up the database part of a deployment so severely that you've been banned from touching every anything that remotely sounds like SQL, at at least three different companies. We've mostly got code deployments solved for, but when it comes to databases we basically rely on desperate hope, with a roll back plan of keeping our resumes up to date. It doesn't have to be that way. Meet Liquibase. It is both an open source project and a commercial offering. Liquibase lets you track, modify, and automate database schema changes across almost any database, with guardrails to ensure you'll still have a company left after you deploy the change. No matter where your database lives, Liquibase can help you solve your database deployment issues. Check them out today at liquibase.com. Offer does not apply to Route 53.Corey: I have to ask because I love what you're doing, don't get me wrong. The counterargument that always comes up in this type of conversation is, “Who in their right mind looks at the state of the industry today and says, ‘You know what we need? That's right; another observability tool.'” what differentiates what you folks are building from a lot of the existing names in the space? And to be clear, a lot of the existing names in the space are treating observability simply as hipster monitoring. I'm not entirely sure they're wrong, but that's a different fight for a different time.Nick: Yeah. I'm happy to come back and talk about that aspect of it, too. What's different about what we're doing is we don't care where the data goes. We don't have a dog in that fight. We want you to have better control over where it goes and what kind of shape it's in when it gets there.And so I'll give an example. One of our customers wanted to deploy a new SIEM—Security Information Event Management—tool. But they didn't want to have to deploy a couple hundred-thousand new agents to go along with it. They already had the data coming in from another agent, they just couldn't get the data to it. So, they use LogStream to send that data to their new desired platform.Worked great. They were able to go from zero to a brand new platform in just a couple days, versus fighting with rolling out agents and having to update them. Did they conflict with existing agents? How much performance did it impact on the servers, and so on? So, we don't care about the destination. We like everybody. We're agnostic when it comes to where that data goes. And—Corey: Oh, it's not about the destination. It's about the journey. Everyone's been saying it, but you've turned it into a product.Nick: It's very spiritual. So, we [laugh] send, we send your observability data on a spiritual [laugh] journey to its destination, and we can do quite a bit with it on the way.Corey: So, you said you offered to go back as well and visit the, “Oh, it's monitoring, but we're going to call it observability because otherwise we get yelled out on Twitter by Charity Majors.” How do you view that?Nick: Monitoring is the things you already know. Right? You know what questions you want to ask, you get an alert if something goes out of bounds or something goes from green to red. Think about monitoring as a data warehouse. You shape your data, you get it all in just the right condition so you can ask the same question over and over again, over different time domains.That's how I think about monitoring. It's prepackaged, you know exactly what you want to do with it. Observability is more like a data lake. I have no idea what I'm going to do with this stuff. I think there's going to be some signals in here that I can use, and I'm going to go explore that data.So, if monitoring is your known knowns, observability is your unknown unknowns. So, an ideal observability solution gives you an opportunity to discover what those are. Once you discover them. Great. Now, you can talk about how to get them into your monitoring system. So, for me, it's kind of a process of discovery.Corey: Which makes an awful lot of sense. The problem I've always had with the monitoring approach is it falls into this terrible pattern of enumerate the badness. In other words, “Imagine all the ways that this system can fail,” and then build an alerting that lets you know when any of those things happen. And what happens next is inevitable to anyone who's ever dealt with the tricksy devils known as computers, and what happens, of course, is that they find new ways to fail and you generally get to add to the list of things to check for, usually at two o'clock in the morning.Nick: On a Sunday.Corey: Oh, absolutely. It almost doesn't matter when. The real problem is when these things happen, it's, “What day, actually, is it?” And you have to check the calendar to figure out because your third time that week being woken up in the dead of night. It's like an infant but less than endearing.So, that has been the old school approach, and there's unfortunately still an awful lot of, we'll just call it nonsense, in the industry that still does exactly the same thing, except now they call it observability because—hearkening back to earlier in our conversation—there's a certain point in the Gartner Hype Cycle that we are all existing within. What's the deal with that?Nick: Well, I think that there are a lot of entrenched interests in the monitoring space. And so I think you always see this when a new term comes around. Vendors will say, “All right, well, there's a lot of confusion about this. Let me back-fit my product into this term so that I can continue to look like I'm on the leading edge and I'm not going to put any of my revenues in jeopardy.” I know, that's a cynical view, but I've seen it over and over again.And I think that's unfortunate because there's a real opportunity to have a better understanding of your systems, to better understand what's happening in all the containers you're deploying and not tearing down the way that you should, to better understand what's happening in distributed systems. And it's going to be a real missed opportunity if that is what happens. If we just call this ‘Monitoring 2.0' it's going to leave a lot of unrealized potential in the market.Corey: The big problem that I've seen in a lot of different areas is—I'll be direct—consolidation where you have a company that starts to do a thing—and that's great—and then they start doing other things that are tied to it. And in turn, they start, I guess, gathering everything in the ecosystem. If you break down observability into various constituent parts, I—know, I know, the pillars thing is going to upset people; ignore that for now—and if you have an offering that's weak in a particular area, okay, instead of building it organically into the product, or saying, “Yeah, that's not what we do,” there's an instinct to acquire a company or build that functionality out. And it turns out that we're building what feels the lot to me like the SaaS equivalent of multifunction printers: they can print, they can scan, they can fax, and none of those three very well, so it winds up with something that dissatisfies everyone, rather than a best-of-breed solution that has a very clear and narrow starting and stopping point. How do you view that?Nick: Well, what you've described is a compromise, right? A compromise is everyone can work and no one's happy. And I think that's the advantage of where LogStream comes in. The reality is best-of-breed. Most enterprises today have 30 or more different monitoring tools—call them observability tools if you want to—and you will never pry those tools from the dead hands of those sysadmins, DevOps engineers, SREs, et cetera.They all integrate those tools into how they work and their processes. So, we're living in a best-of-breed world. It's like that in data and analytics—my former beat—and it's like that in monitoring and observability. People really gravitate towards the tools they like, they gravitate towards the tools their friends are using. And so you need a way to be able to mix and match that stuff.And just because I want to stay [laugh] on message, that's really where the LogStream story kind of blends in because we do that; we allow you to mix and match all those different pieces.Corey: Joke's on you. I use Nagios and I have no friends. I'm not convinced those two things are entirely unrelated, but here we are. So here's, I guess, the big burning question that a lot of folks—certainly not me, but other undefined folks, ‘lots of people are saying'—so you built something interesting that actually works. I want to be clear on this.I have spoken to customers of yours. They swear by it instead of swearing at it, which happens with other companies. Awesome. You have traction, you're moving forward, things are going great. Here's $200 million is the next part of that story, and on some level, my immediate reaction—which does need updating, let's be clear here—is like, all right.I'm trying to build a product. I can see how I could spend a few million bucks. “Well, what can you do with I don't know, 100 times that?” My easy answer is, “Something monstrous.” I don't believe that is the case here. What is the growth plan? What are you doing that makes having that kind of a war chest a useful and valuable thing to have?Nick: Well, if you speak with the co-founders—and they've been open about this—we view ourselves as a generational company. We're not just building one product. We've been thinking about, how do we deliver on observability as this idea of discovery? What does that take? And it doesn't mean that we're going to be less agnostic to other destinations, we still think there's an incredible amount of value there and that's not going away, but we think there's maybe an interim step that we build out, potentially this idea of an observability data lake where you can explore these environments.Certainly, there's other types of options in the space today. Most of them are SQL-based, which is interesting because the audience that uses monitoring and observability tools couldn't care less about SQL right? They want search, they want regex, and so you've got to have the right tool for that audience. And so we're thinking about what that looks like going forward. We're doubling down on people.Surprisingly, this is a very—like anything else in software, it is people-intensive. And so certainly those are other aspects that we're exploring with the recent investment, but definitely, multiproduct company is our future and continued expansion.Corey: Expansion is always a fun one. It's the idea of, great, are you looking at going deeper into the areas you're already active within, or is it more of a, “Ah, so we've solved the, effectively, log routing problem. That's great. Let's solve other problems, too.” Or is it more of a, I guess, a doubling down and focusing on what's working? And again, that probably sounds judgmental in a way I don't intend it to at all. I just have a hard time contextualizing that level of scale coming from a small company perspective the way that I do.Nick: Yeah. Our plan is to focus more intently on the areas that we're in. We have a huge basis of experience there. We don't want to be all things to all people; that dilutes the message down to nothing, so we want to be very specific in the audiences we talk to, the problems we're trying to solve, and how we try to solve them.Corey: The problem I've always found with a lot of the acquisition, growth thrashing of—let me call it what I think it is: companies in decline trying to strain relevancy, it feels almost like a, “We don't see a growth strategy. So, we're going to try and acquire everything that hold still long enough, at some level, trying to add more revenue to the pile, but also thrashing in the sense of, okay. They're going to teach us how to do things in creative, awesome ways,” but it never works out that way. When you have a 50,000 person company acquiring a 200 person company, invariably the bigger culture is going to dominate. And I don't understand why that mistake seems to continually happen again, and again, and again.And people think I'm effectively alluding to—or whenever the spoken word version of subtweeting is—a particular company or a particular acquisition. I'm absolutely not, there are probably 50 different companies listening right now who thinks, “Oh, God. He's talking about us.” It's the common repeating trend. What is that?Nick: It's hard to say. In some cases, these acquisitions might just be talent. “We need to know how to do X. They know how to do X. Let's do it.” They may have very unique niche technology or software that another company thinks they can more broadly apply.Also, some of these big companies, these may not be board-level or CEO-level decisions. A business unit might decide, “Oh, I like what that company is doing. I'm going to go acquire it.” And so it looks like MegaCorp bought TinyCorp, but it's really, this tiny business unit within MegaCorp bought tiny company. The reality is often different from what it looks like on the outside.So, that's one way. Another is, you know, if they're going to teach us to be more effective with tech or something like that, you're never going to beat culture. You're never going to be the existing culture. If it's 50,000, against 200, obviously we know who wins there. And so I don't know if that's realistic.I don't know if the big companies are genuine when they say that, but it could just be the messaging that they use to make people happy and hopefully retain as many of those new employees for as long as they can. Does that make sense?Corey: No, it makes perfect sense. It's the right answer. It does articulate what is happening there, and I think I keep falling prey to the same failure. And it's hard. It's pernicious, but companies are not monolithic entities.There's no one person at all of these companies each who is making these giant unilateral decisions. It's always some product manager or some particular person who has a vision and a strategy in the department. It is not something that the company board is agreeing on every little decision that gets made. They're distributed entities in many respects.Nick: Absolutely. And that's only getting more pervasive as companies get larger [laugh] through acquisition. So, you're going to see more and more of that, and so it's going to look like we're going to put one label on it, one brand. Often, I think internally, that's the exact opposite of what actually happened, how that decision got made.Corey: Nick, I want to thank you for taking so much time to speak with me about what you're up to over there, how your path has shaped, how you view the world, and also what Cribl does these days. If people want to learn more about what you're up to, how you think about the world, or even possibly going to work at Cribl which, having spoken to a number of people over there, I would endorse it. How do they find you?Nick: Best place to find us is by joining our community: cribl.io/community, and Cribl is spelled C-R-I-B-L. You can certainly reach out there, we've got about 2300 people in our community Slack, so it's a great group. You can also reach out to me on Twitter, I'm @nheudecker, N-H-E-U-D-E-C-K-E-R. Tell me what you thought of the episode; love to hear it. And then beyond that, you can also sign up for our free cloud tier at cribl.cloud. It's a pretty generous one terabyte a day processing, so you can start to send data in and send it wherever you'd like to be.Corey: To be clear, this free as in beer, not free as an AWS free tier?Nick: This is free as in beer.Corey: Excellent. Excellent.Nick: I think I'm getting that right. I think it's free as in beer. And the other thing you can try is our hosted solution on AWS, fully managed cloud at cribl.cloud, we offer a free one terabyte per day processing, so you can start to send data into that environment and send it wherever you'd like to go, in whatever shape that data needs to be in when it gets there.Corey: And we will, of course, put links to that in the [show notes 00:35:21]. Thank you so much for your time today. I really appreciate it.Nick: No, thank you for having me. This was a lot of fun.Corey: Nick Heudecker, senior director, market strategy and competitive intelligence at Cribl. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with a comment explaining that the only real reason a startup should raise a $200 million funding round is to pay that month's AWS bill.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

With Covid-19, businesses are going virtual. Whether large or small, every business is starting to sell online through establishing a website or an app. Unfortunately, not every small business owner has the technical knowledge of building a website. Few people know about programming and coding. Today, we host a tech expert who is teaching small businesses technical skills through a non-technical approach and helping them customize their websites. Who is Simon Sorour? Simon Sorour, an analyst, consultant in business innovation and technology, and community philanthropist, shares his SQL programming online course information. His five days course is offering small entrepreneurs' foundational knowledge. How does Simon's course help small businesses? Simon's course is essential in enabling small businesses to improve their product and service provision. For instance, small business owners will struggle to meet or understand some technicalities, details, or configurations in their websites. Simon explains how this course can help the owner resolve such issues by offering them essential skills. Listen to Simon as he shares what they are doing to help small businesses and startups understand how to tackle the market and put themselves out there. Simon's Inner Superpower Simon believes in persistence and adaptability as the pillars of success. To him, persistence is the normalizing of keeping on trying without stopping, while adaptability is learning and changing the way you do things in the process. He rides on the belief that success comes through loss cycles or alterations which make one better with time. Timestamps [00:32] Getting to know Simon [03:46] Simon online technical course and why one should consider it [07:47] Improvements a small business can make using SQL course skills [10:22] How Simon offer support to small businesses [] Simon's plan to reach out to the global small business [15:03] Inner superpowers that keep Simon moving [17:17] Simon's recommendation to people working in innovation jobs Quotes “If you're a small business and you don't have much configurability of your website, it can be hard to achieve some things through business.” “A lot of people are just talking about innovation, but they don't really understand what the meaning of innovation is, which is mainly getting the value from whatever you're putting in the market.” “You don't really fail until you stop trying.” “Through adaptability, you are learning through the way and changing whatever you fail to do, or whatever you did in a way that can be done better.” “Before going out there and putting your product or creating your product, you have to talk to people to understand if they have the same problem that you have.” Connect With Simon: Website - https://www.brainiwi.com/  Linkedin - https://www.linkedin.com/in/msolimanfouad/

About AbbyWith over twenty years in the tech world, Abby Kearns is a true veteran of the technology industry. Her lengthy career has spanned product marketing, product management and consulting across Fortune 500 companies and startups alike. At Puppet, she leads the vision and direction of the current and future enterprise product portfolio. Prior to joining Puppet, Abby was the CEO of the Cloud Foundry Foundation where she focused on driving the vision for the Foundation as well as  growing the open source project and ecosystem. Her background also includes product management at companies such as Pivotal and Verizon, as well as infrastructure operations spanning companies such as Totality, EDS, and Sabre.Links: Cloud Foundry Foundation: https://www.cloudfoundry.org Puppet: https://puppet.com Twitter: https://twitter.com/ab415 TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Liquibase. If you're anything like me, you've screwed up the database part of a deployment so severely that you've been banned from touching every anything that remotely sounds like SQL, at at least three different companies. We've mostly got code deployments solved for, but when it comes to databases we basically rely on desperate hope, with a roll back plan of keeping our resumes up to date. It doesn't have to be that way. Meet Liquibase. It is both an open source project and a commercial offering. Liquibase lets you track, modify, and automate database schema changes across almost any database, with guardrails to ensure you'll still have a company left after you deploy the change. No matter where your database lives, Liquibase can help you solve your database deployment issues. Check them out today at liquibase.com. Offer does not apply to Route 53.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate: is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards, while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other, which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at Honeycomb.io/screaminginthecloud. Observability, it's more than just hipster monitoring.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Once upon a time, I was deep into the weeds of configuration management, which explains a lot, such as why it seems I don't know happiness in any meaningful sense. Then I wound up progressing into other areas of exploration, like the cloud, and now we know for a fact why happiness isn't a thing for me. My guest today is the former CEO of the Cloud Foundry Foundation and today is the CTO over at a company called Puppet, which we've talked about here from time to time. Abby Kearns, thank you for joining me. I appreciate your taking the time out of your day to suffer my slings and arrows.Abby: Thank you for having me. I have been looking forward to this for weeks.Corey: My stars, it seems like things are slow over there, and I kind of envy you for that. So, help me understand something; you went from this world of cloud-native everything, which is the joy of working with Cloud Foundry, to now working with configuration management. How is that not effectively Benjamin Button-ing your career. It feels like the opposite direction that most quote-unquote, “Digital transformations” like to play with. But I have a sneaking suspicion, there's more to it than I might guess from just looking at the label on the tin.Abby: Beyond I just love enterprise infrastructure? I mean, come on, who doesn't?Corey: Oh, yeah. Everyone loves to talk about digital transformation, reading about books like a Head in the Cloud to my children used to be a fun nightly activity before it was formally classified as child abuse. So yeah, I hear you, but it turns out the rest of the world doesn't necessarily agree with us.Abby: I do not understand it. I have been in enterprise infrastructure my entire career, which has been a really, really long time, back when Unix and Sun machines were still a thing. And I'll be a little biased here; I think that enterprise infrastructure is actually the most fascinating part of technology right now. And why is that? Well, we're in the process of actively rewritten everything that got us here.And we talk about infrastructure and everyone's like, “Yeah, sure, whatever,” but at the end of the day, it's the foundation that everything that you think is cool about technology is built on. And for those of us that really enjoy this space, having a front-row seat at that evolution and the innovation that's happening is really, really exciting and it creates a lot of interesting conversation, debate, evolution of technologies, and innovation. And are they all going to be on the money five, ten years from now? Maybe not, but they're creating an interesting space and discussion and just the work ahead for all of us across the board. And I'm kind of bucketing this pretty broadly, intentionally so because I think at the end of the day, all of us play a role in a bigger piece of pie, and it's so interesting to see how these things start to fit together.Corey: One of the things that I've noticed is that the things that get attention on the keynote stage of, “This is this far future, serverless, machine-learning Kubernetes, dingus nonsense,” great is—Abby: You forgot blockchain. [laugh].Corey: Oh, yeah. Oh, yeah blockchain as well. Like, what other things can we wind up putting into the buzzword thing to wind up guaranteeing that your seed round is at least $200 million? Great. There's that.But when you look at the actual AWS bill—my specialty, of course—and seeing where the money is actually going, it doesn't really look that different, as far as percentages go—even though the numbers are higher—than it did ten years ago, at least in the enterprise world. You're still buying a bunch of EC2 instances, you're still potentially modernizing to some of the managed services like RDS—which is Amazon's reimagining of what a database could be if you still had to manage the finicky bits, but had no control over when and how they worked—and of course, data transfer and disk. These are the basic building blocks of everything in cloud. And despite how much we talk about the super neat stuff, what we're doing is not reflected on the conference stage. So, I tend to view the idea of aspirational architecture as its own little world.There are still seasoned companies out there that are migrating from where they are today into this idea of, well, virtualization, we've just finally got our heads around that. Now, let's talk about this cloud thing; seems like a fad—in 2021. And people take longer to get to where they think they're going or where they intend to go than they plan for, and they get stuck somewhere and instead of a cloud migration, they're now hybrid because they can redefine things and declare victory when they plant that flag, and here we are. I'm not here to make fun of these companies because they're doing important work and these are super hard problems. But increasingly, it seems that the technology is not the thing that's holding them back or even responsible for their outcome so much as it is people.The more I work with tech, the more I realized that everything that's hard becomes people issues. Curious to get your take on that, given your somewhat privileged perspective as having a foot standing very deeply in each world.Abby: Yeah, and that's a super great point. And I also realized I didn't fully answer the first question either. So, I'll tie those two things together.Corey: That's okay, we're going to keep circling around until you get there. It's fine.Abby: It's been a long week, and it's only Wednesday.Corey: All day long, as it turns out.Abby: I have a whole soapbox that I drag around behind me about people and process, and how that's your biggest problem, not technology, and if you don't solve for the people in the process, I don't care what technology you choose to use, isn't going to fix your problem. On the other hand, if you get your people and process right, you can borderline use crayons and paper and get [laugh] really close to what you need to solve for.Corey: I have it on good authority that's known as IBM Cloud. Please continue.Abby: [laugh]. And so I think people and process are at the heart of everything. They're our biggest accelerators with technology and they're our biggest limitation. And you can cloud-native serverless your way into it, but if you do not actually do continuous delivery, if you did not actually automate your responses, if you do not actually set up the cross-functional teams—or sometimes fondly referred to as two-pizza teams—if you don't have those things set up, there isn't any technology that's going to make you deliver software better, faster, cheaper. And so I think I care a lot about the focus on that because I do think it is so important, but it's also—the reason a lot of people don't like to talk about it and deal with it because it's also the hardest.People, culture change, digital transformation, whatever you want to call it, is hard work. There's a reason so many books are written around DevOps. And you mentioned Gene Kim earlier, there's a reason he wrote The Phoenix Project; it's the people-process part is the hardest. And I do think technology should be an enabler and an accelerator, but it really has to pair up nicely with the people part. And you asked your earlier question about my move to Puppet.One of the things that I've learned a lot in running the Cloud Foundry Foundation, running an open-source software foundation, is you could a real good crash course in how teams can collaborate effectively, how teams work together, how decisions get made, the need for that process and that practice. And there was a lot of great context because I had access to so much interesting information. I got to see what all of these large enterprises were doing across the board. And I got to have a literal seat at the table for how a lot of the decisions are getting made around not only the open-source technologies that are going into building the future of our enterprise infrastructure but how a lot of these companies are using and leveraging those technologies. And having that visibility was amazing and transformational for myself.It gave me so much richness and context, which is why I have firmly believed that the people and process part were so crucial for many years. And I decided to go to a company that sold products. [laugh]. You're like, “What? What is she talking about now? Where is this going?”And I say that because running an open-source software foundation is great and it gives you so much information and so much context, but you have no access to customers and no access to products. You have no influence over that. And so when I thought about what I wanted to do next, it's like, I really want to be close to customers, I really want to be close to product, and I really want to be part of something that's solving what I look at over the next five to ten years, our biggest problem area, which is that tweener phase that we're going to be in for many years, which we were just talking about, which is, “I have some stuff on-prem and I have some stuff in a cloud—usually more than one cloud—and I got to figure out how to manage all of that.” And that is a really, really, really hard problem. And so when I looked at what Puppet was trying to do, and the opportunity that existed with a lot of the fantastic work that Puppet has done over the last 12 years around Desired State Configuration management, I'm like, “Okay, there's something here.”Because clearly, that problem doesn't go away because I'm running some stuff in the cloud. So, how do we start to think about this more broadly and expansively across the hybrid estate that is all of these different environments? And who is the most well-positioned to actually drive an innovative product that addresses that? So, that's my long way of addressing both of those things.Corey: No, it's a fair question. Friend of the show, Matt Stratton, is famous for saying that, “You cannot buy DevOps, but I sure would like to sell it to you,” and if you're looking at it from that perspective, Puppet is not far from what that product store look like in some ways. My first encounter with Puppet was back around 2009, 2010 or so, and I was using it in an environment I was working within and thought, “Okay, this is terrible, and it's crap, and obviously, I know what I'm doing far better than this, and the problem is the Puppet's a bad product.” So, I was one of the early developers behind SaltStack, which was a terrific, great way of approaching the problem from a novel perspective, and it wasn't crap; it was awesome. Right up until I saw the first time a customer deployed it and looked at their environment, and it wasn't crap, it was worse because it turns out that you can build a super finely crafted precision instrument that makes a fairly bad hammer, but that's how customers are going to use it anyway.Abby: Well, I mean, [sigh] look, you actually hit something that I think we don't actually talk about, which is how hard all of this shit really is. Automation is hard. Automation for distributed systems at scale is super duper hard. There isn't an easy way to solve that problem. And I feel like I learned a lot working with Cloud Foundry.Cloud Foundry is a Platform as a Service and it sits a layer up, but it had the same challenges in that solving the ability to run cloud-native applications and cloud-native workloads at scale and have that ephemerality to it and that resilience to it, and the things everyone wants but don't recognize how difficult it is, actually, to do that well. And I think the same—you know, that really set me up for the way that I think about the problem, even the layer down which is, running and managing desired state, which at the end of the day is a really fancy way of saying, “Does your environment look like the way you think it should? And if it doesn't, what are you going to do about it?” And it seems like, in this year of—what year are we again? 2021, maybe? I don't know. It feels like the last two years of, sort of, munged together?Corey: Yeah, the passing of time is something it's very hard for me to wrap my head around.Abby: But it feels like, I know some people, particularly those of us that have been in tech a long time are probably like, “Why are we still talking about that? Why is that a thing?” But that is still an incredibly hard problem for most organizations, large and small. So, I tend to spend a lot of time thinking about large enterprises, but in the day, you've got more than 20 servers, you're probably sitting around thinking, “Does my environment actually look the way I think it does? There's a new CVE that just came out. Am I able to address that?”And I think at the end of the day, figuring out how you can solve for that on-prem has been one of the things that Puppet has worked for, and done really, really well the last 12 years. Now, I think the next challenge is okay, how do you extend that out across your now bananas complex estate that is—I got a huge data estate, maybe one or two data centers, I got some stuff in AWS, I got some stuff in GCP, oh yeah, got a little thing over here and Azure, and oh, some guy spun up something on OCI. So, we got a little bit of everything. And oh, my God, the SolarWinds breach happened. Are we impacted? I don't know. What does that mean? [laugh].And I think you start to unravel the little pieces of that and it gets more and more complex. And so I think the problems that I was solving in the early aughts with servers seems trite now because you're like, I can see all of my servers; there's eight of them. Things seem fine. To now, you've got hundreds of thousands of applications and workloads, and some of them are serverless, and they're all over the place. And who has what, and where does it sit?And does it look like the way that I think it needs to so that I can run my business effectively? And I think that's really the power of it, but it's also one of those things that I don't feel like a lot of people like to acknowledge the complexity and the hardness of that because it's not just the technology problem—going back to your other question, how do we work? How do we communicate? What are our processes around dealing with this? And I think there's so much wrapped up in that it becomes almost like, how do you eat an elephant story, right? Yes, one bite at a time, but when you first look at the elephant, you're like, “Holy shit. This is big. What do I need to do?” And that I think is not something we all collectively spend enough time talking about is how hard this stuff is.Corey: One of the biggest challenges I see across the board is this idea of conference-ware style architecture; the greatest lie you ever see is someone talking about their infrastructure in public because peel it back a little bit and everything's messy, everything's disastrous, and everything's a tire fire. And we have this cult in tech—Abby: [laugh].Corey: —it's almost a cult where we have this idea that anything that isn't rewritten completely within the last six months based upon whatever is the hot framework now that is designed to run only in Google Chrome running on the latest generation MacBook Pro on a gigabit internet connection is somehow less than. It's like, “So, what does that piece of crap do?” And the answer is, “Well, a few $100 million a quarter in revenue, so how about you watch your mouth?” Moving those things is delicate; moving those things is fraught, and there are a lot of different stakeholders to the point where one of the lessons I keep learning is, people love to ask me, “What is Amazon's opinion of you?” Turns out that there's no Ted Amazon who works over there who forms a single entity's opinion. It's a bunch of small teams. Some of them like me, some of them can't stand me, far and away the majority don't know who I am. And that is okay. In theory; in practice, I find it completely unforgivable because how dare you? But I understand it's—Abby: You write a memo, right now. [laugh].Corey: Exactly. Companies are people and people are messy, and for better or worse, it is impossible to patch them. So, you have to almost route around them. And that was something that I found that Puppet did very well, coming from the olden days of sysadmin work where we spend time doing management [bump 00:15:53] the systems by hand. Like, oh, I'm going to do a for loop. Once I learned how to script. Before that, I use Cluster SSH and inadvertently blew away a University's entire config file what starts up on boot across their entire FreeBSD server fleet.Abby: You only did it once, so it's fine.Corey: Oh, yeah. I'm never going to screw up again. Well, not like that. In other ways. Absolutely, but at least my errors will be novel.Abby: Yeah. It's learning. We all learn. If you haven't taken something down in production in real-time, you have not lived. And also you [laugh] haven't done tech. [laugh].Corey: Oh, yeah, you either haven't been allowed close enough to anything that's important enough to be able to take down, you're lying to me, or thirdly—and this is possible, too—you're not yet at a point in your career where you're allowed to have access to the breaky parts. And that's fine. I mean, my argument has always been about why I'd be a terrible employee at Google, for example, is if I went in maliciously on day one, I would be hard-pressed to take down google.com for one hour. If I can't have that much impact intentionally going in as a bad actor, it feels like there'd be how much possible upside, positive impact can I have what everyone's ostensibly aligned around the same thing?It's the challenge of big companies. It's gaining buy-in, it's gaining investment in the idea and the direction you're going in. Things always take longer, you have to wind up getting multiple stakeholders on board. My consulting practice is entirely around helping save money on the AWS bill. You'd think it would be the easiest thing in the world to sell, but talking to big companies means a series of different sales conversations with different folks, getting them all on the same page. What we do functionally isn't so much look at the computer parts as it is marriage counseling between engineering and finance. Different languages, different ways of thinking about things, ostensibly the same goals.Abby: I mean, I don't think that's a big company problem. I think that's an every company problem if you have more than, like, five people in your company.Corey: The first few years here, it was just me and I had none of those problems. I had very different problems, but you know—and then we started bringing other people in, it's like, “Oh, yeah, things were great until we hired people. Ugh, mistake. Never do that.” And yeah, it turns out that's not particularly sustainable.Abby: Stakeholder management is hard. And you mentioned something about routing around. Well, you can't actually route around people, unfortunately. You have to get people to buy in, you have to bring people along on the journey. And not everybody is at the same place in the way they think about the work you're doing.And that's true at any company, big or small. I think it just gets harder and more complex as the company gets bigger because it's harder to make the changes you need to make fast enough, but I'd say even at a company the size of Puppet, we have the exact same challenges. You know, are the teams aligned? Are we aligned on the right things? Are we focusing on the right things?Or, do we have the right priorities in our backlog? How are we doing the work that we do? And if you're trying to drive innovation, how fast are we innovating? Are we innovating fast enough? How tight are our feedback loops?It's one of those things where the conversations that you and I have had externally with customers are the same conversations I have internally all the time, too. Let's talk about innovators' dilemma. [laugh]. Let's talk about feedback loop. Let's talk about what does it mean to get tighter feedback loops from customers and the field?And how do you align those things to the priorities in your backlog? And it's one of those never-ending challenges that's messy and complicated. And technology can enable it, but the technology is also messy and hard. And I do love going to conferences and seeing how pretty and easy things could look, and it's definitely a great aspiration for us to all shoot for, but at the end of the day, I think we all have to recognize there's a ton of messiness that goes on behind to make that a reality and to make that really a product and a technology that we can sell and get behind, but also one that we buy in, too, and are able to use. So, I think we as a technology industry, and particularly those of us in the Bay Area, we do a disservice by talking about how easy things are and why—you know, I remember a conversation I had in 2014 where someone asked me if Docker was already passe because everybody was doing containerized applications, and I was like, “Are they? Really? Is that an everyone thing? Or is that just an ‘us' thing?” [laugh].Corey: Well, they talk about it on the conference stages an awful lot, but yeah. New problems that continue to arise. I mean, I look back at my early formative years as someone who could theoretically be brought out in public and it was through a consulting project, where I was a traveling trainer for Puppet back in 2014, 2015, and teaching people who hadn't had exposure before what Puppet was about. And there was a definite experience in some of the people attending class where they were very opposed to the idea. And dig down a little bit, it's not that they had a problem with the software, it's not that they had a problem with any of the technical bits.It's that they made the mistake that so many technologists made—I know I have, repeatedly—of identifying themselves with the technology that they work on. And well, in some cases, yeah, the answer was that they ran a particular script a bunch of times and if you can automate that through something like Puppet or something else, well, what does that mean for them? We see it much larger-scale now with people who are, okay, I'm in the data center working on the storage arrays. When that becomes just an API call or—let's be serious, despite what we see in conference stages—when it becomes clicking buttons in the AWS console, then what does that mean for the future of their career? The tide is rising.And I can't blame them too much for this; you've been doing this for 25 years, you don't necessarily want to throw all that away and start over with a whole new set of concepts and the rest because unlike what Twitter believes, there are a bunch of legitimate paths in this industry that do treat it as a job rather than an all-consuming passion. And I have no negative judgment toward folks who walk down that direction.Abby: Most people do. And I think we have to be realistic. It's not just some. A lot of people do. A lot of people, “This is my nine-to-five job, Monday through Friday, and I'm going to go home and I'm going to spend time with my family.”Or I'm going to dare I say—quietly—have a life outside of technology. You know, but this is my job. And I think we have done a disservice to a lot of those individuals who for better or for worse, they just want to go in and do a job. They want to get their job done to the best of their abilities, and don't necessarily have the time—or if you're a single parent, have the flexibility in your day to go home and spend another five, six hours learning the latest technology, the latest programming language, set up your own demo environment at home, play around with AWS, all of these things that you may not have the opportunity to do. And I think we as an industry have done a disservice to both those individuals, as well in putting up really imaginary gates on who can actually be a technologist, too.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of "Hello, World" demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking databases, observability, management, and security.And - let me be clear here - it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself all while gaining the networking load, balancing and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build.With Always Free you can do things like run small scale applications, or do proof of concept testing without spending a dime. You know that I always like to put asterisks next to the word free. This is actually free. No asterisk. Start now. Visit https://snark.cloud/oci-free that's https://snark.cloud/oci-free.Corey: Gatekeeping, on some level, is just—it's a horrible thing. Something I found relatively early on is that I didn't enjoy communities where that was a thing in a big way. In minor ways, sure, absolutely. I wound up gravitating toward Ubuntu rather than Debian because it turned out that being actively insulted when I asked how to do something wasn't exactly the most welcoming, constructive experience, where they, “Read the manual.” “Yeah, I did that and it was incomplete and contradictory, and that's why I'm here asking you that question, but please continue to be a condescending jackwagon. I appreciate that. It really just reminds me that I'm making good choices with my life.”Abby: Hashtag-RTFM. [laugh].Corey: Exactly. In my case, fine, its water off a duck's back. I can certainly take it given the way that I dish it out, but by the same token, not everyone has a quote-unquote, thick skin, and I further posit that not everyone should have to have one. You should not get used to personal attacks as a prerequisite for working in this space. And I'm very sensitive to the idea that people who are just now exploring the cloud somehow feel that they've missed out on their career, and that so there's somehow not appropriate for this field, or that it's not for them.And no, are you kidding me? You know that overwhelming sense of confusion you get when you look at the AWS console and try and understand what all those services do? Yeah, I had the same impression the first time I saw it and there were 12 services; there's over 200 now. Guess what? I've still got it.And if I am overwhelmed by it, I promise there's no shame in anyone else being overwhelmed by it, too. We're long since past the point where I can talk incredibly convincingly about AWS services that don't exist to AWS employees and not get called out on it because who in the world has that entire Rolodex of services shoved into their heads who isn't me?Abby: I'd say you should put out… a call for anyone that does because I certainly do not memorize the services that are available. I don't know that anyone does. And I think even more broadly, is, remember when the landscape diagram came out from the CNCF a couple of years ago, which it's now, like… it's like a NASCAR logo of every logo known to man—Corey: Oh today, there's over 400 icons on it the last time I saw—I saw that thing come out and I realized, “Wow, I thought I was going to shit-posting,” but no, this thing is incredible. It's, “This is great.” My personal favorite was zooming all the way in finding a couple of logos on in the same box three times, which is just… spot on. I was told later, it's like, “Oh, those represent different projects.” I'm like, “Oh, yeah, must have missed that in the legend somewhere.” [laugh]. It's this monstrous, overdone thing.Abby: But the whole point of it was just, if I am running an IT department, and I'm like, “Here you go. Here's a menu of things to choose,” you're just like, “What do I do with this information? Do I choose one of each? All the above? Where do I go? And then, frankly, how do I make them all work together in my environment?” Because they all serve very different problems and they're tackling different aspects of that problem.And I think I get really annoyed with myself as an industry—like, ourselves as an industry because it's like, “What are we doing here?” We're trying to make it harder for people, not only to use the technology, to be part of it. And I think any efforts we can make to make it easier and more simple or clear, we owe it to ourselves to be able to tell that story. Which now the flip side of that is describing cloud-native in the cloud, and infrastructure and automation is really, really hard to do [laugh] in a way that doesn't use any of those words. And I'm just as guilty of this, of describing things we do and using the same language, and all of a sudden you're looking at it this says the same thing is 7500 other websites. [laugh]. So.Corey: Yep. I joke at RSA's Expo Hall is basically about twelve companies selling different things. Sure, each one has a whole bunch of booths with different logos and different marketing copy, but it's the same fundamental product. Same challenge here. And this is, to me, the future of cloud, this is where it's going, where I want something that will—in my case, I built a custom URL shortener out of DynamoDB, API Gateway, Lambda, et cetera, and I built this thing largely as a proof of concept because I wanted to have experience playing with these tools.And that was great, not but if I'm doing something like that in production, I'm going with Bitly or one of the other services that provide this where someone is going to maintain it full time. Unless it is the core of what I'm doing, I don't want to build it myself from popsicle sticks. And moving up the stack to a world of folks who are trying to solve a business problem and they don't want to deal with the ten prerequisite services to understand the cloud, and then a whole bunch of other things tied together, and the billing, and the flow becomes incredibly problematic to understand—not to mention insecure: because we don't understand it, you don't know what your risk exposure is—people don't want that. They—Abby: Or to manage it.Corey: Yeah.Abby: Just the day-to-day management. Care and feeding, beyond security. [laugh].Corey: People's time is free. So, yeah. For example, do I write my own payroll system? Absolutely not. I have the good sense to pay a turnkey company to handle that for me because mistakes will show.I started my career running email systems. I pay for Google workspaces—or GSuite, or Gmail, or whatever the hell they're calling it this week—because it's not core and central to my business. I want a thing that winds up solving a business problem, and I will pay commensurately to the value that thing delivers, not the individual constituent costs of the components that build it together. Because until you're significantly scaled out and it is the core of what you do, you're spending more on people to run the monstrous thing than you are for the thing itself. That's always the way it works.So, put your innovation where it matters for your business. I posit the for an awful lot of the things we're building, in order to achieve those outcomes, this isn't it.Abby: Agreed. And I am a big believer in if I can use off-the-shelf software, I will because I don't believe in reinventing everything. Now, having said that, and coming off my soapbox for just a hot minute, I will say that a lot of what's happening, and going back to where I started around the enterprise infrastructure, we're reinventing so many things that there is a lot of new things coming up. We've talked about containers, we've talked about Kubernetes, around container scheduling, container orchestration, we haven't even mentioned service mesh, and sidecars, and all of the new ways we're approaching solving some of these older problems. So, there is the need for a broad proliferation of technology until the contraction phase, where it all starts to fundamentally clicks together.And that's really where the interesting parts happen, but it's also where the confusion happens because, “Okay, what do I use? How do I use it? How do these pieces fit together? What happens when this changes? What does this mean?”And by the way, if I'm an enterprise company, I'm a payroll company, what's the one thing I care about? My payroll software. [laugh]. And that's the problem I'm solving for. So, I take a little umbrage sometimes with the frame that every company is a software company because every company is not a software company.Every company can use technology in ways to further their business and more and more frequently, that is delivering their business value through software, but if I'm a payroll company, I care about delivering that payroll capabilities to my customer, and I want to do it as quickly as possible, and I want to leverage technology to help me do that. But my endgame is not that technology; my endgame is delivering value to my customers in real and meaningful ways. And I worry, sometimes, that those two things get conflated together. And one is an enabler of the other; the technology is not the outcome.Corey: And that is borderline heresy for an awful lot of folks out there in the space, I wish that people would wake up a little bit more and realize that you have to build a thing that solves customer pain, ideally, an expensive customer pain, and then they will basically rush to hurl money at you. Now, there are challenges and inflections as you go, and there's a whole bunch of nuances that can span entire fields of endeavor that I am hand-waving over here, and that's fine, but this is the direction I think we're going and this is the dawning awareness that I hope and trust we'll see start to take root in this industry.Abby: I mean, I hope so. I do take comfort in the fact that a lot of the industry leaders I'm starting to see, kind of, equate those two things more closely in the top [track 00:31:20]. Because it's a good forcing function for those of us that are technologists. At the end of the day, what am I doing? I am a product company, I am selling software to someone.So clearly, obviously, I have a vested interest in building the best software out there, but at the end of the day, for me, it's, “Okay, how do I make that truly impactful for customers, and how do I help them solve a problem?” And for me, I'm hyper-focused on automation because I honestly feel like that is the biggest challenge for most companies; it's the hardest thing to solve. It's like getting into your auto-driving car for the first time and letting go the steering wheel and praying to the software gods that that software is actually going to work. But it's the same thing with automation; it's like, “Okay, I have to trust that this is going to manage my environment and manage my infrastructure in a factual way and not put me on CNN because I just shut down entire customer environment,” or if I'm an airline and I've just had a really bad week because I've had technology problems. [laugh]. And so I think we have to really take into consideration that there are real customer problems on the other end of that we have to help solve for.Corey: My biggest problem is the failure mode of this is not when people watch the conference-ware presentations is that they're not going to sit there and think, “Oh, yeah, they're just talking about a nuanced thing that doesn't apply to our constraints, and they're hand-waving over a lot of stuff,” it's that, “Wow, we suck.” And that's not the takeaway anyone should ever have. Even Netflix doesn't operate the way that Netflix says that they do in their conference talks. It's always fun sitting next to someone from the company that's currently presenting and saying something to them, like, “Wow, I wish we did things that way.” And they said, “Yeah, I wish we did, too.”And it's always the case because it's very hard to get on stage and talk for 45 minutes about here's what we completely screwed up on, especially at the large publicly traded companies where it's, “Wait, why did our stock price just dive five perce—oh, my God, what did you say on stage?” People care [laugh] about those things, and I get it; there's a risk factor that I don't have to deal with here.Abby: I wish people would though. It would be so refreshing to hear someone like, “You know what? Ohh, we really messed this up, and let me walk you through what we did.” [laugh]. I think that would be nice.Corey: On some level, giving that talk in enough detail becomes indistinguishable from rage-quitting in public.Abby: [laugh].Corey: I mean, I'm there for it. Don't get me wrong. But I would love to see it.Abby: I don't think it has to be rage-quitting. One of the things that I talk to my team a lot about is the safety to fail. You can't take risk if you're too afraid to fail, right? And I think you can frame failure in a way of, “Hey, this didn't work, but let me walk you through all the amazing things we learned from this. And here's how we used that to take this and make this thing better.”And I think there's a positive way to frame it that's not rage-quitting, but I do think we as an industry gloss over those learnings that you absolutely have to do. You fail; everything does not work the first time perfectly. It is not brilliant out the gate. If you've done an MVP and it's perfect and every customer loves it, well then, you sat on that for way too long. [laugh]. And I think it's just really getting comfortable with this didn't work the first time or the fourth, but look, at time seven, this is where we got and this is what we've learned.Corey: I want to thank you for taking so much time out of your day to wind up speaking to me about things that in many cases are challenging to talk about because it's the things people don't talk about in the real world. If people want to learn more about what you're up to, who you are, et cetera, where can they find you?Abby: They can find me on the Twitters at @ab415. I think that's the best way to start, although I will say that I am not as prolific as you are on Twitter.Corey: That's a good thing.Abby: I'm a half-assed Tweeter. [laugh]. I will own it.Corey: Oh, I put my full ass into it every time, in every way.Abby: [laugh]. I do skim it a lot. I get a lot of my tech news from there. Like, “What are people mad about today?” And—Corey: The daily outrage. Oh, yeah.Abby: The daily outrage. “What's Corey ranting about today? Let's see.” [laugh].Corey: We will, of course, put a link to your Twitter profile in the [show notes 00:35:39]. Thank you so much for taking the time to speak with me. I appreciate it.Abby: Hey, it was my pleasure.Corey: Abby Kearns, CTO at Puppet. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with a comment telling me about the amazing podcast content you create, start to finish, at Netflix.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

This is episode 208 recorded on October 14th, 2021 where John & Jason go over the October 2021 Power BI updates including Heat map layer – Azure Maps Visual, improvements to Metamodels, ODBC support for Paginated Reports, and a call back to the September 2021 service update hidden feature. For show notes please visit www.bifocal.show

About EwereCloud, DevOps Engineer, Blogger and AuthorLinks: Infrastructure Monitoring with Amazon CloudWatch: https://www.amazon.com/Infrastructure-Monitoring-Amazon-CloudWatch-infrastructure-ebook/dp/B08YS2PYKJ LinkedIn: https://www.linkedin.com/in/ewere/ Twitter: https://twitter.com/nimboya Medium: https://medium.com/@nimboya My Cloud Series: https://mycloudseries.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate: is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards, while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other, which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at Honeycomb.io/screaminginthecloud. Observability, it's more than just hipster monitoring.Corey: This episode is sponsored in part by Liquibase. If you're anything like me, you've screwed up the database part of a deployment so severely that you've been banned from touching every anything that remotely sounds like SQL, at at least three different companies. We've mostly got code deployments solved for, but when it comes to databases we basically rely on desperate hope, with a roll back plan of keeping our resumes up to date. It doesn't have to be that way. Meet Liquibase. It is both an open source project and a commercial offering. Liquibase lets you track, modify, and automate database schema changes across almost any database, with guardrails to ensure you'll still have a company left after you deploy the change. No matter where your database lives, Liquibase can help you solve your database deployment issues. Check them out today at liquibase.com. Offer does not apply to Route 53.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I periodically make observations that monitoring cloud resources has changed somewhat since I first got started in the world of monitoring. My experience goes back to the original Call of Duty. That's right: Nagios.When you set instances up, it would theoretically tell you when they were unreachable or certain thresholds didn't work. It was janky but it kind of worked, and that was sort of the best we have. The world has progressed as cloud has become more complicated, as technologies have become more sophisticated, and here today to talk about this is the first AWS Hero from Africa and author of a brand new book, Ewere Diagboya. Thank you for joining me.Ewere: Thanks for the opportunity.Corey: So, you recently published a book on CloudWatch. To my understanding, it is the first such book that goes in-depth with not just how to wind up using it, but how to contextualize it as well. How did it come to be, I guess is my first question?Ewere: Yes, thanks a lot, Corey. The name of the book is Infrastructure Monitoring with Amazon CloudWatch, and the book came to be from the concept of looking at the ecosystem of AWS cloud computing and we saw that a lot of the things around cloud—I mostly talked about—most of this is [unintelligible 00:01:49] compute part of AWS, which is EC2, the containers, and all that, you find books on all those topics. They are all proliferated all over the internet, you know, and videos and all that.But there is a core behind each of these services that no one actually talks about and amplifies, which is the monitoring part, which helps you to understand what is going on with the system. I mean, knowing what is going on with the system helps you to understand failures, helps you to predict issues, helps you to also envisage when a failure is going to happen so that you can remedy it and also [unintelligible 00:02:19], and in some cases, even give you a historical view of the system to help you understand how a system has behaved over a period of time.Corey: One of the articles that I put out that first really put me on AWS's radar, for better or worse, was something that I was commissioned to write for Linux Journal, back when that was a print publication. And I accidentally wound up getting the cover of it with my article, “CloudWatch is of the devil, but I must use it.” And it was a painful problem that people generally found resonated with them because no one felt they really understood CloudWatch; it was incredibly expensive; it didn't really seem like it was at all intuitive, or that there was any good way to opt out of it, it was just simply there, and if you were going to be monitoring your system in a cloud environment—which of course you should be—it was just sort of the cost of doing business that you then have to pay for a third-party tool to wind up using the CloudWatch metrics that it was gathering, and it was just expensive and unpleasant all around. Now, a lot of the criticisms I put about CloudWatch's limitations in those days, about four years ago, have largely been resolved or at least mitigated in different ways. But is CloudWatch still crappy, I guess, is my question?Ewere: Um, yeah. So, at the moment, I think, like you said, CloudWatch has really evolved over time. I personally also had that issue with CloudWatch when I started using CloudWatch; I had the challenge of usability, I had the challenge of proper integration, and I will talk about my first experience with CloudWatch here. So, when I started my infrastructure work, one of the things I was doing a lot was EC2, basically. I mean, everyone always starts with EC2 at the first time.And then we had a downtime. And then my CTO says, “Okay, [Ewere 00:04:00], check what's going on.” And I'm like, “How do I check?” [laugh]. I mean, I had no idea of what to do.And he says, “Okay, there's a tool called CloudWatch. You should be able to monitor.” And I'm like, “Okay.” I dive into CloudWatch, and boom, I'm confused again. And you look at the console, you see, it shows you certain metrics, and yet [people 00:04:18] don't understand what CPU metric talks about, what does network bandwidth talks about?And here I am trying to dig, and dig, and dig deeper, and I still don't get [laugh] a sense of what is actually going on. But what I needed to find out was, I mean, what was wrong with the memory of the system, so I delved into trying to install the CloudWatch agent, get metrics and all that. But the truth of the matter was that I couldn't really solve my problem very well, but I had [unintelligible 00:04:43] of knowing that I don't have memory out of the box; it's something that has to set up differently. And trust me, after then I didn't touch CloudWatch [laugh] again. Because, like you said, it was a problem, it was a bit difficult to work with.But fast forward a couple of years later, I could actually see someone use CloudWatch for a lot of beautiful stuff, you know? It creates beautiful dashboards, creates some very well-aggregated metrics. And also with the aggregated alarms that CloudWatch comes with, [unintelligible 00:05:12] easy for you to avoid what to call incident fatigue. And then also, the dashboards. I mean, there are so many dashboards that simplified to work with, and it makes it easy and straightforward to configure.So, the bootstrapping and the changes and the improvements on CloudWatch over time has made CloudWatch a go-to tool, and most especially the integration with containers and Kubernetes. I mean, CloudWatch is one of the easiest tools to integrate with EKS, Kubernetes, or other container services that run in AWS; it's just, more or less, one or two lines of setup, and here you go with a lot of beautiful, interesting, and insightful metrics that you will not get out of the box, and if you look at other monitoring tools, it takes a lot of time for you to set up, for you to configure, for you to consistently maintain and to give you those consistent metrics you need to know what's going on with your system from time to time.Corey: The problem I always ran into was that the traditional tools that I was used to using in data centers worked pretty well because you didn't have a whole lot of variability on an hour-to-hour basis. Sure, when you installed new servers or brought up new virtual machines, you had to update the monitoring system. But then you started getting into this world of ephemerality with auto-scaling originally, and later containers, and—God help us all—Lambda now, where it becomes this very strange back-and-forth story of, you need to be able to build something that, I guess, is responsive to that. And there's no good way to get access to some of the things that CloudWatch provides, just because we didn't have access into AWS's systems the way that they do. The inverse, though, is that they don't have access into things running inside of the hypervisor; a classic example has always been memory: memory usage is an example of something that hasn't been able to be displayed traditionally without installing some sort of agent inside of it. Is that still the case? Are there better ways of addressing those things now?Ewere: So, that's still the case, I mean, for EC2 instances. So before, now, we had an agent called a CloudWatch agent. Now, there's a new agent called Unified Cloudwatch Agent which is, I mean, a top-notch from CloudWatch agent. So, at the moment, basically, that's what happens on the EC2 layer. But the good thing is when you're working with containers, or more or less Kubernetes kind of applications or systems, everything comes out of the box.So, with containers, we're talking about a [laugh] lot of moving parts. The container themselves with their own CPU, memory, disk, all the metrics, and then the nodes—or the EC2 instance of the virtual machines running behind them—also having their own unique metrics. So, within the container world, these things are just a click of a button. Everything happens at the same time as a single entity, but within the EC2 instance and ecosystem, you still find this there, although the setup process has been a bit easier and much faster. But in the container world, that problem has totally been eliminated.Corey: When you take a look at someone who's just starting to get a glimmer of awareness around what CloudWatch is and how to contextualize it, what are the most common mistakes people make early on?Ewere: I also talked about this in my book, and one of the mistakes people make in terms of CloudWatch, and monitoring in generalities: “What am I trying to figure out?” [laugh]. If you don't have that answer clearly stated, you're going to run into a lot of problems. You need to answer that question of, “What am I trying to figure out?” I mean, monitoring is so broad, monitoring is so large that if you do not have the answer to that question, you're going to get yourself into a lot of trouble, you're going to get yourself into a lot of confusion, and like I said, if you don't understand what you're trying to figure out in the first place, then you're going to get a lot of data, you're going to get a lot of information, and that can get you confused.And I also talked about what I call alarm fatigues or incident fatigues. This happens when you configure so many alarms, so many metrics, and you're getting a lot of alarms hitting and notification services—whether it's Slack, whether it's an email—and it causes fatigue. What happens here is the person who should know what is going on with the system gets a ton of messages and in that scenario can miss something very important because there's so many messages coming in, so many integrations coming in. So, you should be able to optimize appropriately, to be able to, like you said, conceptualize what you're trying to figure out, what problems are you trying to solve? Most times you really don't figure this out for a start, but there are certain bare minimums you need to know about, and that's part of what I talked about in the book.One of the things that I highlighted in the book when I talked about monitoring of different layers is, when you're talking about monitoring of infrastructure, say compute services, such as virtual machines, or EC2 instances, the certain baseline and metrics you need to take note of that are core to the reliability, the scalability, and the efficiency of your system. And if you focus on these things, you can have a baseline starting point before you start going deeper into things like observability and knowing what's going on entirely with your system. So, baseline understanding of—baseline metrics, and baseline of what you need to check in terms of different kinds of services you're trying to monitor is your starting point. And the mistake people make is that they don't have a baseline. So, we do not have a baseline; they just install a monitoring tool, configure a CloudWatch, and they don't know the problem they're trying to solve [laugh] and that can lead to a lot of confusion.Corey: So, what inspired you from, I guess, kicking the tires on CloudWatch—the way that we all do—and being frustrated and confused by it, all the way to the other side of writing a book on it? What was it that got you to that point? Were you an expert on CloudWatch before you started writing the book, or was it, “Well, by the time this book is done, I will certainly know [laugh] more about the service than I did when I started.”Ewere: Yeah, I think it's a double-edged sword. [laugh]. So, it's a combination of the things you just said. So, first of all, I have experienced with other monitoring tools; I have love for reliability and scalability of a system. I started Kubernetes at some of the early times Kubernetes came out, when it was very difficult to deploy, when it was very difficult to set up.Because I'm looking at how I can make systems a little bit more efficient, a little bit more reliable than having to handle a lot of things like auto-scaling, having to go through the process of understanding how to scale. I mean, that's a school of its own that you need to prepare yourself for. So, first of all, I have a love for making sure systems are reliable and efficient, and second of all, I also want to make sure that I know what is going on with my system per time, as much as possible. The level of visibility of a system gives you the level of control and understanding of what your system is doing per time. So, those two things are very core to me.And then thirdly, I had a plan of a streak of books I want to write based on AWS, and just like monitoring is something that is just new. I mean, if you go to the package website, this is the first book on infrastructure monitoring AWS with CloudWatch; it's not a very common topic to talk about. And I have other topics in my head, and I really want to talk about things like networking, and other topics that you really need to go deep inside to be able to appreciate the value of what you see in there with all those scenarios because in this book, every chapter, I created a scenario of what a real-life monitoring system or what you need to do looks like. So, being that I have those premonitions, I know that whenever it came to, you know, to share with the world what I know in monitoring, what I've learned in monitoring, I took a [unintelligible 00:12:26]. And then secondly, as this opportunity for me to start telling the world about the things I learned, and then I also learned while writing the book because there are certain topics in the book that I'm not so much of an expert in things, like big data and all that.I had to also learn; I had to take some time to do more research, to do more understanding. So, I use CloudWatch, okay? I'm kind of good in CloudWatch, and also, I also had to do more learning to be able to disseminate this information. And also, hopefully, X-Ray some parts of monitoring and different services that people do not really pay so much attention into.Corey: What do you find that is still the most, I guess, confusing to you as you take a look across the ecosystem of the entire CloudWatch space? I mean, every time I play with it, I take a look, and I get lost in, “Oh, they have contributor analyses, and logs, and metrics.” And it's confusing, and every time I wind up, I guess, spiraling out of control. What do you find that, after all of this, is a lot easier for you, and what do you find that's a lot more understandable?Ewere: I'm still going to go back to the containers part. I'm sorry, I'm in love containers. [laugh].Corey: No, no, it's fair. Containers are very popular. Everyone loves them. I'm just basically anti-container based upon no better reason than I'm just stubborn and bloody-minded most of the time.Ewere: [laugh]. So, pretty much like I said, I kind of had experience with other monitoring tools. Trust me, if you want to configure proper container monitoring for other tools, trust me, it's going to take you at least a week or two to get it properly, from the dashboards, to the login configurations, to the piping of the data to the proper storage engine. These are things I talked about in the book because I took monitoring from the ground up. I mean, if you've never done monitoring before, when you take my book, you will understand the basic principles of monitoring.And [funny 00:14:15], you know, monitoring has some big data process, like an ETL process: extraction, transformation, and writing of data into an analytic system. So, first of all, you have to battle that. You have to talk about the availability of your storage engine. What are you using? An Elasticsearch? Are you using an InfluxDB? Where do you want to store your data? And then you have to answer the question of how do I visualize the data? What method do I realize this data? What kind of dashboards do I want to use? What methods of representation do I need to represent this data so that it makes sense to whoever I'm sharing this data with. Because in monitoring, you definitely have to share data with either yourself or with someone else, so the way you present the data needs to make sense. I've seen graphs that do not make sense. So, it requires some level of skill. Like I said, I've [unintelligible 00:15:01] where I spent a week or two having to set up dashboards. And then after setting up the dashboard, someone was like, “I don't understand, and we just need, like, two.” And I'm like, “Really?” [laugh]. You know? Because you spend so much time. And secondly, you discover that repeatability of that process is a problem. Because some of these tools are click and drag; some of them don't have JSON configuration. Some do, some don't. So, you discover that scalability of this kind of system becomes a problem. You can't repeat the dashboards: if you make a change to the system, you need to go back to your dashboard, you need to make some changes, you need to update your login, too, you need to make some changes across the layer. So, all these things is a lot of overhead [laugh] that you can cut off when you use things like Container Insights in CloudWatch—which is a feature of CloudWatch. So, for me, that's a part that you can really, really suck out so much juice from in a very short time, quickly and very efficiently. On the flip side, when you talk about monitoring for big data services, and monitoring for a little bit of serverless, there might be a little steepness in the flow of the learning curve there because if you do not have a good foundation in serverless, when you get into [laugh] Lambda Insights in CloudWatch, trust me, you're going to be put off by that; you're going to get a little bit confused. And then there's also multifunction insights at the moment. So, you need to have some very good, solid foundation in some of those topics before you can get in there and understand some of the data and the metrics that CloudWatch is presenting to you. And then lastly, things like big data, too, there are things that monitoring is still being properly fleshed out. Which I think that in the coming months and years to come, they will become more proper and they will become more presentable than they are at the moment.Corey: This episode is sponsored by our friends at Oracle HeatWave is a new high-performance accelerator for the Oracle MySQL Database Service. Although I insist on calling it “my squirrel.” While MySQL has long been the worlds most popular open source database, shifting from transacting to analytics required way too much overhead and, ya know, work. With HeatWave you can run your OLTP and OLAP, don't ask me to ever say those acronyms again, workloads directly from your MySQL database and eliminate the time consuming data movement and integration work, while also performing 1100X faster than Amazon Aurora, and 2.5X faster than Amazon Redshift, at a third of the cost. My thanks again to Oracle Cloud for sponsoring this ridiculous nonsense.Corey: The problem I've always had with dashboards is it seems like managers always want them—“More dashboards, more dashboards”—then you check the usage statistics of who's actually been viewing the dashboards and the answer is, no one since you demoed it to the execs eight months ago. But they always claim to want more. How do you square that?I guess, slicing between what people asked for and what they actually use.Ewere: [laugh]. So yeah, one of the interesting things about dashboards in terms of most especially infrastructure monitoring, is the dashboards people really want is a revenue dashboards. Trust me, that's what they want to see; they want to see the money going up, up, up, [laugh] you know? So, when it comes to—Corey: Oh, yes. Up and to the right, then everyone's happy. But CloudWatch tends to give you just very, very granular, low-level metrics of thing—it's hard to turn that into something executives care about.Ewere: Yeah, what people really care about. But my own take on that is, the dashboards are actually for you and your team to watch, to know what's going on from time to time. But what is key is setting up events across very specific and sensitive data. For example, when any kind of sensitive data is flowing across your system and you need to check that out, then you tie a metric to that, and in turn alarm to it. That is actually the most important thing for anybody.I mean, for the dashboards, it's just for you and your team, like I said, for your personal consumption. “Oh, I can see all the RDS connections are getting too high, we need to upgrade.” Oh, we can see that all, the memory, there was a memory spike in the last two hours. I know that's for you and your team to consume; not for the executive team. But what is really good is being able to do things like aggregate data that you can share.I think that is what the executive team would love to see. When you go back to the core principles of DevOps in terms of the DevOps Handbook, you see things like a mean time to recover, and change failure rate, and all that. The most interesting thing is that all these metrics can be measured only by monitoring. You cannot change failure rates if you don't have a monitoring system that tells you when there was a failure. You cannot know your release frequency when you don't have a metric that measures number of deployments you have and is audited in a particular metric or a particular aggregator system.So, we discovered that the four major things you measure in DevOps are all tied back to monitoring and metrics, at minimum, to understand your system from time to time. So, what the executive team actually needs is to get a summary of what's going on. And one of the things I usually do for almost any company I work for is to share some kind of uptime system with them. And that's where CloudWatch Synthetics Canary come in. So, Synthetic Canary is a service that helps you calculate that helps you check for uptime of the system.So, it's a very simple service. It does a ping, but it is so efficient, and it is so powerful. How is it powerful? It does a ping to a system and it gets a feedback. Now, if the status code of your service, it's not 200 or not 300, it considers it downtime.Now, when you aggregate this data within a period of time, say a month or two, you can actually use that data to calculate the uptime of your system. And that uptime [unintelligible 00:19:50] is something you can actually share to your customers and say, “Okay, we have an SLA of 99.9%. We have an SLA of 99.8%.” That data should not be doctored data; it should not be a data you just cook out of your head; it should be based on your system that you have used, worked with, monitored over a period of time so that the information you share with your customers are genuine, they are truthful, and they are something that they can also see for themselves.Hence companies are using [unintelligible 00:20:19] like status page to know what's going on from time to time whenever there is an incident and report back to their customers. So, these are things that executives will be more interested in than just dashboards, [laugh] dashboards, and more dashboards. So, it's more or less not about what they really ask for, but what you know and what you believe you are going to draw value from. I mean, an executive in a meeting with a client and says, “Hey, we got a system that has 99.9% uptime.”He opens the dashboard or he opens the uptime system and say, “You see our uptime? For the past three months, this has been our metric.” Boom. [snaps fingers]. That's it. That's value, instantly. I'm not showing [laugh] the clients and point of graphs, you know? “Can you explain the memory metric?” That's not going to pass the message, send the message forward.Corey: Since your book came out, I believe, if not, certainly by the time it was finished being written and it was in review phase, they came out with Managed Prometheus and Managed Grafana. It looks almost like they're almost trying to do a completely separate standalone monitoring stack of AWS tooling. Is that a misunderstanding of what the tools look like, or is there something to that?Ewere: Yeah. So, I mean by the time those announced at re:Invent, I'm like, “Oh, snap.” I almost told my publisher, “You know what? We need to add three more chapters.” [laugh]. But unfortunately, we're still in review, in preview.I mean, as a Hero, I kind of have some privilege to be able to—a request for that, but I'm like, okay, I think it's going to change the narrative of what the book is talking about. I think I'm going to pause on that and make sure this finishes with the [unintelligible 00:21:52], and then maybe a second edition, I can always attach that. But hey, I think there's trying to be a galvanization between Prometheus, Grafana, and what CloudWatch stands for. Because at the moment, I think it's currently on pre-release, it's not fully GA at the moment, so you can actually use it. So, if you go to Container Insights, you can see that you can still get how Prometheus and Grafana is presenting the data.So, it's more or less a different view of what you're trying to see. It's trying to give you another perspective of how your data is presented. So, you're going to have CloudWatch: it's going to have CloudWatch dashboards, it's going to have CloudWatch metrics, but hey, this different tools, Prometheus, Grafana, and all that, they all have their unique ways of presenting the data. And part of the reason I believe AWS has Prometheus and Grafana there is, I mean, Prometheus is a huge cloud-native open-source monitoring, presentation, analytics tool; it packs a lot of heat, and a lot of people are so used to it. Everybody like, “Why can't I have Prometheus in CloudWatch?”I mean—so instead of CloudWatch just being a simple monitoring tool, [unintelligible 00:22:54] CloudWatch has become an ecosystem of monitoring tool. So, we got—we're not going to see cloud [unintelligible 00:23:00], or just [unintelligible 00:23:00] log, analytics, metrics, dashboards, no. We're going to see it as an ecosystem where we can plug in other services, and then integrate and work together to give us better performance options, and also different perspectives to the data that is being collected.Corey: What do you think is next, as you take a look across the ecosystem, as far as how people are thinking about monitoring and observability in a cloud context? What are they missing? Where's the next evolution lead?Ewere: Yeah, I think the biggest problem with monitoring, which is part of the introduction part of the book, where I talked about the basic types of monitoring—which is proactive and reactive monitoring—is how do we make sure we know before things happen? [laugh]. And one of the things that can help with that is machine learning. There is a small ecosystem that is not so popular at the moment, which talks about how we can do a lot of machine learning in DevOps monitoring observability. And that means looking at historic data and being able to predict on the basic level.Looking at history, [then are 00:24:06] being able to predict. At the moment, there are very few tools that have models running at the back of the data being collected for monitoring and metrics, which could actually revolutionize monitoring and observability as we see it right now. I mean, even the topic of observability is still new at the moment. It's still very integrated. Observability just came into Cloud, I think, like, two years ago, so it's still being matured.But one thing that has been missing is seeing the value AI can bring into monitoring. I mean, this much [unintelligible 00:24:40] practically tell us, “Hey, by 9 p.m. I'm going to go down. I think your CPU or memory is going down. I think I'm line 14 of your code [laugh] is a problem causing the bug. Please, you need to fix it by 2 p.m. so that by 6 p.m., things can run perfectly.” That is going to revolutionize monitoring. That's going to revolutionize observability and bring a whole new level to how we understand and monitor the systems.Corey: I hope you're right. If you take a look right now, I guess, the schism between monitoring and observability—which I consider to be hipster monitoring, but they get mad when I say that—is there a difference? Is it just new phrasing to describe the same concepts, or is there something really new here?Ewere: In my book, I said, monitoring is looking at it from the outside in, observability is looking at it from the inside out. So, what monitoring does not see under, basically, observability sees. So, they are children of the same mom. That's how I put it. One actually needs the other and both of them cannot be separated from each other.What we've been working with is just understanding the system from the surface. When there's an issue, we go to the aggregated results that come out of the issue. Very basic example: you're in a Java application, and we all know Java is very memory intensive, on the very basic layer. And there's a memory issue. Most times, infrastructure is the first hit with the resultant of that.But the problem is not the infrastructure, it's maybe the code. Maybe garbage collection was not well managed; maybe they have a lot of variables in the code that is not used, and they're just filling up unnecessary memory locations; maybe there's a loop that's not properly managed and properly optimized; maybe there's a resource on objects that has been initialized that has not been closed, which will cause a heap in the memory. So, those are the things observability can help you track. Those are the things that we can help you see. Because observability runs from within the system and send metrics out, while basic monitoring is about understanding what is going on on the surface of the system: memory, CPU, pushing out logs to know what's going on and all that.So, on the basic level, observability helps gives you, kind of, a deeper insight into what monitoring is actually telling you. It's just like the result of what happened. I mean, we are told that the symptoms of COVID is coughing, sneezing, and all that. That's monitoring. [laugh].But before we know that you actually have COVID, we need to go for a test, and that's observability. Telling us what is causing the sneezing, what is causing the coughing, what is causing the nausea, all the symptoms that come out of what monitoring is saying. Monitoring is saying, “You have a cough, you have a runny nose, you're sneezing.” That is monitoring. Observability says, “There is a COVID virus in the bloodstream. We need to fix it.” So, that's how both of them act.Corey: I think that is probably the most concise and clear definition I've ever gotten on the topic. If people want to learn more about what you're up to, how you view about these things—and of course, if they want to buy your book, we will include a link to that in the [show notes 00:27:40]—where can they find you?Ewere: I'm on LinkedIn; I'm very active on LinkedIn, and I also shared the LinkedIn link. I'm very active on Twitter, too. I tweet once in a while, but definitely, when you send me a message on Twitter, I'm also going to be very active.I also write blogs on Medium, I write a couple of blogs on Medium, and that was part of why AWS recognized me as a Hero because I talk a lot about different services, I help with comparing services for you so you can choose better. I also talk about setting basic concepts, too; if you just want to get your foot wet into some stuff and you need something very summarized, not AWS documentation per se, something that you can just look at and know what you need to do with the service, I talk about them also in my blogs. So yeah, those are the two basic places I'm in: LinkedIn and Twitter.Corey: And we will, of course, put links to that in the [show notes 00:28:27]. Thank you so much for taking the time to speak with me. I appreciate it.Ewere: Thanks a lot.Corey: Ewere Diagboya, head of cloud at My Cloud Series. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you hated this podcast, please leave a five-star review on your podcast platform of choice along with a comment telling me how many more dashboards you would like me to build that you will never look at.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Imke Feldmann is among the first few to have recognized the incredible value and potential of this thing called Power Pivot in Excel (which was the precursor to Power BI).  And did she ever run with it, launching quite the successful solo consultancy and training service!  She exemplifies the helpful nature of the data community through her blog, The BIccountant, where she shares her amazing Microsoft BI tool knowledge. Her background is in Finance and Accounting, but you'll quickly realize she knows a great deal more than just Finance and Accounting! Contact Imke: The BIccountant Imke's Twitter References in this Episode: Imke's Github MS Power BI Idea - Customizable Ribbon - Please Upvote :) MS Power BI Idea - Speed Up PQ By Breaking Refresh Chain - Please Upvote :) Episode Timeline: 3:00 - The value of outsourcing certain business functions, Imke's path to Power BI starts with Rob's blog, a multi-dimensional cube discussion breaks out! 19:45 - One of Power BI's strengths is collaboration, Imke LOVES her some Power Query and M and loves DAX not so much 33:45 - Imke has a BRILLIANT idea about how to improve Power Query and some other improvements that we'd like to see in PQ 52:30 -  Rob's VS code experience, how COVID has affected the consulting business, Staying solo vs growing a company and how Imke determines which clients she takes on Episode Transcript: Rob Collie (00:00:00): Hello friends. Today's guest is Imke Feldmann. We've been working for a long time, nearly a year to arrange the schedules to get her on the show, and I'm so glad that we finally managed to do it. For a moment, imagine that it's 2010, 2011, that era. During that timeframe, I felt not quite alone, but a member of a very slowly growing and small community of people who had glimpsed what Power Pivot could do. And for those of you who don't know what Power Pivot is, and that was the version of Power BI, the first version that was embedded only in Excel. And at the time, the way the community grew, we'll use a metaphor for this. Imagine that the community was a map of the world and the map is all dark, but slowly, you'd see these little dim lights lighting up like one over here in the UK, one in the Southwest corner of the United States, very faintly. Rob Collie (00:00:51): And these would be people who were just becoming aware of this thing, this Power Pivot thing, and you'd watch them. They'd sort of show up on the radar, very tentatively at first kind of dipping their toe, and then that light would get brighter, and brighter, and brighter over time, as they really leaned in, and they learned more and more, and they became more adept at it. And this was the way things went for a long time. And then in 2011, out of nowhere in Germany on the map, this light comes on at full intensity, brightly declaring itself as super talented and powerful. And that was what it felt like to come across Imke Feldmann. Rob Collie (00:01:27): Like all of our guests, there's a little bit of that accidental path in her career, but also a tremendous sense of being deliberate. When this stuff crossed her radar, she appreciated it immediately. And I didn't know this until this conversation, but she quit her corporate job in 2013, the same year that I founded P3 as a real company, and became a freelancer. So for eight plus years, she has been a full time Power BI professional. There truly aren't that many people who can say that in the world. Our conversation predictably wandered. At one point, we got pretty deep into the notion of M and Power Query and it's screaming need for more buttons on its ribbon. And Imke has some fantastic ideas on how they should be addressing that. Rob Collie (00:02:14): We also, of course, naturally talked about the differences between remaining a solo freelancer as she has, in contrast to the path that I chose, which is scaling up a consulting practice business. Along the way we reprised the old and completely pointless debate of DAX versus M, I even try to get Tom hooked on M as his new obsession. We'll see how well that goes. Most importantly though, it was just a tremendous pleasure to finally get to talk to Imke at length for the first time after all these years, we literally crossed paths 10 years ago. So it was a conversation 10 years in the making compress down to an hour and change. I hope you enjoy it as much as we did, so let's get into it. Announcer (00:02:56): Ladies and gentlemen, may I have your attention, please? Announcer (00:03:00): This is The Raw Data by P3 Adaptive podcast, with your host Rod Collie, and your cohost Thomas LaRock. Find out what the experts at P3 Adaptive can do for your business. Just go to P3adaptive.com. Raw Data by P3 Adaptive is data with the human element. Rob Collie (00:03:24): Welcome to the show Imke Feldmann. How are you today? Imke Feldmann (00:03:27): Thank you, Rob. Great. It's a great day here over in Germany. Rob Collie (00:03:30): We have been talking about doing this for the better part of a year. So I'm glad that we're landing the guest, Imke is here. I really appreciate you doing this. So why don't we start with the basics. What are you up to these days? What do you do for a living? Imke Feldmann (00:03:48): I have people building great Power BI solutions these days. Rob Collie (00:03:55): Ah, yes. Imke Feldmann (00:03:55): That's how I fill my days. Rob Collie (00:03:58): I hear that that's a good business. Imke Feldmann (00:03:58): Yeah, it is. Rob Collie (00:04:03): So, and your website is? Imke Feldmann (00:04:06): Thebiaccountant.com. Rob Collie (00:04:07): Is that what you are on Twitter as well? Imke Feldmann (00:04:08): Yes. That's also my Twitter handle theBIccountant without an A in the middle. I just replaced the A from accountant with a BI. Rob Collie (00:04:17): There you go. Imke Feldmann (00:04:18): Yeah. Rob Collie (00:04:18): That's right. So that means that I'm going to make a tremendous leap here, wait till you see these powers of observation and deduction. You must have an accounting background? Imke Feldmann (00:04:29): I do, yes. Rob Collie (00:04:30): See you look at that. That's why I make the money. Okay, let's start there, was accounting your first career out of school? Imke Feldmann (00:04:39): Yes. I went to university and studied some economics or business stuff there, they'll know it's translated into English. And then I worked as a business controller. After that, I took over a job to lead a bookkeeping departments or to work with an area where the numbers came from basically. And then after that, I worked as the finance director, where I was responsible for a whole bunch of areas, controlling bookkeeping, IT, HR, and production. So that was quite a job with a broad range of responsibilities. Rob Collie (00:05:18): So you mentioned, kind of slipped IT into that list, right? Imke Feldmann (00:05:23): Yeah. Rob Collie (00:05:23): There's all these things in that list of responsibilities that all seemed they belong together, right? Bookkeeping, accounting, control or finance, IT. We've run into this before, with actually a number of people, that a lot of times the accounting or finance function in a company kind of wins the job of IT by default. Imke Feldmann (00:05:45): Yeah. It seems quite common in Germany, at least I would say. Rob Collie (00:05:48): I get multiple examples, but one that I can absolutely point to is Trevor Hardy from the Canadian Football League, he is in accounting, accounting and finance. And just by default, well, that's close to computers. Imke Feldmann (00:06:00): Yes. Rob Collie (00:06:01): And so it just kind of pulls the IT function in. Now is that true at really large organizations in Germany or is it a mid market thing? Imke Feldmann (00:06:09): No I would say a mid market thing. Rob Collie (00:06:12): That's true here too. So when there isn't an IT org yet it ends up being, oftentimes it falls to the finance and accounting function. Hey, that's familiar. It's kind of funny when you think about it, but it's familiar. And isn't finance itself pretty different from accounting? How much of a leap is that? What was that transition like for you taking over the finance function as well? We tend to talk about these things, at least in the US, is like almost like completely separate functions at times. Imke Feldmann (00:06:43): It depends, but at least it had something to do with my former education, which wasn't the case with IT. So, I mean, of course on a certain management level, you are responsible for things that you're not necessarily familiar with in detail. You just have to manage the people that know the details and do the jobs for you. So that was not too big an issue I must admit. Rob Collie (00:07:10): My first job out of school was Microsoft, an organization of that size, I was hyper specialized in terms of what I did. At this company at P, we are nowhere near that scale, and there's a lot more of that multiple hat wearing. I've definitely been getting used to that over the last decade, the first decade plus of my career, not so much. Imke Feldmann (00:07:31): Yeah. That's interesting because I basically went completely the other way around. I see myself now as working as a technical specialist and as a freelancer, I don't have to manage any employees anymore. Rob Collie (00:07:47): Well, so now you wear all the hats? Imke Feldmann (00:07:49): Yes. In a certain way, yes. Rob Collie (00:07:51): Okay. There's no HR department necessarily, right, so it's just you. But marketing, sales, delivery, everything. Imke Feldmann (00:08:01): Yep, that's true. Yep. And when I first started, I tried to do everything by myself, but the test changed as well. So in the past I started to outsource more things, but to external companies, not internal staff. Rob Collie (00:08:17): So you're talking about outsourcing certain functions in your current business, is that correct? Imke Feldmann (00:08:22): Yes, yes. Rob Collie (00:08:22): So it's interesting, right? Even that comes with tremendous risk when you delegate a certain function to an outside party whose incentives and interests they are never going to be 100% aligned with yours. Even we have been taken for a ride multiple times by third-party consulting firms that we've hired to perform certain functions for us. Imke Feldmann (00:08:46): Oh, no I don't outsource and your services that I directly provide to my clients. Rob Collie (00:08:49): Oh, no, no. Imke Feldmann (00:08:50): No. Rob Collie (00:08:50): No, we don't either. But I'm saying for example, our Salesforce implementation for instance- Imke Feldmann (00:08:56): Okay, mm-hmm (affirmative). Rob Collie (00:08:57): ... Has been a tremendous money sink for us over the years. Where we're at is good, but the ROI on that spend has been pretty poor. It's really easy to throw a bunch of money at that and it just grinds and grinds and grinds. And so this contrast that I'm getting around to is really important because that's not what it's like to be a good Power BI consultant, right? You're not that kind of risk for your clients. But if you go out and hire out some sort of IT related services for example, like Salesforce development, we're exposed to that same sort of drag you out into the deep water and drown you business model, that's not how we operate. I'm pretty sure that's not how you operate either. And so anyway, when you start talking about outsourcing, I just thought, oh, we should probably talk about that. Have you outsourced anything for your own sort of back office? Imke Feldmann (00:09:52): Back office stuff, yeah. My blog, WordPress stuff, or computer stuff in the background. So security [inaudible 00:09:59] the stuff and things like that, things that are not my core, I hire consultants to help me out with things that I would formally Google, spend hours Googling with. Rob Collie (00:10:09): Yes. Imke Feldmann (00:10:10): Now I just hire consultants to do that. Or for example, for Power Automate, this is something that I wanted to learn and I saw the big potential for clients. And there I also did private training basically, or coaching, or how you called it, hire specialists. Rob Collie (00:10:27): To kind of getting you going? Imke Feldmann (00:10:29): Exactly, exactly. Rob Collie (00:10:30): And those things that you've outsourced for your back office, have there been any that felt like what I described you end up deep in the spend and deepen the project going, "What's going on here?" Imke Feldmann (00:10:41): I'm usually looking for freelancers on that. And I made quiet good experiences with it, I must say. Rob Collie (00:10:49): Well done. Well done. All right. So let's rewind a bit, we'll get to the point where you're in charge of the finance department, which of course includes IT. Imke Feldmann (00:10:58): Not necessarily so. I felt quite sad for the guys who I had to manage because I said, "Well, I'm really sorry, but you will hear a lot of questions from me, especially at the beginning of our journey," because I had to learn so much in order to be a good manager for them. So that was quite different situation compared to the management roles in finance that I had before, because there I had the impression that I knew something, but IT was basically blank. Rob Collie (00:11:30): I would imagine that that experience turned out to be very important, the good cross pollination, the exposure to the IT function and sort of like seeing it from their side of the table, how valuable is that turned out to be for your career? Imke Feldmann (00:11:45): I think it was a good learning and really interesting experience for me just to feel comfortable with saying that I have no clue and ask the people how things work and just feel relaxed about not being the expert in a certain area and just be open to ask, to get a general understanding of things. Rob Collie (00:12:09): That's definitely the way to do it, is to be honest and transparent and ask all the questions you need to do. It's easier said than done. I think a lot of people feel the need to bluff in those sorts of situations. And that usually comes back to haunt them, not always. Imke Feldmann (00:12:25): No, that's true. Rob Collie (00:12:27): Some people do get away with it, which is a little sad. So at what point did you discover Power BI? Imke Feldmann (00:12:35): I didn't discover Power BI, I discovered Power Pivot, for your blog of course. Rob Collie (00:12:41): Oh, really? Imke Feldmann (00:12:43): Yes, yes, yes, yes. I think it was in, must be 2011, something like that. Rob Collie (00:12:50): Early, yeah. Imke Feldmann (00:12:51): Yeah. Quite early. When I was building a multidimensional cube with a freelancer for our finance department, then I was just searching a bit what is possible, how we should approach this and things like that. So we started with multi-dimensional cube because that was something where I could find literature about and also find experts who could have me building that. But when doing so, I really liked the whole experience and it was a really excellent project that I liked very much. And so I just searched around in the internet and tried to find out what's going on in that area. And this is where I discovered your blog. Rob Collie (00:13:35): I have no idea. First of all, I had no idea that my old blog was where you first crossed paths with this. Imke Feldmann (00:13:42): I think [inaudible 00:13:43]. Rob Collie (00:13:44): And secondly, I had no idea that it was that early. I mean, I remember when you showed up on the radar, Scott [inaudible 00:13:51] had discovered your blog and said, "Hey, Rob, have you seen this? Have you seen what she is doing? She is amazing." That wasn't 2011, that was a little bit later. I don't remember when but... Imke Feldmann (00:14:06): No, I think we've met first. I think we met on the Mr. XR Forum on some crazy stuff I did there. I cannot even remember what that was, but I started blogging in 2015 and we definitely met before. Rob Collie (00:14:21): That's what it was. It was the forums. And Scott was the one that had stumbled upon what you were doing there and brought my attention to it. I was like, whoa. It was like... Imke Feldmann (00:14:34): That last really some crazy stuff. I think I was moving data models from one Excel file to another or something like that. Some crazy stuff with [inaudible 00:14:43] and so on. Rob Collie (00:14:44): You obviously remember a better than I do. But I just remember being jaw dropped, blown away, impressed, by what you were doing. And the thing is the world of Power Pivot interest at that point in time still seems so small. The community still seems so small that for you to emerge on our radar fully formed, already blowing our minds, that was the first thing we ever heard from you. That was a real outlier because usually the way the curve of awareness went with other members of the community is that like, you'd see something modest from them. And you'd sorta like witnessed their upward trajectory as they developed. Of course, you've continued to improve and learn and all of that since then. But as far as our experience of it, it was you just showed up already at the graduate level, just like where did she come from? So cool. So you said that you enjoyed the multi-dimensional cube project? Imke Feldmann (00:15:43): Mm-hmm (affirmative). Yes. I don't know MDX, but I totally enjoyed the project. So being able to build a reporting solution for my own company, basically then for the company I worked for, and doing it live with a consultant with a freelancer on my hand, discussing how things should look like and just seeing the thing form before my eyes and grow. And this was just such an enjoyable experience for me. Rob Collie (00:16:11): So the thing that's striking about that for me is, there's no doubt that the multi-dimensional product from Microsoft was a valuable product. It did good things. But I never have heard someone say that they really enjoyed the implementation process as a client, right? Imke Feldmann (00:16:31): Okay. Rob Collie (00:16:31): You had a freelancer doing the work. So something you said there really jumped out at me, it was, sort of like doing the project live. So the way that this worked traditionally, at least in the US, is the consultant would interview you about your requirements and write a big long requirements document and then disappear and go build a whole bunch of stuff and come back and show it to you, and it's completely not what anyone expected. It's almost like you're on completely different planets. Obviously, if you'd had that experience, you would not be saying that you enjoyed it. So there had to be something different about the way that you and that freelancer interacted. Do you remember what the workflow was like? Imke Feldmann (00:17:16): What we did is that we often met together and just looked at where we're at and what the next steps should be. And we definitely had specific targets in mind. So there were some reports that I had defined as a target, and around these reports I was aware that we needed something that a proper data model, because I also knew that I wanted to have some sort of a general set up that could be carried from Excel as well. So I knew about cube functions, and I knew that on one hand I needed these reports that had formerly been within our ERP system. Also, I wanted them to be in a separate solution that was under my control and independent from the ERP system. And on the other hand, I wanted some more. So I wanted the flexibility to be able to vary this data and for certain other purposes in the controlling department as well. So basically being able to do ad hoc analysis on it. Imke Feldmann (00:18:23): And we met often and I showed a certain interest in how the table logic was created. So I knew that the MDX was over my head at the time, but I showed a very strong interest in which table are created, how they relate to each other, and that was quite unusual. At least this is what the [inaudible 00:18:47] the freelancer told me. Rob Collie (00:18:49): I bet. Imke Feldmann (00:18:50): He said that he doesn't see that very often that clients showed this sort of interest. Rob Collie (00:18:56): Did he say, "Yeah. You really seem to be having fun with this. Most of my clients don't enjoy this." You said that you met very often, so were there times where he was writing MDX while you were in the room? Imke Feldmann (00:19:10): Sometimes yes, because I said, "Well, can we switch this a bit or make some changes?" And sometimes he said, "Well, I can try adjust now." Because he came over for one day or half a day, and then we spoke things through and defined further things. And if we were finishing early, he would just stay and do some coding there. But apart from that, he would work from home and do the big stuff. Rob Collie (00:19:37): OLAP originally it stands for online analytical processing, where online meant not batch, right? It meant you could ask a question and get the answer while you were still sitting there. Imke Feldmann (00:19:51): Okay. Oh, really? Rob Collie (00:19:53): That's what online meant. Imke Feldmann (00:19:54): It's interesting. Rob Collie (00:19:56): It basically meant almost like real time. It's a cousin of real time, that's what online meant at that point, as opposed to offline where you write a query and submit it and come back next week right? So that's what the online and OLAP comes from. Imke Feldmann (00:20:12): Oh, interesting. Rob Collie (00:20:13): We would pick a different terminology of OLAP were it invented today. So something interesting about, it sounds like your experience, and I did not anticipate drilling into your experience with multi-dimensional on this conversation, but I think it's really important is that at least some portion of that project that you sponsored and implemented with the freelancer, at least some portion of the work was similarly performed online. Meaning the two of you were sort of in real time communication as things evolved. And the old model and the vast majority of multidimensional solutions that have ever been built in the world, the MDX powered solutions, were built and an offline model, where the majority of the communication supposedly takes place in the form of a requirements document. Rob Collie (00:21:05): And that was a deeply, deeply, deeply flawed approach to the problem, that just doesn't actually work. So I guess it's not surprising to me that the one time I've ever heard someone say they really enjoyed that multi-dimensional project, that at least a portion of that multidimensional project was sort of almost like real-time collaboratively performed rather than completely asynchronous, right? I guess we want to be really geeky, we could say it was a synchronous model of communication as opposed to an asynchronous one. And Power BI really facilitates that kind of interaction. Imke Feldmann (00:21:41): Absolutely. Rob Collie (00:21:42): The reason why the MDX multi-dimensional model worked the way it did, or there was two reasons, one is a legitimate one on one of them is more cynical. So the legitimate reason is, is that it required reprocessing of the cube for every change, it's just too slow, right? The stakeholder, the business stakeholder doesn't typically have the time or the patience to sit there while the code's being written, because it's so long between even just implementing a formula change sometimes would be, well, we need to wait an hour. And so the attention span of the business person can't be held for good reason there, right? And so that sort of drove it into an asynchronous model. Rob Collie (00:22:23): The other reason is, is that that is asynchronous model turned out to be a really good business model for the consultants, because the fact that it didn't work meant that every project lasted forever. And so that's the cynical reason. But Power BI is not long delays. You change the measure formula, or you add an extra relationship, or heck even bringing in a new table, just a brand new table, bring it in, it wasn't even in the model, now it's in the model. End to end that can sometimes be measured in minutes or even seconds. And so you can retain engaged collaborative interest. Now it's not like you're always doing that, right? There's still room for offline asynchronous work in our business, but really critical portions of it can be performed the other way. And I think that makes a huge difference. Imke Feldmann (00:23:13): Yep. And that's what I like about it. So it's so great to be able to have, as a consultant, to perform really relatively large tasks without any further involvement of other people. Which, I mean, honestly, I don't call myself a team worker, not because I don't love other people also, but teamwork means you have to communicate with other people, make sure that they know what you're working on. So there are so many interfaces that have to be maintained if you're working with other people. And so I really laugh the way I work currently being able to deliver full solutions as a one woman show consultant. That is really a pleasure for me. That's really my preferred way of work, I must say. Because I can really focus on the things that have to be done and I'm able to deliver value in a relatively short time for the clients. Rob Collie (00:24:14): That's a really interesting concept. There are certain kinds of problems in which collaboration, a team collaboration is absolutely necessary. The magic of collaboration sometimes can beat problems that no individual could ever beat. At the same time though, there's this other dynamic, right, where having a team working on a problem is actually a real liability because the communication complexity between the people becomes the majority of the work. Here's a really hyper simplified example. There used to be sort of a three-person committee, if you will, that was running our company P3, me and two other people. Imke Feldmann (00:24:57): Mm-hmm (affirmative). Rob Collie (00:24:58): And so all leadership decisions were essentially handled at that level. Well, things change, people move on, right? And so we went from a three person committee to a two person committee. We didn't anticipate the two of us who stayed, right? We did not anticipate how much simpler that was going to make things. We thought, just do the math, right, it's going to be like, well, it's one less person to get on the same page. So it's going to be a one-third reduction in complexity. It was actually double that because we went from having three pairs of communication, right, the triangle has three sides, to a line that only has one side, right? So there was only one linkage that needed to be maintained as opposed to three geometrically, combinatorially, whatever we're going to say, right? It just became- Imke Feldmann (00:25:45): Exponential. Rob Collie (00:25:45): ... Exponetially simpler. And so for problems that can be soloed, you have this amazing savings in efficiency, in clarity, even, right? Imke Feldmann (00:25:59): Yup. Rob Collie (00:25:59): There's just so many advantages when you can execute as one person, then there's the other examples like our company at our size now, even ignoring the number of consultants that we need to do our business, just the back office alone, we need the difference in skills. We need the difference in talents and interests and everything. We simply could not exist without that kind of collaboration. However, when our consultants were working with a client, usually it's essentially a one-on-one type of thing, right? We don't typically put teams of consultants on the same project. We might have multiple consultants working for the same client and they might be building something that's somehow integrated, but it's still very similar, I think to your model, when you actually watch sort of the work being done, there's this amazing savings and complexities. Imke Feldmann (00:26:50): Yup, that's true. Of course I have a network in the background. So when big problems arise where I need brain input, of course, I have a network, but it's not a former company. Rob Collie (00:27:02): And that's how we work too, right? We have all kinds of internal Slack channels. For some reason we adopted Slack years ago before Teams was really a thing. So Slack is sort of like our internal social network. There's a lot of discussion of problems, and solutions, and a lot of knowledge sharing, and people helping each other out behind the scenes in that same way. Again, we do bring multiple consultants into particularly large projects, but it's not like there's three people working together on the same formula. In Power BI, the things that you do in ETL, the things that you do in power query are intimately interrelated with the data model and the decks that you need to create. And imagine parceling that out to three different people. You have one formula writer, one data modeler, one ETL specialist, you would never ever get anywhere in that kind of approach. Imke Feldmann (00:28:00): Not necessarily. I mean, the tax people are the person responsible for the data model. He could write down his requirements. He could define the tables basically. And then someone could try to get the data from the sources. But of course, then you get some feedback that the data isn't there or that the model has to be shaped in a different way. So it has two sides to it. But that's interesting to see that you have the same experience, that Power BI models or solutions of a certain size that can very well be handled by one person alone. And that really brings speed, and flexibility, and agility to the whole development process I think. Rob Collie (00:28:41): You communicate with yourself at what's above giga? Peta, petabit? you communicate with yourself at petabit speed and you communicate with others through a noisy 2,400 baud modem that's constantly breaking up. It's amazing what that can do for you sometimes. So there comes a point in your journey where you decide to go freelance. Imke Feldmann (00:29:07): Yup. Rob Collie (00:29:08): That's a courageous leap. When did that happen and what led you to that conclusion? Imke Feldmann (00:29:13): I made the decision in 2012 already to do that. Rob Collie (00:29:19): Wow. Imke Feldmann (00:29:20): And I just saw the light. I just saw the light in Power Pivot and then Power Query came along and I saw what Microsoft was after. And as I said, I enjoyed the building of the cube, getting my hands dirty, reading about the technologies behind it and so on. And this was what I felt passionate about. And I also had the idea that I needed some break from company politics. And so I just thought, well, I give it a try. And if it doesn't work, I can find a job after that or find a company where I work for at any time after that. So I just tried it and it worked. Rob Collie (00:30:05): So you decided in 2012, did you make the break in 2012 as well? Imke Feldmann (00:30:12): I prepared it, and then I just in 2013, I started solo. Rob Collie (00:30:18): Okay. 2013 is also when we formally formed our company. For 2010-2013, it was a blog. I had other jobs. I had other clients essentially, but I wasn't really hanging out the shingle so to speak, as you know, we're not an actual business really until 2013. And I guess it's not much accident that we both kind of did the same thing about the same time, it's that demand was finally sufficient I think in 2013 to support going solo. In 2012, there weren't enough clients to even support one consultant. And so, oh, that's great. And I think you really liked Power Query too, does M speak to you? Imke Feldmann (00:31:02): Yes. Yes. Yeah. Rob Collie (00:31:03): It does, doesn't it? Imke Feldmann (00:31:04): I really prefer Power Query or M over DAX, I must admit. It has been much more liable to me than DAX. Rob Collie (00:31:15): Oh, and I liked you so much before you said that. I'm team DAX all the way. Imke Feldmann (00:31:23): I know. I know. I know. I mean, of course I love to use DAX as well, but I really feel very, very strong about Power Query. And I mean, I had such a great journey with it. I mean, it was really [inaudible 00:31:35] work for me personally, that I did with it. And it was just a great journey to understand how things work. I mean, this has been the first coding language for me that I really learned. And it was just a great journey to learn all the things and starting to blog about it. And of course, I started basically helping people in the forum, that's where I basically built my knowledge about it, solving other people's problems. And this was just a great journey. And Polar Query has always been good to me than DAX. Rob Collie (00:32:14): This is really cool, right? So you fell in love with Power Pivot, so DAX and data model, right? There was no Power Query. Imke Feldmann (00:32:21): Mm-hmm (affirmative)-, that's true. Rob Collie (00:32:23): Okay. And because we had no Power Query, there were many, many, many things you couldn't do in Power Pivot unless your data source was a database. Imke Feldmann (00:32:30): Yup. Rob Collie (00:32:31): Because you needed views created that gave you the right shape tables, right? If your original data source didn't have a lookup table, a dimension table, you had to make one. And how are you going to make one without Power Query? It gets crazy, right? At least unbelievable. So try to mentally travel back for a moment to the point in time where you're willing to, and not just, it doesn't sound like you were just willing to, you were eager to go solo to become a freelancer, right, with just DAX and data modeling. And then after that, this thing comes along that you light up when you talk about. You didn't have this thing that you love, but you were already in, that doesn't happen very often. Imke Feldmann (00:33:18): It could be that loved DAX at the beginning, but it just started to disappoint me at sometimes. Rob Collie (00:33:29): Oh, okay. Thomas LaRock (00:33:29): It disappoints everyone. Rob Collie (00:33:29): I'm just devastated. Imke Feldmann (00:33:35): No, I mean, it's amazing what DAX can do, but I mean, we all know it looks easy at the beginning, but then you can really get trapped in certain situations. Rob Collie (00:33:46): Yeah. I described these two things is like the length and width of a rectangle, Power Query and DAX. Take your pick, which one's the width, which one's the length? I don't care. And then we ask which one is more responsible for the area of the rectangle, right? Neither. You can double the length of either of them and it doubles the area of the rectangle. So it's really ironic that I'm so sort of firmly on team DAX for a number of reasons. Number one, is that I'm really not actually that good at it compared to the people who've come along since. Like my book, for instance, I think, I look at it as this is the 100 and maybe the 200 level course at university, maybe the first in the second course, maybe, but it's definitely not the third course. The thing that you take in your third or fourth year of university, that's not covered in my book in terms of DAX. Rob Collie (00:34:44): And basically every one of the consultants at our company is better at DAX than I am. And that's great. That's really good. And the other thing that's ironic about my love of DAX over M, is if these two were in conflict, which they aren't. Imke Feldmann (00:35:00): No they are. Rob Collie (00:35:02): Is that I actually was trying for years to get a Power Query like project started on the Excel team. I knew how much time was being chewed up in the world just transforming data, not analyzing it even, just getting things ready for analysis. It's just ungodly amounts of time. And so I was obsessed with end-user ETL. When I was on the Excel team, it was like a running joke, someone would mention in a meeting, "Well, that's kind of like ETL," and other people would go, "Oh no, no, don't say that in front of Rob, he's going to get started and he won't shut up about it for the next 30 minutes." On the podcast with the Power Query team, I told them I'm really glad that no one ever agreed to fund my project on the Excel team because now that I see what Power Query is like I grossly underestimated how much work needed to go into something like that. And I'm glad that Microsoft isn't saddled with some old and completely inadequate solution to the Power Query space, because now that I've seen what the real thing looks like, I'm like, "Oh my gosh, we would've never been able to pull that off." Rob Collie (00:36:14): So the thing that I was most obsessed with is the thing that now that it's actually been built, for some reason, I just find M to be, I don't know, there's like a reverse gravity there that pushes me away. Imke Feldmann (00:36:26): What I actually would like to see is that there's less need to use M in the Power Query product. So first, the only thing I was dreaming about was finally to have a function library that can easily be shipped from then, or that you can download from internet or wherever, where you can use additional functions in your M code. So this was the first thing that I was really passionate about and thought that we should have such a thing in Power Query to be able to make more cool things, or group steps together. But now what I really think we should actually have and see in Power Query is the ability to build our own ribbons and to the query editor. Rob Collie (00:37:13): Yes. Imke Feldmann (00:37:13): Like we have in an Excel. So this is something that in my eyes would really bring a big push to the product and actually would make so much sense for the people who start using these products. I mean the whole Power platform can have so many benefits for finance department, all departments, but I mean, I'm passionate about finance departments. But have you counted how many low-code languages are in there, if you include Power Apps and Power Automate and all these things? Rob Collie (00:37:50): Low-code. Imke Feldmann (00:37:50): And honestly, in order to come up with any solution that makes sense in a business environment, I would say in all of these solutions, there is no way around the code at the end. I mean, you get quite far with clicky, clicky, but I haven't seen solutions where you get around the languages. And now imagine the typical finance people who really they know the Excel formulas and some of them might know VBA as well. And now their server uses new low-code, no-code word, and just get your head around about five or six new languages that you all have to know and learn in order to get something useful and so on. So I think that's just not feasible for people who have real jobs in the business to learn all that. Rob Collie (00:38:42): Well, that's what you're here for, right? That's what your business is for and that's what P3 is for. Imke Feldmann (00:38:48): We get them started and the products are great. And if there are people in the companies who have a drive to learn things and take the time they get their heads around it, but it could be easier. It could be easier with things like that, where we could provide additional user interfaces and just make it even easier for people to build great solutions for them or adapt solutions that consultants had build initially, but to maintain them by themselves and make adjustments to them if needed. Rob Collie (00:39:19): So [inaudible 00:39:20] has an old joke where he says, when he's doing a presentation or something, he says, "That's a good question. And I define good question as a question I know the answer to, right." And then he says, "But then a great question is a question that is covered by the very next slide." So there's a similar parallel joke to make here, which is that, that idea you just talked about with the ribbons and everything, right? So if I said, it's a smart idea, what I would mean is, again, this is a joke, right? I would mean that that's an idea that I agree with and have kind of already had. But if I say it's a brilliant idea- Imke Feldmann (00:39:55): Okay. Rob Collie (00:39:56): ... Then it's an even better version of an idea that I've already had that has never occurred to me. Your idea is a brilliant idea. Imke Feldmann (00:40:02): Okay. Rob Collie (00:40:06): It goes beyond. So I have been advocating privately behind the scenes with the Power Query team forever telling them that they need about three or four more ribbon tabs. There's just way too many commonly encountered problems for which you can imagine there being a button for, and there's no button. Imke Feldmann (00:40:28): Exactly. Rob Collie (00:40:29): And it's like, I don't understand. I used to be on teams like that, but I don't understand why they haven't gotten to this. Because it seems so low hanging fruit. They've already built the engine, they've built the language, right? The language can already handle this, but you actually had two brilliant ideas in there that had never occurred to me. First of all, I'm used to the idea that the community can't contribute libraries of functions, they can't do that for DAX. Imke Feldmann (00:40:57): Mm-hmm (affirmative). Rob Collie (00:40:58): That's not even like engineering possible for DAX. And the reason for it is, is that the DAX engine is so heavily optimized in so many ways that there'd be no way to plug in some new function that's unpredictable in terms of what it needs to do. All of these things, they're all inherently interrelated and they make changes in the storage and the query engine to make this function work better and vice versa, because it has to take advantage of the index compression scheme and all of that kind of stuff. It's actually not possible, is the wrong word, but it's actually orders of magnitude more difficult, if not impossible to allow DAX to have UDF, user-defined function type of feature. Rob Collie (00:41:42): I don't think Power Query is like that though. Maybe naively, because again, I'm not on the internals team on the Power Query side. But it does seem like a UDF capability is at least much more feasible- Imke Feldmann (00:41:53): Absolutely. Rob Collie (00:41:54): ... For Power Query, which does execute row by row essentially. Other languages have this, right? One of the reasons that R is so popular is not that R is so awesome, is that R has tremendous libraries of commonly solved problems that you can just go grab off the internet or off the shelf and plug into your solution. Imke Feldmann (00:42:14): I have my own library I've created. You can go to my GitHub and you'll see 50, 60 custom M functions. You can package them in a record and [inaudible 00:42:24] them as a library and your M code, or you could even connect live to them and run them with an execute statement. But this is too difficult, although it's just a couple of clicks, but it's too difficult or at least intimidating for the beginners, who really Power Query beginners who start with the products, I think there's so much potential to make their life easier. And that's not through some coding stuff, or I know this function, I know that function, that's really can only come in my eyes through user interface with buttons. Rob Collie (00:42:59): Yeah, I agree. And just as importantly for me, is that I might actually come around and be like, just as much team Power Query as team DAX. Honestly, my frustration is just the M language and just my total lack of desire to learn it. [crosstalk 00:43:16]. It is what it really comes down to. It's not about M, it's not about Power Query, it's about me. Whereas again, I know the need that it fills is massively important. So it's not that I think it's a bad mission, I think it's like the mission in a lot of ways. I was obsessed with it long before I ever crossed paths with business intelligence, I was obsessed with data transformation, end user data transformation. It's just a problem that's about as ubiquitous as it gets. So let's make it happen. We agree, the two of us, that's it, right? It's like we need to go provide a unified front. Imke Feldmann (00:43:52): I think that that's an idea in the idea forum, I might send the link that you can maybe post. Rob Collie (00:43:56): We want that thing up, voted to the moon. I'll even go figure out what my sign in is on the ideas side. Imke Feldmann (00:44:08): Oh, good luck with it. Rob Collie (00:44:09): Which is absolutely impossible. I have no idea which of the 14 counts. And then I'll try to create a new one and it'll go, "Nah, you're not allowed to. We know it's you, but we won't tell you who it is, what your email address is." So I completely agree. So there's so many problems. I always struggle to produce the list. It's like I need to be writing down the list of things that are crucial, but here's an example. Remove duplicates, but control which duplicate you keep. That's a problem that can't be solved in the GUI today. Imke Feldmann (00:44:48): And you need the intimidating type of buffer that you have to write by hand around it, which is just pain. Rob Collie (00:44:56): Remove dups and don't care which one you keep. Okay, fine. That's a great simple button. There should be an advanced section that allows you to specify, oh, but before you keep the dups, sort by this column or sort in the following manner. Imke Feldmann (00:45:10): Exactly. Rob Collie (00:45:10): And then keep the first one of each group. It's easy for us to say outside the team, but apparently that is a, we just make a joke, right? That's apparently a Manhattan project level of software to add that extra button. Anyway, we'll get that. Thomas LaRock (00:45:27): That doesn't make sense to me though. I'm fascinated by all of your conversation and you guys are a hundred miles away from me in a lot of this stuff, but I could listen to it all day. But no, the fact that Excel can't do the remove duplicates, except for like the first of each one of something, that's a simple group by. In my head, I sit there and go that's easily solvable because Excel and DAX does such great stuff that I would never want to do in TSQL, how the hell do we stumble across a thing that's been solved by straight up SQL language that somehow can't get into an Excel? Rob Collie (00:46:01): Well, let's explain the problem very clearly and see if we're on the same page as to what the problem is, but either way it'll be valuable. So let's say you have a whole bunch of orders, a table full of orders. That is a really wide Franken table. It's got things like customer ID, customer address, customer phone number, but also what product they ordered, and how much of it, and how much it cost. Okay, and a date, a date of the order. All right. And you've been given this table because the people that are responsible for this system, they think that what you want is a report and not a data source. And this is incredibly common. Okay. So you need to extract a customer's dimension or lookup table out of this. You need to create a customer's table so that you can build a good star schema model. Okay. And Power Query is right there to help you. Power Query will help you invent a customer's look up table where one wasn't provided, and that's awesome. Rob Collie (00:46:58): Okay. So you say, okay, see customer ID this column. I want to remove duplicates based on that column. Okay, great. But now it's just that the order that the data came in from the report file or the database or whatever that will determine which duplicate is kept. What you really want to do of course is take the most recent customer order of each customer ID because they've probably moved. They may have changed phone numbers, whatever, right? You want their most recent contact information. You don't want their contact information for 15 years ago. And the M language allows you to solve this problem essentially sort by date, and then keep the most recent, but only if you get into the code manually, and as Imke points out, it's not even if you go into the code, the things that you would want to do, if you do a sort, you can add a sort step to the Power Query with the buttons, with the GUI, and then you do the remove duplicates and it ignores the source. Imke Feldmann (00:47:59): Yes. Rob Collie (00:48:02): The GUI almost tries to tell you that it's impossible, but if you know about table dot buffer. Imke Feldmann (00:48:07): So the question is why do we have a sort command in Power Query when it doesn't give the sort order? I mean, that is the question to ask. But that's how it is. Rob Collie (00:48:16): It sorts the results. It sorts the results, it just doesn't sort for the intermediate steps. Imke Feldmann (00:48:20): Why? No, that's quite technical. But would just be great if such a common task could be done with buttons that is reliable at the end. I fully agree. Rob Collie (00:48:35): So Tom, I think this one's really just an example of, again, I truly think that M and Power Query, just like DAX and data modeling, the Power BI data modeling, both of these things belong in the software hall of fame of all time. It is amazing, Power Query, M, is just ridiculously amazing. It's one of the best things ever invented. Remember this is someone who's associated with being a critic of it. Imke Feldmann (00:49:04): Yeah, you're making progress, it's great to see. Rob Collie (00:49:07): And yet I'm telling you that it's one of the top five things ever invented probably. And I think there's a certain tendency when you've done something that amazing to lose track of the last mile. I think it's more of a human thing. Imke Feldmann (00:49:19): Maybe, but I mean, what I see is that they are investing quite a lot in data flows, which makes a lot of sense as well in my eyes. Rob Collie (00:49:27): All that really does though, as far as you and I are concerned, Imke, is it makes it even more important that they solve this problem. Because it's now exposed in two different usage scenarios. Imke Feldmann (00:49:37): Yeah, you're right. Rob Collie (00:49:39): And I want my data flow to be able to control which duplicates are kept too. So that's what I'm saying. There's all these big sort of infrastructural technical challenges that do tend to draw resources. And it's not a neglect thing. Imke Feldmann (00:49:54): No, no. Rob Collie (00:49:54): It isn't like a willful failure or anything like that, I don't want to paint that kind of negative of a picture. Imke Feldmann (00:49:59): No. Rob Collie (00:50:00): It's just that out here in reality, the inability to do, even if we just identified the top 10 things like this, addressing those top 10 things with GUI, with buttons, what have I think in the world, maybe even a bigger impact than the entire data flows project, right? Because you would expand the footprint of human beings that are advocates of this stuff and then you go build data flows. You don't have to think of it as either or, right? They should do both. It's just that I think it's hard to appreciate the impact of those 10 buttons when you're on the software team. It's easier to appreciate the impact of data flows, which is massive. I don't mean to denigrate that. I think it's crazy good. It's just that this other thing is of a similar magnitude in terms of benefit, but it's harder to appreciate when you're on the software team. It's easier to appreciate when you're out here in the trenches, living it every single day. And every time I run into a problem like this, I have to put my hand up and say to my own team, I have to say, " Help." Thomas LaRock (00:51:02): So a casual observation I have is that you wish for there to exist one tool that will handle all of your data janitorial needs. And that tool doesn't necessarily exist because life is dirty, so is your data and you're never going to anticipate everything possible. Now, should that sorting functionality exist in that duplicates, the scenario gave me? Yeah, probably. But there's always going to be something next. And that's why I go to you and I say, the thing that you've described to me is you need your data to be tidy so that it can be consumed and used by a lot of these features that we've talked about today. And in order to get to tidy data, there's no necessarily one tool. Thomas LaRock (00:51:48): You're a big fan of the ETL, Rob. You know that, hey, maybe I need to take the source data and run it through some Python scripts, or some M, or something first before it goes to this next thing. And that's the reality that we really have. What you're wishing for is the one tool, the one button to rule it all. And that's going to take a while before that ever comes around. Rob Collie (00:52:09): The thing is though, is that M is ridiculously complete. Imke Feldmann (00:52:14): Yeah. Rob Collie (00:52:15): You can do anything with it. And it's a language that's optimized for data transformation. So I know you can do anything with C++ too, right? But this is a data crunching, data transformation, specialized language that is really complete. And its UI is woefully under serving the capabilities of the engine. And so I suppose we could imagine and deliberately design a data transformation scenario that maybe M couldn't do it. Imke Feldmann (00:52:45): No. Rob Collie (00:52:46): I think that'd be a very difficult challenge considering how good M is. Imke Feldmann (00:52:49): I think in terms of logic, M can do anything, but in terms of performance, there is some room for improvements. So because there's a streaming semantic running in the background, and as long as the stream runs through all the steps, if you have complex queries, this can really slow things down. And currently there is no button or command in the M language to cut the stream and say, well, stop it here and buffer what you have calculated until here, and then continue from there. So if you have really complex stuff that would benefit from an intermediate buffer, then you can store that in an Azure blob or CSV, or whatever. Specifically if you're working with data flows, you can create some automatic processes that would enable this kind of buffering. Imke Feldmann (00:53:45): And then you will see that the speed of the whole process that can really increase dramatically because in some situations, the speed in M drops exponentially. And these are occasions where a buffer would really helped things, but we don't have it yet in the engine of Power Query. So this was what really be something else that would be fairly beneficial if we wouldn't have to make these work-arounds through things. Rob Collie (00:54:14): Tom, that just occurred to me, I can't believe this is the first time that this thought has crossed my mind. But I think that you might fall into an abyss of love with M. Thomas LaRock (00:54:28): Well, I'm a huge James Bond fan, but... Rob Collie (00:54:30): Oh, no. I think you would really, really just dig it. Thomas LaRock (00:54:38): I don't think I have time to take on a new relationship at this point. I'm still with Python and R, so I mean, I don't know. I'm not going to disagree, I'm just, please don't start a new addiction for me. Rob Collie (00:54:51): Think of the content though, that you could produce over time. The M versus SQL versus Python treatises. Thomas LaRock (00:54:59): Cookbook. Rob Collie (00:55:00): You were made for this mission Tom. Thomas LaRock (00:55:03): Okay. So we'll have to talk later about it. You can sweet talk me. You know I've let you sweet talk me into any [inaudible 00:55:08]. Rob Collie (00:55:08): That's right, that's right. Come on, Tom. Get into M, you know that thing that I have nothing but praise for, that I just love to death, you need to do that. Thomas LaRock (00:55:18): For you. That's what you want to do, is you want to learn it but [inaudible 00:55:21] through me. Rob Collie (00:55:22): Oh, that wouldn't work. I would be, "Oh yeah, well this is still M." Thomas LaRock (00:55:29): You're going to be like, "Tom, where's your latest blog post on M so I can read it and hate upon it even more?" Rob Collie (00:55:37): No, I would not read. Just as the first step. Thomas LaRock (00:55:42): I'm going to read it, but not leave a comment about how much I hate it. Rob Collie (00:55:45): Let's go back to talking about how we did a bunch of big fat Fisher-Price buttons for me to mash my thumbs in the UI. That's what I need. Thomas LaRock (00:55:54): You know what? I'll do that. I'll open up VS code and I'll just build this one big button, it's Rob's button. Rob Collie (00:56:00): Hey, you won't believe this, but I recently installed VS code. Thomas LaRock (00:56:03): I don't believe it, why? Rob Collie (00:56:05): Well, because I needed to edit, not even write, because I'm not capable of it. I needed to edit an interface, add on customization for World of Warcraft. And the only purpose of this World of Warcraft add on interface modification was to allow me to drop snarky comments into a particular channel of the conversation based on the button that I press. I needed a menu of snarky comments to drop at particular points in time. It's hard to type them out all the time, right? So it's just like, now here we go. I dropped one of those. I dropped one of those. Thomas LaRock (00:56:37): We got to get you a real job or something. You got way too much time on your hands. Rob Collie (00:56:42): That was my number one contribution to the World of Warcraft Guild. For a couple of months, there was the snarky rogue chat. Thomas LaRock (00:56:48): You know that is on brand. Rob Collie (00:56:56): It prefixed every comment in the chat with a prefix, you came from rogue chat 9,000. So that people who aren't on the joke were like, "Why is this guy, he's usually very quiet, become so obnoxious. Look at the things he's saying." Anyway. So VS code. And that also involved GitHub. Because my friend who wrote the stub, the shell of this add on for me is a vice president at GitHub. So of course he puts the code in GitHub and points me to it and then points me to VS code, and I'm like, "Oh, you're making me work now? Okay. But you wrote the shell for me, so okay. All right. I'll play ball." So it doesn't sound like you regret your decision to go solo. Imke Feldmann (00:57:40): Absolutely. Rob Collie (00:57:41): You're not looking to go back to corporate life. Imke Feldmann (00:57:43): Absolutely not. Rob Collie (00:57:44): Not missing that. So what can you tell us about the last year or two? What impact, if any, did COVID have on your business? Imke Feldmann (00:57:52): Business has grown especially the last year. So people needed more reports than ever and solutions. So it really, I don't know whether it was COVID effect or just the fact that Power BI is growing and growing. Rob Collie (00:58:07): I'm sure it's both. So the dynamic we saw during 2020. So 2020 would be the, if you're going to have a year that was negatively impacted by COVID, it would have been 2020. And what we saw in 2020 was that we were definitely not acquiring new clients. We weren't making new relationships at nearly the rate we had been people weren't taking risks on meeting a new BI firm. That wasn't something that there was as much appetite for as there had been. However, amongst the clients where we already had a good relationship, we'd already been working with them for a while, their needs for data work expanded as a result of COVID because it did, it created all kinds of new problems and it invalidated so many existing blueprints of tribal knowledge of how we run the business. When reality changes, you need new maps, you need new campuses. Rob Collie (00:59:04): And so on net, we ended up our overall business still grew modestly over the course of 2020, year over year compared to 2019. But then when the new clients started to become viable again, people started looking, we're interested in making new relationships, 2021 has been a very, very strong year of growth, not moderate, really kind of crazy. How do you keep up with increased demand as a one person shop? Imke Feldmann (00:59:35): Saying no. Rob Collie (00:59:36): You have to make your peace with saying no. At one point in my history, I faced sort of the same thing and I decided not to say no, and instead decided to grow the company. That brought an enormous amount of risk and stress- Imke Feldmann (00:59:55): I can imagine. Rob Collie (00:59:55): ... Into my life that I did not anticipate its magnitude. I'm sure I anticipated it, but I didn't anticipate the magnitude of it. I'm very grateful that I'd made that decision though, because where we are today is incredible. That's a rocky transition. So today everything runs like clockwork basically. We have a lot of growth ahead of us that seems almost like it's just going to happen, we're just going to keep growing for a long time. But we had to set the table we had to build our organism as a company into a very different form than what it had been when it was just me. And that molting process it was very painful. I don't pretend that the scaling decision is the right decision, it's very much a personal one. I've certainly lived that. If the version of me that made the decision to scale the company knew everything that was coming, it would have been a much harder decision to make. You kind of have to have a little bit of naive optimism even to make that leap. Imke Feldmann (01:00:57): I can imagine that once you get these things figured out and with the dynamic that the product has, that has a good chance to get it going into a very successful business, I believe. Rob Collie (01:01:10): Well, with your profile and with the growing demand for these sorts of services, the percentage of no that you have to say is just going to keep going up. Imke Feldmann (01:01:20): Yeah. But I made my decision and that's just fine. Rob Collie (01:01:25): I'm very supportive of that decision. I don't have any criticism of it, again, especially knowing what I know now. But if there's going to be come a point where you're going to be saying yes 1% of the time, and the answer to that is ultimately, well, you just raise your rates, which is also very difficult to do. In the end, it's almost like an auction for your services. You need to run yourself like Google. There's a 40 hour block of Imke time coming up for availability. We'll just put it on eBay. Imke Feldmann (01:01:59): I mean, it's just nice to be able to choose with whom you work with. That's just nice. And I earned enough money, so that's fine. So I'm happy with that. Rob Collie (01:02:12): How do you choose who you work with? Is it mostly based on industry? Is it mostly based on job function that you're helping? Or is it more about the specific people? There's all kinds of things that could... Let's say if I came to your website today, I filled out your contact form, what are the things that I could say in that contact for a message that would lead you to say no, versus leads you to say maybe? Imke Feldmann (01:02:37): What I really like to do is to work with finance directors. So basically not people exactly like me, but I like to see that the managers approached me and they have an interest in the product itself and also therefore an interest to push it into their departments. So this is for me, a very, very good starting point because it's an area I'm familiar with. I know that there's enough critical support to get the decisions that have to be made and maybe also push IT to help with certain things. This is really one of my favorite set ups, I would say. Rob Collie (01:03:19): Yeah, we do a lot of work with finance departments as well. How long does sort of your average relationship run with a client? How long do you end up working with the same organization on average? Imke Feldmann (01:03:31): That's hard to say, that's really completely different. It can be the initial five days kickoff where we set up a PNL statement connect all the finance data and they go along with that. And basically, never hear again, or just occasionally hear again, "Can you help me with this problem or that problem?" And it could also be going on for years, basically with breaks in between of course, but some customers, they come every now and then when they want to expand things. Now I have a customer that I'm working on some hours or even days ever week since over a year by now. Rob Collie (01:04:15): That sounds similar to my experience as a freelancer, when it was just me, less similar to our business today, a little bit less. I mean, I think it's still more similar than not. It's just that the dial has moved a little bit. Imke Feldmann (01:04:32): So how long are your engagements then, usually? Rob Collie (01:04:35): Most of our engagements are, if we start out doing kind of that kickoff you're talking about, we started like a project with people, that tends to not be the end. We don't typically have people just immediately vanish after that because that's usually the point at which, I mean, they've got something working already, very often after the first week or so of working with a client, they've usually got some really amazing things built already at that point. But at the same time, that's really just at the beginning of the appetite. Usually there are things that are

About TimTim's tech career spans over 20 years through various sectors. Tim's initial journey into tech started as a US Marine. Later, he left government contracting for the private sector, working both in large corporate environments and in small startups. While working in the private sector, he honed his skills in systems administration and operations for largeUnix-based datastores.Today, Tim leverages his years in operations, DevOps, and Site Reliability Engineering to advise and consult with clients in his current role. Tim is also a father of five children, as well as a competitive Brazilian Jiu-Jitsu practitioner. Currently, he is the reigning American National and 3-time Pan American Brazilian Jiu-Jitsu champion in his division.Links: Twitter: https://twitter.com/elchefe The Duckbill Group: https://duckbillgroup.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate: is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards, while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other, which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at Honeycomb.io/screaminginthecloud. Observability, it's more than just hipster monitoring.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Periodically, I have a whole bunch of guests come on up, second time. Now, it's easy to take the naive approach of assuming that it's because it's easier for me to find a guest if I know them and don't have to reach out to brand new people all the time. This is absolutely correct; I'm exceedingly lazy. But I don't have too many folks on a third time, but that changes today.My guest is Tim Banks. I've had him on the show twice before, both times it led to really interesting conversations around a wide variety of things. Since those episodes, Tim has taken the job as a principal cloud economist here at The Duckbill Group. Yes, that is probably the strangest interview process you can imagine, but here we are. Tim, thank you so much for joining me both on the show and in the business.Tim: My pleasure, Corey. It was definitely an interesting interview process, you know, but I was glad to be here. So, I'm happy to be here a third time. I don't know if you get a jacket like you do in Saturday Night Live, if you host, like, a fifth time, but we'll see. Maybe it's a vest. A cool vest would be nice.Corey: We can come up with something.[ effectively, it can be like reverse hangman where you wind up getting a vest and every time you come on after that you get a sleeve, then you get a second sleeve, and then you get a collar, and we can do all kinds of neat stuff.Tim: I actually like that idea a lot.Corey: So, I'm super excited to be able to have this conversation with you because I don't normally talk a lot on this show about what cloud economics is because my guest usually is not as deep into the space as I am, and that's fine; people should never be as deep into this space as I am, in the general sense, unless they work here. Awesome. But I do guest on other shows, and people ask me all kinds of questions about AWS billing and cloud economics, and that's fine, it's great, but they don't ask the questions about the space in the same way that I would and the way that I think about it. So, it's hard for me to interview myself. Now, I'm not saying I won't try it someday, but it's challenging. But today, I get to take the easy path out and talk to you about it. So Tim, what the hell is a principal cloud economist?Tim: So, a principal cloud economist, is a cloud computing expert, both in architecture and practice, who looks at cloud cost in the same way that a lot of folks look at cloud security, or cloud resilience, or cloud performance. So, the same engineering concerns you have about making sure that your API stays up all the time, or to make sure that you don't have people that are able to escape containers or to make sure that you can have super, super low response times, is the same engineering fundamentals that I look at when I'm trying to find a way to reduce your AWS bill.Corey: Okay. When we say cloud cost and cloud economics, the natural picture that leads to mind is, “Oh, I get it. You're an Excel jockey.” And sometimes, yeah, we all kind of play those roles, but what you're talking about is something else entirely. You're talking about engineering expertise.And sure enough, if you look at the job postings we have for roles on the team from time to time, we have not yet hired anyone who does not have an engineering and architecture background. That seems odd to folks who do not spend a lot of time thinking about the AWS bill. I'm told those people are what is known as ‘happy.' But here we are. Why do we care about the engineering aspect of any of this?Tim: Well, I think first and foremost because what we're doing in essence, is still engineering. People aren't putting construction paper up on [laugh] AWS; sometimes they do put recipes up on there, but it still involves working on a computer, and writing code, and deploying it somewhere. So, to have that basic understanding of what it is that folks are doing on the platform, you have to have some engineering experience, first and foremost. Secondly, the fact of the matter is that most cost optimization, in my opinion, can be done on the whiteboard, before anything else, and really I think should be done on the whiteboard before anything else. And so the Excel aspect of it is always reactive. “We have now spent this much. How much was it? Where did it go?” And now we have to figure out where it went.I like to figure out and get a ballpark on how much something is going to cost before I write the first line of code. I want to know, hey, we have a tier here, we're using this kind of storage, it's going to take this kind of instance types. Okay, well, I've got an idea of how much it's going to cost. And I was like, “You know, that's going to be expensive. Before we do anything, is there a way that we can reduce costs there?”And so I'm reverse engineering that on already deployed workloads. Or when customers want to say, “Hey, we were thinking about doing this, and this is our proposed architecture,” I'm going to look at it and say, “Well, if you do this and this and this and this, you can save money.”Corey: So, it sounds like you and I have a bit of a philosophical disagreement in some ways. One of my recurring talking points has always been that, “Oh, by and large, application developers don't need to think overly much about cloud cost. What they need to know generally fits on an index card.” It's, okay, big things cost more than small things; if you turn something on, it will never get turned off and will bill you in perpetuity; data transfer has some weird stuff; and if you store data, you pay for data, like, that level of baseline understanding. When I'm trying to build something out my immediate thought is, great, is this thing possible?Because A, I don't always know that it is, and B, I'm super bad at computers so for me, it may absolutely not be, whereas you're talking about baking cost assessments into the architecture as a day one type of approach, even when sketching ideas out on the whiteboard. I'm curious as to how we diverge there. Can you talk more about your philosophy?Tim: Sure. And the reason I do that is because, as most folks that have an engineering background in cloud infrastructure will tell you, you want to build resilience in, on the whiteboard. You certainly want to build performance in, on the whiteboard, right? And security folks will tell you you want to do security on the whiteboard. Because those things are hard to fix after they're deployed.As soon as they're deployed, without that, you now have technical debt. If you don't consider cost optimization and cost efficiency on the whiteboard, and then you try and do it after it's deployed, you not only have technical debt, you may have actual real debt.Corey: One of the comments I tend to give a lot is that architecture and cost are the same thing in the world of cloud. And I think that we might be in violent agreement, as Liz Fong-Jones is fond of framing it, where I am acutely aware of aspects of cost and that does factor into how I build things on the whiteboard—let's also be very clear, most of the things that I build are very small scale; the largest cost by a landslide is the time I spend building it—in practice, that's an awful lot of environments; people are always more expensive than the AWS environment they're working on. But instead, it's about baking in the assumptions and making sure you're not coming up with something that is going to just be wasteful and horrible out of the gate, and I guess part of that also is the fact that I am at a level of billing understanding that I sort of absorbed these concepts intrinsically. Because to me, there is no difference between cost and architecture in an environment like this. You're right, there's always an inherent trade-off between cost and durability. On the one hand, I don't like that. On the other, it feels like it's been true forever and I don't see a way out of it.Tim: It is inescapable. And it's interesting because you talk about the level of an application developer or something like that, like what is your level of concern, but retroactively, we'll go in for cost optimization houses—and I've done this as far back as when I was working at AWS has a TAM—and I'll ask the question to an application developer or database administrator, and I'm like, “Why do you do this? What do you have a string value for something that could be a Boolean?” And you'll ask, “Well, what difference does that make?” Well, it makes a big difference when you're talking about cycles for CPU.You can reduce your CPU consumption on a database instance by changing a string to a Boolean, you need fewer instances, or you need a less powerful instance, or you need less memory. And now you can run a less expensive instance for your database architecture. Well, maybe for one node it's not that biggest difference, but if you're talking about something that's multi-AZ and multi-node, I mean, that can be a significant amount of savings just by making one simple change.Corey: And that might be the difference right there. I didn't realize that, offhand. It makes sense if you think about it, but just realizing that I've made that mistake on one of my DynamoDB tables. It costs something like seven cents a month right now, so it's not something I'm rushing to optimize, but you're right, expand that out by a factor of a million or so, and we're talking serious money, and then that sort of optimization makes an awful lot of sense. I think that my position on it is that when you're building out something small scale as a demo or a proof of concept, spending time on optimizations like this is not the best use of anyone's time or brain sweat, for lack of a better term. How do you wind up deciding when it's time to focus on stuff like that?Tim: Well, first, I will say that—I daresay that somewhere in the 80% of production workloads are just—were the POC, [laugh] right? Because, like, “It worked for this to get funding, let's run it,” right?Corey: Let they who does not have a DynamoDB table in production with the word ‘test' or ‘dev' in it cast the first stone.Tim: It's certainly not me. So, I understand how some of those decisions get made. And that's why I think it's better to think about it early. Because as I mentioned before, when you start something and say, “Hey, this works for now,” and you don't give consideration to that in the future, or consideration for what it's going to be like in the future, and when you start doing it, you'll paint yourself into corners. That's how you get something like static values put in somewhere, or that's how you get something like, well, “We have to run this instance type because we didn't build in the ability to be more microservice-based or stateless or anything like that.”You've seen people that say, “Hey, we could save you a lot of money if you can move this thing off to a different tier.” And it's like, “Well, that would be an extensive rewrite of code; that'd be very expensive.” I daresay that's the main reason why most AS/400s are still being used right now is because it's too expensive to rewrite the code.Corey: Yeah, and there's no AWS/400 that they can migrate to. Yet. Re:Invent is nigh.Tim: So, I think that's why, even at the very beginning, even if you were saying, “Well, this is something we will do later.” Don't make it impossible for you to do later in your code. Don't make it impossible for you to do later in your architecture. Make things as modular as possible, so that way you can say, “Hey”—later on down the road—“Oh, we can switch this instance type.” Or, “Here's a new managed service that we can maybe save money on doing this.”And you allow yourself to switch things out, or turn different knobs, or change the way you do things, and give yourself more options in the future, whether those options are for resilience, or those options or for security, or those options are for performance, or they're for cost optimizations. If you make binding decisions earlier on, you're going to have debt that's going to build up at some point in the future, and then you're going to have to pay the piper. Sometimes that piper is going to be AWS.Corey: One thing that I think gets lost in a lot of conversations about cloud economics—because I know that it happened to me when I first started this place—where I am planning to basically go out and be the world's leading expert in AWS cost analysis and understanding and optimization. Great. Then I went out into the world and started doing some of my first engagements, and they looked a lot less like far-future cost attribution projections and a lot more like, “What's a reserved instance?” And, “We haven't bought any of those in 18 months.” And, “Oh, yeah, we shut down an entire project six months ago. We should probably delete all the resources, huh?”The stuff that I was preparing for at the high end of the maturity curve are great and useful and terrific to have conversations about in some very nuanced depth, but very often there's a walk before you can run style of conversation where, okay, let's do the easy stuff first before we start writing a whole bunch of bespoke internal stuff that maps your business needs to the AWS bill. How do you, I guess, reconcile those things where you're on the one hand, you see the easy stuff and on the other, you see some of the just the absolutely challenging, very hard, five-years-of-engineering-effort-style problems on the other?Tim: Well, it's interesting because I've seen one customer very recently who has brilliant analyses as to their cost; just well-charted, well-tagged, well-documented, well—you know, everything is diagrammed quite nicely and everything like that, and they're very, very aware of their costs, but they leave test instances running all weekend, you know, and their associated volumes and things like that. And that's a very easy thing to fix. That is a very, very low-hanging fruit. And so sometimes, you just have to look at where they're spending their efforts where sometimes they do spend so much time chasing those hard to do things because they are hard to do and they're exciting in an engineering aspect, and then something as simple as, “Hey, how about we delete these old volumes?” It just isn't there.Or, “How about we switch to your S3 bucket storage type?” Those are easy, low-hanging fruits, and you would be surprised how sometimes they just don't get that. But at the same time, sometimes customers have, like, “Hey, we could knock this thing out, we knock this thing out,” because it's Trusted Advisor. Every AI cost optimization recommendation you can get will tell you these five things to do, no matter who you are or where you are, but they don't do the conceptual things like understanding some of the principles behind cost optimization and cost optimization architecture, and proactive cost optimization versus react with cost optimizations. So, you're doing very conceptual education and conversations with folks rather than the, “Do these five things.” And I've not often found a customer that you have to do both on; it's usually one or the other.Corey: It's funny that you made that specific reference to that example. One of my very first projects—not naming names. Generally, when it comes to things like this, you can tell stories or you can name names; I bias for stories—I was talking to a company who was convinced that their developer environments were incredibly overwrought, expensive, et cetera, and burning money. Okay, great. So, I talked about the idea of turning those things off at night or between test runs, deleting volumes to snapshot, and restore them on a schedule when people come in in the morning because all your developers sit in the same building in the same time zones. Great. They were super on board with the idea, and it was going to be a little bit of work, but all right, this was in the days before the EC2 Instance Scheduler, for example.But first, let's go ahead and do some analysis. This is one of those early engagements that really reinforced my idea of, yeah, before we start going too far down the rabbit hole, let's double-check what's going on in the account. Because periodically you encounter things that surprise people. Like, “What's up with those Australia instances?” “Oh, we don't have anything in that region.” “I believe you're being sincere when you say this, however, the API generally doesn't tell lies.”So, that becomes a, oh, security incident time. But looking at this, they were right; they had some fairly sizable developer instances that were running all the time, but doing some analysis, their developer environment was 3% of their bill at the time and they hadn't bought RIs in a year-and-a-half. And looking at what they were doing, there was so much easier stuff that they could do to generate significant savings without running the potential of turning a developer environment off at night in the middle of an incident or something like that. The risk factor and effort were easier just do the easy stuff, then do another pass and look at the deep stuff. And to be clear, they weren't lying to me; they weren't wrong.Back when they started building this stuff out, their developer environments were significantly large and were a significant portion of their spend. And then they hit product-market fit, and suddenly their production environment had to scale significantly in a short period of time. Which, yay, cloud. It's good at that. Then it just became such a small portion that developer environments weren't really a thing. But the narrative internally doesn't get updated very often because once people learn something, they don't go back to relearn whether or not it's still true. It's a constant mistake; I make it myself frequently.Tim: I think it's interesting, there are things that we really need to put into buckets as far as what's an engineering effort and what's an administrative effort. And when I say ‘administrative effort,' I mean if I can save money with a stroke of a pen, well, that's going to be pretty easy, and that's usually going to be RIs; that's going to be EDPs, or PPAs or something like that, that don't require engineering effort. It just requires administrative effort, I think RIs being the simplest ones. Like, “Oh, all I have to do is go in here and click these things four times and I'm going to save money?” “Well, let's do that.”And it's surprising how often people don't do that. But you still have to understand that, and whether it's RIs or whether it's a savings plan, it's still a commitment of some kind, but if you are willing to make that commitment, you can save money with no engineering effort whatsoever. That's almost free money.Corey: So, much of what we do here comes down to psychology, in many ways, more than it does math. And a lot of times you're right, everything you say is right, but in a large-scale environment, go ahead and click that button to buy the savings plan or the reserved instance, and that's a $20 million purchase. And companies will stall for months trying to run a different series of analyses on this and what if this happens, what if that happens, and I get it because, “Yeah, I'm going to click this button that's going to cost more money than I'll make in my lifetime,” that's a scary thing to do; I get it. But you're going to spend the money, one way or the other, with the provider, and if you believe that number is too high, I get it; I am right there with you. Buy half of them right now and then you can talk about the rest until you get to a point of being comfortable with it.Do it incrementally; it's not all or nothing, you have one shot to make the buy. Take pieces out of it that makes sense. You know you're probably not going to turn off your database cluster that handles all of production in the next year, so go ahead and go for it; it saves some money. Do the thing that makes sense. And that doesn't require deep-dive analytics that requires, on some level, someone who's seen a lot of these before who gets what customers are going through. And honestly, it's empathy in many respects, becomes one of those powerful things that we can apply to our customer accounts.Tim: Absolutely. I mean, people don't understand that decision paralysis, about making those commitments costs you money. You can spend months doing analysis, but those months doing analysis, you're going to spend 30, 40, 50, 60, 70% more on your EC2 instances or other compute than you would otherwise, and that can be quite significant. But it's one of those cases where we talk about psychology around perfect being the enemy of good. You don't have to make the perfect purchase of RIs or savings plans and have that so tuned perfectly that you're going to get one hundred percent utilization and zero—like, you don't have to do that.Just do something. Do a little bit. Like you said, buy half; buy anything; just something, and you're going to save money. And then you can run analysis later on, while you're saving money [laugh] and get a little better and tune it up a little more and get more analysis on and maybe fine-tune it, but you don't actually ever need to have it down to the penny. Like, it never has to be that good.Corey: At some point, one of the value propositions we have for our customers has always been that we tell you when to stop focusing on saving money because there's a theoretical cap of a hundred percent of the cloud bill that you can save, but you can make so much more than that by launching the right feature to the right market a little sooner; focus on that. Be responsible stewards of the money that's invested with you, but by and large, as a general piece of guidance, at some point, stop cutting and go back to doing the thing that makes your company work. It's not all about saving money at all costs for almost all of us. It is for us, but we're sort of a special case.Tim: Well, it's a conversation I often have. It's like, all right, are you trying to save money on AWS or are you trying to save money overall? So, if you're going to spend $400,000 worth of engineering effort to save $10,000 on your AWS bill, that doesn't make no sense. So—[laugh]—Corey: Right. There has to be a strategic reason to do things like that—Tim: Exactly.Corey: —and make sure you understand the value of what you're getting for this. One reason that we wind up charging the way that we do—and we've gotten questions on this for a while—has been that we charge a fixed fee for what we do on engagements. And similarly—people have asked this, but haven't tied the two things together—you talk about cost optimization, but never cost-cutting. Why is that? Is that just a negative term?And the answer has been no, they're aligned. What we do focuses on what is best for the customer. Once that fixed fee is decided upon, every single thing that we say is what we would do if we were in the customer's position. There are times we'll look at what they have going on and say, “Ah, you really should spend more money here for resiliency, or durability,” or, “Okay, that is critical data that's not being backed up. You should consider doing that.”It's why we don't take percentages of things because, at that point, we're not just going with the useful stuff, it's, well we're going to basically throw the entire kitchen sink at you. We had an early customer and I was talking to their AWS account manager about what we were going to be doing and their comment was, “Oh, saving money on AWS bills is great, make sure you check the EBS snapshots.” Yeah, I did that. They were spending 150 bucks a month on EBS snapshots, which is basically nothing. It's one of those stories where if, in the course of an hour-long meeting, I can pay for that entire service, by putting a quarter on the table, I'm probably not going to talk about it barring [laugh] some extenuating circumstances.Focus on the big things, not the things that worked in a different environment with a different account and different constraints. It's hard to context switch like that, but it gets a lot easier when it is basically the entirety of what we do all day.Tim: The difference I draw between cost optimization and cost-cutting is that cost optimization is ensuring that you're not spending money unnecessarily, or that you're maximizing your dollar. And so sometimes we get called in there, and we're just validation for the measures they've already done. Like, “Your team is doing this exactly right. You're doing the things you should be doing. We can nitpick if you want to; we're going to save you $7 a year, but who cares about that? But y'all are doing what you should be doing. This is great. Going forward, you want to look for these things and look for these things and look for these things. We're going to give you some more concepts so that you are cost-optimized in the future.” But it doesn't necessarily mean that we have to cut your bill. Because if you're already spending efficiently, you don't need your bill cut; you're already cost-optimized.Corey: Oh, we're not going to nitpick on that, you're mostly optimized there. It's like, “Yeah, that workload's $140 million a year and rising; please, pick nits.” At which point? “Okay, great.” That's the strategic reason to focus on something. But by and large, it comes down to understanding what the goals of clients are. I think that is widely misunderstood about what we do and how we do it.The first question I always ask when someone does outreach of, “Hey, we'd like to talk about coming in here and doing a consulting engagement with us.” “Great.” I always like to ask the quote-unquote, “Foolish question” of, “Why do you care about the AWS bill?” And occasionally I'll get people who look at me like I have two heads of, “Why wouldn't I care about the AWS bill?” Because there are more important things to care about for the business, almost certainly.Tim: One of the things I try and do, especially when we're talking about cost optimization, especially trying to do something for the right now so they can do things going forward, it's like, you know, all right, so if we cut this much from your bill—if you just do nothing else, but do reserved instances or buy a savings plan, right, you're going to save enough money to hire four engineers. Think about what four engineers would do for your overall business? And that's how I want you to frame it; I want you to look at what cost optimization is going to allow you to do in the future without costing you any more money. Or maybe you save a little more money and you can shift it; instead of paying for your AWS bill, maybe you can train your developers, maybe you can get more developers, maybe you can get some ProServ, maybe you can do whatever, buy newer computers for your people so they can do—whatever it is, right? We're not saying that you no longer have to spend this money, but saying, “You can use this money to do something other than give it to Jeff Bezos.”Corey: This episode is sponsored in part by Liquibase. If you're anything like me, you've screwed up the database part of a deployment so severely that you've been banned from touching every anything that remotely sounds like SQL, at at least three different companies. We've mostly got code deployments solved for, but when it comes to databases we basically rely on desperate hope, with a roll back plan of keeping our resumes up to date. It doesn't have to be that way. Meet Liquibase. It is both an open source project and a commercial offering. Liquibase lets you track, modify, and automate database schema changes across almost any database, with guardrails to ensure you'll still have a company left after you deploy the change. No matter where your database lives, Liquibase can help you solve your database deployment issues. Check them out today at liquibase.com. Offer does not apply to Route 53.Corey: There was an article recently, as of the time of this recording, where Pinterest discussed what they had disclosed in one of their regulatory filings which was, over the next eight years, they have committed to pay AWS $3.2 billion. And in this article, they have the head of engineering talking to the reporter about how they're thinking about these things, how they're looking at things that are relevant to their business, and they're talking about having a dedicated team that winds up doing a whole bunch of data analysis and running some analytics on all of these things, from piece to piece to piece. And that's great. And I worry, on some level, that other companies are saying, “Oh, Pinterest is doing that. We should, too.” Yeah, for the course of this commitment, a 1% improvement is $32 million, so yeah, at that scale I'm going to hire a team of data scientists, too, look at these things. Your bill is $50,000 a month. Perhaps that's not worth the effort you're going to put into it, barring other things that contribute to it.Tim: It's interesting because we will get folks that will approach us that have small accounts—very small, small spend—and like, “Hey, can you come in and talk to us about this whatever.” And we can say very honestly, “Look, we could, but the amount of money we're going to charge you is going to—it's not going to be worth your while right now. You could probably get by on the automated recommendations, on the things that already out there on the internet that everybody can do to optimize their bill, and then when you grow to a point where now saving 10% is somebody's salary, that's when it, kind of, becomes more critical.” And it's hard to say what point that is in anyone's business, but I can say sometimes, “Hey, you know what? That's not really what you need to focus on.” If you need to save $100 a month on your AWS bill, and that's critical, you've got other concerns that are not your AWS bill.Corey: So, back when you were interviewing to work here, one of the areas of focus that you kept bringing up was the concept of observability, and my response to this was, “Ah, hell. Another one.” Because let's be clear, Mike Julian—my business partner and our CEO—has written a book called Practical Monitoring, and apparently what we learned from this is as soon as you finish writing a book on the topic, you never want to talk about that topic ever again, which yeah, in hindsight makes sense. Why do you care about observability when you're here to look at cloud costs?Tim: Because cloud costs is another metric, just like you would use for performance, or resilience, or security. You do real-time monitoring to see if somebody has compromised the system, you do real-time monitoring to see if you have bad performance, if response times are too slow. You do real-time monitoring to know if something has gone down and then you need to make adjustments, or that the automated responses you have in response to that downtime are working. But cloud costs, you send somebody a report at the end of the month. Can you imagine, if you will—just for a second—if you got a downtime report at the end of month, and then you can react to something that has gone down?Or if you get a security report at the end of the month, and then you can react to the fact that somebody has your root keys? Or if you get [laugh] a report at the end of month, this said, “Hey, the CPU on this one was pegged. You should probably scale up.” That's outrageous to anybody in this industry right now. But why do we accept that for cloud cost?Corey: It's worse than that. There are a number of startups that talk about, “Oh, real-time cloud cost monitoring. Okay, the only way you're going to achieve such a thing is if you build an API shim that interprets everything that you're telling your cloud control plane to do, taking cost metrics out of it, and then passing it on to the actual cloud control plane.” Otherwise, you're talking about it showing up in the billing record in—ideally, eight hours; in practice, several days, or you're talking about the CloudTrail events, which is not holistic but gives you some rough idea, but it's also in some cases, 5 to 20 minutes delayed. There's no real-time way to do this without significant disruption to what's going on in your environment.So, when I hear about, “Oh, we do real-time bill analysis.” Yeah, it feels—to be very direct—you don't know enough about the problem space you're working within to speak intelligently about it because anyone who's played in this space for a while knows exactly how hard it is to get there. Now, I've talked to companies that have built real-time-ish systems that take that shim approach and acts sort of as a metadata sidecar ersatz billing system that tracks all of this so they can wind up intercepting potentially very expensive configuration mistakes. And that's great. That's also a bit beyond for a lot of folks today, but it's where the industry is going. But there is no way to get there today, short of effectively intercepting all of those calls, in a way that is cohesive and makes sense. How do you square that circle given the complete lack of effective tooling?Tim: Honestly, I'm going to point that right back at the cloud provider because they know how much you're spending, real-time. They know exactly how much you spend in real-time. They've figured it out. They have the buckets, they have APIs for it internally. I'm sure they do; it would make no sense for them not to. Without giving anything anyway, I know that when I was at AWS, I knew how much they were spending, almost real-time.Corey: That's impressive. I wish that existed. My never having worked at AWS perspective on it is that they, of course, have the raw data effective immediately, or damn close to it, but the challenge for the billing system is distilling and summarizing and attributing all of that in a reasonable timeframe; it is an exabyte-scale problem. I've talked to folks there who have indicated it is comfortably north of a petabyte in raw data per day. And that was a couple of years ago, so one can only imagine as the footprint has increased, so has all of this.I mean, the billing system is fundamentally magic from the outside. I'm not saying it's good magic, but it is magic, and it's something that is unappreciated, that every customer uses, and is one of those areas that doesn't get the attention it deserves. Because, let's be clear, here, we talk about observability; the bill is still the only thing that AWS offers that gives you a holistic overview of everything running in your account, in one place.Tim: What I think is interesting is that you talk about this, the scale of the problem and that it makes it difficult to solve. At the same time, I can have a conversation with my partner about kitty litter, and then all of a sudden, I'm going to start getting ads about kitty litter within minutes. So, I feel like it's possible to emit cost as a metric like you would CPU or disk. And if I'm going to look at who's going to do that, I'm going to look right back at AWS. The fun part about that, though, is I know from AWS's business model, that if that's something they were to emit, it would also cost you, like, 25 cents per call, and then you would actually, like, triple your cloud costs just trying to figure out how much it costs you.Corey: Only with 16 other billing dimensions because of course it would. And again, I'm talking about stuff, because of how I operate and how I think about this stuff, that is inherently corner case, or [vertex 00:31:39] case in many cases. But for the vast majority of folks, it's not the, “Oh, you have this really weird data transfer paradigm between these two resources,” which yeah, that's a problem that needs to be addressed in an awful lot of cases because data transfer pricing is bonkers, but instead it's the, “Huh. You just spun up a big cluster that's going to cost $20,000 a month.” You probably don't need to wait a full day to flag that.And you also can't put this on the customer in the sense of, “Oh, just set some budget alarms, that's great. That's the first thing you should do in a new AWS account.” “Well, jackhole, I've done an awful lot of first things I'm supposed to do in an AWS account, in my dedicated test account for these sorts of things. It's been four months, I'm not done yet with all of those first things I'm supposed to do.” It's incredibly secure, increasingly expensive, and so far all it runs is a single EC2 instance that is mostly there just so that everything else doesn't error out trying to divide by zero.Tim: There are some things that are built-in. If I stand up an EC2 instance and it goes down, I'm going to get an alert that this instance terminated for some reason. It's just going to show up informationally.Corey: In the console. You're not going to get called about it or paged about it, unless—Tim: Right.Corey: —you have something else in the business that will, like a boss that screams at you two o'clock in the morning. This is why we have very little that's production-facing here.Tim: But if I know that alert exists somewhere in the console, that's easy for me to write a trap for. That's easy for me to write, say hey, I'm going to respond to that because this call is going to come out somewhere; it's going to get emitted somewhere. I can now, as an engineer, write a very easy trap that says, “Hey, pop this in the Slack. Send an alert. Send a page.”So, if I could emit a cost metric, and I could say, “Wow. Somebody has spun up this thing that's going to cost X amount of money. Someone should get paged about this.” Because if they don't page about this and we wait eight hours, that's my month's salary. And you would do that if your database server went down; you would do that if someone rooted that database server; you would do that if the database server was [bogging 00:33:48] you to scale up another one. So, why can't you do that if that database server was all of sudden costing you way more than you had calculated?Corey: And there's a lot of nuance here because what you're talking about makes perfect sense for smaller-scale accounts, but even some of the very large accounts where we're talking hundreds of millions a year in spend, you can set compromised keys up on GitHub, put them in Payspin, whatever, and then people start spinning up Bitcoin miners everywhere. Great. It takes a long time to materially move the needle on that level of spend; it gets lost in the background noise. I lose my mind when I wind up leaving a managed NAT gateway running and it cost me 70 bucks a month in my $5 a month test account. Yeah, but you realize you could basically buy an island and it gets lost in the AWS bill at some of the high watermarks for some of these larger accounts.“Oh, someone spun up a cluster that's going to cost $400,000 a year?” Yeah, do I need to re-explain to you what a data science team does? They light money on fire in return for questionable returns, as a general rule. You knew that when you hired them; leave them alone. Whereas someone in their developer account does this, yeah, you kind of want to flag that immediately.It always comes down to rules and context. But I'd love to have some templates ready to go of, “I'm a starving student, please alert me anytime it looks like I might possibly exceed the free tier,” or better yet, “Don't let me, and if I do, it's on you and you eat the cost.” Conversely, it's, “Yeah, this is a Netflix sub-account or whatnot. Maybe don't bother me for anything whatsoever because freedom and responsibility is how we roll.” I imagine that's what they do internally on a lot of their cloud costing stuff because freedom and responsibility is ingrained in their culture. It's great. It's the freedom from having to think about cloud bills and the responsibility for paying it, of the cloud bill.Tim: Yeah, we will get internally alerted if things are [laugh] up too long, and then we will actually get paged, and then our manager would get paged, [laugh] and it would go up the line. If you leave something that's running too expensive, too long. So, there is a system there for it.Corey: Oh, yeah. The internal AWS systems for employees are probably my least favorite AWS service, full stop. And I've seen things posted about it; I believe it's called Isengard, for spinning up internal accounts and the rest—there's a separate one, I think, called Conduit, but I digress—that you spin something up, and apparently if it doesn't wind up—I don't need you to comment on this because you worked there and confidentiality is super important, but to my understanding it's, great, it has a whole bunch of formalized stuff like that and it solves for a whole lot of nifty features that bias for the way that AWS focuses on accounts and how they've view security and the rest. And, “Oh, well, we couldn't possibly ship this to customers because it's not how they operate.” And that's great.My problem with this internal provisioning system is it isolates and insulates AWS employees from the real pain of working with multiple accounts as a customer. You don't have to deal with the provisioning process of Control Tower or whatnot; you have your own internal thing. Eat your own dog food, gargle your own champagne, whatever it takes to wind up getting exposure to the pain that hits customers and suddenly you'll see those things improve. I find that the best way to improve a product is to make the people building it live with the painful parts.Tim: I think it's interesting that the stance is, “Well, it's not how the customers operate, and we wouldn't want the customers to have to deal with this.” But at the same time, you have to open up, like, 100 accounts if you need more than a certain number of S3 buckets. So, they are very comfortable with burdening the customer with a lot of constraints, and they say, “Well, constraints drive innovation.” Certainly, this is a constraint that you could at least offer and let the customers innovate around that.Corey: And at least define who the customer is. Because yeah, “I'm a Netflix sub-account is one story,” “I'm a regulated bank,” is another story, and, “I'm a student in my dorm room, trying to learn how this whole cloud thing works,” is another story. From risk tolerance, from a data protection story, from a billing surprise story, from a, “I'm trying to learn what the hell this is, and all these other service offerings you keep talking to me about confuse the hell out of me; please streamline the experience.” There's a whole universe of options and opportunity that isn't being addressed here.Tim: Well, I will say it very simply like this: we're talking about a multi-trillion dollar company versus someone who, if their AWS bill is too high, they don't pay rent; maybe they don't eat; maybe they have other issues, they don't—medical bill doesn't get paid; child care doesn't get paid. And if you're going to tell me that this multi-trillion dollar company can't solve for that so that doesn't happen to that person and tells them, “Well, if you come in afterwards, after your bill gets there, maybe we can do something about it, but in the meantime, suffer through this.” That's not ethical. Full stop.Corey: There are a lot of things that AWS gets right, and I want to be clear that I'm not sitting here trying to cast blame and say that everything they're doing is terrible. I feel like every time I talk about billing in any depth, I have to throw this disclaimer in. Ninety to ninety-five percent of what they do is awesome. It's just the missing piece that is incredibly painful for customers, and that's what I spend most of my time focusing on. It should not be interpreted to think that I hate the company.I just want them to do better than they are, and what they're doing now is pretty decent in most respects. I just want to fix the painful parts. Tim, thank you for joining me for a third time here. I'm certain I'll have you back in the somewhat near future to talk about more aspects of this, but until then, where can people find you slash retain your services?Tim: Well, you can find me on Twitter at @elchefe. If you want to retain my services for which you would be very, very happy to have, you can go to duckbillgroup.com and fill out a little questionnaire, and I will magically appear after an exchange of goods and services.Corey: Make sure to reference Tim by name just so that we can make our sales team facepalm because they know what's coming next. Tim, thank you so much for your time; it's appreciated.Tim: Thank you so much, Corey. I loved it.Corey: Principal cloud economist here at The Duckbill Group, Tim Banks. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, wait at least eight hours—possibly as many as 48 to 72—and then leave a comment explaining what you didn't like.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About EmmaEmma Bostian is a Software Engineer at Spotify in Stockholm. She is also a co-host of the Ladybug Podcast, author of Decoding The Technical Interview Process, and an instructor at LinkedIn Learning and Frontend Masters.Links: Ladybug Podcast: https://www.ladybug.dev LinkedIn Learning: https://www.linkedin.com/learning/instructors/emma-bostian Frontend Masters: https://frontendmasters.com/teachers/emma-bostian/ Decoding the Technical Interview Process: https://technicalinterviews.dev Twitter: https://twitter.com/emmabostian TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friends at Jellyfish. So, you're sitting in front of your office chair, bleary eyed, parked in front of a powerpoint and—oh my sweet feathery Jesus its the night before the board meeting, because of course it is! As you slot that crappy screenshot of traffic light colored excel tables into your deck, or sift through endless spreadsheets looking for just the right data set, have you ever wondered, why is it that sales and marketing get all this shiny, awesome analytics and inside tools? Whereas, engineering basically gets left with the dregs. Well, the founders of Jellyfish certainly did. That's why they created the Jellyfish Engineering Management Platform, but don't you dare call it JEMP! Designed to make it simple to analyze your engineering organization, Jellyfish ingests signals from your tech stack. Including JIRA, Git, and collaborative tools. Yes, depressing to think of those things as your tech stack but this is 2021. They use that to create a model that accurately reflects just how the breakdown of engineering work aligns with your wider business objectives. In other words, it translates from code into spreadsheet. When you have to explain what you're doing from an engineering perspective to people whose primary IDE is Microsoft Powerpoint, consider Jellyfish. Thats Jellyfish.co and tell them Corey sent you! Watch for the wince, thats my favorite part.Corey: This episode is sponsored in part by Liquibase. If you're anything like me, you've screwed up the database part of a deployment so severely that you've been banned from touching every anything that remotely sounds like SQL, at at least three different companies. We've mostly got code deployments solved for, but when it comes to databases we basically rely on desperate hope, with a roll back plan of keeping our resumes up to date. It doesn't have to be that way. Meet Liquibase. It is both an open source project and a commercial offering. Liquibase lets you track, modify, and automate database schema changes across almost any database, with guardrails to ensure you'll still have a company left after you deploy the change. No matter where your database lives, Liquibase can help you solve your database deployment issues. Check them out today at liquibase.com. Offer does not apply to Route 53.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. One of the weird things that I've found in the course of, well, the last five years or so is that I went from absolute obscurity to everyone thinking that I know everyone else because I have thoughts and opinions on Twitter. Today, my guest also has thoughts and opinions on Twitter. The difference is that what she has to say is actually helpful to people. My guest is Emma Bostian, software engineer at Spotify, which is probably, if we can be honest about it, one of the least interesting things about you. Thanks for joining me.Emma: Thanks for having me. That was quite the intro. I loved it.Corey: I do my best and I never prepare them, which is a blessing and a curse. When ADHD is how you go through life and you suck at preparation, you've got to be good at improv. So, you're a co-host of the Ladybug Podcast. Let's start there. What is that podcast? And what's it about?Emma: So, that podcast is just my three friends and I chatting about career and technology. We all come from different backgrounds, have different journeys into tech. I went the quote-unquote, “Traditional” computer science degree route, but Ali is self-taught and works for AWS, and Kelly she has, like, a master's in psychology and human public health and runs her own company. And then Sydney is an awesome developer looking for her next role. So, we all come from different places and we just chat about career in tech.Corey: You're also an instructor at LinkedIn Learning and Frontend Masters. I'm going to guess just based upon the name that you are something of a frontend person, which is a skill set that has constantly eluded me for 20 years, as given evidence by every time I've tried to build something that even remotely touches frontend or JavaScript in any sense.Emma: Yeah, to my dad's disdain, I have stuck with the frontend; he really wanted me to stay backend. I did an internship at IBM in Python, and you know, I learned all about assembly language and database, but frontend is what really captures my heart.Corey: There's an entire school of thought out there from a constituency of Twitter that I will generously refer to as shitheads that believe, “Oh, frontend is easy and it's somehow less than.” And I would challenge anyone who holds that perspective to wind up building an interface that doesn't look like crap first, then come and talk to me. Spoiler, you will not say that after attempting to go down that rabbit hole. If you disagree with this, you can go ahead and yell at me on Twitter so I know where you're hiding, so I can block you. Now, that's all well and good, but one of the most interesting things that you've done that aligns with topics near and dear to my heart is you wrote a book.Now, that's not what's near and dear to my heart; I have the attention span to write a tweet most days. But the book was called Decoding the Technical Interview Process. Technical interviewing is one of those weird things that comes up from time to time, here and everywhere else because it's sort of this stylized ritual where we evaluate people on a number of skills that generally don't reflect in their day-to-day; it's really only a series of skills that you get better by practicing, and you only really get to practice them when you're interviewing for other jobs. That's been my philosophy, but again, I've written a tweet on this; you've written a book. What's the book about and what drove you to write it?Emma: So, the book covers everything from an overview of the interview process, to how do you negotiate a job offer, to systems design, and talks about load balancing and cache partitioning, it talks about what skills you need from the frontend side of things to do well on your JavaScript interviews. I will say this, I don't teach HTML, CSS, and JavaScript in-depth in the book because there are plenty of other resources for that. And some guy got mad at me about that the other day and wanted a refund because I didn't teach the skills, but I don't need to. [laugh]. And then it covers data structures and algorithms.They're all written in JavaScript, they have easy to comprehend diagrams. What drove me to write this is that I had just accepted a job offer in Stockholm for a web developer position at Spotify. I had also just passed my Google technical interviews, and I finally realized, holy crap, maybe I do know what I'm doing in an interview now. And this was at the peak of when people were getting laid off due to COVID and I said, “You know what? I have a lot of knowledge. And if I have a computer science degree and I was able to get through some of the hardest technical interviews, I think I should share that with the community.”Because some people didn't go through a CS degree and don't understand what a linked list is. And that's not their fault. It's just unfortunately, there weren't a lot of great resources—especially for web developers out there—to learn these concepts. Cracking the Coding Interview is a great book, but it's written in backend language and it's a little bit hard to digest as a frontend developer. So, I decided to write my own.Corey: How much of the book is around the technical interview process as far as ask, “Here's how you wind up reversing linked lists,” or, “Inverting a binary tree,” or whatever it is where you're tracing things around without using a pointer, how do you wind up detecting a loop in a recursive whatever it is—yeah, as you can tell, I'm not a computer science person at all—versus how much of it is, effectively, interview 101 style skills for folks who are even in non-technical roles could absorb?Emma: My goal was, I wanted this to be approachable by anyone without extensive technical knowledge. So, it's very beginner-friendly. That being said, I cover the basic data structures, talking about what traditional methods you would see on them, how do you code that, what does that look like from a visual perspective with fake data? I don't necessarily talk about how do you reverse a binary tree, but I do talk about how do you balance it if you remove a node? What if it's not a leaf node? What if it has children? Things like that.It's about [sigh] I would say 60/40, where 40% is coding and technical stuff, but maybe—eh, it's a little bit closer to 50/50; it kind of depends. I do talk about the take-home assessment and tips for that. When I do a take-home assessment, I like to include a readme with things I would have done if I had more time, or these are performance trade-offs that I made; here's why. So, there's a lot of explanation as to how you can improve your chances at moving on to the next round. So yeah, I guess it's 50/50.I also include a section on tips for hiring managers, how to create an inclusive and comfortable environment for your candidates. But it's definitely geared towards candidates, and I would say it's about 50/50 coding tech and process stuff.Corey: One of the problems I've always had with this entire industry is it feels like we're one of the only industries that does this, where we bring people in, and oh, you've been an engineer for 15 years at a whole bunch of companies I've recognized, showing career progression, getting promoted at some of them transitioning from high-level role to high-level role. “Great, we are so glad that you came in to interview. Now, up to the whiteboard, please, and implement FizzBuzz because I have this working theory that you don't actually know how to code, and despite the fact that you've been able to fake your way through it at big companies for 15 years, I'm the one that's going to catch you out with some sort of weird trivia question.” It's this adversarial, almost condescending approach and I don't see it in any other discipline than tech. Is that just because I'm not well-traveled enough? Is that because I'm misunderstanding the purpose of all of these things? Or, what is this?Emma: I think partially it was a gatekeeping solution for a while, for people who are comfortable in their roles and may be threatened by people who have come through different paths to get to tech. Because software engineer used to be an accredited title that you needed a degree or certification to get. And in some countries it still is, so you'll see this debate sometimes about calling yourself a software engineer if you don't have that accreditation. But in this day and age, people go through boot camps, they can come from other industries, they can be self-taught. You don't need a computer science degree, and I think the interview process has not caught up with that.I will say [laugh] the worst interview I had was at IBM when I was already working there. I was already a web developer there, full-time. I was interviewing for a role, and I walked into the room and there were five guys sitting at a table and they were like, “Get up to the whiteboard.” It was for a web development job and they quizzed me about Java. And I was like, “Um, sir, I have not done Java since college.” And they were like, “We don't care.”Corey: Oh, yeah, coding on a whiteboard in front of five people who already know the answer—Emma: Horrifying.Corey: —during a—for them, it's any given Tuesday, and for you, it is a, this will potentially determine the course that your career takes from this point forward. There's a level of stress that goes into that never exists in our day-to-day of building things out.Emma: Well, I also think it's an artificial environment. And why, though? Like, why is this necessary? One of the best interviews I had was actually with Gatsby. It was for an open-source maintainer role, and they essentially let me try the product before I bought it.Like, they let me try out doing the job. It was a paid process, they didn't expect me to do it for free. I got to choose alternatives if I wanted to do one thing or another, answer one question or another, and this was such an exemplary process that I always bring it up because that is a modern interview process, when you are letting people try the position. Now granted, not everyone can do this, right? We've got parents, we've got people working two jobs, and not everyone can afford to take the time to try out a job.But who can also afford a five-stage interview process that still warrants taking vacation days? So, I think at least—at the very least—pay your candidates if you can.Corey: Oh, yeah. One of the best interviews I've ever had was at a company called Three Rings Design, which is now defunct, unfortunately, but it was fairly typical ops questions of, “Yeah, here's an AWS account. Spin up a couple EC2 instances, load balance between them, have another one monitored. You know, standard op stuff. And because we don't believe in asking people to work for free, we'll pay you $300 upon completion of the challenge.”Which, again, it's not huge money for doing stuff like that, but it's also, this shows a level of respect for my time. And instead of giving me a hard deadline of when it was due, they asked me, “When can we expect this by?” Which is a great question in its own right because it informs you about a candidate's ability to set realistic deadlines and then meet them, which is one of those useful work things. And they—unlike most other companies I spoke with in that era—were focused on making it as accommodating for the candidate as possible. They said, “We're welcome to interview you during the workday; we can also stay after hours and have a chat then, if that's more convenient for your work schedule.”Because they knew I was working somewhere else; an awful lot of candidates are. And they just bent over backwards to be as accommodating as possible. I see there's a lot of debate these days in various places about the proper way to interview candidates. No take-home because biases for people who don't have family obligations or other commitments outside of work hours. “Okay, great, so I'm going to come in interview during the day?” “No. That biases people who can't take time off.” And, on some level, it almost seems to distill down to no one likes any way that there is of interviewing candidates, and figuring out a way that accommodates everyone is a sort of a fool's errand. It seems like there is no way that won't get you yelled at.Emma: I think there needs to be almost like a choose your own adventure. What is going to set you up for success and also allow you to see if you want to even work that kind of a job in the first place? Because I thought on paper, open-source maintainer sounds awesome. And upon looking into the challenges, I'm like, “You know what? I think I'd hate this job.”And I pulled out and I didn't waste their time and they didn't waste mine. So, when you get down to it, honestly, I wish I didn't have to write this book. Did it bring me a lot of benefit? Yeah. Let's not sugarcoat that. It allowed me to pay off my medical debt and move across a continent, but that being said, I wish that we were at a point in time where that did not need to exist.Corey: One of the things that absolutely just still gnaws at me even years later, is I interviewed at Google twice, and I didn't get an offer either time, I didn't really pass their technical screen either time. The second one that really sticks out in my mind where it was, “Hey, write some code in a Google Doc while we watch remotely,” and don't give you any context or hints on this. And just it was—the entire process was sitting there listening to them basically, like, “Nope, not what I'm thinking about. Nope, nope, nope.” It was… by the end of that conversation, I realized that if they were going to move forward—which they didn't—I wasn't going to because I didn't want to work with people that were that condescending and rude.And I've held by it; I swore I would never apply there again and I haven't. And it's one of those areas where, did I have the ability to do the job? I can say in hindsight, mostly. Were there things I was going to learn as I went? Absolutely, but that's every job.And I'm realizing as I see more and more across the ecosystem, that they were an outlier in a potentially good way because in so many other places, there's no equivalent of the book that you have written that is given to the other side of the table: how to effectively interview candidates. People lose sight of the fact that it's a sales conversation; it's a two-way sale, they have to convince you to hire them, but you also have to convince them to work with you. And even in the event that you pass on them, you still want them to say nice things about you because it's a small industry, all things considered. And instead, it's just been awful.Emma: I had a really shitty interview, and let me tell you, they have asked me subsequently if I would re-interview with them. Which sucks; it's a product that I know and love, and I've talked about this, but I had the worst experience. Let me clarify, I had a great first interview with them, and I was like, “I'm just not ready to move to Australia.” Which is where the job was. And then they contacted me again a year later, and it was the worst experience of my life—same recruiter—it was the ego came out.And I will tell you what, if you treat your candidates like shit, they will remember and they will never recommend people interview for you. [laugh]. I also wanted to mention about accessibility because—so we talked about, oh, give candidates the choice, which I think the whole point of an interview should be setting your candidates up for success to show you what they can do. And I talked with [Stephen 00:14:09]—oh, my gosh, I can't remember his last name—but he is a quadriplegic and he types with a mouthstick. And he was saying he would go to technical interviews and they would not be prepared to set him up for success.And they would want to do these pair programming, or, like, writing on a whiteboard. And it's not that he can't pair program, it's that he was not set up for success. He needed a mouthstick to type and they were not prepared to help them with that. So, it's not just about the commitment that people need. It's also about making sure that you are giving candidates what they need to give the best interview possible in an artificial environment.Corey: One approach that people have taken is, “Ah, I'm going to shortcut this and instead of asking people to write code, I'm going to look at their work on GitHub.” Which is, in some cases, a great way to analyze what folks are capable of doing. On the other, well, there's a lot of things that play into that. What if they're working in environment where they don't have the opportunity to open-source their work? What if people consider this a job rather than an all-consuming passion?I know, perish the thought. We don't want to hire people like that. Grow up. It's not useful, and it's not helpful. It's not something that applies universally, and there's an awful lot of reasons why someone's code on GitHub might be materially better—or worse—than their work product. I think that's fine. It's just a different path toward it.Emma: I don't use GitHub for largely anything except just keeping repositories that I need. I don't actively update it. And I have, like, a few thousand followers; I'm like, “Why the hell do you guys follow me? I don't do anything.” It's honestly a terrible representation.That being said, you don't need to have a GitHub repository—an active one—to showcase your skills. There are many other ways that you can show a potential employer, “Hey, I have a lot of skills that aren't necessarily showcased on my resume, but I like to write blogs, I like to give tech talks, I like to make YouTube videos,” things of that nature.Corey: I had a manager once who refused to interview anyone who didn't have a built-out LinkedIn profile, which is also one of these bizarre things. It's, yeah, a lot of people don't feel the need to have a LinkedIn profile, and that's fine. But the idea that, “Oh, yeah, they have this profile they haven't updated in a couple years, it's clearly they're not interested in looking for work.” It's, yeah. Maybe—just a thought here—your ability to construct a resume and build it out in the way that you were expecting is completely orthogonal to how effective they might be in the role. The idea that someone not having a LinkedIn profile somehow implies that they're sketchy is the wrong lesson to take from all of this. That site is terrible.Emma: Especially when you consider the fact that LinkedIn is primarily used in the United States as a social—not social networking—professional networking tool. In Germany, they use Xing as a platform; it's very similar to LinkedIn, but my point is, if you're solely looking at someone's LinkedIn as a representation of their ability to do a job, you're missing out on many candidates from all over the world. And also those who, yeah, frankly, just don't—like, they have more important things to be doing than updating their LinkedIn profile. [laugh].Corey: On some level, it's the idea of looking at a consultant, especially independent consultant type, when their website is glorious and up-to-date and everything's perfect, it's, oh, you don't really have any customers, do you? As opposed to the consultants you know who are effectively sitting there with a waiting list, their website looks like crap. It's like, “Is this Geocities?” No. It's just that they're too busy working on the things that bring the money instead of the things that bring in business, in some respects.Let's face it, websites don't. For an awful lot of consulting work, it's word of mouth. I very rarely get people finding me off of Google, clicking a link, and, “Hey, my AWS bill is terrible. Can you help us with it?” It happens, but it's not something that happens so frequently that we want to optimize for it because that's not where the best customers have been coming from. Historically, it's referrals, it's word of mouth, it's people seeing the aggressive shitposting I engage in on Twitter and saying, “Oh, that's someone that should help me with my Amazon bill.” Which I don't pretend to understand, but I'm still going to roll with it.Emma: You had mentioned something about passion earlier, and I just want to say, if you're a hiring manager or recruiter, you shouldn't solely be looking at candidates who superficially look like they're passionate about what they do. Yes, that is—it's important, but it's not something that—like, I don't necessarily choose one candidate over the other because they push commits, and open pull requests on GitHub, and open-source, and stuff. You can be passionate about your job, but at the end of the day, it's still a job. For me, would I be working if I had to? No. I'd be opening a bookstore because that's what I would really love to be doing. But that doesn't mean I'm not passionate about my job. I just show it in different ways. So, just wanted to put that out there.Corey: Oh, yeah. The idea that you must eat, sleep, live, and breathe is—hell with that. One of the reasons that we get people to work here at The Duckbill Group is, yeah, we care about getting the job done. We don't care about how long it takes or when you work; it's oh, you're not feeling well? Take the day off.We have very few things that are ‘must be done today' style of things. Most of those tend to fall on me because it's giving a talk at a conference; they will not reschedule the conference for you. I've checked. So yeah, that's important, but that's not most days.Emma: Yeah. It's like programming is my job, it's not my identity. And it's okay if it is your primary hobby if that is how you identify, but for me, I'm a person with actual hobbies, and, you know, a personality, and programming is just a job for me. I like my job, but it's just a job.Corey: And on the side, you do interesting things like wrote a book. You mentioned earlier that it wound up paying off some debt and helping cover your move across an ocean. Let's talk a little bit about that because I'm amenable to the idea of side projects that accidentally have a way of making money. That's what this podcast started out as. If I'm being perfectly honest, and started out as something even more self-serving than that.It's, well if I reach out to people in this industry that are doing interesting things and ask them to grab a cup of coffee, they'll basically block me, whereas if I ask them to, would you like to appear on my podcast, they'll clear time on their schedule. I almost didn't care if my microphone was on or not when I was doing these just because it was a chance to talk to really interesting people and borrow their brain, people reached out asking they can sponsor it, along with the newsletter and the rest, and it's you want to give me money? Of course, you can give me money. How much money? And that sort of turned into a snowball effect over time.Five years in, it's turned into something that I would never have predicted or expected. But it's weird to me still, how effective doing something you're actually passionate about as a side project can sort of grow wings on its own. Where do you stand on that?Emma: Yeah, it's funny because with the exception of the online courses that I've worked with—I mentioned LinkedIn Learning and Frontend Masters, which I knew were paid opportunities—none of my side projects started out for financial reasonings. The podcast that we started was purely for fun, and the sponsors came to us. Now, I will say right up front, we all had pretty big social media followings, and my first piece of advice to anyone looking to get into side projects is, don't focus so much on making money at the get-go. Yes, to your point, Corey, focus on the stuff you're passionate about. Focus on engaging with people on social media, build up your social media, and at that point, okay, monetization will slowly find its way to you.But yeah, I say if you can monetize the heck out of your work, go for it. But also, free content is also great. I like to balance my paid content with my free content because I recognize that not everyone can afford to pay for some of this information. So, I generally always have free alternatives. And for this book that we published, one of the things that was really important to me was keeping it affordable.The first publish I did was $10 for the book. It was like a 250-page book. It was, like, $10 because again, I was not in it for the money. And when I redid the book with the egghead.io team, the same team that did Epic React with Kent C. Dodds, I said, “I want to keep this affordable.” So, we made sure it was still affordable, but also that we had—what's it called? Parity pricing? Pricing parity, where depending on your geographic location, the price is going to accommodate for how the currency is doing. So, yes, I would agree. Side project income for me allows me to do incredible stuff, but it wasn't why I got into it in the first place. It was genuinely just a nice-to-have.Corey: I haven't really done anything that asks people for money directly. I mean, yeah, I sell t-shirts on the website, and mugs, and drink umbrellas—don't get me started—but other than that and the charity t-shirt drive I do every year, I tend to not be good at selling things that don't have a comma in the price tag. For me, it was about absolutely building an audience. I tend to view my Twitter follower count as something of a proxy for it, but the number I actually care about, the audience that I'm focused on cultivating, is newsletter subscribers because no social media platform that we've ever seen has lasted forever. And I have to imagine that Twitter will one day wane as well.But email has been here since longer than we'd been alive, and by having a list of email addresses and ways I can reach out to people on an ongoing basis, I can monetize that audience in a more direct way, at some point should I need them to. And my approach has been, well, one, it's a valuable audience for some sponsors, so I've always taken the asking corporate people for money is easier than asking people for personal money, plus it's a valuable audience to them, so it tends to blow out a number of the metrics that you would normally expect of, oh, for this audience size, you should generally be charging Y dollars. Great. That makes sense if you're slinging mattresses or free web hosting, but when it's instead, huh, these people buy SaaS enterprise software and implement it at their companies, all of economics tend to start blowing apart. Same story with you in many respects.The audience that you're building is functionally developers. That is a lucrative market for the types of sponsors that are wise enough to understand that—in a lot of cases these days—which product a company is going to deploy is not dictated by their exec so much as it is the bottom-up adoption path of engineers who like the product.Emma: Mm-hm. Yeah, and I think once I got to maybe around 10,000 Twitter followers is when I changed my mentality and I stopped caring so much about follower count, and instead I just started caring about the people that I was following. And the number is a nice-to-have but to be honest, I don't think so much about it. And I do understand, yes, at that point, it is definitely a privilege that I have this quote-unquote, “Platform,” but I never see it as an audience, and I never think about that “Audience,” quote-unquote, as a marketing platform. But it's funny because there's no right or wrong. People will always come to you and be like, “You shouldn't monetize your stuff.” And it's like—Corey: “Cool. Who's going to pay me then? Not you, apparently.”Emma: Yeah. It's also funny because when I originally sold the book, it was $10 and I got so many people being like, “This is way too cheap. You should be charging more.” And I'm like, “But I don't care about the money.” I care about all the people who are unemployed and not able to survive, and they have families, and they need to get a job and they don't know how.That's what I care about. And I ended up giving away a lot of free books. My mantra was like, hey if you've been laid off, DM me. No questions asked, I'll give it to you for free. And it was nice because a lot of people came back, even though I never asked for it, they came back and they wanted to purchase it after the fact, after they'd gotten a job.And to me that was like… that was the most rewarding piece. Not getting their money; I don't care about that, but it was like, “Oh, okay. I was actually able to help you.” That is what's really the most rewarding. But yeah, certainly—and back really quickly to your email point, I highly agree, and one of the first things that I would recommend to anyone looking to start a side product, create free content so that you have a backlog that people can look at to… kind of build trust.Corey: Give it away for free, but also get emails from people, like a trade for that. So, it's like, “Hey, here's a free guide on how to start a podcast from scratch. It's free, but all I would like is your email.” And then when it comes time to publish a course on picking the best audio and visual equipment for that podcast, you have people who've already been interested in this topic that you can now market to.This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of "Hello, World" demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking databases, observability, management, and security.And - let me be clear here - it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself all while gaining the networking load, balancing and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build.With Always Free you can do things like run small scale applications, or do proof of concept testing without spending a dime. You know that I always like to put asterisks next to the word free. This is actually free. No asterisk. Start now. Visit https://snark.cloud/oci-free that's https://snark.cloud/oci-free.Corey: I'm not sitting here trying to judge anyone for the choices that they make at all. There are a lot of different paths to it. I'm right there with you. One of the challenges I had when I was thinking about, do I charge companies or do I charge people was that if I'm viewing it through a lens of audience growth, well, what stuff do I gate behind a paywall? What stuff don't I? Well, what if I just—Emma: Mm-hm.Corey: —gave it all away? And that way I don't have to worry about the entire class of problems that you just alluded to of, well, how do I make sure this is fair? Because a cup of coffee in San Francisco is, what, $14 in some cases? Whereas that is significant in places that aren't built on an economy of foolishness. How do you solve for that problem? How do you deal with the customer service slash piracy issues slash all the other nonsense? And it's just easier.Emma: Yeah.Corey: Something I've found, too, is that when you're charging enough money to companies, you don't have to deal with an entire class of customer service problem. You just alluded to the other day that well, you had someone who bought your book and was displeased that it wasn't a how to write code from scratch tutorial, despite the fact that he were very clear on what it is and what it isn't. I don't pretend to understand that level of entitlement. If I spend 10 or 20 bucks on an ebook, and it's not very good, let's see, do I wind up demanding a refund from the author and making them feel bad about it, or do I say, “The hell with it.” And in my case, I—there is privilege baked into this; I get that, but it's I don't want to make people feel bad about what they've built. If I think there's enough value to spend money on it I view that as a one-way transaction, rather than chasing someone down for three months, trying to get a $20 refund.Emma: Yeah, and I think honestly, I don't care so much about giving refunds at all. We have a 30-day money-back guarantee and we don't ask any questions. I just asked this person for feedback, like, “Oh, what was not up to par?” And it was just, kind of like, BS response of like, “Oh, I didn't read the website and I guess it's not what I wanted.” But the end of the day, they still keep the product.The thing is, you can't police all of the people who are going to try to get your content for free if you're charging for it; it's part of it. And I knew that when I got into it, and honestly, my thing is, if you are circulating a book that helps you get a job in tech and you're sending it to all your friends, I'm not going to ask any questions because it's very much the sa—and this is just my morals here, but if I saw someone stealing food from a grocery store, I wouldn't tell on them because at the end of the day, if you're s—Corey: Same story. You ever see someone's stealing baby formula from a store? No, you didn't.Emma: Right.Corey: Keep walking. Mind your business.Emma: Exactly. Exactly. So, at the end of the day, I didn't necessarily care that—people are like, “Oh, people are going to share your book around. It's a PDF.” I'm like, “I don't care. Let them. It is what it is. And the people who wants to support and can, will.” But I'm not asking.I still have free blogs on data structures, and algorithms, and the interview stuff. I do still have content for free, but if you want more, if you want my illustrated diagrams that took me forever with my Apple Pencil, fair enough. That would be great if you could support me. If not, I'm still happy to give you the stuff for free. It is what it is.Corey: One thing that I think is underappreciated is that my resume doesn't look great. On paper, I have an eighth-grade education, and I don't have any big tech names on my resume. I have a bunch of relatively short stints; until I started this place, I've never lasted more than two years anywhere. If I apply through the front door the way most people do for a job, I will get laughed out of the room by the applicant tracking system, automatically. It'll never see a human.And by doing all these side projects, it's weird, but let's say that I shut down the company for some reason, and decide, ah, I'm going to go get a job now, my interview process—more or less, and it sounds incredibly arrogant, but roll with it for a minute—is, “Don't you know who I am? Haven't you heard of me before?” It's, “Here's my website. Here's all the stuff I've been doing. Ask anyone in your engineering group who I am and you'll see what pops up.”You're in that same boat at this point where your resume is the side projects that you've done and the audience you've built by doing it. That's something that I think is underappreciated. Even if neither one of us made a dime through direct monetization of things that we did, the reputational boost to who we are and what we do professionally seems to be one of those things that pays dividends far beyond any relatively small monetary gain from it.Emma: Absolutely, yeah. I actually landed my job interview with Spotify through Twitter. I was contacted by a design systems manager. And I was in the interview process for them, and I ended up saying, “You know, I'm not ready to move to Stockholm. I just moved to Germany.”And a year later, I circled back and I said, “Hey, are there any openings?” And I ended up re-interviewing, and guess what? Now, I have a beautiful home with my soulmate and we're having a child. And it's funny how things work out this way because I had a Twitter account. And so don't undervalue [laugh] social media as a tool in lieu of a resume because I don't think anyone at Spotify even saw my resume until it actually accepted the job offer, and it was just a formality.So yeah, absolutely. You can get a job through social media. It's one of the easiest ways. And that's why if I ever see anyone looking for a job on Twitter, I will retweet, and vouch for them if I know their work because I think that's one of the quickest ways to finding an awesome candidate.Corey: Back in, I don't know, 2010, 2011-ish. I was deep in the IRC weed. I was network staff on the old freenode network—not the new terrible one. The old, good one—and I was helping people out with various things. I was hanging out in the Postfix channel and email server software thing that most people have the good sense not to need to know anything about.And someone showed up and was asking questions about their config, and I was working with them, and teasing them, and help them out with it. And at the end of it, his comment was, “Wow, you're really good at this. Any chance you'd be interested in looking for jobs?” And the answer was, “Well, sure, but it's a global network. Where are you?”Well, he was based in Germany, but he was working remotely for Spotify in Stockholm. A series of conversations later, I flew out to Stockholm and interviewed for a role that they decided I was not a fit for—and again, they're probably right—and I often wonder how my life would have gone differently if the decision had gone the other way. I mean, no hard feelings, please don't get me wrong, but absolutely, helping people out, interacting with people over social networks, or their old school geeky analogs are absolutely the sorts of things that change lives. I would never have thought to apply to a role like that if I had been sitting here looking at job ads because who in the world would pick up someone with relatively paltry experience and move them halfway around the world? This was like a fantasy, not a reality.Emma: [laugh].Corey: It's the people you get to know—Emma: Yeah.Corey: —through these social interactions on various networks that are worth… they're worth gold. There's no way to describe it other than that.Emma: Yeah, absolutely. And if you're listening to this, and you're discouraged because you got turned down for a job, we've all been there, first of all, but I remember being disappointed because I didn't pass my first round of interviews of Google the first time I interviewed with them, and being, like, “Oh, crap, now I can't move to Munich. What am I going to do with my life?” Well, guess what, look where I am today. If I had gotten that job that I thought was it for me, I wouldn't be in the happiest phase of my life.And so if you're going through it—obviously, in normal circumstances where you're not frantically searching for a job; if you're in more of a casual life job search—and you've been let go from the process, just realize that there's probably something bigger and better out there for you, and just focus on your networking online. Yeah, it's an invaluable tool.Corey: One time when giving a conference talk, I asked, “All right, raise your hand if you have never gone through a job interview process and then not been offered the job.” And a few people did. “Great. If your hand is up, aim higher. Try harder. Take more risks.”Because fundamentally, job interviews are two-way streets and if you are only going for the sure thing jobs, great, stretch yourself, see what else is out there. There's no perfect attendance prize. Even back in school there wasn't. It's the idea of, “Well, I've only ever taken the easy path because I don't want to break my streak.” Get over it. Go out and interview more. It's a skill, unlike most others that you don't get to get better at unless you practice it.So, you've been in a job for ten years, and then it's time to move on—I've talked to candidates like this—their interview skills are extremely rusty. It takes a little bit of time to get back in the groove. I like to interview every three to six months back when I was on the job market. Now that I, you know, own the company and have employees, it looks super weird if I do it, but I miss it. I miss those conversations. I miss the aspects—Emma: Yes.Corey: —of exploring what the industry cares about.Emma: Absolutely. And don't underplay the importance of studying the foundational language concepts. I see this a lot in candidates where they're so focused on the newest and latest technologies and frameworks, that they forgot foundational JavaScript, HTML, and CSS. Many companies are focused primarily on these plain language concepts, so just make sure that when you are ready to get back into interviewing and enhance that skill, that you don't neglect the foundation languages that the web is built on if you're a web developer.Corey: I'd also take one last look around and realize that every person you admire, every person who has an audience, who is a known entity in the space only has that position because someone, somewhere did them a favor. Probably lots of someones with lots of favors. And you can't ever pay those favors back. All you can do is pay it forward. I repeatedly encourage people to reach out to me if there's something I can do to help. And the only thing that surprises me is how few people in the audience take me up on that. I'm talking to you, listener. Please, if I can help you with something, please reach out. I get a kick out of doing that sort of thing.Emma: Absolutely. I agree.Corey: Emma, thank you so much for taking the time to speak with me today. If people want to learn more, where can they find you?Emma: Well, you can find me on Twitter. It's just @EmmaBostian, I'm, you know, shitposting over there on the regular. But sometimes I do tweet out helpful things, so yeah, feel free to engage with me over there. [laugh].Corey: And we will, of course, put a link to that in the [show notes 00:35:42]. Thank you so much for taking the time to speak with me today. I appreciate it.Emma: Yeah. Thanks for having me.Corey: Emma Bostian, software engineer at Spotify and oh, so very much more. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an incoherent ranting comment mentioning that this podcast as well failed to completely teach you JavaScript.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

In this Hasty Treat, Scott and Wes talk about how much modern web development has taken from PHP! Freshbooks - Sponsor Get a 30 day free trial of Freshbooks at freshbooks.com/syntax and put SYNTAX in the “How did you hear about us?” section. LogRocket - Sponsor LogRocket lets you replay what users do on your site, helping you reproduce bugs and fix issues faster. It's an exception tracker, a session re-player and a performance monitor. Get 14 days free at logrocket.com/syntax. Show Notes 03:56 - Why much of modern web development is just recreating PHP Everyone loves to hate on PHP, but modern Web dev takes a lot from PHP 05:44 - Mixing templating and logic We do this with JSX 07:39 - Each request has its own scope 08:57 - Massive standard lib Format a date? No sweat! Image resizing? Sure! Audio bindings? Sure! 10:16 - URL-based routing Next.js pages Serverless functions 11:13 - Server-rendered Hotwire 11:38 - $_GET, $_POST, are just available Next.js hooks 12:29 - Variable interpolation 12:59 - All-in-one frameworks Laravel did it CakePHP CodeIgnighter 13:32 - Direct DB access SQL statements 14:37 - Why do people hate PHP? WordPress Inconsistent API Their first code was PHP and they sucked PHP has come a long way It used to not be safe Blocking by default - no async/await 17:48 - Why is JS still better? Shared code between frontend and backend Single language Huge ecosystem (could be a con) Links Syntax 267: Hasty Treat - Turbolinks + Server Generated HTML + JS Sprinkles https://vuejs.org/ https://www.hey.com/ Tweet us your tasty treats! Scott's Instagram LevelUpTutorials Instagram Wes' Instagram Wes' Twitter Wes' Facebook Scott's Twitter Make sure to include @SyntaxFM in your tweets

Applications write data to persistent storage like a database.  The most popular database query language is SQL which has many similar dialects.  SQL is expressive and powerful for describing what data you want.  What you do with that data requires a solution in the form of a data pipeline.  Ideally, these analytical workflows can follow The post DBT: Data Build Tool with Tristan Handy appeared first on Software Engineering Daily.