Podcasts about Serverless Framework

Play Episode Listen Later Jun 30, 2022 33:34

About ChrisChris is a robotics engineer turned cloud security practitioner. From building origami robots for NASA, to neuroscience wearables, to enterprise software consulting, he is a passionate builder at heart. Chris is a cofounder of Common Fate, a company with a mission to make cloud access simple and secure.Links: Common Fate: https://commonfate.io/ Granted: https://granted.dev Twitter: https://twitter.com/chr_norm TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Let's face it, on-call firefighting at 2am is stressful! So there's good news and there's bad news. The bad news is that you probably can't prevent incidents from happening, but the good news is that incident.io makes incidents less stressful and a lot more valuable. incident.io is a Slack-native incident management platform that allows you to automate incident processes, focus on fixing the issues and learn from incident insights to improve site reliability and fix your vulnerabilities. Try incident.io, recover faster and sleep more.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate. Is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other; which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at honeycomb.io/screaminginthecloud. Observability: it's more than just hipster monitoring.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. It doesn't matter where you are on your journey in cloud—you could never have heard of Amazon the bookstore—and you encounter AWS and you spin up an account. And within 20 minutes, you will come to the realization that everyone in this space does. “Wow, logging in to AWS absolutely blows goats.”Today, my guest, obviously had that reaction, but unlike most people I talked to, decided to get up and do something about it. Chris Norman is the co-founder of Common Fate and most notably to how I know him is one of the original authors of the tool, Granted. Chris, thank you so much for joining me.Chris: Hey, Corey, thank you for having me.Corey: I have done podcasts before; I have done a blog post on it; I evangelize it on Twitter constantly, and even now, it is challenging in a few ways to explain holistically what Granted is. Rather than trying to tell your story for you, when someone says, “Oh, Granted, that seems interesting and impossible to Google for in isolation, so therefore, we know it's going to be good because all the open-source projects with hard to find names are,” what is Granted and what does it do?Chris: Granted is a command-line tool which makes it really easy for you to get access and assume roles when you're working with AWS. For me, when I'm using Granted day-to-day, I wake up, go to my computer—I'm working from home right now—crack open the MacBook and I log in and do some development work. I'm going to go and start working in the cloud.Corey: Oh, when I start first thing in the morning doing development work and logging into the cloud, I know. All right, I'm going to log in to AWS and now I know that my day is going downhill from here.Chris: [laugh]. Exactly, exactly. I think maybe the best days are when you don't need to log in at all. But when you do, I go and I open my terminal and I run this command. Using Granted, I ran this assume command and it authenticates me with single-sign-on into AWS, and then it opens up a console window in a particular account.Now, you might ask, “Well, that's a fairly standard thing.” And in fact, that's probably the way that the console and all of the tools work by default with AWS. Why do you need a third-party tool for this?Corey: Right. I've used a bunch of things that do varying forms of this and unlike Granted, you don't see me gushing about them. I want to be very clear, we have no business relationship. You're not sponsoring anything that I do. I'm not entirely clear on what your day job entails, but I have absolutely fallen in love with the Granted tool, which is why I'm dragging you on to this show, kicking and screaming, mostly to give me an excuse to rave about it some more.Chris: [laugh]. Exactly. And thank you for the kind words. And I'd say really what makes it special or why I've been so excited to be working on it is that it makes this access, particularly when you're working with multiple accounts, really, really easy. So, when I run assume and I open up that console window, you know, that's all fine and that's very similar to how a lot of the other tools and projects that are out there work, but when I want to open that second account and that second console window, maybe because I'm looking at like a development and a staging account at the same time, then Granted allows me to view both of those simultaneously in my browser. And we do that using some platform sort of tricks and building into the way that the browser works.Corey: Honestly, one of the biggest differences in how you describe what Granted is and how I view it is when you describe it as a CLI application because yes, it is that, but one of the distinguishing characteristics is you also have a Firefox extension that winds up leveraging the multi-container functionality extension that Firefox has. So, whenever I wind up running a single command—assume with a-c' flag, then I give it the name of my AWS profile, it opens the web console so I can ClickOps my heart's content inside of a tab that is locked to a container, which means I can have one or two or twenty different AWS accounts and/or regions up running simultaneously side-by-side, which is basically impossible any other way that I've ever looked at it.Chris: Absolutely, yeah. And that's, like, the big differentiating factor right now between Granted and between this sort of default, the native experience, if you're just using the AWS command line by itself. With Granted, you can—with these Firefox containers, all of your cookies, your profile, everything is all localized into that one container. It's actually it's a privacy features that are built into Firefox, which keeps everything really separate between your different profiles. And what we're doing with Granted is that we make it really easy to open a specific profiles that correspond with different AWS profiles that you're using.So, you'd have one which could be your development account, one which could be production or staging. And you can jump between these and navigate between them just as separate tabs in your browser, which is a massive improvement over, you know, what I've previously had to use in the past.Corey: The thing that really just strikes me about this is first, of course, the functionality and the rest, so I saw this—I forget how I even came across it—and immediately I started using it. On my Mac, it was great. I started using it when I was on the road, and it was less great because you built this thing in Go. It can compile and install on almost anything, but there were some assumptions that you had built into this in its early days that did not necessarily encompass all of the use cases that I use. For example, it hadn't really occurred to you that some lunatic would try and only use an iPad when they're on the road, so they have to be able to run this to get federated login links via SSHing into an EC2 instance running somewhere and not have it open locally.You seemed almost taken aback when I brought it up. Like, “What lunatic would do that?” Like, “Hi, I'm such a lunatic. Let's talk about this.” And it does that now, and it's awesome. It does seem to me though, and please correct me if I'm wrong on this assumption slash assessment that this is first and foremost aimed at desktop users, specifically people running Mac on the desktop, is that the genesis of it?Chris: It is indeed. And I think part of the cause behind that is that we originally built a tool for ourselves. And as we were building things and as we were working using the cloud, we were running things—you know, we like to think that we're following best practices when we're using AWS, and so we'd set up multiple accounts, we'd have a special account for development, a separate one for staging, a separate one for production, even internal tools that we would build, we would go and spin up an individual account for those. And then you know, we had lots of accounts. and to go and access those really easily was quite difficult.So, we definitely, we built it for ourselves first and I think that that's part of when we released it, it actually a little bit of cause for some of the initial problems. And some of the feedback that we had was that it's great to build tools for yourself, but when you're working in open-source, there's a lot of different diversity with how people are using things.Corey: We take different approaches. You want to try to align with existing best practices, whereas I am a loudmouth white guy who works in tech. So, what I do definitionally becomes a best practice in the ecosystem. It's easier to just comport with the ones that are already existing that smart people put together rather than just trying to competence your way through it, so you took a better path than I did.But there's been a lot of evolution to Granted as I've been using it for a while. I did a whole write-up on it and that got a whole bunch of eyes onto the project, which I can now admit was a nefarious plan on my part because popping into your community Slack and yelling at you for features I want was all well and good, but let's try and get some people with eyes on this who are smarter than me—which is not that high of a bar when it comes to SSO, and IAM, and federated login, and the rest—and they can start finding other enhancements that I'll probably benefit from. And sure enough, that's exactly what happened. My sneaky plan has come to fruition. Thanks for being a sucker, I guess. I mean—[laugh] it worked. I'm super thrilled by the product.Chris: [laugh]. I guess it's a great thing I think that the feedback and particularly something that's always been really exciting is just seeing new issues come through on GitHub because it really shows the kinds of interesting use cases and the kinds of interesting teams and companies that are using Granted to make their lives a little bit easier.Corey: When I go to the website—which again is impossible to Google—the website for those wondering is granted.dev. It's short, it's concise, I can say it on a podcast and people automatically know how to spell it. But at the top of the website—which is very well done by the way—it mentions that oh, you can, “Govern access to breakglass roles with Common Fate Cloud,” and it also says in the drop shadow nonsense thing in the upper corner, “Brought to you by Common Fate,” which is apparently the name of your company.So, the question I'll get to in a second is what does your company do, but first and foremost, is this going to be one of those rug-pull open-source projects where one day it's, “Oh, you want to log into your AWS accounts? Insert quarter to continue.” I'm mostly being a little over the top with that description, but we've all seen things that we love turn into molten garbage. What is the plan around this? Are you about to ruin this for the rest of us once you wind up raising a round or something? What's the deal?Chris: Yeah, it's a great question, Corey. And I think that to a degree, releasing anything like this that sits in the access workflow and helps you assume roles and helps you day-to-day, you know, we have a responsibility to uphold stability and reliability here and to not change things. And I think part of, like, not changing things includes not [laugh] rug-pulling, as you've alluded to. And I think that for some companies, it ends up that open-source becomes, like, a kind of a lead-generation tool, or you end up with, you know, now finally, let's go on add another login so that you have to log into Common Fate to use Granted. And I think that, to be honest, a tool like this where it's all about improving the speed of access, the incentives for us, like, it doesn't even make sense to try and add another login for to try to get people to, like, to say, login to Common Fate because that would make your signing process for AWS take even longer than it already does.Corey: Yeah, you decided that you know, what's the biggest problem? Oh, you can sleep at night, so let's go ahead and make it even worse, by now I want you to be this custodian of all my credentials to log into all of my accounts. And now you're going to be critical path, so if you're down, I'm not able to log into anything. And oh, by the way, I have to trust you with full access to my bank stuff. I just can't imagine that is a direction that you would be super excited about diving head-first into.Chris: No, no. Yeah, certainly not. And I think that the, you know, building anything in this space, and with what we're doing with Common Fate, you know, we're building a cloud platform to try to make IAM a little bit easier to work with, but it's really sensitive around granting any kind of permission and I think that you really do need that trust. So, trying to build trust, I guess, with our open-source projects is really important for us with Granted and with this project, that it's going to continue to be reliable and continue to work as it currently does.Corey: The way I see it, one of the dangers of doing anything that is particularly open-source—or that leans in the direction of building in Amazon's ecosystem—it leads to the natural question of, well, isn't this just going to be some people say stolen—and I don't think those people understand how open-source works—by AWS themselves? Or aren't they going to build something themselves at AWS that's going to wind up stomping this thing that you've built? And my honest and remarkably cynical answer is that, “You have built a tool that is a joy to use, that makes logging into AWS accounts streamlined and efficient in a variety of different patterns. Does that really sound like something AWS would do?” And followed by, “I wish they would because everyone would benefit from that rising tide.”I have to be very direct and very clear. Your product should not exist. This should be something the provider themselves handles. But nope. Instead, it has to exist. And while I'm glad it does, I also can't shake the feeling that I am incredibly annoyed by the fact that it has to.Chris: Yeah. Certainly, certainly. And it's something that I think about a little bit. I like to wonder whether there's maybe like a single feature flag or some single sort of configuration setting in AWS where they're not allowing different tabs to access different accounts, they're not allowing this kind of concurrent access. And maybe if we make enough noise about Granted, maybe one of the engineers will go and flick that switch and they'll just enable it by default.And then Granted itself will be a lot less relevant, but for everybody who's using AWS, that'll be a massive win because the big draw of using Granted is mainly just around being able to access different accounts at the same time. If AWS let you do that out of the box, hey, that would be great and, you know, I'd have a lot less stuff to maintain.Corey: Originally, I had you here to talk about Granted, but I took a glance at what you're actually building over at Common Fate and I'm about to basically hijack slash derail what probably is going to amount the rest of this conversation because you have a quick example on your site for by developers, for developers. You show a quick Python script that tries to access a S3 bucket object and it's denied. You copy the error message, you paste it into what you're building over a Common Fate, and in return, it's like, “Oh. Yeah, this is the policy that fixes it. Do you want us to apply it for you?”And I just about fell out of my chair because I have been asking for this explicit thing for a very long time. And AWS doesn't do it. Their IAM access analyzer claims to. Like, “Oh, just go look at CloudTrail and see what permissions it uses and we'll build a policy to scope it down.” “Okay. So, it's S3 access. Fair enough. To what object or what bucket?” “Guess,” is what it tells you there.And it's, this is crap. Who thinks this is a good user experience? You have built the thing that I wish AWS had built in natively. Because let's be honest here, I do what an awful lot of people do and overscope permissions massively just because messing around with the bare minimum set of permissions in many cases takes more time than building the damn thing in the first place.Chris: Oh, absolutely. Absolutely. And in fact, this—was a few years ago when I was consulting—I had a really similar sort of story where one of the clients that we were working with, the CTO of this company, he was needing to grant us access to AWS and we were needing to build a particular service. And he said, “Okay, can you just let me know the permissions that you will need and I'll go and deploy the role for this.” And I came back and I said, “Wait. I don't even know the permissions that I'm going to need because the damn thing isn't even built yet.”So, we went sort of back and forth around this. And the compromise ended up just being you know, way too much access. And that was sort of part of the inspiration for, you know, really this whole project and what we're building with Common Fate, just trying to make that feedback loop around getting to the right level of permissions a lot faster.Corey: Yeah, I am just so overwhelmingly impressed by the fact that you have built—and please don't take this as a criticism—but a set of very simple tools. Not simple in the terms of, “Oh, that's, like, three lines of bash, and a fool could write that on a weekend.” No. Simple in the sense of it solves a problem elegantly and well and it's straightforward—well, straightforward as anything in the world of access control goes—to wrap your head around exactly what it does. You don't tend to build these things by sitting around a table brainstorming with someone you met at co-founder dating pool or something and wind up figuring out, “Oh, we should go and solve that. That sounds like a billion-dollar problem.”This feels very much like the outcome of when you're sitting around talking to someone and let's start by drinking six beers so we become extraordinarily honest, followed immediately by let's talk about what sucks. What pisses you off the most? It feels like this is sort of the low-hanging fruit of things that upset people when it comes to AWS. I mean, if things had gone slightly differently, instead of focusing on AWS bills, IAM was next on my list of things to tackle just because I was tired of smacking my head into it.This is very clearly a problem space that you folks have analyzed deeply, worked within, and have put a lot of thought into. I want to be clear, I've thrown a lot of feature suggestions that you for Granted from start to finish. But all of them have been around interface stuff and usability and expanding use cases. None of them have been, “Well, that seems screamingly insecure.” Because it hasn't been.Chris: [laugh].Corey: It has been effective, start to finish, I think that from a security posture, you make terrific choices, in many cases better than ones I would have made a starting from scratch myself. Everything that I'm looking at in what you have built is from a position of this is absolutely amazing and it is transformative to my own workflows. Now, how can we improve it?Chris: Mmm. Thank you, Corey. And I'll say as well, maybe around the security angle, that one of the goals with Granted was to try and do things a little bit better than the default way that AWS does them when it comes to security. And it's actually been a bit of a source for challenges with some of the users that we've been working with with Granted because one of the things we wanted to do was encrypt the SSO token. And this is the token that when you sign in to AWS, kind of like, it allows you to then get access to all of the rest of the accounts.So, it's like a pretty—it's a short-lived token, but it's a really sensitive one. And you know, by default, it's just stored in plain text on your disk. So, we dump to a file and, you know, anything that can go and read that, they can go and get it. It's also a little bit hard to revoke and to lock people out. There's not really great workflows around that on AWS's side.So, we thought, “Okay, great. One of the goals for Granted can be that we will go and store this in your keychain in your system and we'll work natively with that.” And that's actually been a cause for a little bit of a hassle for some users, though, because by doing that and by storing all of this information in the keychain, it's actually broken some of the integrations with the rest of the tooling, which kind of expects tokens and things to be in certain places. So, we've actually had to, as part of dealing with that with Granted, we've had to give users the ability to opt out for that.Corey: DoorDash had a problem. As their cloud-native environment scaled and developers delivered new features, their monitoring system kept breaking down. In an organization where data is used to make better decisions about technology and about the business, losing observability means the entire company loses their competitive edge. With Chronosphere, DoorDash is no longer losing visibility into their applications suite. The key? Chronosphere is an open-source compatible, scalable, and reliable observability solution that gives the observability lead at DoorDash business, confidence, and peace of mind. Read the full success story at snark.cloud/chronosphere. That's snark.cloud slash C-H-R-O-N-O-S-P-H-E-R-E.Corey: That's why I find this so, I think, just across the board, fantastic. It's you are very clearly engaged with your community. There's a community Slack that you have set up for this. And I know, I know, too many Slacks; everyone has this problem. This is one of those that is worth hanging in, at least from my perspective, just because one of the problems that you have, I suspect, is on my Mac it's great because I wind up automatically updating it to whatever the most recent one is every time I do a brew upgrade.But on the Linux side of the world, you've discovered what many of us have discovered, and that is that packaging things for Linux is a freaking disaster. The current installation is, “Great. Here's basically a curl bash.” Or, “Here, grab this tarball and install it.” And that's fine, but there's no real way of keeping that updated and synced.So, I was checking the other day, oh wow, I'm something like eight versions behind on this box. But it still just works. I upgraded. Oh, wow. There's new functionality here. This is stuff that's actually really handy. I like this quite a bit. Let's see what else we can do.I'm just so impressed, start to finish, by just how receptive you've been to various community feedbacks. And as well—I want to be very clear on this point, too—I've had folks who actually know what they're doing in an InfoSec sense look at what you're up to, and none of them had any issues of note. I'm sure that they have a pile of things like, with that curl bash, they should really be doing a GPG check. Yes, yes, fine. Whatever. If that's your target threat model, okay, great. Here in reality-land for what I do, this is awesome.And they don't seem to have any problems with, “Oh, yeah. By the way, sending analytics back up”—which, okay, fine, whatever. “And it's not disclosing them.” Okay, that's bad. “And it's including the contents of your AWS credentials.”Ahhhh. I did encounter something that was doing that on the back-end once. [cough]—Serverless Framework—sorry, something caught in my throat for a second.Chris: [laugh].Corey: No faster way I can think of to erode trust in that. But everything you're doing just makes sense.Chris: Oh, I do remember that. And that was a little bit of a fiasco, really, around all of that, right? And it's great to hear actually around that InfoSec folks and security people being, you know, not unhappy, I guess, with a tool like this. It's been interesting for me personally. We've really come from a practitioner's background.You know, I wouldn't call myself a security engineer at all. I would call myself as a sometimes a software developer, I guess. I have been hacking my way around Go and definitely learning a lot about how the cloud has worked over the past seven, eight years or so, but I wouldn't call myself a security engineer, so being very cautious around how all of these things work. And we've really tried to defer to things like the system keychain and defer to things that we know are pretty safe and work.Corey: The thing that I also want to call out as well is that your licensing is under the MIT license. This is not one of those, “Oh, you're required to wind up doing a bunch of branding stuff around it.” And, like some people say, “Oh, you have to own the trademark for all of these things.” I mean, I'm not an expert in international trademark law, let's be very clear, but I also feel that trademarking a term that is already used heavily in the space such as the word ‘Granted,' feels like kind of an uphill battle. And let's further be clear that it doesn't matter what you call this thing.In fact, I will call attention to an oddity that I've encountered a fair bit. After installing it, the first thing you do is you run the command ‘granted.' That sets it up, it lets you configure your browser, what browser you want to use, and it now supports standard out for that headless, EC2 use case. Great. Awesome. Love it. But then the other binary that ships with it is Assume. And that's what I use day-to-day. It actually takes me a minute sometimes when it's been long enough to remember that the tool is called Granted and not Assume what's up with that?Chris: So, part of the challenge that we ran into when we were building the Granted project is that we needed to export some environment variables. And these are really important when you're logging into AWS because you have your access key, your secret key, your session token. All of those, when you run the assume command, need to go into the terminal session that you called it. This doesn't matter so much when you're using the console mode, which is what we mentioned earlier where you can open 100 different accounts if you want to view all of those at the same time in your browser. But if you want to use it in your terminal, we wanted to make it look as really smooth and seamless as possible here.And we were really inspired by this approach from—and I have to shout them out and kind of give credit to them—a tool called AWSume—they're spelled A-W-S-U-M-E—Python-based tool that they don't do as much with single-sign-on, but we thought they had a really nice, like, general approach to the way that they did the scripting and aliasing. And we were inspired by that and part of that means that we needed to have a shell script that called this executable, which then will export things back out into the shell script. And we're doing all this wizardry under the hood to make the user experience really smooth and seamless. Part of that meant that we separated the commands into granted and assume and the other part of the naming for everything is that I felt Granted had a far better ring to it than calling the whole project Assume.Corey: True. And when you say assume, is it AWS or not? I've used the AWSume project before; I've used AWS Vault out of 99 Designs for a while. I've used—for three minutes—the native AWS SSO config, and that is just trash. Again, they're so good at the plumbing, so bad at the porcelain, I think is the criticism that I would levy toward a lot of this stuff.Chris: Mmm.Corey: And it's odd to think there's an entire company built around just smoothing over these sharp, obnoxious edges, but I'm saying this as someone who runs a consultancy and have five years that just fixes the bill for this one company. So, there's definitely a series of cottage industries that spring up around these things. I would be thrilled, on some level, if you wound up being completely subsumed by their product advancements, but it's been 15 years for a lot of this stuff and we're still waiting. My big failure mode that I'm worried about is that you never are.Chris: Yeah, exactly, exactly. And it's really interesting when you think about all of these user experience gaps in AWS being opportunities for, I guess, for companies like us, I think, trying to simplify a lot of the complexity for things. I'm interested in sort of waiting for a startup to try and, like, rebuild the actual AWS console itself to make it a little bit faster and easier to use.Corey: It's been done and attempted a bunch of different times. The problem is that the console is a lot of different things to a lot of different people, and as you step through that, you can solve for your use case super easily. “Yeah, what do I care? I use RDS, I use some VPC nonsense, and I use EC2. The end.” “Great. What about IAM?”Because I promise you're using that whether you know it or not. And okay, well, I'm talking to someone else who's DynamoDB, and someone else is full-on serverless, and someone else has more money than sense, so they mostly use SageMaker, and so on and so forth. And it turns out that you're effectively trying to rebuild everything. I don't know if that necessarily works.Chris: Yeah, and I think that's a good point around maybe while we haven't seen anything around that sort of space so far. You go to the console, and you click down, you see that list of 200 different services and all of those have had teams go and actually, like, build the UI and work with those individual APIs. Yeah.Corey: Any ideas as far as what's next for features on Granted?Chris: I think that, for us, it's continuing to work with everybody who's using it, and with a focus of stability and performance. We actually had somebody in the community raise an issue because they have an AWS config file that's over 7000 lines long. And I kind of pity that person, potentially, for their day-to-day. They must deal with so much complexity. Granted is currently quite slow when the config files get very big. And for us, I think, you know, we built it for ourselves; we don't have that many accounts just yet, so working to try to, like, make it really performant and really reliable is something that's really important.Corey: If you don't mind a feature request while we're at it—and I understand that this is more challenging than it looks like—I'm willing to fund this as a feature bounty that makes sense. And this also feels like it might be a good first project for a very particular type of person, I would love to get tab completion working in Zsh. You have it—Chris: Oh.Corey: For Fish because there's a great library that automatically populates that out, but for the Zsh side of it, it's, “Oh, I should just wind up getting Zsh completion working,” and I fell down a rabbit hole, let me tell you. And I come away from this with the perception of yeah, I'm not going to do it. I have not smart enough to check those boxes. But a lot of people are so that is the next thing I would love to see. Because I will change my browser to log into the AWS console for you, but be damned if I'm changing my shell.Chris: [laugh]. I think autocomplete probably should be higher on our roadmap for the tool, to be honest because it's really, like, a key metric and what we're focusing on is how easy is it to log in. And you know, if you're not too sure what commands to use or if we can save you a few keystrokes, I think that would be the, kind of like, reaching our goals.Corey: From where I'm sitting, you definitely have. I really want to thank you for taking the time to not only build this in the first place, but also speak with me about it. If people want to learn more, where's the best place to find you?Chris: So, you can find me on Twitter, I'm @chr_norm, or you can go and visit granted.dev and you'll have a link to join the Slack community. And I'm very active on the Slack.Corey: You certainly are, although I will admit that I fall into the challenge of being in just the perfectly opposed timezone from you and your co-founder, who are in different time zones to my understanding; one of you is on Australia and one of you was in London; you're the London guy as best I'm aware. And as a result, invariably, I wind up putting in feature requests right when no one's around. And, for better or worse, in the middle of the night is not when I'm usually awake trying to log into AWS. That is Azure time.Chris: [laugh]. Yeah, no, we don't have the US time zone properly covered yet for our community support and help. But we do have a fair bit of the world timezone covered. The rest of the team for Common Fate is all based in Australia and I'm out here over in London.Corey: Yeah. I just want to thank you again, for just being so accessible and, like, honestly receptive to feedback. I want to be clear, there's a way to give feedback and I do strive to do it constructively. I didn't come crashing into your Slack one day with a, “You know what your problem is?” I prefer to take the, “This is awesome. Here's what I think would be even better. Does that make sense?” As opposed to the imperious demands and GitHub issues and whatnot? It's, “I'd love it if it did this thing. Doesn't do this thing. Can you please make it do this thing?” Turns out that's the better way to drive change. Who knew?Chris: Yeah. [laugh]. Yeah, definitely. And I think that one of the things that's been the best around our journey with Granted so far has been listening to feedback and hearing from people how they would like to use the tool. And a big thank you to you, Corey, for actually suggesting changes that make it not only better for you, but better for everybody else who's using Granted.Corey: Well, at least as long as we're using my particular byzantine workload patterns in some way, or shape, or form, I'll hear that. But no, it's been an absolute pleasure and I really want to thank you for your time as well.Chris: Yeah, thank you for having me.Corey: Chris Norman, co-founder of Common Fate, as well as one of the two primary developers originally behind the Granted project that logs you into AWS without you having to lose your mind. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry, incensed, raging comment that talks about just how terrible all of this is once you spend four hours logging into your AWS account by hand first.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

love amazon australia google simple mit drop nasa cloud mac fate ipads context i am cto slack granted designs screaming python ui doordash linux aws github assume apis macbook devops azure firefox dns s3 govern functionality infosec rds honeycomb cli observability sso ec2 slacks vpc dynamodb corey quinn gpg chronosphere chris norman chris yeah chris so chris oh sagemaker chris no zsh serverless framework duckbill group chief cloud economist last week in aws humblepod

Serverless Should be Simple with Tomasz Łakomy

Play Episode Listen Later May 10, 2022 38:43

About TomaszTomasz is a Frontend Engineer at Stedi, Co-Founder/Head of React at Cloudash, egghead.io instructor with over 200 lessons published, a tech speaker, an AWS Community Hero and a lifelong learner.Links Referenced: Cloudash: https://cloudash.dev/ Twitter: https://twitter.com/tlakomy TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate. Is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other; which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at honeycomb.io/screaminginthecloud. Observability: it's more than just hipster monitoring.Corey: This episode is sponsored in part by our friends at ChaosSearch. You could run Elasticsearch or Elastic Cloud—or OpenSearch as they're calling it now—or a self-hosted ELK stack. But why? ChaosSearch gives you the same API you've come to know and tolerate, along with unlimited data retention and no data movement. Just throw your data into S3 and proceed from there as you would expect. This is great for IT operations folks, for app performance monitoring, cybersecurity. If you're using Elasticsearch, consider not running Elasticsearch. They're also available now in the AWS marketplace if you'd prefer not to go direct and have half of whatever you pay them count towards your EDB commitment. Discover what companies like Equifax, Armor Security, and Blackboard already have. To learn more, visit chaossearch.io and tell them I sent you just so you can see them facepalm, yet again.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. It's always a pleasure to talk to people who ask the bold questions. One of those great bold questions is, what if CloudWatch's web page didn't suck? It's a good question. It's one I ask myself all the time.And then I stumbled across a product that wound up solving this for me, and I'm a happy customer. To be clear, they're not sponsoring anything that I do, nor should they. It's one of those bootstrapped, exciting software projects called Cloudash. Today, I'm joined by the Head of React at Cloudash, Tomasz Łakomy. Tomasz, thank you for joining me.Tomasz: It's a pleasure to be here.Corey: So, where did this entire idea come from? Because I sit and I get upset every time I have to go into the CloudWatch dashboard because first, something's broken. In an ideal scenario, I don't have to care about monitoring or observability or anything like that. But then it's quickly overshadowed by the fact that this interface is terrible. And the reason I know it's terrible is that every time I'm in there, I feel dumb.My belief is—for the longest time, I thought that was a problem with me. But no, invariably, when you wind up working with something and consistently finding it a bad—you don't know enough to solve for it, it's not you. It is, in fact, the signs of a poorly designed experience, start to finish. “You should be smarter to use this tool,” is very rarely correct. And there are a bunch of observability tools and monitoring tools for serverless things that have made sense over the years and made this easier, but one of the most—and please don't take this the wrong way—stripped down, bare essentials of just the facts, style of presentation is Cloudash. It's why I continue to pay for it every month with a smile on my face. How did you get here from there?Tomasz: Yeah that's a good question. I would say that. Cloudash was born out of desire for simple things to be simple. So, as you mentioned, Cloudash is basically the monitoring and troubleshooting tool for serverless applications, made for serverless developers because I am very much into serverless space, as is Maciej Winnicki, who is the another half of Cloudash team. And, you know, the whole premise of serverless was things are going to be simpler, right?So, you know, you have a bunch of code, you're going to dump it into a Lambda function, and that's it. You don't have to care about servers, you don't have to care about, you know, provisioning stuff, you don't have to care about maintenance, and so on. And that is not exactly true because why PagerDuty still continues to be [unintelligible 00:02:56] business even in serverless spaces. So, you will get paged every now and then. The problem is—what we kind of found is once you have an incident—you know, PagerDuty always tends to call it in the middle of the night; it's never, like, 11 a.m. during the workday; it's always the middle of the night.Corey: And no one's ever happy when it calls them either. It's, “Ah, hell.” Whatever it rings, it's yeah, the original Call of Duty. PagerDuty hooked up to Nagios. I am old enough to remember those days.Tomasz: [unintelligible 00:03:24] then business, like, imagine paying for something that's going to wake you up in the middle of the night. It doesn't make sense. In any case—Corey: “So, why do you pay for that product? Because it's really going to piss me off.” “Okay, well… does that sound like a good business to you? Well, AWS seems to think so. No one's happy working with that stuff.” “Fair. Fair enough.”Tomasz: So, in any case, like we've established an [unintelligible 00:03:43]. So you wake up, you go to AWS console because you saw a notification that this-and-this API has, you know, this threshold was above it, something was above the threshold. And then you go to the CloudWatch console. And then you see, okay, those are the logs, those are the metrics. I'm going to copy this request ID. I'm going to go over here. I'm going to go to X-Ray.And again, it's 3 a.m. so you don't exactly remember what do you investigate; you have, like, ten minutes. And this is a problem. Like, we've kind of identified that it's not simple to do these kinds of things, too—it's not simple to open something and have an understanding, okay, what exactly is happening in my serverless app at this very moment? Like, what's going on?So, we've built that. So, Cloudash is a desktop app; it lives on your machine, which is a single pane of glass. It's a single pane of glass view into your serverless system. So, if you are using CloudFormation in order to provision something, when you open Cloudash, you're going to see, you know, all of the metrics, all the Lambda functions, all of the API Gateways that you have provisioned. As of yesterday, API Gateway is no longer cool because they did launch the direct integration, so you have—you can call Lambda functions with [crosstalk 00:04:57]—Corey: Yeah, it's the one they released, and then rolled back and somehow never said a word—because that's an AWS messaging story, and then some—right around re:Invent last year. And another quarter goes by and out it goes.Tomasz: It's out yesterday.Corey: Yeah, it's terrific. I love that thing. The only downside to it is, ah, you have to use one of their—you have to use their domain; no custom domain support. Really? Well, you can hook up CloudFront to it, but the pricing model that way makes it more expensive than API Gateway.Okay, so I could use Cloudflare in front of it, and then it becomes free, so I bought a domain just for that purpose. That's right, my serverl—my direct Lambda URLs now live behind the glorious domain of cheapass.cloud because of course. They are. It's a day-one product from AWS, so of course, it's not feature-complete.But one of the things I like about the serverless model, and it's also a challenge when it comes to troubleshooting stuff is that it's very much set it and forget it style because serverless in many cases, at least the way that I tend to use it, is back-office stuff, its back-end things, it's processing on things that are not necessarily always direct front and center. So, these things can run on their own for years until finally, you find a strange bug in a new use case, or you want to go and change something. And then it's how the hell did this ever work? And it's still working, kind of, but what fool built this? Of course, it was me; it's always me.But what happened here? You're basically excavating your own legacy code, trying to understand what's going on. And so, you're already upset then. Cloudash makes this easier to find the things, to navigate through a whole bunch of different accounts. And there are a bunch of decisions that you made while building the app that are so clearly correct, that I get actively annoyed when others don't because oh, it looks at your AWS configuration file in your user home directory. Great, awesome. It's a desktop app, but it still consults that file. Yay, integration between ClickOps and the terminal. Wonderful.But ah, use SSO for a lot of stuff, so that's going to fix your little red wagon. I click on that app, and suddenly, bam, a browser opens asking me to log in and authenticate, allow the request. It works, and then suddenly, it goes back to doing exactly what you'd expect it to. It's really nice. The affordances behind this are glorious.Tomasz: Like I said, one of our kind of design goals when building Cloudash was to make simple things simple again. The whole purpose is to make sure that you can get into the root cause of an issue within, like, five minutes, if not less. And this is kind of the app that you're going to tend to open whenever that—as I said, because some of the systems can be around for, like, ages, literally without any incident whatsoever, then the data is going to change because somebody [unintelligible 00:07:30] got that the year is 2020 and off you go, we have an incident.But what's important about Cloudash is that we don't send logs anywhere. And that's kind of important because you don't pay for [PUT 00:07:42] metric API because we are not sending those logs anywhere. If you install Cloudash on your machine, we are not going to get your logs from the last ten years, put them in into a system, charge you for that, just so you are able to, you know, find out what happened in this particular hour, like, two weeks ago. We genuinely don't care about your logs; we have enough of our own logs at work to, you know, to analyze, to investigate, and so on; we are not storing them anywhere.In fact, you know, whatever happens on your machine stays on the machine. And that is partially why this is a desktop app. Because we don't want to handle your credentials. We don't—absolutely, we don't want you to give us any of your credentials or access keys, you know, whatever. We don't want that.So, that is why you install Cloudash, it's going to run on your machine, it's going to use your local credentials. So, it's… effectively, you could say that this is a much more streamlined and much more laser-focused browser or like, an eye into AWS systems, which live on the serverless side of things.Corey: I got to deal with it in a bit of an interesting way, recently. I have a detector in my company's production AWS org, to detect when ClickOps is afoot. Now, I'm a big proponent of ClickOps, but I also want to know what's going on, so I have a whole thing that [runs detects 00:09:04] when people are doing things in the console versus via API. And it alerts on certain subsets of them. I had to build a special case for the user agent string coming out of Cloudash because no, no, this is an app, this is not technically ClickOps—it is also read-only, which is neither here nor there, to my understanding.But it was, “Oh yeah, this is effectively an Electron app.” It just wraps, effectively, a browser and presents that as an application. And cool. From my perspective, that's an implementation detail. It feels like a native app—because it is—and I can suddenly see the things I care about in a way that is much more straightforward without having to have four different browser tabs open where, okay, here's the CloudTrail log for this thing, here's the metrics next to it. Oh, those are two separate windows already, and so on and so forth. It just makes hunting down to the obnoxious problems so much nicer.It's also, you're one of those rare products where if I don't use it for a month, I don't get the bill at the end of the month and think, “Ooh, that's going to—did I waste the money?” It's no, nice. I had a whole month where I didn't have to mess with this. It's great.Tomasz: Exactly. I feel like, you know, it's one of those systems where, as you said, we send you an email at the end of every month that we're going to charge you X dollars for the month—by the way, we have fixed pricing and then you can cancel anytime—and it's like one of those things that, you know, I didn't have to open this up for a month. This is awesome because I didn't have any incidents. But I know whenever again, PagerDuty is going to decide, “Hey, dude, wake up. You know, if slept for three hours. That is definitely long enough,” then you know that; you know, this app is there and you can use that.We very much care about, you know, building this stuff, not only for our customers, but we also use that on a daily basis. In fact, I… every single time that I have to—I want to investigate something in, like, our serverless systems at Stedi because everything that we do at work, at Stedi, since this incident serverless paradigm. So, I tend to open Cloudash, like, 95% of the time whenever I want to investigate something. And whenever I am not able to do something in Cloudash, this goes, like, straight to the top of our, you know, issue lists or backlog or whatever you want to call it. Because we want to make this product, not only awesome, you know, for customers to buy a [unintelligible 00:11:22] or whatever, but we also want to be able to use that on a daily basis.And so far, I think we've kind of succeeded. But then again, we have quite a long way to go because we have more ideas, than we have the time, definitely, so we have to kind of prioritize what exactly we're going to build. So, [unintelligible 00:11:39] integrations with alarms. So, for instance, we want to be able to see the alarms directly in the Cloudash UI. Secondly, integration with logs insights, and many other ideas. I could probably talk for hours about what we want to build.Corey: I also want to point out that this is still your side gig. You are by day a front-end engineer over at Stedi, which has a borderline disturbing number of engineers with side gigs, generally in the serverless space, doing interesting things like this. Dynobase is another example, a DynamoDB desktop client; very similar in some respects. I pay for that too. Honestly, for a company in Stedi's space, which is designed as basically a giant API for deep, large enterprise business stuff, there's an awful lot of stuff for small-scale coming out of that.Like, I wind up throwing a disturbing amount of money in the general direction of Stedi for not being their customer. But there's something about the culture that you folks have built over there that's just phenomenal.Tomasz: Yeah. For the record, you know, having a side gig is another part of interview process at Stedi. You don't have to have [laugh] a side project, but yeah, you're absolutely right, you know, the amount of kind of side projects, and you know, some of those are monetized, as you mentioned, you know, Cloudash and Dynobase and others. Some of those—because for instance, you talked to Aidan, I think a couple of weeks ago about his shenanigans, whenever you know, AWS is going to announce something he gets in and try to [unintelligible 00:13:06] this in the most amusing ways possible. Yeah, I mean, I could probably talk for ages about why Stedi is by far the best company I've ever worked at, but I'm going to say this: that this is the most talented group of people I've ever met, and myself, honestly.And, you know, the fact that I think we are the second largest, kind of, group of AWS experts outside of AWS because the density of AWS Heroes, or ex-AWS employees, or people who have been doing cloud stuff for years, is frankly, massive, I tend to learn something new about cloud every single day. And not only because of the Last Week in AWS but also from our Slack.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of “Hello, World” demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking, databases, observability, management, and security. And—let me be clear here—it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself, all while gaining the networking, load balancing, and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build. With Always Free, you can do things like run small-scale applications or do proof-of-concept testing without spending a dime. You know that I always like to put asterisks next to the word free? This is actually free, no asterisk. Start now. Visit snark.cloud/oci-free that's snark.cloud/oci-free.Corey: There's something to be said for having colleagues that you learn from. I have never enjoyed environments where I did not actively feel like the dumbest person in the room. That's why I love what I do now. I inherently am. I have to talk about so many different things, that whenever I talk to a subject matter expert, it is a certainty that they know more about the thing than I do, with the admitted and depressing exception of course of the AWS bill because it turns out the reason I had to start becoming the expert in that was because there weren't any. And here we are now.I want to talk as well about some of—your interaction outside of work with AWS. For example, you've been an Egghead instructor for a while with over 200 lessons that you published. You're an AWS Community Hero, which means you have the notable distinction of volunteering for a for-profit company—good work—no, the community is very important. It's helping each other make sense of the nonsense coming out of there. You've been involved within the ecosystem for a very long time. What is it about, I guess—the thing I'm wondering about myself sometimes—what is it about the AWS universe that drew you in, and what keeps you here?Tomasz: So, give you some context, I've started, you know, learning about the cloud and AWS back in early-2019. So, fun fact: Maciej Winnicki—again, the co-founder of Cloudash—was my manager at the time. So, we were—I mean, the company I used to work for at the time, OLX Group, we are in the middle of cloud transformation, so to speak. So, going from, you know, on-premises to AWS. And I was, you know, hired as a senior front-end engineer doing, you know, all kinds of front-end stuff, but I wanted to grow, I wanted to learn more.So, the idea was, okay, maybe you can get AWS Certified because, you know, it's one of those corporate goals that you have to have something to put that checkbox next to it. So, you know, getting certified, there you go, you have a checkbox. And off you go. So, I started, you know, diving in, and I saw this whole ocean of things that, you know, I was not entirely aware of. To be fair, at the time I knew about this S3, I knew that you can put a file in an S3 bucket and then you can access it from the internet. This is, like, the [unintelligible 00:16:02] idea of my AWS experiences.Corey: Ideally, intentionally, but one wonders sometimes.Tomasz: Yeah, exactly. That is why you always put stuff as public, right? Because you didn't have to worry about who [unintelligible 00:16:12] [laugh] public [unintelligible 00:16:15]. No, I'm kidding, of course. But still, I think what's [unintelligible 00:16:20] to AWS is what—because it is this endless ocean of things to learn and things to play with, and, you know, things to teach.I do enjoy teaching. As you said, I have quite a lot of, you know, content, videos, blog posts, conference talks, and a bunch of other stuff, and I do that for two reasons. You know, first of all, I tend to learn the best by teaching, so it helps me very much, kind of like, solidify my own knowledge. Whenever I record—like, I have two courses about CDK, you know, when I was recording those, I definitely—that kind of solidify my, you know, ideas about CDK, I get to play with all those technologies.And secondly, you know, it's helpful for others. And, you know, people have opinions about certificates, and so on and so forth, but I think that for somebody who's trying to get into either the tech industry or, you know, cloud stuff in general, being certified helps massively. And I've heard stories about people who are basically managed to double or triple their salaries by going into tech, you know, with some of those certificates. That is why I strongly believe, by the way, that those certificates should be free. Like, if you can pass the exam, you shouldn't have to worry about this $150 of the fee.Corey: I wrote a blog post a while back, “The Dumbest Dollars a Cloud Provider Can Make,” and it's charging for training and certification because if someone's going to invest that kind of time in learning your platform, you're going to try and make $150 bucks off them? Which in some cases, is going to put people off from even beginning that process. “What cloud provider I'm not going to build a project on?” Obviously, the one I know how to work with and have a familiarity with, in almost every case. And the things you learn in your spare time as an independent learner when you get a job, you tend to think about your work the same way. It matters. It's an early on-ramp that pays off down the road and the term of years.I used to be very anti-cert personally because it felt like I was jumping through hoops, and paying, in some cases, for the privilege. I had a CCNA for a while from Cisco. There were a couple of smaller companies, SaltStack, for example, that I got various certifications from at different times. And that was sort of cheating because I helped write the software, but that's neither here nor there. It's the—and I do have a standing AWS cert that I get a different one every time—mine is about to expire—because it gets me access to lounges at physical events, which is the dumbest of all reasons to get certs, but here you go. I view it as the $150 lounge pass with a really weird entrance questionnaire.But in my case it certs don't add anything to what I do. I am not the common case. I am not early in my career. Because as you progress through your career, things—there needs to be a piece of paper that says you know things, and early on degree or certifications are great at that. In the time it becomes your own list of experience on your resume or CV or LinkedIn or God knows what. Polywork if you're doing it the right way these days.And it shows a history of projects that are similar in scope and scale and impact to the kinds of problems that your prospective employer is going to have to solve themselves. Because the best answer to hear—especially in the ops world—when there's a problem is, “Oh, I've seen this before. Here's how you fix it.” As opposed to, “Well, I don't know. Let me do some research.”There's value to that. And I don't begrudge anyone getting certs… to a point. At least that's where I sit on it. At some point when you have 25 certs, it's when you actually do any work? Because it's taking the tests and learning all of these things, which in many ways does boil down to trivia, it stands in counterbalance to a lot of these things.Tomasz: Yeah. I mean, I definitely, totally agree. I remember, you know, going from zero to—maybe not Hero; I'm not talking about AWS Hero—but going from zero to be certified, there was the Solutions Architect Associate. I think it took me, like, 200 hours. I am not the, you know, the brightest, you know, the sharpest tool in the shed, so it probably took me, kind of, somewhat more.I think it's doable in, like, 100 hours, but I tend to over-prepare for stuff, so I didn't actually take the actual exam until I was able to pass the sample exams with, like, 90% pass, just to be extra sure that I'm actually going to pass it. But still, I think that, you know, at some point, you probably should focus on, you know, getting into the actual stuff because I hold two certificates, you know, one of those is going to expire, and I'm not entirely sure if I want to go through the process again. But still, if AWS were to introduce, like, a serverless specialty exam, I would be more than happy to have that. I genuinely enjoy, kind of, serverless, and you know, the fact that I would be able to solidify my knowledge, I have this kind of established path of the things that I should learn about in order to get this particular certificate, I think this could be interesting. But I am not probably going to chase all the 12 certificates.Maybe if AWS IQ was available in Poland, maybe that would change because I do know that with IQ, those certs do matter. But as of [unintelligible 00:21:26] now, I'm quite happy with my certs that I have right now.Corey: Part of the problem, too, is the more you work with these things, the harder it becomes to pass the exams, which sounds weird and counterintuitive, but let me use myself as an example. When I got the cloud practitioner cert, which I believe has lapsed since then, and I got one of the new associate-level betas—I'll keep moving up the stack until I start failing exams. But I got a question wrong on the cloud practitioner because it was, “How long does it take to restore an RDS database from a snapshot backup?” And I gave the honest answer of what I've seen rather than what it says in the book, and that honest answer can be measured in days or hours. Yeah.And no, that's not the correct answer. Yeah, but it is the real one. Similarly, a lot of the questions get around trivia, syntax of which of these is the correct argument, and which ones did we make up? It's, I can explain in some level of detail, virtually every one of AWS has 300 some-odd services to you. Ask me about any of them, I could tell you what it is, how it works, how it's supposed to work and make a dumb joke about it. Fine, whatever.You'll forgive me if I went down that path, instead of memorizing what is the actual syntax of this YAML construct inside of a CloudFormation template? Yeah, I can get the answer to that question in the real world, with about ten seconds of Googling and we move on. That's the way most of us learn. It's not cramming trivia into our heads. There's something broken about the way that we do certifications, and tech interviews in many cases as well.I look back at some of the questions I used to ask people for Linux sysadmin-style jobs, and I don't remember the answer to a lot of these things. I could definitely get back into it, but if I went through one of these interviews now, I wouldn't get the job. One would argue I shouldn't because of my personality, but that's neither here nor there.Tomasz: [laugh]. I mean, that's why you use CDK, so you'd have to remember random YAML comments. And if you [unintelligible 00:23:26] you don't have YAML anymore. [unintelligible 00:23:27].Corey: Yes, you're quite the CDK fanboy, apparently.Tomasz: I do like CDK, yes. I don't like, you know, mental overhead, I don't like context switching, and the way we kind of work at Stedi is everything is written in TypeScript. So, I am a front-end engineer, so I do stuff in the front-end line in TypeScript, all of our Lambda functions are written in TypeScript, and our [unintelligible 00:23:48] is written in TypeScript. So, I can, you know, open up my Visual Studio Code and jump between all of those files, and the language stays the same, the syntax stays the same, the tools stay the same. And I think this is one of the benefits of CDK that is kind of hard to replicate otherwise.And, you know, people have many opinions about the best to deploy infrastructure in the cloud, you know? The best infrastructure-as-code tool is the one that you use at work or in your private projects, right? Because some people enjoy ClickOps like you do; people—Corey: Oh yeah.Tomasz: Enjoy CloudFormation by hand, which I don't; people are very much into Terraform or Serverless Framework. I'm very much into CDK.Corey: Or the SAM CLI, like, three or four more, and I use—Tomasz: Oh, yeah. [unintelligible 00:24:33]—Corey: —all of these things in various ways in some of my [monstrous 00:24:35] projects to keep up on all these things. I did an exploration with the CDK. Incidentally, I think you just answered why I don't like it.Tomasz: Because?Corey: Because it is very clear that TypeScript is a first-class citizen with the CDK. My language of choice is shitty bash because, grumpy old sysadmin; it happens. And increasingly, that is switching over to terrible Python because I'm very bad at that. And the problem that I run into as I was experimenting with this is, it feels like the Python support is not fully baked, most people who are using the CDK are using a flavor of JavaScript and, let's be very clear here, the every time I have tried to explore front-end, I have come away more confused than I was when I started, part of me really thinks I should be learning some JavaScript just because of its versatility and utility to a whole bunch of different problems. But it does not work the way I think, on some level, that it should because of my own biases and experiences. So, if you're not a JavaScript person, I think that you have a much rockier road with the CDK.Tomasz: I agree. Like I said, I tend to talk about my own experiences and my kind of thoughts about stuff. I'm not going to say that, you know, this tool or that tool is the best tool ever because nothing like that exists. Apart from jQuery, which is the best thing that ever happened to the web since, you know, baked bread, honestly. But you are right about CDK, to the best of my knowledge, kind of, all the other languages that are supported by CDK are effectively transpiled down from TypeScript. So it's, like, first of all, it is written in TypeScript, and then kind of the Python, all of the other languages… kind of come second.You know, and afterwards, I tend to enjoy CDK because as I said, I use TypeScript on a daily basis. And you know, with regards to front-end, you mentioned that you are, every single time you is that you end up being more confused. It never goes away. I've been doing front-end stuff for years, and it's, you know, kind of exactly the same. Fun story, I actually joined Cloudash because, well, Maciej started working on Cloudash alone, and after quite some time, he was so frustrated with the modern front-end landscape that he asked me, “Dude, you need to help me. Like, I genuinely need some help. I am tired of React. I am tired of React hooks. This is way too complex. I want to go back to doing back-end stuff. I want to go back, you know, thinking about how we're going to integrate with all those APIs. I don't want to do UI stuff anymore.”Which was kind of like an interesting shift because I remember at the very beginning of my career, where people were talking about front-end—you know, “Front-end is not real programming. Front-end is, you know, it's easy, it's simple. I can learn CSS in an hour.” And the amount of people who say that CSS is easy, and are good at CSS is exactly zero. Literally, nobody who's actually good at CSS says that, you know, CSS, or front-end, or anything like that is easy because it's not. It's incredibly complex. It's getting probably more and more complex because the expectations of our front-end UIs [unintelligible 00:27:44].Corey: It's challenging, it is difficult, and one of the things I find most admirable about you is not even your technical achievements, it's the fact that you're teaching other people to do this. In fact, this gets to the last point I want to cover on our conversation today. When I was bouncing topic ideas off of you, one of the points you brought up that I'm like, “Oh, we're keeping that and saving that for the end,” is why—to your words—why speaking at tech events gets easier, but never easy. Let's dive into that. Tell me more about it.Tomasz: Basically, I've accidentally kickstarted my career by speaking at meetups which later turned into conferences, which later turned into me publishing courses online, which later turned into me becoming an AWS Hero, and here we are, you know, talking to each other. I do enjoy, you know, going out in public and speaking and being on stage. I think, you know, if somebody has, kind of, the heart, the ability to do that, I do strongly recommend, you know, giving it a shot, not only to give, like, an honestly life-changing experience because the first time you go in front of hundreds of people, this is definitely, you know, something that's going to shake you, while at the same time acknowledging that this is absolutely, definitely not for everyone. But if you are able to do that, I think this is definitely worth your time. But as you said—by quoting me—that it gets easier, so every single time you go on stage, talk at a meetup or at a conference or online conferences—which I'm not exactly a fan of, for the record—it's—Corey: It's too much like work, too much like meetings. There's nothing different about it.Tomasz: Yeah, exactly. Like, there's no journey. There's no adventure in online conferences. I know that, of course, you know, given all of that, you know, we had to kind of switch to online conferences for quite some time where I think we are pretending that Covid is not a thing anymore, so we, you know, we're effectively going back, but kind of the point I wanted to make is that I am a somewhat experienced public speaker—I'd like to say that because I've been doing that for years—but I've been, you know, talking to people who actually get paid to speak at the conferences, to actually kind of do that for a living, and they all say the same thing. It gets simpler, it gets easier, but it's never freaking easy, you know, to go out there, and you know, to share whatever you've learned.Corey: I'm one of those people. I am a paid public speaker fairly often, even ignoring the podcast side, and I've spoken on conference stages a couple hundred times at least. And it does get easier but never easy. That's a great way of framing it. You… I get nervous before every talk I give.There are I think two talks I've given that I did not have an adrenaline hit and nervous energy before I went onstage, and both of those were duds. Because I think that it's part of the process, at least for me. And it's like, “Oh, how do you wind up not being scared for before you go on stage?” You don't. You really don't.But if that appeals to you and you enjoy the adrenaline rush of the rest, do it. If you're one of those people who've used public speaking as, “I would prefer death over that,” people are more scared of public speaking their death, in some cases, great. There are so many ways to build audiences and to reach people that fine, if you don't like doing it on stage, don't force yourself to. I'd say try it once; see how it feels meetups are great for this.Tomasz: Yeah. Meetups are basically the best way to get started. I'm yet to meet a meetup, either, you know, offline or online, who is not looking for speakers. It's always quite the opposite, you know? I was, you know, co-organizing a meetup in my city here in Poznań, Poland, and the story always goes like this: “Okay, we have a date. We have a venue. Where are the speakers?” And then you know, the tumbleweed is going to roll across the road and, “Oh, crap, we don't have any speakers.” So, we're going to try to find some, reach out to people. “Hey, I know that you did this fantastic project at your workplace. Come to us, talk about this.” “No, I don't want to. You know, I'm not an expert. I am, you know, I have on the 50 years of experience as an engineer. This is not enough.” Like I said, I do strongly recommend it, but as you said, if you're more scared of public speaking than, like, literally dying, maybe this is not for you.Corey: Yeah. It comes down to stretching your limits, finding yourself interesting. I find that there are lots of great engineers out there. The ones that I find myself drawn to are the ones who aren't just great at building something, but at storytelling around the thing that they are built of, yes, you build something awesome, but you have to convince me to care about it. You have to show me the thing that got you excited about this.And if you can't inspire that excitement in other people, okay. Are you really excited about it? Or what is the story here? And again, it's a different skill set. It is not for everyone, but it is absolutely a significant career accelerator if it's leveraged right.Tomasz: [crosstalk 00:32:45].Corey: [crosstalk 00:32:46] on it.Tomasz: Yeah, absolutely. I think that we don't talk enough about, kind of, the overlap between engineering and marketing. In the good sense of marketing, not the shady kind of marketing. The kind of marketing that you do for yourself in order to elevate yourself, your projects, your successes to others. Because, you know, try as you might, but if you are kind of like sitting in the corner of an office, you know, just jamming on your keyboard 40 hours per week, you're not exactly likely to be promoted because nobody's going to actively reach out to you to find out about your, you know, recent successes and so on.Which at the same time, I'm not saying that you should go @channel in Slack every single time you push a commit to the main branch, but there's definitely, you know, a way of being, kind of, kind to yourself by letting others know that, “Okay, I'm here. I do exist, I have, you know, those particular skills that you may be interested about. And I'm able to tell a story which is, you know, convincing.” So it's, you know, you can tell a story on stage, but you can also tell your story to your customers by building a future that they're going to use. [unintelligible 00:33:50].Corey: I really want to thank you for taking the time to speak with me today. If people want to learn more, where's the best place to find you?Tomasz: So, the best place to find me is on Twitter. So, my Twitter handle is @tlakomy. So, it's T-L-A-K-O-M-Y. I'm assuming this is going to be in the [show notes 00:34:06] as well.Corey: Oh, it absolutely is. You beat me to it.Tomasz: [laugh]. So, you can find Cloudash at cloudash.dev. You can probably also find my email, but don't email me because I'm terrible, absolutely terrible at email, so the best way to kind of reach out to me is via my Twitter DMs. I'm slightly less bad at those.Corey: Excellent. And we will, of course, put links to that in the [show notes 00:34:29]. Thank you so much for being so generous with your time. I appreciate it.Tomasz: Thank you. Thank you for having me.Corey: Tomasz Łakomy, Head of React at Cloudash. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, and if you're on the YouTubes, smash the like and subscribe button, as the kids say. Whereas if you've hated this episode, please do the exact same thing—five-star reviews smash the buttons—but this time also leave an insulting and angry comment written in the form of a CloudWatch log entry that no one is ever able to find in the native interface.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

covid-19 god amazon head world discover simple fun hero drop cloud front id poland honestly context oracle call of duty counting iq react slack cv api cisco last week screaming python ui linux aws apis javascript devops invent googling elk css dns x ray equifax s3 cloudflare meetups lambda tomasz maciej electron rds serverless honeycomb terraform typescript pozna jquery observability eggheads blackboard sso elasticsearch visual studio code uis cdk pagerduty yaml oracle cloud dynamodb ccna corey quinn always free api gateway edb cloudfront saltstack cloudformation polywork cloudwatch nagios serverless framework duckbill group aws hero elastic cloud chief cloud economist last week in aws humblepod

Episode #130: Serverless Framework v3 with Matthieu Napoli and Mariusz Nowak

personal cloud packed aws napoli faa javascript bref matthieu php fascinated lambda nowak senior product manager baas serverless mariusz serverless framework

Play Episode Listen Later Mar 28, 2022 47:00

About Matthieu NapoliMatthieu is a software engineer passionate about helping developers to create. He's the founder of Null, and currently a Senior Product Manager for Serverless Framework at Serverless Inc. Fascinated by how serverless unlocks creativity, he works on making serverless accessible to everyone.Apart from consulting for clients, Matthieu also spends his time maintaining open-source projects. That includes Bref, a framework for creating serverless PHP applications on AWS. Alongside Bref, he sends a monthly newsletter containing serverless news relevant to PHP developers.After years of talking at conferences and training teams on serverless, Matthieu created the Serverless Visually Explained course. Packed with use cases, visual explanations, and code samples, the course focuses on being practical and accessible. Twitter: https://twitter.com/matthieunapoli LinkedIn: https://www.linkedin.com/in/matthieunapoli GitHub: https://github.com/mnapoli/ Personal website: https://mnapoli.fr/ Serverless Explained course: https://serverless-visually-explained.com/ Null: https://null.tc/ Serverless Framework: serverless.com About Mariusz NowakMariusz has been involved with full-stack development of web applications since 2004 and actively engaged in the open source community. He developed and published many JavaScript tools and modules, which play important part in implementation of modern web applications (client & server side) that he's worked with.He also implemented a light, highly configurable, in-memory database engine that allows decentralized, network independent and (while in network connection) a real-time distribution/replication of database data: https://github.com/medikoo/dbjs. Twitter: https://twitter.com/medikoo LinkedIn: https://www.linkedin.com/in/mariusznowak GitHub: https://github.com/medikoo

11. How do you move away from the management console?

AWS Bites

Play Episode Listen Later Nov 19, 2021 8:47

In this episode, Luciano and Eoin discuss the good and the bad of the AWS Management Console (a.k.a. the web console) and why you should consider migrating to Infrastructure as Code (IaC) as soon as possible, especially for your production applications. In this episode we mentioned the following resources: - Cloudformation: https://aws.amazon.com/cloudformation/ - CDK: https://aws.amazon.com/cdk/ - Serverless Framework: https://www.serverless.com/ - SAM: https://aws.amazon.com/serverless/sam - Terraform: https://www.terraform.io/ - Former2: https://former2.com/ - Import or create (e.g. if the resource already exists in production). With CDK: https://loige.co/create-resources-conditionally-with-cdk This episode is also available on YouTube: https://www.youtube.com/AWSBites You can listen to AWS Bites wherever you get your podcasts: - Apple Podcasts: https://podcasts.apple.com/us/podcast/aws-bites/id1585489017 - Spotify: https://open.spotify.com/show/3Lh7PzqBFV6yt5WsTAmO5q - Google: https://podcasts.google.com/feed/aHR0cHM6Ly9hbmNob3IuZm0vcy82YTMzMTJhMC9wb2RjYXN0L3Jzcw== - Breaker: https://www.breaker.audio/aws-bites - RSS: https://anchor.fm/s/6a3312a0/podcast/rss Do you have any AWS questions you would like us to address? Leave a comment here or connect with us on Twitter: - https://twitter.com/eoins - https://twitter.com/loige

spotify management infrastructure import aws console eoin terraform move away cdk cloudformation serverless framework aws management console

Austen Collins: Serverless Framework, Serverless Cloud, Serverless Ops Console, and the Future of Serverless

AWS FM

Play Episode Listen Later Nov 9, 2021

Austen joins Adam to discuss the evolution of Serverless, Inc, the many products they're building now including the Serverless Framework, Serverless Cloud, and Serverless Ops Console, as well as the future of serverless development and monitoring.

cloud console austen serverless serverless framework

Episode #117: Serverless Cloud with Doug Moscrop, Eslam Hefnawy, and Ben Miner

Play Episode Listen Later Nov 1, 2021 58:01

Doug Moscrop is the Lead Software Engineer at Serverless, Inc. working on Serverless Cloud. He is a pragmatic programmer with a strong engineering discipline, a thirst for knowledge and a desire for accuracy. He's the author of several Serverless Framework plugins and the creator of serverless-http.Eslam Hefnawy is a Principal Software Engineer at Serverless Inc. working on Serverless Cloud. He's been writing software since the age of 14 and is passionate about open source, dev tools, and serverless technologies. In 2015, he joined Serverless, Inc as their first hire and co-created the Serverless Framework with founder, Austen Collins. In 2018, he lead the design and development of Serverless Components, a next-generation Serverless Framework. Ben Miner is a Software Engineer at Serverless, Inc. working on Serverless Cloud and has experience all over the spectrum, including DevOps, backend, and frontend technologies. He's also a pursuer of SAAS architectures, functional programming, and great end user experiences. Twitter: Eslam: @eahefnawy Doug: @dougmoscrop Ben: @devvyben Serverless Cloud: https://serverless.com/cloud

cloud saas aws miners faa devops software engineers lambda baas serverless principal software engineer lead software engineer serverless framework ben miner eslam

Severless Hero, Got Severs in His Eyes with Ant Stanley

Play Episode Listen Later Aug 31, 2021 37:02

About AntAnt Co-founded A Cloud Guru, ServerlessConf, JeffConf, ServerlessDays and now running Senzo/Homeschool, in between other things. He needs to work on his decision making.Links: A Cloud Guru: https://acloudguru.com homeschool.dev: https://homeschool.dev aws.training: https://aws.training learn.microsoft.com: https://learn.microsoft.com Twitter: https://twitter.com/iamstan TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Thinkst. This is going to take a minute to explain, so bear with me. I linked against an early version of their tool, canarytokens.org in the very early days of my newsletter, and what it does is relatively simple and straightforward. It winds up embedding credentials, files, that sort of thing in various parts of your environment, wherever you want to; it gives you fake AWS API credentials, for example. And the only thing that these things do is alert you whenever someone attempts to use those things. It's an awesome approach. I've used something similar for years. Check them out. But wait, there's more. They also have an enterprise option that you should be very much aware of canary.tools. You can take a look at this, but what it does is it provides an enterprise approach to drive these things throughout your entire environment. You can get a physical device that hangs out on your network and impersonates whatever you want to. When it gets Nmap scanned, or someone attempts to log into it, or access files on it, you get instant alerts. It's awesome. If you don't do something like this, you're likely to find out that you've gotten breached, the hard way. Take a look at this. It's one of those few things that I look at and say, “Wow, that is an amazing idea. I love it.” That's canarytokens.org and canary.tools. The first one is free. The second one is enterprise-y. Take a look. I'm a big fan of this. More from them in the coming weeks.Corey: This episode is sponsored in part my Cribl Logstream. Cirbl Logstream is an observability pipeline that lets you collect, reduce, transform, and route machine data from anywhere, to anywhere. Simple right? As a nice bonus it not only helps you improve visibility into what the hell is going on, but also helps you save money almost by accident. Kind of like not putting a whole bunch of vowels and other letters that would be easier to spell in a company name. To learn more visit: cribl.ioCorey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Every once in a while I talk to someone about, “Oh, yeah, remember that time that you appeared on Screaming in the Cloud?” And it turns out that they didn't; it was something of a fever dream. Today is one of those guests that I'm, frankly, astonished I haven't had on before: Ant Stanley. Ant, thank you so much for indulging me and somehow forgiving me for not having you on previously.Ant: Hey, Corey, thanks for that. Yeah, I'm not too sure why I haven't been on previously. You can explain that to me over a beer one day.Corey: Absolutely, and I'm sure I'll be the one that buys it because that is just inexcusable. So, who are you? What do you do? I know that you're a Serverless Hero at AWS, which is probably the most self-aggrandizing thing you can call someone because who in the world in their right mind is going to introduce themselves that way? That's what you have me for. I'll introduce you that way. So, you're an AWS Serverless Hero. What does that mean?Ant: So, the Serverless Hero, effectively I've been recognized for my contribution to the serverless community, what that contribution is potentially dubious. But yeah, I was one of the original co-founders of A Cloud Guru. We were a serverless-first company, way back when. So, from 2015 to 2016, I was with A Cloud Guru with Ryan and Sam, the two other co-founders.I left in 2016 after we'd run ServerlessConf. So, I led and ran the first ServerlessConf. And then for various reasons, I decided, hey, the pressure was too much; I needed a break, and a few other reasons I decided to leave A Cloud Guru. A very amicable split with my former co-founders. And then yeah, I kind of took a break, took some time off, de-stressed, got the serverless user group in London up and running; ran a small conference in London called JeffConf, which was a take on a blog that Paul Johnson, who was one of the folks who ran JeffConf with me, wrote a while ago saying we could have called it serverless—and we might as well have called it Jeff. Could have called it anything; might as well have called it Jeff. So, we had this joke about JeffConf. Not a reference to Mr. Bazos.Corey: No, no. Though they do have an awful lot of Jeffs working over there. But that's neither here nor there. ‘The Land of the Infinite Jeffs' as it were.Ant: Yeah, exactly. There are more Jeffs than women in the exec team if I remember correctly.Corey: I think it's now it's a Dave problem instead.Ant: Yeah, it's a Dave problem. Yeah. [laugh]. It's not a problem either way. Yeah. So, JeffConf morphed into SeverlessDays, which is a group of community events around the world. So, I think AWS said, “Hey, this guy likes running serverless events for some silly reason. Let's make him a Serverless Hero.”Corey: And here we are. Which is interesting because a few directions you can take this in. One of them, most recently, we were having a conversation, and you were opining on your thoughts of the current state of serverless, which can succinctly be distilled down to ‘serverless sucks,' which is not something you'd expect to hear from a Serverless Hero—and I hope you can hear the initial caps when I say ‘Serverless Hero'—or the founder of a serverless conference. So, what's the deal with that? Why does it suck?Ant: So, whole serverless movement started to gather momentum in 2015. The early adopters were all extremely experienced technologists, folks like Ben Kehoe, the chief robotics scientist at iRobot—he's incredibly smart—and folks of that caliber. And those were the kinds of people who spoke at the first serverless conference, spoke at all the first serverless events. And, you know, you'd kind of expect that with a new technology where there's not a lot of body of knowledge, you'd expect these high-level, really advanced folks being the ones putting themselves out there, being the early adopters. The problem is we're in 2021 and that's still the profile of the people who are adopting serverless, you know? It's still not this mass adoption.And part of the reason for me is because of the complexity around it. The user experience for most serverless tools is not great. It's not easy to adopt. The patterns aren't standardized and well known—even though there are a million websites out there saying that there are serverless patterns—and the concepts aren't well explained. I think there's still a fair amount of education that needs to happen.I think folks have focused far too much on the technical aspects of serverless, and what is serverless and not serverless, or how you deploy something, or how you monitor something, observability, instead of going back to basics and first principles of what is this thing? Why should you do it? How do you do it? And how do we make that easy? There's no real focus on user experience and adoption for inexperienced folks.The adoption curve, the learning curve for serverless, no matter what platform you do, if you want to do anything that's beyond a side project it's really difficult because there's no easy path. And I know there's going to be folks that are going to complain about it, but the Serverless Stack just got a million dollars to solve this problem.Corey: I love the Serverless Stack. They had a great way of building things out.Ant: Yeah.Corey: I cribbed a fair bit from what they built when I was building out my own serverless project of the newsletter production pipeline system. And that's awesome. And I built that, and I run it mostly as a technology testbed. But my website, lastweekinaws.com?I pay WP Engine to host it on WordPress and the reason behind that is not that I can't figure out the serverless pieces of it, it's because when I want to hire someone to do something that's a bit off the beaten path on WordPress, I don't have to spend $400 an hour for a consultant to do it because there's more than 20 people in the world who understand how all this stuff fits together and integrates well. There's something to be said for going in the direction the rest of the market is when there's not a lot of reason to differentiate yourselves. Yeah, could I save thousands of dollars a year in infrastructure costs if I'd gone with serverless? Of course, but people's time is worth more than that. It's expensive to have people work on these things.And even on the serverless stuff that I've built, if it's been more than six months since I've touched a component, someone else may have written it; I have to rediscover what the hell I was thinking and what the constraints are, what the constraints I thought existed there in the platform. And every time I deal with Lambda or API Gateway, I come away with a spiraling sense of complexity tied to all of it. And the vision of serverless I believe in, truly, but the execution has lagged from all providers.Ant: Yeah. I agree with that completely. The execution is just not there. I look at the situation—so Datadog had their report, “The State of Serverless Report” that came out about a month or two ago; I think it's the second year they've done it, now, might be the third. And in the report, one of the sections, they talked about tooling.And they said, “What's the most adopted tools?” And they had the Serverless Framework in there, they had SAM in there, they had CloudFormation, I think they had Terraform in there. But basically, Serverless Framework had 70% of the respondents. 70% of folks using Datadog and using serverless tools were using Serverless Framework. But SAM, AWS's preferred solution, was like 12%.It was really tiny and this is the thing that every single AWS demo example uses, that the serverless developer advocates push heavily. And it's the official solution, but the Serverless Application Model is just not being adopted and there are reasons for that, and it's because it's the way they approach the market because it's highly opinionated, and they don't really listen to end-users that much. And their CDK out there. So, that's the other AWS organizational complexity as well, you've got another team within AWS, another product team who've developed this different way—CDK—doing things.Corey: This is all AWS's fault, by the way. For the longest time, I've been complaining about Lambda edge functions because they are not at all transparent; you have to wait for a CloudFront deployment for it to update every time, only to figure out that in my case, I forgot a comma because I've never heard of a linter. And it becomes this awful thing. Only recently did I find out they only run at regional edge caches, not just in all of the CloudFront pop, so I said, “The hell with it,” ripped it out of everything I was using it with, and wound up implementing it in bog-standard Lambda because it was easier. But then rather than fixing that, they've created their—what was it—their CloudFront Workers. Or is it—is it CloudFront Workers, or is it CloudFront Functions?Ant: No, CloudFront Functions.Corey: I don't even remember it because rather than fixing the thing, you just released a different thing that addresses these problems in very different ways that aren't directly compatible. And it's oh, great, awesome. Terrific. As a customer, I want absolutely not this. It's one of these where, honestly, I've left in many cases with the resigned position of, if you're not going to take this seriously, why am I?Ant: Yeah, exactly. And it's bizarre. So, the CloudFront Functions thing, it's based on Nginx's [little 00:08:39] JavaScript engine. So, it's the Nginx team supporting it—the engine—which is really small number of users; it's tiny, there's no foundation behind it. So, you've got these massive companies reliant on some tiny organization to support the runtime of one of their businesses, one of their services.And they expect people to adopt it. And on top of that, that engine supports primary language is JavaScript's ES5 or ES2015, which is the 2015 edition of JavaScript, so it's a six-year-old version of JavaScript. You cannot use one JavaScript with it which also means you can't use any other tools in the JavaScript ecosystem for it. So basically, anything you write for that is going to be vanilla, you're going to write yourself, there's no tooling, no community to really leverage to use that thing. Again, like, why have you even done that? Why if you now gone off and taken an engine no one uses—they will say someone uses it, but basically no one uses—Corey: No one willingly uses or knowingly uses it.Ant: Yeah. No one really uses. And then decided to run that. Why not look at WebAssembly—it's crazy—which has a foundation behind it and they're doing great things, and other providers are using WebAssembly on the edge. I just don't understand the thought process—well, I say I don't understand, but I do understand the thought processes behind Amazon. Every single GM in Amazon is effectively incentivized to release stuff, and build stuff, and to get stuff out the door. That's how they make money. You hear the stories—Corey: Oh, it's been clear for years. They only recently stopped—in their keynotes every year—talking about the number of feature releases that they've had over the past 12 months. And I think they finally had it clued into them by someone snarky on Twitter—ahem—that the only people that feel good about that are people internal to AWS because customers see that and get horrified of, “I haven't kept up with most of those things. How many of those are important? How many of them are nonsense?”And I'm sure somewhere you have released a serverless that will solve my business problem perfectly so I don't have to build it together myself out of Lambda functions, and string, and popsicle sticks, but I'll never hear about it because you're too busy talking about nonsense. And that problem still exists and it's writ large. There's a philosophy around not breaking existing workloads—which I get; that's a hard problem to solve for—but their solution is, rather than fixing existing services will launch a new one that doesn't have those constraints and takes a different approach to it. And it's horrible.Ant: Yeah, exactly. If you compare Amazon to Apple, Apple releases a net-new product once a year, once every two years.Corey: You're talking about new generations of products, that comes out on an annualized basis, but when you're talking about actual new product, not that frequently. The last one—Ant: Yeah.Corey: —I can really think of is probably going to be AirPods, at least of any significance.Ant: AirTags is the new one.Corey: Oh, AirTags. AirTags is recent, which is a neat—but it's an accessory to the rest of those things. It is—Ant: And then there's AirPods. But yeah, it's once—because they—everything works. If you're in that Apple ecosystem, everything works. And everything's back-ported and supported. My four-year-old phone still works and had a five-year-old MacBook before this current one, still worked, you know, not a problem.And those two philosophies—and the Amazon folk are heavily incentivized to release products and to grow the usage of those products. And they're all incentivized within their bubbles. So, that's why you get competing products. That's why Proton exists when CodeBuild and CodePipeline, and all of those things exist, and you have all these competing products. I'm waiting for the container team to fully recreate AWS on top of containers. They're not far away.Corey: They're already in the process of recreating AWS on top of Lightsail. It's more or less the, “Oh, we're making this the simpler version.” Which is great. You know who likes simplicity? Freaking everyone.So, it's the vision of a cloud, we could have had but didn't. “Oh, you want a virtual machine. Spin up a Lightsail instance; you're going to get a fixed amount of compute, disk, RAM, and CPU that you can adjust, and it's going to cost you a flat fee per month until you exceed some fairly high limits.” Why can't everything be like that, on some level? Because in many cases, I don't care about wanting to know exactly to the penny shave things off.I want to spin up a fleet of 20 virtual machines, and if they cost me 20 bucks a pop each a month, I can forecast that, I can budget for that, I can do a lot and I don't actually care in any business context about the money there, but dialing it in and having the variable charges and the rest, and, “Oh, you went through a managed NAT gateway. That's going to double your bandwidth price and it's going to be expensive. Surprise, you should have looked more closely at it,” is sort of the lesson of the original AWS services. At some level, they've deviated away from anything resembling simplicity and increasingly we're seeing a world where in order to do something effectively with cloud, you have to spend 12 weeks going to cloud school first.Ant: Oh, yeah. Completely. See, that's one of the major barriers with serverless. You can't use serverless for any of the major cloud providers until you understand that cloud provider. So yeah, do your 12 weeks of cloud school. And there's more than enough providers.Corey: Whoa, whoa, whoa. Before you spin up a function that runs code, you have to understand the identity and security model, and how the network works, and a bunch of other ancillary nonsense that isn't directly tied to business value.Ant: And all these fun things. How are you're going to test this, and how are you're going to do all that?Corey: How do you write the entry point? Where is it going to enter? What is it expecting? What objects are getting passed in, if any? What format is it going to take?I've spent days, previously, trying to figure out the exact invocation for working with a JSON object in Python, what that's going to show up as, and how specifically to refer to it. And once you've done that a couple of times, great, fine, it's easy. Copy and paste it from the last time you did it. But figuring it out from first principles, particularly in a time when there isn't a lot of good public demonstrations of this—especially early days—it's hard to do.Ant: Yeah. And they just love complexity. Have you looked at the second edition—so the third version of the AWS SDK for JavaScript?Corey: I don't touch JavaScript with my hands most days, just because I'm bad at it and I don't understand the asynchronous model and computers are really not my thing most.Ant: So, unfortunately for my sins, I do use JavaScript a lot. So, version two of the SDK is effectively the single most popular Cloud SDK of any language, anything out there; 20 million downloads a week. It's crazy. It's huge—version two. And JavaScript's a very fast-evolving language, though.Basically, it's a bit like the English language in that it adopts things from other languages through osmosis, and co-opts various other features of other languages. So, JavaScript has—if there's a feature you love in your language, it's going to end up in JavaScript at some point. So, it becomes a very broad Swiss Army knife that can do almost anything. And there's always better ways to do things. So, the problem is, the version two was written in old JavaScript from years twenty fifteen years five years six kind of level.So, from 2015, 2016, I—you know, 2020, 2021, JavaScript has changed. So, they said, “Oh, we're going to rewrite this.” Which good; you should do. But they absolutely broke all compatibility with version two. So, there is no path from version two to version three without rewriting what you've got.So, if you want to take anything you've written—not even serverless—anything in JavaScript you've written and you want to upgrade it to get some of the new features of JavaScript in the SDK, you have to rewrite your code to do that. And some instances, if you're using hexagonal architecture and you're doing all the right things, that's a really small thing to do. But most people aren't doing that.Corey: But let's face it, a lot of things grow organically.Ant: Yeah.Corey: And again, I can sit here and tell you how to build things appropriately and then I look at my own environment and… yeah, pay no attention to that burning dumpster fire behind the camera. And it's awful. You want to make sure that you're doing things the right way but it's hard to do and taking on additional toil because the provider decides the time to focus on this is a problem.Ant: But it's completely not a user-centric way of thinking. You know, they've got all their 14—is it 16 principles now? Did they add two principles, didn't they?Corey: They added two to get up to 16; one less than the numbers of ways to run containers in AWS.Ant: Yeah. They could barely contain themselves. [laugh]. It's just not customer-centric. They've moved themselves away from that customer-centric view of the world because the reality is, they are centered on the goals of the team, the goals of the GM, and the goals of that particular product.That famous drawing of all the different organizational charts, they got the Facebook chart, and the Google Chart, and the Amazon chart has all these little circles, everyone pointing guns at each other. And the more Amazon grows, the more you feel like that's reality. And it's hurting users, it's massively hurting users. And we feel the pain every day, absolutely every day, which is not great. And it's going to hurt Amazon in the long run, but short-term, they're not going to see that pain quarterly, they're not going to see that pain, probably within 12 months.But they will see the pain long run. And if they want to fix it, they probably should have started fixing it two years ago. But it's going to take years to fix because that's a massive cultural shift to say, “Okay, how do we get back to being more customer-focused? How do we stop that organizational targets and goals from getting in the way of delivering value to the customer?”Corey: It's a good question. The hard part is getting customers to understand enough of what you put out there to be able to disambiguate what you've built, and what parts to trust, what parts not the trust, what parts are going to be hard, et cetera, et cetera, et cetera, et cetera. The concern that I've got across the board here is, how do you learn? How do you get started with this? And the way that I came into this was I started off, in the early days of AWS, there were a dozen services, and okay, I could sort of stumble my way through it.And the UI was rough, but it got better with time. So, the answer for a lot of folks these days is training, which makes sense. In the beginning, we learned through things like podcasts. Like there was a company called Jupiter Broadcasting which did a bunch of Linux-oriented podcasts and learned how this stuff works. And then they were acquired by Linux Academy which really focused on training.And then A Cloud Guru acquired Linux Academy. And then Pluralsight acquired A Cloud Guru and is now in the process of itself being acquired by Vista Equity Partners. There's always a bigger fish eating something somewhere. It feels like a tremendous, tremendous consolidation in the training market. Given that you were one of the founders of A Cloud Guru, where do you stand on that?Ant: So, in terms of that actual transaction, I don't know the details because I'm a long time out of A Cloud Guru, but I've stayed within the whole training sphere, and so effectively, the bigger fish scenario, it's making the market smaller in terms of providers are there. You really don't have many providers doing cloud-specific training anymore. On one level you don't, but then another level, you've got lots of independent folks doing tons of stuff. So, you've got this explosion at the bottom end. If you go to Udemy—which is where A Cloud Guru started, on Udemy—you will see tons of folks offering courses at ten bucks a pop.And then there's what I'm doing now on homeschool.dev; there's serverless-focused training on there. But that's really focused on a really small niche. So, there's this explosion at the bottom end of lots of small people doing lots of things, and then you've got this consolidation at the top end, all the big providers buying each other, which leaves a massive gap in the middle.And on top of that, you've got AWS themselves, and all the other cloud providers, offering a lot of their own free training, whether it's on their own platforms—there's aws.training now, and Microsoft have similar as well—I think it's learn.microsoft.com is theirs. And you've got all these different providers doing their own training, so there's lots out there.There's actually probably more training for lower costs than ever before. The problem is, it's like the complexity of too many services, it's the 17 container problem. Which training do you use because the actual cost of the training is your time? It's not the cost of the course. Your time is always going to be more expensive.Corey: Yeah, the course is never going to be anywhere comparable to the time you spend on it. And I've never understood, frankly, why these large companies charge money for training on their own platform and also charge money for certifications because I don't care what you're going to pay for those things, once you know a platform well enough to hit a certification, you're going to use the thing you know, in most cases; it's a great bottom-up adoption story.Ant: Yeah, completely. That was actually one of Amazon's first early problems with their trainings, why A Cloud Guru even exists, and Linux Academy, and Cloud Academy all actually came into being is because Amazon hired a bunch of folks from VMware to set up their training program. And VMware's training, back in the day, was a profit center. So, you'd have a one-and-a-half thousand, two thousand dollar training course you'd go on for three to five days, and then you'd have a couple hundred dollars to do the certification. It was a profit center because VMware didn't really have that much competition. Zen and Microsoft's Hyper V were so late to the market, they basically own the market at the time. So—Corey: Oh, yeah. They still do in some corners.Ant: Yeah. They're still massively doing in this place as they still exist. And so they Amazon hired a bunch of ex-VMware folk, and they said, “We're just going to do what we did at VMware and do it at Amazon,” not realizing Amazon didn't own the market at the time, was still growing, and they tried to make it a profit center, which basically left a huge gap for folks who just did something at a reasonable price, which was basically everyone else. [laugh].This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of "Hello, World" demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking databases, observability, management, and security.And - let me be clear here - it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself all while gaining the networking load, balancing and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build.With Always Free you can do things like run small scale applications, or do proof of concept testing without spending a dime. You know that I always like to put asterisks next to the word free. This is actually free. No asterisk. Start now. Visit https://snark.cloud/oci-free that's https://snark.cloud/oci-free.Corey: The challenge I found with a few of these courses as well, is that they teach you the certification, and the certifications are, in some ways, crap when it comes to things you actually need to know to intelligently use a platform. So, many of them distill down not to the things you need to know, but to the things that are easy to test in a multiple-choice format. So, it devolves inherently into trivia such as, “Which is the right syntax for this thing?” Or, “Which one of these CloudFormations stanzas or functions isn't real?” Things like that where it's, no one in the real world needs to know any of those things.I don't know anyone these days—sensible—who can write CloudFormation from scratch without pulling up some reference somewhere because most people don't have that stuff in their head. And if you do, I'd suggest forgetting it so you can use that space to remember something that's more valuable. It doesn't make sense for how people interact with these things. But I do see the value as well in large companies trying to upskill thousands and thousands of people. You have 5000 people that are trying to come up to speed because you're migrating into cloud. How do you judge people's progress? Well, certifications are an easy answer.Ant: Yeah, massively. Probably the most successful blog post ever written—I don't think it's up anymore, but it was when I was at A Cloud Gurus—like, what's the value of a certification? And ultimately, it came down to, it's a way for companies that are hiring to filter people easily. That's it. That's really it. It's if you've got to hire ten people and you get 1000 CVs or resumes for those ten roles, first thing you do is you filter by who's certified for that role. And then you go through anything else. Does the certification mean you can actually do the job? Not really. There are hundreds of people who are not cer—thousands, millions of people who are not certified to do jobs that they do. But when you're getting hired and there's lots of people applying for the same role, it's literally the first thing they will filter on. And it's—so you want to get certified, it's hard to get through that filter. That's what the certification does, it's how you get through that first filter of whatever the talent tracking system they're using is. That's it. And how to get into the dev lounge at re:Invent.Corey: Oh yeah, that's my reason for getting a certification, originally. And again, for folks who learn effectively that way, I have no problem with people getting certifications. If you're trying to advance in your career, especially early stage, and you need a piece of paper that says you know what you're talking about, a certification is a decent approach. In time, with seniority, that gets replaced by a piece of paper, it's called your resume or your CV, but that is a longer-term more senior-focused approach. I don't begrudge people getting certifications and I don't think that they're foolish for doing it.But in time, it feels like the market for training is simultaneously contracting into only a few players left, and also, I'm curious as to whether or not the large companies out there are increasing their spend with the training providers or not. On the community side, the direct-to-consumer approach, that is exploding, but at the same time, you're then also dealing—forgive me, listeners—with the general public and there is nothing worse than a customer, from a customer service perspective, who was only paying a little money to you. I used to work in a web hosting company that $3,000 a month customers were great to work with. The $2999 a month customers were hell on earth who expected that they were entitled to 80 hours a month of systems engineering time. And you see something similar in the training space. It's always the small individual customers who are spending personal money instead of corporate money that are more difficult to serve. You've been in the space for a while. What do you see around that?Ant: Yeah, I definitely see that. So, the smaller customers, there's a correlation between the amount of money you spend and the amount of hand-holding that someone needs. The more money someone spends, the less hand-holding they need, generally. But the other side of it, what training businesses—particularly for subscription-based business—it's the same model as most gyms. You pay for it and you never use it.And it's not just subscription; like, Udemy is a perfect example of that, you know, people who have hundreds of Udemy courses they've never done, but they spend ten bucks on each. So, there's a lot of that at the lower end, which is why people offer courses at that level. So, there's people who actually do the course who are going to give you a lot of a headache, but then you're going to have a bunch of folk who never do the course and you're just taking their money. Which is also not great, either, but those folks don't feel bad because I only spent 10, 20 bucks on it. It's like, oh, it's their fault for not doing it, and you've made the money.So, that's kind of how a lot of the training works. So, the other problem with training as well is you get the quality is so variable at the bottom end. It's so, so variable. You really struggle to find—there's a lot of people just copying, like, you see instances where folks upload videos to Udemy that are literally they've downloaded someone's, video resized it, cut out a logo or something like that, and re-uploaded it and it's taken a few weeks for them to get caught. But they made money in the meantime.That's how blatant it does get to some level, but there are levels where people will copy someone else's content and just basically make it their own slides, own words, that kind of thing; that happens a lot. At the low end, it's a bit all over the place, but you still have quality, as well, at the low end, where you have these cheapest smaller courses. And how do you find that quality, as well? That's the other side of it. And also people will just trade in their name.That's the other problem you see. Someone has a name for doing X whatever, and they'll go out and bring a course on whatever that is. Doesn't mean they're a good teacher; it means they're good at building a brand.Corey: Oh, teaching is very much its own skill set.Ant: Oh, yeah.Corey: I learned to speak publicly by being a corporate trainer for Puppet and it teaches you an awful lot. But I had the benefit, in that case, of a team of people who spent their entire careers building curricula, so it wasn't just me throwing together some slides; I would teach a well-structured curriculum that was built by someone who knew exactly what they're doing. And yeah, I needed to understand failure modes, and how to get things to work when they weren't working properly, and how to explain it in different ways for folks who learn in different ways—and that is the skill of teaching right there—but curriculum development is usually not the same thing. And when you're bootstrapping, learning—I'm going to build my own training course, you have to do all of those things, and more. And it lends itself to, in many cases, what can come across as relatively low-quality offerings.Ant: Yeah, completely. And it's hard. But one thing you will often see is sometimes you'll see a course that's really high production quality, but actually, the content isn't great because folks have focused on making it look good. That's another common, common problem I see. If you're going to do training out there, just get referrals, get references, find people who've done it.Don't believe the references you see on a website; there's a good chance they might be fake or exaggerated. Put something out on Twitter, put out something on Reddit, whatever communities—and Slack or Discord, whatever groups you're in, ask questions. And folks will recommend. In the world of Google where you could search for anything, [laugh], the only way to really find out if something is any good is to find out if someone else has done it first and get their opinion on it.Corey: That's really the right answer. And frankly, I think that is sort of the network effect that makes a lot of software work for folks. Because you don't want to wind up being the first person on your provider trying to do a certain thing. The right answer is making sure that you are basically 8,000th person to try and do this thing so you can just Google it and there's a bunch of results and you can borrow code on GitHub—which is how we call ‘thought leadership' because plagiarism just doesn't work the same way—and effectively realizing this has been solved before. If you find a brand new cloud that has no customers, you are trailblazing every time you do anything with the platform. And that's personally never where I wanted to spend my innovation points.Ant: We did that at Cloud Guru. I think when we were—in 2015 and we had problems with Lambda and you go to Stack Overflow, and there was no Lambda tag on Stack Overflow, no serverless tag on Stack Overflow, but you asked a question and Tim Wagner would probably be the one answering. And he was the former head of product on Lambda. But it was painful, and in general you don't want to do it. Like [sigh] whenever AWS comes out with a new product, I've done it a few times, I'll go, “I think I might want to use this thing.”AWS Proton is a really good example. It's like, “Hey, this looks awesome. It looks better than CodeBuild and CodePipeline,” the headlines or what I thought it would be. I basically went while the keynote was on, I logged in to our console, had a look at it, and realized it was awful. And then I started tweeting about it as well and then got a lot of feedback [laugh] on my tweets on that.And in general, my attitude from whatever the new shiny thing is if I'm going to try it, it needs to work perfectly and it needs to live up to its billing on day one. Otherwise, I'm not going to touch it. And in general with AWS products now, you announce something, I'm not going to look at it for a year.Corey: And it's to their benefit that you don't look at it for a year because the answer is going to be, ah, if you're going to see that it's terrible, that's going to form your opinion and you won't go back later when it's actually decent and reevaluate your opinion because no one ever does. We're all busy.Ant: Yeah, exactly.Corey: And there's nothing wrong with doing that, but it is obnoxious they're not doing themselves favors here.Ant: Yeah, completely. And I think that's actually a failure of marketing and communication more than anything else. I don't blame the product teams too much there. Don't bill something as a finished glossy product when it's not. Pitch it at where it is.Say, “Hey, we are building”—like, I don't think at the re:Invent stage they should announce anything that's not GA and anything that it does not live up to the billing, the hype they're going to give it to. And they're getting more and more guilty of that the last few re:Invents, of announcing products that do not live up to the hype that they promote it at and that are not GA. Literally, they should just have a straight-up rule, they can announce products, but don't put it on the keynote stage if it's not GA. That's it.Corey: The whole re:Invent release is a whole separate series of arguments.Ant: [laugh]. Yeah, yeah.Corey: There are very few substantial releases throughout the year and then they drop a whole bunch of them at re:Invent, and it doesn't matter what you're talking about, whose problem it solves, how great it is, it gets drowned out in the flood. The only thing more foolish that I see than that is companies that are not AWS releasing things during re:Invent that are not on the re:Invent keynote stage, which in turn means that no one pays attention. The only thing you should be releasing is news about your data breach.Ant: [laugh]. Yeah. That's exactly it.Corey: What do I want to bury? Whenever Adam Selipsky gets on stage and starts talking, great, then it's time to push the button on the, “We regret to inform you,k” dance.Ant: Yeah, exactly. Microsoft will announce yet another print spooler bug malware.Corey: Ugh, don't get me started on that. Thank you so much for taking the time to speak with me today. If people want to hear more about your thoughts and how you view these nonsenses, and of course to send angry emails because they are serverless fans, where can they find you?Ant: Twitter is probably the easiest place to find me, @iamstan—Corey: It is a place for outrage. Yes. Your Twitter user account is?Ant: [laugh], my Twitter user account's all over the place. It's probably about 20% serverless. So, yeah @iamstan. Tweet me; I will probably respond to you… unless you're rude, then I probably won't. If you're rude about something else, I probably will. But if you're rude about me, I won't.And I expect a few DMs from Amazon after this. I'm waiting for you, [unintelligible 00:32:02], as I always do. So yeah, that's probably the easiest place to get hold of me. I check my email once a month. And I'm actually not joking about that; I really do check my email once a month.Corey: Yeah, people really need me then they'll find me. Thank you so much for taking the time to speak with me. I appreciate it.Ant: Yes, Corey. Thank you.Corey: Ant Stanley, AWS Serverless Hero, and oh so much more. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment defending serverless's good name just as soon as you string together the 85 components necessary to submit that comment.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

amazon world english google apple state land simple microsoft hero surprise ga discord reddit cloud gm pitch oracle dms ram ant counting zen slack copy cv wordpress puppets screaming python ui airpods linux aws github cvs macbook javascript devops freaking invent udemy cpu vmware terrific sdks lambda irobot airtags proton stack overflow paul johnson json swiss army terraform jeffs datadog pluralsight webassembly invents wp engine cdk nginx oracle cloud vista equity partners hyper v corey quinn always free api gateway lightsail cloudfront nmap cloud guru cloudformation jupiter broadcasting serverless framework duckbill group linux academy thinkst es5 aws sdk aws serverless hero es2015 ben kehoe chief cloud economist last week in aws humblepod

Keep on Rockin' in the Server-Free World

Play Episode Listen Later Jul 15, 2021 36:01

About MichaelMichael Garski is the Director of Platform Engineering at Fender Musical Instruments, where he leads the teams responsible for service development & testing, devops, and data. He's been with Fender for over 5 years and prior to that worked as a software engineer & architect on back-end systems at Viant, MySpace, Countrywide Home Loans & Fandango. He is passionate about application reliability and observability and their impact on customer satisfaction.Links:LinkedIn: https://www.linkedin.com/in/mgarski/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Your company might be stuck in the middle of a DevOps revolution without even realizing it. Lucky you! Does your company culture discourage risk? Are you willing to admit it? Does your team have clear responsibilities? Depends on who you ask. Are you struggling to get buy in on DevOps practices? Well, download the 2021 State of DevOps report brought to you annually by Puppet since 2011 to explore the trends and blockers keeping evolution firms stuck in the middle of their DevOps evolution. Because they fail to evolve or die like dinosaurs. The significance of organizational buy in, and oh it is significant indeed, and why team identities and interaction models matter. Not to mention weither the use of automation and the cloud translate to DevOps success. All that and more awaits you. Visit: www.puppet.com to download your copy of the report now!Corey: If your familiar with Cloud Custodian, you'll love Stacklet. Which is made by the same people who made Cloud Custodian, but put something useful on top of it so you don't have to be a need to be a YAML expert to work with it. They're hosting a webinar called “Governance as Code: The Guardrails for Cloud at Scale” because its a new paradigm that enables organizations to use code to manage and automate various aspects of governance. If you're interested in exploring this you should absolutely make it a point to sign up, because they're going to have people who know what they're talking about—just kidding they're going to have me talking about this. Its doing to be on Thursday, July 22nd at 1pm Eastern. To sign up visit snark.cloud/stackletwebinar and I'll talk to you on Thursday, July 22nd.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. We talk to a lot of people here on this show who are deep in the weeds of SaaS companies, or cloud vendors, or cloud vendors cosplaying as SaaS companies. Today, we're taking a bit of a different direction. My guest is Michael Garski, Director of Platform Engineering at Fender Musical Instruments. They make guitars among many other things. Michael, thank you for joining me.Michael: Oh, thanks for having me on, Corey.Corey: So, one of the things that I really appreciate about what you do as a company is I can, at least presumably, explain it to someone who is not super deep in technical weeds without 45 minutes of explainer first. The easy answer is, “Oh, Fender. You folks make guitars.” These days, no one just does one thing, I have to imagine. How do you describe what the company does?Michael: Oh, well, to quote Leo Fender, his view was that artists are angels and it's our job to give them wings. So, in addition to actually making and developing guitars and amplifiers, we've branched off into consumer-facing products to actually teach people how to play those instruments.Corey: You folks have been relatively outspoken about the various things you're doing at different AWS events. I mean, my approach to that tends to be that if AWS is great at making bricks that you can use to build amazing things with, “Well, great, can you draw a picture of the house that you can build with this?” “No, we're going to have a customer come out and talk about that stuff instead.” You folks have been focusing on a lot of serverless work, and you've been very public about the fact that you are almost entirely serverless-driven in terms of architecture if I'm not mistaken.Michael: That is true.Corey: Tell me about that. How did you get there and what brought it about?Michael: So, I work in the digital division in Fender. We started, let's see, we're coming up on five years I've been there. So, what we did was, initially, we started building services that could run within a container, or on an EC2 instance, but we started looking at Lambda functions. We had need to ingest a product catalog, so the IT team was able to drop us off a product catalog into an S3 bucket, and the easiest thing to do then was just trigger a Lambda function to then process that file. And it just kind of snowballed in from there.Corey: I think the common problem when people hear ‘serverless' is they think, “Oh, great. More discussions about Lambda functions.” And Lambda is almost getting something of a tarred reputation in some circles because when we can build amazing things with it ourselves, we love it, but when we ask AWS how to wind up integrating two services, or about a feature gap, their response is, “Oh, use a Lambda function for it,” It starts to feel like they're using it as spackle and the spackle has become load-bearing. Do you view serverless as being purely function-driven or is it broader than that?Michael: It's much broader than that. Serverless is a mindset where you're looking beyond just Lambda functions to using a lot of third-party services so that you can actually focus on your core business. Like, we use Zuora as a subscription provider for web-based subscriptions; we use Algolia for full-text search; we use a variety of other services so that we can just focus on the core business.Corey: One thing that's been on everyone's mind, somewhat recently, has been the idea of dramatic changes as far as user behavior goes. And in the more traditional environments where you see things like EC2 instances or on-premises data centers, back when the pandemic first hit and companies that were very focused on a model of business that aligned directly with people behaving in certain ways that they suddenly didn't, would the 80% drop-offs or more in their user traffic, but their infrastructure spend just kept hanging out exactly where it was, in a straight line. So, at some level, it feels like yes, the whole point of cloud is that it can be elastic, except no one builds it that way for a variety of reasons. When COVID hit, what changed for your business?Michael: Change for our business is we launched a program called Playthrough, okay we did this about a year ago; we started it, we gave away three months of Fender Play for free. It was a single-use code that a user would redeem and no credit card required, and over a period of five days, we saw our traffic increase by more than ten times. And we had very little changes we needed to make. Everything scaled up, we had no issue with—we used a lot of Lambda functions, DynamoDB, everything just scaled up fine. The only point that became a bottleneck was our Elasticsearch cluster. However, beefing up the nodes and adding a few more nodes that resolved that issue immediately.Corey: So, I'm going to go out on a limb and postulate that you folks increased pickup when the lockdowns hit, if for no other reason then, “Well, I'm trapped at home and I'm tired of staring at the guitar on the wall. I may as well learn to play it.” I would guess. I could be way off base on that.Michael: No, no, that's very true. Even since then, even after that program has expired—of course, not everyone then converts and sticks around—but many, many did, many more than we thought would did stick around, and our usage and our goals were exceeded for this last year, and we're in a healthy place, and looking at continuing to grow and expand in the future.Corey: So, one of the applications that I think gets a fair bit of attention—rightfully so—lately, is something called Fender Play, and as best I can tell, that is a app that works in web, it works on mobile, and it's a video-based instruction tool for guitar at least, but some other instruments as well. How did that come to be? Did that exist before COVID hit? Has that been something that's been in the works for a while? Or was it, “Well, we're going to do a two-week sprint and build this thing from scratch?”Michael: No, we launched that—this June we're coming up on the fourth anniversary since it's been launched, so we launched this in summer of 2017.Corey: One of the problems I've always found is that it's challenging to learn to do something that is as, I guess, physical and intricate, et cetera, as playing an instrument without having someone in the room looking at you and smacking you with a stick whenever you do things that are wrong. “Nope, that's a bad habit. If you keep doing that it's going to hurt you.” How do you approach that as a company from a non-interactive perspective of someone who's going to watch a video and do things and maybe it'll work, maybe it won't? Particularly in light of things like, well, the competition is YouTube, which, you know, I'm going to roll the dice and sometimes I'll see a great tutorial, sometimes I'll see one that I don't realize teaching me terrible things, and then it's going to recommend some baseless conspiracy theory because YouTube. How do you differentiate that? What makes Fender Play different?Michael: So currently, you're right; it's just a video-based instruction app. There's not any way to, like, provide direct feedback to students within the web and mobile applications. However, we do have an online community, and our Fender Play instructors do an office hours feature, is where they'll actually answer questions live and talk to students. We are investigating and doing some earlier research in some, possibly, being able to provide that type of feedback to users, but it's very challenging problem, just due to the nature of you're playing an instrument that has multiple strings, so you're trying to pick out the chord that they're playing in, and the timing. But it's something we definitely need to add.Corey: There's something to be said as well for the kind of care and attention that you folks wind up putting into your media where, “This is how you finger a chord,” and someone on the YouTube video will do it for two-tenths of a second, and they're filming it with a potato that isn't focused properly and pointing at the wrong part of the guitar. You folks have a high bar for quality on this. Is that done in-house? Do you wind up just going through a bunch of random folks that you just wind up offering a bunch of gift cards to, or free guitars to do this? How does the program work on the back end?Michael: So, we have an in-house curriculum team that puts together the lesson plans to really help people learn in small bite-sized lessons so that it's not too overwhelming at once. And that curriculum then is shot and filmed by an in-house video team that put that together; they upload the data into S3 for the final cut, then that gets transcoded via MediaConvert, and we serve it up via CloudFront.Corey: It's rare to wind up talking to a company that is something of a household name about something that they're doing, and hear the AWS services that they're using not trend toward a baseline mean if I can be so bold. Normally, you'll see some of the case studies, like, “Oh, this is an online bank. What services are they using?” “Oh, they're using EC2, and S3, and load balancing because did you miss the part where it's a bank?” They're not going to use these far-future services due to regulatory risk, among other things, in many cases.You're using Elemental MediaConvert, which is one of those relatively high-up-the-stack offerings that isn't broadly known. It's one of those services that is focused on specific use cases and specific industry verticals in a way that a baseline primitive service isn't. What does MediaConvert do?Michael: What it does is it takes the final edit of the video, and we have several different presets so that it will put it into an HLS format with different bitrates so that the user is getting the best quality video depending on their bandwidth.Corey: When I looked into it in the early days when it was first launching, I found that it looked an awful lot like Elastic Transcoder, which is a service that they've had for a while, only they changed up some of the capabilities. It's obviously far more capable as a service, but they also added something that felt like 15 different billing dimensions to it, “So, what is this going to cost me?” “Well, we're going to run it for a month and find out if we're still in business.” And it seemed like it was one of those very difficult to get started with and run experiments with service. Now, obviously, services evolve over time. When you started looking into it was that experience roughly akin to what you felt, or am I completely and unfairly slandering in the product?Michael: We actually started out using Elastic Transcoder and then moved over to MediaConvert, I believe it was last year. We found it to be a little bit easier to use, and the pricing overall in transcoding the videos for us is really a drop in the bucket as compared to actually hosting them and serving them up via CloudFront. And when we switched over to MediaConvert, we adjusted our settings to lower the maximum bitrate for a given video, we found that after a certain point, the quality to the user just doesn't really improve, and yet we're paying to serve the larger video.Corey: One statistic that I found was that in March of 2020—you know which I believe we're still in at this point; just, it's the Endless September model, applied to March—you wound up seeing over an order of magnitude in traffic increase within five days, and looking at that through a lens of traditional architecture, that means that nobody sleeps a whole heck of a lot. Given that you're in on the serverless story, and you have been since before that hit, what was that scaling experience like for you?Michael: Scaling experience was completely seamless. We use a lot of Lambda, DynamoDB, Kinesis, SNS, to glue things together, and no problems whatsoever. Just had to bump up our Elasticsearch cluster a bit, that was really the only thing because we saw some latency starting to rise on some of our APIs.Corey: Let me ask the uncomfortable question then because whenever I tried to scale things up quickly in a cloud environment, what was your experience with smacking into various AWS service limits as the traffic grew?Michael: Initially, we actually requested some service limits increase to make sure we weren't hitting the concurrent Lambda invocation limit, and same thing with Cognito, making sure that we weren't going to hit any limits as far as sign-ins and things like that. So, we were able to just put in requests, and they served us around pretty quick turnaround time on that, as well.Corey: It really does seem like there's a strong benefit on the serverless space, but I had to double-check before we started recording that you do, in fact, work at Fender because you are a staunch advocate for observability. And usually, when someone is that passionate about observability, you can guess that they work at an observability-slash-monitoring company. It's akin to the idea of someone selling mattresses telling you that mattresses are great and you should have four of them. You're on the customer side of that and still very passionate about it. Where'd that come from?Michael: Came from my time years ago, when I worked at MySpace—if anyone can still remember that—working on the search systems there. And as the company started winding down, to laying people off, and being one of the only people left working on those systems, being able to know and understand them, you just have to, so you have to continue to monitor and find ways to monitor, and that really ingrained how important instrumentation is and being able to really understand the health of your application as it's running so that you can see, yes, everything is good, and then when something doesn't look right so that you can know where to start looking, and you can be alerted of a problem.Corey: So, I tend to view the world in olden terms where monitoring was what we did, and we use something like Nagios, which was the second-worst option out there because everything else felt like it was tied for first. I also take a somewhat regressive view that observability is to monitoring as DevOps is to being a systems administrator. It's the same thing, but by using the more modern terminology, you can charge more for it. I'm going to go out on a limb and guess that you take a somewhat contrarian [laugh] view to that.Michael: Yes, yes, I do. It's about really understanding how your applications is running. It's not just looking at, oh, how many HTTP 500s am I serving up per hour, if I hit a threshold for the last hour? It's a lot more than that. It's really being able to really dig in and see what the issue is or what's working really well.And to that end, we rely on two services for this. We use Honeycomb and Epsagon. Honeycomb, kind of, acts as our top layer because it gives us the really good high-cardinality metrics where I can punch in a user ID and I can see all the API traffic that this user has performed. As well as, even just like when we launched the Playthrough when our traffic rose, that the reason we discovered that our latency was dropping was due to a service-level objective being triggered in Honeycomb on latency. And we were able to respond to that using that before customers really noticed anything at all.Corey: As an Epsagon customer myself, I'm always conflicted when I find myself going into their service and using it to figure out what the heck's going on with my giant pile of Lambda functions, and API gateways, and whatnot, wired together because the experience is uniformly excellent, but I'm also frustrated in that it needs a third-party to even begin to allude to what's going on. It feels, on some level, like the vendor that is providing this service to me should be reasonably effective at telling me what it's doing, and when it's breaking. I understand that how I wish the world is and how it actually is are two radically different things but does that ever strike you as well?Michael: Whether or not AWS should be providing that type of level, that seems… that seems like more of a service that you can have competition and other vendors that really specialize and get in the weeds on it. I don't think AWS needs to provide every service you could possibly use for your application. That's not something I'm too concerned about. I don't really even think it's their place, frankly.Corey: No, no, I understand. The problem I keep running into, on some level, whenever I try and diagnose it natively is, I look at CloudWatch and it's difficult to understand that is this—in my case because again, I'm still early days with a lot of these things—is it the API gateway that's having the problem? Is it the CloudFront distribution that is tied to that? Is it the Lambda function? Where's the handoff?Trying to understand where in a complicated application the failure is occurring is a challenge. And let's be clear, most of that is a problem of my own making because I didn't have the good sense to instrument this thing in a reliable repeatable way when I built it. It feels like everything is tied together with duct tape, and baling wire, and spit, and a bit of luck. As a counterpoint, the more companies I talk to, the more I realize that no, no, this is actually how most people feel [laugh] when they look at things that are working. It's, yeah, it's terrible. It's a trash fire, but it makes money so we're going to roll with it.And there's always, on some level, a sense of what we've built is very far from the platonic ideal of what we should have built. Does that resonate with you, or do you take a step back and look at what you've achieved with a perspective of, “This is awesome. More people should do it exactly like this.” And honestly, if it's that one, I'd love to take a look at what you've built.Michael: I think there's always room for us to improve on what we're doing because we're constantly learning and evolving to improve both, even at such a low level of like, “Okay, how do we lay out the files in our service repository to make the best organization to make sense?” All the way up to, “Okay, how are we going to do tracing? And what kind of information do we need to get from that so that we can find problems when they occur?” We're always looking to learn what others are doing, and talking to others in this space. No one will ever be a hundred percent right. There's always room for improvement everywhere.Corey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I'm going to just guess that it's awful because it's always awful. No one loves their deployment process. What if launching new features didn't require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren't what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Corey: One thing that you folks have done that I think was really interesting and didn't get as much play as I think it really deserved, was that, especially in the early days of the pandemic, you wound up seeing that massive increase due to giving out almost a million free three-month subscriptions to Playthrough. Additionally, you also worked closely with LAUSD, the Los Angeles Unified School District, to add Fender Play to their middle school music program's curriculum to help supplement their remote learning programs. First, was that all in the same timeframe? Or—and, two, what has it been like, I guess, working with a organization that is, I guess, on some level, not particularly cloud-first. I would imagine. When I lived in Los Angeles, I never got the sense that LAUSD was full-on serverless, full on-board with cloud, full on-board with remote learning. And then the pandemic of course exacerbates all of that.Michael: Yeah, so those were really two different projects. So, that the Playthrough project that started in March, and we started working with Los Angeles Unified School District last year during their summer school program; started out with 1500 students and we put it together very quickly. Essentially, we use the same three-month codes that we used for that Playthrough promotion so that we could set things up very quickly for students and gave out, through our nonprofit arm of Fender, the Fender Play Foundation, gave out 1500 instruments to these students to use during the summer school program. And that program became so successful, we continued on with them in the fall, and now in the current semester, and we will be again this summer. I believe there's 7000 students in the program now.And working with their IT team has actually been quite nice. And in dealing with partners, you wouldn't think much of, “Oh, it's a school district, what do they have?” But as far as just ease of working with them, we actually hooked into their SAML provider in Cognito so that LAUSD students could authenticate when they come in through the remote learning systems. And they were great to work with and very helpful and cooperative.Corey: One of the arguments that you'll see that comes up against serverless, from time to time, is that you are now indelibly linked to your provider, but you can't take what you've built with all of these services and just move it over to Azure or GCP on a moment's whim. Now, in practice, people who tend to build for that, just build everything on top of EC2 and very little else, and then run it entirely in AWS and never move it to any of those other places. But was there friction with making that, I guess, architectural commitment to a single vendor?Michael: Oh, you're bringing up the vendor lock-in Boogeyman.Corey: Oh, I absolutely am. Most people who bring that—when I bring it up as a straw man so you can attack it, most people who bring up the vendor lock-in Boogeyman, “Oh, you have to go multi-cloud,” are either trying to sell you something that is required if you want to go multi-cloud, or they're a cloud provider themselves who know that if you go all-in on one provider, it will certainly not be theirs.Michael: I think if you properly architect your applications with separations of concerns that you could move to, say—okay, say Lambda wasn't working out for us anymore, and we needed to take our applications and, where, we're going to put them into a container, but we're going to stay in AWS. Our applications are set up in such a way that Lambda is basically a deployment pattern. We could easily convert those individual function handlers into route handlers with a minimal effort because the business logic and then the underlying data storage are separated. So, it would be feasible for us if we wanted to, say, move to Azure and use Azure Functions and whatever comparable service they have to DynamoDB. I'm not too familiar with a lot of their offerings.But that would certainly be possible to do it with, obviously, some effort and really, at the end of the day, the resources you have working on the applications are end up going to costing you much more than any, sort of like, software licensing or specific savings you're going to get from a cloud vendor, so might as well go ahead and just use those service that they're providing. So that you can just focus on the business.Corey: My approach has almost universally been that looking at an awful lot of companies and their AWS bills, it is a challenge to find an environment where the resources in the environment cost more than the people who are operating them. In the context of business, AWS bills seemed giant and enormous, right up until you look at payroll and then it's, “Oh, okay.” That's counterintuitive for folks who are learning this, and I fall prey to it myself is, when I'm playing around as a hobbyist trying to build something I value, my time is free because I'm learning as this goes, and then in that context, especially when I was starting out as a student, it was, “Oh, great. So, this winds up costing me $7 a month. Oh, that's a lot of money. That's my ramen budget, so I'm instead going to wind up spending eight hours avoiding it charging me anything.” It's the exact opposite from the direction you want staff that you're paying to work on these things to go in. How do you approach the idea of increasing the cloud cost if it will save time for your team?Michael: It's a balance between, where do we need to build this ourselves? And then not only build it, you have to operate it and maintain it? Or what is the cost of getting this third-party service? And that's really what it comes down to in all of them. And do we actually want to spend time working on this piece of infrastructure that these other people are specializing in and do so well? I've got better things I can have people doing than that.Corey: Speaking of people, one thing that you talk about, as you self-describe, is that you wind up not writing a whole lot of code anymore, but you're something of a stickler for observability and enforcing consistency between services, so you'll periodically do things like submit a PR to tweak a log message to put your mind at ease, was one example that you gave. Given that you're a director, which is generally manager of managers style approaches, how do you avoid having those PRs come across to your team as either micromanagement or a condemnation of what they've built? Because I get it; when I see something that's easy and small to tweak, I want to go ahead and get it fixed immediately. I don't want to go back and forth and play those games; I just want it done. But I'm also always weighing that against, I don't want to have people think that I'm judging them somehow for something I'm very much not.Michael: That's a very good point. The larger technical decisions on how things are laid out, I generally just try to—I don't insert myself into. I let the team go ahead, and make those decisions, and leave that direction, and let them take the charge on that, and I take the approach of looking at it as more of a guiding, and mentoring and teaching to really hone and instill that discipline in really being able to understand what the applications are doing. And as our team is growing, I have less and less time to even do those things, but I can go through the systems and go, “Hey, how come we're not tracing this call to the reCAPTCHA servers? Let's add that in there.” And I'll just at this point now, I mainly just write Jira tickets to have someone else actually do the work.Corey: The more I do this, the more I realize that as complicated as the technology is, the people are in many ways, far more complicated. And let's be fair here, non-deterministic things that work super well on one person one month could work entirely differently a following month, or even with the same person, or between teams. It's a constant balancing act, on some level. And giving people a sense of psychological safety has always been the biggest challenge. The thing that surprised me about management, back when I was running ops teams was the more, I guess, responsibility you accrue as you rise from individual contributor into the management—or ‘rise' is sort of a wrong term; it's an orthogonal transition—is that you spend a lot more time on the people problems, and your ability to directly control or affect change diminishes because you have to do everything via influence. You get a lot more responsibility with a lot less direct power [laugh] over the outcome in some ways. Does that align with how you see it, or am I just—do I have very strange approaches on management? Which may be true, and why I got out of it as fast as I could.Michael: No, that is a good point because you are having to [unintelligible 00:27:05], like, influence, and guide, and more take a higher-level view, as opposed to really getting into the weeds of like, “Okay, what methods are we going to put on this interface? How are we going to, say, architect the internals of an application?” Those are details I just really don't have time for anymore. But larger things as to making sure that we're okay, it's like, “What's the performance of this?” And, “Overall, is something that can be adapted as the business needs change, and as we change? And as we learn, what can we do to modify it?” And more just things like guiding, and mentoring, and really taking a higher-level view of that.Corey: I'm going to selfishly ask about something that I struggle with myself. That goes a bit more into the technical area, but you talk about enforcing consistency across all of your different services. What does that mean? Similar coding style? Similar instrumentation?Because I look at the things I built and microservices that power my internal nonsense, and each one of those is very different than all the rest. So, whatever your version of consistency is, I know I'm not doing it. But how do you view it?Michael: So, there's really two types of consistency. The one I really refer to the most is in observability. So that, if you've got a thousand Lambda functions out there, and each one is logging things slightly differently, that's just a pain to deal with, and realistically, dealing with a thousand unicorns is a real pain. So, through that observability, at least in Lambda, we use an internally developed middleware to make sure that the logging is consistent, and it's easy enough to use. And then other consistency, like, just within projects of how we lay things out.That's something that's been consistently evolving. What's the folder structure in how we organize the code? And we've kind of been evolving that over the last three years. And within about the last six months, we've come up with a really good pattern and a template for the future. And it's not much different from what we started out with, but it's a little bit easier, really, to comprehend as a new engineer coming in. It makes more sense.Corey: I have to ask—and I understand if you don't want to give a particular endorsement in any direction—but do you go through Serverless Framework, SAM CLI, the CDK, using the console and then lying about it? What is the template that you wind up using for that uniformity? Because even internally, I use three or four of those different things and professional advice: don't do that.Michael: Let's see. So, in our development, QA, production environments, infrastructure is all managed with Terraform. Each engineer has their own personal AWS account so that they can work on things there—Corey: Oh, that makes billing granularity super easy.Michael: Oh, yes. You can tell who's got EC2 instances running up for too long. But for the most part, we'll use Serverless Framework in that regard to say—for the engineer can just deploy into your local environment. Although we are working on ways to reuse the Terraform infrastructure and deploy that. But we have our own build and deployment pipeline that we built using CircleCI, and all of our Lambda functions are in Go.And so having to compile, say, 20 binaries in a service, that gets kind of slow, one of our DevOps engineers actually came up with a way to use Lambda to build the Lambdas, so that we can build them all in a distributed parallel fashion during the build process.Corey: One thing that I do love about the whole serverless approach—and it is a neat part about Lambda—is no two people ever seem to do it quite the same way. You can tie things together in so many different and exciting ways, and it's fun. It's almost like a modern version of playing with Lego. And I know that if Jeff Barr is listening, he just perked up at that. But I love the concept that you can take so many different ways to achieve similar outcomes. And it almost gives a bigger sense of creativity in how you approach problems. Has that been your experience?Michael: Oh, definitely. It's not only the creativity; it's also the flexibility in how you solve it, and the ability to adapt and evolve as services evolve, or change, or there's new ones are added. And to the point of using AWS, kind of, saying, “Oh, using a Lambda function to do this.” Like, using Lambda functions for customizing behavior of Cognito with the Cognito triggers, is to me, I think, a perfect way to customize the service to do exactly what you need to do.Corey: I want to thank you so much for taking the time to speak with me today. It's always appreciated. If people want to hear more about what you have to say and how you view these things or even, possibly, decide to work with you, okay can they find you?Michael: I'm somewhat active on LinkedIn. LinkedIn is the best place to find me. Please go ahead and connect to me; tell me you heard me on the podcast here.And yes, we are hiring. We have, all within our technical organization, from client, to web, and mobile engineers, data engineers, DevOps, API, we're always hiring and if we don't have something right now that fits your experience, let me know that you're interested and I'll put you on the list so that when we do have an opening, we'll reach out right away.Corey: And we will, of course, include links to that in the [show notes 00:32:20]. Thank you so much for being so generous with your time. I appreciate it.Michael: Thanks for having me on, Corey. It was nice talking to you.Corey: Michael Garski, Director of Platform Engineering at Fender Musical Instruments. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with a comment telling me that I'm almost certainly doing that chord incorrectly.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

covid-19 director amazon los angeles pr state cloud scale id lego saas similar governance api myspace puppets screaming aws server apis qa devops azure boogeyman sns s3 prs fender fandango lambda free world serverless jira honeycomb terraform gcp lausd elasticsearch los angeles unified school district playthrough cdk ec2 circleci yaml algolia saml platform engineering zuora kinesis launchdarkly azure functions hls recaptcha dynamodb cognito corey quinn leo fender cloudfront lambdas michael it michael oh michael yeah cloudwatch michael yes michael so nagios viant serverless framework duckbill group fender play jeff barr michael thanks fender musical instruments chief cloud economist last week in aws humblepod keep on rockin'

Episode #107: Serverless Infrastructure as Code with Ben Kehoe

Play Episode Listen Later Jun 28, 2021 79:04

About Ben KehoeBen Kehoe is a Cloud Robotics Research Scientist at iRobot and an AWS Serverless Hero. As a serverless practitioner, Ben focuses on enabling rapid, secure-by-design development of business value by using managed services and ephemeral compute (like FaaS). Ben also seeks to amplify voices from dev, ops, and security to help the community shape the evolution of serverless and event-driven designs.Twitter: @ben11kehoeMedium: ben11kehoeGitHub: benkehoeLinkedIn: ben11kehoeiRobot: www.irobot.comWatch this episode on YouTube: https://youtu.be/B0QChfAGvB0 This episode is sponsored by CBT Nuggets and Lumigo.TranscriptJeremy: Hi, everyone. I'm Jeremy Daly.Rebecca: And I'm Rebecca Marshburn.Jeremy: And this is Serverless Chats. And this is a momentous occasion on Serverless Chats because we are welcoming in Rebecca Marshburn as an official co-host of Serverless Chats.Rebecca: I'm pretty excited to be here. Thanks so much, Jeremy.Jeremy: So for those of you that have been listening for hopefully a long time, and we've done over 100 episodes. And I don't know, Rebecca, do I look tired? I feel tired.Rebecca: I've never seen you look tired.Jeremy: Okay. Well, I feel tired because we've done a lot of these episodes and we've published a new episode every single week for the last 107 weeks, I think at this point. And so what we're going to do is with you coming on as a new co-host, we're going to take a break over the summer. We're going to revamp. We're going to do some work. We're going to put together some great content. And then we're going to come back on, I think it's August 30th with a new episode and a whole new show. Again, it's going to be about serverless, but what we're thinking is ... And, Rebecca, I would love to hear your thoughts on this as I come at things from a very technical angle, because I'm an overly technical person, but there's so much more to serverless. There's so many other sides to it that I think that bringing in more perspectives and really being able to interview these guests and have a different perspective I think is going to be really helpful. I don't know what your thoughts are on that.Rebecca: Yeah. I love the tech side of things. I am not as deep in the technicalities of tech and I come at it I think from a way of loving the stories behind how people got there and perhaps who they worked with to get there, the ideas of collaboration and community because nothing happens in a vacuum and there's so much stuff happening and sharing knowledge and education and uplifting each other. And so I'm super excited to be here and super excited that one of the first episodes I get to work on with you is with Ben Kehoe because he's all about both the technicalities of tech, and also it's actually on his Twitter, a new compassionate tech values around humility, and inclusion, and cooperation, and learning, and being a mentor. So couldn't have a better guest to join you in the Serverless Chats community and being here for this.Jeremy: I totally agree. And I am looking forward to this. I'm excited. I do want the listeners to know we are testing in production, right? So we haven't run any unit tests, no integration tests. I mean, this is straight test in production.Rebecca: That's the best practice, right? Total best practice to test in production.Jeremy: Best practice. Right. Exactly.Rebecca: Straight to production, always test in production.Jeremy: Push code to the cloud. Here we go.Rebecca: Right away.Jeremy: Right. So if it's a little bit choppy, we'd love your feedback though. The listeners can be our observability tool and give us some feedback and we can ... And hopefully continue to make the show better. So speaking of Ben Kehoe, for those of you who don't know Ben Kehoe, I'm going to let him introduce himself, but I have always been a big fan of his. He was very, very early in the serverless space. I read all his blogs very early on. He was an early AWS Serverless Hero. So joining us today is Ben Kehoe. He is a cloud robotics research scientist at iRobot, as I said, an AWS Serverless Hero. Ben, welcome to the show.Ben: Thanks for having me. And I'm excited to be a guinea pig for this new exciting format.Rebecca: So many observability tools watching you be a guinea pig too. There's lots of layers to this.Jeremy: Amazing. All right. So Ben, why don't you tell the listeners for those that don't know you a little bit about yourself and what you do with serverless?Ben: Yeah. So I mean, as with all software, software is people, right? It's like Soylent Green. And so I'm really excited for this format being about the greater things that technology really involves in how we create it and set it up. And serverless is about removing the things that don't matter so that you can focus on the things that do matter.Jeremy: Right.Ben: So I've been interested in that since I learned about it. And at the time saw that I could build things without running servers, without needing to deal with the scaling of stuff. I've been working on that at iRobot for over five years now. As you said early on in serverless at the first serverless con organized by A Cloud Guru, now plural sites.Jeremy: Right.Ben: And yeah. And it's been really exciting to see it grow into the large-scale community that it is today and all of the ways in which community are built like this podcast.Jeremy: Right. Yeah. I love everything that you've done. I love the analogies you've used. I mean, you've always gone down this road of how do you explain serverless in a way to show really the adoption of it and how people can take that on. Serverless is a ladder. Some of these other things that you would ... I guess the analogies you use were always great and always helped me. And of course, I don't think we've ever really come to a good definition of serverless, but we're not talking about that today. But ...Ben: There isn't one.Jeremy: There isn't one, which is also a really good point. So yeah. So welcome to the show. And again, like I said, testing in production here. So, Rebecca, jump in when you have questions and we'll beat up Ben from both sides on this, but, really ...Rebecca: We're going to have Ben from both sides.Jeremy: There you go. We'll embrace him from both sides. There you go.Rebecca: Yeah. Yeah.Jeremy: So one of the things though that, Ben, you have also been very outspoken on which I absolutely love, because I'm in very much closely aligned on this topic here. But is about infrastructure as code. And so let's start just quickly. I mean, I think a lot of people know or I think people working in the cloud know what infrastructure as code is, but I also think there's a lot of people who don't. So let's just take a quick second, explain what infrastructure as code is and what we mean by that.Ben: Sure. To my mind, infrastructure as code is about having a definition of the state of your infrastructure that you want to see in the cloud. So rather than using operations directly to modify that state, you have a unified definition of some kind. I actually think infrastructure is now the wrong word with serverless. It used to be with servers, you could manage your fleet of servers separate from the software that you were deploying onto the servers. And so infrastructure being the structure below made sense. But now as your code is intimately entwined in the rest of your resources, I tend to think of resource graph definitions rather than infrastructure as code. It's a less convenient term, but I think it's worth understanding the distinction or the difference in perspective.Jeremy: Yeah. No, and I totally get that. I mean, I remember even early days of cloud when we were using the Chefs and the Puppets and things like that, that we were just deploying the actual infrastructure itself. And sometimes you deploy software as part of that, but it was supporting software. It was the stuff that ran in the runtime and some of those and some configurations, but yeah, but the application code that was a whole separate process, and now with serverless, it seems like you're deploying all those things at the same time.Ben: Yeah. There's no way to pick it apart.Jeremy: Right. Right.Rebecca: Ben, there's something that I've always really admired about you and that is how strongly you hold your opinions. You're fervent about them, but it's also because they're based on this thorough nature of investigation and debate and challenging different people and yourself to think about things in different ways. And I know that the rest of this episode is going to be full with a lot of opinions. And so before we even get there, I'm curious if you can share a little bit about how you end up arriving at these, right? And holding them so steady.Ben: It's a good question. Well, I hope that I'm not inflexible in these strong opinions that I hold. I mean, it's one of those strong opinions loosely held kind of things that new information can change how you think about things. But I do try and do as much thinking as possible so that there's less new information that I have to encounter to change an opinion.Rebecca: Yeah. Yeah.Ben: Yeah. I think I tend to try and think about how people ... But again, because it's always people. How people interact with the technology, how people behave, how organizations behave, and then how technology fits into that. Because sometimes we talk about technology in a vacuum and it's really not. Technology that works for one context doesn't work for another. I mean, a lot of my strong opinions are that there is no one right answer kind of a thing, or here's a framework for understanding how to think about this stuff. And then how that fits into a given person is just finding where they are in that more general space. Does that make sense? So it's less about finding out here's the one way to do things and more about finding what are the different options, how do you think about the different options that are out there.Rebecca: Yeah, totally makes sense. And I do want to compliment you. I do feel like you are very good at inviting new information in if people have it and then you're like, "Aha, I've already thought of that."Ben: I hope so. Yeah. I was going to say, there's always a balance between trying to think ahead so that when you discover something you're like, "Oh, that fits into what I thought." And the danger of that being that you're twisting the information to fit into your preexisting structures. I hope that I find a good balance there, but I don't have a principle way of determining that balance or knowing where you are in that it's good versus it's dangerous kind of spectrum.Jeremy: Right. So one of the opinions that you hold that I tend to agree with, I have some thoughts about some of the benefits, but I also really agree with the other piece of it. And this really has to do with the CDK and this idea of using CloudFormation or any sort of DSL, maybe Terraform, things like that, something that is more domain-specific, right? Or I guess declarative, right? As opposed to something that is imperative like the CDK. So just to get everybody on the same page here, what is the top reasons why you believe, or you think that DSL approach is better than that iterative approach or interpretive approach, I guess?Ben: Yeah. So I think we get caught up in the imperative versus declarative part of it. I do think that declarative has benefits that can be there, but the way that I think about it is with the CDK and infrastructure as code in general, I'm like mildly against imperative definitions of resources. And we can get into that part, but that's not my smallest objection to the CDK. I'm moderately against not being able to enforce deterministic builds. And the CDK program can do anything. Can use a random number generator and go out to the internet to go ask a question, right? It can do anything in that program and that means that you have no guarantees that what's coming out of it you're going to be able to repeat.So even if you check the source code in, you may not be able to go back to the same infrastructure that you had before. And you can if you're disciplined about it, but I like tools that help give you guardrails so that you don't have to be as disciplined. So that's my moderately against. My strongly against piece is I'm strongly against developer intent remaining client side. And this is not an inherent flaw in the CDK, is a choice that the CDK team has made to turn organizational dysfunction in AWS into ownership for their customers. And I don't think that's a good approach to take, but that's also fixable.So I think if we want to start with the imperative versus declarative thing, right? When I think about the developers expressing an intent, and I want that intent to flow entirely into the cloud so that developers can understand what's deployed in the cloud in terms of the things that they've written. The CDK takes this approach of flattening it down, flattening the richness of the program the developer has written into ... They think of it as assembly language. I think that is a misinterpretation of what's happening. The assembly language in the process is the imperative plan generated inside the CloudFormation engine that says, "Here's how I'm going to take this definition and turn it into an actual change in the cloud.Jeremy: Right.Ben: They're just translating between two definition formats in CDK scene. But it's a flattening process, it's a lossy process. So then when the developer goes to the Console or the API has to go say, "What's deployed here? What's going wrong? What do I need to fix?" None of it is framed in terms of the things that they wrote in their original language.Jeremy: Right.Ben: And I think that's the biggest problem, right? So drift detection is an important thing, right? What happened when someone went in through the Console? Went and tweaked some stuff to fix something, and now it's different from the definition that's in your source repository. And in CloudFormation, it can tell you that. But what I would want if I was running CDK is that it should produce another CDK program that represents the current state of the cloud with a meaningful file-level diff with my original program.Jeremy: Right. I'm just thinking this through, if I deploy something to CDK and I've got all these loops and they're generating functions and they're using some naming and all this kind of stuff, whatever, now it produces this output. And again, my naming of my functions might be some function that gets called to generate the names of the function. And so now I've got all of these functions named and I have to go in. There's no one-to-one map like you said, and I can imagine somebody who's not familiar with CloudFormation which is ultimately what CDK synthesizes and produces, if you're not familiar with what that output is and how that maps back to the constructs that you created, I can see that as being really difficult, especially for younger developers or developers who are just getting started in that.Ben: And the CDK really takes the attitude that it's going to hide those things from those developers rather than help them learn it. And so when they do have to dive into that, the CDK refers to it as an escape hatch.Jeremy: Yeah.Ben: And I think of escape hatches on submarines, where you go from being warm and dry and having air to breathe to being hundreds of feet below the sea, right? It's not the sort of thing you want to go through. Whereas some tools like Amplify talk about graduation. In Amplify they aim to help you understand the things that Amplify is doing for you, such that when you grow beyond what Amplify can provide you, you have the tools to do that, to take the thing that you built and then say, "Okay, I know enough now that I understand this and can add onto it in ways that Amplify can't help with."Jeremy: Right.Ben: Now, how successful they are in doing that is a separate question I think, but the attitude is there to say, "We're looking to help developers understand these things." Now the CDK could also if the CDK was a managed service, right? Would not need developers to understand those things. If you could take your program directly to the cloud and say, "Here's my program, go make this real." And when it made it real, you could interact with the cloud in an understanding where you could list your deployed constructs, right? That you can understand the program that you wrote when you're looking at the resources that are deployed all together in the cloud everywhere. That would be a thing where you don't need to learn CloudFormation.Jeremy: Right.Ben: Right? That's where you then end up in the imperative versus declarative part where, okay, there's some reasons that I think declarative is better. But the major thing is that disconnect that's currently built into the way that CDK works. And the reason that they're doing that is because CloudFormation is not moving fast enough, which is not always on the CloudFormation team. It's often on the service teams that aren't building the resources fast enough. And that's AWS's problem, AWS as an entire company, as an organization. And this one team is saying, "Well, we can fix that by doing all this client side."What that means is that the customers are then responsible for all the things that are happening on the client side. The reason that they can go fast is because the CDK team doesn't have ownership of it, which just means the ownership is being pushed on customers, right? The CDK deploys Lambda functions into your account that they don't tell you about that you're now responsible for. Right? Both the security and operations of. If there are security updates that the CDK team has to push out, you have to take action to update those things, right? That's ownership that's being pushed onto the customer to fix a lack of ACM certificate management, right?Jeremy: Right. Right.Ben: That is ACM not building the thing that's needed. And so AWS says, "Okay, great. We'll just make that the customer's problem."Jeremy: Right.Ben: And I don't agree with that approach.Rebecca: So I'm sure as an AWS Hero you certainly have pretty good, strong, open communication channels with a lot of different team members across teams. And I certainly know that they're listening to you and are at least hearing you, I should say, and watching you and they know how you feel about this. And so I'm curious how some of those conversations have gone. And some teams as compared to others at AWS are really, really good about opening their roadmap or at least saying, "Hey, we hear this, and here's our path to a solution or a success." And I'm curious if there's any light you can shed on whether or not those conversations have been fruitful in terms of actually being able to get somewhere in terms of customer and AWS terms, right? Customer obsession first.Ben: Yeah. Well, customer obsession can mean two things, right? Customer obsession can mean giving the customer what they want or it can mean giving the customer what they need and different AWS teams' approach fall differently on that scale. The reason that many of those things are not available in CloudFormation is that those teams are ... It could be under-resourced. They could have a larger majority of customer that want new features rather than infrastructure as code support. Because as much as we all like infrastructure as code, there are many, many organizations out there that are not there yet. And with the CDK in particular, I'm a relatively lone voice out there saying, "I don't think this ownership that's being pushed onto the customer is a good thing." And there are lots of developers who are eating up CDK saying, "I don't care."That's not something that's in their worry. And because the CDK has been enormously successful, right? It's fixing these problems that exists. And I don't begrudge them trying to fix those problems. I think it's a question of do those developers who are grabbing onto those things and taking them understand the full total cost of ownership that the CDK is bringing with it. And if they don't understand it, I think AWS has a responsibility to understand it and work with it to help those customers either understand it and deal with it, right? Which is where the CDK takes this approach, "Well, if you do get Ops, it's all fine." And that's somewhat true, but also many developers who can use the CDK do not control their CI/CD process. So there's all sorts of ways in which ... Yeah, so I think every team is trying to do the best that they can, right?They're all working hard and they all have ... Are pulled in many different directions by customers. And most of them are making, I think, the right choices given their incentives, right? Given what their customers are asking for. I think not all of them balance where customers ... meeting customers where they are versus leading them where they should, like where they need to go as well as I would like. But I think ... I had a conclusion to that. Oh, but I think that's always a debate as to where that balance is. And then the other thing when I talk about the CDK, that my ideal audience there is less AWS itself and more AWS customers ...Rebecca: Sure.Ben: ... to understand what they're getting into and therefore to demand better of AWS. Which is in general, I think, the approach that I take with AWS, is complaining about AWS in public, because I do have the ability to go to teams and say, "Hey, I want this thing," right? There are plenty of teams where I could just email them and say, "Hey, this feature could be nice", but I put it on Twitter because other people can see that and say, "Oh, that's something that I want or I don't think that's helpful," right? "I don't care about that," or, "I think it's the wrong thing to ask for," right? All of those things are better when it's not just me saying I think this is a good thing for AWS, but it being a conversation among the community differently.Rebecca: Yeah. I think in the spirit too of trying to publicize types of what might be best next for customers, you said total cost of ownership. Even though it might seem silly to ask this, I think oftentimes we say the words total cost of ownership, but there's actually many dimensions to total cost of ownership or TCO, right? And so I think it would be great if you could enumerate what you think of as total cost of ownership, because there might be dimensions along that matrices, matrix, that people haven't considered when they're actually thinking about total cost of ownership. They're like, "Yeah, yeah, I got it. Some Ops and some security stuff I have to do and some patches," but they might only be thinking of five dimensions when you're like, "Actually the framework is probably 10 to 12 to 14." And so if you could outline that a bit, what you mean when you think of a holistic total cost of ownership, I think that could be super helpful.Ben: I'm bad at enumeration. So I would miss out on dimensions that are obvious if I was attempting to do that. But I think a way that I can, I think effectively answer that question is to talk about some of the ways in which we misunderstand TCO. So I think it's important when working in an organization to think about the organization as a whole, not just your perspective and that your team's perspective in it. And so when you're working for the lowest TCO it's not what's the lowest cost of ownership for my team if that's pushing a larger burden onto another team. Now if it's reducing the burden on your team and only increasing the burden on another team a little bit, that can be a lower total cost of ownership overall. But it's also something that then feeds into things like political capital, right?Is that increased ownership that you're handing to that team something that they're going to be happy with, something that's not going to cause other problems down the line, right? Those are the sorts of things that fit into that calculus because it's not just about what ... Moving away from that topic for a second. I think about when we talk about how does this increase our velocity, right? There's the piece of, "Okay, well, if I can deploy to production faster, right? My feedback loop is faster and I can move faster." Right? But the other part of that equation is how many different threads can you be operating on and how long are those threads in time? So when you're trying to ship a feature, if you can ship it and then never look at it again, that means you have increased bandwidth in the future to take on other features to develop other new features.And so even if you think about, "It's going to take me longer to finish this particular feature," but then there's no maintenance for that feature, that can be a lower cost of ownership in time than, "I can ship it 50% faster, but then I'm going to periodically have to revisit it and that's going to disrupt my ability to ship other things," right? So this is where I had conversations recently about increasing use of Step Functions, right? And being able to replace Lambda functions with Step Functions express workflows because you never have to go back to those Lambdas and update dependencies in them because dependent bot has told you that you need to or a version of Python is getting deprecated, right? All of those things, just if you have your Amazon States Language however it's been defined, right?Once it's in there, you never have to touch it again if nothing else changes and that means, okay, great, that piece is now out of your work stream forever unless it needs to change. And that means that you have more bandwidth for future things, which serverless is about in general, right? Of say, "Okay, I don't have to deal with this scaling problems here. So those scaling things. Once I have an auto-scaling group, I don't have to go back and tweak it later." And so the same thing happens at the feature level if you build it in ways that allow you to do that. And so I think that's one of the places where when we focus on, okay, how fast is this getting me into production, it's okay, but how often do you have to revisit it ...Jeremy: Right. And so ... So you mentioned a couple of things in there, and not only in that question, but in the previous questions as you were talking about the CDK in general, and I am 100% behind you on this idea of deterministic builds because I want to know exactly what's being deployed. I want to be able to audit that and map that back. And you can audit, I mean, you could run CDK synth and then audit the CloudFormation and test against certain things. But if you are changing stuff, right? Then you have to understand not only the CDK but also the CloudFormation that it actually generates. But in terms of solving problems, some of the things that the CDK does really, really well, and this is something where I've always had this issue with just trying to use raw CloudFormation or Serverless Framework or SAM or any of these things is the fact that there's a lot of boilerplate that you often have to do.There's ways that companies want to do something specifically. I basically probably always need 1,400 lines of CloudFormation. And for every project I do, it's probably close to the same, and then add a little bit more to actually make it adaptive for my product. And so one thing that I love about the CDK is constructs. And I love this idea of being able to package these best practices for your company or these compliance requirements, excuse me, compliance requirements for your company, whatever it is, be able to package these and just hand them to developers. And so I'm just curious on your thoughts on that because that seems like a really good move in the right direction, but without the deterministic builds, without some of these other problems that you talked about, is there another solution to that that would be more declarative?Ben: Yeah. In theory, if the CDK was able to produce an artifact that represented all of the non-deterministic dependencies that it had, right? That allowed you to then store that artifacts as you'd come back and put that into the program and say, "I'm going to get out the same thing," but because the CDK doesn't control upstream of it, the code that the developers are writing, there isn't a way to do that. Right? So on the abstraction front, the constructs are super useful, right? CloudFormation now has modules which allow you to say, "Here's a template and I'm going to represent this as a CloudFormation type itself," right? So instead of saying that I need X different things, I'm going to say, "I packaged that all up here. It is as a type."Now, currently, modules can only be playing CloudFormation templates and there's a lot of constraints in what you can express inside a CloudFormation template. And I think the answer for me is ... What I want to see is more richness in the CloudFormation language, right? One of the things that people do in the CDK that's really helpful is say, "I need a copy of this in every AZ."Jeremy: Right.Ben: Right? There's so much boilerplate in server-based things. And CloudFormation can't do that, right? But if you imagine that it had a map function that allowed you to say, "For every AZ, stamp me out a copy of this little bit." And then that the CDK constructs allowed to translate. Instead of it doing all this generation only down to the L one piece, instead being able to say, "I'm going to translate this into more rich CloudFormation templates so that the CloudFormation template was as advanced as possible."Right? Then it could do things like say, "Oh, I know we need to do this in every AZ, I'm going to use this map function in the CloudFormation template rather than just stamping it out." Right? And so I think that's possible. Now, modules should also be able to be defined as CDK programs. Right? You should be able to register a construct as a CloudFormation tag.Jeremy: It would be pretty cool.Ben: There's no reason you shouldn't be able to. Yeah. Because I think the declarative versus imperative thing is, again, not the most important piece, it's how do we move ... It's shifting right in this case, right? That how do you shift what's happening with the developer further into the process of deployment so that more of their context is present? And so one of the things that the CDK does that's hard to replicate is have non-local effects. And this is both convenient and I think of code smell often.So you can pass a bucket resource from another stack into a piece of code in your CDK program that's creating a different stack and you say, "Oh great, I've got this Lambda function, it needs permissions to that bucket. So add permissions." And it's possible for the CDK programs to either be adding the permissions onto the IAM role of that function, or non-locally adding to that bucket's resource policy, which is weird, right? That you can be creating a stack and the thing that you do to that stack or resource or whatever is not happening there, it's happening elsewhere. I don't think that's a great approach, but it's certainly convenient to be able to do it in a lot of situations.Now, that's not representable within a module. A module is a contained piece of functionality that can't touch anything else. So things like SAM where you can add events onto a function that can go and create ... You create the API events on different functions and then SAM aggregates them and creates an API gateway for you. Right? If AWS serverless function was a module, it couldn't do that because you'd have these in different places and you couldn't aggregate something between all of them and put them in the top-level thing, right?This is what CloudFormation macros enable, but they don't have a... There's no proper interface to them, right? They don't define, "This is what I'm doing. This is the kind of resources I can create." There's none of that that would help you understand them. So they're infinitely flexible, but then also maybe less principled for that reason. So I think there are ways to evolve, but it's investment in the CloudFormation language that allows us to shift that burden from being a flattening inside client-side code from the developer and shifting it to be able to be represented in the cloud.Jeremy: Right. Yeah. And I think from that standpoint too if we go back to the solving people's problems standpoint, that everything you explained there, they're loaded with nuances, it's loaded with gotchas, right? Like, "Oh, you can't do this, you can't do that." So that's just why I think the CDK is so popular because it's like you can do so much with it so quickly and it's very, very fast. And I think that trade-off, people are just willing to make it.Ben: Yes. And that's where they're willing to make it, do they fully understand the consequences of it? Then does AWS communicate those consequences well? Before I get into that question of, okay, you're a developer that's brand new to AWS and you've been tasked with standing up some Kubernetes cluster and you're like, "Great. I can use a CDK to do this." Something is malfunctioning. You're also tasked with the operations and something is malfunctioning. You go in through the Console and maybe figure out all the things that are out there are new to you because they're hidden inside L3 constructs, right?You're two levels down from where you were defining what you want, and then you find out what's wrong and you have no idea how to turn that into a change in your CDK program. So instead of going back and doing the thing that infrastructure as code is for, which is tweaking your program to go fix the problem, you go and you tweak it in the Console ...Jeremy: Right. Which you should never do.Ben: ... and you fix it that way. Right. Well, and that's the thing that I struggle with, with the CDK is how does the CDK help the developer who's in that situation? And I don't think they have a good story around that. Now, I don't know. I haven't talked with enough junior developers who are using the CDK about how often they get into that situation. Right? But I always say client-side code is not a replacement for a managed service because when it's client-side code, you still own the result.Jeremy: Right.Ben: If a particular CDK construct was a managed service in AWS, then all of the resources that would be created underneath AWS's problem to make work. And the interface that the developer has is the only level of ownership that they have. Fargate is this. Because you could do all the things that Fargate does with a CDK construct, right? Set up EC2, do all the things, and represent it as something that looks like Fargate in your CDK program. But every time your EC2 fleet is unhealthy that's your problem. With Fargate, that's AWS's problem. If we didn't have Fargate, that's essentially what CDK would be trying to do for ECS.And I think we all recognize that Fargate is very necessary and helpful in that case, right? And I just want that for all the things, right? Whenever I have an abstraction, if it's an abstraction that I understand, then I should have a way of zooming into it while not having to switch languages, right? So that's where you shouldn't dump me out the CloudFormation to understand what you're doing. You should help me understand the low-level things in the same language. And if it's not something that I need to understand, it should be a managed service. It shouldn't be a bunch of stuff that I still own that I haven't looked at.Jeremy: Makes sense. Got a question, Rebecca? Because I was waiting for you to jump in.Rebecca: No, but I was going to make a joke, but then the joke passed, and then I was like, "But should I still make it?" I was going to be like, "Yeah, but does the CDK let you test in production?" But that was a 32nd ago joke and then I was really wrestling with whether or not I should tell it, but I told it anyway, hopefully, someone gets a laugh.Ben: Yeah. I mean, there's the thing that Charity Majors says, right? Which is that everybody tests in production. Some people are lucky enough to have a development environment in production. No, sorry. I said that the wrong way. It's everybody has a test environment. Some people are lucky enough that it's not in production.Rebecca: Yeah. Swap that. Reverse it. Yeah.Ben: Yeah.Jeremy: All right. So speaking of talking to developers and getting feedback from them, so I actually put a question out on Twitter a couple of weeks ago and got a lot of really interesting reactions. And essentially I asked, "What do you love or hate about infrastructure as code?" And there were a lot of really interesting things here. I don't know, maybe it might be fun to go through a couple of these and get your thoughts on them. So this is probably not a great one to start with, but I thought it was interesting because this I think represents the frustration that a lot of us feel. And it was basically that they love that automation minimizes future work, right? But they hate that it makes life harder over time. And that pretty much every approach to infrastructure in, sorry, yeah, infrastructure in code at the present is flawed, right? So really there are no good solutions right now.Ben: Yeah. CloudFormation is still a pain to learn and deal with. If you're operating in certain IDEs, you can get tab completion.Jeremy: Right.Ben: If you go to CDK you get tab completion, which is, I think probably most of the value that developers want out of it and then the abstraction, and then all the other fancy things it does like pipelines, which again, should be a managed service. I do think that person is absolutely right to complain about how difficult it is. That there are many ways that it could be better. One of the things that I think about when I'm using tools is it's not inherently bad for a tool to have some friction to use it.Jeremy: Right.Ben: And this goes to another infrastructure as code tool that goes even further than the CDK and says, "You can define your Lambda code in line with your infrastructure definition." So this is fine with me. And there's some other ... I think Punchcard also lets you do some of this. Basically extracts out the bits of your code that you say, "This is a custom thing that glues together two things I'm defining in here and I'll make that a Lambda function for you." And for me, that is too little friction to defining a Lambda function.Because when I define a Lambda function, just going back to that bringing in ownership, every time I add a Lambda function, that's something that I own, that's something that I have to maintain, that I'm responsible for, that can go wrong. So if I'm thinking about, "Well, I could have API Gateway direct into DynamoDB, but it'd be nice if I could change some of these fields. And so I'm just going to drop in a little sprinkle of code, three lines of code in between here to do some transformation that I want." That is all of sudden an entire Lambda function you've brought into your infrastructure.Jeremy: Right. That's a good point.Ben: And so I want a little bit of friction to do that, to make me think about it, to make me say, "Oh, yeah, downstream of this decision that I am making, there are consequences that I would not otherwise think about if I'm just trying to accomplish the problem," right? Because I think developers, humans, in general, tend to be a bit shortsighted when you have a goal especially, and you're being pressured to complete that goal and you're like, "Okay, well I can complete it." The consequences for later are always a secondary concern.And so you can change your incentives in that moment to say, "Okay, well, this is going to guide me to say, "Ah, I don't really need this Lambda function in here. Then I'm better off in the long term while accomplishing that goal in the short term." So I do think that there is a place for tools making things difficult. That's not to say that the amount of difficult that infrastructure as code is today is at all reasonable, but I do think it's worth thinking about, right?I'd rather take on the pain of creating an ASL definition by hand for express workflow than the easier thing of writing Lambda code. Because I know the long-term consequences of that. Now, if that could be flipped where it was harder to write something that took more ownership, it'd be just easy to do, right? You'd always do the right thing. But I think it's always worth saying, "Can I do the harder thing now to pay off to pay off later?"Jeremy: And I always call those shortcuts "tomorrow-Jeremy's" problem. That's how I like to look at those.Ben: Yeah. Yes.Jeremy: And the funny thing about that too is I remember right when EventBridge came out and there was no CloudFormation support for a long time, which was super frustrating. But Serverless Framework, for example, implemented a custom resource in order to do that. And I remember looking at a clean stack and being like, "Why are there two Lambda functions there that I have no idea?" I'm like, "I didn't publish ..." I honestly thought my account was compromised that somebody had published a Lambda function in there because I'm like, "I didn't do that." And then it took me a while to realize, I'm like, "Oh, this is what this is." But if it is that easy to just create little transform functions here and there, I can imagine there being thousands of those in your account without anybody knowing that they even exist.Ben: Now, don't get me wrong. I would love to have the ability to drop in little transforms that did not involve Lambda functions. So in other words, I mean, the thing that VTL does for API Gateway, REST APIs but without it being VTL and being ... Because that's hard and then also restricted in what you can do, right? It's not, "Oh, I can drop in arbitrary code in here." But enough to say, "Oh, I want to flip ... These fields should go from a key-value mapping to a list of key-value, right? In the way that it addresses inconsistent with how tags are defined across services, those kinds of things. Right? And you could drop that in any service, but once you've defined it, there's no maintenance for you, right?You're writing JavaScript. It's not actually a JavaScript engine underneath or something. It's just getting translated into some big multi-tenant fancy thing. And I have a hypothesis that that should be possible. You should be able to do it where you could even do it in the parsing of JSON, being able to do transforms without ever having to have the whole object in memory. And if we could get that then, "Oh, sure. Now I have sprinkled all over the place all of these little transforms." Now there's a little bit of overhead if the transform is defined correctly or not, right? But once it is, then it just works. And having all those little transforms everywhere is then fine, right? And that incentive to make it harder it doesn't need to be there because it's not bringing ownership with it.Rebecca: Yeah. It's almost like taking the idea of tomorrow-Jeremy's problem and actually switching it to say tomorrow-Jeremy's celebration where tomorrow-Jeremy gets to look back at past-Jeremy and be like, "Nice. Thank you for making that decision past-Jeremy." Because I think we often do look at it in terms of tomorrow-Jeremy will think of this, we'll solve this problem rather than how do we approach it by saying, how do I make tomorrow-Jeremy thankful for it today-Jeremy? And that's a simple language, linguistic switch, but a hard switch to actually make decisions based on.Ben: Yeah. I don't think tomorrow-Ben is ever thankful for today-Ben. I think it's tomorrow-Ben is thankful for yesterday-Ben setting up the incentives correctly so that today-Ben will do the right thing for tomorrow-Ben. Right? When I think about people, I think it's easier to convince people to accept a change in their incentives than to convince them to fight against their incentives sustainably.Jeremy: Right. And I think developers and I'm guilty of this too, I mean, we make decisions based off of expediency. We want to get things done fast. And when you get stuck on that problem you're like, "You know what? I'm not going to figure it out. I'm just going to write a loop or I'm going to do whatever I can do just to make it work." Another if statement here, "Isn't going to hurt anybody." All right. So let's move to ... Sorry, go ahead.Ben: We shouldn't feel bad about that.Jeremy: You're right.Ben: I was going to say, we shouldn't feel bad about that. That's where I don't want tomorrow-Ben to have to be thankful for today-Ben, because that's the implication there is that today-Ben is fighting against his incentives to do good things for tomorrow-Ben. And if I don't need to have to get to that point where just the right path is the easiest path, right? Which means putting friction in the right places than today-Ben ... It's never a question of whether today-Ben is doing something that's worth being thankful for. It's just doing the job, right?Jeremy: Right. No, that makes sense. All right. I got another question here, I think falls under the category of service discovery, which I know is another topic that you love. So this person said, "I love IaC, but hate the fuzzy boundaries where certain software awkwardly fall. So like Istio and Prometheus and cert-manager. That they can be considered part of the infrastructure, but then it's awkward to deploy them when something like Terraform due to circular dependencies relating to K8s and things like that."So, I mean, I know that we don't have to get into the actual details of that, but I think that is an important aspect of infrastructure as code where best practices sometimes are deploy a stack that has your permanent resources and then deploy a stack that maybe has your more femoral or the ones that are going to be changing, the more mutable ones, maybe your Lambda functions and some of those sort of things. If you're using Terraform or you're using some of these other services as well, you do have that really awkward mix where you're trying to use outputs from one stack into another stack and trying to do all that. And really, I mean, there are some good tools that help with it, but I mean just overall thoughts on that.Ben: Well, we certainly need to demand better of AWS services when they design new things that they need to be designed so that infrastructure as code will work. So this is the S3 bucket notification problem. A very long time ago, S3 decided that they were going to put bucket notifications as part of the S3 bucket. Well, CloudFormation at that point decided that they were going to put bucket notifications as part of the bucket resource. And S3 decided that they were going to check permissions when the notification configuration is defined so that you have to have the permissions before you create the configuration.This creates a circular dependency when you're hooking it up to anything in CloudFormation because the dependency depends on the resource policy on an SNS topic, and SQS queue or a Lambda function depends on the bucket name if you're letting CloudFormation name the bucket, which is the best practice. Then bucket name has to exist, which means the resource has to have been created. But the notification depends on the thing that's notifying, which doesn't have the names and the resource policy doesn't exist so it all fails. And this is solved in a couple of different ways. One of which is name your bucket explicitly, again, not a good practice. Another is what SAM does, which says, "The Lambda function will say I will allow all S3 buckets to invoke me."So it has a star permission in it's resource policy. So then the notification will work. None of which is good or there's custom resources that get created, right? Now, if those resources have been designed with infrastructure as code as part of the process, then it would have been obvious, "Oh, you end up with a circular pendency. We need to split out bucket notifications as a separate resource." And not enough teams are doing this. Often they're constrained by the API that they develop first ...Jeremy: That's a good point.Ben: ... they come up with the API, which often makes sense for a Console experience that they desire. So this is where API Gateway has this whole thing where you create all the routes and the resources and the methods and everything, right? And then you say, "Great, deploy." And in the Console you only need one mutable working copy of that at a time, but it means that you can't create two deployments or update two stages in parallel through infrastructure as code and API Gateway because they both talk to this mutable working copy state and would overwrite each other.And if infrastructure as code had been on their list would have been, "Oh, if you have a definition of your API, you should be able to go straight to the deployment," right? And so trying to push that upstream, which to me is more important than infrastructure as code support at launch, but people are often like, "Oh, I want CloudFormation support at launch." But that often means that they get no feedback from customers on the design and therefore make it bad. KMS asymmetric keys should have been a different resource type so that you can easily tell which key types are in your template.Jeremy: Good point. Yeah.Ben: Right? So that you can use things like CloudFormation Guard more easily on those. Sure, you can control the properties or whatever, but you should be able to think in terms of, "I have a symmetric key or an asymmetric key in here." And they're treated completely separately because you use them completely differently, right? They don't get used to the same place.Jeremy: Yeah. And it's funny that you mentioned the lacking support at launch because that was another complaint. That was quite prevalent in this thread here, was people complaining that they don't get that CloudFormation support right away. But I think you made a very good point where they do build the APIs first. And that's another thing. I don't know which question asked me or which one of these mentioned it, but there was a lot of anger over the fact that you go to the API docs or you go to the docs for AWS and it focuses on the Console and it focuses on the CLI and then it gives you the API stuff and very little mention of CloudFormation at all. And usually, you have to go to a whole separate set of docs to find the CloudFormation. And it really doesn't tie all the concepts together, right? So you get just a block of JSON or of YAML and you're like, "Am I supposed to know what everything does here?"Ben: Yeah. I assume that's data-driven. Right? And we exist in this bubble where everybody loves infrastructure as code.Jeremy: True.Ben: And that AWS has many more customers who set things up using Console, people who learn by doing it first through the Console. I assume that's true, if it's not, then the AWS has somehow gotten on the extremely wrong track. But I imagine that's how they find that they get the right engagement. Now maybe the CDK will change some of this, right? Maybe the amount of interest that is generating, we'll get it to the point where blogs get written with CDK programs being written there. I think that presents different problems about what that CDK program might hide from when you're learning about a service. But yeah, it's definitely not ... I wrote a blog for AWS and my first draft had it as CloudFormation and then we changed it to the Console. Right? And ...Jeremy: That must have hurt. Did you die a little inside when that happened?Ben: I mean, no, because they're definitely our users, right? That's the way in which they interact with data, with us and they should be able to learn from that, their company, right? Because again, developers are often not fully in control of this process.Jeremy: Right. That's a good point.Ben: And so they may not be able to say, "I want to update this through CloudFormation," right? Either because their organization says it or just because their team doesn't work that way. And I think AWS gets requests to prevent people from using the Console, but also to force people to use the Console. I know that at least one of them is possible in IAM. I don't remember which, because I've never encountered it, but I think it's possible to make people use the Console. I'm not sure, but I know that there are companies who want both, right? There are companies who say, "We don't want to let people use the API. We want to force them to use the Console." There are companies who say, "We don't want people using the Console at all. We want to force them to use the APIs."Jeremy: Interesting.Ben: Yeah. There's a lot of AWS customers, right? And there's every possible variety of organization and AWS should be serving all of them, right? They're all customers. And certainly, I want AWS to be leading the ones that are earlier in their cloud journey and on the serverless ladder to getting further but you can't leave them behind, I think it's important.Jeremy: So that people argument and those different levels and coming in at a different, I guess, level or comfortability with APIs versus infrastructure as code and so forth. There was another question or another comment on this that said, "I love the idea of committing everything that makes my solution to text and resurrect an entire solution out of nothing other than an account key. Loved the ability to compare versions and unit tests, every bit of my solution, and not having to remember that one weird setting if you're using the Console. But hate that it makes some people believe that any coder is now an infrastructure wizard."And I think this is a good point, right? And I don't 100% agree with it, but I think it's a good point that it basically ... Back to your point about creating these little transformations in Pulumi, you could do a lot of damage, I mean, good or bad, right? When you are using these tools. What are your thoughts on that? I mean, is this something where ... And again, the CDK makes it so easy for people to write these constructs pretty quickly and spin up tons of infrastructure without a lot of guard rails to protect them.Ben: So I think if we tweak the statement slightly, I think there's truth there, which isn't about the self-perception but about what they need to be. Right? That I think this is more about serverless than about infrastructure as code. Infrastructure as code is just saying that you can define it. Right? I think it's more about the resources that are in a particular definition that require that. My former colleague, Aaron Camera says, "Serverless means every developer is an architect" because you're not in that situation where the code you write goes onto something, you write the whole thing. Right?And so you do need to have those ... You do need to be an infrastructure wizard whether you're given the tools to do that and the education to do that, right? Not always, like if you're lucky. And the self-perception is again an even different thing, right? Especially if coders think that there's nothing to be learned ... If programmers, software developers, think that there's nothing to be learned from the folks who traditionally define the infrastructure, which is Ops, right? They think, "Those people have nothing to teach me because now I can do all the things that they did." Well, you can create the things that they created and it does not mean that you're as good at it ...Jeremy: Or responsible for monitoring it too. Right.Ben: ... and have the ... Right. The monitoring, the experience of saying these are the things that will come back to bite you that are obvious, right? This is how much ownership you're getting into. There's very much a long-standing problem there of devaluing Ops as a function and as a career. And for my money when I look at serverless, I think serverless is also making the software development easier because there's so much less software you need to write. You need to write less software that deals with the hard parts of these architectures, the scaling, the distributed computing problems.You still have this, your big computing problems, but you're considering them functionally rather than coding things that address them, right? And so I see a lot of operations folks who come into serverless learn or learn a new programming language or just upscale, right? They're writing Python scripts to control stuff and then they learn more about Python to be able to do software development in it. And then they bring all of that Ops experience and expertise into it and look at something and say, "Oh, I'd much rather have step functions here than something where I'm running code for it because I know how much my script break and those kinds of things when an API changes or ... I have to update it or whatever it is."And I think that's something that Tom McLaughlin talks about having come from an outside ground into serverless. And so I think there's definitely a challenge there in both directions, right? That Ops needs to learn more about software development to be more engaged in that process. Software development does need to learn much more about infrastructure and is also at this risk of approaching it from, "I know the syntax, but not the semantics, sort of thing." Right? We can create ...Jeremy: Just because I can doesn't mean I should.Ben: ... an infrastructure. Yeah.Rebecca: So Ben, as we're looping around this conversation and coming back to this idea that software is people and that really software should enable you to focus on the things that do matter. I'm wondering if you can perhaps think of, as pristine as possible, an example of when you saw this working, maybe it was while you've been at iRobot or a project that you worked on your own outside of that, but this moment where you saw software really working as it should, and that how it enabled you or your team to focus on the things that matter. If there's a concrete example that you can give when you see it working really well and what that looks like.Ben: Yeah. I mean, iRobot is a great example of this having been the company without need for software that scaled to consumer electronics volumes, right? Roomba volumes. And needing to build a IOT cloud application to run connected Roombas and being able to do that without having to gain that expertise. So without having to build a team that could deal with auto-scaling fleets of servers, all of those things was able to build up completely serverlessly. And so skip an entire level of organizational expertise, because that's just not necessary to accomplish those tasks anymore.Rebecca: It sounds quite nice.Ben: It's really great.Jeremy: Well, I have one more question here that I think could probably end up ... We could talk about for another hour. So I will only throw it out there and maybe you can give me a quick answer on this, but I actually had another Twitter thread on this not too long ago that addressed this very, very problem. And this is the idea of the feedback cycle on these infrastructure as code tools where oftentimes to deploy infrastructure changes, I mean, it just takes time. In many cases things can run in parallel, but as you said, there's race conditions and things like that, that sometimes things have to be ... They just have to be synchronous. So is this something where there are ways where you see in the future these mutations to your infrastructure or things like that potentially happening faster to get a better feedback cycle, or do you think that's just something that we're going to have to deal with for a while?Ben: Yeah, I think it's definitely a very extensive topic. I think there's a few things. One is that the deployment cycle needs to get shortened. And part of that I think is splitting dev deployments from prod deployments. In prod it's okay for it to take 30 seconds, right? Or a minute or however long because that's at the end of a CI/CD pipeline, right? There's other things that are happening as part of that. Now, you don't want that to be hours or whatever it is. Right? But it's okay for that to be proper and to fully manage exactly what's going on in a principled manner.When you're doing for development, it would be okay to, for example, change the Lambda code without going through CloudFormation to change the Lambda code, right? And this is what an architect does, is there's a notion of a dirty deploy which just packages up. Now, if your resource graph has changed, you do need to deploy again. Right? But if the only thing that's changing is your code, sure, you can go and say, "Update function code," on that Lambda directly and that's faster.But calling it a dirty deploy is I think important because that is not something that you want to do in prod, right? You don't want there to be drift between what the infrastructure as code service understands, but then you go further than that and imagine there's no reason that you actually have to do this whole zip file process. You could be R sinking the code directly, or you could be operating over SSH on the code remotely, right? There's many different ways in which the loop from I have a change in my Lambda code to that Lambda having that change could be even shorter than that, right?And for me, that's what it's really about. I don't think that local mocking is the answer. You and Brian Rue were talking about this recently. I mean, I agree with both of you. So I think about it as I want unit tests of my business logic, but my business logic doesn't deal with AWS services. So I want to unit test something that says, "Okay, I'm performing this change in something and that's entirely within my custom code." Right? It's not touching other services. It doesn't mean that I actually need adapters, right? I could be dealing with the native formats that I'm getting back from a given service, but I'm not actually making calls out of the code. I'm mocking out, "Well, here's what the response would look like."And so I think that's definitely necessary in the unit testing sense of saying, "Is my business logic correct? I can do that locally. But then is the wiring all correct?" Is something that should only happen in the cloud. There's no reason to mock API gateway into Lambda locally in my mind. You should just be dealing with the Lambda side of it in your local unit tests rather than trying to set up this multiple thing. Another part of the story is, okay, so these deploys have to happen faster, right? And then how do we help set up those end-to-end test and give you observability into it? Right? X-Ray helps, but until X-Ray can sort through all the services that you might use in the serverless architecture, can deal with how does it work in my Lambda function when it's batching from Kinesis or SQS into my function?So multiple traces are now being handled by one invocation, right? These are problems that aren't solved yet. Until we get that kind of inspection, it's going to be hard for us to feel as good about cloud development. And again, this is where I feel sometimes there's more friction there, but there's bigger payoff. Is one of those things where again, fighting against your incentives which is not the place that you want to be.Jeremy: I'm going to stop you before you disagree with me anymore. No, just kidding! So, Rebecca, you have any final thoughts or questions for Ben?Rebecca: No. I just want to say to both of you and to everyone listening that I hope your today self is celebrating your yesterday-self right now.Jeremy: Perfect. Well, Ben, thank you so much for joining us and being a guinea pig as we said on this new format that we are trying. Excellent guinea pig. Excellent.Rebecca: An excellent human too but also great guinea pig.Jeremy: Right. Right. Pretty much so. So if people want to find out more about you, read some of the stuff you're doing and working on, how do they do that?Ben: I'm on Twitter. That's the primary place. I'm on LinkedIn, I don't post much there. And then I write articles that show up on Medium.Rebecca: And just so everyone knows your Twitter handle I'll say it out loud too. It's @ben11kehoe, K-E-H-O-E, ben11kehoe.Jeremy: Right. Perfect. All right. Well, we will put all that in the show notes and hopefully people will like this new format. And again, we'd love your feedback on this, things that you'd like us to do in the future, any ideas you have. And of course, make sure you reach out to Ben. He's an amazing resource for serverless. So again, thank you for everything you do, and thank you for being on the show.Ben: Yeah. Thanks so much for having me. This was great.Rebecca: Good to see you. Thank you.

technology moving chefs code loved software medium cloud infrastructure reverse i am iot api puppets swap aha python aws console amplify prometheus apis faa javascript ops sns x ray roomba asl ides s3 kubernetes lambda irobot acm baas serverless ci cd json iac soylent green terraform cli dsl ssh ecs l3 tco rest apis kms cdk ec2 yaml k8s kinesis istio charity majors dynamodb api gateway cloud guru fargate lambdas cloudformation sqs tom mclaughlin ben it vtl serverless framework aws hero ben there eventbridge jeremy it jeremy you aws serverless hero ben well cbt nuggets ben kehoe ben thanks ben right rebecca so jeremy so

Episode #106: Building Apps on the Decentralized Web with Nader Dabit

Play Episode Listen Later Jun 21, 2021 59:04

About Nader DabitNader Dabit is a web and mobile developer, author, and Developer Relations Engineer building the decentralized future at Edge and Node. Previously, he worked as a Developer Advocate at AWS Mobile working with projects like AWS AppSync and AWS Amplify. He is also the author and editor of React Native in Action and OpenGraphQL.Nader Dabit Twitter: @dabit3Edge and Node Twitter: @edgeandnodeGraph protocol Twitter: @graphprotocolEdge and Node: edgeandnode.com Everest: everest.link YouTube: YouTube.com/naderdabitWhat is Web3? The Decentralized Internet of the Future ExplainedWatch this episode on YouTube: https://youtu.be/pSv_cCQyCPQ This episode is sponsored by CBT Nuggets and Fauna. TranscriptJeremy: Hi everyone. I'm Jeremy Daly and this is Serverless Chats. Today I am joined again by Nader Dabit. Hey Nader, thanks for joining me.Nader: Hey Jeremy. Thanks for having me.Jeremy: You are now a developer relations engineer at Edge & Node. I would love it if you could tell the listeners a little bit about yourself. I think a lot of people probably know you already, but a little bit about your background and then what Edge & Node is.Nader: Yeah, totally. My name is Nader Dabit like you mentioned, and I've been a developer for about, I guess, nine or ten years now. A lot of people might know me from my work with AWS, where I worked with the Amplify team with the front end web and mobile team, doing a lot of full stack stuff there as well as serverless. I've been working as a developer relations person, developer advocate, actually, leading the front end web and mobile team at AWS for a little over three years I was there. I was a manager for the last year and I became really, really interested in serverless while I was there. It led to me writing a book, which is Full Stack Serverless. It also just led me down the rabbit hole of managed services and philosophy and all this stuff.It's been really, really cool to learn about everything in the space. Edge & Node is my next step, I would say, in doing work and what I consider maybe a serverless area, but it's an area that a lot of people might not associate with the traditional, I would say definition of serverless or the types of companies they often associate with serverless. But Edge & Node is a company that was spun off from a team that created a decentralized API protocol, which is called the Graph protocol. And the Graph protocol started being built in 2017. It was officially launched in a decentralized way at the end of 2020. Now we are currently finalizing that migration from a hosted service to a decentralized service actually this month.A lot of really exciting things going on. We'll talk a lot about that and what all that means. But Edge & Node itself, we do support the Graph protocol, that's part of what we do, but we also build out decentralized applications ourselves. We have a couple of applications that we're building as engineers. We're also doing a lot of work within the Web3 ecosystem, which is known as the decentralized web ecosystem by investing in different people and companies and supporting different things and spreading awareness around some of the things that are going on here because it does have a lot to do with maybe the work that people are doing in the Web2 space, which would be the traditional webspace, the space that I was in before.Jeremy: Right, right. Here I am. I follow you on Twitter. Love the videos that you do on your YouTube channel. You're like a shining example of what a really good developer relations dev advocate is. You just produce so much content, things like that, and you're doing all this stuff on serverless and I'm loving it. And then all of a sudden, I see you post this thing saying, hey, I'm leaving AWS Amplify. And you mentioned something about blockchain and I'm like, okay, wait a minute. What is this that Nader is now doing? Explain to me this, or maybe explain to me and hopefully the audience as well. What is the blockchain have to do with this decentralized applications or decentralized, I guess Web3?Nader: Web3 as defined by definition, what you might see if you do some research, would be what a lot of people are talking about as the next evolution of the web as we know it. In a lot of these articles and stuff that people are trying to formalize ideas and stuff, the original web was the read-only web where we were not creators, the only creators were maybe the developers themselves. Early on, I might've gone and read a website and been able to only interact with the website by reading information. The current version that we're currently experiencing might be considered as Web2 where everyone's a creator. All of the interfaces, all of the applications that we interact with are built specifically for input. I can actually create a comment, I can upload a video, I can share stuff, and I can write to the web. And I can read.And then the next evolution, a lot of people are categorizing, yes, is Web3. It's like taking a lot of the great things that we have today and maybe improving upon those. A lot of people and everyone kind of, this is just a really, a very old discussion around some of the trade-offs that we currently make in today's web around our data, around advertising, around the way a lot of business models are created for monetization. Essentially, they all come down to the manipulation of user data and different tricks and ways to steal people's data and use that essentially to create targeted advertising. Not only does this lead to a lot of times a negative experience. I just saw a tweet yesterday that resonated a lot with me that said, "YouTube is no longer a video platform, it's now an ad platform with videos in between." And that's the way I feel about YouTube. My kids ...Jeremy: Totally.Nader: ... I have kids that use YouTube and it's interesting to watch them because they know exactly what to do when the ads come up and exactly how to time it because they're used to, ads are just part of their experience. That's just what they're used to. And it's not just YouTube, it's every site that's out there, that's a social site, Instagram, LinkedIn. I think that that's not the original vision that people had, right, for the web. I don't think this was part of it. There have been a lot of people proposing solutions, but the core fundamental problem is how these applications are engineered, but also how the applications are paid for. How do these companies pay for developers to build. It's a really complex problem that, the simplest solution is just sell ads or maybe create something like a developer platform where you're charging a weekly or monthly or yearly or something like that.I would say a lot of the ideas around Web3 are aiming to solve this exact problem. In order to do that you have to rethink how we build applications. You have to rethink how we store data. You have to rethink about how we think about identity as well, because again, how do you build an application that deals with user data without making it public in some way? Right? How do we deal with that? A lot of those problems are the things that people are thinking about and building ways to address those in this decentralized Web3 world. It became really fascinating to me when I started looking into it because I'm very passionate about what I'm doing. I really enjoy being a developer and going out and helping other people, but I always felt there was something missing because I'm sitting here and I love AWS still.In fact, I would 100% go back and work there or any of these big companies, right? Because you can't really look at a company as, in my opinion, a black or white, good or bad thing, there's companies are doing good things and bad things at the same time. For instance, at AWS, I would meet a developer, teach them something at a workshop, a year later they would contact me and be like, hey, I got my first job or I created a business, or I landed my first client. So you're actually helping improve people's lives, at the same time you're reading these articles about Amazon in the news with some of the negative stuff going on. The way that I look at it is, I can't sit there and say any company is good or bad, but I felt a lot of the applications that people were building were also, at the end goal when you hear some of these VC discussions or people raising money, a lot of the end goal for some of the people I was working with were just selling advertising.And I'm like, is this really what we're here to do? It doesn't feel fulfilling anymore when you start seeing that over and over and over. I think the really thing that fascinated me was that people are actually building applications that are monetized in a different way. And then I started diving into the infrastructure that enabled this and realized that there was a lot of similarities between serverless and how developers would deploy and build applications in this way. And it was the entry point to my rabbit hole.Jeremy: I talked to you about this and I've been reading some of the stuff that you've been putting out and trying to educate myself on some of this. It seems very much so that show Silicon Valley on HBO, right? This decentralized web and things like that, but there's kind of, and totally correct me if I'm wrong here, but I feel there's two sides of this. You've got one side that is the blockchain, that I think some people are familiar with in the, I guess in the context of cryptocurrency, right? This is a very popular use of the blockchain because you have that redundancy and you have the agreement amongst multiple places, it's decentralized. And so you have that security there around that. But there's other uses for the blockchain as well.Especially things like banking and real estate and some of those other use cases that I'd like to talk about. And then there's another side of it that is this decentralized piece. Is the decentralized piece of it like building apps? How is that related to the blockchain or are those two separate things?Nader: Yeah, absolutely. I'm a big fan of Silicon Valley. Working in tech, it's almost like every single episode resonates with you if you've been in here long enough because you've been in one of those situations. The blockchain is part of the discussion. Crypto is part of the discussion, and those things never really interested me, to be honest. I was a speculator in crypto from 2015 until now. It's been fun, but I never really looked at crypto in any other way other than that. Blockchain had a really negative, I would say, association in my mind for a long time, I just never really saw any good things that people were doing with it. I just didn't do any research, maybe didn't understand what was going on.When I started diving into it originally what really got me interested is the Graph protocol, which is one of the things that we work on at Edge & Node. I started actually understanding, why does this thing exist? Why is it there? That led me to understanding why it was there and the fact that 90% of dApps, decentralized apps in the Ethereum ecosystem are using it. And billions of queries, companies with billions of dollars in transactions are all using this stuff. I'm like, okay, this whole world exists, but why does it exist? I guess to give you an example, I guess we can talk about the Graph protocol. And there are a lot of other web, I would say Web3 or decentralized infrastructure protocols that are out there that are similar, but they all are doing similar things in the sense of how they're actually built and how they allow participation and stuff like that.When you think of something like AWS, you think of, AWS has all of these different services. I want to build an app, I need storage. I need some type of authentication layer, maybe with Cognito, and then maybe I need someplace to execute some business logic. So maybe I'll spin up some serverless functions or create an EC2 instance, whatever. You have all these building blocks. Essentially what a lot of these decentralized protocols like the Graph are doing, are building out the same types of web infrastructure, but doing so in a decentralized way. Why does that even matter? Why is that important? Well, for instance, when you live, let's say for example in another country, I don't know, in South America and outside the United States, or even in the United States in the future, you never know. Let's say that you have some application and you've said something rude about maybe the president or something like that.Let's say that for whatever reason, somebody hacks the server that you're dealing with or whatever, at the end of the day, there is a single point of failure, right? You have your data that's controlled by the cloud provider or the government can come in and they can have control over that. The idea around some of, pretty much all of the decentralized protocols is that they are built and distributed in a way that there is no single point of failure, but there's also no single point of control. That's important when you're living in areas that have to even worry about stuff like that. So maybe we don't have to worry about that as much here, but in other countries, they might.Building something like a server is not a big deal, right? With AWS, but how would you build a server and make it available for anyone in the world to basically deploy and do so in a decentralized way? I think that's the problem that a lot of these protocols are trying to solve. For the Graph in particular, if you want to build an application using data that's stored on a blockchain. There's a lot of applications out there that are basically using the blockchain for mainly, right now it's for financial, transactional reasons because a lot of the transactions actually cost a lot of money. For instance, Uniswap is one of these applications. If you want to basically query data from a blockchain, it's not as easy as querying data from a traditional server or database.For us we are used to using something like DynamoDB, or some type of SQL database, that's very optimized for queries. But on the blockchain, you're basically having these blocks that add up every time. You create a transaction, you save it. And then someone comes behind them and they save another transaction. Over time you build up this data that's aggregated over time. But let's say you want to hit that database with the, quote-unquote, database with a query and you want to retrieve data over time, or you want to have some type of filtering mechanism. You can't do that. You can't just query blockchains the way you can from a regular database. Similar to how a database basically indexes data and stores it and makes it efficient for retrieval, the Graph protocol basically does that, but for blockchain data.Anyone that wants to build an application, one of these decentralized apps on top of blockchain data has a couple of options. They can either build their own indexing server and deploy it to somewhere like AWS. That takes away the whole idea of decentralization because then you have a single point of failure again. You can query data directly from the blockchain, from your client application, which takes a very long time. Both of those are not, I would say the most optimal way to build. But also if you're building your own indexing server, every time you want to come up with a new idea also, you have to think about the resources and time that go into it. Basically, I want to come up with a new idea and test it out, I have to basically build a server index, all this data, create APIs around it. It's time-intensive.What the Graph protocol allows you to do is, as a developer you can basically define a subgraph using YAML, similar to something like cloud formation or a very condensed version of that maybe more Serverless Framework where you're defining, I want to query data from this data source, and I want to save these entities and you deploy that to the network. And that subgraph will basically then go and look into that blockchain. And will look for all the transactions that have happened, and it will go ahead and save those and make those available for public retrieval. And also, again, one of the things that you might think of is, all of this data is public. All of the data that's on the blockchain is public.Jeremy: Right. Right. All right. Let me see if I could repeat what you said and you tell me if I'm right about this. Because this was one of those things where blockchain ... you're right. To me, it had a negative connotation. Why would you use the blockchain, unless you were building your own cryptocurrency? Right. That just seemed like that's what it was for. Then when AWS comes out with QLDB or they announced that or whatever it was. I'm like, okay, so this is interesting, but why would you use it, again, unless you're building your own cryptocurrency or something because that's the only thing I could think of you would use the blockchain for.But as you said, with these blockchains now, you have highly sensitive transactions that can be public, but a real estate transaction, for example, is something really interesting, where like, we still live in a world where if Bank of America or one of these other giant banks, JPMorgan Chase or something like that gets hacked, they could wipe out financial data. Right? And I know that's backed up in multiple regions and so forth, but this is the thing where if you're doing some transaction, that you want to make sure that transaction lives forever and isn't manipulated, then the blockchain is a good place to do that. But like you said, it's expensive to write there. But it's even harder to read off the blockchain because it's that ledger, right? It's just information coming in and coming in.So event storming or if you were doing event sourcing or something that, it's that idea. The idea with these indexers are these basically separate apps that run, and again, I'm assuming that these protocols, their software, and things that you don't have to build this yourself, essentially you can just deploy these things. Right? But this will read off of the blockchain and do that aggregation for you and then make that. Basically, it caches the blockchain. Right? And makes that available to you. And that you could deploy that to multiple indexers if you wanted to. Right? And then you would have access to that data across multiple providers.Nader: Right. No single point of failure. That's exactly right. You basically deploy a very concise configuration file that defines how you want your data stored and made available. And then it goes, and it just starts at the very beginning and it queries all those blocks or reads all those blocks, saves the data in a database, and then it keeps up with additional new updates. If someone writes a new transaction after that, it also saves that and makes it available for efficient retrieval. This is just for blockchain data. This is the data layer for, but it's not just a blockchain data in the future. You can also query from IPFS, which is a file storage layer, somewhat S3. You can query from other chains other than Ethereum, which is kind of like the main chamber.In the future really what we're hoping to have is a complete API on top of all public data. Anybody that wants to have some data set available can basically deploy a subgraph and index it and then anyone can then essentially query for it. It's like when you think of public data, we're not really used to thinking of data in this way. And also I think a good thing to talk about in a moment is the types of apps that you can build because you wouldn't want to store private messages on a blockchain or something like that. Right? The types of apps that people are building right now at least are not 100% in line with everything. You can't do everything I would say right now in Web3 that you can do in Web2.There are only certain types of applications, but those applications that are successful seem to be wildly successful and have a lot of people interested in them and using them. That's the general idea, is like you have this way to basically deploy APIs and the technology that we use to query is GraphQL. That was one of the reasons that I became interested as well. Right now the main data sources are blockchains like Ethereum, but in the future, we would like to make that available to other data sources as well.Jeremy: Right. You mentioned earlier too because there are apps obviously being built on this that you said are successful. And the problem though, I think right now, because I remember I speculated a little bit with Bitcoin and I bought a whole bunch of Ripple, so I'm still hanging on to it. Ripple XPR whatever, let's go. Anyways, but it was expensive to make a transaction. Right? Reading off of the blockchain itself, I think just connecting generally doesn't cost money, but if you're, and I know there's some costs with indexers and that's how that works. But in terms of the real cost, it's writing to the blockchain. I remember moving some Bitcoin at one point, I think cost me $30 to make one transaction, to move something like that.I can see if you're writing a $300,000 real estate transaction, or maybe some really large wire transfer or something that you want to record, something that makes sense where you could charge a fee of $30 or $40 in order to do that. I can't see you doing that for ... certainly not for web streaming or click tracking or something like that. That wouldn't make sense. But even for smaller things there might be writing more to it, $30 or whatever that would be ... seems quite expensive. What's the hope around that?Nader: That was one of the biggest challenges and that was one of the reasons that when I first, I would say maybe even considered this as a technology back in the day, that I would be considering as something that would possibly be usable for the types of applications I'm used to seeing. It just was like a no-brainer, like, no. I think right now, and that's one of the things that attracted me right now to some of the things that are happening, is a lot of those solutions are finally coming to fruition for fixing those sorts of things. There's two things that are happening right now that solve that problem. One of them is, they are merging in a couple of updates to the base layer, layer one, which would be considered something like Ethereum or Bitcoin. But Ethereum is the main one that a lot of the financial stuff that I see is happening.Basically, there are two different updates that are happening, I think the main one that will make this fee transactional price go down a little bit is sharding. Sharding is basically going to increase the number of, I believe nodes that are basically able to process the transactions by some number. Basically, that will reduce the cost somewhat, but I don't think it's ever going to get it down to a usable level. Instead what the solutions seem to be right now and one of the solutions that seems to actually be working, people are using it in production really recently, this really just started happening in the last couple of months, is these layer 2 solutions. There are a couple of different layer 2 solutions that are basically layers that run on top of the layer one, which would be something like Ethereum.And they treat Ethereum as the settlement layer. It's almost like when you interact with the bank and you're running your debit card. You're probably not talking to the bank directly and they are doing that. Instead, you have something like Visa who has this layer 2 on top of the banks that are managing thousands of transactions per second. And then they take all of those transactions and they settle those in an underlying layer. There's a couple different layer 2s that seem to be really working well right now in the Ethereum ecosystem. One of those is Arbitrum and then the other is I think Matic, but I think they have a different name now. Both of those seem to be working and they bring the cost of a transaction down to a fraction of a penny.You have, instead of paying $20 or $30 for a transaction, you're now paying almost nothing. But now that's still not cheap enough to probably treat a blockchain as a traditional database, a high throughput database, but it does open the door for a lot of other types of applications. The applications that you see building on layer one where the transactions really are $5 to $20 or $30 or typically higher value transactions. Things like governance, things like financial transactions, you've heard of NFTs. And that might make sense because if someone's going to spend a thousand bucks or 500 bucks, whatever ...Jeremy: NFTs don't make sense to me.Nader: They're not my thing either, the way they're being, I would say, talked about today especially, but I think in the future, the idea behind NFTs is interesting, but yeah, I'm in the same boat as you. But still to those people, if you're paying a thousand dollars for something then that 5 or 10 or 20 bucks might make sense, but it's not going to make sense if I just want to go to an e-commerce store and pay $5 for something. Right? I think that these layer 2s are starting to unlock those potential opportunities where people can start building these true financial applications that allow these transactions to happen at the same cost or actually a lot cheaper maybe than what you're paying for a credit card transaction, or even what those vendors, right? If you're running a store, you're paying percentages to those companies.The idea around decentralization comes back to this discussion of getting rid of the middleman, and a lot of times that means getting rid of the inefficiencies. If you can offload this business logic to some type of computer, then you've basically abstracted away a lot of inefficiencies. How many billions of dollars are spent every year by banks flying their people around the world and private jets and these skyscrapers and stuff. Now, where does that money come from? It comes from the consumer and them basically taking fees. They're taking money here and there. Right? That's the idea behind technology in general. They're like whenever something new and groundbreaking comes in, it's often unforeseen, but then you look back five years later and you're like, this is a no-brainer. Right?For instance Blockbuster and Netflix, there's a million of them. I don't have to go into that. I feel this is what that is for maybe the financial institutions and how we think about finance, especially in a global world. I think this was maybe even accelerated by COVID and stuff. If you want to build an application today, imagine limiting yourself to developers in your city. Unless you're maybe in San Francisco or New York, where that might still work. If I'm here in Mississippi and I want to build an application, I'm not going to just look for developers in a 30-mile radius. That is just insane. And I don't use that word mildly, it's just wild to think about that. You wouldn't do that.Instead, you want to look in your nation, but really you might want to look around the world because you now have things like Slack and Discord and all these asynchronous ways of doing work. And you might be able to find the best developer in the world for 25% or 50% of what you would typically find locally and an easy way to pay them might just be to just send them some crypto. Right? You don't have to go find out all their banking information and do all the wiring and all this other stuff. You just open your wallet, you send them the money and that's it. It's a done deal. But that's just one thing to think about. To me when I think about building apps in Web2 versus Web3, I don't think you're going to see the Facebook or Instagram use case anytime in the next year or two. I think the killer app for right now, it's going to be financial and e-commerce stuff.But I do think in maybe five years you will see someone crack that application for, something like a social media app where we're basically building something that we use today, but maybe in a better way. And that will be done using some off-chain storage solution. You're not going to be writing all these transactions again to a blockchain. You're going to have maybe a protocol like Graph that allows you to have a distributed database that is managed by one of these networks that you can write to. I think the ideas that we're talking about now are the things that really excite me anyway.Jeremy: Let's go back to GraphQL for a second, though. If you were going to build an app on top of this, and again, that's super exciting getting those transaction fees down, because I do feel every time you try to move money between banks or it's the $3 fee, if you go to a foreign ATM and you take money out of an ATM, they charge you. Everybody wants to take a cut somewhere along, and there's probably reasons for it, but also corporate jets cost money. So that makes sense as well. But in terms of the GraphQL protocol here, so if I wanted to build an application on top of it, and maybe my application doesn't write to the blockchain, it just reads from it, with one of these indexers, because maybe I'm summing up some financial transactions or something, or I've got an app we can look things up or whatever, I'm building something.I'm querying using the GraphQL, this makes sense. I have to use one of these indexers that's aggregating that data for me. But what if I did want to write to the blockchain, can I use GraphQL to do a mutation and actually write something to the blockchain? Or do I have to write to it directly?Nader: Yeah, that's actually a really, really good question. And that's one of the things that we are currently working on with the Graph. Right now if you want to write a transaction, you typically are going to be using one of these JSON RPC wallets and using some type of client library that interacts with the wallet and signs the transaction with the private key. And then that sends the transaction to the blockchain directly. And you're talking to the blockchain and you're just using something like the Graph to query. But I think what would be ideal and what we think would be ideal, is if someone could use a single technology, a single language, and a single abstraction to do everything, not only with reading and writing but also with subscriptions for real-time updates.That's where we think the whole idea for this will ultimately be, and that's what we're working on now. Right now you can only query. And if you want to write a transaction, you basically are still going to be using something like ethers.js or Web3 or one of these other libraries that allows you to sign a transaction using your wallet. But in the future and in fact, we're already building this right now as having an end-to-end GraphQL library that allows you to write transactions as well as read. That way someone just learns a single API and it's a lot easier. It would also make it easier for developers that are coming from a traditional web background to come in because there's a little bit of learning curve for understanding how to create one of these signed providers and write the transaction. It's not that much code, but it is a new way of thinking about things.Jeremy: Well I think both of us coming from the serverless space, we know that new way of thinking about things certainly can throw a wrench in the system when a new developer is trying to pick that stuff up.Nader: Yeah.Jeremy: All right. So that's the blockchain side of things with the data piece of it. I think people could wrap their head around that. I think it makes a lot of sense. But I'm still, the decentralized, the other things that you talked about. You mentioned an S3, something that's sort of an S3 type protocol that you can use. And what are some of the other ones? I think I've written some of them down here. Acash was one, Filecoin, Livepeer. These are all different protocols or services that are hosted by the indexers, or is this a different thing than the indexers? How does that work? And then how would you use that to save data, maybe save some blob, a blob storage or something like that?Nader: Let's talk about the tokenomics idea around how crypto fits into this and how it actually powers a protocol like this. And then we'll talk about some of those other protocols. How do people actually build all this stuff and do it for, are they getting paid for it? Is it free? How does that work and how does this network actually stay up? Because everything costs money, developers' time costs money, and so on and so forth. For something like the Graph, basically during the building phase of this protocol, basically, there was white papers and there was blog posts, and there was people in Discords talking about the ideas that were here. They basically had this idea to build this protocol. And this is a very typical life cycle, I would say.You have someone that comes up with an idea, they document some of it, they start building it. And the people that start building it are going to be basically part of essentially the founding team you could think of, in the sense of they're going to be having equity. Because at the end of the day, to actually launch one of these decentralized protocols, the way that crypto comes into it, there's typically some type of a token offering. The tokens need to be for a network like this, some type of utility token to keep the network running in the future. You're not just going to create some crypto and that's it like, right? I think that's the whole idea that I thought was going on when in reality, these tokens are typically used for powering the protocol.But let's say early on you have let's say 20 developers and they all build 5% of the system, whatever percentage that you want to talk about, whatever. Let's say you have these people helping out and then you actually build the thing and you want to go ahead and launch it and you have something that's working. A lot of times what people will do is they'll basically have a token offering, where they'll basically say, okay, let's go ahead and we're going to mint X number of tokens, and we're going to put these on the market and we're going to also pay these people that helped build this system, X number of tokens, and that's going to be their payment. And then they can go and sell those or keep those or trade those or whatever they would like to do.And then you have the tokens that are then put on the public market essentially. Once you've launched the protocol, you have to have tokens to basically continue to power the protocol and fund it. There are different people that interact with the protocol in different ways. You have the indexers themselves, which are basically software engineers that are deploying whatever infrastructure to something like AWS or GCP. These people are still using these cloud providers or they're maybe doing it at their house, whatever. All you basically need is a server and you want to basically run this indexer node, which is software that is open source, and you run this node. Basically, you can go ahead and say, okay, I want to start being an indexer and I want to be one of the different nodes on the network.To do that you basically buy some GRT, Graph Token, and in our case you stake it, meaning you are putting this money up to basically affirm that you are an indexer on the protocol and you are going to be accepting subgraph developers to deploy their subgraphs to your indexer. You stake that money and then when people use the API, they're basically paying money just like they might pay money to somewhere like API gateway or AppSync. Instead, they're paying money for their subgraph and that money is paid in GRT and it's distributed to the people in the ecosystem. Like me as a developer, I'm deploying the subgraph, and then if I have a million people using it, then I make some money. That's one way to use tokens in the system.Another way is basically to, as an outside person looking in, I can say, this indexer is really, really good. They know what they're doing. They're a very strong engineer. I'm going to basically put some money into their indexer and I'm basically backing them as an indexer. And then I will also share the money that comes in from the query fees. And then there are also people that are subgraph developers, which is the stuff that I've been working with mainly, where I can basically come up with a new API. I can be like, it'd be cool if I took data from this blockchain and this file system and merged it together, and I made this really cool API that people can use to build their apps with. I can deploy that. And basically, people can signal to this subgraph using tokens. And when people do that, they can say that they believe that this is a good subgraph to use.And then when people use that, I can also make money in that way. Basically, people are using tokens to be part of the system itself, but also to use that. If I'm a front end application like Uniswap and I want to basically use the Graph, I can basically say, okay, I'm going to put a thousand dollars in GRT tokens and I'm going to be using this API endpoint, which is a subgraph. And then all of the money that I have put up as someone that's using this, is going to be taken as the people start using it. Let's say I have a million queries and each query is one, 1000th of a cent, then after those million queries are up, I've spent $100 or something like that. Kind of similar to how you might pay AWS, you're now paying, you know, subgraph developers and indexers.Jeremy: Right. Okay. That makes sense. So then that's the payment method of that. So then these other protocols that get built on top of it, the Acash and Filecoin and Livepeer. So those ...Nader: They're all operating in a very similar fashion.Jeremy: Okay. All right. And so it's ...Nader: They have some type of node software that's run and people can basically run this node on some server somewhere and make it available as part of the network. And then they can use the tokens to participate. There's Filecoin for file storage. There's also IPFS, which is actually more of, it's a completely free service, but it's also not something that's as reliable as something like S3 or Filecoin. And then you have, like you mentioned, I believe Acash, which is a way to execute arbitrary code, business logic, and stuff like that. You have Ceramic Network, which is something that you can use for authentication. You have Livepeer which is something you use for live streaming. So you have all these ideas, these decentralized services fitting in these different niches.Jeremy: Right, right. Okay. So then now you've got a bunch of people. Now you mentioned this idea of, you could say, this is a good indexer. What about bad indexers? Right?Nader: That's a really good question.Jeremy: Yeah. You're relying on people to take data off of a public blockchain, and then you're relying on them to process it correctly and give you back good data. I'm assuming they could manipulate that data if they wanted to. I don't know why, but let's say they did. Is there a way to guarantee that you're getting the correct data?Nader: Yeah. That's a whole part of how the system works. There's this whole idea and this whole, really, really deep rabbit hole of crypto-economics and how these protocols are structured to incentivize and also disincentivize. In our protocol, basically, you have this idea of slashing and this is also a fairly known and used thing in the ecosystem and in the space. It's this idea of slashing. Basically, you incentivize people to go out and find people that are serving incorrect data. And if that person finds someone that's serving incorrect data, then the person that's serving the incorrect data is, quote-unquote, slashed. And that basically means that they're not only not going to receive the money from the queries that they were serving, but they also might lose the money that they put up to be a part of the network.I mentioned you have to actually put up money to deploy an indexer to the network, that money could also be at risk. You're very, very, very much so financially disincentivized to do that. And there's actually, again, incentives in the network for people to go and find those people. It's all-around incentives, game theory, and things like that.Jeremy: Which makes a ton of sense. That's good to know. You mentioned, you threw out the number, five years from now, somebody might build the killer app or whatever, they'll figure out some of these things. Where are we with this though? Because this sounds really early, right? There's still things that need to be figured out. Again, it's public data on the blockchain. How do you see this evolving? When do you think Web3 will be more accessible to the masses?Nader: Today people are actually building really, really interesting applications that are fitting the current technology stack, what are the things that you can build? People are already building those. But when you think about the current state of the web, where you have something like Twitter, or Facebook or Instagram, where I would say, especially maybe something like Facebook, that's extremely, extremely complex with a lot of UI interaction, a lot of private data, messages and stuff. I think to build something like that, yeah, it's going to be a couple of years. And then you might not even see certain types of applications being built. I don't think there is going to be this thing where there is no longer these types of applications. There are only these new types. I think it's more of a new type of application that people are going to be building, and it's not going to be a winner takes all just like in all tech in my opinion.I wouldn't say all but in many areas of tech where you're thinking of something as a zero-sum game where I don't think this is. But I do think that the most interesting stuff is around how Web3 essentially enables native payments and how people are going to use these native payments in interesting ways that maybe we haven't thought of yet. One of the ways that you're starting to see people doing, and a lot of venture capitalists are now investing in a lot of these companies, if you look at a lot of the companies coming out of YC and a lot of the new companies that these traditional venture capitalists are investing in, are a lot of TOMS crypto companies.When you think about the financial incentives, the things that we talked about early on, let's say you want to have the next version of YouTube and you don't want to have ads. How would that even work? Right? You still need to enable payments. But there's a couple of things that could happen there. Well, first of all, if you're building an application in the way that I've talked about, where you basically have these native payments or these native tokens that can be part of the whole process now, instead of waiting 10 years to do an IPO for an application that has been around for those 10 years and then paying back all his investors and all of those people that had been basically pulling money out their pockets to take part in.What if someone that has a really interesting idea and maybe they have a really good track record, they come out with a new application and they're basically saying, okay, if you want to own a piece of this, we're going to basically create a token and you can have ownership in it. You might see people doing these ICO's, initial coin offerings, or whatever, where basically they're offering portions of the company to anyone that wants to own it and then incentivizing people to basically use those, to govern how the application is built in the future. Let's say I own 1% of this company and a proposal is put up to do something new. I can basically say, I can use that portion of my ownership to vote on things. And then people that are speculating can say, this company is doing interesting things. I'm going to buy into it, therefore driving the price up or down.Kind of like the same way that you see the traditional stock market there, but without all of the regulation and friction that comes with that. I think that's interesting and you're already seeing companies doing that. You're not seeing the majority of companies doing that or anything like that, but you are starting to see those types of things happening. And that brings around the discussion of regulations. Is ... can you even do something like that in the United States? Well, maybe, maybe not. Does that mean people are going to start building these companies elsewhere? That's an interesting discussion as well. Right now if you want to build an application this way, you need to have some type of utility that these tokens are there for. You can't just do them purely on speculation, at least right now. But I think it's going to be interesting for sure, to watch.Jeremy: Right. And I think too that, I'm just thinking if you're a bank, right? And you maybe have a bunch of private transactions that you want to keep private. Because again, I don't even know how, I don't know how we get to private transactions on the blockchain. I could see you wanting to have some transactions that were public blockchain and some that were private and maybe a hybrid approach would make sense for some companies.Nader: I think the idea that we haven't really talked about at all is identity and how identity works compared to how we're used to identity. The way that we're used to identity working is, we basically go to a new website and we're like, this looks awesome. Let me try it out. And they're like, oh wait, we need your name, your email address, your phone number, and possibly your credit card and all this other stuff. We do that over and over and over, and over time we've now given our personal information to 500 people. And then you start getting these emails, your data has been breached, every week you get one of these emails, if you're someone like me, I don't know. Maybe I'm just signing up for too much stuff. Maybe not every week, but maybe every month or two. But you're giving out your personal data.But we're used to identity as being tied to our own physical name and address and things like that. But what if identity was something that was more abstract? And I think that that's the way that you typically see identity managed in Web3. When you're dealing with authentication mechanisms, one of the most interesting things that I think that is part of this whole discussion is this idea of a single sign-on mechanism, that you own your identity and you can transfer it across all the applications and no one else is in control of it. When you use something like an Ethereum wallet, like MetaMask, for example, it's an extension you can just download and put crypto in and basically make payments on the web with. When you create a wallet, you're given a wallet address. And the wallet address is basically created using public key cryptography, where basically you start with this private key, your public key is derived from the private key, and then your address is dropped from the public key.And when you send a transaction, you basically sign the transaction with your private key and you send your public key along with the transaction, and the person that receives that can decode the transaction with the public key to verify that that's who signed the transaction. Using this public key cryptography that only you can basically sign with your own address and your own password, it's all stored on the blockchain or in some decentralized manner. Actually in this case stored on the blockchain or it depends on how you use it really, I guess. But anyway, the whole idea here is that you completely own your identity. If you never decide to associate that identity with your name and your phone number, then who knows who's sending these transactions and who knows what's going on, because why would you need to associate your own name and phone number with all of these types of things, in these situations where you're making payments and stuff like that. Right?What is the idea of a user profile anyway, and why do you actually need it? Well, you might need it on certain applications. You might need it or want it on social network, or maybe not, or you might come up with a pseudonym, because maybe you don't want to associate yourself with whatever. You might want to in other cases, but that's completely up to you and you can have multiple wallet addresses. You might have a public wallet address that you associate your name with that you are using on social media. You might have a private wallet address that you're never associating with your name, that you're using for financial transactions. It's completely up to you, but no one can change that information. One of the applications that I recently built was called Decentralized Identity. I built it and release it a few days ago.And it's an implementation of this and it's using some of these Web3 technologies. One of them is IDX. One of them is Ceramic, which is a decentralized protocol similar to the Graph but for identity. And then it's using something called DIDs, which are decentralized identifiers, which are a way to have a completely unique ID based off of your address. And then you own the control over that. You can basically go in and make updates to that profile. And then any application across the web that you choose to use can then access that information. You're only dealing with it stored in one place. You have full control over it, at any time you can go in and delete that. You can go in and change it. No one has control over it except for you.The idea of identity is a mind-bending thing in this space because I think we're so used to just handing everybody our real names and our real phone numbers and all of our personal information and just having our fingers crossed, that we're just not used to anything else.Jeremy: It's all super interesting. You mentioned earlier about, would it be legal in the United States? I'm thinking of all these recent ransomware attacks and I think they were able to trace back some Bitcoin transaction, they were actually able to trace it back to the individual group that accepted the payment. It opens up a whole can of worms. I love this idea of being anonymous and not being tracked, but then it's also like, what could bad actors do with anonymous financial transactions and things like that? So ...Nader: There kind of has been anonymous transactional layer for a long time. Cash brought in, you can't really do a lot of illegal stuff these days without cash. So should we get rid of cash? I think with any technology ...Jeremy: No, but I mean, there's a limit though, right? You can't withdraw more than $10,000 worth of cash without the FBI being flagged and you can't deposit more, you know what I mean?Nader: You can't take a million dollars worth of Bitcoin that you've gotten from ransomware and turn it into cash either.Jeremy: That's also true. Right.Nader: Because it's all tracked on the blockchain, that's probably how they caught those people. Right? They somehow had their personal information tied to a transaction, because if you follow these transactions long enough, you're going to find some origination point. I agree though. There's definitely trade-offs with everything. I don't think I'm ever the type to argue that. There's good things and there's bad things. I think you have to look at the whole picture and decide for yourself, what you think. I'm the type that's like, let's lay out all of the ideas and let the market decide.Jeremy: Right. Yeah. I totally agree with that. All this stuff is fascinating, there is way too much more for me to learn at this point. I think my brain is filled at this point. Anything else about Edge & Node? Any cool things you're working on there or anything you want people to know?Nader: We're working on a couple of different projects. I can't really talk about some of them because they're not released yet, but we are working on a new version of something called Everest, and Everest is already out. If you want to check it out, it's at everest.link. It's basically a repository of a bunch of different applications that have already been built in the Web3 ecosystem. It also ties in a lot of the stuff that we talked about, like identity and stuff like that. You can basically sign in with your Ethereum wallet. You can basically interact with different applications and stuff, but you can also just see the types of stuff people are building. It's categorized into games, financial apps. If you've listened to this and you're like, this sounds cool, but are people actually building stuff? This is a place to see hundreds of apps that people have are already built and that are out there and successful.Jeremy: Awesome. All right. Well, listen, Nader, this was awesome. Thank you so much for sharing this with me. I know I learned a ton. I hope the listeners learned a ton. If people want to learn more about this or just follow you and keep up with what you're doing, what's the best way to do that?Nader: I would say check out Twitter, we're on Twitter @dabit3 for me, @edgeandnode for Edge & Node, and of course @graphprotocol for Graph protocol.Jeremy: Okay. And then edgeandnode.com. Your YouTube channel is just youtube.com/naderdabit, N-A-D-E-R D-A-B-I-T. And then you had an article on Web3 and I'll put it in the show notes.Nader: Yeah. Put it in the show notes. For freeCodeCamp, it's called what is Web3. And it's really a condensed version of a lot of the stuff we talked about. Maybe go into a little bit more depth around native payments and how people might build companies in the way that we've talked about here.Jeremy: Awesome. All right. Well, I will get all that stuff into the show notes. Thanks again, Nader.Nader: Thanks for having me. It was good to talk.

covid-19 united states america love new york amazon netflix action san francisco building reading hbo bank fbi bitcoin nfts silicon valley discord cloud crypto mississippi id south america blockchain mount everest explain vc similar visa slack ipo blockbuster api ethereum web3 ui atm aws amplify apis faa ripple jp morgan chase decentralized ico graphs fauna s3 sql node nader web2 lambda yc matic baas serverless ceramic dapps graphql metamask uniswap gcp developer advocate react native ipfs filecoin ec2 arbitrum yaml sharding dynamodb cognito idx discords building apps grt decentralized internet dids nader dabit decentralized web aws amplify livepeer developer relations engineer serverless framework aws appsync appsync jeremy it jeremy you cbt nuggets aws mobile

412 Serverless at Via

Reversim Podcast

Play Episode Listen Later Jun 21, 2021

שלום וברוכים הבאים לפודקאסט מספר 412 של רברס עם פלטפורמה. התאריך היום הוא ה-13 ביוני 2021, - הייתי אומר שזה תאריך קצת היסטורי: ככל הנראה היום הוקמה איזושהי ממשלה, אנחנו לא יודעים האם היא תחזיק מעמד, אבל ההצבעה הייתה ממש היום ואנחנו במתח לקראת מה שהולך להיות [מחשש שמספר הממשלות יעקוף את מספר הפרקים של רברסים?].היום אנחנו מקליטים פודקאסט עם ינון מחברת Via - תיכף תציג את עצמך - ויש לנו גם אורח מיוחד היום: כמחליף לאורי יש לנו היום את יונתן מחברת Outbrain - היי יונתן!שלום וברוך הבא - יונתן עובד ב-Outbrain כבר כך-וכך שנים וגם התארח בעבר בפרקים שלנו, אבל זה כבר ממש ממש היסטוריה עתיקה [נגיד 328 The tension between Agility and Ownership או Final Class 23: IDEs או 131 uijet או 088 Final Class 2 ... יש עוד].אז קודם כל - כיוון ש-412 זה Precondition Failed - כולם יודעים, נכון? לא הייתי צריך לבדוק את זה לפני השידור, ממש לא, זכרתי בע”פ . . . - אז יונתן, בוא וספר לנו על ה-Precondition שלך. או מי אתה, במילים אחרות . . . (יונתן) אז קודם כל - אני מאזין ותיק של רברסים, אני חושב שכשהגעתי ל-Outbrain לפני 10 שנים, אחת הסיבות הייתה הפודקאסט.הגעתי כמהנדס Backend, ובחמש השנים האחרונות אני מוביל ב-Outbrain את הפיתוח.(רן) “מוביל ב-Outbrain את הפיתוח” זו הדרך שלך להצטנע ולהגיד שאתה מנהל הפיתוח?(יונתן) מנהל הפיתוח . . .(רן) יפה, טוב - פיתוח ב-Outbrain זו קבוצה גדולה, יש לך הרבה עבודה, ותודה שבאת(יונתן) בשמחה.(רן) אז ינון מחברת Via, והיום אנחנו הולכים לדבר על הנושא של Serverless - אבל מזויות אחרות, זויות שעדיין לא כיסינו.לפני שנכנס ככה לנושא - ספר לנו קצת על עצמך.(ינון) אז אהלן, אני ינון, נעים מאוד.אני נמצא ב-Via משהו כמו שלוש שנים וקצת.סתם כרקע - דיברנו על Precondition - הגעתי ל-Via מחברת Ravello לפני כן - למי שלא מכיר, Ravello תומכת די גדולה ברברסים וככה גם התוודעתי גם לפודקאסט, וככה הגעתי אליך, רן . . .אספר קצת על Via, אני חושב שהם התארחו כבר בפודקסט [אכן - 360 Via] אבל עוד פעם, אולי מזוית אישית שלי, אני תמיד אוהב לספר לכולם צ'יזבט על Via . . .(רן) אז, דרך אגב, אירחנו איש מוצר, אז זה היה פחות טכנולוגי - והיום אנחנו הולכים להיות הרבה יותר טכנולוגיים.(ינון) מעולה - זו ההקדמה שאני עושה.למעשה, Via ככה התחילה . . . אני מספר תמיד צ'יזבט כזה, שאף אחד לא אישר או הכחיש במסדרונות של Via.לפי הסיפור, ה-CTO וה-Founder שלנו, אורן, הסתובב יום אחד ברחוב אלנבי וניסה לתפוס מונית שירות לכיוון תל אביב, לכיוון האוניברסיטה - ולמעשה גילה שזה די מסובך, בתקופה ההיא.לפני 7-8 שנים, משהו כזה, לתפוס מונית שירות זה לא כזה קל - צריך למצוא, ולאן להגיע, ואיך לעלות עליה ומה עושים איתה, ואיך שעולים צריך לשלם את הכסף הזה וזה נורא מסובך . . .ומה שעלה לו בראש זה ש”וואלה, זה רעיון מגניב הדבר הזה, זה סוג של בלגן מזרח-תיכוני כזה מגניב”, של מי שהיה כבר בתוך התחבורה הציבורית - ומצד שני, איזה כיף היה אם הייתה איזו אפליקציה או משהו שהיה מאפשר לפחות להזמין מונית, להגיע איתה מאיפה שאתה רוצה לאן שאתה רוצה, שאומרת לך איך להגיע, מה לעשות, זה היה משלם בשבילך . . . מרגיש כמו חלום.אז הוא הלך והגשים אותו, וככה Via התחילה את חייה - בניו-יורק . . .משם התגלגלנו להרבה מאוד מקומות אחרים - התחלנו כשירות סוג-של-Consumer ומשם הלאה זה התגלגל לשירותים שונים של Pre-booking ו-Paratransit.היום אנחנו מתעסקים גם ברכבים כמו School Bus, כלומר - כל מערך שירות האוטובוסים של עיריית ניו-יורק, כך שבעצם אנחנו יכולים לעשות המון המון דברים.ואת כל זה אנחנו בונים בעצם מעל Stack טכנולוגי אחד ויחיד, שכמו שרן קודם רמז - רובו ככולו נעשה מעל Serverless.(רן) כן, אז מיד ניכנס לסיפור הזה . . . דרך אגב, אני מניח שהרבה מהמאזינים מכירים את Via, וגם היו הרצאות של עובדים שלכם בעבר בכנסים - יש שם ערבוב מעניין של טכנולוגיה, אלגוריתמיקה, Data Science ודברים אחרים - וכמובן גם סיפור מוצרי.יכול להיות שמי ששומע את ה-Pitch שלך אומר “אה! זה Uber!” או אחרים - אבל זה לא . . . הפעם לא ניכנס לזה, כי אנחנו לא עושים פרק על מוצר. יש הבדלים, אבל לא ניכנס אליהם [וכמו כן - 360 Via].ועכשיו - בוא ניכנס לטכנולוגיה: אז אחד הדברים המעניינים ב-Via זה שה-Stack הטכנולוגי כולו, או רובו, רץ מעל פלטפורמת Serverless.אז בוא ספר לנו - איך בנוי ה-Stack שלכם?(ינון) אז בוא נתחיל אולי גם כן היסטורית -אז היסטורית, Via, כמובן, כמו כל חברת היי-טק ישראלית טובה, סטארטאפ ישראלי, התחילה עם Monolith, שלשמחתינו או לצערינו קיים עד היום, אותו Monolith מפורסם שנקרא “Via Server”, שם מאוד מקורי . . . ואותו Via Server רץ במקור על הרבה מכונות EC2 באמאזון - די סטנדרטי, כמו שאתה מצפה, הרבה לפני Kubernetes .מה שגילינו זה שככל שמתפתחים, והזכרתי קודם מקומות ש-Via התפתחה - בעצם ה-Stack עצמו התחיל להיות נורא-נורא יקר . . .מצד אחד, היינו נתקלים בהרבה מאוד בעיות של Scale-up - זאת אומרת שהיה צריך לעשות הרבה Scale-up ולהגדיל את ה-Monolith הזה כדי לתמוך ב-Traffic שכל הזמן גדל, ומצד שני, אם היינו משאירים אותו ב-Scale מאוד גבוה, אז AWS היו מאוד נהנים ו-Via פחות . . . מה גם שב-Via מסתכלים גם על צורת השימוש, בעיקר במקור אבל גם היום - יש תקופות שיש בהן Peak, אפשר לחשוב על זה גם בתל אביב, אנחנו מפעילים גם את שירות Bubble, אז אפשר לחשוב שבשעות הבוקר, בערך מ-07:00 עד 09:30 - מי שמנסה לתפוס Bubble יודע שזה לא פשוט, הרכבים מאוד מאוד עמוסים, ואתם בעצם מפציצים את השרתים שלנו . . . ואותו הדבר קורה בשעות הערב.מצד שני - מי שמנסה לתפוס Bubble בסביבות השעה 12:00, אז החיים שלו מאוד קלים, הוא מוצא אותו מאוד מאוד מהר - וזה פשוט כי יש פחות ביקוש, יש פחות אנשים שרוצים להגיד אל ומ-העבודה.(רן) אבל נשמע שאתם יודעים מה הולכות להיות שעות העומס . . . למה לא פשוט לעשות Scale-up ו-Scale-Down, כשאפילו יש לך חמש דקות לעשות Warm-up לשרתים, אם אתם יודעים מראש . . . ?(ינון) מעולה - אז ככה אמרנו גם אנחנו . . . “מעולה! החיים קלים - אנחנו יודעים לעשות warm-up ו-Scale-Down”הבעיה היא שבשביל זה צריך קצת לחזות ולהבין מה באמת יהיה ה-Traffic - ובכל פעם צריך כמו בפיתוח - לשים באפרים (Buffers) . . .“כמה יהיה מחר בדיוק? אז יהיו מחר 1,000 נוסעים? אז בוא נשים 10 שרתים, או 15 שרתים . . . “ואז למחרת קמים בבוקר - ויש גשם, וגשם זה מכה . . . אז פתאום ה-1,000 נוסעים הופכים ל-2,000 . . . ואם מסתכלים על עיר ב-Scale של ניו-יורק, שהיא עיר מן הסתם הרבה יותר גדולה, אז שם זה הופך מ-10,000 נוסעים ל-100,000 פתאום . . . במכה אחת פתאום כולם מפציצים.אז אם אין גשם - נהדר, זה אומר שמישהו צריך ב-06:00 בבוקר לשים לב ולהגיד “וואו, כדאי שנגדיל עוד יותר את השרתים” - ואתם יכולים לדמיין כמה פעמים מישהו פספס, או פספס לכמה שעות, פספס את ההערכה שלו, ובמקום 100 שרתים הוסיף רק 50 - והפסדנו.(רן) אני מבין את הנקודה הכואבת - זה לקום ב-06:00 בבוקר . . . זאת אומרת, אם זו הייתה חברה שפעילה בצהריים, אז לאף אחד אולי לא היה אכפת, ולא הייתם מגיעים ל-Serverless, אבל לקום ב-06:00 בבוקר זה כבר סיפור . . . (ינון) זו סיבה ממש מעולה לעבור למשהו אחר . . . הבעיה השנייה היא שגם אחרי כשהיינו מעלים את ה-50 שרתים - מישהו גם היה צריך לזכור להוריד את זה אחר כך . . . זה לא תמיד כזה קליל של “נרים עכשיו ואחר כך נזכור”, כי שוכחים, ולמחרת לא . . . והחשבון AWS נראה פתאום לא להיט.אז חפשנו פתרון שיאפשר לנו לעשות גם את ה-Scaling האוטומטי.מצד אחד אתה אומר “אחלה, אז יש פתרונות יותר מודרניים” . . . Kubernetes הגיע בשלב יותר מאוחר, אבל גם לפני כן היו פתרונות של Auto-Scaling Groups ב-AWS, אפשר היה להרים גם איתם.הבעיה היא שכשמסתכלים על זה רגע - אז Monolith שכזה, אמנם כתוב ב-Python, שזה עולה יחסית מהר - ועדיין עד שהוא עולה ועושה את ה-Init שלו, ומוריד . . . אפשר לחשוב קצת על מה ש-Via עושה, אז צריך להוריד את את המפות, צריך להוריד קונפיגורציות, צריך להכין כל מיני דברים . . . וזמן ה-Warm-up והבנייה של ה-Container הוא לא קצר - זה יכול לקחת גם דקות, תלוי כמובן בגודל ה-Traffic ובגודל הדברים שצריך להעלות - וזה כמובן די כואב.לעשות Scale-Up שמסתמך רק על ה-Auto-Scaling הזה מראש זה לא מספיק מהר, ויש תקופה לא מספיק קצרה שמפסידים.מפסידים כסף, מפסידים תנועה - וגם יש שירות ממש לא מוצלח למשתמש שמנסה לנסוע.וזו הסיבה שהתחלנו לחפש דברים אחרים.(רן) אבל זמן טעינה כזה - בטח יבוא יונתן תיכף ויטען - “רגע! אבל אתם לא Monolith! יש Microservices!” . . . אז למה ה-Monolith צרך לטעון את כל המפות של כל העולם ואת כל שאר הדברים? נכון, זה כבד - אבל יש לזה גם פתרונות אחרים, לא רק Serverless . . .(ינון) מעולה - זה השלב שבאמת הסתכלנו - ותודה יונתן על השאלה . . . - הסתכלנו ואמרנו “אוקיי, פתרון אחד זה באמת להגיד יופי, בוא נבנה את זה עם כל מיני Microservices”למעשה, אפשר להסתכל על זה ולהגיד שזה הפתרון שבחרנו - השאלה רק עכשיו היא רק מה ה-Transport שלו, מה ה-Pipeline שבאמצעותו אנחנו בעצם מרימים את אותו הדבר.אופציה אחת הייתה להגיד “אוקיי, נכתוב את הקוד באוסף של שרתים קטנים, ב-Python . . .”, ואגב - בחלק מהדברים זה מה שעשינויש מקומות שבהם . . . Via לא דוגמאטית ואומרת “Serverless is the only way”, זה לא הדרך הנכונה שלנו להסתכל על זה.אנחנו אומרים שבמקומות שבהם אפשר לעשות את זה בצורה קלה דווקא שלא על ידי להרים Service שלם וכבד מעל Kubernetes אלא להסתכל על יתרונות של דברים אחרים, זה היה המקום שבו הסתכלנו על Serverless.אם מסתכלים רק על למה בחרנו ללכת עם Serverless בחלקים ספציפיים, אז בעצם שמנו לעצמנו כמה נקודות מעניינות - אמרנו שאנחנו רוצים כמה שפחות התערבות של DevOps, כי DevOps זה דבר יקר וזה דבר מסובך - לא רק מצד האנשים אלא גם עצם הזמן שמושקע ב-DevOps, בלהרים סביבות ולסדר אותן - מאוד יקר.גם בסביבה מאוד מוצלחת כמו Kubernetes, שבאמת יש לה הרבה יתרונות - עדיין יש הרבה מאוד קונפיגורציה שצריך לעשותצריך להבין מהם הפרמטרים שבעזרתם אנחנו קובעים Scale-up ו-Scale-Down ו-Scale-In - ובעצם לקנפג (Configure) את השרת, לעשות Fine-tuning כל הזמן, כדי להגיע בעצם לתוצאות שהיינו רוצים להגיע אליהן.אז גם בסביבה של Microservices קלאסית כזאת, שבה יש Containers ו-Pods, רוב עבודת הקונפיגורציה הזאת היא עלינו, אחריות שלנו . . . (רן) אני חושב שיש חוק, כלל שימור האנרגיה בטכנולוגיה: עבודה לא נעלמת - היא משנה צורה . . . אם לפני כן היית צריך לחווט כבלים, אז היום אתה צריך לקנפג VPCs, ואם לפני כן היית צריך לקנפג איזושהי מכונה, אז היום אתה צריך לייצר איזשהו Script או לעשות איזושהי קונפיגורציה ב-Terraform או כל כלי אחר . . .ההתמחות משתנה, אבל העבודה לא נעלמת(יונתן) אמרתם שאתם לא דוגמאטיים, זאת אומרת - אתם לא אומרים שזו הדרך היחידה. יש דברים שבהם אתם כן משתמשים עדיין ב . . . ה-Monolith עדיין משחק תפקיד? ה-Microservices עדיין באיזור? או שזה . . .(ינון) קודם כל, ה-Monolith עדיין קיים - לא בכל Deployment ולא בכל מקום, אבל עדיין קיים בלב של חלק מה-Deployment שלנו.ויש לנו עדיין כמה מה-Services האחרים, שהם Microservices סטנדרטיים, עם Containers, חלק כתובים ב-Java, חלקם ב-Python, ועדיין קיימים כ-microServices קלאסיים, Docker Containers בתוך Kubernetes.בעיקר במקומות שבהם יש צריכת זכרון מאוד גבוה - אנחנו צריכים להוריד . . . לצורך העניין להחזיק את המפה - מפה, מן הסתם, זה אובייקט שלוקח הרבה מאוד זכרון, ושם דווקא יוצא לנו יותר נוח להחזיק אותה למשל בתוך Container.כך שיש לנו גם Kubernetes stacks שלמים.עם זאת, במקומות שבהם זו לוגיקה או שהוא “Container Glue” - וזה, אגב, הרבה ממה ש-Via עושה . . .תחשבו, לצורך העניין, על נהגים שמסתובבים בעיר ומדווחים לנו מיקום - הם צריכים כל הזמן לדווח איפה הם נמצאים ולקבל הוראות - זה משהו שלא מצריך הרבה זכרוןמה שהוא באמת מצריך זה את היכולת לעשות Scale-up ו-Scale-In, לפי כמות הנהגים שמסתובבים כרגע בכביש.אז במקום, בעצם, להרים Containers שיודעים לטפל בדבר הזה, גילינו שהרבה יותר קל לנו להרים “micro-micro-micro-Containers”, או “Nano-Containers” כאלה - שזה, בתכל'ס, Lambda . . . אז זה בדיוק מה שזה עושה.(יונתן) אז זה Use-case של, נגיד, לקבל את המיקומים של הנהגים ולכתוב אותם איפשהו, אני מניח? יש Use-cases אחרים, נניח אם אני רוצה להזמין מונית, זה גם . . .(ינון) גם זה על Serverless, לגמרי. גם זה ירוץ Serverless.בעצם תגיע בקשה, לאיזושהי Lambda, שיושבת, לצורך העניין, מאחורי או ALB או איזשהו API Gateway.היא מחוברת ישירות לתוך ה-Lambda - ומשם, למעשה, יכולה לרוץ “שרשרת של Lambda-ות” . . .הגיעה בקשה - מזהה מי הנוסע - משם זה רץ למוקד ההזמנות שלנו, שזה בעצם המוקד “שמדבר” מול הרכבים - תבוצע הזמנה - זה יעבור לאיזושהי Lambda שיודעת לנהל תשלומים, מן הסתם צריך לבדוק שאתה גם רשאי לעלות על הנסיעה . . . זה גם יעבור משם לאיזשהו מקום שהנהג מקבל בו את ההוראותמשם זה יפנה לשירות המיפוי - שזה, להזכיר לכם, Container - נגיד לו “נא לייצר לנהג מסלול חדש”, שיוביל אתכם לאיסוף של אותו נוסע.ובסופו של דבר זה יתורגם כהוראות בחזרה לנהג - והנהג מקבל הוראה וימשיך לשדר לנו את אותם דיווחים, שאנחנו קוראים להם Heartbeat, בשם המקורי . . .וזה יגיע חזרה, בעצם, למערכות שלנו, וימשיך את אותו Flow שהזכרתי קודם.(רן) בוא נדבר רגע על כסף, כלכלה . . . קודם הזכרת שהיו Instances של EC2, והייתם צריך לעשות Scale-up ואז אולי שכחתם לעשות להם Scale-Down וזה עולה כסף וכו' . . . לי יצא לעבוד Serverless, בסטארטאפ הרבה יותר קטן מ-Via, וזה גם היה לפני כמה שנים, לפני חמש שנים או משהו כזה, ואז היה ברור ש-Serverless זה יקר . . . זאת אומרת - יש יתרונות בצד של האופרציה, יש מודל תפעולי, יש מודל תכנותי שהוא בריא, את כל הדברים האלה מאוד מאוד אהבתי - אבל דבר אחד היה ברור: שזה הולך להיות מאוד יקר כשנעשה Scale-up.אם אין כלום באוויר, אז נכון, זה לא עולה - אם לא קוראים לפונקציה שלך, אז עץ שנופל ביער ואף לא שומע אותו אז הוא לא באמת נופל . . . אז זה ברור שיותר קל מאשר לתחזק Container של EC2.אבל ברגע שיש Traffic משמעותי, והפונקציה כל הזמן נקראית, אז היה ברור, לפחות אז, שזה גם הולך להיות הרבה-הרבה-יותר יקר מאשר לתחזק Microservice משלך.איך נראית הכלכלה של זה היום?(ינון) אז בוא נספר לכם את זה ככה . . . נתחיל דווקא מסיפור ואז ניכנס לאט לאט לכיוון הזה.בעצם, בתחילת משבר הקורונה [הסיבוב הראשון . . .], אתם יכולים לדמיין מה הייתה ההשפעה של זה על שירותי ההסעה . . . במכה אחת, בוקר אחד, בתוך שבוע פחות או יותר, עברנו ממאות אלפי נוסעים בניו-יורק לבערך עשרה.אף אחד לא נסע, אף אחד לא זז - וזה היה בכל העולם, לא רק בניו-יורק.מצד שני, אם מסתכלים רגע עכשיו על הכלכלה, או על מה שעלה כסף - במכה אחת כל ה-Lambda-ות שלנו ירדו לאפס, הפסקנו לשלם עליהן לחלוטין,ולעומת זאת כל אותם Containers - שנשארו ב-Monolith וכל מיני כאלה - השאירו שם לא מעט דולרים, שהמשיכו לזרום ישירות לכיסים של מיסטר בזוס . . . [קצת אמפטיה, לאיש יש חללית לבנות].כך שלפחות ברמת ה-Scale-Up / Scale-In, יש לזה כלכלה שהיא סופר-מוצלחת - אנחנו לא צריכים להשאיר בשום שלב “ספיירים” כדי להתמודד עם עומס “למקרה ש…"אפשר לדבר על Warn-ups, אבל לא משאיריםומצד שני, גם בזמן שהיא באוויר והיא כן עושה את הפעילות, אנחנו רואים שזמן העיבוד בפועל, שבו ה-Lambda בעצם עובדת, הוא מאוד נמוך - בעיקר כי כי משתמשים בקריאות א-סינכרוניות.אם עובדים בצורה שהיא יותר א-סינכרונית - כלומר, קריאות שמגיעות עוברות . . . משקיעות את רוב הזמן שלהן במעבר בין Lambda-ות בתוך תורים, ואין בו קריאות סינכרוניות החוצה - פתאום הזמן שבעצם ה-Lambda רצה הוא מאוד מאוד קטן [קצר].ספציפית, אגב - לפני כמה חודשים AWS החליפו את צורת ה-Billing שלהם ממינימום של 100 מילי-שניות למינימום של 1 מילי-שנייה ל-Lambda, וזה שיפר משמעותית את העלות שלהן, בעיקר של Lambda-ות קצרות, שזה הרבה ממה שאנחנו עושים.(רן) הבנתי - זאת אומרת שאם, לדוגמא, ה-Lambda-ות . . . נגיד, אני אתאר איזשהו Flow של -Lambda שקוראת ל--Lambda וכו' - אם כל אחת מהן מחכה לשניה בצורה סינכרונית, אז אתה משלם את החשבון של כולן, אם היא בדרך; אבל אם זה קורה בצורה א-סינכרונית, במעבר דרך SQS או כל מכניזם אחר - אז אתה משלם רק על זמן העיבוד המינימלי. בסדר, אני מבין . . .(יונתן) גם מה שמעניין פה, רן, זה שיש קשר ישיר בין ה-Business - שזה ה-Traffic שאתם מקבלים - לבין העלות, מה שעם Services יותר קשה לעשות את הקשר הזה.הוא חי כל הזמן, גם אם הוא לא יקבל Traffic, גם אם אתה לא “מקבל כסף”.(ינון) בדיוק - בערים של Via, יש ערים שלמות שבהן אין לך שירות בשעות מסויימות של היוםגם השירות ב-Bubble, לצורך העניין, הוא רק בשעות היום, הוא נגמר סביב 22:00-23:00 בלילה [אפילו קצת יותר].תחשוב שכל הלילה יש איזשהו שרת פעיל, וכן צריך לענות תשובות לשאלות: אם איזשהו נוסע פותח אפליקציה של Bubble ב-02:00 לפנות בוקר, אנחנו ניתן לו תשובה שאין כרגע שירות - אבל בשביל זה צריך שיהיה איזשהו שרת באוויר . . .אז ככל שנעביר יותר מהדברים האלה ל-Serverless, אם מישהו יבקש בקשה אז הוא יקבל, אבל אם לא - אז אין צורך בכלל להרים את ה-Service.(רן) אז אתה אומר שבגלל שאתם מאופיינים באלסטיות מאוד גדולה - אולי קורונה זו דוגמא קיצונית, אבל עדיין ביום-יום יש אלסטיות - יש סופי-שבוע, יש שעות שונות במהלך היום, יש חגים . . . בכל אופן, יש אלסטיות בצורה יחסית משמעותית - זה עושה את המודל של Serverless ליותר משתלם אצלכם.אני תוהה - אני לא יודע אם יש בכלל את התשובה, אבל אני תוהה - האם למישהו עם Workload יחסית מאוזן לאורך היממה, האם גם לו זה הולך להשתלם?(ינון) זו תמיד שאלה של מה באמת ה-Workload שלך - וכמה באמת ממה שאתה עושה הוא באמת Broken-down ל-Microservices עד הסוף.למה אני מתכוון? לצורך העניין, אם מסתכלים לרגע על אותו שירות של Via, אז מצד אחד מה שבאמת לוקח המון מהקריאות ומה-Traffic אלו אותם Hearbeat-ים של הנהגים - זה משהו שאנחנו יודעים עליו שהוא מאוד מאוד כבד מבחינת כמות הקריאות שנעשות ומבחינת זמן העיבוד שרץ שם בפנים - גם אם העיבוד עצמו הוא מאוד קצר.אז אם יש לך איזשהו שירות שבו את מחזיק את ה-Heartbeats האלה יחד עם עוד שירות ביחד, בעצם אתה עושה פה Coupling מאוד חזק של של שרת אחד יחד לשתי שכבות יחד.בעולם של Serverless, נורא קל לעשות את ה-Breakdown הזה ממש ל-”Nano-Services” - זה Service שאולי אין לו בכלל זכות חיים משל עצמו, אבל לעשות Scale-up של חתיכה קטנה מתוך ה-Service זה נורא קל.(רן) אוקיי, כן, בסדר - זה היה השיקול הכלכלי. עכשיו, בוא נסתכל על השיקול המתודולוגי.אני אשאל את זה ככה - האם המפתחים שלכם ניהיו יותר טובים, כי הם עובדים Stateless? הם נהיו יותר טובים כי הם נאלצים לרוץ תחת Constraints כאלה של פונקציות Lambda? או במילים אחרות - איך אתה רואה שמתודולוגיה כזאת משפיעה על צורת הפיתוח, איכות הפיתוח, איכות הקוד וכו'?(ינון) אז יש לזה כמה תשובות . . .מצד אחד, כן - המפתחים, בלית ברירה, צריכים לחשוב על עולם שבוא אין זכרון מרכזי, אין שיתוף בין . . . השיתוף היחידי בין Containers הוא בעצם משהו חיצוני, כך שזה גורם לאנשים לחשוב כמה שיותר על איך מחזיקים State ומה עושים איתו.באמת עלינו עם הרבה כיוונים ופתרונות לזה, שגם חלקם, אגב, זה שיקולים כלכליים גם כן . . . לצורך העניין, גילינו שעבודה עם Databases שהם יותר Serverless באופי שלהם, כמו DynamoDB, יוצא לנו הרבה יותר זול - וגם נוח מבחינת Burst-ים של Traffic - מאשר להשתמש ב-Database שהוא “MySQL-כזה”, ושיותר קשה לו לעשות Scale-up.ו-Dynamo, לצורך העניין, הוא גם “אם לא השתמשת - לא שילמת”, אז אם לא קראת אז לא קרה כלום - ולעומת זאת ב-MySQL, גם בגרסאות מוצלחות כמו Aurora, אתה משלם כל עוד ה-Instance למעלה, לא יעזור כלום.בנוסף לכך, גילינו שיש הרבה דרכים גם לשפר את הקריאה מה-Database - אנחנו עובדים כמובן גם עם איזשהו ElastiCache או עם איזשהו S3 כ-Cache מקומי, שעוזר לנו להתמודד בעצם עם Burst-ים של “פתאום אלפי Lambda-ות מנסים לתקוף את ה-Database” [הסרט הבא של Netflix?].ו-Dynamo, לצורך העניין, מתמודד עם זה די יפה, בשביל זה הוא בנוי.כש-Aurora, שהוא Database די מוצלח, קצת פחות נהנה מכזה Burst של Traffic.יש כמה פתרונות ל-AWS, שעובדים ויודעים לפתור את הבעיה הזו - חלק מהם זה אנחנו בנינו בעצמנו לצורך העניין - הרמנו Cache ב-Redis מעל הדבר הזה, בעזרת ElastiCache.אופציה אחרת זה שיש לשים Proxy לפני ה-Database - ובעצם לעשות Connection Pooling לפני ה-Database עצמו.זה באמת מאפשר לנו להריץ, שוב, הרבה Load עם הרבה מאוד Spikiness . . .(רן) בוא רגע נתעכב על הסיטואציה הזאת של Connection Pooling - אני חייב להגיב שגם אני נכוותי מזה . . . ממש אותה סיטואציה שאתה מתאר: פונקציות Lambda, עם Aurora ו-MySQL מאחור - ואלפי פונקציות Lambda שמנסות להתחבר אל ה-Database . . . עכשיו - אם כל האלפים הללו היו בסך הכל Thread-ים בתוך אותו Process, אז יש Connection Pooling ולפי . . . נניח שאתה מחליט שה-Database מרשה שיהיו 300 Connections, אז 300 Thread-ים ידברו עם ה-Database, והאחרים יחכו בתור.אבל פה - Lambda לא יודעת “לחכות בתור” . . . . אז הן מתחילות להיכשל . . .(ינון) ה-Lambda-ות הן באמת יצור קצת אנוכי בקטע הזה - הן לא כל כך “מסתכלות מסביב”ובאמת יש שני פתרונות שאנחנו גילינו והשתמשנו בהם - אחד מהם זה פתרון Built-in של AWS, יש להם Proxy שנועד לפתור בדיוק את הבעיות האלה.בעצם שמים . . . תחשוב על זה כעל סוג של מכונה ששמים לפני ה-EC2, בעצם EC2 לפני ה-Database.ה-Lambda מתחברת למכונה הזאת - והמכונה עצמה מחזיקה Connection Pool - והיא אומרת ל-Lambda “אוקיי, חכי שנייה, אני אתפוס אותך על ה-Connection Pool הבא”.וזה מתנהג, בעצם, מבחינת התפיסה, מאוד דומה למה שהזכרת קודם - בתוך Monolith שכזה . . . זה מאפשר להשתמש בעצם ב-Aurora [מחייב רפרנס ל-The Robots of Dawn . . . .](יונתן) דרך אגב, אתה יודע, רק כדי להשלים את התמונה ואת המוטיבציה - זה לא רק שהן מפגיזות את ה-Database וחלקן נכשלות, אלה למה בכלל מייצרים Connection Pool? כדי לחסוך את זמן יצירת ה-Connection, שב-DCP זה זמן יקר - אבל Lambda לא יכולה לעשות את זה . . . Lambda חייבת בכל פעם לייצר את ה-Connection מחדשואז אתה משלם שוב על Latency - וגם דולרים בסופו של דבר . . .וזו רק דוגמא אחת של Connection Pool - אני חושב שכל Local Cache . . . .כל מה שב-Microservices אתה יכול להשתמש ב-Local Cache, פה אתה בבעיה, אתה צריך פתרונות אחרים.(ינון) נכון . . . אז יש כמה דרכים . . . שוב, כשנתקלנו בבעיה דומה, אגב במקום שבו ה-Database היה Read-Mostly, הפתרון היה בעצם להשתמש ב-ElastiCache כסוג-של-Cache מעל ה-Database.אפשרנו להכריז את ה-Database עצמו כהרבה יותר קטן - ומה שצריך זה Lambda ש”פעם ב” . . . פעם בזמן ה-Refresh-הרלוונטי הייתה פשוט מרפרשת (Refresh) את ה-Cache.די פשוט - לקרוא מה-Database, לדחוף ל-Cache . . . בעצם לעבוד ישירות מול ה-ElastiCache.עלו על כמה פתרונות בדרך, אגב - יש פתרון שנקרא EFS, שבעצם מאפשר להחזיק File System, כשה-Lambda-ות בעצם חולקות, ויש גישה שהיא הרבה יותר קלה, לא צריך להחזיק Connection אלה פשוט זה ניגש ישירות ל-Data.וגם כשהוא באותו Proxy, באמת זה שימושי כדי להחזיק DCP Connection פתוח מול ה-Database ואז רק צריך ליצור Connection קטן מול “הדברצ'יק” הזה.(רן) לפעמים קורה שאתה כן רוצה לשלוט על Server . . . זאת אומרת: אנחנו מדברים על Serverless, ואתה רץ בתוך איזשהו Container. אבל וואלה - לפעמים אתה רוצה לקבוע את כמות הזכרון, לשחק ב-TCP Stack, לעשות כל מיני אופטימיזציות על File Descriptors וכו' . . . מה אתה עושה כשאתה מגיע למצב כזה? מה אתה עושה כשאתה מרגיש שאתה כבר “מגרד את תקרת הזכוכית” בתוך ה-Lambda שלכם?(ינון) קודם כל, אני אשאל אותך - למה? מה המניע?כי בדרך אצלנו, At the end of the day, the business is not that . . . אנחנו לא מתעסקים בלהתעסק עם הקרביים של איזשהו קובץ . . . אם אין ברירה אז אין ברירה, אבל לא מצאנו, עד עכשיו, שום מקום שבו היה צורך בזה.כלומר, העדפנו את הקלות של ה-No-Ops, כשכל מה שצריך לקבוע ב-Lambda זה את כמות הזיכרון שלה - והיא רצה.אני כמובן מגזים, ואפשר לקבוע עוד כמה דברים - האם היא רצה בתוך vPC או מחוץ ל-vPC, יש Security Groups וכו' - אבל בגדול, ברגע שקבעת אותם Once אז גמרנו, ואין מה להתעסק עם זה כמעט.לפי כמות הזכרון בעצם אתה קובע את ה . . . לא את הזכרון אלא את ה-Performance הכללי של ה-Lambda.בגדול, כשאני חושב על זה - כשאתה קובע את הזכרון אתה קובע כמה Lambda-ות רצות על Container אחד של AWSוככל שרצות פחות Lambda-ות, כלומר תופסות יותר זיכרון ורצות פחות Lambda-ות על ה-Container - אז יש לך יותר משאבים בתוך ה-Container: גם CPU, גם Network card - וזו בעצם השליטה שיש לך.בסך הכל - גילינו שכשקצת משחקים עם הזכרון אז זה ממש מעל ומעבר למה שאנחנו צריכים מבחינת השליטה שיש שלנו בתוך ה-Server.לדברים שהם ממש Fine grained - אני מסכים, Lambda לא מתאימה.בשביל זה בדיוק אנחנו הולכים למקומות אחרים כמו Containers ו-Pods.(יונתן) אם מסתכלים קצת אחורה, אז פעם היו “מפלצות כאלה” - היה WebSphere ו-JBOSS ואפילו Tomcat . . . אתה היית כותב את הקוד שלך, עושה לו איזשהו . . . היו קוראים לזה WAR ו-EAR וכל מיני קללות . . . ועושה לזה Deploy בתוך איזשהו Container.וכשהגיעו ה-Microservices זה די הלך לכיוון אחר . . . במקום להיות “אורח” בתוך איזשהו Run-time שמישהו אחר מתחזק, והוא מאוד גדול ומורכב ומוטת השליטה שלך היא קטנה, אתה ניהיה בעל בית של ה . . . אם אתה רץ ב-Python אז אתה ניהיה הבעל-בית של ה-Process של ה-Python של ה-JVM - ובאיזשהו אופן ה-Serverless קצת מחזיר אותך אחורה, לפחות “אחורה” מבחינת האופנה . . . אתה עדיין ניהיה אורח בתוך איזשהו Run-time שמישהו אחר מחזיק ומקנפג (Configure) - איך אתה פה עם “הרטרו” הזה? . . .(ינון) אני מת על הרטרו הזה, כי מי שמחזיק ומקפנג את ה-Server הענק הזה זה לא אני . . . זה התותחים ב-AWS, שיודעים בדיוק מה רוצים - והם די טובים בעולם הזה . . . בגלל זה אני חי עם זה בשלום.אם אני הייתי צריך לתחזק את ה-JBOSS הזה או את ה-WebSphere הזה, אז כנראה שלא היינו מדברים היום . . .מכיוון ש-AWS עושים את זה ואנחנו . . . בסופו של דבר הם באמת יודעים בדיוק מה הם עושים והם טובים בזה, אז אני חי עם זה די בשלום..אני אמנם נתון לחסדיהם, וזה נשמע קצת פטאליסטי, אבל at the end of the day, אם יש מישהו שטוב להיות בידיים שלו זה כנראה החבר'ה ב-AWS שעושים עבודה די טובה.ומה שאני מרוויח מזה באמת זה שאני לא צריך להתעסק יותר עם קונפיגורציות מסובכות, אני בסך הכל I Deploy my code, it works - וזה די הסיפור.בטח כאשר אנחנו מרגישים . . . זאת אומרת, AWS הם מאוד פתוחים מבחינת האימפלמנטציה (Implementation) שלהם ומה שהם מוכנים לספר, ברמה כזאת שהם מאפשרים לך להריץ Any Run-time you want, bring your own Run-time . . . אם אתה רוצה ממש להריץ קוד Fortran מעל Lambda אז No problem, you can do it [ברצינות…] כך שזה אמנם סגור מצד אחד - אבל יש לזה הרבה פתיחות מהצד השני.(יונתן) ואני מניח שאם אתה באמת רוצה להיות בעל הבית של ה-Process, אתה תעשה Microservice שיפתור את הבעיה, אם אתה צריך לקנפג (Configure) את הלא-יודע-כמה Descriptors שאתה צריך . . .(ינון) בדיוק, ואגב - גם שם, זה קצת שונה, אבל במובן מסויים אתה Hosted בתוך Kubernetes . . כלומר - יש לך יותר שליטה על ה-Process, אתה שולט באמת על ה-Run-time, יש לך את ה-Pod . . .מצד שני, יש לך עדיין איזשהו “בעל-בית” שאומר לך “שמע, אתה לא בדיוק עושה את מה שאתה . . . אני עדיין בעל הבית פה”.(יונתן) נכון.(רן) אתם עדיין “Python-Shop”? או ש . . .(ינון) עדיין Python-Shop . . .(רן) זאת אומרת - ברמת העיקרון, Lambda מאפשר לך, אפילו יותר בקלות, לגוון בשפות היעד - אבל זו הזדמנות שעוד לא לקחתם.(ינון) נכון - חוץ מהעובדה שבעצם מה שבאמת משפיע, ואפשר לדבר גם על זה קצת, זה Cold-Start . . . מסתכלים רגע על מתי Lambda עולה, אז כש-Lambda מרימה את עצמה, היא צריכה לעשות הרבה קונפיגורציות ו-Setup.ובעצם העלאת Run-time של Python זה ה-Run-time, אולי חוץ מ-Node, הכי מהיר שיש.משמעותית, לצורך העניין, יותר מהיר מאשר לעלות Lambda של Javaוגם כאלה, אגב, יש לנו כמה, מסיבות הסטוריות - ובאמת רואים שה-Run-time של Java, עד שהוא עולה . . . הוא כבד.מצד שני, אם משתמשים ב-Lambda, אחד מהחסרונות - שהוא גם יתרון, במובן מסויים - הוא שה-Lambda Run-time הוא Single-threaded, או לפחות Single-Core - אין שם באמת תמיכה מלאה ב-Multi-threading, שרצים במקביל.אפשר להסתכל על כמה Processes בתוך Lambda, אבל לא Multi-thread - מה שמייתר, לצורך העניין, את הצורך להשתמש ב-asyncio או בכל מני Thread-ים מסובכים ב-Javaוגם הסתכלנו קצת בעבר על Go-lang - שפה כזו מודרנית ומגניבה [חכה לבאמפרס הבא . . . ]אז לכתוב בה Containers זה די מגניב, אבל לכתוב אותה בתוך Lambda זה די מיותר . . . זאת אומרת - אי אפשר להרוויח שם בכלל מכל ה-asyncio שיושב בתוך Go, כל ה-Async Functions(רן) כן . . . טכנית זה אפשרי, רק שאתה לא מרוויח(ינון) בדיוק - זה עדיין Single-threaded אז זה סינכרוני לחלוטין.(רן) איך נראית חווית המפתח? זאת אומרת - מה קורה אם פתאום ה-Production איטי, או פתאום דברים אובדים, או פתאום . . . לא יודע, בקשה מקבלת Time-out או דברים כאלה? איך מדבגים (Debug) תהליך? איך עושים Tracing? . . איך מדבגים פונקציות Lambda שמפוזרות, אני לא יודע כמה . . . כמה יש לכם?(ינון) יש לנו, בפעם האחרונה שספרתי - כמה אלפים טובים של פונקציות.(יונתן) . . . בטח אתה מתחיל להתגעגע ל-Monolith, שיכולת לשים Break-point ולראות בדיוק מה קורה . . .(ינון) . . . בדיוק, זו אכן שחוויה שהיא . . . At first daunting . . . כשמסתכלים על זה בפעם הראשונה, אני זוכר שאני הסתכלתי על זה ואמרתי “אוקיי, מה אני עושה?” . . . אני פותח את CloudWatch ומנסה לחפור בלוגים . . .לא חווייה מאוד מעניינת, לא כיפית כל כך.ובאמת, אחד הדברים ש-Lambda מחייב זה Clear observability - אז יש כלים פנימיים של AWS - לצורך העניין X-Ray, ש-AWS מאוד דוחפיםכלי חביב כזה, שעוזר בעצם לעשות Distributed Tracing.הבעיה העיקרית עם X-Ray זה שצריך לעבוד בשביל לגרום לזה לעבוד . . . כלומר, חלק מהעבודה היא גם להכניס בעצם יכולת של Tracing בפנים . . .(רן) . . . אינסטרומנטציה (Instrumentation)(ינון) . . אינסטרומנטציה שכזאת, בדיוק . . .ואנחנו העדפנו כלי שעושה את אינסטרומנטציה בשבילנוחפרנו קצת מסביב והתלבשנו בסוף על Epsagonלמי שלא מכיר את Epsagon - כלי מעניין מאוד, שבעצם, עם מעט מאוד עבודה, מאפשר להיכנס ולעשות Tracing של כל ה-Lambda-ות שלנו יחד ולחבר אותן ביחד.הוא משתמש בספרייה שנקראת Jaeger כדי לעשות בעצם Distributed Tracing, זו ספריית open-source די מוכרת, פשוט המימוש שלהם די מוצלח.בעצם, זה מאפשר לנו לראות קריאות שמתחילות ב-Lambda אחת ונגמרות ב-Lambda אחרת, בקצה ה-stack, דרך כל ה-Lambda-ות האחרות.גם מבחינת Tracing שלהן, גם מבחינת ה-Payloads שעברו בתוך ה-Lambda-ות - מבחוץ פנימה, דרך ה-SQS-ים השונים, קריאות ל-Database וכן הלאה.בעצם, זה מאפשר גם לראות את הלוגים - וגם לראות Performance: כמה כל קריאה לקחה, בפנים.(רן) אבל מההיכרות שלי עם Jaeger - הוא מצויין כשמדובר על gRPC או HTTP - כל הדברים הסינכרוניים, אבל דברים א-סינכרוניים, למשל המעבר ב-SQS או מעבר ב-Kafka - שם אתה צריך כבר להמציא בעצמך פתרון . . . אז הם עטפו לכם את זה?(ינון) הם עטפו את כל העסק, הם טיפלו בזה מאוד יפה - אפשר לראות ממש את הקריאות ל-SQS ואת המעבר החוצה, את הקריאה החוצה מתוכו.בעצם הכניסו לא מעט מה-Tracing . . . הרחיבו Jaeger לתוך ה-Tracing שלהם - על זה אולי יהיה מעניין לעשות פרק אחר . . .אבל אנחנו כן משתמשים ב-Epsagon ורואים Observability מלא, End-to-End.המקומות היחידים שבהם זה נשבר הם מקומות שבהם לא הוספנו איזה ארבע שורות לתוך ה-Serverless Framework, שבאמצעותו אנחנו עושים Deploy ל-Lambda-ות, ששם בעצם אנחנו לא עושים את ה-Automatic wrapping שלהם - ושם באמת רואים מתי זה נשבר וכמה זה קשה.ובמקומות כאלה שאנחנו מזהים, זה מאוד פשוט להוסיף Tracing אוטומטי שכזה - זה ממש כמה שורות, להוסיף מודול קטן ב-Node וזה הופ! עושה Tracing אוטומטי ובעצם מאפשר לנו Observability מלא ממש של הכל.(רן) אוקיי, אז זה Tracing ב-Production - אבל איך נראית חוויית הפיתוח? אני עכשיו צריך לכתוב איזשהו Service חדש, או פונקציה חדשה - מה, אני פשוט פורש את זה לענן ורואה מה קורה, או שיש איזשהו משהו מקומי?(ינון) . . . That's pretty much itיש כמובן דברים בסיסיים - אם כותבים ב-Python אז Unit Testing זה דבר די סטנדרטי, שאנחנו מן הסתם חייבים לכתוב, זה אפילו סוג-של-תחליף-Complier, בלאית ברירה.יש קצת Linting וכאלה - אבל למי שלא מכיר את Python, ואני מניח שיש מעט מאוד כאלה, יודע שבלי איזה Unit Test אחד או שניים כדי לראות שהקוד באוויר אתה מאוד בקלות פורש איזו שטות . . . אפשר להריץ Unit Testing בשביל לעשות Local Debugging פשוטבדרך כלל, מה שאנחנו עושים זה פשוט פורשים את זה ישירות לענן מהסביבת Dev, מריצים אוסף של קריאות HTTP ל-Lambda-ות כדי לראות שזה עובד, PostPlan עובד שעות נוספות . . .כן יש פה כמה אופציות להריץ לוקאלית, זאת אומרת - גם ל-AWS יש אופציה להריץ סוג-של Local Lambda Server, “להרים את Lambda מקומית”להגיד שזה מאוד נוח? זה לא . . . זה לא להיט, וגילינו שהרבה יותר קל ונוח לנו לפרוש ישירות ל-Dev Environment ולהריץ הכל משם.(רן) זאת אומרת שיש לכם איזשהו עותק של סביבת ה-Production . . . זאת אומרת, להריץ את הפונקציה שלך זה . . השאלה היא האם היא תלויה בפונקציות אחרות? בתורים אחרים? ב-Databases אחרים? שם הדברים יותר מתחילים להסתבך.אז בעצם, את כל זה אתם עושים ישר בענן? לא על תחנה מקומית?(ינון) נכון.אפשר להסתכל על זה בעצם כעל סוג של Sandbox, שמכיל את ה-Lambda-ות שיש לנו בעולם.בעצם, אנחנו פורשים את הקוד ישירות לשם ובודקים אחד מול השני.מן הסתם, כל Lambda היא באחריות של איזשהו צוות, כל אוסף Lambda-ות או כל Service, בעצם - זה לא רק Lambda, אנחנו מסתכלים על אוסף של X [כמה] Lambda-ות כעל Service מסויים, שיש לו איזושהי מטרה.אנחנו בעצם פורשים את השירותים השונים אל תוך הענן ועושים . . . משתמשים ב-Convention כדי להגיע משירות לשירות ולעשות את כל ה-Wiring בין ה-Lambda-ות השונות.(יונתן) במובן מסוים, גם ב-microServices, החל מ-Scale מסויים, אתה בבעיה די דומה . . .זאת אומרת - כשיש לך כבר כמה מאות אתה כבר לא מרים את כולם על ה-Laptop, וגם פה תלוי ב-Cloud שלך, בעצם.(רן) מסכים ב-100% . . .אני חושב שהבעיה, או האתגר, של שירותים מבוססי-דאטה זה לשחזר איזושהי סביבת Production, כלומר - אם אתה רוצה איזשהו Copy של סביבת ה-Production, עם הדאטה של Production, אבל בלי להזיק ל-Production, וגם לא לשלם את העלות של Production.לפעמים, ה-Databases הם ענקיים, ואתה לא באמת רוצה עותק מלא - אז תיקח את ה-Sub-set של הדאטה, שהוא בדיוק מה שאתה צריך אבל לא יותר מזה - וגם לא תזיק ל-Production - זה אתגר לא פשוט לכל מי שמתעסק עם כמויות גדולות של דאטה, בלי קשר ל-Lambda או לא Lambda.כמה זמן אתה ב-Via?(ינון) שלוש שנים . . .(רן) אוקיי . . . כשהגעת, כבר הייתם בעולם ה-Serverless?(ינון) זו בדיוק הייתה ההתחלה, בשלב שבו הסתכלנו על זה בפעם הראשונה.(רן) אני אגיד לך למה אני שואל - אני מנסה לדמיין מפתח ותיק, מפתח מנוסה אחר, שעכשיו נכנס ל-Via. האם אתה מוצא, נגיד כשאתה מסתכל על מפתחים שגוייסו בזמן האחרון, ואני לא מדבר על צעירים שבחיים לא כתבו קוד אלא על כאלה שכבר . . . אתה יודע, “שועלי קרבות” . . .(יונתן) הפילו את ה-Production כבר כמה פעמים . . .(רן) כן . . . האם אתה רואה שהם, אתה יודע - הם מסתכלים על כל עולם ה-Serverless הזה, ועכשיו צריכים לכתוב איזושהי פונקציה חדשה - האם אתה רואה שהם נלחמים ביצר הטבעי שלהם, או שזה פשוט בא להם בטבעיות, והם “משילים מעליהם” איזשהו משקל כבד שהם נשאו עד עכשיו על הכתפיים ופורחים סביבת ה-Serverless?(ינון) אז הייתי אומר שהם די פורחים . . . זאת אומרת, יש תמיד את המעבר המסוחרר הראשון הזה שאומר, כמו שהזכרת קודם: “רגע, אין לי Connection Pool”, “אין לי פה . . . אני צריך להבין רגע איך זה מגיע, אני פורש את זה כבר לענן? מה קרה לי? זה קצת מוקדם?”.אז באמת יש את הכמה ימים האלה של “רגע-רגע, איך אני עושה פה דברים?”אבל באמת זה לוקח ממש מעט זמן.ב-Via, אתה בדרך כלל מתחיל לכתוב קוד תוך פחות משבועייםכלומר - צריך להרים איזושהי סביבה מקומית, צריך לראות שהכל עובד, שהכל מותקן והכל בסדר - ואז טיפה ללמוד את העולם, גם של Via וגם את עולם של Serverless.אבל תוך באמת פחות משבועיים הוא מקבל משימה ואוקיי - פורש בפעם הראשונה ובפעם השנייה ומשם בעצם זה מגיע ל-Production די מהר.(יונתן) יש גם, אני מניח, יתרון שאולי ה-Scope של הקוד שאתה צריך להכיר כדי לעשות שינוי הוא, כנראה, יותר קטן - זאת אומר, הוא כנראה תלוי בעוד הרבה דברים אחרים, אבל כבר מראש צמצמו לך אותו לסט מסויים של פונקציות או של Services . . . (ינון) כן, אז יותר קל, כנראה, לפרק ל-microServices קטנים, כי העלות של להרים microService היא כמעט כלום.לא צריך פה לפרוש איזה Pod חדש או לייצר משהו חדש - זה “אוקיי, מעתיקים את ה-Serverless.yml”, שזה yml פשוט שרק מגדיר את ה-Service עצמו, יש בו ממש-ממש כלום הגדרות.ומשם פורשים Service חדש מאוד-מאוד בקלות, מה שמאפשר לנו בעצם להריץ הרבה מאוד microServicesרק אצלי בקבוצה יש בין 80 ל-100 microServices ו… And Growing . . . (יונתן) יש איזו אופטימיזציה ש-AWS נותנים, נניח שהם מזהים Lambda-ות שקוראות אחת לשנייה בצורה . . . באופן תדיר, ובעצם להוריד את ה-Network ביניהן ושהן תרוצנה In-process?(ינון) רעיון מדליק . . . אבל לא שאני מכיר . . . מה שאנחנו עושים הרבה באמת זה שאם יש לנו הרבה מקומות שבהם אנחנו קוראים לאותו קוד שוב ושוב ושוב, אנחנו פשוט אורזים אותו כ-Packages.כלומר, במקום לארוז אותו כ-Lambda נפרדת, אנחנו אורזים את זה ב-Package כזה, ואז משתמשים בו, ב-Re-use, במקומות שונים.(רן) שזה, ”בשפת Lambda”, זה ספרייה, נכון? זאת אומרת, יש מגבלה על גודל הפונקציה, אז בשביל זה AWS מציעים Packages, שזה כמו Library . . . (ינון) לא . . . ל-Lambda עצמו, Built-in, יש את מה שנקרא Layers . . .עם Layers, ב-AWS בעצם מאפשרים לך להרים, בהגדרות של AWS, ממש “שכבות” כאלה של Lambda, שמאפשרות לפרוש כחלק מה-Lambda, כאשר ה-Lambda עצמה נפרשת לתוך ה-Container.רעיון די מדליק - אנחנו לא משתמשים בו הרבה . . . אנחנו משתמשים ממש ב-Node Packages בשביל לארוז מחדש את ה-Packages אצלנו, מכמה סיבות.ל-Layers היו כל מיני מגבלות טכניות - היה אפשר רק 5 Layers, ואם אתה צריך את השישית אז אתה כבר נתקע.יש עניין ש-Cold start לא מתחיל מחדש את ה-Layer תמיד . . . כך שהשליטה שלך היא לא מספיק חזקה שם.הרגשנו יותר בנוח לעבוד עם Node Packages, עם גרסאות מסודרות, כשכל Lambda תדע מתי היא מתקדמת לגירסא הבאה . . .(רן) רגע, אמרת Node Packages? אנחנו לא ב-Python? . . . (ינון) סליחה . . . Python . . . אתה צודק, 100% . . .(רן) כמעט תפסנו אותך . . . (ינון) כמעט תפסתם אותי . . . אגב - יש לנו Lambda-ות גם ב-Node, כתבנו כמה Lambda-ות ב-JavaScriptיש צוותים שהעדיפו לעבוד ב-JavaScriptלא הרבה . . . זה עובד, אגב, טוב ממש כמו Python, אם כי אני חובב Python יותר מאשר Node, ולכן אצלי בצוות עובדים בעיקר ב-Python.(רן) כן . . . דרך אגב, יונתן אולי נתייחס לשאלה שלך - שאלת האם כשיש פונקציות שקוראות אחת לשנייה באופן תכוף, האם אפשר לעשות כזאת אופטימיזציה, אבל אז, זאת אומרת . . (א) זה רעיון טובאבל כנראה שבמקרה של ינון זה לא יעזור, כי הם עושים את הכל א-סינכרוני ושמים את הכל ב-Queue, אז בכל מקרה צריך לשלוח ל-Queue . . . (יונתן) Wix בדיוק נתנו הרצאה לא מזמן, על ה-Vision שלהם בעולם ה-Serverless, והם נתנו את הדוגמא הזאת.[לפני חודש - Beyond Serverless and DevOps, Aviran Mordo]זאת אומרת - את ההזדמנות לאופטימיזציה הזאתאז שווה ל . . .(רן) אז מה - גם ה-Queue נמצא בתוך ה-Host?(יונתן) אני לא יודע, אני חושב שזה היה . . .צריך לשאול את אבירן, זה היה יותר “תכנונים עתידיים”, כמו שאני הבנתי . . .(רן) הבנתיטוב, ינון, שמע - מרתק . . . אז אנחנו ממש, ככה, לקראת סיום - תן כמה “מילים סוגרות”, אני בטוח שאתם מגייסים . . .(ינון) כמובן . . . אנחנו בהחלט - כמו, כנראה, כל חברה אחרת בארץ - אבל בטח, אצלנו מגייסים.אנחנו מגייסים, אגב, בכל מיני מקומות בארץ - גם בתל אביב, גם בירושליםואנחנו גם די עובדים, כזה, From anywhere - אז אנחנו מאוד נשמח, אם מעניין אתכם.וכן - העולם של Serverless הוא מרתק בעיני, הוא מאוד שינה לי את החשיבה, מרגע שהגעתי ל-Via, ומאפשר לי באמת לעשות כמה דברים מאוד מאוד מהר ובקלות.ושוב - אנחנו באמת, אם נסכם את זה - אנחנו לא דוגמאטיים בעניין.אנחנו מאוד מאוד מאמינים ב-Serverless כאחת מהטכנולוגיות שעוזרות לנו לקדם את המוצראבל, אתה יודע: מה שעובד - עובד . . . If it works, don't break it . . . (רן) טוב, אחלה - אז תודה רבה, היה מעניין, ונתראה.תודה רן, תודה יונתן.כנס רברסים 2021: נפתחה הקריאה להגשות!הקובץ נמצא כאן, האזנה נעימה ותודה רבה לעופר פורר על התמלול

time founders service state performance data single network robots uber cold built run services software production bubbles cloud scale connections library setup pitch traffic scaling consumer peak warm script pipeline package transport laptops implementation stack layers refresh python data science scope aws java server databases heartbeat automatic dev devops tracing layer container agility packages deployment constraints warn sandbox proxy queue containers x ray ides workload cache monolith node dynamo wix kubernetes backend school bus lambda heartbeats serverless microservices coupling mysql terraform observability stateless instrumentation debug configure ec2 efs filesystem outbrain dynamodb instance unit tests ravello sqs cloudwatch paratransit scale down serverless framework vpcs elasticache

Episode #105: Building a Serverless Banking Platform with Patrick Strzelec

Play Episode Listen Later Jun 14, 2021 66:01

About Patrick StrzelecPatrick Strzelec is a fullstack developer with a focus on building GraphQL gateways and serverless microservices. He is currently working as a technical lead at NorthOne making banking effortless for small businesses.LinkedIn: Patrick StrzelecNorthOne Careers: www.northone.com/about/careersWatch this episode on YouTube: https://youtu.be/8W6lRc03QNU This episode sponsored by CBT Nuggets and Lumigo. TranscriptJeremy: Hi everyone. I'm Jeremy Daly, and this is Serverless Chats. Today, I'm joined by Patrick Strzelec. Hey, Patrick, thanks for joining me.Patrick: Hey, thanks for having me.Jeremy: You are a lead developer at NorthOne. I'd love it if you could tell the listeners a little bit about yourself, your background, and what NorthOne does.Patrick: Yeah, totally. I'm a lead developer here at NorthOne, I've been focusing on building out our GraphQL gateway here, as well as some of our serverless microservices. What NorthOne does, we are a banking experience for small businesses. Effectively, we are a deposit account, with many integrations that act almost like an operating system for small businesses. Basically, we choose the best partners we can to do things like check deposits, just your regular transactions you would do, as well as any insights, and the use cases will grow. I'd like to call us a very tailored banking experience for small businesses.Jeremy: Very nice. The thing that is fascinating, I think about this, is that you have just completely embraced serverless, right?Patrick: Yeah, totally. We started off early on with this vision of being fully event driven, and we started off with a monolith, like a Python Django big monolith, and we've been experimenting with serverless all the way through, and somewhere along the journey, we decided this is the tool for us, and it just totally made sense on the business side, on the tech side. It's been absolutely great.Jeremy: Let's talk about that because this is one of those things where I think you get a business and a business that's a banking platform. You're handling some serious transactions here. You've got a lot of transactions that are going through, and you've totally embraced this. I'd love to have you take the listeners through why you thought it was a good idea, what were the business cases for it? Then we can talk a little bit about the adoption process, and then I know there's a whole bunch of stuff that you did with event driven stuff, which is absolutely fascinating.Then we could probably follow up with maybe a couple of challenges, and some of the issues you face. Why don't we start there. Let's start, like who in your organization, because I am always fascinated to know if somebody in your organization says, “Hey we absolutely need to do serverless," and just starts beating that drum. What was that business and technical case that made your organization swallow that pill?Patrick: Yeah, totally. I think just at a high level we're a user experience company, we want to make sure we offer small businesses the best banking experience possible. We don't want to spend a lot of time on operations, and trying to, and also reliability is incredibly important. If we can offload that burden and move faster, that's what we need to do. When we're talking about who's beating that drum, I would say our VP, Blake, really early on, seemed to see serverless as this amazing fit. I joined about three years ago today, so I guess this is my anniversary at the company. We were just deciding what to build. At the time there was a lot of architecture diagrams, and Blake hypothesized that serverless was a great fit.We had a lot of versions of the world, some with Apache Kafka, and a bunch of microservices going through there. There's other versions with serverless in the mix, and some of the tooling around that, and this other hypothesis that maybe we want GraphQL gateway in the middle of there. It was one of those things that we wanted to test our hypothesis as we go. That ties into this innovation velocity that serverless allows for. It's very cheap to put a new piece of infrastructure up in serverless. Just the other day we wanted to test Kinesis for an event streaming use case, and that was just a half an hour to set up that config, and you could put it live in production and test it out, which is completely awesome.I think that innovation velocity was the hypothesis. We could just try things out really quickly. They don't cost much at all. You only pay for what you use for the most part. We were able to try that out, and as well as reliability. AWS really does a good job of making sure everything's available all the time. Something that maybe a young startup isn't ready to take on. When I joined the company, Blake proposed, “Okay, let's try out GraphQL as a gateway, as a concept. Build me a prototype." In that prototype, there was a really good opportunity to try serverless. They just ... Apollo server launched the serverless package, that was just super easy to deploy.It was a complete no-brainer. We tried it out, we built the case. We just started with this GraphQL gateway running on serverless. AWS Lambda. It's funny because at first, it's like, we're just trying to sell them development. Nobody's going to be hitting our services. It was still a year out from when we were going into production. Once we went into prod, this Lambda's hot all the time, which is interesting. I think the cost case breaks down there because if you're running this thing, think forever, but it was this GraphQL server in front of our Python Django monolift, with this vision of event driven microservices, which has fit well for banking. If you just think about the banking world, everything is pretty much eventually consistent.Just, that's the way the systems are designed. You send out a transaction, it doesn't settle for a while. We were always going to do event driven, but when you're starting out with a team of three developers, you're not going to build this whole microservices environment and everything. We started with that monolith with the GraphQL gateway in front, which scaled pretty nicely, because we were able to sort of, even today we have the same GraphQL gateway. We just changed the services backing it, which was really sweet. The adoption process was like, let's try it out. We tried it out with GraphQL first, and then as we were heading into launch, we had this monolith that we needed to manage. I mean, manually managing AWS resources, it's easier than back in the day when you're managing your own virtual machines and stuff, but it's still not great.We didn't have a lot of time, and there was a lot of last-minute changes we needed to make. A big refactor to our scheduling transactions functions happened right before launch. That was an amazing serverless use case. And there's our second one, where we're like, “Okay, we need to get this live really quickly." We created this work performance pattern really quickly as a test with serverless, and it worked beautifully. We also had another use case come up, which was just a simple phone scheduling service. We just wrapped an API, and just exposed some endpoints, but it was just a lot easier to do with serverless. Just threw it off to two developers, figure out how you do it, and it was ready to be live. And then ...Jeremy: I'm sorry to interrupt you, but I want to get to this point, because you're talking about standing up infrastructure, using infrastructure as code, or the tools you're using. How many developers were working on this thing?Patrick: How many, I think at the time, maybe four developers on backend functionality before launch, when we were just starting out.Jeremy: But you're building a banking platform here, so this is pretty sophisticated. I can imagine another business case for serverless is just the sense that we don't have to hire an operations team.Patrick: Yeah, exactly. We were well through launching it. I think it would have been a couple of months where we were live, or where we hired our first dev ops engineer. Which is incredible. Our VP took a lot of that too, I'm sure he had his hands a little more dirty than he did like early on. But it was just amazing. We were able to manage all that infrastructure, and scale was never a concern. In the early stages, maybe it shouldn't be just yet, but it was just really, really easy.Jeremy: Now you started with four, and I think, what are you now? Somewhere around 25 developers? Somewhere in that space now?Patrick: About 25 developers now, we're growing really fast. We doubled this year during COVID, which is just crazy to think about, and somehow have been scaling somewhat smoothly at least, in terms of just being able to output as a dev team promote. We'll probably double again this year. This is maybe where I shamelessly plug that we're hiring, and we always are, and you could visit northone.com and just check out the careers page, or just hit me up for a warm intro. It's been crazy, and that's one of the things that serverless has helped with us too. We haven't had this scaling bottleneck, which is an operations team. We don't need to hire X operations people for a certain number of developers.Onboarding has been easier. There was one example of during a major project, we hired a developer. He was new to serverless, but just very experienced developer, and he had a production-ready serverless service ready in a month, which was just an insane ramp-up time. I haven't seen that very often. He didn't have to talk to any of our operation staff, and we'd already used serverless long enough that we had all of our presets and boilerplates ready, and permissions locked down, so it was just super easy. It's super empowering just for him to be able to just play around with the different services. Because we hit that point where we've invested enough that every developer when they opened a branch, that branch deploys its own stage, which has all of the services, AWS infrastructure deployed.You might have a PR open that launches an instance of Kinesis, and five SQS queues, and 10 Lambdas, and a bunch of other things, and then tear down almost immediately, and the cost isn't something we really worry about. The innovation velocity there has been really, really good. Just being able to try things out. If you're thinking about something like Kinesis, where it's like a Kafka, that's my understanding, and if you think about the organizational buy-in you need for something like Kafka, because you need to support it, come up with opinions, and all this other stuff, you'll spend weeks trying it out, but for one of our developers, it's like this seems great.We're streaming events, we want this to be real-time. Let's just try it out. This was for our analytics use case, and it's live in production now. It seems to be doing the thing, and we're testing out that use case, and there isn't that roadblock. We could always switch off to a different design if you want. The experimentation piece there has been awesome. We've changed, during major projects we've changed the way we've thought about our resources a few times, and in the end it works out, and often it is about resiliency. It's just jamming queues into places we didn't think about in the first place, but that's been awesome.Jeremy: I'm curious with that, though, with 25 developers ... Kinesis for the most part works pretty well, but you do have to watch those iterator ages, and make sure that they're not backing up, or that you're losing events. If they get flooded or whatever, and also sticking queues everywhere, sounds like a really good idea, and I'm a big fan of that, but it also, that means there's a lot of queues you have to manage, and watch, and set alarms and all that kind of stuff. Then you also talked about a pretty, what sounds like a pretty great CI/CD process to spin up new branches and things like that. There's a lot of dev ops-y ops work that is still there. How are you handling that now? Do you have dedicated ops people, or do you just have your developers looking after that piece of it?Patrick: I would say we have a very spirited group of developers who are inspired. We do a lot of our code-sharing via internal packages. A few of our developers just figured out some of our patterns that we need, whether it's like CI, or how we structure our events stores, or how we do our Q subscriptions. We manage these internal packages. This won't scale well, by the way. This is just us being inspired and trying to reduce some of this burden. It is interesting, I've listened to this podcast and a few others, and this idea of infrastructure as code being part of every developer's toolbox, it's starting to really resonate with our team.In our migration, or our swift shift to full, I'd say doing serverless properly, we've learned to really think in it. Think in terms of infrastructure in our creating solutions. Not saying we're doing serverless the right way now, but we certainly did it the wrong way in the past, where we would spin up a bunch of API gateways that would talk to each other. A lot of REST calls going around the spider web of communication. Also, I'll call these monster Lambdas, that have a whole procedure list that they need to get through, and a lot of points of failure. When we were thinking about the way we're going to do Lambda now, we try to keep one Lambda doing one thing, and then there's pieces of infrastructure stitching that together. EventBridge between domain boundaries, SQS for commands where we can, instead of using API gateway. I think that transitions pretty well into our big break. I'm talking about this as our migration to serverless. I want to talk more about that.Jeremy: Before we jump into that, I just want to ask this question about, because again, I call those fat, some people call them fat Lambdas, I call them Lambda lifts. I think there's Lambda lifts, then fat Lambdas, then your single-purpose functions. It's interesting, again, moving towards that direction, and I think it's super important that just admitting that you're like, we were definitely doing this wrong. Because I think so many companies find that adopting serverless is very much so an evolution, and it's a learning thing where the teams have to figure out what works for them, and in some cases discovering best practices on your own. I think that you've gone through that process, I think is great, so definitely kudos to you for that.Before we get into that adoption and the migration or the evolution process that you went through to get to where you are now, one other business or technical case for serverless, especially with something as complex as banking, I think I still don't understand why I can't transfer personal money or money from my personal TD Bank account to my wife's local checking account, why that's so hard to do. But, it seems like there's a lot of steps. Steps that have to work. You can't get halfway through five steps in some transaction, and then be like, oops we can't go any further. You get to roll that back and things like that. I would imagine orchestration is a huge piece of this as well.Patrick: Yeah, 100%. The banking lends itself really well to these workflows, I'll call them. If you're thinking about even just the start of any banking process, there's this whole application process where you put in all your personal information, you send off a request to your bank, and then now there's this whole waterfall of things that needs to happen. All kinds of checks and making sure people aren't on any fraud lists, or money laundering lists, or even just getting a second dive from our compliance department. There's a lot of steps there, and even just keeping our own systems in sync, with our off-provider and other places. We definitely lean on using step functions a lot. I think they work really, really well for our use case. Just the visual, being able to see this is where a customer is in their onboarding journey, is very, very powerful.Being able to restart at any point of their, or even just giving our compliance team a view into that process, or even adding a pause portion. I think that's one of the biggest wins there, is that we could process somebody through any one of our pipelines, and we may need a human eye there at least for this point in time. That's one of the interesting things about the banking industry is. There are still manual processes behind the scenes, and there are, I find this term funny, but there are wire rooms in banks where there are people reviewing things and all that. There are a lot of workflows that just lend themselves well to step functions. That pausing capability and being able to return later with a response, so that allows you to build other internal applications for your compliance teams and other teams, or just behind the scenes calls back, and says, "Okay, resume this waterfall."I think that was the visualization, especially in an events world when you're talking about like sagas, I guess, we're talking about distributed transactions here in a way, where there's a lot of things happening, and a common pattern now is the saga pattern. You probably don't want to be doing two-phase commits and all this other stuff, but when we're looking at sagas, it's the orchestration you could do or the choreography. Choreography gets very messy because there's a lot of simplistic behavior. I'm a service and I know what I need to do when these events come through, and I know which compensating events I need to dump, and all this other stuff. But now there's a very limited view.If a developer is trying to gain context in a certain domain, and understand the chain of events, although you are decoupled, there's still this extra coupling now, having to understand what's going on in your system, and being able to share it with external stakeholders. Using step functions, that's the I guess the serverless way of doing orchestration. Just being able to share that view. We had this process where we needed to move a lot of accounts to, or a lot of user data to a different system. We were able to just use an orchestrator there as well, just to keep an eye on everything that's going on.We might be paused in migrating, but let's say we're moving over contacts, a transaction list, and one other thing, you could visualize which one of those are in the red, and which one we need to come in and fix, and also share that progress with external stakeholders. Also, it makes for fun launch parties I'd say. It's kind of funny because when developers do their job, you press a button, and everything launches, and there's not really anything to share or show.Jeremy: There's no balloons or anything like that.Patrick: Yeah. But it was kind of cool to look at these like, the customer is going through this branch of the logic. I know it's all green. Then I think one of the coolest things was just the retry ability as well. When somebody does fail, or when one of these workflows fails, you could see exactly which step, you can see the logs, and all that. I think one of the challenges we ran into there though, was because we are working in the banking space, we're dealing with sensitive data. Something I almost wish AWS solved out of the box, would be being able to obfuscate some of that data. Maybe you can't, I'm not sure, but we had to think of patterns for tokenization for instance.Stripe does this a lot where certain parts of their platform, you just get it, you put in personal information, you get back a token, and you use that reference everywhere. We do tokenization, as well as we limit the amount of details flowing through steps in our orchestrators. We'll use an event store with identifiers flowing through, and we'll be doing reads back to that event store in between steps, to do what we need to do. You lose some of that debug-ability, you can't see exactly what information is flowing through, but we need to keep user data safe.Jeremy: Because it's the use case for it. I think that you mentioned a good point about orchestration versus choreography, and I'm a big fan of choreography when it makes sense. But I think one of the hardest lessons you learn when you start building distributed systems is knowing when to use choreography, and knowing when to use orchestration. Certainly in banking, orchestration is super important. Again, with those saga patterns built-in, that's the kind of thing where you can get to a point in the process and you don't even need to do automated rollbacks. You can get to a failure state, and then from there, that can be a pause, and then you can essentially kick off the unwinding of those things and do some of that.I love that idea that the token pattern and using just rehydrating certain steps where you need to. I think that makes a ton of sense. All right. Let's move on to the adoption and the migration process, because I know this is something that really excites you and it should because it is cool. I always know, as you're building out applications and you start to add more capabilities and more functionality and start really embracing serverless as a methodology, then it can get really exciting. Let's take a step back. You had a champion in your organization that was beating the drum like, "Let's try this. This is going to make a lot of sense." You build an Apollo Lambda or a Lambda running Apollo server on it, and you are using that as a strangler pattern, routing all your stuff through now to your backend. What happens next?Patrick: I would say when we needed to build new features, developers just gravitated towards using serverless, it was just easier. We were using TypeScript instead of Python, which we just tend to like as an organization, so it's just easier to hop into TypeScript land, but I think it was just easier to get something live. Now we had all these Lambdas popping up, and doing their job, but I think the problem that happened was we weren't using them properly. Also, there was a lot of difference between each of our serverless setups. We would learn each time and we'd be like, okay, we'll use this parser function here to simplify some of it, because it is very bare-bones if you're just pulling the Serverless Framework, and it took a little ...Every service looked very different, I would say. Also, we never really took the time to sit back and say, “Okay, how do we think about this? How do we use what serverless gives us to enable us, instead of it just being an easy thing to spin up?" I think that's where it started. It was just easy to start. But we didn't embrace it fully. I remember having a conversation at some point with our VP being like, “Hey, how about we just put Express into one of our Lambdas, and we create this," now I know it's a Lambda lift. I was like, it was just easier. Everybody knows how to use Express, why don't we just do this? Why are we writing our own parsers for all these things? We have 10 versions of a make response helper function that was copy-pasted between repos, and we didn't really have a good pattern for sharing that code yet in private packages.We realized that we liked serverless, but we realized we needed to do it better. We started with having a serverless chapter reading between some of our team members, and we made some moves there. We created a shared boilerplate at some point, so it reduced some of the differences you'd see between some of the repositories, but we needed a step-change difference in our thinking, when I look back, and we got lucky that opportunity came up. At this point, we probably had another six Lambda services, maybe more actually. I want to say around, we'd probably have around 15 services at this point, without a governing body around patterns.At this time, we had this interesting opportunity where we found out we're going to be re-platforming. A big announcement we just made last month was that we moved on to a new bank partner called Bancorp. The bank partner that supports Chime, and they're like, I'll call them an engine boost. We put in a much larger, more efficient engine for our small businesses. If you just look at the capabilities they provide, they're just absolutely amazing. It's what we need to build forward. Their events API is amazing as well as just their base banking capabilities, the unit economics they can offer, the times on there, things were just better. We found out we're doing an engine swap. The people on the business side on our company trusted our technical team to do what we needed to do.Obviously, we need to put together a case, but they trusted us to choose our technology, which was awesome. I think we just had a really good track record of delivering, so we had free reign to decide what do we do. But the timeline was tight, so what we decided to do, and this was COVID times too, was a few of our developers got COVID tested, and we rented a house and we did a bubble situation. How in the NHL or MBA you have a bubble. We had a dev bubble.Jeremy: The all-star team.Patrick: The all-star team, yeah. We decided let's sit down, let's figure out what patterns are going to take us forward. How do we make the step-change at the same time as step-change in our technology stack, at the same time as we're swapping out this bank, this engine essentially for the business. In this house, we watched almost every YouTube video you can imagine on event driven and serverless, and I think leading up. I think just knowing that we were going to be doing this, I think all of us independently started prototyping, and watching videos, and reading a lot of your content, and Alex DeBrie and Yan Cui. We all had a lot of ideas already going in.When we all got to this house, we started off with this exercise, an event storming exercise, just popular in the domain-driven design community, where we just threw down our entire business on a wall with sticky notes, and it would have been better to have every business stakeholder there, but luckily we had two people from our product team there as representatives. That's how invested we were in building this outright, that we have products sitting in the room with us to figure it out.We slapped down our entire business on a wall, this took days, and then drew circles around it and iterated on that for a while. Then started looking at what the technology looks like. What are our domain boundaries, and what prototypes do we need to make? For a few weeks there, we were just prototyping. We built out what I'd called baby's first balance. That was the running joke where, how do we get an account opened with a balance, with the transactions minimally, with some new patterns. We really embraced some of this domain-driven-design thinking, as well as just event driven thinking. When we were rethinking architecture, three concepts became very important for us, not entirely new, but important. Item potency was a big one, dealing with distributed transactions was another one of those, as well as the eventual consistency. The eventual consistency portion is kind of funny because we were already doing it a lot.Our transactions wouldn't always settle very quickly. We didn't know about it, but now our whole system becomes eventually consistent typically if you now divide all of your architecture across domains, and decouple everything. We created some early prototypes, we created our own version of an event store, which is, I would just say an opinionated scheme around DynamoDB, where we keep track of revisions, payload, timestamp, all the things you'd want to be able to do event sourcing. That's another thing we decided on. Event sourcing seemed like the right approach for state, for a lot of our use cases. Banking, if you just think about a banking ledger, it is events or an accounting ledger. You're just adding up rows, add, subtract, add, subtract.We created a lot of prototypes for these things. Our events store pattern became basically just a DynamoDB with opinions around the schema, as well as a package of a shared code package with a simple dispatch function. One dispatch function that really looks at enforcing optimistic concurrency, and one that's a little bit more relaxed. Then we also had some reducer functions built into there. That was one of the packages that we created, as well as another prototype around that was how do we create the actual subscriptions to this event store? We landed on SNS to SQS fan-out, and it seems like fan-out first is the serverless way of doing a lot of things. We learned that along the way, and it makes sense. It was one of those things we read from a lot of these blogs and YouTube videos, and it really made sense in production, when all the data is streaming from one place, and then now you just add subscribers all over the place. Just new queues. Fan-out first, highly recommend. We just landed on there by following best practices.Jeremy: Great. You mentioned a bunch of different things in there, which is awesome, but so you get together in this house, you come up with all the events, you do this event storming session, which is always a great exercise. You get a pretty good visualization of how the business is going to run from an event standpoint. Then you start building out this event driven architecture, and you mentioned some packages that you built, we talked about step functions and the orchestration piece of this. Just give me a quick overview of the actual system itself. You said it's backed by DynamoDB, but then you have a bunch of packages that run in between there, and then there's a whole bunch of queues, and then you're using some custom packages. I think I already said that but you're using ... are you using EventBridge in there? What's some of the architecture behind all that?Patrick: Really, really good question. Once we created these domain boundaries, we needed to figure out how do we communicate between domains and within domains. We landed on really differentiating milestone events and domain events. I guess milestone events in other terms might be called integration events, but this idea that these are key business milestones. An account was open, an application was approved or rejected, things that every domain may need to know about. Then within our domains, or domain boundaries, we had these domain events, which might reduce to a milestone event, and we can maintain those contracts in the future and change those up. We needed to think about how do we message all these things across? How do we communicate? We landed on EventBridge for our milestone events. We have one event bus that we talked to all of our, between domain boundaries basically.EventBridge there, and then each of our services now subscribed to that EventBridge, and maintain their own events store. That's backed by DynamoDB. Each of our services have their own data store. It's usually an event stream or a projection database, but it's almost all Dynamo, which is interesting because our old platform used Postgres, and we did have relational data. It was interesting. I was really scared at first, how are we going to maintain relations and things? It became a non-issue. I don't even know why now that I think about it. Just like every service maintains its nice projection through events, and builds its own view of the world, which brings its own problems. We have DynamoDB in there, and then SNS to SQS fan-out. Then when we're talking about packages ...Jeremy: That's Office Streams?Patrick: Exactly, yeah. We're Dynamo streams to SNS, to SQS. Then we use shared code packages to make those subscriptions very easy. If you're looking at doing that SNS to SQS fan-out, or just creating SQS queues, there is a lot of cloud formation boilerplate that we were creating, and we needed to move really quick on this project. We got pretty opinionated quick, and we created our own subscription function that just generates all this cloud formation with naming conventions, which was nice. I think the opinions were good because early on we weren't opinionated enough, I would say. When you look in your AWS dashboard, the read for these aren't prefixed correctly, and there's all this garbage. You're able to have consistent naming throughout, make it really easy to subscribe to an event.We would publish packages to help with certain things. Our events store package was one of those. We also created a Lambda handlers package, which leverages, there's like a Lambda middlewares compose package out there, which is quite nice, and we basically, all the common functionality we're doing a lot of, like parsing a body from S3, or SQS or API gateway. That's just the middleware that we now publish. Validation in and out. We highly recommend the library Zod, we really embrace the TypeScript first object validation. Really, really cool package. We created all these middlewares now. Then subscription packages. We have a lot of shared code in this internal NPM repository that we install across.I think one challenge we had there was, eventually you extracted away too much from the cloud formation, and it's hard for new developers to ... It's easy for them to create events subscriptions, it's hard for them to evolve our serverless thinking because they're so far removed from it. I still think it was the right call in the end. I think this is the next step of the journey, is figuring out how do we share code effectively while not hiding away too much of serverless, especially because it's changing so fast.Jeremy: It's also interesting though that you take that approach to hide some of that complexity, and bake in some of that boilerplate that, someone's mostly didn't have to write themselves anyways. Like you said, they're copying and pasting between services, is not the best way to do it. I tried the whole shared packages thing one time, and it kind of worked. It's just like when you make a small change to that package and you have 14 services, that then you have to update to get the newest version. Sometimes that's a little frustrating. Lambda layers haven't been a huge help with some of that stuff either. But anyways, it's interesting, because again you've mentioned this a number of times about using queues.You did mention resiliency in there, but I want to touch on that point a little bit because that's one of those things too, where I would assume in a banking platform, you do not want to lose events. You don't want to lose things. and so if something breaks, or something gets throttled or whatever, having to go and retry those events, having the alerts in place to know that a queue is backed up or whatever. Then just, I'm thinking ordering issues and things like that. What kinds of issues did you face, and tell me a little bit more about what you've done for reliability?Patrick: Totally. Queues are definitely ... like SQS is a workhorse for our company right now. We use a lot of it. Dropping messages is one of the scariest things, so you're dead-on there. When we were moving to event driven, that was what scared me the most. What if we drop an event? A good example of that is if you're using EventBridge and you're subscribing Lambdas to it, I was under the impression early on that EventBridge retries forever. But I'm pretty sure it'll retry until it invokes twice. I think that's what we landed on.Jeremy: Interesting.Patrick: I think so, and don't quote me on this. That was an example of where drop message could be a problem. We put a queue in front of there, an SQS queue as the subscription there. That way, if there's any failure to deliver there, it's just going to retry all the time for a number of days. At that point we got to think about DLQs, and that's something we're still thinking about. But yeah, I think the reason we've been using queues everywhere is that now queues are in charge of all your retry abilities. Now that we've decomposed these Lambdas into one Lambda lift, into five Lambdas with queues in between, if anything fails in there, it just pops back into the queue, and it'll retry indefinitely. You can drop messages after a few days, and that's something we learned luckily in the prototyping stage, where there are a few places where we use dead letter queues. But one of the issues there as well was ordering. Ordering didn't play too well with ...Jeremy: Not with DLQs. No, it does not, no.Patrick: I think that's one lesson I'd want to share, is that only use ordering when you absolutely need it. We found ways to design some of our architecture where we didn't need ordering. There's places we were using FIFO SQS, which was something that just launched when we were building this thing. When we were thinking about messaging, we're like, "Oh, well we can't use SQS because they don't respect ordering, or it doesn't respect ordering." Then bam, the next day we see this blog article. We got really hyped on that and used FIFO everywhere, and then realized it's unnecessary in most use cases. So when we were going live, we actually changed those FIFO queues into just regular SQS queues in as many places as we can. Then so, in that use case, you could really easily attach a dead letter queue and you don't have to worry about anything, but with FIFO things get really, really gnarly.Ordering is an interesting one. Another place we got burned I think on dead-letter queues, or a tough thing to do with dead letter queues is when you're using our state machines, we needed to limit the concurrency of our state machines is another wishlist item in AWS. I wish there was just at the top of the file, a limit concurrent executions of your state machine. Maybe it exists. Maybe we just didn't learn to use it properly, but we needed to. There's a few patterns out there. I've seen the [INAUDIBLE] pattern where you can use the actual state machine flow to look back at how many concurrent executions you have, and pause. We landed on setting reserved concurrency in a number of Lambdas, and throwing errors. If we've hit the max concurrency and it'll pause that Lambda, but the problem with DLQs there was, these are all errors. They're coming back as errors.We're like, we're fine with them. This is a throttle error. That's fine. But it's hard to distinguish that from a poison message in your queue, so when do you dump those into DLQ? If it's just a throttling thing, I don't think it matters to us. That was another challenge we had. We're still figuring out dead letter queues and alerting. I think for now we just relied on CloudWatch alarms a lot for our alerting, and there's a lot you could do. Even just in the state machines, you can get pretty granular there. I know once certain things fail, and announced to your Slack channel. We use that Slack integration, it's pretty easy. You just go on a Slack channel, there's an email in there, you plop it into the console in AWS, and you have your very early alerting mechanism there.Jeremy: The thing with Elasticsearch ... not Elasticsearch, I'm sorry. I'm totally off-topic here. The thing with EventBridge and Lambda, these are one of those things that, again, they're nuances, but event bridge, as long as it can deliver to the Lambda service, then the Lambda service kicks off and queues it automatically. Then that will retry at a certain number of times. I think you can control that now. But then eventually if that retries multiple times and eventually fails, then that kicks it over to the DLQ or whatever. There's all different ways that it works like that, but that's why I always liked the idea of putting a queue in between there as well, because I felt you just had a little bit more control over exactly what happens.As long as it gets to the queue, then you know you haven't lost the message, or you hope you haven't lost a message. That's super interesting. Let's move on a little bit about the adoption issues. You mentioned a few of these things, obviously issues with concurrency and ordering, and some of that other stuff. What about some of the other challenges you had? You mentioned this idea of writing all these packages, and it pulls devs away from the CloudFormation a little bit. I do like that in that it, I think, accelerates a lot of things, but what are some of the other maybe challenges that you've been having just getting this thing up and running?Patrick: I would say IAM is an interesting one. Because we are in the banking space, we want to be very careful about what access do you give to what machines or developers, I think machines are important too. There've been cases where ... so we do have a separate developer set up with their own permissions, in development's really easy to spin up all your services within reason. But now when we're going into production, there's times where our CI doesn't have the permissions to delete a queue or create a queue, or certain things, and there's a lot of tweaking you have to do there, and you got to do a lot of thinking about your IAM policies as an organization, especially because now every developer's touching infrastructure.That becomes this shared operational overhead that serverless did introduce. We're still figuring that out. Right now we're functioning on least privilege, so it's better to just not be able to deploy than deploy something you shouldn't or read the logs that you shouldn't, and that's where we're starting. But that's something that, it will be a challenge for a little while I think. There's all kinds of interesting things out there. I think temporary IAM permissions is a really cool one. There are times we're in production and we need to view certain logs, or be able to access a certain queue, and there's tooling out there where you can, or at least so I've heard, you can give temporary permissions. You have this queue permission for 30 minutes, and it expires and it's audited, and I think there's some CloudTrail tie-in you could do there. I'm speaking about my wishlist for our next evolution here. I hope my team is listening ...Jeremy: Your team's listening to you.Patrick: ... will be inspired as well.Jeremy: What about ... because this is something too that I always found to be a challenge, especially when you start having multiple services, and you've talked about these domain events, but then milestone events. You've got different services that need to communicate across services, or across domains, and realize certain things like that. Service discovery in and of itself, and which queue are we mapping to, or which service am I talking to, and which version of the service am I talking to? Things like that. How have you been dealing with that stuff?Patrick: Not well, I would say. Very, very ad hoc. I think like right now, at least we have tight communication between the teams, so we roughly know which service we need to talk to, and we output our URLs in the cloud formation output, so at least you could reference the URLs across services, a little easier. Really, a GraphQL is one of the only service that really talks to a lot of our API gateways. At least there's less of that, knowing which endpoint to hit. Most of our services will read into EventBridge, and then within services, a lot of that's abstracted away, like the queue subscription's a little easier. Service discovery is a bit of a nightmare.Once our services grow, it'll be, I don't know. It'll be a huge challenge to understand. Even which services are using older versions of Node, for instance. I saw that AWS is now deprecating version 10 and we'll have to take a look internally, are we using version 10 anywhere, and how do we make sure that's fine, or even things like just knowing which services now have vulnerabilities in their NPM packages because we're using Node. That's another thing. I don't even know if that falls in service discovery, but it's an overhead of ...Jeremy: It's a service management too. It's a lot there. That actually made me, it brings me to this idea of observability too. You mentioned doing some CloudWatch alerts and some of that stuff, but what about using some observability tool or tracing like x-ray, and things like that? Have you been implementing any of that, and if you have, have you had any success and or problems with it?Patrick: I wish we had a better view of some of the observability tools. I think we were just building so quickly that we never really invested the time into trying them out. We did use X-Ray, so we rolled our own tooling internally to at least do what we know. X-Ray was one of those, but the problem with X-Ray is, we do subscribe all of our services, but X-Ray isn't implemented everywhere internally in AWS, so we lose our trail somewhere in that Dynamo stream to SNS, or SQS. It's not a full trace. Also, just digesting that huge graph of information is just very difficult. I don't use it often, I think it's a really cool graphic to show, “Hey, look, how many services are running, and it's going so fast."It's a really cool thing to look at, but it hasn't been very useful. I think our most useful tool for debugging and observability has been just our logging. We created a JSON logger package, so we get up JSON logs and we can actually filter off of different properties, and we ship those to Elasticsearch. Now you can have a view of all of the functions within a given domain at any point in time. You could really see the story. Because I think early on when we were opening up CloudWatch and you'd have like 10 tabs, and you're trying to understand this flow of information, it was very difficult.We also implemented our own trace ID pattern, and I think we just followed a Lumigo article where we introduced some properties, and in each of our Lambdas at a higher level, and one of our middlewares, and we were able to trace through. It's not ideal. Observability is something that we'll probably have to work on next. It's been tolerable for now, but I can't see the scaling that long.Jeremy: That's the other thing too, is even the shared package issue. It's like when you have an observability tool, they'll just install a layer or something, where you don't necessarily have to worry about updating your own tool. I always find if you are embracing serverless and you want to get rid of all that undifferentiated heavy lifting, observability tools, there's a lot of really good ones out there that are doing some great stuff, and they're specializing in it. It might be worth letting someone else handle that for you than trying to do it yourself internally.Patrick: Yeah, 100%. Do you have any that you've used that are particularly good? I know you work with serverless so-Jeremy: I played around with all of them, because I love this stuff, so it's always fun, but I mean, obviously Lumigo and Epsagon, and Thundra, and New Relic. They're all great. They all do things slightly differently, but they all follow a similar implementation pattern so that it's very easy to install them. We can talk more about some recommendations. I think it's just one of those things where in a modern application not having that insight is really hard. It can be really hard to debug stuff. If you look at some of the tools that AWS offers, I think they're there, it's just, they are maybe a little harder to implement, and not quite as refined and targeted as some of the observability tools. But still, you got to get there. Again, that's why I keep saying it's an evolution, it's a process. Maybe one time you get burned, and you're like, we really needed to have observability, then that's when it becomes more of a priority when you're moving fast like you are.Patrick: Yeah, 100%. I think there's got to be a priority earlier than later. I think I'll do some reading now that you've dropped some of these options. I have seen them floating around, but it's one of those things that when it's too late, it's too late.Jeremy: It's never too late to add observability though, so it should. Actually, a lot of them now, again, it makes it really, really easy. So I'm not trying to pitch any particular company, but take a look at some of them, because they are really great. Just one other challenge that I also find a lot of people run into, especially with serverless because there's all these artificial account limits in place. Even the number of queues you can create, and the number of concurrent Lambda functions in a particular region, and stuff like that. Have you run into any of those account limit issues?Patrick: Yeah. I could give you the easiest way to run into an account on that issue, and that is replay your entire EventBridge archive to every subscriber, and you will find a bottleneck somewhere. That's something ...Jeremy: Somewhere it'll fall over? Nice.Patrick: 100%. It's a good way to do some quick check and development to see where you might need to buffer something, but we have run into that. I think the solution there, and a lot of places was just really playing with concurrency where we needed to, and being thoughtful about where is their main concurrency in places that we absolutely needed to stay functioning. I think the challenge there is that eats into your total account concurrency, which was an interesting learning there. Definitely playing around there, and just being thoughtful about where you are replaying. A couple of things. We use replays a lot. Because we are using these milestone events between service boundaries, now when you launch a new service, you want to replay that whole history all the way through.We've done a lot of replaying, and that was one of the really cool things about EventBridge. It just was so easy. You just set up an archive, and it'll record everything coming through, and then you just press a button in the console, and it'll replay all of them. That was really awesome. But just being very mindful of where you're replaying to. If you replay to all of your subscriptions, you'll hit Lambda concurrency limits real quick. Even just like another case, early on we needed to replace ... we have our own domain events store. We want to replace some of those events, and those are coming off the Dynamo stream, so we were using dynamo to kick those to a stream, to SNS, and fan-out to all of our SQS queues. But there would only be one or two queues you actually needed to subtract to those events, so we created an internal utility just to dump those events directly into the SQS queue we needed. I think it's just about not being wasteful with your resources, because they are cheap. Sure.Jeremy: But if you use them, they start to cost money.Patrick: Yeah. They start to cost some money as well as they could lock down, they can lock you out of other functionality. If you hit your Lambda limits, now our API gateway is tapped.Jeremy: That's a good point.Patrick: You could take down your whole system if you just aren't mindful about those limits, and now you could call up AWS in a panic and be like, “Hey, can you update our limits?" Luckily we haven't had to do that yet, but it's definitely something in your back pocket if you need it, if you can make the case to AWS, that maybe you do need bigger limits than the default. I think just not being wasteful, being mindful of where you're replaying. I think another interesting thing there is dealing with partners too. It's really easy to scale in the Lambda world, but not every partner could handle that volume really quickly. If you're not buffering any event coming through EventBridge to your new service that hits a partner every time, you're going to hit their API rate limit really quickly, because they're just going to just go right through it.You might be doing thousands of API calls when you're instantiating a new service. That's one of those interesting things that we have to deal with, and particularly in our orchestrators, because they are talking to different partners, that's why we need to really make sure we could limit the concurrent executions of the state machines themselves. In a way, some of our architecture is too fast to scale.Jeremy: It's too good.Patrick: You still have to consider downstream. That, and even just, if you are using relational databases or anything else in your system, now you have to worry about connection limits and ...Jeremy: I have a whole talk I gave on that.Patrick: ... spikes in traffic.Jeremy: Yes, absolutely.Patrick: Really cool.Jeremy: I know all about it. Any final advice for companies like you that are trying to bite off a piece of the serverless apple, I guess, That's really bad. Anyways, any advice for people looking to get into this?Patrick: Yeah, totally. I would say start small. I think we were wise to just try it out. It might not land with your development team. If you don't really buy in, it's one of those things that could just end up unnecessarily messy, so start small, see if you like it in-shop, and then reevaluate, once you hit a certain point. That, and I would say shared boilerplate packages sooner than later. I know shared code is a problem, but it is nice to have an un-opinionated starter pack, that you're at least not doing anything really crazy. Even just things like having opinions around logging. In our industry, it's really important that you're not logging sensitive details.For us doing things like wrapping our HTTP clients to make sure we're not logging sensitive details, or having short Lambda packages that make sure out-of-the-box you're opinionated about not doing something terribly awful. I would say those two things. Start small and a boiler package, and maybe the third thing is just pay attention to the code smell of a growing Lambda. If you are doing three API calls in one Lambda, chances are you could probably break that up, and think about it in a more resilient way. If any one of those pieces fail, now you could have retry ability in each one of those. Those are the three things I would say. I could probably talk forever about the rest of our journey.Jeremy: I think that was great advice, and I love hearing about how companies are going through this process, what that process looks like, and I hope, I hope, I hope that companies listen to this and can skip a lot of these mistakes. I don't want to call them all mistakes, and I think it's just evolution. The stuff that you've done, we've all made them, we've all gone through that process, and the more we can solidify these practices and stuff like that, I think that more companies will benefit from hearing stories like these. Thank you very much for sharing that. Again, thank you so much for spending the time to do this and sharing all of this knowledge, and this journey that you've been on, and are continuing to be on. It would great to continue to get updates from you. If people want to contact you, I know you're not on Twitter, but what's the best way to reach out to you?Patrick: I almost wish I had a Twitter. It's the developer thing to have, so maybe in the future. Just on LinkedIn would be great. LinkedIn would be great, as well as if anybody's interested in working with our team, and just figuring out how to take serverless to the next level, just hit me up on LinkedIn or look at our careers page at northone.com, and I could give you a warm intro.Jeremy: That's great. Just your last name is spelled S-T-R-Z-E-L-E-C. How do you say that again? Say it in Polish, because I know I said it wrong in the beginning.Patrick: I guess for most people it would just be Strzelec, but if there are any Slavs in the audience, it's "Strzelec." Very intense four consonants last name.Jeremy: That is a lot of consonants. Anyways again, Patrick, thanks again. This was great.Patrick: Yeah, thank you so much, Jeremy. This has been awesome.

covid-19 pr service event mba cloud nhl id platform express apollo i am banking dropping polish slack validation api python aws onboarding faa stripe item kafka ordering chime sns x ray s3 node dynamo urls choreography lambda zod baas serverless ci cd json typescript queues graphql td bank observability fifo npm elasticsearch new relic postgres aws lambda inaudible slavs apache kafka kinesis dynamodb lambdas cloudformation bancorp our vp sqs cloudwatch northone serverless framework yan cui strzelec eventbridge thundra jeremy it jeremy you cbt nuggets patrick you patrick yeah jeremy great

Serverless Observerless with Aviad Mor

Play Episode Listen Later May 25, 2021 35:20

About AviadAviad Mor is the Co-Founder & CTO at Lumigo. Lumigo’s SaaS platform helps companies monitor and troubleshoot serverless applications while providing actionable insights that prevent business disruptions. Aviad has over a decade of experience in technology leadership, heading the development of core products in Check Point from inception to wide adoption.Links:Lumigo: https://lumigo.io/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored by ExtraHop. ExtraHop provides threat detection and response for the Enterprise (not the starship). On-prem security doesn’t translate well to cloud or multi-cloud environments, and that’s not even counting IoT. ExtraHop automatically discovers everything inside the perimeter, including your cloud workloads and IoT devices, detects these threats up to 35 percent faster, and helps you act immediately. Ask for a free trial of detection and response for AWS today at extrahop.com/trial.Corey: This episode is sponsored in part by our friends at Lumigo. If you’ve built anything from serverless, you know that if there’s one thing that can be said universally about these applications, it’s that it turns every outage into a murder mystery. Lumigo helps make sense of all of the various functions that wind up tying together to build applications.It offers one-click distributed tracing so you can effortlessly find and fix issues in your serverless and microservices environment. You’ve created more problems for yourself; make one of them go away. To learn more, visit lumigo.io.Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. I periodically talk about how I bolt together a whole bunch of different serverless tools in horrifying ways to write my newsletter every week. At last count, I was up to something like four API Gateways, twenty-nine Lambda functions, and counting. How do I figure out if something’s broken in there? Well, honestly, I just keep clicking the button until it works, which is a depressingly honest story.Now, that doesn’t work for everyone. Today’s promoted episode is brought to us by Lumigo. And my guest today is Aviad Mor, their CTO, and co-founder. Aviad, thanks for taking the time to suffer my slings and arrows.Aviad: Thank you, Corey. I’m very glad to be here today.Corey: So, let’s begin at, I guess, the very easy, high-level question: what is Lumigo and is ‘loom-ago’ an accepted alternate pronunciation?Aviad: [laugh]. So, Lumigo is a monitoring and debugging platform for serverless environments. And yes, you can call it whatever you want as long as it’s Lu-mi-go. What we do is we integrate with the customer’s AWS account, we do a very quick connection to its Lambdas, and then we’re able to show him exactly what’s going on in his system: what’s going well, what’s going wrong, and how we can fix it.Corey: So, let’s make sure that we hit a few points here at the beginning. It is AWS specific at this time?Aviad: Yes, it is. We’re not officially exclusive with AWS, but right now we see the most interesting serverless environments in AWS, so it’s a pretty easy call. But we are keeping our eye open to, you know, Google, Microsoft, even Oracle.Corey: Oh, Oracle Cloud has some phenomenally interesting serverless stories that I don’t think the world is paying enough attention to yet. But one of these days, I’m hoping that that’s going to change just because they have so much savvy locked up in that platform.Aviad: Right. They do have serverless functions. Yeah, so.Corey: They acquired the iron.io folks a while back, and those people were way ahead of Lambda at the time.Aviad: Right, right. So, we’re waiting for the big breakout of serverless in Oracle, and then we’ll build the best monitoring solution for them.Corey: So, something else, I think, that you have successfully navigated as far as, I guess, the traps that various observability tooling falls into, you also talk on your site about monitoring AWS Lambda as the center around which everything winds up being captured. You also, of course, integrate with the things that tied directly into it, such as API Gateway—or ‘opi-gateway,’ as I’m sure they mispronounce it at AWS—but that’s sort of where you stop. You don’t also show all of the container workloads that folks are running, and, “Oh, hey. While we have access to your API, here’s a whole story about ECS, and RDS, and all the rest.” And eventually, it feels like everything, in the fullness of time, tries to become Datadog version two.And that always drove me nuts because I want something that’s going to talk to me specifically about what it is that I’m looking at in a serverless application context, not trying to be all things to all people at once. Is that a fair assessment of the product strategy you folks have pursued?Aviad: Right. So, we’re very focused on serverless. We think there’s a lot of interesting things that we can do there, and we’re actually seeing more and more use cases of serverless. And it is important to say that when we say serverless, it’s very clear what is serverless. So Lambda, of course, and API Gateway, DynamoDB, S3, and so on.There’s a lot of services in data ecosystem, and seeing them all being tied together in a serverless cloud application, we’re able to do all of that to monitor it; not only monitor it at the high level, but also get into the details and show you things which are very specific because this is what we do all day, and sometimes all night. And then there are those boundaries of where do we go beyond serverless. So, there are some hybrid environments out there. And when I say ‘hybrid,’ the easy hybrid, which is you have two different applications which just happen to be on the same AWS account; one of them is completely serverless, and then the other one is EC2. So, that’s kind of hybrid.But the more interesting hybrid is those applications which start with an API Gateway in the Lambda, and then are directly connected to something else, which is maybe Fargate, ECS, EKS, and so on. So, we are very much focused on serverless, but we are getting also a lot of requests from our customers, “So, show us, also, the other parts.” We’re starting to look at that, but we’re not losing our focus. Our focus is still very much on the serverless while allowing you to tie together if you do have some other aspects in your environment to see them all together.Corey: So, you’ve done a number of things that I would consider best in class as you’ve gone through this. First and foremost, let’s begin with the easy stuff. It doesn’t appear that your dashboard, your tooling itself, is purely serverless itself. I can tell this because when I click around in your site, the site loads super quickly. It’s not waiting for cold starts or waiting for the latency inherent to the Lambda.It’s clear that you have not gone so far down the path of being, I guess, religiously correct around everything must be serverless all the times in favor of improving customer experience. That’s something that I’ve seen a number of different vendors fall into the trap of, of, “Why is the dashboard so slow to load?” “Ah, because everything is itself a Lambda function.” Is that accurate, or if you just found a way to improve Lambda [laugh] function performance in an ungodly way?Aviad: [laugh]. We are serverless—we call ourselves serverless first, but the customer is always—he’s really the first. So, if there’s a place where serverless is not the best solution, we’re going to use whatever is the best solution. But the truth is, we’re, I’d say, something like 99% serverless. And specifically, anything which is dashboard-facing customer-facing, that’s actually completely serverless.So, we did have to put in a lot of work, but also, I have to say that AWS went a very long way, like, in the last two years, allowing us to give much better latencies in different parts of the dashboard. So, all of that is serverless, and it goes together with the new features of Lambdas, and API Gateways, and a lot of small things we had to do in order to provide the best experience to the customer.Corey: The next thing that I think was interesting, as far as, I guess, capturing the way in which people use these things. One of the earliest problems I had, in the early days of these, I guess, new breed of serverless tools was getting everything instrumented correctly. It felt like it took in some cases more time to get the observability pieces working than it did to write the thing in the first place. So, you’re integrating out of the gate with a lot of the right things as best I can tell. Your website advertises that you integrate with the Serverless Framework, you integrate with a bunch of other [processes 00:07:52] as well. Chalice, which I haven’t seen used in anger too much, but okay; Terraform, which everyone’s using; Stackery, et cetera. Is AWS’s SAM on the list as well?Aviad: Yes, it actually is. And once we started seeing more and more users using SAM, we had to provide a way to allow them to easily do the integration. Because one of the things that we learned is, no, our users are developers and, just like you said, they don’t want to spend time on anything, which is not like doing the thing that they want to do. Especially in serverless, because the whole serverless premise is work on what you do best, and don’t spend time on everything else. So, we actually spend a lot of time ourselves in order to make the integrations as easy as possible, as quickly as possible, and that also means that working with a lot of different tools to fit all the different environments our users are using out there.Corey: It looks like you’re doing this through the application—judiciously—of a bunch of custom layers. In other words, whatever you wind up using winds up being built as an underpinning of the existing Lambda functions, so it’s super easy to wind up grabbing the necessary dependencies for anything that you folks support without having to go back and refactor existing applications. Is that directionally correct?Aviad: Right. That’s correct. We’re using layers in order to, on one hand, do this deep integration we do with the Lambda, allowing us to do different instrumentations, collecting the data that’s being passed into the Lambda, being passed out of the Lambda, on one hand. But on the other hand, so the developer doesn’t have to make any code changes, and he can do whatever changes he wants to do. Doesn’t have to think about Lumigo at any point, and serverless layer does everything for him automatically.Corey: How do you handle support of the Lambda@Edge functions, which seem an awful lot like regular Lambda functions, except they’re worse in nearly every single way, every time I’ve tried to use them? In fact, in my experience, the best practice has been to immediately rip out Lambda@Edge and replace it with something else. Recently, it was formally disclosed that they only ran in a subset of 13 regional cache locations, and they still took a full CloudFront distribution update cycle every time you did a deployment, which dramatically slowed everything down for deploying it; they were massively expensive to run at significant scale, and they would log to whatever region was closest so it was a constant game of whack a mole to figure out what was going on. But, you know, other than that, they were great. How do you approach those?Aviad: Lambda@Edge are not very easy to use, and they’re, like, let’s say they’re full of surprises [laugh] because not everything they do is exactly what you find in the documentation. But again, since our users are using them, we had to make sure that we give them proper support. And giving them the proper support—other than running and collecting the data—is things that you mentioned, like the fact that it will log to the specific region it’s running in, so you have to go and collect all this data from different places, and you don’t really know exactly where it’s going to run. So, the main thing here is just to make things easy. It’s a bit of a mess when you’re looking at it directly, and taking all the information, putting it in one place so you as a user can just go ahead and read it and you don’t care where it’s running and what it’s doing, that was the main challenge which we worked on and added to the product.Corey: So, across the board, it seems like you folks have been evolving in lockstep with the underlying platform itself. Have you had time to evaluate their new CloudFront Functions, I believe is what they’re calling it. Or is it CloudFront Workers? I can never quite keep it straight; between all the different providers, all the words start to sound alike. But the thing that only runs for a millisecond or two, only in JavaScript, only in the actual all the CloudFront edge locations, et cetera, et cetera. Rather than fixing Lambda@Edge, they decided to roll something completely different out, and I haven’t looked at anything approaching the observability story yet because I’m still too angry about it.Aviad: [laugh]. Right. So, there’s a lot of things coming out, and we’re also very close partners with AWS, so in many cases, we’re actually beta users of new services or new functionality in Lambda. And one of the hardest parts is—and then we cannot spend all our time checking everything new. So, this is one of the things which is still in the to-do list; we’re going to check it very close, in a very close time.I think it’s interesting to see how we can actually use it and is it as quickly as they say. What they say usually works; we’ll see if it works already today, or do we have to wait a little bit until it works exactly like they said. But no, that’s one of the things that are on my to-do list. I’m really looking forward to checking it out.Corey: So, it looks like once I set this up and it starts monitoring my account—or observing my account. I know I know, observability is just hipster monitoring, but no one agrees with me on that, so I’m going to keep rolling with it anyway just to irritate people—it looks like I can effectively more-or-less click a button, and suddenly, you will roll out an underlying Lambda layer to all of my existing Lambda functions. How does that get maintained whenever I wind up, for example, doing a new deployment with the serverless framework or something like it that isn’t aware of that underlying layer, so it—presumably—would revert that layer itself in the definition? Or am I misunderstanding how that works?Aviad: No, no. You’re actually getting it right. So, unless you, for example, are using a serverless plugin, so this is an integral part of your deployment, one of the things that we need to do is to automatically identify that a deployment is happening so we can automatically update the Lambda layer to be the right one, so you won’t miss anything. And this deep integration, which is happening without the user having to know anything about it, this is, I think, one of the most important parts because in serverless, as you know, you have so many components, and very easily you can reach, you know, hundreds of Lambdas, which are things that we’re seeing. So, if a user has to take care and maintain something across a hundred Lambdas or more, you can be sure that it won’t be maintained because he has, like, something much more important to do.So, behind the scenes, immediately as the deployment is happening, we can recognize that it’s happening, and then update the layer that’s required. And by the way, now the layers have a new part called extensions, which allow us and everybody else to do a lot more with those layers, basically allowing the code to run in parallel to the Lambda. So, this is a new thing that AWS has started to roll out, and we think will allow us to give even better experience to our users.Corey: Let’s have a look across the, I guess, the ecosystem of have different approaches to this stuff. One thing that has always annoyed me about a whole raft of observability and monitoring tools is they wind up charging me whatever it is they charge me; it’s generally fine—and I don’t really have a problem with that. You know, in advance going in what things are going to cost you. Incidentally, what is your pricing model?Aviad: So, our pricing model is according to the number of invocations you have. So, we have basically two models which we’re using right now, and each one can decide what he wants better. So, if you want to know in advance exactly how much you’re going to pay, you can go with the tiered model meaning, I want to pay for, let’s say, a million invocations each month, and then you’re sure that you’re paying exactly for what you have a budget for. And it’s always related to how much your AWS account is working, so similar to how much you’re paying for your Lambdas. And then there’s another way, which is dynamic pricing, which is very similar to serverless payment.So, it’s really according to the number of invocations you have; you don’t need to decide in advance, and at the end of each month, according to the number of invocations you have, you get the bill. And that way it’s not based on the invocation in general; it’s exactly according to the number of invocations.Corey: And let’s be clear, if I wind up exceeding the number of invocations under my plan, it just stops tracing and observing these things, it doesn’t break my app.Aviad: Yeah, right. [laugh].Corey: Always good to triple-check those things. It seems like that might hurt.Aviad: That’s very important. You’re totally correct. And, yeah, we never do anything bad to your Lambdas. That’s written on the top of our door: “Never hurt a Lambda.” And we make sure that nothing bad happens, we just stopped collecting data.And by the way, even as you pass your limit, we still collect the basic metrics so you can see what’s going on in your system. But you won’t be able to see the rich information, all the information that allowing you to do the debugging, or seeing the full traceability end-to-end of all the invocations and see how they’re connected to each other.Corey: This episode is sponsored in part by Thinkst. This is going to take a minute to explain, so bear with me. I linked against an early version of their tool, canarytokens.org in the very early days of my newsletter, and what it does is relatively simple and straightforward. It winds up embedding credentials, files, that sort of thing in various parts of your environment, wherever you want to; it gives you fake AWS API credentials, for example. And the only thing that these things do is alert you whenever someone attempts to use those things. It’s an awesome approach. I’ve used something similar for years. Check them out. But wait, there’s more. They also have an enterprise option that you should be very much aware of canary.tools. You can take a look at this, but what it does is it provides an enterprise approach to drive these things throughout your entire environment. You can get a physical device that hangs out on your network and impersonates whatever you want to. When it gets Nmap scanned, or someone attempts to log into it, or access files on it, you get instant alerts. It’s awesome. If you don’t do something like this, you’re likely to find out that you’ve gotten breached, the hard way. Take a look at this. It’s one of those few things that I look at and say, “Wow, that is an amazing idea. I love it.” That’s canarytokens.org and canary.tools. The first one is free. The second one is enterprise-y. Take a look. I’m a big fan of this. More from them in the coming weeks.Corey: So, the pricing makes perfect sense, and that is in line with what I would expect, but the thing that irritates me then is, “Great. I know what I’m going to be paying you folks on a monthly basis, and that’s fine.” And then I use the monitoring tool and it cost me over three times as much in AWS charges, both direct and indirect, where it’s, “Oh, now CloudWatch is going to suddenly be the largest component of my bill and data transfer for sending everything externally winds up spiking into the stratosphere.” What’s your experience been around that?Aviad: So, since we are collecting data and we are doing API calls, it will affect your AWS bill. But because we don’t want to irritate you, or anybody else, we are putting a lot of focus to see that we’re doing the absolute minimal possible effect on your system. So, for example, as we’re collecting data from your Lambda, we’re doing our best to add only milliseconds to the running time of your Lambda so you don’t end up paying a lot more for the runtime. Or for the API calls or data transfer, we have a lot of optimizations that we did, so the billing on your AWS account is really very, very small; it’s not something that you will notice. And sometimes when people do ask us, we go together with them into their account and show them exactly how their billing was affected by Lumigo so they’ll have assurance that nothing crazy is going on there.Corey: Which is I guess one of the fundamental problems of the underlying platform itself. I have a hard time blaming you for any of this stuff. This is the perpetual joyless story of getting to work with a variety of AWS services. It’s not something that I see that you folks have a good way around just on basis of how the underlying platform works.Aviad: Yeah. And then there are a lot of different prices for a lot of small things that you do, and you need to be able to collect it all in order to have the big picture of the effect. And yeah, we don’t have a silver bullet for it, but we can show exactly where we’re going, what we’re adding, to show how low it is.Corey: One of the things that I think is not well understood for folks who are not into the serverless ecosystem is just how these applications tend to look. In most organic environments, you’ll see a whole bunch of Lambda functions that are all tied together with basically spit and baling wire. They talk to each other, either directly on the back end—which is an anti-pattern in many respects, let’s not kid ourselves—or alternately, they’re approaching through a lens of, we’re going to now talk to each other through hardened REST APIs, which is generally preferred, but also a little finicky to get up and running. So, at some point, you have a request come in, and it winds up bouncing around through a whole bunch of different subsystems. Tracing, and a lot of the observability story around serverless is figuring out, all right, somewhere in that rat’s nest, it winds up failing.Where did it break? What was it that actually threw the exception? What was it that prevented something from working? Or alternately, adding latency: where is the bulk of the time serving that request being spent? And you would think that this is the sort of thing that AWS could handle itself.And they’ve tried with their X-Ray distributed tracing option, which more or less feels like a proof of concept demonstrating what not to do. And if you take a look from their application view, and all the rest, it is the best sales pitch I can possibly imagine for any of the serverless monitoring tools that I’ve seen because it is so badly articulated. You have to instrument all of your stuff by hand. There’s none of this, oh, I’ll look at it and figure out what it’s talking to and build an automated trace approach, the way that Lumigo does. And that’s always frustrated me because I shouldn’t have to turn every weird analysis into a murder mystery. Am I missing something obvious in how this could be done using native tools directly, or is it really as bad as I believe it is?Aviad: [laugh]. I won’t say it as bad as you’re saying it is. I think X-Ray is a great place to start with. So, if you have, like, just a few Lambdas; you’re starting to check out the serverless world, X-Ray can be good enough if you don’t want to start with a third-party tool right at the beginning. But then as it gets a little bit complex, it’s going to get hard, especially if you’re trying to do it yourself.That’s usually the wow part when people start using Lumigo when we show them a demo, is seeing how everything is tied together. So, once you see how everything is tied together: the whole system, which components are talking to each other, and how they’re affecting each other. And for example, if one of them goes down, does it mean that the whole system now is not working, or maybe, eh, wasn’t that important, and everything is working. I’ll fix it next week. But I think the most important part is actually what we call the transactions.So, as you said, there’s an API call at the very beginning with an API Gateway or AppSync, and then it can go through dozens of components. Some of them are not directly related, so it’s like, Lambda calling, putting something into a DynamoDB, which triggers a DynamoDB stream. And then another Lambda is being called, and so on, and so on. It’s crucial to be able to see how everything is connected, both very visually, so you can understand it. There’s only so much you can understand when looking at a list as a human being, right?You need to see it visually how everything is connected. But then after you understand how everything is connected in this specific transaction, if, for example, you have an issue in a specific invocation, you need to understand the story of that invocation. And maybe you’re looking at a Lambda which starts to throw an exception, and you didn’t change anything in its code today, yesterday, or the day before that, so take care of that exception, but the root cause is probably not in that Lambda, it’s probably upstream. So, you need to be able to understand exactly what was the chain of events, all the calls being made until that specific Lambda was called to see the data being passed, including the data that Lambda maybe passed to a third-party API—like Stripe or PayPal—and what it got in return. And only when you’re able to see all of that you’re able to solve an issue quickly, not a murder mystery like it might be. Time over time without having to think about how will I make sure that I make all the code changes in order to keep getting these transactions?Corey: So, taking a look at the somewhat crowded space—if I’m being perfectly honest with you—of the entirety of, let’s call it the serverless observability space—or ‘observerless,’ as I’m a big fan of calling it—what is it the differentiates Lumigo from a number of other offerings that people could wind up pulling out of the hat?Aviad: Right. So, that’s a great question. And every time somebody asks me, the first thing I can say is, the more I see people getting into this space, I think that that’s a great sign. Because that means there’s more serverless activity, there’s more companies doing serverless and it means that our serverless space is interesting. People see an opportunity there, and they want to try and solve the issues that we’re seeing there.And I think that there’s a few things: one of them is the serverless expertise. So, if you look at a lot of the big companies—like I’ll mention Datadog and New Relic—they’re doing a lot of great things, but in the end, in the serverless environment, there are very specific things which you need to know, have to do in order to be able to do that distributed tracing, the distributed tracing which allows you to correlate specific transactions together and then bring in those metrics which are relevant and bring in the logs which are relevant for a specific transaction. That’s a lot of hard work which we put in in order to be able to do the transactions with a distributed tracing in the best way possible, and then showing it to you in the simplest way possible. And today, I think that Lumigo does that in a very good way. And also, if we’re looking around at other players, which are not only the big ones, also players, which are doing more specifically serverless, I think that if we’re looking at companies which are very focused on serverless, and serverless is the thing that they do, you’ll still see that Lumigo is the one which is doing serverless the most, let’s call it.So, as serverless is expanding, we’re still not becoming generic—something that we mentioned before—and this allows us not only to do the best distributed tracing but also allow us to show you, out of the box, a lot of issues which might be hiding in your environment. So, it’s not only, “Okay, you have an exception here,” it’s also more specific things to serverless. Like for example, because it’s event-driven, so sometimes you’ll get duplicate events that Kinesis or SQL might send you over and over the same event. The fact that we can show you it automatically and put a spotlight on it can save you a lot of time in trying to understand why things are not working the way you think they should be working. And allowing us to scan your environment and show you misconfigurations which are specific to serverless, this is the kind of things that once you use Lumigo, you get automatically without having to do anything special and that can save you a lot of time.Corey: I think that’s a relatively astute position to take. I’m a big believer in getting the right tool for the right job. I don’t necessarily want the one single pane of glass to look at everything. I think that is something that is very often misunderstood. Yeah, I might be using three or four different clouds in different ways.I don’t need to see a roundup of all of them; I don’t necessarily care what the billing looks like on all of them; I don’t necessarily want to spend my time thinking about different aspects of these things juxtaposed to one another, and it’s a pain in the butt to have to sort through to find the thing I actually care about. So yeah, on some level, I definitely want there to be a specific tool. And let’s be clear, you have a terrific stack of things that you integrate with for alerting, for opening tickets, for remediation—or issues, as the case may be. Nomenclature is always a distraction. Don’t at me—but yeah, across the board, I see that you’re doing a lot of things right that if I were going to be entering the space, I would make a lot of those decisions very similarly. And then expect to hear it from the internet. You’ve been around for years now and are continuing to grow. What’s next for you, folks?Aviad: So, that’s a great question, which I asked myself every morning. I’ll actually take together the two things that you mentioned. One is how we’re focused on serverless, and the second is where do we want to grow from there? And when you do this great focus, you have to make sure that what you’re focusing on is big enough. So, as we’re growing, we’re very happy to see that the serverless is growing with us.We’re seeing more and more places using serverless. We see a lot more users, companies, developers going into serverless. And we see new types of users. So, it’s not only those bleeding-edge technologies that people want to use, and they are really trying to find out how they can use it. We’re seeing more and more places, for example, enterprises that had maybe one architect in the beginning that said, “Okay, I’m going to use serverless.”And now a year or two afterwards, they see that it’s working, and it saving them money. They’re able to build faster, and now it’s both spreading virally to other teams which are starting to use that, and also the initial project, which was started two years ago, is now growing and becoming bigger and more complex. And also, that team which was just starting with serverless two years ago now has maybe a second and third product. So, what we’re doing is we’re looking how we can give serverless better and better monitoring for the new services that are entering that field. And also, we’re very strong believers that developers today are doing much of that monitoring—or observability, you can choose whatever you want—and that means that it goes all the way into debugging.So, we think that doing those two together, bringing together the monitoring and debugging is a great opportunity for our users just to save them more time because it’s the same person who’s going to do both those things, and trying to keep being best of breed in serverless, and doing those two together, I think that’s going to be hard. And that’s exactly the challenge that we’re taking, and we want to see how we’re doing it to the best.Corey: And I think that that is probably the best way to approach it. If people want to learn more about what you’re up to, how you view these things, and ideally, kick the tires on Lumigo and see for themselves, where can they find you?Aviad: So, easiest thing you can do, just search for Lumigo in Google, you’ll get to lumigo.io. And from there, it’s very easy to try us out.Corey: And we will, of course, put links to that in the [show notes 00:31:24]. Thank you so much for taking the time to speak with me today. I really appreciate it.Aviad: Thank you, Corey. It was great fun and looking forward for the next time.Corey: Absolutely. Aviad Mor, co-founder and CTO at Lumigo. I’m Chief Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you hated this podcast, please leave a five-star review on your podcast platform of choice along with a long rambling comment telling me how very wrong I am on the wonder that is Lambda@Edge.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

amazon time google co founders microsoft cloud paypal oracle saas enterprise cto iot api screaming aws stripe javascript devops tracing checkpoint x ray s3 sql lambda rds serverless chalice terraform ecs datadog new relic rest apis nomenclature eks aws lambda ec2 oracle cloud kinesis dynamodb corey quinn api gateway cloudfront nmap fargate lambdas extrahop cloudwatch serverless framework duckbill group thinkst appsync chief cloud economist stackery last week in aws humblepod

#10: Gdzie idziemy? K8s, serverless i edge - opinia!

IT i TO

Play Episode Listen Later May 21, 2021 33:01

K8s rządzi, przynajmniej w ogłoszeniach o pracę i w głowach developerów i edytorach ludzi od DevOps. Ale czy tak zostanie? Czy to jest najlpesza droga? Co czeka nas poza K8s i kontenerami? Moja opinia, wyrażana na blogach i konferencjach w tym odcinku podcastu:- Dlaczego K8s jest trudny i co się stanie z tą technologią w długim okresie czasu? - Gdzie spotyka się K8s i serverless i kto wygra? - Co czeka nas na krawędzi, czyli może popatrzmy w Edge ComputingBONUS! Zapytany o tak zwany "burnout" odpowiadam! Pamiętaj, jeżeli masz jakieś pytanie, nagraj je i wyślij na podcast@onyszko.com. Możesz też wysłać je jako e-mail, ale fajnie by było gdybyś je nagrał(a) i stał się częścią tego podcastu. Podziel się! Link do odcinka, gotowy do udostępnienia niezależnie od platformy: https://share.transistor.fm/s/5ec86315Zasoby wymienione w odcinku:Cloudyna 2019: Navigating between servers: why servereless is not lambda or functions Cloud Forum Podcast o trendach i buzzwords Nagrania z konferencji "Architektura i kontenery" - https://architekturaikontenery.pl/watchProgram Poznaj Kuberenetes (płatny, nie jest to reklama i nie jest to afiljacja, znam autorów) - https://poznajkubernetes.pl/Serverless Framework - https://www.serverless.com/Serverless Polska - https://serverlesspolska.pl/Edge Computing:CloudFlare Workers - https://workers.cloudflare.com/CloudFlare on the Edge - https://stratechery.com/2021/cloudflare-on-the-edge/AWS CloudFront Functions - https://aws.amazon.com/blogs/aws/introducing-cloudfront-functions-run-your-code-at-the-edge-with-low-latency-at-any-scale/Zagraj w DOOM na Edge CloudFlare Workers - https://blog.cloudflare.com/doom-multiplayer-workers/Link do gry: https://silentspacemarine.com/KCP - a minimal K8s Server API: https://github.com/kcp-dev/kcpCoding over Cocktails: Edge Computing in Plain English with Lori MacVittie (dzięki Gutek za podesłanie): Pozostańmy w kontakcie >> Zapisz się do mojego newsletter (EN)

navigating blog cloud doom ale devops czy trendy gdzie cloudflare moja pami serverless edge computing plain english kcp pozosta zapisz k8s architektura podziel opinia idziemy nagrania serverless framework gutek lori macvittie

Episode #101: How Serverless is Becoming More Extensible with Julian Wood

Play Episode Listen Later May 17, 2021 64:40

About Julian WoodJulian Wood is a Senior Developer Advocate for the AWS Serverless Team. He loves helping developers and builders learn about, and love, how serverless technologies can transform the way they build and run applications at any scale. Julian was an infrastructure architect and manager in global enterprises and start-ups for more than 25 years before going all-in on serverless at AWS.Twitter: @julian_woodAll things Serverless @ AWS: ServerlessLandServerless Patterns CollectionServerless Office Hours – every Tuesday 10am PTLambda ExtensionsLambda Container ImagesWatch this episode on YouTube: https://youtu.be/jtNLt3Y51-gThis episode sponsored by CBT Nuggets and Lumigo.TranscriptJeremy: Hi everyone, I'm Jeremy Daly and this is Serverless Chats. Today I'm joined by Julian Wood. Hey Julian, thanks for joining me.Julian: Hey Jeremy, thank you so much for inviting me.Jeremy: Well, I am super excited to have you here. I have been following your work for a very long time and of course, big fan of AWS. So you are a Serverless Developer Advocate at AWS, and I'd love it if you could just tell the listeners a little bit about your background, so they get to know you a bit. And then also, sort of what your role is at AWS.Julian: Yeah, certainly. Well, I'm Julian Wood. I am based in London, but yeah, please don't let my accent fool you. I'm actually originally from South Africa, so the language purists aren't scratching their heads anymore. But yeah, I work within the Serverless Team at AWS, and hopefully do a number of things. First of all, explain what we're up to and how our sort of serverless things work and sort of, I like to sometimes say a bit cheekily, basically help the world fall in love with serverless as I have. And then also from the other side is to be a proxy and sort of be the voice of builders, and developers and whoever's building service applications, and be their voices internally. So you can also keep us on our toes to help build the things that will brighten your days.And just before, I've worked for too many years probably, as an infrastructure racker, stacker, architect, and manager. I've worked in global enterprises babysitting their Windows and Linux servers, and running virtualization, and doing all the operations kind of stuff to support that. But, I was always thinking there's a better way to do this and we weren't doing the best for the developers and internal customers. And so when this, you know in inverted commas, "serverless way" of things started to appear, I just knew that this was going to be the future. And I could happily leave the server side to much better and cleverer people than me. So by some weird, auspicious alignment of the stars, a while later, I managed to get my current dream job talking about serverless and talking to you.Jeremy: Yeah. Well, I tell you, I think a lot of serverless people or people who love serverless are recovering ops and infrastructure people that were doing racking and stacking. Because I too am also recovering from that and I still have nightmares.I thought that it was interesting too, how you mentioned though, developer advocacy. It's funny, you work for a specific company, AWS obviously, but even developer advocacy in general, who is that for? Who are you advocating for? Are you advocating for the developers to use the service from the company? Are you advocating for the developers so that the company can provide the services that they actually need? Interesting balance there.Julian: Yeah, it's true. I mean, the honest answer is we don't have great terms for this kind of role, but yeah, I think primarily we are advocating for the people who are developing the applications and on the outside. And to advocate for them means we've got to build the right stuff for them and get their voices internally. And there are many ways of doing that. Some people raise support requests and other kind of things, but I mean, sometimes some of our great ideas come from trolling Twitter, or yes, I know even Hacker News or that kind of thing. But also, we may get responses from 10 different people about something and that will formulate something in our brain and we'll chat with other kind of people. And that sort of starts a thing. It's not just necessarily each time, some good idea in Twitter comes in, it gets mashed into some big surface database that we all pick off.But part of our job is to be out there and try and think and be developers in whatever backgrounds we come from. And I mean, I'm not a pure software developer where I've come from, and I come, I suppose, from infrastructure, but maybe you'd call that a bit of systems engineering. So yeah, I try and bring that background to try and give input on whatever we do, hopefully, the right stuff.Jeremy: Right. Yeah. And then I think part of the job too, is just getting the information out there and getting the examples out there. And trying to create those best practices or at least surface those best practices, and encourage the community to do a lot of that work and to follow that. And you've done a lot of work with that, obviously, writing for the AWS blog. I know you have a series on the Serverless Lens and the Well-Architected Framework, and we can talk about that in a little while. But I really want to talk to you about, I guess, just the expansion of serverless over the last couple of years.I mean, it was very narrowly focused, probably, when it first came out. Lambda was ... FaaS as a whole new concept for a lot of people. And then as this progressed and we've gotten more APIs, and more services and things that it can integrate with, it just becomes complex and complicated. And that's a good thing, but also maybe a bad thing. But one of the things that AWS has done, and I think this is clearly in reaction to the developers needing it, is the ability to extend what you can do with a Lambda function, right? I mean, the idea of just putting your code in there and then, boom, that's it, that's all you have to do. That's great. But what if you do need access to lifecycle hooks? Or what if you do want to manipulate the underlying runtime or something like that? And AWS, I think has done a great job with that.So maybe we can start there. So just about the extensibility of Lambda in general. And one of the new things that was launched recently was, and recently, I don't know what was it? Seven months ago at this point? I'm not even sure. But was launched fairly recently, let's say that, is Lambda Extensions, and a couple of different flavors of that as well. Could you kind of just give the users an over, the users, wow, the listeners an overview of what Lambda Extensions are?Julian: I could hear the ops background coming in, talking about our users. Yeah. But I mean, from the get-go, serverless was always a terrible term because, why on earth would you name something for what it isn't? I mean, you know? I remember talking to DBAs, talking about noSQL, and they go, "Well, if it's not SQL, then what is it?" So we're terrible at that, serverless as well. And yeah, Lambda was very constrained when it came out. Lambda was never built being a serverless thing, that's what was the outcome. Sometimes we focus too much on the tools rather than the outcome. And the story is S3, just turning 15. And the genesis of Lambda was being an event trigger for S3, and people thought you'd upload something to S3, fire off a Lambda function, how cool is that? And then obviously the clever clubs at the time were like, "Well, hang on, let's not just do this for S3, let's do this for a whole bunch of kind of things."So Lambda was born out of that, as that got that great history, which is created an arc sort of into the present and into the future, which I know we're also going to get on about, the power of event driven applications. But the power of Lambda has always been its simplicity, and removing that operational burden, and that heavy lifting. But, sometimes that line is a bit of a gray area and there're people who can be purists about serverless and can be purists about FaaS and say, "Everything needs to be ephemeral. Lambda functions can't extend to anything else. There shouldn't be any state, shouldn't be any storage, shouldn't be any ..." All this kind of thing.And I think both of us can agree, but I don't want to speak for you, but I think both of us would agree that in some sense, yeah, that's fine. But we live in the real world and there's other stuff that needs to connect to and we're not here about building purist kind of stuff. So Lambda Extensions is a new way basically to integrate Lambda with your favorite tools. And that's the sort of headline thing we like to talk about. And the big idea is to open up Lambda to more effectively work mainly with partners, but also your own tools if you want to write them. And to sort of have deeper hooks into the Lambda lifecycle.And other partners are awesome and they do a whole bunch of stuff for serverless, plus customers also have connections to on-prem staff, or EC2 staff, or containers, or all kind of things. How can we make the tools more seamless in a way? How can we have a common set of tools maybe that you even use on-prem or in the cloud or containers or whatever? Why does Lambda have to be unique or different or that kind of thing? And Extensions is sort of one of the starts of that, is to be able to use these kind of tools and get more out of Lambda. So I mean, just the kind of tools that we've already got on board, there's things like Splunk and AppDynamics. And Lumigo, Epsagon, HashiCorp, Honeycomb, CoreLogic, Dynatrace, I can't think. Thundra and Sumo Logic, Check Point. Yeah, I'm sorry. Sorry for any partners who I've forgotten a few.Jeremy: No, right, no. That's very good. Shout them out, shout them out. No, I mean just, and not to interrupt you here, but ...Julian: No, please.Jeremy: ... I think that's great. I mean, I think that's one of the things that I like about the way that AWS deals with partners, is that ... I mean, I think AWS knows they can't solve all these problems on their own. I mean, maybe they could, right? But they would be their own way of solving the problems and there's other people who are solving these problems differently and giving you the ability to extend your Lambda functions into those partners is, there's a huge win for not only the partners because it creates that ecosystem for them, but also for AWS because it makes the product itself more valuable.Julian: Well, never mind the big win for customers because ultimately they're the one who then gets a common deployment tool, or a common observability tool, or a HashiCorp Vault that you can manage secrets and a Lambda function from HashiCorp Vault. I mean, that's super cool. I mean, also AWS services are picking this up because that's easy for them to do stuff. So if anybody's used Lambda Insights or even seen Lambda Insights in the console, it's somewhere in the monitoring thing, and you just click something over and you get this tool which can pull stuff that you can't normally get from a Lambda function. So things like CPU time and network throughput, which you couldn't normally get. But actually, under the hoods, Lambda Insights is using Lambda extensions. And you can see that if you look. It automatically adds the Lambda layer and job done.So anyway, this is how a lot of the tools work, that a layer is just added to a Lambda function and off you go, the tool can do its work. So also there's a very much a simplicity angle on this, that in a lot of cases you don't have to do anything. You configure some of the extensions via environment variables, if that's cooled you may just have an API key or a log retention value or something like that, I don't know, any kind of example of that. But you just configure that as a normal Lambda environment variable at this partner extension, which is just a Lambda layer, and off you go. Super simple.Jeremy: Right. So explain Extensions exactly, because I think that's one of those things because now we have Lambda layers and we have Lambda Extensions. And there's also like the runtime API and then something else. I mean, even I'm not 100% sure what all of the naming conventions. I'm pretty sure I know what they do ...Julian: Yeah, fair enough.Jeremy: ... but maybe we could say the names and say exactly what they do as well.Julian: Yeah, cool. You get an API, I get an API, everybody gets an API. So Lambda layers, let's just start, because that's, although it's not related to Extensions, it's how Extensions are delivered to the power core functions. And Lambda layers is just another way to add code to a Lambda function or not even code, it can be a dependency. It's just a way that you could, and it's cool because they are shareable. So you have some dependencies, or you have a library, or an SDK, or some training data for something, a Lambda layer just allows you to add some bits and bobs to your Lambda function. That's a horrible explanation. There's another word I was thinking of, because I don't want to use the word code, because it's not necessarily code, but it's dependency, whatever. It's just another way of adding something. I'll wake up in a cold sweat tonight thinking of the word I was thinking of, but anyway.But Lambda Extensions introduces a whole new companion API. So the runtime API is the little bit of code that allows your function to talk to the Lambda service. So when an event comes in, this is from the outside. This could be via API gateway or via the Lambda API, or where else, EventBridge or Step Functions or wherever. When you then transports that data rise in the Lambda services and HTTP call, and Lambda transposes that into an event and sends that onto the Lambda function. And it's that API that manages that. And just as a sidebar, what I find it cool on a sort of geeky, technical thing is, that actually API sits within the execution environment. People are like, "Oh, that's weird. Why would your Lambda API sit within the execution environment basically within the bubble that contains your function rather than it on the Lambda service?"And the cool answer for that is it's actually for a security mechanism. Like your function can then only ever talk to the Lambda runtime API, which is in that secure execution environment. And so our security can be a lot stronger because we know that no function code can ever talk directly out of your function into the Lambda service, it's all got to talk locally. And then the Lambda service gets that response from the runtime API and sends it back to the caller or whatever. Anyway, sidebar, thought that was nerdy and interesting. So what we've now done is we've released a new Extensions API. So the Extensions API is another API that an extension can use to get information from Lambda. And they're two different types of extensions, just briefly, internal and external extensions.Now, internal extensions run within the runtime process so that it's just basically another thread. So you can use this for Python or Java or something and say, when the Python runtime starts, let's start it with another parameter and also run this Java file that may do some observability, or logging, or tracing, or finding out how long the modules take to launch, for example. I know there's an example for Python. So that's one way of doing extensions. So it's internal extensions, they're two different flavors, but I'll send you a link. I'll provide a link to the blog posts before we go too far down the rabbit hole on that.And then the other part of extensions are external extensions. And this is a cool part because they actually run as completely separate processes, but still within that secure bubble, that secure execution environment that Lambda runs it. And this gives you some superpowers if you want. Because first of all, an extension can run in any language because it's a separate process. So if you've got a Node function, you could run an extension in other kind of languages. Now, what do we do recommend is you do run your extension in a compiled binary, just because you've got to provide the runtime that the extensions got to run in any way, so as a compiled binary, it's super easy and super useful. So is something like Go, a lot of people are doing because you write a single extension and Go, and then you can use it on your Node functions, your Java functions, your PowerShell functions, whatever. So that's a really good, simple way that you can have the portability.But now, what can these extensions do? Well, the extensions basically register with extensions API, and then they say to Lambda, "Lambda, I want to know about what happens when my functions invoke?" So the extension can start up, maybe it's got some initialization code, maybe it needs to connect to a database, or log into an observability platform, or pull down a secret order. That it can do, it's got its own init that can happen. And then it's basically ready to go before the function invokes. And then when the extension then registers and says, "I want to know when the function invokes and when it shuts down. Cool." And that's just something that registers with the API. Then what happens is, when a functioning invoke comes in, it tells the runtime API, "Hello, you now have an event," sends it off to the Lambda function, which the runtime manages, but also extension or extensions, multiple ones, hears information about that event. And so it can tell you the time it's going to run and has some metadata about that event. So it doesn't have the actual event data itself, but it's like the sort of Lambda context, a version of that that it's going to send to the extension.So the extension can use that to do various things. It can start collecting telemetry data. It can alter instrument some of your code. It could be managing a secret as a separate process that it is going to cache in the background. For example, we've got one with AppConfig, which is really cool. AppConfig is a service where you manage parameters external to your Lambda function. Well, each time your Lambda function warm invokes if you've got to do an external API call to retrieve that, well, it's going to be a little bit efficient. First of all, you're going to pay for it and it's going to take some time.So how about when the Lambda function runs and the extension could run before the Lambda function, why don't we just cache that locally? And then when your Lambda function runs, it just makes a local HTTP call to the extension to retrieve that value, which is going to be super quick. And some extensions are super clever because they're their own process. They will go, "Well, my value is for 30 minutes and every 30 minutes if I haven't been run, I will then update the value from that." So that's useful. Extensions can then also, when the runtime ... Sorry, let me back up.When the runtime is finished, it sends its response back to the runtime API, and extensions when they're done doing, so the runtime can send it back and the extension can carry on processing saying, "Oh, I've got the information about this. I know that this Lambda function has done X, Y, Z, so let me do, do some telemetry. Let me maybe, if I'm writing logs, I could write a log to S3 or to Kinesis or whatever. Do some kind of thing after the actual function invocation has happened." And then when it says it's ready, it says, "Hello, extensions API, I'm telling you I'm done." And then it's gone. And then Lambda freezes the execution environment, including the runtime and the extensions until another invocation happens. And the cycle then will happen.And then the last little bit that happens is, instead of an invoke coming in, we've extended the Lambda life cycles, so when the environment is going to be shut down, the extension can receive the shutdown and actually do some stuff and say, "Okay, well, I was connected to my observer HTTP platform, so let me close that connection. I've got some extra logs to flush out. I've got whatever else I need to do," and just be able to cleanly shut down that extra process that is running in parallel to the Lambda function.Jeremy: All right.Julian: So that was a lot of words.Jeremy: That was a lot and I bet you that would be great conversation for a dinner party. Really kicks things up. Now, the good news is that, first of all, thank you for that though. I mean, that's super technical and super in-depth. And for anyone listening who ...Julian: You did ask, I did warn you.Jeremy ... kind of lost their way ... Yes, but something that is really important to remember is that you likely don't have to write these yourself, right? There is all those companies you mentioned earlier, all those partners, they've already done this work. They've already figured this out and they're providing you access to their tools via this, that allows you to build things.Julian: Exactly.Jeremy: So if you want to build an extension and you want to integrate your product with Lambda or so forth, then maybe go back and listen to this at half speed. But for those of you who just want to take advantage of it because of the great functionality, a lot of these companies have already done that for you.Julian: Correct. And that's the sort of easiness thing, of just adding the Lambda layer or including in a container image. And yeah, you don't have to worry any about that, but behind the scenes, there's some really cool functionality that we're literally opening up our Lambda operates and allowing you to impact when a function responds.Jeremy: All right. All right. So let me ask another, maybe an overly technical question. I have heard, and I haven't experienced this, but that when it runs the life cycle that ends the Lambda function, I've heard something like it doesn't send the information right away, right? You have to wait for that Lambda to expire or something like that?Julian: Well, yes, for now, about to change. So currently Extensions is actually in preview. And that's not because it's in Beta or anything like that, but it's because we spoke to the partners and we didn't want to dump Extensions on the world. And all the partners had to come out with their extensions on day one and then try and figure out how customers are going to use them and everything. So what we really did, which I think in this case works out really well, is we worked with the partners and said, "Well, let's release this in preview mode and then give everybody a whole bunch of months to work out what's the best use cases, how can we best use this?" And some partners have said, "Oh, amazing. We're ready to go." And some partners have said, "Ah, it wasn't quite what we thought. Maybe we're going to wait a bit, or we're going to do something differently, or we've got some cool ideas, just give us time." And so that's what this time has been.The one other thing that has happened is we've actually added some performance enhancements during it. So yes, currently during the preview, the runtime and all extensions need to finish before we give you your response back to your Lambda function. So if you're in an asynchronous mode, you don't really care, but obviously if you're in a synchronous mode behind an API, yeah, you don't really want that. But when Extensions goes GA, which isn't going to be long, then that is no longer the case. So basically what'll happen is the runtime will respond and the result goes directly back to whoever's calling that, maybe API gateway, and the extensions can carry on, partly asynchronously in the background.Jeremy: Yep. Awesome. All right. And I know that the plan is to go GA soon. I'm not sure when around when this episode comes out, that that will be, but soon, so that's good to know that that is ...Julian: And in fact, when we go GA that performance enhancement is part of the GA. So when it goes GA, then you know, it's not something else you need to wait for.Jeremy: Perfect. Okay. All right. So let's move on to another bit of, I don't know if this is extensibility of the actual product itself or more so I think extensibility of maybe the workflow that you use to deploy to Lambda and deploy your serverless applications, and that's container image support. I mean, we've discussed it a lot. I think people kind of have an idea, but just give me your quick overview of what that is to set some context here.Julian: Yeah, sure. Well, container image support in a simple sort of headline thing is to be able to build and package your functions as a container image. So you basically build a function using a Docker file. So before if you use a zip function, but a lot of people use Serverless Framework or SAM, or whatever, that's all abstracted away from you, but it's actually creating a zip file and uploading it to Lambda or S3. So with container image support, you use a Docker file to build your Lambda function. That's the headline of what's happening.Jeremy: Right. And so the idea of creating, and this is also, and again, you mentioned packaging, right? I mean, that is the big thing here. This is a packaging format. You're not actually running the container in a Lambda function.Julian: Correct. Yeah, let's maybe think, because I mean, "containers," in inverted commas again for people who are on the audio, is ...Jeremy: What does it even mean?Julian: Yeah, exactly. And can be quite an overload of terms and definitely causes some confusion. And I sort of think maybe there's sort of four things that are in the container world. One, containers is an isolation mechanism. So on Linux, this is UNC Group, seccomp, other bits and pieces that can be used to isolate processes or maybe groups of processes. And then a second one, containers as the packaging mechanism. This is what Docker really popularized and this is about taking some code and the dependencies needed to run the code, and then packaging them all out together, maybe with some metadata to describe it.And then, three is containers as also a design philosophy. This is the idea, if we can package and isolate software, it's easier to run. Maybe smaller pieces of software is easy to reason about and manage independently. So I don't want to necessarily use microservices, but there's some component of that with it. And the emphasis here is on software rather than services, and standardized tooling to simplify your ops. And then the fourth thing is containers as an ecosystem. This is where all the products, tools, know how, all the actual things to how to do containers. And I mean, these are certain useful, but I wouldn't say there're anything about the other kind of things.What is cool and worth appreciating is how maybe independent these things are. So when I spoke about containers as isolation, well, we could actually replace containers as isolation with micro VMs such as we do with Firecracker, and there's no real change in the operational properties. So one, if we think, what are we doing with containers and why? One of those is in a way ticked off with Lambda. Lambda does have secure isolation. And containers as a packaging format. I mean, you could replace it with static linking, then maybe won't really be a change, but there's less convenience. And the design philosophy, that could really be applicable if we're talking microservices, you can have instances and certainly functions, but containers are all the same kind of thing.So if we talk about the packaging of Lambda functions, it's really for people who are more familiar with containers, why does Lambda have to be different? You've got, why does Lambda to have to be a snowflake in a way that you have to manage differently? And if you are packaging dependencies, and you're doing npm or pip install, and you're used to building Docker files, well, why can't we just do that for Lambda at the same things? And we've got some other things that come with that, larger function sizes, up to 10 gig, which is enabled with some of this technology. So it's a packaging format, but on the backend, there's a whole bunch of different stuff, which has to be done to to allow this. Benefits are, use your tooling. You've got your CI/CD pipelines already for containers, well, you can use that.Jeremy: Yeah, yeah. And I actually like that idea too. And when I first heard of it, I was like, I have nothing against containers, the containers are great. But when I was thinking about it, I'm like, "Wait container? No, what's happening here? We're losing something." But I will say, like when Lambda layers came out, which was I think maybe 2019 or something like that, maybe 2018, the idea of it made a lot of sense, being able to kind of supplement, add additional dependencies or code or whatever. But it always just seemed awkward. And some of the publishing for it was a little bit awkward. The versioning used like a numbered versioning instead of like semantic versioning and things like that. And then you had to share it to multiple places and if you published it as a SAR app, then you got global distri ... Anyways, it was a little bit hard to use.And so when you're trying to package large dependencies and put those in a layer and then combine them with a Lambda function, the other problem you had was you still had a maximum size that you could use for those, when those were combined. So I like this idea of saying like, "Look, I'd like to just kind of create this little isolate," like you said, "put my dependencies in there." Whether that's PyCharm or some other thing that is a big dependency that maybe I don't want to install, directly in a Lambda layer, or I don't want to do directly in my Lambda function. But you do that together and then that whole process just is a lot easier. And then you can actually run those containers, you could run those locally and test those if you wanted to.Julian: Correct. So that's also one of the sort of superpowers of this. And that's when I was talking about, just being able to package them up. Well, that now enables a whole bunch of extra kind of stuff. So yes, first of all is you can then use those container images that you've created as your local testing. And I know, it's silly for anyone to poo poo local testing. And we do like to say, "Well, bring your testing to the cloud rather than bringing the cloud to your testing." But testing locally for unit tests is super great. It's going to be super fast. You can iterate, have your Lambda functions, but we don't want to be mocking all of DynamoDB, all of building harebrained S3 options locally.But the cool thing is you've got the same Docker file that you're going to run in Lambda can be the same Docker file to build your function that you run locally. And it is literally exactly the same Lambda function that's going to run. And yes, that may be locally, but, with a bit of a stretch of kind of stuff, you could also run those Lambda functions elsewhere. So even if you need to run it on EC2 instances or ECS or Fargate or some kind of thing, this gives you a lot more opportunities to be able to use the same Lambda function, maybe in different way, shapes or forms, even if is on-prem. Now, obviously you can't recreate all of Lambda because that's connected to IM and it's got huge availability, and scalability, and latency and all that kind of things, but you can actually run a Lambda function in a lot more places.Jeremy: Yeah. Which is interesting. And then the other thing I had mentioned earlier was the size. So now the size of these container or these packages can be much, much bigger.Julian: Yeah, up to 10 gig. So the serverless purists in the back are shouting, "What about cold starts? What about cold starts?"Jeremy: That was my next question, yes.Julian: Yeah. I mean, back on zip functional archives are also all available, nothing changes with that Lambda layers, many people use and love, that's all available. This isn't a replacement it's just a new way of doing it. So now we've got Lambda functions that can be up to 10 gig in size and surely, surely that's got to be insane for cold starts. But actually, part of what I was talking about earlier of some of the work we've done on the backend to support this is to be able to support these super large package sizes. And the high level thing is that we actually cache those things really close to where the Lambda layer is going to be run.Now, if you run the Docker ecosystem, you build your Docker files based on base images, and so this needs to be Linux. One of the super things with the container image support is you don't have to use Amazon Linux or Amazon Linux 2 for Lambda functions, you can actually now build your Lambda functions also on Ubuntu, DBN or Alpine or whatever else. And so that also gives you a lot more functionality and flexibility. You can use the same Linux distribution, maybe across your entire suite, be it on-prem or anywhere else.Jeremy: Right. Right.Julian: And the two little components, there's an interface client, what you install, it's just another Docker layer. And that's that runtime API shim that talks to the runtime API. And then there's a runtime interface emulator and that's the thing that pretends to be Lambda, so you can shunt those events between HTTP and JSON. And that's the thing you would use to run locally. So runtime interface client means you can use any Linux distribution at the runtime interface client and you're compatible with Lambda, and then the interface emulators, what you would use for local testing, or if you want to spread your wings and run your Lambda functions elsewhere.Jeremy: Right. Awesome. Okay. So the other thing I think that container support does, I think it opens up a broader set of, or I guess a larger audience of people who are familiar with containerization and how that works, bringing those two Lambda functions. And one of the things that you really don't get when you run a container, I guess, on EC2, or, not EC2, I'm sorry, ECS, or Fargate or something like that, without kind of adding another layer on top of it, is the eventing aspect of it. I mean, Lambda just is naturally an event driven, a compute layer, right? And so, eventing and this idea of event driven applications and so forth has just become much more popular and I think much more mainstream. So what are your thoughts? What are you seeing in terms of, especially working with so many customers and businesses that are using this now, how are you seeing this sort of evolution or adoption of event driven applications?Julian: Yeah. I mean, it's quite funny to think that actually the event of an application was the genesis of Lambda rather than it being Serverless. I mentioned earlier about starting with S3. Yeah, the whole crux of Lambda has been, I respond to an event of an API gateway, or something on SQS, or via the API or anything. And so the whole point in a way of Lambda has been this event driven computing, which I think people are starting to sort of understand in a bigger thing than, "Oh, this is just the way you have to do Lambda." Because, I do think that serverless has a unique challenge where there is a new conceptual learning maybe that you have to go through. And one other thing that holds back service development is, people are used to a client's server and maybe ports and sockets. And even if you're doing containers or on-prem, or EC2, you're talking IP addresses and load balances, and sockets and firewalls, and all this kind of thing.But ultimately, when we're building these applications that are going to be composed of multiple services talking together through using APIs and events, the events is actually going to be a super part of it. And I know he is, not for so much longer, but my ultimate boss, but I can blame Jeff Bezos just a little bit, because he did say that, "If you want to talk via anything, talk via an API." And he was 100% right and that was great. But now we're sort of evolving that it doesn't just have to be an API and it doesn't have to be something behind API gateway or some API that you can run. And you can use the sort of power of events, particularly in an asynchronous model to not just be "forced" again in inverted commas to use APIs, but have far more flexibility of how data and information is going to flow through, maybe not just your application, but your suite of applications, or to and from your partners, or where that is.And ultimately authentications are going to be distributed, and maybe that is connecting to partners, that could be SaaS partners, or it's going to be an on-prem component, or maybe things in other kind of places. And those things need to communicate. And so the way of thinking about events is a super powerful way of thinking about that.Jeremy: Right. And it's not necessarily new. I mean, we've been doing web hooks for quite some time. And that idea of, something is going to happen somewhere and I want to be notified of it, is again, not a new concept. But I think certainly the way that it's evolved with Lambda and the way that other FaaS products had done eventing and things like that, is just those tight integrations and just all of the, I guess, the connective tissue that runs between those things to make sure that the events get delivered, and that you can DLQ them, and you can do all these other things with retries and stuff like that, is pretty powerful.I know you have, I actually just mentioned this on the last episode, about one of my favorite books, I think that changed my thinking and really got me thinking about how microservices communicate with one another. And that was Building Microservices by Sam Newman, which I actually said was sort of like my Bible for a couple of years, yes, I use that. So what are some of the other, like I know you have a favorite book on this.Julian: Well, that Building Microservices, Sam Newman, and I think there's a part two. I think it's part two, or there's another one ...Jeremy: Hopefully.Julian: ... in the works. I think even on O'Riley's website, you can go and see some preview copies of it. I actually haven't seen that. But yeah, I mean that is a great kind of Bible talking. And sometimes we do conflate this microservices things with a whole bunch of stuff, but if you are talking events, you're talking about separating things. But yeah, the book recommendation I have is one called Flow Architectures by James Urquhart. And James Urquhart actually works with VMware, but he's written this book which is looking sort of at the current state and also looking into the future about how does information flow through our applications and between companies and all this kind of thing.And he goes into some of the technology. When we talk about flow, we are talking about streams and we're talking about events. So streams would be, let's maybe put some AWS words around it, so streams would be something like Kinesis and events would be something like EventBridge, and topics would be SNS, and SQS would be queues. And I know we've got all these things and I wish some clever person would create the one flow service to rule them all, but we're not there. And they've got also different properties, which are helpful for different things and I know confusingly some of them merge. But James' sort of big idea is, in the future we are going to be able to moving data around between businesses, between applications. So how can we think of that as a flow? And what does that mean for designing applications and how we handle that?And Lambda is part of it, but even more nicely, I think is even some of the native integrations where you don't have to have a Lambda function. So if you've got API gateway talking to Step Functions directly, for example, well, that's even better. I mean, you don't have any code to manage and if it's certainly any code that I've written, you probably don't want to manage it. So yeah. I mean this idea of flow, Lambda's great for doing some of this moving around. But we are even evolving to be able to flow data around our applications without having to do anything and just wire up some things in a console or in a terminal.Jeremy: Right. Well, so you mentioned, someone could build the ultimate sort of flow control system or whatever. I mean, I honestly think EventBridge is very close to that. And I actually had Mike Deck on the show. I think it was like episode five. So two years ago, whenever it was when the show came out. I mean, when EventBridge came out. And we were talking and I sort of made the joke, I'm like, so this is like serverless web hooks, essentially being able, because there was the partner integrations where partners could push events onto an event bus, which they still can do. But this has evolved, right? Because the issue was always sort of like, I would have to subscribe to web books, I'd have to build a web hook to get events from a particular company. Which was great, always worked fine, but you're still maintaining that infrastructure.So EventBridge comes along, it creates these partner integrations and now you can just push an event on that now your applications, whether it's a Lambda function or other services, you can push them to an SQS queue, you can push them into a Kinesis stream, all these different destinations. You can go ahead and pull that data in and that's just there. So you don't have to worry about maintaining that infrastructure. And then, the EventBridge team went ahead and released the destination API, I think it's called.Julian: Yeah, API destinations.Jeremy: Event API destinations, right, where now you can set up these integrations with other companies, so you don't even have to make the API call yourself anymore, but instead you get all of the retries, you get the throttling, you get all that stuff kind of built in. So I mean, it's just really, really interesting where this is going. And actually, I mean, if you want to take a second to tell people about EventBridge API destinations, what that can do, because I think that now sort of creates both sides of that equation for you.Julian: It does. And I was just thinking over there, you've done a 10 times better job at explaining API destinations than I have, so you've nailed it on the head. And packet is that kind of simple. And it is just, events land up in your EventBridge and you can just pump events to any arbitrary endpoint. So it doesn't have to be in AWS, it can be on-prem. It can be to your Raspberry PI, it can literally be anywhere. But it's not just about pumping the events over there because, okay, how do we handle failover? And how do we handle over throttling? And so this is part of the extra cool goodies that came with API destinations, is that you can, for instance, if you are sending events to some external API and you only licensed for 1,000 invocations, not invocations, that could be too Lambda-ish, but 1,000 hits on the API every minute.Jeremy: Quotas. I think we call them quotas.Julian: Quotas, something like that. That's a much better term. Thank you, Jeremy. And some sort of quota, well, you can just apply that in API destinations and it'll basically store the data in the meantime in EventBridge and fire that off to the API destination. If the API destination is in that sort of throttle and if the API destination is down, well, it's going to be able to do some exponential back-off or calm down a little bit, don't over-flood this external API. And then eventually when the API does come back, it will be able to send those events. So that does just really give you excellent power rather than maintaining all these individual API endpoints yourself, and you're not handling the availability of the endpoint API, but of whatever your code is that needs to talk to that destination.Jeremy: Right. And I don't want to oversell this to anybody, but that also ...Julian: No, keep going. Keep going.Jeremy: ... adds the capability of enhanced security, because you're not exposing those API keys to your developers or anybody else, they're all baked in and stored within, the API destinations or within an EventBridge. You have the ability, you mentioned this idea of not needing Lambda to maybe talk directly, API gateway to DynamoDB or to step function or something like that. I mean, the cool thing about this is you do have translation capabilities, or transformation capabilities in EventBridge where you can transform the event. I haven't tried this, but I'm assuming it's possible to say, get an event from Salesforce and then pipe it into Stripe or some other API that you might want to pipe it into.So I mean, just that idea of having that centralized bus that can communicate with all these different things. I mean, we're talking about distributed systems here, right? So why is it different sending an event from my microservice A to my microservice B? Why can't I send it from my microservice A to company-wise, microservice B or whatever? And being able to do that in a secure, reliable, just with all of that stuff kind of built in for you, I think it's amazing. So I love EventBridge. To me EventBridge is one of those services that rivals Lambda. It's as, I guess as important as Lambda is, in this whole serverless equation.Julian: Absolutely, Jeremy. I mean, I'm just sitting here. I don't actually have to say anything. This is a brilliant interview and Jeremy, you're the expert. And you're just like laying down all of the excellent use cases. And exactly it. I mean, I like to think we've got sort of three interlinked services which do three different things, but are awesome. Lambda, we love if you need to do some processing or you need to do something that's literally your business logic. You've got EventBridge that can route data from in and out of SaaS partners to any other kind of API. And then you've got Step Functions that can do some coordination. And they all work together, but you've got three different things that really have sort of superpowers in terms of the amount of stuff you can do with it. And yes, start with them. If you land up bumping up against any kind of things that it doesn't work, well, first of all, get in touch with me, I'll work on that.But then you can maybe start thinking about, is it containers or EC2, or that kind of thing? But using literally just Lambda, Step Functions and EventBridge, okay. Yes, maybe you're going to need some queues, topics and APIs, and that kind of thing. But ...Jeremy: I was just going to say, add DynamoDB in there for some permanent state or for some data persistence. Right? Yeah. But other than that, no, I think you nailed it. Honestly, sometimes you're starting to build applications and yeah, you're right. You maybe need a queue here and there and things like that. But for the most part, no, I mean, you could build a lot with those three or four services.Julian: Yeah. Well, I mean, even think of it what you used to do before with API destinations. Maybe you drop something on a queue, you'd have Lambda pull that from a queue. You have Lambda concurrency, which would be set to five per second to then send that to an external API. If it failed going to that API, well, you've got to then dump it to Lambda destinations or to another SQS queue. You then got something ... You know, I'm going down the rabbit hole, or just put it on EventBridge ...Jeremy: You just have it magically happen.Julian: ... or we talk about removing serverless infrastructure, not normal infrastructure, and just removing even the serverless bits, which is great.Jeremy: Yeah, no. I think that's amazing. So we talked about a couple of these different services, and we talked about packaging formats and we talked about event driven applications, and all these other things. And a lot of this stuff, even though some of it may be familiar and you could probably equate it or relate it to things that developers might already know, there is still a lot of new stuff here. And I think, my biggest complaint about serverless was not about the capabilities of it, it was basically the education and the ability to get people to adopt it and understand the power behind it. So let's talk about that a little bit because ... What's that?Julian: It sounds like my job description, perfectly.Jeremy: Right. So there we go. Right, that's what you're supposed to be doing, Julian. Why aren't you doing it? No, but you are doing it. You are doing it. No, and that's why I want to talk to you about it. So you have that series on the Well-Architected Framework and we can talk about that. There's a whole bunch of really good resources on this. Obviously, you're doing videos and conferences, well, you used to be doing conferences. I think you probably still do some of those virtual ones, right? Which are not the same thing.Julian: Not quite, no.Jeremy: I mean, it was fun seeing you in Cardiff and where else were you?Julian: Yeah, Belfast.Jeremy: Cardiff and Northern Ireland.Julian: Yeah, exactly.Jeremy: Yeah, we were all over the place together.Julian: With the Guinness and all of us. It was brilliant.Jeremy: Right. So tell me a little bit about, sort of, the education process that you're trying to do. Or maybe even where you sort of see the state of Serverless education now, and just sort of where it's evolved, where we're getting best practices from, what's out there for people. And that's a really long question, but I don't know, maybe you can distill that down to something usable.Julian: No, that's quite right. I'm thinking back to my extensions explanation, which is a really long answer. So we're doing really long stuff, but that's fine. But I like to also bring this back to also thinking about the people aspect of IT. And we talk a lot about the technology and Lambda is amazing and S3 is amazing and all those kinds of things. But ultimately it is still sort of people lashing together these services and building the serverless applications, and deciding what you even need to do. And so the education is very much tied with, of course, having the products and features that do lots of kinds of things. And Serverless, there's always this lever, I suppose, between simplicity and functionality. And we are adding lots of knobs and levers and everything to Lambda to make it more feature-rich, but we've got to try and keep it simple at the same time.So there is sort of that trade-off, and of course with that, that obviously means not just the education side, but education about Lambda and serverless, but generally, how do I build applications? What do I do? And so you did mention the Well-Architected Framework. And so for people who don't know, this came out in 2015, and in 2017, there was a Serverless Lens which was added to it; what is basically serverless specific information for Well-Architected. And Well-Architected means bringing best practices to serverless applications. If you're building prod applications in the cloud, you're normally looking to build and operate them following best practices. And this is useful stuff throughout the software life cycle, it's not just at the end to tick a few boxes and go, "Yes, we've done that." So start early with the well-architected journey, it'll help you.And just sort of answer the question, am I well architected? And I mean, that is a bit of a fuzzy, what is that question? But the idea is to give you more confidence in the architecture and operations of your workloads, and that's not a goal it's in, but it's to reduce and minimize the impact of any issues that can happen. So what we do is we try and distill some of our questions and thoughts on how you could do things, and we built that into the Well-Architected Framework. And so the ServiceLens has a few questions on its operational excellence, security, reliability, performance, efficiency, and cost optimization. Excellent. I knew I was going to forget one of them and I didn't. So yeah, these are things like, how do you control access to an API? How do you do lifecycle management? How do you build resiliency into your application? All these kinds of things.And so the Well-Architected Framework with Serverless Lens there's a whole bunch of guidance to help you do that. And I have been slowly writing a blog series to literally cover all of the questions, they're nine questions in the Well-Architected Serverless Lens. And I'm about halfway through, and I had to pause because we have this little conference called re:Invent, which requires one or two slides to be created. But yeah, I'm desperately keen to pick that up again. And yeah, that's just providing some really and sort of more opinionated stuff, because the documentation is awesome and it's very in-depth and it's great when you need all that kind of stuff. But sometimes you want to know, well, okay, just tell me what to do or what do you think is best rather than these are the seven different options.Jeremy: Just tell me what to do.Julian: Yeah.Jeremy: I think that's a common question.Julian: Exactly. And I'll launch off from that to mention my colleague, James Beswick, he writes one or two things on serverless ...Jeremy: Yeah, I mean, every once in a while you see something from it. Yeah.Julian: ... every day. The Besbot machine of serverless. He's amazing. James, he's so knowledgeable and writes like a machine. He's brilliant. Yeah, I'm lucky to be on his team. So when you talk about education, I learn from him. But anyway, in a roundabout way, he's created this blog series and other series called the Lambda Operations Guide. And this is literally a whole in-depth study on how to operate Lambda. And it goes into a whole bunch of things, it's sort of linked to the Serverless Lens because there are a lot of common kind of stuff, but it's also a great read if you are more nerdily interested in Lambda than just firing off a function, just to read through it. It's written in an accessible way. And it has got a whole bunch of information on how to operate Lambda and some of the stuff under the scenes, how to work, just so you can understand it better.Jeremy: Right. Right. Yeah. And I think you mentioned this idea of confidence too. And I can tell you right now I've been writing serverless applications, well, let's see, what year is it? 2021. So I started in 2015, writing or building applications with Lambda. So I've been doing this for a while and I still get to a point every once in a while, where I'm trying to put something in cloud formation or I'm using the Serverless Framework or whatever, and you're trying to configure something and you think about, well, wait, how do I want to do this? Or is this the right way to do it? And you just have that moment where you're like, well, let me just search and see what other people are doing. And there are a lot of myths about serverless.There's as much good information is out there, there's a lot of bad information out there too. And that's something that is kind of hard to combat, but I think that maybe we could end it there. What are some of the things, the questions people are having, maybe some of the myths, maybe some of the concerns, what are those top ones that you think you could sort of ...Julian: Dispel.Jeremy: ... to tell people, dispel, yeah. That you could say, "Look, these are these aren't things to worry about. And again, go and read your blog post series, go and read James' blog post series, and you're going to get the right answers to these things."Julian: Yeah. I mean, there are misconceptions and some of them are just historical where people think the Lambda functions can only run for five minutes, they can run for 15 minutes. Lambda functions can also now run up to 10 gig of RAM. At re:Invent it was only 3 gig of RAM. That's a three times increase in Lambda functions within a three times proportional increase in CPU. So I like to say, if you had a CPU-intensive job that took 40 minutes and you couldn't run it on Lambda, you've now got three times the CPU. Maybe you can run it on Lambda and now because that would work. So yeah, some of those historical things that have just changed. We've got EFS for Lambda, that's some kind of thing you can't do state with Lambda. EFS and NFS isn't everybody's cup of tea, but that's certainly going to help some people out.And then the other big one is also cold starts. And this is an interesting one because, obviously we've sort of solved the cold start issue with connecting Lambda functions to VPC, so that's no longer an issue. And that's been a barrier for lots of people, for good reason, and that's now no longer the case. But the other thing for cold starts is interesting because, people do still get caught up at cold starts, but particularly for development because they create a Lambda function, they run it, that's a cold start and then update it and they run it and then go, oh, that's a cold start. And they don't sort of grok that the more you run your Lambda function the less cold starts you have, just because they're warm starts. And it's literally the number of Lambda functions that are running at exactly the same time will have a cold start, but then every subsequent Lambda function invocation for quite a while will be using a warm function.And so as it ramps up, we see, in the small percentages of cold starts that are actually going to happen. And when we're talking again about the container image support, that's got a whole bunch of complexity, which people are trying to understand. Hopefully, people are learning from this podcast about that as well. But also with the cold starts with that, those are huge and they're particular ways that you can construct your Lambda functions to really reduce those cold starts, and it's best practices anyway. But yeah, cold starts is also definitely one of those myths. And the other one ...Jeremy: Well, one note on cold starts too, just as something that I find to be interesting. I know that we, I even had to spend time battling with that earlier on, especially with VPC cold starts, that's all sort of gone away now, so much more efficient. The other thing is like provision concurrency. If you're using provision concurrency to get your cold starts down, I'm not even sure that's the right use for provision concurrency. I think provision concurrency is more just to make sure you have enough capacity because of the ramp-up time for Lambda. You certainly can use it for cold starts, but I don't think you need to, that's just my two cents on that.Julian: Yeah. No, that is true. And they're two different use cases for the same kind of thing. Yeah. As you say, Lambda is pretty scalable, but there is a bit of a ramp-up to get up to many, many, many, many thousands or tens of thousands of concurrent executions. And so yeah, using provision currency, you can get that up in advance. And yeah, some people do also use it for provision concurrency for getting those cold starts done. And yet that is another very valid use case, but it's only an issue for synchronous workloads as well. Anything that is synchronous you really shouldn't be carrying too much. Other than for cost perspective because it's going to take longer to run.Jeremy: Sure. Sure. I have a feeling that the last one you were going to mention, because this one bugs me quite a bit, is this idea of no ops or some people call it ops-less, which I think is kind of funny. But that's one of those things where, oh, it drives me nuts when I hear this.Julian: Yeah, exactly. And it's a frustrating thing. And I think often, sometimes when people are talking about no ops, they either have something to sell you. And sometimes what they're selling you is getting rid of something, which never is the case. It's not as though we develop serverless applications and we can then get rid of half of our development team, it just doesn't work like that. And it's crazy, in fact. And when I was talking about the people aspect of IT, this is a super important thing. And me coming from an infrastructure background, everybody is dying in their jobs to do more meaningful work and to do more interesting things and have the agility to try those experiments or try something else. Or do something that's better or even improve the way your build or improve the way your CI/CD pipeline runs or anything, rather than just having to do a lot of work in the lower levels.And this is what serverless really helps you do, is to be able to, we'll take over a whole lot of the ops for you, but it's not all of the ops, because in a way there's never an end to ops. Because you can always do stuff better. And it's not just the operations of deploying Lambda functions and limits and all that kind of thing. But I mean, think of observability and not knowing just about your application, but knowing about your business. Think of if you had the time that you weren't just monitoring function invocations and monitoring how long things were happening, but imagine if you were able to pull together dashboards of exactly what each transaction costs as it flows through your whole entire application. Think of the benefit of that to your business, or think of the benefit that in real-time, even if it's on Lambda function usage or something, you can say, "Well, oh, there's an immediate drop-off or pick-up in one region in the world or one particular application." You can spot that immediately. That kind of stuff, you just haven't had time to play with to actually build.But if we can take over some of the operational stuff with you and run one or two or trillions of Lambda functions in the background, just to keep this all ticking along nicely, you're always going to have an opportunity to do more ops. But I think the exciting bit is that ops is not just IT infrastructure, plumbing ops, but you can start even doing even better business ops where you can have more business visibility and more cool stuff for your business because we're not writing apps just for funsies.Jeremy: Right. Right. And I think that's probably maybe a good way to describe serverless, is it allows you to focus on more meaningful work and more meaningful tasks maybe. Or maybe not more meaningful, but more impactful on the business. Anyways, Julian, listen, this was a great conversation. I appreciate it. I appreciate the work that you're doing over at AWS ...Julian: Thank you.Jeremy: ... and the stuff that you're doing. And I hope that there will be a conference soon that we will be able to attend together ...Julian: I hope so too.Jeremy: ... maybe grab a drink. So if people want to get a hold of you or find out more about serverless and what AWS is doing with that, how do they do that?Julian: Yeah, absolutely. Well, please get hold of me anytime on Twitter, is the easiest way probably, julian_wood. Happy to answer your question about anything Serverless or Lambda. And if I don't know the answer, I'll always ask Jeremy, so you're covered twice over there. And then, three different things. James is, if you're talking specifically Lambda, James Beswick's operations guide, have a look at that. Just so much nuggets of super information. We've got another thing we did just sort of jump around, you were talking about cloud formation and the spark was going off in my head. We have something which we're calling the Serverless Patterns Collection, and this is really super cool. We didn't quite get to talk about it, but if you're building applications using SAM or serverless application model, or using the CDK, so either way, we've got a whole bunch of patterns which you can grab.So if you're pulling something from S3 to Lambda, or from Lambda to EventBridge, or SNS to SQS with a filter, all these kind of things, they're literally copy and paste patterns that you can put immediately into your cloud formation or your CDK templates. So when you are down the rabbit hole of Hacker News or Reddit or Stack Overflow, this is another resource that you can use to copy and paste. So go for that. And that's all hosted on our cool site called serverlessland.com. So that's serverlessland.com and that's an aggregation site that we run because we've got video talks, and we've got blog posts, and we've got learning path series, and we've got a whole bunch of stuff. Personally, I've got a learning path series coming out shortly on Lambda extensions and also one on Lambda observability. There's one coming out shortly on container image supports. And our team is talking all over as many things as we can virtually. I'm actually speaking about container images of DockerCon, which is coming up, which is exciting.And yeah, so serverlessland.com, that's got a whole bunch of information. That's just an easy one-stop-shop where you can get as much information about AWS services as you can. And if not yet, get in touch, I'm happy to help. I'm happy to also carry your feedback. And yeah, at the moment, just inside, we're sort of doing our planning for the next cycle of what Lambda and what all the service stuff we're going to do. So if you've got an awesome idea, please send it on. And I'm sure you'll be super excited when something pops out in the near issue, maybe just in future for a cool new functionality you could have been involved in.Jeremy: Well, I know that serverlessland.com is an excellent resource, and it's not that the AWS Compute blog is hard to parse through or anything, but serverlessland.com is certainly a much easier resource to get there. S

bible benefits south africa fall in love ga reddit cloud honestly windows jeff bezos ip saas beta personally ram salesforce northern ireland api belfast guinness python linux aws java cardiff apis faa stripe alpine invent sar ubuntu cpu vmware extensions checkpoint raspberry pi sns s3 sql docker node sdks lambda baas stack overflow splunk firecrackers serverless ci cd json honeycomb vms ecs nosql powershell corelogic hashicorp woodall hacker news nfs cdk ec2 efs dynatrace dbn appdynamics dbas kinesis sam newman vpc sumo logic dynamodb senior developer advocate extensible fargate pycharm dockercon sqs building microservices serverless framework amazon linux julian wood james urquhart eventbridge thundra jeremy you cbt nuggets jeremy so julian it

Episode #99: You already have a Multi-Cloud Strategy with Rob Sutter

neste nesse epis cesta serverless abaixo taverna serverless framework

Play Episode Listen Later May 3, 2021 62:29

About Rob SutterRob Sutter, a Principal Developer Advocate at Fauna, has woven application development into his entire career, from time in the U.S. Army and U.S. Government to stints with the Big Four and Amazon Web Services. He has started his own company – twice – once providing consulting services and most recently with WorkFone, a software as a service startup that provided virtual digital identities to government clients. Rob loves to build in public with cloud architectures, Node.js or Go, and all things serverless!Twitter: @rts_robPersonal email: rob@fauna.com Personal website: robsutter.comFauna Homepage Learn more about Fauna Supported Languages and Frameworks Try Fauna for FreeThe Calvin PaperThis episode sponsored by CBT Nuggets: https://www.cbtnuggets.com/Watch this video on YouTube: https://youtu.be/CUx1KMJCbvkTranscriptJeremy: Hi, everyone. I'm Jeremy Daly, and this is Serverless Chats. Today, I'm joined by Rob Sutter. Hey, Rob. Thanks for joining me.Rob: Hey, Jeremy. Thanks for having me.Jeremy: So you are now the or a Principal Developer Advocate at Fauna. So I'd love it if you could tell the listeners a little bit about your background and what Fauna is all about.Rob: Right. So as you've said, I'm a DA at Fauna. I've been a serverless user in production since 2017, started the Serverless User Group in Dubai. So that's how I got into serverless in general. Previously, I was a DA on the Serverless Team at AWS, and I've been a SaaS startup co-founder, a government employee, an IT auditor, and like a lot of serverless people I find, I have a lot of Ops in my background, which is why I don't want to do it anymore. There's a lot of us that end up here that way, I think. Fauna is the data API for modern applications. So it's a database that you access as an API just as you would access Stripe for payments, Twilio for messaging. You just put your data into Fauna and access it that way. It's flexible, serverless. It's transactional. So it's a distributed database with asset transactions, and again, it's as simple as accessing any other API that you're already accessing as a developer so that you can simplify your code and ship faster.Jeremy: Awesome. All right. Well, so I want to talk more about Fauna, but I want to talk about it actually in a broader ... I think in the broader ecosystem of what's happening in the cloud right now, and we hear this term "multicloud" all the time. By the way, I'm super excited to have you on. I wanted to have you on for the longest time, and then just schedules, and it's like ...Rob: Yeah.Jeremy: You know how it is, but anyways.Rob: Thank you.Jeremy: No. But seriously, I'm super excited because your tweets, and everything that you've written, and the things that you were doing at AWS and things like that I think just all reinforced this idea that we are living in this multicloud world, right, and that when people think of multicloud ... and this is something I try to be very clear on. Multicloud is not cloud-agnostic, right?Rob: Right.Jeremy: It's a very different thing, right? We're not talking about running the same work load in parallel on multiple service providers or whatever.Rob: Right.Jeremy: We're talking about this idea of using the best services that are available to you across the spectrum of providers, whether those are cloud service providers, whether those are SaaS companies, or even to some degree, some open-source projects that are out there that make up this strategy. So let's start there right from the beginning. Just give me your thoughts on this idea of what multicloud is.Rob: Right. Well, it's sort of a dirty word today, and people like to rail against it. I think rightly so because it's that multicloud 1.0, the idea of, as you said, cloud-agnostic that "write once, run everywhere." All that is, is a race to the bottom, right? It's the lowest common denominator. It's, "What do I have available on every cloud service provider? And then let me write for that as a risk management strategy." That's a cost center when you want to put it in business terms.Jeremy: Right.Rob: Right? You're not generating any value there. You're managing risk by investing against that. In contrast, what you and I are talking about today is this idea of, "Let me use best in class everywhere," and that's a value generation strategy. This cloud service provider offers something that this team understands, and wants to build with, and creates value for the customer more quickly. So they're going to write on that cloud service provider. This team over here has different needs, different customers. Let them write over there. Quite frankly, a lot of this is already happening today at medium businesses and enterprises. It's just not called multicloud, right?Jeremy: Right.Rob: So it's this bottom-up approach that individual teams are consuming according to their needs to create the greatest value for customers, and that's what I like to see, and that's what I like to promote.Jeremy: Yeah, yeah. I love that idea of bottom-up because I think that is absolutely true, and I don't think you've seen this as aggressively as you have in the last probably five years as more SaaS companies have become or SaaS has become a household name. I mean, probably 10 years ago, I think Salesforce was around, and some of these other things were around, right?Rob: Yeah.Jeremy: But they just weren't ... They weren't the household names that they are now. Now, you watch any sport, any professional sport, and you see advertisements for all these SaaS companies now because that seems to be the modern economy. But the idea of the bottom-up approach, that is something where you basically give a developer or maybe you don't give them, but the developer takes the liberties, I would say, to maybe try and experiment with something new without having to do years of research, go through procurement, get approval to use some platform. Even companies trying to move to AWS, or on to Azure, or something like that, they have to go through hoops. Basically, jump through hoops in order to get them there. So this idea of the bottom-up approach, the developers are the ones who are experimenting. Very low-risk experiments, by the way, with some of these other services. That approach, that seems like the right marketing approach for companies that are building these services, right?Rob: Yeah. It seems like a powerful approach for them. Maybe not necessarily the only one, but it is a good one. I mean, there's a historical lesson here as well, right? I want to come back to your point about the developers after, but I think of this as shadow cloud. Right? You saw this with the early days of SaaS where people would go out and sign up for accounts for their business and use them. They weren't necessarily regulated, but we saw even before that with shadow IT, right, where people were bringing their own software in?Jeremy: Right.Rob: So for enterprises that are afraid of this that are heavily risk-focused or control-focused top-down, I would say don't be so afraid because there's an entire set of lessons you can learn about this as you bring it, as you come forward to it. Then, with the developers, I think it's even more powerful than the way you put it because a lot of times, it's not an experiment. I mean, you've seen the same things on Twitter I've seen, the great tech turnover of 2021, right? That's normal for tech. There's such a turnover that a lot of times, people are coming in already having the skills that they know will enhance delivery and add customer value more quickly. So it's not even an experiment. They already have the evidence, and they're able to get their team skilled up and building quickly. If you hire someone who's coming from an AWS shop, you hire someone who's coming from an Azure shop on to two different teams, they're likely going to evolve that excellence or that capability independently, and I don't necessarily think there's a reason to stop that as long as you have the right controls around it.Jeremy: Right. I mean, and you mentioned controls, and I think that if I'm the CTO of some company or whatever, or I'm the CIO because we're dealing in a super enterprise-y world, that you have developers that are starting to use tools ... Maybe not Stripe, but maybe like a Twilio or maybe they're using, I don't know, ChaosSearch or something, something where data that is from within their corporate walls are going out somewhere or being stored somewhere else, like the security risk around that. I mean, there's something there though, right?Rob: Yeah, there absolutely is. I think it's incumbent on the organizations to understand what's going on and adapt. I think it's also imcu,bent on the cloud service providers to understand those organizational concerns and evolve their product to address them, right?Jeremy: Right.Rob: This is one thing. My classic example of this is data exfiltration in a Lambda function. Some companies get ... I want to be able to inspect every packet that leaves, and they have that hard requirement for reasons, right?Jeremy: Right.Rob: You can't argue with them that they're right or wrong. They made that decision as a company. But then, they have to understand the impact of that is, "Okay. Well, every single Lambda function that you ever create is going to run inside of VPC or is going to run connected to a VPC." Now, you have the added complexity of managing a VPC, managing your firewall rules, your NACLs, your security groups. All of this stuff that ... Maybe you still have to do it. Maybe it really is a requirement. But if you examine your requirements from a business perspective and say, "Okay. There's another way we can address this with tightly-scoped IAM permissions that only allow me to read certain records or from certain tables, or access certain keys, or whatever." Then, we assume all that traffic goes out and that's okay. Then, you get to throw all of that complexity away and get back to delivering value more quickly. So they have to meet together, right? They have to meet.Jeremy: Right.Rob: This led to a lot of the work that AWS did with VPC networking for Lambda functions or removing the cold start because a lot ... Those enterprises weren't ready to let go of that requirement, and AWS can't tell them, "You're wrong." It's their business. It's their requirement. So AWS built that for them to reduce the cold start so that Lambda became a viable platform for them to build against.Jeremy: Right, and so if you're a developer and you're looking at some solution because ... By the way, I mean, like you said, choosing the best of breed. There are just a lot of good services out there. There are thousands and thousands of SaaS companies, and I think ... I don't know if we made this point, but I certainly consider SaaS companies themselves to be part of the cloud. I would think you would probably agree with that, right?Rob: Yeah.Jeremy: It might as well be cloud providers themselves. Most of them run on top of the cloud providers anyways, but they found ...Rob: But they don't have to, and that's interesting to me and another truth that you could be consuming services from somebody else's data center and that's still multicloud.Jeremy: Right, right. So, anyway. So my thought here or I guess the question I have is if you're a developer and you're trying to choose something best in breed, right? Whatever that is. Let's say I'm trying to send text messages, and I just think Twilio is ... It's got all the features that I need. I want to go with Twilio. If you're a developer, what are the things that you need to look for in some of these companies that maybe don't have ... I mean, I would say Twilio does, but like don't necessarily have the trust or the years of experience or I guess years under their belts of providing these services, and keeping data secure, and things like that. What's the advice that you give to developers looking to choose something like that to be aware of?Rob: To developers in particular I think is a different answer ...Jeremy: Well, I mean, yeah. Well, answer it both ways then.Rob: Yeah, because there's the builder and the buyer.Jeremy: Right.Rob: Right?Jeremy: Right.Rob: Whoever the buyer is, and a lot of times, that could just be the software development manager who's the buyer, and they still would approach it different ways. I think the developer is first going to be concerned with, "Does it solve my problem?" Right? "Overall, does it allow me to ship faster?" The next thing has to be stability. You have to expect that this company will be around, which means there is a certain level of evidence that you need of, "Okay. This company has been around and has serviced," and that's a bit of a chicken and an egg problem.Jeremy: Right.Rob: I think the developer is going to be a lot less concerned with that and more concerned with the immediacy of the problem that they're facing. The buyer, whether that's a manager, or CIO, or anywhere in between, is going to need to be concerned with other things according to their size, right? You even get the weird multicloud corner cases of, "Well, we're a major supplier of Walmart, and this tool only runs on a certain cloud service provider that they don't want us to use. So we're not going to use it." Again, that's a business decision, like would I build my software that way? No, but I'm not subject to that constraint. So that means nothing in that equation.Jeremy: Right. So you mentioned a little bit earlier this idea of bringing people in from different organizations like somebody comes in and they can pick up where somebody else left off. One of the things that I've noticed quite a bit in some of the companies that I've worked with is that they like to build custom tools. They build custom tools to solve a job, right? That's great. But as soon as Fred or Sarah leave, right, then all of a sudden, it's like, "Well, who takes over this project?" That's one of the things where I mentioned ... I said "experiments," and I said "low-risk." I think something that's probably more low-risk than building your own thing is choosing an API or a service that solves your problem because then, there's likely someone else who knows that API or that service that can come in, and can replace it, and then can have that seamless transition.And as you said, with all the turnover that's been happening lately, it's probably a good thing if you have some backup, and even if you don't necessarily have that person, if you have a custom system built in-house, there is no one that can support that. But if you have a custom ... or if you have a system you've used, you're interfacing with Twilio, or Stripe, or whatever it is, you can find a lot of developers who could come in even as consultants and continue to maintain and solve your problems for you.Rob: Yeah, and it's not just those external providers. It's the internal tooling as well.Jeremy: Right.Rob: Right?Jeremy: Right.Rob: We're guilty of this in my company. We wrote a lot of stuff. Everybody is, right, like you like to do it?Jeremy: Right.Rob: It's a problem that you recognized. It feels good to solve it. It's a quick win, and it's almost always the wrong answer. But when you get into things like ... a lot of cases it doesn't matter what specific tool you use. 10 years ago, if you had chosen Puppet, or Chef, or Ansible, it wouldn't be as important which one as the fact that you chose one of those so that you could then go out and find someone who knew it. Today, of course, we've got Pulumi, Terraform, and all these other things that you could choose from, and it's just better than writing a bunch of Bash scripts to tile the stuff together. I believe Bash should more or less be banned in the cloud, but that's another ... That's my hot take for another time. Come at me on Twitter if you don't like that one.Jeremy: So, yeah. So I think just bringing up this idea of tooling is important because the other thing that you potentially run into is with the variety of tooling that's out there, and you mentioned the original IAC. I guess they would... Right? We call those like Ansible and those sort of things, right?Rob: Right.Jeremy: All of those things, the Chefs and the Puppets. Those were great because you could have repeatable deployments and things like that. I mean, there's still work to be done there, but that was great because you made the choice to not building something yourself, right?Rob: Right.Jeremy: Something that somebody else could jump in on. But now with Terraform and with ... You mentioned Pulumi. We've got CloudFormation and even Microsoft has their own ... I think it's called ARM or something like that that is infrastructure as code. We've got the Serverless Framework. We've got SAM. We've got Begin. You've got ... or Architect, right? You've got all of these choices, and I think what happens too is that if teams don't necessarily ... If they don't rally around a tool, then you end up with a bunch of people using a bunch of different tools. Maybe not all these tools are going to be compatible, but I've seen really interesting mixes of people using Terraform, and CloudFormation, and SAM, and Serverless Framework, like binding it all together, and I think that just becomes ... I think that becomes a huge mess.Rob: It does, and I get back to my favorite quote about complexity, right? "Simplicity before complexity is worthless. Simplicity beyond complexity is priceless." I find it hard to get to one tool that's like artificial, premature optimization, fake simplicity.Jeremy: Yeah.Rob: If you force yourself into one tool all the time, then you're going to find it doing what it wasn't built to do. A good example of this, you talked about Terraform and the Serverless Framework. My opinion, they weren't great together, but your Terraform comes for your persistent infrastructure, and your Serverless Framework comes in and consumes the outputs of those Terraform stacks, but then builds the constantly churning infrastructure pieces of it. Right? There's a blast radius issue there as well where you can't take down your database, or S3 bucket, or all of this from a bad deploy when all of that is done in Terraform either by your team, or by another team, or by another process. Right? So there's a certain irreducible complexity that we get to, but you don't want to have duplication of effort with multiple tools, right?Jeremy: Right.Rob: You don't want to use CloudFormation to manage your persistent data over here and Terraform to manage your persistent data over here because then you're not ... That's like that agnostic model where you're not benefiting from the excellent features in each. You're only using whatever is common between them.Jeremy: Right, right, and I totally agree with you. I do like the idea of consuming. I mean, I have been using AWS for a very, very long time like 2007, 2008.Rob: Yeah, same. Oh, yeah.Jeremy: Right when EC2 instances were a thing. I guess 2008. But the biggest thing for me looking at using Terraform or something like that, I always felt like keeping it in, keeping it in the family. That's the wrong way to say it, but like using CloudFormation made a lot of sense, right, because I knew that CloudFormation ... or I thought I knew that CloudFormation would always support the services that needed to be built, and that was one of my big complaints about it. It was like you had this delay between ... They would release some service, and you had to either do it through the CLI or through the console. But then, CloudFormation support came months later. The problem that you have with some of that was then again other tools that were generating CloudFormation, like a Serverless Framework, that they would have to wait to get CloudFormation support before they could support it, and that would be another delay or they'd have to build something custom, which is not always the cleanest way to do it.Rob: Right.Jeremy: So anyways, I've always felt like the CloudFormation route was great if you could get to that CloudFormation, but things have happened with CDK. We didn't even mention CDK, but CDK, and Pulumi, and Terraform, and all of these other things, they've all provided these different ways to do things. But the thing that I always thought was funny was, and this is ... and maybe you have some insight into this if you can share it, but with SAM, for example, SAM wasn't extensible, right? You would just run into issues where you're like, "Oh, I can't do that with SAM." Whereas the Serverless Framework had this really great third-party plug-in system that allowed you to do some of these other things. Now, granted not all third-party plug-ins were super stable and were the best way to do something, right, because they'd either interact with APIs directly or whatever, but at least it gave you ... It unblocked you. Whereas I felt like with SAM and even CloudFormation when it didn't support something would block you.Rob: Yeah. Yeah, and those are just two different implementation philosophies from two different companies at two different stages of their existence, right? Like AWS ... and let's separate the reality from the theory here. The theory is that a large company can exert control over release cycles and limit what it delivers, but deliver it with a bar of excellence. A small company can open things up, and it depends on its community members for contributions to solve problems. It's very much like this is the cathedral and the Bazaar of cloud tooling, right?AWS has that CloudFormation architecture that they're working around with its own goals and approach, the one way to do it. Serverless Framework is, "Look, you need to ... You want to set up a stall here and insert IAM policies per function. Set up a stall. It will be great. Maybe people come and maybe they don't," and the system inherently sorts or bubbles up the value, right? So you see things like the Step Functions plug-in for Serverless Framework. It was one of the early ones that became very popular very quickly, whereas Step Functions supporting SAM trailed, but eventually came in. I think that team, by the way, deserves a lot of credit for really being focused on developers, but that's not the point of the difference between the two.A small young company like Serverless Framework that is moving very quickly can't have that cathedral approach to it, and both are valid, right? They're both just different strategies and good for the marketplace, quite frankly. I have my preferred approach, which is not about AWS or SAM vs Serverless Framework. It's the extensibility of plug-in frameworks to me are a key component of tooling that adapts as quickly as the clouds change, and you see this. Like Terraform was the first place that I really learned about plug-ins, and their plug-in framework is fantastic, the way they do providers. Serverless Framework as well is another good example, but you can't know how developers are going to build with your services. You just can't.You do customer development. You talk to them ahead of time. You get all this research. You talk to a thousand customers, and then you release it to 14 million customers, right? You're never going to guess, so let them. Let them build it, and if people ... They put the work in. People find there's value in it. Sometimes you can bring it in. Sometimes you leave it up to the community to maintain, but you just ... You have to be willing to accept that customers are going to use your product in different ways than you envisioned, and that's a good thing because it means customers are using your product.Jeremy: Right, right. Yeah. So I mean, from your perspective though ... because let's talk about SAM for a minute because I was excited when SAM came out. I was thinking to myself. I'm like, "All right. A simplified tooling that is focused on serverless. Right? Like gives me all the things that I think I'm going to need." And then I did ... from a developer experience standpoint, and let's call out the elephant in the room. AWS and developer experience are not always the same. They don't always give you that developer experience that you would want. They give you tons of tools, right, but they don't always give you that ...Rob: Funny enough, you can spell "developer experience" without AWS.Jeremy: Right. So I mean, that's my ... I was disappointed when I started using SAM, and I immediately reverted back to the Serverless Framework. Not because I thought that it wasn't good or that it wasn't well-thought-out. Like you said, there was a level of excellence there, which certainly you cannot diminish, It just didn't do the things I needed it to do, and I'm just curious if that was a consistent feedback that you got as being someone on the dev advocate team there. Was that something that you felt as well?Rob: I need to give two answers to this to be fair, to be honest. That was something that I felt as well. I never got as comfortable with SAM as I am with the Serverless Framework, but there's another side to this coin, and that's that enterprise uptake of SAM CLI has been really strong.Jeremy: Right.Rob: Enterprise it, it does what they need it to do, and it addresses their concerns, and they liked getting tooling from AWS. It just goes back to there being a place for both, right?Jeremy: Right. Rob: Enterprises are much more likely to build cathedrals. They want that top-down, "Okay, everybody. This is how you define something. In fact, we've created a module for you. Consume it here. Thou shalt not write new S3 to web server configuration in your SAM templates. Thou shalt consume this." That's not wrong, and the usage numbers don't lie with SAM. It's got a lot of fans, and it's got a lot of uptake, but that's an entirely different answer from how I feel about it. I think it also goes back to I'm not running an enterprise. I've never run an enterprise. The biggest I've got in terms of responsibility is at best a small company, right? So I think it's natural for me to feel that way when I try to use a tool that has such popularity amongst enterprise. Now, of course, you have the switch, right? You have enterprises using Serverless Framework, and you have small builders using SAM. But in general, I think the success there was with the enterprise, and it's a validation of their strategy.Jeremy: Right, right. So let's talk about enterprises for a second because this is where we look at tools like the CDK and SAM, Serverless Framework, things like that. You look at all those different tools, and like you said, there's adoption across some of those. But at the end of the day, most of those tools are compiling down to a CloudFormation or they're compiling down to ... What's it called? The Azure Resource Manager Language or whatever the heck it is, right?Rob: ARM templates.Jeremy: ARM templates. What's the value now in CloudFormation and those sort of things that the final product that you get to ... I mean, certainly, it's so much easier to build those using these frameworks, but do we need CloudFormation in those things anymore? Do we need to know those? Does an individual developer need to be able to understand those, or can they just basically take a step back and say, "Look, CDK does it for me," or, "Pulumi does it for me. So why do I need to know what's baked into those templates?"Rob: Yeah. So let's set Terraform aside and talk about it after because it's different. I think the choice of JSON and YAML as implementation languages for most of this tooling, most of this tooling is very ... It was a very effective choice because you don't necessarily have to know CloudFormation to look at a template and define what it's doing.Jeremy: Right.Rob: Right? You don't have to understand transforms. You don't have to understand parameter replacement and all of this stuff to look at the final transformed template in CloudFormation in the console and get a very quick reasoning about what's happening. That's good. Do I think there's value in learning to create multi-thousand-line CloudFormation templates by hand? I don't. It's the assembly language of the cloud, right?Jeremy: Right.Rob: It's there when you need it, and just like with procedural languages, you might want to look underneath at the instructions, how it unrolled certain loops, how it decided not to unroll others so that you can make changes at the next level. But I think that's rare, and that's optimization. In terms of getting things done and getting things shipped and delivered, to start, I wouldn't start with plain CloudFormation for any of these, especially of anything for any meaningful production size. That's not a criticism of CloudFormation. It's just like you said. It's all these other tooling is there to help us generate what we want consistently.The other benefit of it is once you have that underlying lingua franca of the cloud, you can build visualization, and debugging, and monitoring, and like ... I mean, all of these other evaluatory forensic. "Evaluatory?" Is that a word? It's a word now. You heard it here first on this podcast. Like forensic, cloud forensic type tooling that lets you see what's going on because it is a universal language among all of the tools.Jeremy: Yeah, and I want to get back to Terraform because I know you mentioned that, but I also want to be clear. I don't suggest you write CloudFormation. I think it is horribly verbose, but probably needs to be, right?Rob: Yeah.Jeremy: It probably needs to have that level of fidelity there or that just has all that descriptive information. Yeah. I would not suggest. I'm with you, don't suggest that people choose that as their way to go. I'm just wondering if it's one of those things where we don't need to be able to look at ones and zeroes anymore, and understand what they do, right?Rob: Right.Jeremy: We've got higher-level constructs that do that for us. I wouldn't quite put ... I get the assembly language comparison. I think that's a good comparison, but it's just that if you are an enterprise, right, is that ... Do you trust? Do you trust something like CDK to do everything you need it to do and make sure that it's covering all its bases because, again, you're writing layers of abstraction on top of a layer of abstraction that then abstracts it even more. So yeah, I'm just wondering. You had mentioned forensic tools. I think there's value there in being able to understand exactly what gets put into the cloud.Rob: Yeah, and I'd agree with that. It takes 15 seconds to run into the limit of my knowledge on CDK, by the way. But that knowledge includes the fact that CDK Synth is there, which generates the CloudFormation template for you, and that's actually the thing, which is uploaded to the CloudFormation service and executed. You'd have to bring in somebody like Richard Boyd or someone to talk about the guard rails that are there around it. I know they exist. I don't know what they are. It's wildly popular, and adoption is through the roof. So again, whether I philosophically think it's a good idea or not is irrelevant. Developers want it and want to build with it, right?Jeremy: Yeah.Rob: It's a Bazaar-type tool where they give you some basic constructs, and you can write your own constructs around it and get whatever you need. But ultimately, that comes back to CloudFormation, which is then subject to all the controls that your organization puts around CloudFormation, so it is ... There's value there. It can't be denied.Jeremy: Right. No, and the thing that I like about the CDK is the idea of being able to create those constructs because I think, especially from a ... What's the right word? Compliance standpoint or something like that, that you can write in these constructs that you say, "You need to use these constructs when you deploy a microservice or you deploy this," or whatever it is, and then you have those guard rails as you mentioned or whatever, but those ... All of those checkboxes are ticked because you can put that all into one construct. So I totally think that's great. All right. So let's talk about Terraform.Rob: Yeah, so there's ... First, it's a completely different model, right?Jeremy: Right.Rob: This is an interesting discussion to have because it's API calls. You write your provider, whatever your infrastructure is, and anything that can be an API call can now be a Terraform declarative resource. So it's that mapping between declarative and imperative that I find fascinating. Well, also building the dependency graph for you. So it handles all of those aspects of it, which is a really powerful tool. The thing that they did so well ... Terraform is equally verbose as CloudFormation. You've got to configure all the same options. You get the same defaults, et cetera. It can be terribly verbose, but it's modular. Every Terraform file that you have in one directory is concatenated, and that is a huge distinction between how CloudFormation wants everything in one template, or well, you can refer to something in an S3 bucket, but that's not actually useful to me as a developer.Jeremy: Right, right.Rob: I can't mount an S3 bucket as a drive on my workstation and compose all of these independent files at once and do them that way. Sidebar here. Maybe I can. Maybe it supports that, and I haven't been able to discover it, right? Whereas Terraform by default, out of the box put everything in its own file according to function. It's very easy to look in your databases.tf and understand what's in there, look in your VPC.tf and understand what's in there, and not have to go through thousands of lines of code at once. Yes, we have find and replace. Yes, we have search, and you ... Anybody who's ever built any of this stuff knows that's not the same thing. It's not the same thing as being able to open a hundred lines in your text editor, and look at everything all at once, and gain an understanding of it, and then dive into the next level of detail a hundred lines at a time and understand that.Jeremy: Right. But now, just a question here, without... because the API thing, I love that idea, and actually, Serverless Components used an API thing to do it and bypass CloudFormation. Actually, I believe Architect originally was using APIs, and then switched to CloudFormation, but the question I have about that is, if you don't have a CloudFormation template, if you don't have that assembly language of the web, and that's not sitting in your CloudFormation tool built into the dashboard, you don't get the drift protection, right, or the detection, and you don't get ... You don't have that resource map necessarily up there, right?Rob: First, I don't think CloudFormation is the assembly language of the web. I think it's the assembly language of AWS.Jeremy: I'm sorry. Right. Yes. Yeah. Yeah.Rob: That leads into my point here, which is, "Okay. AWS gives you the CloudFormation dashboard, but what if you're now consuming things from Datadog, or from Fauna, or from other places that don't map this the same way?Jeremy: Right.Rob: Terraform actually does manage that. You can do a plan against your existing file, and it will go out and check the actual existing state of all of your resources, and compare them to what you've asked for declaratively, and show you what the changeset will be. If it's zero, there's no drift. If there is something, then there's either drift or you've added new functionality. Now, with Terraform Cloud, which I've only used at a basic level, I'm not sure how automatic that is or whether it provides that for you. If you're from HashiCorp and listening to this, I would love to learn more. Get in touch with me. Please tell me. But the tooling is there to do that, but it's there to do it across anything that can be treated as an API that has ... really just create and retrieve. You don't even necessarily need the update and delete functionality there.Jeremy: Right, right. Yeah, and I certainly ... I am a fan of Terraform and all of these services that make it easier for you to build clouds more easily, but let's talk about APIs because you mentioned anything with an API. Well, everything has APIs now, right? I mean, essentially, we are living in a ... I mentioned SaaS, but now we're sort of ... This whole idea of the API economy. So just overall, what are your thoughts on this idea of basically anything you want to do is available now via an API?Rob: It's not anything you want to do. It's everything you want to do short of fulfilling your customer's needs, short of solving your customer's specific problem is available as an API. That means that you get to choose best-in-class for everything, right? Your customer's need isn't, "I want to spend $25 on my credit card." Your customer's need is, "I need a book."Jeremy: Right.Rob: So it's not, "I want to store information about books in a database." It's, "I need a book." So everything and every step of the way there can now be consumed from an API. Again, it's like serverless in general, right? It allows you to focus purely or as close to purely as we can right now on generating customer value and addressing customer problems so that you can ship faster, and so that you have it as a competitive advantage. I can write a payment processing program. I know I can because I've done it back in 2004, and it was horrible, and it was awful. It wasn't a very good one, and it worked. It took your money, but this was like pre-PCIDSS.If I had to comply with all of those things, why would I do that? I'm not a credit card payment processor. Stripe is, and they have specialists in all of the areas related to the problem of, "I need to take and process payments." That's the customer problem that they're solving. The specialization of labor that comes along with the API economy is fantastic. Ops never went away. All the ops people work at the cloud service providers now.Jeremy: Right.Rob: Right? Audit never went away. All the auditors have disappeared from view and gone into internal roles in payments companies. All of this continues to happen where the specialists are taking their deep, deep knowledge and bringing it inside companies that specialize in that domain.Jeremy: Right, and I think the domain expertise value that you get from whatever it is, whether it's running a database company or whether it's running a payment company, the number of people that you would need to hire to have a level of specialization for what you're paying for two cents per transaction or whatever, $50 a month for some service, you couldn't even begin. The total cost of ownership on those things are ... It's not even a conversation you would want to have, but I also built a payment processing system, and I did have to pass PCI, which we did pass, but it was ...Rob: Oh, good for you.Jeremy: Let's put it this way. It was for a customer, and we lost money on that customer because we had to go through PCI compliance, but it was good. It was a good experience to have, and it's a good experience to have because now I know I never want to do it again.Rob: Yeah, yeah. Back to my earlier point on Ops and serverless.Jeremy: Right. Exactly.Rob: These things are hard, right?Jeremy: Right, right.Rob: Sorry.Jeremy: No, no. Go ahead.Rob: Not to interrupt, but these are all really hard problems that people with graduate degrees and post-graduate research who have ... They're 30 years old when they start working on the problem are solving. There's a supply question there as well, right? There's just not enough people, and so you and I can like ... Well, I'm not going to project this on to you. I can stumble through an implementation and check off the requirements just like I worked in an optical microscopy lab in college, and I could create computer programs that modeled those concepts, but I was not an optical microscopist. I was not a PhD-level generating understandings of these things, and all of these, they're just so hard. Why would you do that when customer problems are equally hard and this set is solved?Jeremy: Right, right.Rob: This set of problems over here is solved, and you can't differentiate yourself by solving it better, and you're not likely to solve it better either. But even if you did, it wouldn't matter. This set of problems are completely unsolved. Why not just assemble the pieces from best-in-class so that you can attack those problems yourself?Jeremy: Again, I think that makes a ton of sense. So speaking about expertise, let's talk about what you might have to pay say a database administrator. If you were to hire a database administrator to maintain all the databases for you and keep all that uptime, and maybe you have to hire six database administrators in order for them to ... Well, I'm thinking multi-region and all that kind of stuff. Maybe you have to hire a hundred, depending on what it is. I mean, I'm getting a little ahead of myself, but ... So if I can buy a service like Fauna, so tell me a little bit more about just how that works.Rob: Right. Well, I mean, six database engineers in the US, you're over a million dollars a year easily, right?Jeremy: Right.Rob: I don't know what the exact number is, but when you consider benefits, and total cost, and all of that, it's a million dollars a year for six database engineers. Then, there are some very difficult problems in especially distributed databases and database scaling that Fauna solves. A number of other products or services solve some of them. I'm biased, of course, but I happen to think Fauna solves all of them in a way that no other product does, but you're looking ... You mentioned distributed transactions. Fauna is built atop the Calvin paper, which came out of Yale. It's a very brief, but dense academic research paper. It's a PHC research paper, and it talks about a model for distributed transactions and databases. It's a layer, a serialization layer, that sits atop your database.So let's say you wanted to replicate something like Fauna. So not only do you need to get six database engineers who understand the underlying database, but you need to find engineers that understand this paper, understand the limitations of the theory in the paper, and how to overcome them in operations. In reality, what happens when you actually start running regions around the world, replicating transactions between those regions? Quite frankly, there's a level of sophistication there that most of the set of people who satisfy that criteria already work at Fauna. So there's not much of a supply there. Now, there are other database competitors that solve this problem in different ways, and most of the specialists already work at those companies as well, right? I'm not saying that they aren't equally competent database engineers. I'm just saying there's not a lot of them.Jeremy: Right.Rob: So if you're thing is to sell books at a certain scale like Amazon, then that makes sense to have them because you are a database creator as well. But if your thing is to sell books at some level below that, why would you compete for that talent rather than just consuming it?Jeremy: Right. Yeah, and I would say unless you're a horse with a horn on your head, it's probably not worth maintaining your own database and things like that. So let's talk a little bit more, though, about that. I guess just this idea of maybe a shortage of people, like if you're ... You're right. There's a limited number of resources, right? I'm sure there's brilliant database engineers all around the world, and they have the experience where, right, they could come in and they could probably really help you maintain your database. Even if you could afford six of them and you wanted to do that, I think the problem is it's got to be the interestingness of the problem. I don't think "interestingness" is a word either, but like if I'm a database engineer, wouldn't I want to be working on something like Fauna that I could help millions and millions of people as opposed to helping some trucking company maintain their internal database or something like that?Rob: Yeah, and I would hope so. I hope it's okay that I mention we're hiring. So come to Fauna.com and look at our roles database engineers.Jeremy: You just read that Calvin paper first. Go ahead.Rob: But read the Calvin paper first. I think it's only like 12 pages, and even just the first page is enough. I'm happy to talk about that at any length because I find it fascinating and it's public. It is an interesting problem and the ... It's the reification or the implementation of theory. It's bringing that theory to the real world and ... Okay. First off, the theory is brilliant. This is not to take away from it, but the theory is conceived inside someone's mind. They do some tests, they prove it, and there's a world of difference between that point, which is foundational, and deploying it to production where people are trusting their workloads on it around the world. You're actually replicating across multiple cloud providers, and you're actually replicating across multiple regions, and you're actually delivering on the promise of the paper.What's described in the paper is not what we run at Fauna other than as a kernel, as a nugget, right, as the starting point or the first principle. That I think is wildly interesting for that class of talent like you talked about, the really world-class engineers who want to do something that can't be done anywhere else. I think one thing that Fauna did smartly early was be a remote-first company, which means that they can take advantage of those world-class engineers and their thirst for innovation regardless of wherever Fauna finds them. So that's a big deal, right? Would you rather work on a world-class or global problem or would you rather work on a local problem? Now, look, we need people working on local problems too. It's not to disparage that, but if this is your wheelhouse, if innovation is the thing that you want to do, if you want to be doing things with databases that nobody else is doing, this is where you want to be. So I think there's a strong argument there for coming to work in a place like Fauna.Jeremy: Yeah, and I want to make sure I apologize to any database engineer working at a trucking company because I'm sure there are actually probably really interesting problems with logistics and things like that that they are solving. So maybe not the best example. Maybe. I don't know. I can't think of another example. I don't want to offend anybody who's chosen a more local problem because you're right. I mean, there are local problems that need to be solved, but I do think that there are people ... I mean, even someone like me. I want to work on a bigger problem. You know what I mean? I owned a web development company for 12 years, and I was solving other people's problems by building them a website or whatever, and it just got to a point where I'm like, "I'm not making enough of an impact here." You're not solving a big enough problem. You want to work on something more interesting.Rob: Yeah. Humans crave challenge, right? Challenge is a necessary precondition for growth, and at least most of us, we want to grow. We want to be better at whatever it is we're doing or just however we think of ourselves next year that we aren't today, and you can't do that with challenge. If you build other people's websites for 12 years, eventually, you get to a point where maybe you're too good at it. Maybe that's great from a business perspective, but it's not so great from a personal fulfillment perspective.Jeremy: Right.Rob: If it's, "Oh, look, another brochure website. Okay. Here you go. Oh, you need a contact form?" Again, it's not to disparage this. It's the fact that if you do anything for 12 years, sometimes mastery is stasis. Not always.Jeremy: Right, and I have nightmares of contact forms, of building contact forms, by the way, but...Rob: It makes sense. Yeah. You know what you should do is just put all of those directly into Fauna and don't worry about it.Jeremy: Easy enough. Easy enough.Rob: Yeah, but it's not necessarily stasis, but I think about craftsmen and people who actually make things with their hands, physical builders, and I think a lot of that ... Like if you're making furniture, you're a cabinet maker. I think a lot of that is every time, it's just a little bit wrong, right? Not wrong, but just a little bit off from your optimum no matter how long you do it, and so everything has a chance to evolve. That's there with software to a certain extent, but the problem is never changing.Jeremy: Right.Rob: So, yeah. I can see both sides of it, but for me, I ... You can see it when I was on serverless four years ago and now that I'm on a serverless database now. I like to be out at the edge, pushing that edge out for everyone who's coming behind. It can be challenging because sometimes there's just no way forward, and sometimes everybody is not ready to come with you. In a lot of ways, being early is the same as being wrong.Jeremy: Right. Well, I've been ...Rob: Not an original statement, but ...Jeremy: No, but I've been early on many things as well where like five years after we tried to do something, like then, all of a sudden, it was like this magical thing where everybody is doing it, but you mentioned the edge. That would be something ... or you said on the edge. I know you mean this way, but the edge in terms of the actual edge. That's going to be an interesting data problem to solve.Rob: Oh, that's a fascinating data problem, especially for us at Fauna. Yeah, compute, and Andy Jassy, when he was at AWS, talked about how compute was bifurcating, right? It's either moving all the way out to the edge or it's moving all the way into the cloud, and that's true. But I think at Fauna, we take that a step further, right? The edge part is true, and a lot of the work that we've done recently, announcements with CloudFlare workers. We're ready for that. We believe in it, and we like pushing that out as close to the user as possible. One thing that we do uniquely is we have this concept of user-defined functions, and anybody who's written T-SQL back in the day, who wrote store procedures is going to be familiar with this, but it's ... You bring that business logic and that code to your data. Not near your data, to your data.Jeremy: Right.Rob: So you bring the compute not just to the cloud where it still needs to pass through top-of-rack and all of this. You bring it literally on to the same instance as your data where these functions execute against them there. So you get not just the database, but you get a compute layer in there, and this helps for things like filtering for things like the equivalent of joins, stuff that just ... If you've got to load gigabytes of data and move it somewhere, compute against it, reduce it to something, and then store that back, the speed of light still matters. Even if it's the speed of light across a couple switches, it still matters, and so there are some really interesting things that you can begin to do as you pull more and more of that logic into your data layer, and you also protect that logic from other components of your application.So I like that because things like GraphQL that endpoints already speak and already understand, just send it over, and again, they don't care about the architectural, quite frankly, genius--I can say that because I didn't create it--the genius behind all of this stuff. They just care that, "Look, I send this request over and I get it back," and entire workflows, and complex processes, and everything are executing behind the scenes just so that the endpoints can send and retrieve what they need more effectively and more quickly. The edge is fascinating. The thing I regret the most about the edge is I have no hardware skills, right? So I can't make fun things to do fun things in my house. I have to buy them, but you can't do everything.Jeremy: Yeah. Well, no. I think you make a good point though about bringing the compute to the data side, and other people have said there's no ... Ben Kehoe has been talking about this for a while too where like it just makes sense. Run the compute where the data is, and then send that data somewhere else, right, because there's more things that can be done with data after that initial bit of compute. But certainly, like you said, filtering it down or getting the bits that are relevant and moving a small amount of data as opposed to a large amount of data I think is hugely important.Now, the other thing I just want to mention before I let you go or I want to talk about quickly is this idea of going back to the API economy aspect of things and buying versus building. If you think about what you've had to do at Fauna, and I know you're relatively recent there, but you know what they've done and the work that had to go in in order to build this distributed system. I mean, I think about most systems now, and I think like anything I'm going to build now, I got to think about scale, right?I don't necessarily have to build to scale right away, especially if I'm doing an MVP or something like that. But if I was going to build a service that did something, I need to think about multi-region, and I need to think about failover, and I need to think about potentially providing it at the edge, and all these other things. So you come down to this thing, and I will just use the database example. But even if you were say using like MySQL, or Postgres, or something like that, that's going to scale. That's going to scale pretty well to get to a certain point, and then you're going to have to start sharding, right? When data gets hard, it's time to shard, right? You just have to start sharding everything.Rob: Yeah.Jeremy: Essentially, what you end up doing is rebuilding DynamoDB, or trying to rebuild Fauna, or something like that. So just thinking about that, anything you're building ... Maybe you have some advice for developers who ... I know we've talked about this a little bit, but I just go back to this idea of like, if you think about how complex some of these SaaS companies and these services that are being built out right now, why would you ever want to take that complexity on yourself?Rob: Pride. Hubris. I mean, the correct answer is you wouldn't. You shouldn't.Jeremy: People do.Rob: Yeah, they do. I would beg and plead with them like, "Look, we did take a lot of that on. Fauna scales. You don't need to plan for sharding. You don't need to plan for global replication. All of these things are happening." I raise that as an example of understanding the customer's problem. The customer didn't want to think about, "Okay, past a thousand TPS, I need to create a new read replica. Past a million TPS, I need to have another region with active-active." The customer wanted to store some data and get that data, knowing that they had the ASA guarantees around it, right, and that's what the customer has.So get that good understanding of what your customer really wants. If you can buy that, then you don't have a product yet. This is even out of software development and into product ideation at startups, right? If you can go ... Your customer's problem isn't they can't send text messages programmatically. They can do that through Twilio. They can do that through Amazon. They can do that through a number of different services, right? Your customer's problem is something else. So really get a good understanding of it. This is where knowing a little ... Like Joe Emison loves to rage against senior developers for knowing not quite enough. This is where knowing like, "Oh, yeah, Postgres. You can just shard it." "Just," the worst word in computer science, right?Jeremy: Right.Rob: "You can just shard it." Okay. Now, we're back to those database engineers that you talked about, and your customer doesn't want to shard a database. Your customer wants to store and retrieve data.Jeremy: Right.Rob: So any time that you can really focus in, and I guess I really got this one, this customer obsession beaten into me from my time at AWS. Really focus in on what the customer is asking you to do or what the customer needs even if they don't know how to express it, and build for that.Jeremy: Right, right. Like the saying. I forgot who said it. Somebody from Harvard Business Review, but, "Your customers don't want a quarter-inch drill. They want a quarter-inch hole."Rob: Right, right.Jeremy: That's basically true. I mean, the complexity that goes behind the scenes are something that I think a vast majority of customers don't necessarily want, and you're right. If you focus on that product ideation thing, I think that's a big piece of that. All right. Well, anyway. So I have one more question for you just to ...Rob: Please.Jeremy: We've been talking for a while here. Hopefully, we haven't been boring people to death with our talk about APIs and stuff like that, but I would like to get a little bit academic here and go into that Calvin paper just a tiny bit because I think most people probably will not want to read it. Not because they don't want to, but because people are busy, right, and so they're listening to the podcast.Rob: Yeah.Jeremy: Just give us a quick summary of this because I think this is actually really fascinating and has benefits beyond just I think solving data problems.Rob: Yeah. So what I would say first. I actually have this paper on my desk at all times. I would say read Section 1. It's one page, front and back. So if you're interested in it, you don't have to read the whole paper. Read that, and then listeners to this podcast will probably understand when I say this. Previously, for distributed databases and distributed transactions, you had what was called a two-phase commit. The first was you'd go out to all of your replicas, and you say, "Hey, I need lock." When everybody comes back, and acknowledges, and says, "Okay. You have the lock," then you do your transaction, and then you replicate it, and then you say, "Hey, everybody. I'm done. Release the lock." So it's a two-phase commit. If something went wrong, you rolled it all the way back and you said, "Hey, everybody. Forget it."Calvin is event-sourcing for databases. So if I could distill the entire paper down into one concept, that's it. Right? Instead of saying, "Hey, everybody. Give me a lock, I'm going to do something," you say, "Hey, everybody. Here's what we're going to do." It's a deterministic application of the transaction so that you can ... You both create the lock and execute the transaction at the same time. So rather than having this outbound round trip, and then doing the thing in an outbound round trip, you have the outbound round trip, and you're done.They all apply it independently, and then it gets into how you structure the guarantees around that, which again is very similar to event-sourcing in that you use snapshotting or checkpointing. "So, hey. At this point, we all agree. So we can forget all of our old transactions, and we roll forward from here." If somebody leaves the cluster, they come back in at that checkpoint. They reapply all of the events that occurred prior to that. Those events are transactions. They'd get up to working speed, and then off they go. The paper I think uses Paxos. That's an implementation detail, but the really interesting thing about it is you're not having a double round trip.Jeremy: Yeah.Rob: Again, I love the idea of event-sourcing. I think Amazon EventBridge is the best service that they've released in the past couple years.Jeremy: Totally.Rob: If you understand all of that and are already building serverless applications that way, well, that's what we're doing, just event-sourcing for database. That's it. That's it.Jeremy: Just event-sourcing. It's easy. Simple. All right. All the words you never want to hear. Simple, easy, just. Right. Yeah. Perfect.Rob: Yeah, but we do the hard work, so you don't have to. You don't care about all of that. You want to write your data somewhere, and you want to retrieve your data from an API, and that's what Fauna gives you.Jeremy: Which I think is the main point here. So awesome. All right. Well, Rob, listen. This was great, and I'm super happy that I finally got you on the show. Congratulations for the new role at Fauna and on what's happening over there because it is pretty exciting.Rob: Thank you.Jeremy: I love companies that are innovating, right? It's not just another hosted database. You're actually building something here that is innovative, which is pretty amazing. So if people want to find out more about you, follow you on Twitter, or find out more about Fauna, how do they do that?Rob: Right. Twitter, rts_rob. Probably the easiest way, go to my website, robsutter.com, and you will link to me from there. From there, of course, you'll get to fauna.com and all of our resources there. Always open to answer questions on Twitter. Yeah. Oh, rob@fauna.com. If you're old-school like me and you prefer the email, there you go.Jeremy: All right. Awesome. Well, I will get all that into the show notes. Thanks again, Rob.Rob: Thank you, Jeremy. Thanks for having me.

amazon personal phd government simple microsoft army chefs run humans mvp walmart cloud dubai congratulations yale i am saas developers architects cto simplicity compliance arm salesforce api audit cio puppets harvard business review aws bash apis faa consume stripe azure bazaar amazon web services ops fauna s3 node big four hubris cloudflare twilio sutter lambda tps pci baas sidebar serverless json mysql iac terraform cli graphql multicloud datadog hashicorp andy jassy ansible postgres cdk paxos ec2 yaml pci dss cloud strategy vpc dynamodb phc cloudformation rob it richard boyd serverless framework rob you t sql jeremy it jeremy you cbt nuggets ben kehoe jeremy so rob so

Episódio #13 - Serverless no SeniorLabs

Sem Servidor

Play Episode Listen Later Feb 16, 2021 53:31

Neste episódio conversamos com os pesquisadores Leonardo Fiedler e Flavio Losada, que representaram o time do SeniorLabs, que é o laboratório de pesquisa aplicada da Senior Sistemas. Nesse papo, falamos sobre como é utilizar serverless no time de pesquisa, quais projetos já utilizaram e também quais serviços foram envolvidos. Abaixo algumas referências do que foi citado no episódio: SeniorLabs: https://www.senior.com.br/noticias/seniorlabs Cesta de Compras Inteligente: https://www.linkedin.com/feed/update/urn:li:activity:6737354823051575296/ Arquiteturas de referência serverless: https://www.jeremydaly.com/serverless-reference-architectures/ Episódio que fala sobre plugin do Serverless Framework: https://open.spotify.com/episode/795F0tcr8NEWWNcRf8hKbV?si=YE0xNcYnTrCQomNcOese4A Episódio sobre serverless da Taverna da Programação: https://open.spotify.com/episode/4Rtrd8h2hj4lWDVa4cVjIJ?si=eliB9tHsSh28HHARa1TiIA

1.1 - Infrastructure as Code with Pulumi

Purrfect.dev

Play Episode Listen Later Nov 11, 2020 46:16

Today we are going to talk with Lee Briggs from Pulumi. Pulumi allows you to create modern Infrastructure as Code, using any cloud using familiar programming languages and tools. The BEST part is that Pulumi is 100% open source. Guest Details Lee Briggs Links https://www.linkedin.com/in/briggsl/ General Questions What is Infrastructure as Code (IaC)? Why would you want to adopt IaC? Who made changes to the code Ticket that was linked Collaboration Documentation Does IaC document your infrastructure? Does IaC allow for solid Dev Ops? Pulumi What is Pulumi? What makes Pulumi different than Serverless Framework - https://www.pulumi.com/docs/intro/vs/serverless/ Cloud Formation - https://www.pulumi.com/docs/intro/vs/cloud_templates/ Terraform - https://www.pulumi.com/docs/intro/vs/terraform/ Arm Will IaC take longer? Should it be used for an MVP? How large of a team would you have before using IaC? Is it worth the time to set up if it is just something quick. Is Pulumi good for single code shops that know only a single language? Speed to market is easier. No need to learn markup language How big is the learning curve of IaC? Serverless is YAML Pulumi support “ecosystems” NodeJS Go .Net Purrfect Picks Lee Great examples https://github.com/pulumi/examples https://www.pulumi.com/resources/introduction-to-pulumi?utm_source=codingcat Queen’s Gambit Alex FUN Stuff https://reinvent.awsevents.com/ --- This episode is sponsored by · Anchor: The easiest way to make a podcast. https://anchor.fm/app --- Send in a voice message: https://anchor.fm/purrfect-dev/message

code mvp speed ticket infrastructure terraform serverless framework

Episode #67: The Story of the Serverless Framework with Austen Collins (PART 2)

cloud aws faa austen lambda baas serverless serverless framework

Play Episode Listen Later Sep 21, 2020 42:50

In part 2, Jeremy finishes his chat with Austen Collins about the origins of the Serverless Framework, how it was able to grow a passionate developer community, build a company around it, and where the framework and serverless are headed in the future.

Episode #66: The Story of the Serverless Framework with Austen Collins (PART 1)

cloud aws faa austen lambda baas serverless serverless framework

Play Episode Listen Later Sep 14, 2020 51:10

In this two part episode, Jeremy chats with Austen Collins about the origins of the Serverless Framework, how it was able to grow a passionate developer community, build a company around it, and where the framework and serverless are headed in the future.

Moving to a Cloud Native Architecture with Tolga Tarhan

CTO Connection

Play Episode Listen Later Apr 16, 2020 33:19

Tolga Tarhan got his start single-handedly running all the technology in his high school. Fast forward to today in which his experience during the dot com bust and incremental steps at various development firms have led him to the belief that the cloud is the ultimate goal of any organization.On today’s episode of CTO Connection, Tolga shares wisdom and advice on how to find the right mechanisms for your business so you can gradually shift to a cloud native architecture. For Tolga, the key is not “lift and shift,” but making a gradual shift that will enable your company to innovate quickly.[01:26] - How Tolga became a technologist and engineering leader[04:01] - Specializing in AWS at Rackspace[05:55] - Getting started adopting the cloud[09:39] - Dealing with problems of scale[11:11] - Bringing professional software development to the serverless world[13:28] - Experimenting with Serverless Framework[20:45] - What are microservices?[22:41] - When to consider more than one service[26:58] - Talking to clients about resilience[28:34] - Observability requirements[29:40] - Becoming cloud nativeSpecial thanks to our global partner – Amazon Web Services(AWS). AWS offers a broad set of global cloud-based products to equip technology leaders to build better and more powerful solutions, reach out to aws-cto-program@amazon.com if you’re interested to learn more about their offerings.CTO Connection is where you can learn from the experiences of successful engineering leaders at fast-growth tech startups. Whether you want to learn more about hiring, motivating or managing an engineering team, if you're technical and manage engineers, the CTO Connection podcast is a great resource for learning from your peers!If you'd like to receive new episodes as they're published, please subscribe to CTO Connection in Apple Podcasts, Google Podcasts, Spotify or wherever you get your podcasts. If you enjoyed this episode, please consider leaving a review in Apple Podcasts. It really helps others find the show.This podcast episode was produced by Dante32.

spotify moving google podcasts architecture aws specializing experimenting amazon web services cloud native observability rackspace tolga serverless framework

4: #4 - James Hall, AWS Hero and Founder/Director at Parallax

One Zero One

Play Episode Listen Later Jul 30, 2019 22:00

For Episode #4 of the One Zero One podcast, Jamie Linford from Version 1 took a deep-dive into the rising popularity of Serverless, discussing why it is being embraced by businesses worldwide with James Hall, 'AWS Hero' and the Founder/Director of the Digital agency Parallax. James Hall has been working in the digital sector for over a decade. He is the author of the popular js PDF library, and is a founder/Director of Parallax, a digital agency in the UK. James has contributed to and promotes the Serverless Framework (https://serverless.com/) which allows you to build web applications on top of Lambda and related services. Notably, James built an online recording studio for David Guetta and UEFA using Serverless technology shortly after API Gateway was released, which won him the title of an 'AWS Hero' from Amazon Web Services (https://www.version1.com/about-us/our-technology-partnerships/amazon-web-services/) themselves. As he discusses with Jamie in this episode - he didn't receive a cape for this accolade, but he was happy with the news all the same! Listen or download this episode to hear more from James Hall regarding Serverless, why he was asked to call up NASA, and his work on a wide variety of projects, from LED Billboards, car unlocking apps, to large web applications and tools.

director digital united kingdom hero nasa uefa notably david guetta amazon web services lambda founder director serverless parallax james hall api gateway serverless framework

Ep84: The Serverless Framework, with Nick Gottlieb

Cloud Unfiltered

Play Episode Listen Later Jun 28, 2019 34:27

The Serverless Framework open source project had 150K downloads in 2016, 3.9 million in 2018, and 3.7 million so far this year. Something must be going on. We talk about it with Nick Gottlieb on this week’s episode.

gottlieb 150k serverless framework

#4 Desmistificando Serverless com Igor Halfeld e Lucas Santos

O Universo da Programação

Play Episode Listen Later Jun 21, 2019 55:19

# Convidados: Lucas Santos: https://twitter.com/_StaticVoid Igor Halfeld: https://twitter.com/igorhalfeld # Links citados: Serverless Framework: https://serverless.com/acceleration/ https://github.com/igorhalfeld/crediup-api https://slsweek.netlify.com/ https://abcdevelopers.org/ --- Contribua Faça parte do grupo no Telegram Apoie minha criação de artigos Apoie minha produção de vídeos Proponha pautas Tenha super poderes Se você está querendo entrar na área de programação, confere meu livro: O Universo da Programação

igor serverless o universo desmistificando serverless framework lucas santos

Episode 64: Serverless Runs on Serverless Framework with Austen Collins

Play Episode Listen Later Jun 12, 2019 40:07

Serverless deployment is picking up steam in the industry, and Austen Collins has been leading the charge since 2015. In this episode, Collins talks about his work with AWS, building the Serverless Framework, and why it’s solving so many problems.

amazon cloud runs aws devops austen serverless serverless framework last week in aws

160: Serverless PHP using Bref with Matthieu Napoli and Neal Brooks

Three Devs and a Maybe

Play Episode Listen Later Feb 23, 2019 51:21

In this weeks episode we are lucky to have both Matthieu Napoli and Neal Brooks on the show to discuss all things Serverless PHP. We start off by discussing what drew Matthieu to Serverless, the creation of the Bref project and the technical challenges encountered with getting PHP to work within the Lambda environment. From here, we touch upon the reasons behind moving from the Serverless Framework to SAM (for the 0.3 release) and how Bref uses the new Lambda Layers and Runtime API. This leads us on to highlight how a typical PHP project would use Bref, the decision to be opinionated in order to stay minimal and the experimental Loop SAPI. Finally, we envision what the future holds for the Bref project and Serverless compute.

napoli bref matthieu php lambda serverless serverless framework lambda layers

RR 399: Jets Ruby Serverless Framework with Tung Nguyen

jets nguyen tung sentry serverless framework

Play Episode Listen Later Feb 12, 2019 73:48

Sponsors Sentry use the code "devchat" for $100 credit Panel Andrew Mason Eric Berry Dave Kimura Charles Max Wood Nate Hopkins Special Guest: Tung Nguyen Episode Summary In this episode of Ruby Rogues, the panelists talk with Tung Nguyen, President and Founder of BoltOps AWS Cloud Infrastructure Consultancy, a Bay Area based DevOps infrastructure consultancy. Tung is also the creator of Ruby on Jets. Jets is a Ruby Serverless Framework, allowing you to to create serverless applications with Ruby. It includes everything needed to build and deploy applications to AWS Lambda. Tung explains how Jets works and that even before AWS Lambda supported Ruby, Jets used a shim to run Ruby. The shim was written in a language that is natively supported by AWS Lambda and called out to Ruby. Tung describes this process using the dream in dream concept in the movie Inception. Since AWS Lambda has started supporting Ruby, Jets has since moved to the official AWS version of Ruby. They discuss Tung’s decision to open source Jets and his end goal with it. Tung explains he created Jets because he needed it, he wanted to run Ruby functions without managing a server. So by building tools like Jets he is able to help his clients and his consulting company. By open sourcing them, he is able to give back to the community. Tung talks about the development process of Jets and explains that he has already re-written Jets a couple of times. Finally, for people who want to find out more about Jets, Tung directs them to the documentation and support links on the Jets website and the YouTube videos he has posted. Links Jets Ruby Serverless Framework Jets Blog Post http://rubyonjets.com/docs/crud-html-activerecord/ http://rubyonjets.com/docs/crud-json-activerecord/ https://asyncy.com/ BoltOps BoltOps Nuts and Bolts Blog AWS Lambda Serverless Framework Tung's LinkedIn Tung's GitHub Tung's Twitter Tung's YouTube Channel Support Jets Introducing Jets: A Ruby Serverless Framework on AWS Lambda Build an API with Jets Ruby Serverless Framework Picks Nate Hopkins: Influence: The Psychology of Persuasion Eric Berry: https://asyncy.com/ https://scoutapp.com Light Therapy Lamp Andrew Mason: H68G Drone Dave Kimura: Microsoft Sculpt Ergonomic Keyboard for Business (5KV-00001) Inversion Table Charles Max Wood: The 1-Page Marketing Plan bu Allan Dib Cholesterol Clarity by Jimmy Moore The Keto Reset Diet by Mark Sisson Deseret Book Company Tung Nguyen: Profit First by Mike Michalowicz

founders president influence panel jets bay area api persuasion inception aws nguyen github devops profit first mike michalowicz tung sentry mark sisson influence the psychology aws lambda jimmy moore eric berry andrew mason page marketing plan allan dib keto reset diet cholesterol clarity charles max wood ruby rogues serverless framework nate hopkins deseret book company cholesterol clarity what wrong numbers

Sponsors Sentry use the code "devchat" for $100 credit

RR 399: Jets Ruby Serverless Framework with Tung Nguyen

jets nguyen tung sentry serverless framework

Play Episode Listen Later Feb 11, 2019 73:48

Sponsors Sentry use the code "devchat" for $100 credit

Episode #3: Internship Stories

Do Not Merge

Play Episode Listen Later Dec 31, 2018 28:00

In this episode, Nishan, Yankee, Kishor and Saugat talk about life as an Intern at Leapfrog. They go through the process and the challenges with regards to the 6-week long Internship bootcamp at Leapfrog. Show Notes: Leapfrog Internship: https://www.lftechnology.com/internships/ Internship at Leapfrog: https://blog.lftechnology.com/internship-at-leapfrog-technology-66e0f2da2049 Re-creating Memorable Games: https://blog.lftechnology.com/internship-stories-re-creating-memorable-games-af3af08f29be FlappyBotML: https://kishorliv.github.io/js-experiments/FlappyBotML/index.html MetalSlug: https://yankeexe.github.io/MetalSlug/ More: https://github.com/leapfroglets/repositories Competitive Programming: https://en.wikipedia.org/wiki/Competitive_programming Apollo GraphQL: https://www.apollographql.com/ The Serverless Framework: https://serverless.com/ Music by Peyruis: https://soundcloud.com/peyruis Send us your feedback at podcast@lftechnology.com

music stories intern yankee internship leapfrog kishor peyruis serverless framework

253: Jets: Ruby Serverless Framework with Tung Nguyen

Ruby on Rails Podcast

Play Episode Listen Later Dec 19, 2018 31:15

Jets is a framework that allows you to create serverless applications with Ruby. Tung Nguyen joined Brittany to discuss his passion for contributing to open source, DevOps and joining them together in the Ruby community.

jets nguyen devops tung serverless ruby on rails serverless framework

253: Jets: Ruby Serverless Framework with Tung Nguyen

Ruby on Rails Podcast

Play Episode Listen Later Dec 18, 2018 31:15

jets nguyen devops tung serverless ruby on rails serverless framework

50 выпуск 06 сезона. Ruby 2.6.0-rc2, Mruby 2.0.0, Action Mailbox for Rails 6, GraphQL code generator, Matchit и прочее

RWpod - подкаст про мир Ruby и Web технологии

Play Episode Listen Later Dec 16, 2018 27:43

Добрый день уважаемые слушатели. Представляем новый выпуск подкаста RWpod. В этом выпуске: Ruby Ruby 2.6.0-rc2 Released, Mruby 2.0.0, Introducing Action Mailbox for Rails 6 и Announcing GitLab Serverless Building AWS Lambdas for Real World using Ruby and Serverless Framework, Serverless Slack bot on Lambda with Ruby (and what's the less pleasant part about it) и How to build a serverless Twitter bot with Ruby and AWS Lambda Rails 5 Active Record attributes API и Project Health Indicators JavaScript A Recap of Frontend Development in 2018 и Our learnings from adopting GraphQL Let's talk JS ⚡: documentation и Performance Anti-Patterns: Base64 Encoding GraphQL code generator и Matchit - quickly parse & match URLs

action code released real world rails generator js urls mailbox lambda graphql front end development activerecord ruby ruby serverless framework

25: Palvelimettomuus

Webbidevaus.fi

Play Episode Listen Later Dec 9, 2018 66:25

Vieraana Cihan Bebek, aiheena "functions as a service" ja Serverless Framework. Pre-show Synthwave-soittolista ARRAY DISPATCH_1980 Edge vaihtaa enginensä EdgeHTML:stä Blinkiin Microsoft is building a Chromium-powered web browser that will replace Edge on Windows 10 Microsoft Edge: Making the web better through more open source collaboration github.com/MicrosoftEdge/MSEdge The Ecological Impact of Browser Diversity Edge dies a death of a thousand cuts as Microsoft switches to Chromium Goodbye Edge React Finland 2019 Serverless AWS Lambda Google Cloud Functions Azure Functions now.sh Netlify Functions Serverless Framework AWS SAM CLI serverless-stack.com Jakson valinnat Cihan: The Fable of the Dragon Tyrant Antti: Wolfenstein 2: The New Colossus Riku: Innolux Rondo 400 Kysymyksiä? Lähetä tänne: bit.ly/webbidevaus Tai twiittaa meille!

microsoft chromium serverless framework

The Future of FaaS

software cloud faa serverless framework

Play Episode Listen Later Nov 15, 2018 14:55

Jared and Forrest return together for a special fifteen-minute episode discussing the future of FaaS and Jared's new gig at the Serverless Framework.

The Future of FaaS

software cloud faa serverless framework

Play Episode Listen Later Nov 15, 2018 14:55

Jared and Forrest return together for a special fifteen-minute episode discussing the future of FaaS and Jared's new gig at the Serverless Framework.

Serverless and the Serverless Framework with David Wells

The InfoQ Podcast

Play Episode Listen Later May 27, 2018 32:23

The Serverless Framework is quickly becoming one of the more popular frameworks used in managing serverless deployments. David Wells, an engineer working on the framework, talks with Wes Reisz about serverless adoption and the use of the open source Serverless Framework. On this week’s podcast, the two dive into what it looks like to use the tool, the development experience, why a developer might want to consider a tool like the serverless framework, and finally wraps up with what the tool offers in areas like CI/CD, canaries, and blue/green deployment. Why listen to this podcast: - Serverless allows you to focus on the core business functionality and less on the infrastructure required to run your systems. - Serverless Framework allows you to simplify the amount of configuration you need for each cloud provider (for example, you can automate much of the configuration required for CloudFormation with AWS) - Serverless Framework is an open source CLI tool that supports all major cloud providers and several on-prem solutions for managing serverless functions. - The serverless space has room to grow in offering a local development space. Much of the workflow today involves frequent deploy and scoping the deployment for different stages. - Serverless Framework is open source and invites contributions from the community. More on this: Quick scan our curated show notes on InfoQ https://bit.ly/2IWayeE You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq Subscribe: www.youtube.com/infoq Like InfoQ on Facebook: bit.ly/2jmlyG8 Follow on Twitter: twitter.com/InfoQ Follow on LinkedIn: www.linkedin.com/company/infoq Check the landing page on InfoQ: https://bit.ly/2IWayeE

serverless ci cd cli david wells infoq cloudformation serverless framework

Zip It And Ship It - Serverless Deployments on AWS

development ship python aws serverless deployments serverless framework

Play Episode Listen Later Mar 29, 2018 4:26

AWS SAM and the Serverless Framework go head to head in our latest five-minute episode, where we discuss the challenges of deploying complex serverless apps on AWS.

Zip It And Ship It - Serverless Deployments on AWS

development ship python aws serverless deployments serverless framework

Play Episode Listen Later Mar 29, 2018 4:26

AWS SAM and the Serverless Framework go head to head in our latest five-minute episode, where we discuss the challenges of deploying complex serverless apps on AWS.

MJS 042: Kassandra Perch

All JavaScript Podcasts by Devchat.tv

Play Episode Listen Later Jan 5, 2018 50:27

Panel: Charles Max Wood Guest: Kassandra Perch This week on My JavaScript Story, Charles speaks with Kassandra Perch. Kassandra is a return guest from JavaScript Jabber episode 197. Kassandra is a developer relations engineer for IOpipe, that does AWS Lambda monitoring and visibility in the server-less space. Kassandra talks about her journey into program through game sharks or programming game cartridges. Also, furthering her interest in programming was taking computer science courses in college, and getting a part-time job in the technology field during college while networking. Kassandra shares her favorite contributions to javascript and open source projects. In particular, we dive pretty deep on: How did you get into programming? Game Sharks Game Cartridges Austin Meetup Group and JavaScript Working in the open source community College courses Contributions - Nodebotanist Interest in education and being autistic Child of a teacher Serving the community Helping people with projects IOT - Internet of Things Building Robots Serverless What are you working on now? AVR Girl and much, much more! Links: https://www.iopipe.com https://github.com/nodebotanist https://github.com/noopkat/avrgirl http://johnny-five.io IOpipe Picks Kassandra Sue Hitten Johnny 5 Serverless Framework Charles VS Code Azure pluggin Serverless Framework Amanda Silver interview More VS Code Interviews on Dev Chat TV

internet college child serving panel internet of things javascript contributions serverless aws lambda iot internet amanda silver itemname charles max wood serverless framework javascript jabber devchattv iopipe my javascript story kassandra perch

MJS 042: Kassandra Perch

Play Episode Listen Later Jan 4, 2018 50:27

MJS 042: Kassandra Perch

My JavaScript Story

Play Episode Listen Later Jan 4, 2018 50:27

JSJ 291: Serverless For JavaScript with Gareth McCumskey

JavaScript Jabber

Play Episode Listen Later Dec 12, 2017 54:26

Panel: Charles Max Wood Aimee Knight AJ O’Neal Joe Eames Special Guests: Gareth McCumskey In this episode, JavaScript Jabber speaks with Gareth McCumskey about Serverless For JavaScript. Gareth leads the dev team at Expat Explore in Cape Town, South Africa. Gareth and this team specialize in exploring the Serverless realm in JavaScript. The JavaScript Jabbers panel and Gareth discuss the many different types of serverless systems, and when to implement them, how serverless system work, and when to go in the direction of using Serverless. In particular, we dive pretty deep on: What does it mean to be Serverless? Since platform as a service. Microservice on Docker Firebase “no backend” Backend systems Cloud functions and failure in systems How do you start to think about a serverless system? How do decide what to do? AWS Lambda Working in a different vendor Node 4 Programming JS to deploy Using libraries for NPM How is works with AWS Lambda Where is the database? More point of failure? Calls to Slack? Authentication Micro Services Elastic Bean Stalk Static Assets, S3, Managing Testing the services Integration testing And much more! Links: @garethmcc @expatexplore gareth.mccumskey.com https://github.com/garethmcc serverless.com Picks: Aimee Serverless Architectures NG-BE Conference AJ Documentary on Enron Hard Thing about Hard Things Charles Serverless Framework The Storm Light Achieves Avengers: Infinity War Gareth Building MicroServices Skeptics Guide To The Universe Podcast Expate Explore Joe Wonder - Movie Gloom In Space - Board Game

JSJ 291: Serverless For JavaScript with Gareth McCumskey

All JavaScript Podcasts by Devchat.tv

Play Episode Listen Later Dec 12, 2017 54:26

JSJ 291: Serverless For JavaScript with Gareth McCumskey

Play Episode Listen Later Dec 12, 2017 54:26

Our Stacks Explained

Syntax - Tasty Web Development Treats

Play Episode Listen Later Oct 4, 2017 79:34

Scott and Wes detail their current stacks that run their training platforms. From front end code linting to the server side and databases. Sponsor Intro to The Serverless Framework by Loren Stewart. The first 20 people to use the code SYNTAX_FREE will get the course for free! After that make sure to use the code SYNTAX for an extra $10 off. Show Notes Wes' Stack Youtube Video Meteor Node.js Level Up Tutorials is fast! Express Learn Node Passport JS MongoDB Mongoose mLab Hosting Mongohub MongoDB Compass Studio 3T MiniMongo React Styled Components Stylus Lang Metor Sessions Prerender.io React Apollo Cross Storage Victory Charts Cloudinary Tim Thumb Amazon S3 Amazon Cloudfront Backblaze Vimeo Pro Jest Mocha Fixer.io Curreny Conversion API Brain Tree Stripe Mandrill Drip Amazon SES PostMark App (THE BEST) Zurb Inky Juice CSS Inliner Meteor Hosting Meteor Hosting Digital Ocean Zeit Now Heroku Bluehost Sucks Let's Encrypt Cloudflare OOPS I SAID CLOUDFRONT Sick Picks Scott: Focusrite Scarlett 2i2 Wes: Better Bidding Tweet us your tips! Wes Bos Scott Tolinski Make sure to include @SyntaxFM Shameless Plugs Level Up Tuts - check out scott's new shopping cart! Wes just updated his ES6 course!

stacks syntax es6 serverless framework

The ServerlessCast #8 - Managing Serverless Performance

The Cloudcast

Play Episode Listen Later Jul 28, 2017 25:35

NOTE: Sorry about the patchy audio quality in spots on this episode, we didn't notice until we edited the show that both Adam's and Aaron's connections weren't the best this time. We cleaned it up the best we could. Thanks for listening! Aaron and Brian talk with Adam Johnson (@adjohn, CO-Founder/CEO of @IOpipe) about the momentum of Serverless, the challenge of identify performance bottlenecks, if NoOps is real or not, how APMs have to evolve in a Serverless world. Show Links: Get a free eBook from O'Reilly media or use promo code PC20CLOUD for a discount - 40% off Print Books and 50% off eBooks and videos [DISCOUNT] Start Serverless Skills Bundle (4 courses) - (only $49 instead of $79) [FREE] Alexa Development for Absolute Beginners IOpipe website Show Notes Topic 1 - Welcome to the show. Give us a little bit of your background and walk us through the process of being part of the TechStars incubator program. Tell us a little bit about IOpipe. Topic 2 - It took a little while for people to grasp the concept of serverless (e.g. AWS Lambda), but now it has a lot of momentum. What made you think it was the the right focus for the company a few years back? Topic 3 - Why IOpipe? What problem does is solve? How is it “Included” in existing FaaS code today? How is that different from the Serverless Framework plugin? Topic 4 - Serverless seems to be associated with the NoOps movement, since the underlying platform tends to do many things the Ops team used to do. Where does an APM platform like IOpipe fit into the discussion? Is it used by a Serverless Ops team, or mostly by Developers? Topic 5 - What are some of the common performance challenges with Serverless that people should be aware of? Follow Up: What are some of the most common emerging use cases? Feedback? Email: show at thecloudcast dot net Twitter: @thecloudcastnet

19 выпуск 05 сезона. Rails 5.0.3 и 5.1.1, GraphQL в Rails, Serverless-ruby, Webpack CLI, Britecharts, Sticky Sidebar и прочее

RWpod - подкаст про мир Ruby и Web технологии

Play Episode Listen Later May 14, 2017 39:21

Добрый день уважаемые слушатели. Представляем новый выпуск подкаста RWpod. В этом выпуске: Ruby Rails 5.0.3 and 5.1.1 have been released, Fixing Unicode for Ruby Developers, How to find and fix bug in JRuby and don't go crazy и Don't Use Objects as Hash Keys in Ruby Start using GraphQL in Rails, The hidden cost of the invisible queries in Rails и Faster Rails: Is Your Database Properly Indexed? Trailblazer on Rails: Basics - Setup & Use, Serverless-ruby - a Serverless Framework example using an AWS lambda which runs a Ruby function и Bundleup - a Ruby project containing a Gemfile to see what gem dependencies need updating JavaScript Announcing the new webpack CLI, Methods for Contrasting Text Against Backgrounds и The Issue with Preprocessing CSS Custom Properties Distributing a self-replicating malicious code using NPM, The Hitchhiker's Guide to d3.js и Making Maps With React JavaScript implementations of different famous Computer Science algorithms, Britecharts - a client-side reusable Charting Library based on D3.js, Pingy CLI - simple frontend build tool и Sticky Sidebar - jQuery plugin for making smart and high performance sticky sidebars

guide methods computer science sticky aws rails trailblazer hitchhiker d3 sidebar serverless cli graphql npm webpack jruby serverless framework ruby rails

HJDL4: Serverless (and a theme song!)

HARAJUKU DATA LAKE

Play Episode Listen Later Feb 28, 2017 62:41

In our longest episode yet, Morris and Sergio discuss serverless the architectural concept and Serverless the open-source framework. Show NotesCagpie.net Shun Kobayashi’s portfolioPicotune Shun’s chiptune-style MIDI player and piano roll visualizer, hand-coded in vanilla JSServerless Architectures by Martin FowlerServerless, the company and the frameworkStackOverflow: Lock, mutex, semaphore… what’s the difference?Serverless Framework with Austen Collins on Software Engineering DailyThe Docker Weekly newsletter

morris theme songs midi serverless serverless framework

The ServerlessCast #2 - Kubeless - Serverless Framework for Kubernetes

The Cloudcast

Play Episode Listen Later Feb 17, 2017 27:31

Brian talks with Sebastien Goasguen (@sebgoa, Founder @skippbox)about his experience with containers, the focus of Skippbox, market demand for serverless, the architecture of Kubeless, and how the emerging serverless+kubernetes projects need to evolve. Show Links: Get a free eBook from O'Reilly media or use promo code PCBW for a discount - 40% off Print Books and 50% off eBooks and videos Skippbox acquired by Bitnami on March 7, 2017 Skippbox website Kubeless - Serverless Framework for Kubernetes (GitHub) “Docker Cookbook” by Sebastien Goasguen (O’Reilly) Show Notes: Topic 1 - Welcome to the show. Give us some of your background, as well as what you’re doing at Skippbox these days. Topic 2 - Before we jump into “Kubeless”, let’s talk about what you’re hearing around serverless in the market today. Topic 3 - Tell us about the Kubeless architecture. Topic 4 - With different serverless functions, they initially support a limited subset of languages. Since Kubernetes is language agnostic, why does the limitation exist, or how complex is it to add new languages? Topic 5 - You mention on the project’s GitHub page that there are other “serverless on Kubernetes” alternatives out there (Funktions, Fission, OpenWhisk, etc.). Do you expect that one of these projects will emerge, or do you see these starting to merge and just become a job type within Kubernetes? Topic 6 - Let’s come back to Skippbox. You have a focus on Kubernetes and the tooling around making it easier to deploy and run. What are you seeing in the Kubernetes market and when are people engaging with Skippbox? Feedback? Email:show at thecloudcast dot net Twitter:@thecloudcastnet or @serverlesscast YouTube:Cloudcast Channel

founders ebooks open source github functions containers docker kubernetes lambda serverless fission kubecon bitnami print books serverless framework funktions openwhisk

Ep. #4, The Serverless Framework & AWS Lambda

JAMstack Radio

Play Episode Listen Later Oct 28, 2016 32:35

In Episode 4 of JAMstack Radio, Brian and Ryan are joined by engineer David Wells who explains the Serverless Framework and automation using AWS Lambda. The three cover topics including potential pain points of complex microservices, advantages of event-driven architectures, and writing Kanye skills for Amazon's Alexa. Plus a new round of JAMPicks.

amazon kanye west david wells aws lambda serverless framework

Serverless Architecture with Ben Godwin

.NET Rocks!

Play Episode Listen Later Oct 18, 2016 52:10

Serverless is the new hot buzzword - but what does it really mean? Carl and Richard talk to Ben Godwin about his work building serverless applications - no servers, but lots of services! Ben talks about Amazon Lambda, which is similar to Azure Functions. Both these environments allow individual bits of code to run within them, written in a variety of languages, but often that language is Javascript in the Node style. The advantage of this approach is eliminating a lot of the ceremony around your services set, but at the price of some new working patterns and organization. Ben also mentions the Serverless Framework as a great free tool for getting started!Support this podcast at — https://redcircle.com/net-rocks/donations

microsoft studio visual vista javascript node orcas asp serverless vb azure functions serverless framework serverless architecture ben godwin

Serverless Architecture with Ben Godwin

.NET Rocks!

Play Episode Listen Later Oct 18, 2016 52:09

javascript node serverless azure functions serverless framework serverless architecture ben godwin

227 JSJ Fostering Community Through React with Benjamin Dunphy, Berkeley Martinez, and Ian Sinnott

JavaScript Jabber

Play Episode Listen Later Aug 31, 2016 51:06

03:08 - Benjamin Dunphy Introduction Twitter GitHub 04:07 - Berkeley Martinez Introduction Twitter GitHub Free Code Camp 04:19 - Ian Sinnott Introduction Twitter GitHub Blog TruSTAR Technology 05:19 - The React Codebase 12:38 - Other Important Parts of the React Ecosystem 14:22 - The Angular vs the React Ecosystem and Community The Learning Curve create-react-app 22:07 - Community Developer Experience Functional Programming 26:56 - Getting Connected to the React Community Meetup: Real World React @rwreact ReactJS San Francisco Bay Area Meetup Meetup Eventbrite Calagator Twitter Dan Abramov: My React List 29:34 - Conferences React.js Conf React Rally ReactNext ReactiveConf ReactEurope 33:28 - Technology From the Community redux ThunderCats.js 38:23 - Choices Are Expanding; Not Shrinking Linting 40:19 - The Future of React 42:39 - Starting More Communities Picks This Developing Story (Aimee) Nashville (Aimee) Nodevember (Aimee) egghead.io: React in 7 Minutes (Ben) Lee Byron: Immutable User Interfaces @ Render 2016 (Ben) Nick Schrock: React.js Conf 2016 Keynote (Ben) create-react-app (Ian) Functional Programming Jargon (Ian) The Serverless Framework (Ian) Ben's Blog (Berkeley) Isaac Asimov’s Robot Series (Berkeley) Vsauce: The Zipf Mystery (Berkeley) Kinesis Advantage for PC & Mac (Dave)

community future nashville blog robots pc mac berkeley martinez react keynote functional conferences conf meetup github eventbrite thundercats isaac asimov learning curves asimov angular developer experience dunphy functional programming freecodecamp sinnott fostering community getting connected codebase linting serverless framework react rally nodevember trustar technology kinesis advantage reacteurope

227 JSJ Fostering Community Through React with Benjamin Dunphy, Berkeley Martinez, and Ian Sinnott