Podcasts about Lambda

Share on
Share on Facebook
Share on Twitter
Share on Reddit
Copy link to clipboard

Letter in the Greek alphabet

  • 564PODCASTS
  • 3,115EPISODES
  • 1h 8mAVG DURATION
  • 5WEEKLY NEW EPISODES
  • Nov 26, 2021LATEST
Lambda

POPULARITY

20112012201320142015201620172018201920202021


Best podcasts about Lambda

Show all podcasts related to lambda

Latest podcast episodes about Lambda

Lambda3 Podcast
Lambda3 Podcast 275 – Diversidade Racial

Lambda3 Podcast

Play Episode Listen Later Nov 26, 2021 40:37


Na série de diversidade hoje vamos falar sobre diversidade racial, essa sequência de conteúdos também tem conexão com o Blog da Lambda, onde lançaremos artigos sobre o assunto de cada episódio do podcast. Feed do podcast: www.lambda3.com.br/feed/podcast Feed do podcast somente com episódios técnicos: www.lambda3.com.br/feed/podcast-tecnico Feed do podcast somente com episódios não técnicos: www.lambda3.com.br/feed/podcast-nao-tecnico Lambda3 · #275 - Diversidade Racial Pauta: Representatividade em outros ambientes e mídias e o impacto na autoestima da pessoa negra Diversidade Racial - O quanto falar sobre diversidade racial impacta diretamente na representatividade e autoestima? Depoimentos - Realidades enfrentadas seja na vida pessoal e/ou no mercado de trabalho Impactos no mercado de trabalho - Que ações as empresas precisam tomar para que se diminua os preconceitos raciais enfrentados pelas pessoas não brancas? Que ações tomamos na Lambda?  Participantes: Sarah Barros Bianca Paiva Victor Hugo Germando Edição: Compasso Coolab Créditos das músicas usadas neste programa: Music by Kevin MacLeod (incompetech.com) licensed under Creative Commons: By Attribution 3.0 - creativecommons.org/licenses/by/3.0

The Cloud Pod
144: Oh the Places You'll Go at re:Invent 2021

The Cloud Pod

Play Episode Listen Later Nov 24, 2021 61:35


The Cloud Pod: Oh the Places You'll Go at re:Invent 2021 — Episode 144 On The Cloud Pod this week, as a birthday present to Ryan, the team didn't discuss his advanced age, and focused instead on their AWS re:Invent predictions. Also, the Google Cybersecurity Action Team launches a product, and Microsoft announces a new VM series in Azure. A big thanks to this week's sponsors: Foghorn Consulting, which provides full-stack cloud solutions with a focus on strategy, planning and execution for enterprises seeking to take advantage of the transformative capabilities of AWS, Google Cloud and Azure. JumpCloud, which offers a complete platform for identity, access, and device management — no matter where your users and devices are located.  This week's highlights

Equity
Co-founder titles aren't a formality, they're a morality

Equity

Play Episode Listen Later Nov 19, 2021 30:01


Good news, everyone: Mary Ann is back! Yes, after a short absence we have our third Friday host back where she belongs. Namely right in front of a microphone, talking to us.And it was good to have her back, as we had a small mountain of news to talk through:OpenSea could be worth $10 billion: Right before we recorded, news broke that OpenSea could raise new capital at a far-greater valuation than it did earlier this year. While we had some jokes, we also did a little math on why OpenSea could really be worth so much money.Other rapid-fire funding rounds: If OpenSea does raise again, it will join good company in putting together several rounds in under a year. Facily has done the same, as has Justt. In short, the old 18-month cycle for raising capital is so dated it might as well get delivered by telegram.Lambda School rebrands:Casper goes private: After a tough time as a public concern, DTC mattress company Casper is going private. Notably its issues have not dampened market interest in taking other DTC companies public.What makes a founder, a founder:We're entering the holiday period, so expect the usual disruptions to Equity's production calendar. We'll be taking short breaks as per usual, but will be recharging some ahead of what should prove to be a very, very busy 2022.

Coders Campus Podcast
EP51 - Let's Talk Lambdas in Java

Coders Campus Podcast

Play Episode Listen Later Nov 12, 2021 57:32


In this episode we'll talk about a super useful feature that was introduced back in Java version 8, known as Lambdas. The Lambda feature is something you didn't know you desperately wanted or needed until you understood it. The Lambda syntax allows you to write much cleaner and more readable code, while also empowering you to get more done with less code. In this lecture, I'll be referring to some code that you can download via this github repository. Interested in starting your coding career? I'm now accepting students into an immersive programming Bootcamp where I guarantee you a job offer upon graduation. It is a 6 month, part-time, online Bootcamp that teaches you everything you need to know to get a job as a Java developer in the real-world. You can learn more via https://www.coderscampus.com/bootcamp

Screaming in the Cloud
Building a Partnership with Your Cloud Provider with Micheal Benedict

Screaming in the Cloud

Play Episode Listen Later Nov 10, 2021 54:44


About Micheal Micheal Benedict leads Engineering Productivity at Pinterest. He and his team focus on developer experience, building tools and platforms for over a thousand engineers to effectively code, build, deploy and operate workloads on the cloud. Mr. Benedict has also built Infrastructure and Cloud Governance programs at Pinterest and previously, at Twitter -- focussed on managing cloud vendor relationships, infrastructure budget management, cloud migration, capacity forecasting and planning and cloud cost attribution (chargeback). Links: Pinterest: https://www.pinterest.com Teletraan: https://github.com/pinterest/teletraan Twitter: https://twitter.com/micheal Pinterestcareers.com: https://pinterestcareers.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: You know how git works right?Announcer: Sorta, kinda, not really. Please ask someone else!Corey: Thats all of us. Git is how we build things, and Netlify is one of the best way I've found to build those things quickly for the web. Netlify's git based workflows mean you don't have to play slap and tickle with integrating arcane non-sense and web hooks, which are themselves about as well understood as git. Give them a try and see what folks ranging from my fake Twitter for pets startup, to global fortune 2000 companies are raving about. If you end up talking to them, because you don't have to, they get why self service is important—but if you do, be sure to tell them that I sent you and watch all of the blood drain from their faces instantly. You can find them in the AWS marketplace or at www.netlify.com. N-E-T-L-I-F-Y.comCorey: This episode is sponsored in part by our friends at Vultr. Spelled V-U-L-T-R because they're all about helping save money, including on things like, you know, vowels. So, what they do is they are a cloud provider that provides surprisingly high performance cloud compute at a price that—while sure they claim its better than AWS pricing—and when they say that they mean it is less money. Sure, I don't dispute that but what I find interesting is that it's predictable. They tell you in advance on a monthly basis what it's going to going to cost. They have a bunch of advanced networking features. They have nineteen global locations and scale things elastically. Not to be confused with openly, because apparently elastic and open can mean the same thing sometimes. They have had over a million users. Deployments take less that sixty seconds across twelve pre-selected operating systems. Or, if you're one of those nutters like me, you can bring your own ISO and install basically any operating system you want. Starting with pricing as low as $2.50 a month for Vultr cloud compute they have plans for developers and businesses of all sizes, except maybe Amazon, who stubbornly insists on having something to scale all on their own. Try Vultr today for free by visiting: vultr.com/screaming, and you'll receive a $100 in credit. Thats v-u-l-t-r.com slash screaming.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Every once in a while, I like to talk to people who work at very large companies that are not in fact themselves a cloud provider. I know it sounds ridiculous. How can you possibly be a big company and not make money by selling managed NAT gateways to an unsuspecting public? But I'm told it can be done here to answer that question. And hopefully at least one other is Pinterest. It's head of engineering productivity, Micheal Benedict. Micheal, thank you for taking the time to join me today.Micheal: Hi, Corey, thank you for inviting me today. I'm really excited to talk to you.Corey: So, exciting times at Pinterest in a bunch of different ways. It was recently reported—which of course, went right to the top of my inbox as 500,000 people on Twitter all said, “Hey, this sounds like a ‘Corey would be interested in it' thing.” It was announced that you folks had signed a $3.2 billion commitment with AWS stretching until 2028. Now, if this is like any other large-scale AWS contract commitment deal that has been made public, you were probably immediately inundated with a whole bunch of people who are very good at arithmetic and not very good at business context saying, “$3.2 billion? You could build massive data centers for that. Why would anyone do this?” And it's tiresome, and that's the world in which we live. But I'm guessing you heard at least a little bit of that from the peanut gallery.Micheal: I did, and I always find it interesting when direct comparisons are made with the total amount that's been committed. And like you said, there's so many nuances that go into how to perceive that amount, and put it in context of, obviously, what Pinterest does. So, I at least want to take this opportunity to share with everyone that Pinterest has been on the cloud since day one. When Ben initially started the company, that product was launched—it was a simple Django app—it was launched on AWS from day one, and since then, it has grown to support 450-plus million MAUs over the course of the decade.And our infrastructure has grown pretty complex. We started with a bunch of EC2 machines and persisting data in S3, and since then we have explored an array of different products, in fact, sometimes working very closely with AWS, as well and helping them put together a product roadmap for some of the items they're working on as well. So, we have an amazing partnership with them, and part of the commitment and how we want to see these numbers is how does it unlock value for Pinterest as a business over time in terms of making us much more agile, without thinking about the nuances of the infrastructure itself. And that's, I think, one of the best ways to really put this into context, that it's not a single number we pay at the end [laugh] of the month, but rather, we are on track to spending a certain amount over a period of time, so this just keeps accruing or adding to that number. And we basically come out with an amazing partnership in AWS, where we have that commitment and we're able to leverage their products and full suite of items without any hiccups.Corey: The most interesting part of what you said is the word partner. And I think that's the piece that gets lost an awful lot when we talk about large-scale cloud negotiations. It's not like buying a car, where you can basically beat the crap out of the salesperson, you can act as if $400 price difference on a car is the difference between storm out of the dealership and sign the contract. Great, you don't really have to deal with that person ever again.In the context of a cloud provider, they run your production infrastructure, and if they have a bad day, I promise you're going to have a bad day, too. You want to handle those negotiations in a way that is respectful of that because they are your partner, whether you want them to be or not. Now, I'm not suggesting that any cloud provider is going to hold an awkward negotiation against the customer, but at the same time, there are going to be scenarios in which you're going to want to have strong relationships, where you're going to need to cash in political capital to some extent, and personally, I've never seen stupendous value in trying to beat the crap out of a company in order to get another tenth of a percent discount on a service you barely use, just because someone decided that well, we didn't do well in the last negotiation so we're going to get them back this time.That's great. What are you actually planning to do as a company? Where are you going? And the fact that you just alluded to, that you're not just a pile of S3 and EC2 instances speaks, in many ways, to that. By moving into the differentiated service world, suddenly you're able to do things that don't look quite as much like building a better database and start looking a lot more like servicing your users more effectively and well.Micheal: And I think, like you said, I feel like there's like a general skepticism in viewing that the cloud providers are usually out there to rip you apart. But in reality, that's not true. To your point, as part of the partnership, especially with AWS and Pinterest, we've got an amazing relationship going on, and behind the scenes, there's a dedicated team at Pinterest, called the Infrastructure Governance Team, a cross-functional team with folks from finance, legal, engineering, product, all sitting together and working with our AWS partners—even the AWS account managers at the times are part of that—to help us make both Pinterest successful, and in turn, AWS gets that amazing customer to work with in helping build some of their newer products as well. And that's one of the most important things we have learned over time is that there's two parts to it; when you want to help improve your business agility, you want to focus not just on the bottom line numbers as they are. It's okay to pay a premium because it offsets the people capital you would have to invest in getting there.And that's a very tricky way to look at math, but that's what these teams do; they sit down and work through those specifics. And for what it's worth, in our conversations, the AWS teams always come back with giving us very insightful data on how we're using their systems to help us better think about how we should be pricing or looking things ahead. And I'm not the expert on this; like I said, there's a dedicated team sitting behind this and looking through and working through these deals, but that's one of the important takeaways I hope the users—or the listeners of this podcast then take away that you want to treat your cloud provider as your partner as much as possible. They're not always there to screw you. That's not their goal. And I apologize for using that term. It is important that you set that expectations that it's in their best interest to actually make you successful because that's how they make money as well.Corey: It's a long-term play. I mean, they could gouge you this quarter, and then you're trying to evacuate as fast as possible. Well, they had a great quarter, but what's their long-term prospect? There are two competing philosophies in the world of business; you can either make a lot of money quickly, or you can make a little bit of money and build it over time in a sustained way. And it's clear the cloud providers are playing the long game on this because they basically have to.Micheal: I mean, it's inevitable at this point. I mean, look at Pinterest. It is one of those success stories. Starting as a Django app on a bunch of EC2 machines to wherever we are right now with having a three-plus billion dollar commitment over a span of couple of years, and we do spend a pretty significant chunk of that on a yearly basis. So, in this case, I'm sure it was a great successful partnership.And I'm hoping some of the newer companies who are building the cloud from the get-go are thinking about it from that perspective. And one of the things I do want to call out, Corey, is that we did initially start with using the primitive services in AWS, but it became clear over time—and I'm sure you heard of the term multi-cloud and many of that—you know, when companies start evaluating how to make the most out of the deals they're negotiating or signing, it is important to acknowledge that the cost of any of those evaluations or even thinking about migrations never tends to get factored in. And we always tend to treat that as being extremely simple or not, but those are engineering resources you want to be spending more building on the product rather than these crazy costly migrations. So, it's in your best interest probably to start using the most from your cloud provider, and also look for opportunities to use other cloud providers—if they provide more value in certain product offerings—rather than thinking about a complete lift-and-shift, and I'm going to make DR as being the primary case on why I want to be moving to multi-cloud.Corey: Yeah. There's a question, too, of the numbers on paper look radically different than the reality of this. You mentioned, Pinterest has been on AWS since the beginning, which means that even if an edict had been passed at the beginning, that, “Thou shalt never build on anything except EC2 and S3. The end. Full stop.”And let's say you went down that rabbit hole of, “Oh, we don't trust their load balancers. We're going to build our own at home. We have load balancers at home. We'll use those.” It's terrible, but even had you done that and restricted yourselves just to those baseline building blocks, and then decide to do a cloud migration, you're still looking back at over a decade of experience where the app has been built unconsciously reflecting the various failure modes that AWS has, the way that it responds to API calls, the latency in how long it takes to request something versus it being available, et cetera, et cetera.So, even moving that baseline thing to another cloud provider is not a trivial undertaking by any stretch of the imagination. But that said—because the topic does always come up, and I don't shy away from it; I think it's something people should go into with an open mind—how has the multi-cloud conversation progressed at Pinterest? Because there's always a multi-cloud conversation.Micheal: We have always approached it with some form of… openness. It's not like we don't want to be open to the ideas, but you really want to be thinking hard on the business case and the business value something provides on why you want to be doing x. In this case, when we think about multi-cloud—and again, Pinterest did start with EC2 and S3, and we did keep it that way for a long time. We built a lot of primitives around it, used it—for example, my team actually runs our bread and butter deployment system on EC2. We help facilitate deployments across a 100,000-plus machines today.And like you said, we have built that system keeping in mind how AWS works, and understanding the nuances of region and AZ failovers and all of that, and help facilitate deployments across 1000-plus microservices in the company. So, thinking about leveraging, say, a Google Cloud instance and how that works, in theory, we can always make a case for engineering to build our deployment system and expand there, but there's really no value. And one of the biggest cases, usually, when multi-cloud comes in is usually either negotiation for price or actually a DR strategy. Like, what if AWS goes down in and us-east-1? Well, let's be honest, they're powering half the internet [laugh] from that one single—Corey: Right.Micheal: Yeah. So, if you think your business is okay running when AWS goes down and half the internet is not going to be working, how do you want to be thinking about that? So, DR is probably not the best reason for you to be even exploring multi-cloud. Rather, you should be thinking about what the cloud providers are offering as a very nuanced offering which your current cloud provider is not offering, and really think about just using those specific items.Corey: So, I agree that multi-cloud for DR purposes is generally not necessarily the best approach with the idea of being able to failover seamlessly, but I like the idea for backups. I mean, Pinterest is a publicly-traded company, which means that among other things, you have to file risk disclosures and be responsive to auditors in a variety of different ways. There are some regulations to start applying to you. And the idea of, well, AWS builds things out in a super effective way, region separation, et cetera, whenever I talk to Amazonians, they are always surprised that anyone wouldn't accept that, “Oh, if you want backups use a different region. Problem solved.”Right, but it is often easier for me to have a rehydrate the business level of backup that would take weeks to redeploy living on another cloud provider than it is for me to explain to all of those auditors and regulators and financial analysts, et cetera why I didn't go ahead and do that path. So, there's always some story for okay, what if AWS decides that they hate us and want to kick us off the platform? Well, that's why legal is involved in those high-level discussions around things like risk, and indemnity, and termination for convenience and for cause clauses, et cetera, et cetera. The idea of making an all-in commitment to a cloud provider goes well beyond things that engineering thinks about. And it's easy for those of us with engineering backgrounds to be incredibly dismissive of that of, “Oh, indemnity? Like, when does AWS ever lose data?” “Yeah, but let's say one day they do. What is your story going to be when asked some very uncomfortable questions by people who wanted you to pay attention to this during the negotiation process?” It's about dotting the i's and crossing the t's, especially with that many commas in the contractual commitments.Micheal: No, it is true. And we did evaluate that as an option, but one of the interesting things about compliance, and especially auditing as well, we generally work with the best in class consultants to help us work through the controls and how we audit, how we look at these controls, how to make sure there's enough accountability going through. The interesting part was in this case, as well, we were able to work with AWS in crafting a lot of those controls and setting up the right expectations as and when we were putting proposals together as well. Now, again, I'm not an expert on this and I know we have a dedicated team from our technical program management organization focused on this, but early on we realized that, to your point, the cost of any form of backups and then being able to audit what's going in, look at all those pipelines, how quickly we can get the data in and out it was proving pretty costly for us. So, we were able to work out some of that within the constructs of what we have with our cloud provider today, and still meet our compliance goals.Corey: That's, on some level, the higher point, too, where everything is everything comes down to context; everything comes down to what the business demands, what the business requires, what the business will accept. And I'm not suggesting that in any case, they're wrong. I'm known for beating the ‘Multi-cloud is a bad default decision' drum, and then people get surprised when they'll have one-on-one conversations, and they say, “Well, we're multi-cloud. Do you think we're foolish?” “No. You're probably doing the right thing, just because you have context that is specific to your business that I, speaking in a general sense, certainly don't have.”People don't generally wake up in the morning and decide they're going to do a terrible job or no job at all at work today, unless they're Facebook's VP of Integrity. So, it's not the sort of thing that lends itself to casual tweet size, pithy analysis very often. There's a strong dive into what is the level of risk a business can accept? And my general belief is that most companies are doing this stuff right. The universal constant in all of my consulting clients that I have spoken to about the in-depth management piece of things is, they've always asked the same question of, “So, this is what we've done, but can you introduce us to the people who are doing it really right, who have absolutely nailed this and gotten it all down?” “It's, yeah, absolutely no one believes that that is them, even the folks who are, from my perspective, pretty close to having achieved it.”But I want to talk a bit more about what you do beyond just the headline-grabbing large dollar figure commitment to a cloud provider story. What does engineering productivity mean at Pinterest? Where do you start? Where do you stop?Micheal: I want to just quickly touch upon that last point about multi-cloud, and like you said, every company works within the context of what they are given and the constraints of their business. It's probably a good time to give a plug to my previous employer, Twitter, who are doing multi-cloud in a reasonably effective way. They are on the data centers, they do have presence on Google Cloud, and AWS, and I know probably things have changed since a couple of years now, but they have embraced that environment pretty effectively to cater to their acquisitions who were on the public cloud, help obviously, with their initial set of investments in the data center, and still continue to scale that out, and explore, in this case, Google Cloud for a variety of other use cases, which sounds like it's been extremely beneficial as well.So, to your point, there is probably no right way to do this. There's always that context, and what you're working with comes into play as part of making these decisions. And it's important to take a lot of these with a grain of salt because you can never understand the decisions, why they were made the way they were made. And for what it's worth, it sort of works out in the end. [laugh]. I've rarely heard a story where it's never worked out, and people are just upset with the deals they've signed. So, hopefully, that helps close that whole conversation about multi-cloud.Corey: I hope so. It's one of those areas where everyone has an opinion and a lot of them do not necessarily apply universally, but it's always fun to take—in that case, great, I'll take the lesser trod path of everyone's saying multi-cloud is great, invariably because they're trying to sell you something. Yeah, I have nothing particularly to sell, folks. My argument has always been, in the absence of a compelling reason not to, pick a provider and go all in. I don't care which provider you pick—which people are sometimes surprised to hear.It's like, “Well, what if they pick a cloud provider that you don't do consulting work for?” Yeah, it turns out, I don't actually need to win every AWS customer over to have a successful working business. Do what makes sense for you, folks. From my perspective, I want this industry to be better. I don't want to sit here and just drum up business for myself and make self-serving comments to empower that. Which apparently is a rare tactic.Micheal: No, that's totally true, Corey. One of the things you do is help people with their bills, so this has come up so many times, and I realize we're sort of going off track a bit from that engineering productivity discussion—Corey: Oh, which is fine. That's this entire show's theme, if it has one.Micheal: [laugh]. So, I want to briefly just talk about the whole billing and how cost management works because I know you spend a lot of time on that and you help a lot of these companies be effective in how they manage their bills. These questions have come up multiple times, even at Pinterest. We actually in the past, when I was leading the infrastructure governance organization, we were working with other companies of our similar size to better understand how they are looking into getting visibility into their cost, setting sort of the right controls and expectations within the engineering organization to plan, and capacity plan, and effectively meet those plans in a certain criteria, and then obviously, if there is any risk to that, actively manage risk. That was like the biggest thing those teams used to do.And we used to talk a lot trade notes, and get a better sense of how a lot of these companies are trying to do—for example, Netflix, or Lyft, or Stripe. I recall Netflix, content was their biggest spender, so cloud spending was like way down in the list of things for them. [laugh]. But regardless, they had an active team looking at this on a day-to-day basis. So, one of the things we learned early on at Pinterest is that start investing in those visibility tools early on.No one can parse the cloud bills. Let's be honest. You're probably the only person who can reverse… [laugh] engineer an architecture diagram from a cloud bill, and I think that's like—definitely you should take a patent for that or something. But in reality, no one has the time to do that. You want to make sure your business leaders, from your finance teams to engineering teams to head of the executives all have a better understanding of how to parse it.So, investing engineering resources, take that data, how do you munch it down to the cost, the utilization across the different vectors of offerings, and have a very insightful discussion. Like, what are certain action items we want to be taking? It's very easy to see, “Oh, we overspent EC2,” and we want to go from there. But in reality, that's not just that thing; you will start finding out that EC2 is being used by your Hadoop infrastructure, which runs hundreds of thousands of jobs. Okay, now who's actually responsible for that cost? You might find that one job which is accruing, sort of, a lot of instance hours over a period of time and a shared multi-tenant environment, how do you attribute that cost to that particular cost center?Corey: And then someone left the company a while back, and that job just kept running in perpetuity. No one's checked the output for four years, I guess it can't be that necessarily important. And digging into it requires context. It turns out, there's no SaaS tool to do this, which is unfortunate for those of us who set out originally to build such a thing. But we discovered pretty early on the context on this stuff is incredibly important.I love the thing you're talking about here, where you're discussing with your peer companies about these things because the advice that I would give to companies with the level of spend that you folks do is worlds apart from what I would advise someone who's building something new and spending maybe 500 bucks a month on their cloud bill. Those folks do not need to hire a dedicated team of people to solve for these problems. At your scale, yeah, you probably should have had some people in [laugh] here looking at this for a while now. And at some point, the guidance changes based upon scale. And if there's one thing that we discover from the horrible pages of Hacker News, it's that people love applying bits of wisdom that they hear in wildly inappropriate situations.How do you think about these things at that scale? Because, a simple example: right now I spend about 1000 bucks a month at The Duckbill Group, on our AWS bill. I know. We have one, too. Imagine that. And if I wind up just committing admin credentials to GitHub, for example, and someone compromises that and start spinning things up to mine all the Bitcoin, yeah, I'm going to notice that by the impact it has on the bill, which will be noticeable from orbit.At the level of spend that you folks are at, at company would be hard-pressed to spin up enough Bitcoin miners to materially move the billing needle on a month-to-month basis, just because of the sheer scope and scale. At small bill volumes, yeah, it's pretty easy to discover the thing that spiking your bill to three times normal. It's usually a managed NAT gateway. At your scale, tripling the bill begins to look suspiciously like the GDP of a small country, so what actually happened here? Invariably, at that scale, with that level of massive multiplier, it's usually the simplest solution, an error somewhere in the AWS billing system. Yes, they exist. Imagine that.Micheal: They do exist, and we've encountered that.Corey: Kind of heartstopping, isn't it?Micheal: [laugh]. I don't know if you remember when we had the big Spectre and the Meltdown, right, and those were interesting scenarios for us because we had identified a lot of those issues early on, given the scale we operate, and we were able to, sort of, obviously it did have an impact on the builds and everything, but that's it; that's why you have these dedicated teams to fix that. But I think one of the points you made, these are large bills and you're never going to have a 3x jump the next day. We're not going to be seeing that. And if that happens, you know, God save us. [laugh].But to your point, one of the things we do still want to be doing is look at trends, literally on a week-over-week basis because even a one percentage move is a pretty significant amount, if you think about it, which could be funding some other aspects of the business, which we would prefer to be investing on. So, we do want to have enough rigor and controls in place in our technical stack to identify and alert when something is off track. And it becomes challenging when you start using those higher-order services from your public cloud provider because there's no clear insights on how do you, kind of, parse that information. One of the biggest challenges we had at Pinterest was tying ownership to all these things.No, using tags is not going to cut it. It was so difficult for us to get to a point where we could put some sense of ownership in all the things and the resources people are using, and then subsequently have the right conversation with our ads infrastructure teams, or our product teams to help drive the cost improvements we want to be seeing. And I wouldn't be surprised if that's not a challenge already, even for the smaller companies who have bills in the tunes of tens and thousands, right?Corey: It is. It's predicting the spend and trying to categorize it appropriately; that's the root of all AWS bill panic on the corporate level. It's not that the bill is 20% higher, so we're going to go broke. Most companies spend far more on payroll than they do on infrastructure—as you mentioned with Netflix, content is a significantly larger [laugh] expense than any of those things; real estate, it's usually right up there too—but instead it's, when you're trying to do business forecasting of, okay, if we're going to have an additional 1000 monthly active users, what will the cost for us be to service those users and, okay, if we're seeing a sudden 20% variance, if that's the new normal, then well, that does change our cost projections for a number of years, what happens? When you're public, there starts to become the question of okay, do we have to restate earnings or what's the deal here?And of course, all this sidesteps past the unfortunate reality that, for many companies, the AWS bill is not a function of how many customers you have; it's how many engineers you hired. And that is always the way it winds up playing out for some reason. “It's why did we see a 10% increase in the bill? Yeah, we hired another data science team. Oops.” It's always seems to be the data science folks; I know I'd beat up on those folks a fair bit, and my apologies. And one day, if they analyze enough of the data, they might figure out why.Micheal: So, this is where I want to give a shout out to our data science team, especially some of the engineers working in the Infrastructure Governance Team putting these charts together, helping us derive insights. So, definitely props to them.I think there's a great segue into the point you made. As you add more engineers, what is the impact on the bottom line? And this is one of the things actually as part of engineering productivity, we think about as well on a long-term basis. Pinterest does have over 1000-plus engineers today, and to large degree, many of them actually have their own EC2 instances today. And I wouldn't say it's a significant amount of cost, but it is a large enough number, were shutting down a c5.9xl can actually fund a bunch of conference tickets or something else.And then you can imagine that sort of the scale you start working with at one point. The nuance here is though, you want to make sure there's enough flexibility for these engineers to do their local development in a sustainable way, but when moving to, say production, we really want to tighten the flexibility a bit so they don't end up doing what you just said, spin up a bunch of machines talking to the API directly which no one will be aware of.I want to share a small anecdote because when back in the day, this was probably four years ago, when we were doing some analysis on our bills, we realized that there was a huge jump every—I believe Wednesday—in our EC2 instances by almost a factor of, like, 500 to 600 instances. And we're like, “Why is this happening? What is going on?” And we found out there was an obscure job written by someone who had left the company, calling an EC2 API to spin up a search cluster of 500 machines on-demand, as part of pulling that ETL data together, and then shutting that cluster down. Which at times didn't work as expected because, you know, obviously, your Hadoop jobs are very predictable, right?So, those are the things we were dealing with back in the day, and you want to make sure—since then—this is where engineering productivity as team starts coming in that our job is to enable every engineer to be doing their best work across code building and deploying the services. And we have done this.Corey: Right. You and I can sit here and have an in-depth conversation about the intricacies of AWS billing in a bunch of different ways because in different ways we both specialize in it, in many respects. But let's say that Pinterest theoretically was foolish enough to hire me before I got into this space as an engineer, for terrifying reasons. And great. I start day one as a typical software developer if such a thing could be said to exist. How do you effectively build guardrails in so that I don't inadvertently wind up spinning up all the EC2 instances available to me within an account, which it turns out are more than one might expect sometimes, but still leave me free to do my job without effectively spending a nine-month safari figuring out how AWS bills work?Micheal: And this is why teams like ours exist, to help provide those tools to help you get started. So today, we actually don't let anyone directly use AWS APIs, or even use the UI for that matter. And I think you'll soon realize, the moment you hit, like, probably 30 or 40 people in your organization, you definitely want to lock it down. You don't want that access to be given to anyone or everyone. And then subsequently start building some higher-order tools or abstraction so people can start using that to control effectively.In this case, if you're a new engineer, Corey, which it seems like you were, at some point—Corey: I still write code like I am, don't worry.Micheal: [laugh]. So yes, you would get access to our internal tool to actually help spin up what we call is a dev app, where you get a chance to, obviously, choose the instance size, not the instance type itself, and we have actually constrained the instance types we have approved within Pinterest as well. We don't give you the entire list you get a chance to choose and deploy to. We actually have constraint to based on the workload types, what are the instance types we want to support because in the future, if we ever want to move from c3 to c5—and I've been there, trust me—it is not an easy thing to do, so you want to make sure that you're not letting people just use random instances, and constrain that by building some of these tools. As a new engineer, you would go in, you'd use the tool, and actually have a dev app provisioned for you with our Pinterest image to get you started.And then subsequently, we'll obviously shut it down if we see you not being using it over a certain amount of time, but those are sort of the guardrails we've put in over there so you never get a chance to directly ever use the EC2 APIs, or any of those AWS APIs to do certain things. The similar thing applies for S3 or any of the higher-order tools which AWS will provide, too.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of "Hello, World" demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking databases, observability, management, and security.And - let me be clear here - it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself all while gaining the networking load, balancing and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build.With Always Free you can do things like run small scale applications, or do proof of concept testing without spending a dime. You know that I always like to put asterisks next to the word free. This is actually free. No asterisk. Start now. Visit https://snark.cloud/oci-free that's https://snark.cloud/oci-free.Corey: How does that interplay with AWS launches yet another way to run containers, for example, and that becomes a valuable potential avenue to get some business value for a developer, but the platform you built doesn't necessarily embrace that capability? Or they release a feature to an existing tool that you use that could potentially be a just feature capability story, much more so than a cost savings one. How do you keep track of all of that and empower people to use those things so they're not effectively trying to reimplement DynamoDB on top of EC2?Micheal: That's been a challenge, actually, in the past for us because we've always been very flexible where engineers have had an opportunity to write their own solutions many a times rather than leveraging the AWS services, and of late, that's one of the reasons why we have an infrastructure organization—an extremely lean organization for what it's worth—but then still able to achieve outsized outputs. Where we evaluate a lot of these use cases, as they come in and open up different aspects of what we want to provide say directly from AWS, or build certain abstractions on top of it. Every time we talk about containers, obviously, we always associate that with something like Kubernetes and offerings from there on; we realized that our engineers directly never ask for those capabilities. They don't come in and say, “I need a new container orchestration system. Give that to me, and I'm going to be extremely productive.”What people actually realize is that if you can provide them effective tools and that can help them get their job done, they would be happy with it. For example, like I said, our deployment system, which is actually an open-source system called Teletraan. That is the bread and butter at Pinterest at which my team runs. We operate 100,000-plus machines. We have actually looked into container orchestration where we do have a dedicated Kubernetes team looking at it and helping certain use cases moved there, but we realized that the cost of entire migrations need to be evaluated against certain use cases which can benefit from being on Kubernetes from day one. You don't want to force anyone to move there, but give them the right incentives to move there. Case in point, let's upgrade your OS. Because if you're managing machines, obviously everyone loves to upgrade their OSes.Corey: Well, it's one of the things I love savings plans versus RIs; you talk about the c3 to c5 migration and everyone has a story about one of those, but the most foolish or frustrating reason that I ever saw not to do the upgrade was what we bought a bunch of Reserved Instances on the C3s and those have a year-and-a-half left to run. And it's foolish not on the part of customers—it's economically sound—but on the part of AWS where great, you're now forcing me to take a contractual commitment to something that serves me less effectively, rather than getting out of the way and letting me do my job. That's why it's so important to me at least, that savings plans cover Fargate and Lambda, I wish they covered SageMaker instead of SageMaker having its own thing because once again, you're now architecturally constrained based upon some ridiculous economic model that they have imposed on us. But that's a separate rant for another time.Micheal: No, we actually went through that process because we do have a healthy balance of how we do Reserved Instances and how we look at on-demand. We've never been big users have spot in the past because just the spot market itself, we realized that putting that pressure on our customers to figure out how to manage that is way more. When I say customers, in this case, engineers within the organization.Corey: Oh, yes. “I want to post some pictures on Pinterest, so now I have to understand the spot market. What?” Yeah.Micheal: [laugh]. So, in this case, when we even we're moving from C3 to C5—and this is where the partnership really plays out effectively, right, because it's also in the best interest of AWS to deprecate their aging hardware to support some of these new ones where they could also be making good enough premium margins for what it's worth and give the benefit back to the user. So, in this case, we were able to work out an extremely flexible way of moving to a C5 as soon as possible, get help from them, actually, in helping us do that, too, allocating capacity and working with them on capacity management. I believe at one point, we were actually one of the largest companies with a C3 footprint and it took quite a while for us to move to C5. But rest assured, once we moved, the savings was just immense. We were able to offset any of those RI and we were able to work behind the scenes to get that out. But obviously, not a lot of that is considered in a small-scale company just because of, like you said, those constraints which have been placed in a contractual obligation.Corey: Well, this is an area in which I will give the same guidance to companies of your scale as well as small-scale companies. And by small-scale, I mean, people on the free tier account, give or take, so I do mean the smallest of the small. Whenever you wind up in a scenario where you find yourself architecturally constrained by an economic barrier like this, reach out to your account manager. I promise you have one. Every account, even the tiny free tier accounts, have an account manager.I have an account manager, who I have to say has probably one of the most surreal jobs that AWS, just based upon the conversations I throw past him. But it's reaching out to your provider rather than trying to solve a lot of this stuff yourself by constraining how you're building things internally is always the right first move because the worst case is you don't get anywhere in those conversations. Okay, but at least you explored that, as opposed to what often happens is, “Oh, yeah. I have a switch over here I can flip and solve your entire problem. Does that help anything?”Micheal: Yeah.Corey: You feel foolish finding that out only after nine months of dedicated work, it turns out.Micheal: Which makes me wonder, Corey. I mean, do you see a lot of that happening where folks don't tend to reach out to their account managers, or rather treat them as partners in this case, right? Because it sounds like there is this unhealthy tension, I would say, as to what is the best help you could be getting from your account managers in this case.Corey: Constantly. And the challenge comes from a few things, in my experience. The first is that the quality of account managers and the technical account managers—the folks who are embedded many cases with your engineering teams in different ways—does vary. AWS is scaling wildly and bursting at the seams, and people are hard to scale.So, some are fantastic, some are decidedly less so, and most folks fall somewhere in the middle of that bell curve. And it doesn't take too many poor experiences for the default to be, “Oh, those people are useless. They never do anything we want, so why bother asking them?” And that leads to an unhealthy dynamic where a lot of companies will wind up treating their AWS account manager types as a ticket triage system, or the last resort of places that they'll turn when they should be involved in earlier conversations.I mean, take Pinterest as an example of this. I'm not sure how many technical account managers you have assigned to your account, but I'm going to go out on a limb and guess that the ratio of technical account managers to engineers working on the environment is incredibly lopsided. It's got to be a high ratio just because of the nature of how these things work. So, there are a lot of people who are actively working on things that would almost certainly benefit from a more holistic conversation with your AWS account team, but it doesn't occur to them to do it just because of either perceived biases around levels of competence, or poor experiences in the past, or simply not knowing the capabilities that are there. If I could tell one story around the AWS account management story, it would be talk to folks sooner about these things.And to be clear, Pinterest has this less than other folks, but AWS does themselves no favors by having a product strategy of, “Yes,” because very often in service of those conversations with a number of companies, there is the very real concern of are they doing research so that they can launch a service that competes with us? Amazon as a whole launching a social network is admittedly one of the most hilarious ideas I [laugh] can come up with and I hope they take a whack at it just to watch them learn all these lessons themselves, but that is again, neither here nor there.Micheal: That story is very interesting, and I think you mentioned one thing; it's just that lack of trust, or even knowing what the account managers can actually do for you. There seems to be just a lack of education on that. And we also found it the hard way, right? I wouldn't say that Pinterest figured this out on day one. We evolved sort of a relationship over time. Yes, our time… engagements are, sort of, lopsided, but we were able to negotiate that as part of deals as we learned a bit more on what we can and we cannot do, and how these individuals are beneficial for Pinterest as well. And—Corey: Well, here's a question for you, without naming names—and this might illustrate part of the challenge customers have—how long has your account manager—not the technical account managers, but your account manager—been assigned to your account?Micheal: I've been at Pinterest for five years and I've been working with the same person. And he's amazing.Corey: Which is incredibly atypical. At a lot of smaller companies, it feels like, “Oh, I'm your account manager being introduced to you.” And, “Are you the third one this year? Great.” What happens is that if the account manager excels, very often they get promoted and work with a smaller number of accounts at larger spend, and whereas if they don't find that AWS is a great place for them for a variety of reasons, they go somewhere else and need to be backfilled.So, at the smaller account, it's, “Great. I've had more account managers in a year than you've had in five.” And that is often the experience when you start seeing significant levels of rotation, especially on the customer engineering side where you wind up with you have this big kickoff, and everyone's aware of all the capabilities and you look at it three years later, and not a single person who was in that kickoff is still involved with the account on either side, and it's just sort of been evolving evolutionarily from there. One thing that we've done in some of our larger accounts as part of our negotiation process is when we see that the bridges have been so thoroughly burned, we will effectively request a full account team cycle, just because it's time to get new faces in where the customer, in many cases unreasonably, is not going to say, “Yeah but a year-and-a-half ago you did this terrible thing and we're still salty about it.” Fine, whatever. I get it. People relationships are hard. Let's go ahead and swap some folks out so that there are new faces with new perspectives because that helps.Micheal: Well, first off, if you had so many switches in account manager, I think that's something speaks about [laugh] how you've been working, too. I'm just kidding. There are a bu—Corey: Entirely possible. In seriousness, yes. But if you talk to—like, this is not just me because in my case, yeah, I feel like my account manager is whoever drew the short straw that week because frankly, yeah, that does seem like a great punishment to wind up passing out to someone who is underperforming. But for a lot of folks who are in the mid-tier, like, spending $50 to $100,000 a month, this is a very common story.Micheal: Yeah. Actually, we've heard a bit about this, too. And like you said, I think maintaining context is the most thing. You really want your account manager to vouch for you, really be your champion in those meetings because AWS, like you said is so large, getting those exec time, and reviews, and there's so many things that happen, your account manager is the champion for you, or right there. And it's important and in fact in your best interest to have a great relationship with them as well, not treat them as, oh yet another vendor.And I think that's where things start to get a bit messy because when you start treating them as yet another vendor, there is no incentive for them to do the best for you, too. You know, people relationships are hard. But that said though, I think given the amount of customers like these cloud companies are accruing, I wouldn't be surprised; every account manager seems to be extremely burdened. Even in our case, although I've been having a chance to work with this one person for a long time, we've actually expanded. We have now multiple account managers helping us out as we've started scaling to use certain aspects of AWS which we've never explored before.We were a bit constrained and reserved about what service we want to use because there have been instances where we have tried using something and we have hit the wall pretty immediately. API rate limits, or it's not ready for primetime, and we're like, “Oh, my God. Now, what do we do?” So, we have a bit more cautious. But that said, over time, having an account manager who understands how you work, what scale you have, they're able to advocate with the internal engineering teams within the cloud provider to make the best of supporting you as a customer and tell that success story all the way out.So yeah, I can totally understand how this may be hard, especially for those small companies. For what it's worth, I think the best way to really think about it is not treat them as your vendor, but really go out on a limb there. Even though you signed a deal with them, you want to make sure that you have the continuing relationship with them to have—represent your voice better within the company. Which is probably hard. [laugh].Corey: That's always the hard part. Honestly, if this were the sort of thing that were easy to automate, or you could wind up building out something that winds up helping companies figure out how to solve these things programmatically, talk about interesting business problems that are only going to get larger in the fullness of time. This is not going away, even if AWS stopped signing up new customers entirely right now, they would still have years of growth ahead of them just from organic growth. And take a company with the scale of Pinterest and just think of how many years it would take to do a full-on exodus, even if it became priority number one. It's not realistic in many cases, which is why I've never been a big fan of multi-cloud as an approach for negotiation. Yeah, AWS has more data on those points than any of us do; they're not worried about it. It just makes you sound like an unsophisticated negotiator. Pick your poison and lean in.Micheal: That is the truth you just mentioned, and I probably want to give a call out to our head of infrastructure, [Coburn 00:42:13]. He's also my boss, and he had brought this perspective as well. As part of any negotiation discussions, like you just said, AWS has way more data points on this than what we think we can do in terms of talking about, “Oh, we are exploring this other cloud provider.” And it's—they would be like, “Yeah. Do tell me more [laugh] how that's going.”And it's probably in the best interest to never use that as a negotiation tactic because they clearly know the investments that's going to build on what you've done, so you might as well be talking more—again, this is where that relationship really plays together because you want both of them to be successful. And it's in their best interest to still keep you happy because the good thing about at least companies of our size is that we're probably, like, one phone call away from some of their executive team, where we could always talk about what didn't work for us. And I know not everyone has that opportunity, but I'm really hoping and I know at least with some of the interactions we've had with the AWS teams, they're actively working and building that relationship more and more, giving access to those customer advisory boards, and all of them to have those direct calls with the executives. I don't know whether you've seen that in your experience in helping some of these companies?Corey: Have a different approach to it. It turns out when you're super loud and public and noisy about AWS and spend too much time in Seattle, you start to spend time with those people on a social basis. Because, again, I'm obnoxious and annoying to a lot of AWS folks, but I'm also having an obnoxious habit of being right in most of the things I'm pointing out. And that becomes harder and harder to ignore. I mean, part of the value that I found in being able to do this as a consultant is that I begin to compare and contrast different customer environments on a consistent ongoing basis.I mean, the reason that negotiation works well from my perspective is that AWS does a bunch of these every week, and customers do these every few years with AWS. And well, we do an awful lot of them, too, and it's okay, we've seen different ways things can get structured and it doesn't take too long and too many engagements before you start to see the points of commonality in how these things flow together. So, when we wind up seeing things that a customer is planning on architecturally and looking to do in the future, and, “Well, wait a minute. Have you talked to the folks negotiating the contract about this? Because that does potentially have bearing and it provides better data than what AWS is gathering just through looking at overall spend trends. So yeah, bring that up. That is absolutely going to impact the type of offer you get.”It just comes down to understanding the motivators that drive folks and it comes down to, I think understanding the incentives. I will say that across the board, I have never yet seen a deal from AWS come through where it was, “Okay, at this point you're just trying to hoodwink the customer and get them to sign on something that doesn't help them.” I've seen mistakes that can definitely lead to that impression, and I've seen areas where they're doing data is incomplete and they're making assumptions that are not borne out in reality. But it's not one of those bad faith type—Micheal: Yeah.Corey: —of negotiations. If it were, I would be framing a lot of this very differently. It sounds weird to say, “Yeah, your vendor is not trying to screw you over in this sense,” because look at the entire IT industry. How often has that been true about almost any other vendor in the fullness of time? This is something a bit different, and I still think we're trying to grapple with the repercussions of that, from a negotiation standpoint and from a long-term business continuity standpoint, when your faith is linked—in a shared fate context—with your vendor.Micheal: It's in their best interest as well because they're trying to build a diversified portfolio. Like, if they help 100 companies, even if one of them becomes the next Pinterest, that's great, right? And that continued relationship is what they're aiming for. So, assuming any bad faith over there probably is not going to be the best outcome, like you said. And two, it's not a zero-sum game.I always get a sense that when you're doing these negotiations, it's an all-or-nothing deal. It's not. You have to think they're also running a business and it's important that you as your business, how okay are you with some of those premiums? You cannot get a discount on everything, you cannot get the deal or the numbers you probably want almost everything. And to your point, architecturally, if you're moving in a certain direction where you think in the next three years, this is what your usage is going to be or it will come down to that, obviously, you should be investing more and negotiating that out front rather than managed NAT [laugh] gateways, I guess. So, I think that's also an important mindset to take in as part of any of these negotiations. Which I'm assuming—I don't know how you folks have been working in the past, but at least that's one of the key items we have taken in as part of any of these discussions.Corey: I would agree wholeheartedly. I think that it just comes down to understanding where you're going, what's important, and again in some cases knowing around what things AWS will never bend contractually. I've seen companies spend six weeks or more trying to get to negotiate custom SLAs around services. Let me save everyone a bunch of time and money; they will not grant them to you.Micheal: Yeah.Corey: I promise. So, stop asking for them; you're not going to get them. There are other things they will negotiate on that they're going to be highly case-dependent. I'm hesitant to mention any of them just because, “Well, wait a minute, we did that once. Why are you talking about that in public?” I don't want to hear it and confidentiality matters. But yeah, not everything is negotiable, but most things are, so figuring out what levers and knobs and dials you have is important.Micheal: We also found it that way. AWS does cater to their—they are a platform and they are pretty clear in how much engagement—even if we are one of their top customers, there's been many times where I know their product managers have heavily pushed back on some of the requests we have put in. And that makes me wonder, they probably have the same engagement even with the smallest of customers, there's always an implicit assumption that the big fish is trying to get the most out of your public cloud providers. To your point, I don't think that's true. We're rarely able to negotiate anything exclusive in terms of their product offerings just for us, if that makes sense.Case in point, tell us your capacity [laugh] for x instances or type of instances, so we as a company would know how to plan out our scale-ups or scale-downs. That's not going to happen exclusively for you. But those kind of things are just, like, examples we have had a chance to work with their product managers and see if, can we get some flexibility on that? For what it's worth, though, they are willing to find a middle ground with you to make sure that you get your answers and, obviously, you're being successful in your plans to use certain technologies they offer or [unintelligible 00:48:31] how you use their services.Corey: So, I know we've gone significantly over time and we are definitely going to do another episode talking about a lot of the other things that you're involved in because I'm going to assume that your full-time job is not worrying about the AWS bill. In fact, you do a fair number of things beyond that; I just get stuck on that one, given that it is but I eat, sleep, breathe, and dream about.Micheal: Absolutely. I would love to talk more, especially about how we're enabling our engineers to be extremely productive in this new world, and how we want to cater to this whole cloud-native environment which is being created, and make sure people are doing their best work. But regardless, Corey, I mean, this has been an amazing, insightful chat, even for me. And I really appreciate you having me on the show.Corey: No, thank you for joining me. If people want to learn more about what you're up to, and how you think about things, where can they find you? Because I'm also going to go out on a limb and assume you're also probably hiring, given that everyone seems to be these days.Micheal: Well, that is true. And I wasn't planning to make a hiring pitch but I'm glad that you leaned into that one. Yes, we are hiring and you can find me on Twitter at twitter dot com slash M-I-C-H-E-A-L. I am spelled a bit differently, so make sure you can hit me up, and my DMs are open. And obviously, we have all our open roles listed on pinterestcareers.com as well.Corey: And we will, of course, put links to that in the [show notes 00:49:45]. Thank you so much for taking the time to speak with me today. I really appreciate it.Micheal: Thank you, Corey. It was really been great on your show.Corey: And I'm sure we'll do it again in the near future. Micheal Benedict, Head of Engineering Productivity at Pinterest. I am Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with a long rambling comment about exactly how many data centers Pinterest could build instead.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Screaming in the Cloud
That Datadog Will Hunt with Dann Berg

Screaming in the Cloud

Play Episode Listen Later Nov 4, 2021 41:24


About DannDann Berg is a Senior CloudOps Analyst at Datadog, and has nearly a decade of experience working in the cloud and optimizing multi-million dollar budgets. He is also an active member of the larger technical community, hosting the monthly New York City FinOps Meetup, and has been published multiple times in places such as MSNBC, Fox News, NPR, and others. When he's not saving companies millions of dollars, he's writing plays, and has had two full-lengh plays produced in New York City and China.Links: Datadog: https://www.datadoghq.com Personal Website: https://dannb.org LinkedIn: https://www.linkedin.com/in/dannberg/ Twitter: https://twitter.com/dannberg Monthly newsletter: https://dannb.org/newsletter/ Previous SITC episode with Dann Berg, Episode 51: https://www.lastweekinaws.com/podcast/screaming-in-the-cloud/episode-51-size-of-cloud-bill-not-about-number-of-customers-but-number-of-engineers-you-ve-hired/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friends at Vultr. Spelled V-U-L-T-R because they're all about helping save money, including on things like, you know, vowels. So, what they do is they are a cloud provider that provides surprisingly high performance cloud compute at a price that—while sure they claim its better than AWS pricing—and when they say that they mean it is less money. Sure, I don't dispute that but what I find interesting is that it's predictable. They tell you in advance on a monthly basis what it's going to going to cost. They have a bunch of advanced networking features. They have nineteen global locations and scale things elastically. Not to be confused with openly, because apparently elastic and open can mean the same thing sometimes. They have had over a million users. Deployments take less that sixty seconds across twelve pre-selected operating systems. Or, if you're one of those nutters like me, you can bring your own ISO and install basically any operating system you want. Starting with pricing as low as $2.50 a month for Vultr cloud compute they have plans for developers and businesses of all sizes, except maybe Amazon, who stubbornly insists on having something to scale all on their own. Try Vultr today for free by visiting: vultr.com/screaming, and you'll receive a $100 in credit. Thats v-u-l-t-r.com slash screaming.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of "Hello, World" demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking databases, observability, management, and security.And - let me be clear here - it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself all while gaining the networking load, balancing and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build.With Always Free you can do things like run small scale applications, or do proof of concept testing without spending a dime. You know that I always like to put asterisks next to the word free. This is actually free. No asterisk. Start now. Visit https://snark.cloud/oci-free that's https://snark.cloud/oci-free.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. If there's one thing that I love, it is certainly not AWS billing, but for better or worse, that's where my career has led me. Way back in Episode 51, I had Dann Berg, the CloudOps analyst at Datadog. And now he's back for more. Things have changed. He's now a senior CloudOps analyst, and I'm hoping my jokes have gotten better. Dann, thanks for being bold enough to come out and find out.Dann: Yeah. I'm excited to see if these jokes have gotten better. That's the main reason for coming back.Corey: Exactly. Because it turns out that death, taxes, and AWS bills are the things that are inevitable and never seem to change.Dann: Yeah. They just keep coming. They never stop, and they're always slightly different than you expect. I guess, just like death and taxes.Corey: So, when we spoke back in, I want to say 2019 is when it aired, so probably that—ish—is when we had the conversation, if not a little bit before that, you were effectively a team of one, and as mentioned, had the CloudOps analyst title. Now, you're a senior CloudOps analyst, which I assume just means you're older. Is the team larger as well? What does that process look like? How has it evolved in the last couple years?Dann: Yeah, it's been interesting, especially being a single organization and that organization being Datadog, that to be able to grow the team a little bit. So, as you said, it was just me. Now, it's a total of four people, including myself, so three others. And, yeah, it's been interesting just in terms of my own professional development, being able to identify what needs to be done, how much capacity I have, and being able to grow it over time, especially in this fairly new space of being specifically focused on cloud cost billing. So, kind of that bridge between engineering and finance, which itself is kind of a fairly new space, still.Corey: It is. And my favorite part of having these conversations with folks who have no idea what this space is, is learning—when I was starting—out how to talk about this in a way that didn't lead down weird paths. It's, “Oh, you save money on Amazon bills? Can you help me save money on socks?” It's like, “No. Well, yes. Get the Prime card, it gives you 5% off. But no.” And yeah, I talk about camelcamelcamel and other ways of working around the retail side, but that's not really what I do.It's similar to back when I was doing SRE-style work. I made it a point never to talk about being someone involved in working in tech, or suddenly you're the neighborhood printer repair person. Similarly, you have, I guess, gone in a strange direction because you weren't, to my recollection, someone who had a strong SRE background. That's not where you came from in the traditional sense, is it?Dann: No, not an SRE background at all. Yeah, I mean, it's really interesting. So, talking about this space, I mean, people are calling it a lot of different things, cloud economics, the term FinOps—financial operations—is being used a lot, now—Corey: Cloud financial management is another popular one. Oh, swing a dead cat, you'll hit 15 different words, and I give—my advice on that, even though I hate some of the terms is, cool. If people are going to pay you to have a title, even if you think it's ridiculous, you can take the money or you can die on a petty naming hill and here we are.Dann: Yeah. And it's interesting because the role that I was hired for at Datadog was very much this niche, very specific role that I didn't realize was a niche, very specific role at the time. So previously, I was at a company and I was building out their data centers, so I was working with vendors, buying servers, sometimes going on-site, installing, racking those, dealing with RMAs. And I was getting more involved as their cloud usage was growing and bringing some of those hardware capitalization cost procedures to the cloud. And so I found myself in this kind of niche role in my previous company.And a Datadog, they basically had the exact same role that was dealing with all of the billing stuff around the cloud—kind of from an engineering perspective because it was on the engineering team—but working closely with finance, and I was like, “Oh, these are the skills that I have.” And it kind of fit perfectly. And it wasn't until after I got to Datadog and was doing more research about this specific space that I discovered just how wide open it was. And I mean, meeting you was one of the earliest things that I did in the industry. Discovering the FinOps Foundation and a few other things has kind of like opened my eyes to this as an actual career path.Corey: It's an expensive problem that isn't going away anytime soon, and it is foundational and core to the entire rest of how companies are building things these days. My argument has been for a while that when it comes to cloud, cost and architecture are the exact same thing. You don't have the deep SRE architect background, but you're also now a member of a four-person team. Does everyone in the team have the same skill set as you, or do you wind up effectively tagging in subject matter experts from different areas? How is the team composed? People love to ask me this question, and I strongly believe there's no one way to do it. But what's your answer?Dann: Yeah, I mean, the team works very much in terms of everybody kind of taking on tasks that they need to do, but we did hire for specific skill sets when we tried to find people. So, the first person that we hired, we wanted them to have more of a developer engineer type background, writing code, stuff like that. The third hire, we were looking for somebody that was more of a generalist. I've seen myself more as a generalist in the space; anything that's going on, I can pick it up and make some progress on it and build something out. And then the fourth person, we were lacking some of the deeper FP&A or FinOps experience, and so we found somebody with more of that kind of background and less of the engineering experience, but they were eager to, kind of, move from finance into more of an engineering role. And I feel like this is the perfect role for that because I feel like there are a lot of non-engineers that want to break into engineering and don't really know how to do it. And if you are in finance, in FP&A, finding one of these more cloud-cost-optimization-specific roles is the great way to bridge that gap, I feel.Corey: The last time we spoke, I was independent, doing this all myself, and it turns out that taking all of the things that make me and trying to find those in other people is a relatively heavy lift, even if you discount the things like ‘obnoxious on Twitter.' So, how do you start decomposing that? Well, now we're a dozen people and we've found ways to do it. But by and large in our experience, for the way that we interact—and I want to get to that in a second—is that it's easier for us to teach engineers how finance works than it is the opposite direction. And there are exceptions to that, and as we scale, I can easily see a day in the near future where that is no longer the case.However, we also have two very specific styles of engagement. We do our cost optimization projects, where we go into an environment and, “Oh, fix this. Turn that thing off. Do you really need eight copies of those four petabytes of data? Oh, you didn't realize they were there. Great, maybe delete it.” And we look like wizards from the future and things are great.The other project that we do is contract negotiation with AWS, especially at large scale. It's never as simple as people would have you believe because, “Oh, you're doing co-marketing efforts, and you have a very specific use case, and there are business partnerships on 15 different levels, and that all factors into how this works.” It's nuanced and challenging, and of course, because it's a series of anecdata, I can't really tell too many stories in public about that. But those are the two things that we wind up focusing on. You are focusing on a very different problem.You're not moving from company to company, basically reimplementing the same global problem, solving it locally for them. You are embedded in an account for the duration, almost four years now by my count. And, “Okay, I guess I could just do a whole bunch of cost optimization projects on a quarterly basis in an environment like that,” doesn't seem like it solves the problem in any meaningful way. What does your team do?Dann: Yeah. Well, I mean, that's such an interesting question. Just in terms of—yeah, if you're doing consulting, you're starting from square one every time you get a new contract, a new engagement, and being at the same company for, like you said, about four years, going on four years now, you really have a chance to dive in and think about, “Okay, what does it mean to work cloud cost optimization into just the regular business cycle of how it works?” Because I mean, you have the triangle that everybody's familiar with: things can either be cheaper, faster, efficient and at different stages in the product lifecycle, you want to be focusing on these areas, more or less. And so, on our team, the different things that I'm thinking about is, first is visibility, is you want to provide engineers visibility into their cost. And not just numbers, right? Actionable visibility where if something needs to change, they need to do something, they know what that is.And a lot of the times, that means not just costs, but also efficiency. So, these are the metrics that this particular application should be scaling against. As this application grows, as usage grows, are we remaining as cost-efficient? Then there's also the piece—as you're saying—like discovering things within the infrastructure that, “Hey, if we make this change, or if you turn this off, if we do things this way, we'll save a bunch of money. Let's do those.”There's things like reservations, committed use discounts for GCP, all of those kinds of things we manage. And then dealing closely with verifying our bill, working with finance—FP&A—on cost modeling forecasting, both short-term—like, within a month; like, what are we going to be at the end of this month and it's the 10th right now?—and also, what does our next quarter look like? What are our next two years look like? And that bleeds into the contract negotiations, those kind of things as well.So, I mean, it's setting up the cycles of how do you prioritize this work? What is the company focusing on at the time? And what can you do when the company is not focusing explicitly on deciding to save money?Corey: One of the more interesting aspects of my work that I didn't expect is, whenever I wind up starting an engagement, or even in the prospect stage, I love asking the dumbest possible questions I can think of because it turns out they're not. And the most common one that I always love to start with is, “Oh, okay. Your AWS bill is too high. Why do you care?” And that often takes people aback, but once you dig down underneath the surface just a little bit, it becomes pretty clear that the actual goal is not that it's too much money—because spoiler, payroll always cost more than infrastructure—instead, it's, “How do I think about this? How do I rationalize what the additional costs are going to be per thousand monthly active users or whatever metric it is you're choosing to use?”And how do you wind up forecasting that because the old days of data centers where you—“Well, we're going to spend a boatload of money, and then we'll have capacity for the next, ehh, two years, maybe down to eighteen months, depending on growth,” that's easier for companies to rationalize around, rather than this idea of incremental cost on a per-unit basis, but not exactly because it also turns out that architecture changes, problems of scale, AWS pricing changes from time to time, all tend to impact that. What I think is not well understood in this space is that yeah, if you have a 20% overage this month, people are going to have some serious questions, but they're also going to have those same questions if you're 20% low.Dann: Yeah. I mean, understanding why people care about the cost is definitely the first step because with a single company, so it's just constantly looking at the numbers rather than understanding exactly what motivations a company has to contact somebody like you, like a consultant, right? Because usually, I imagine that it's going to be a bill, maybe two bills, three bills come in, and they keep going up and up and up, and they need to go down. And they're going to have an explicit reason why it needs to go down; finance is going to say, “Margins are x, y, and z,” or, “Revenue has done this; our costs can't do this.” There's going to be explicit reasons because if there aren't reasons, then they shouldn't necessarily be focusing on costs at that moment in time.What you want to do is have—I mean, this is way more complicated than just saying it out loud, but have a culture of cloud cost mindfulness, where people aren't just spinning up resources willy nilly. But also, my goal is for people not to have to really think about cost that much other than just in a way that helps them do their work. Because I mean, I want engineers to be able to build stuff and build stuff fast—that's what the cloud is all about—but I also want to be able to do it in a way that isn't inappropriately high in cost.Corey: I have my thoughts on this, and I've shared them before and I'll dive into them again, but how do you approach that? If Datadog makes a grievous error and hires me to write code somewhere as an engineer, what is the, I guess, cost approach training for me as I wind up going through my onboarding as part of an SRE team or an application team?Dann: I mean, this feels so basic as to not even be the right answer, but honestly, visibility is the easiest and best thing that you can give people, and so we've built out some visibility reports that engineers get on a regular basis. We also meet with our top—what is it—ten or fifteen spending internal engineering teams on a monthly basis to go over those costs so that they understand what they're looking at so that we understand the context behind it, so that we can understand what's on the roadmap going forward so that when things in the cost happen, we're aware. And then we're just staying on top of things. And if we have questions, we have an open dialogue with engineers and things like that.In an ideal space, it would be great to have cost, I guess, more fit into the product development lifecycle in a more deeply ingrained way, but at the same time, I really don't want to serve as a gatekeeper. Our goal is not to stop any sort of engineering process. And we haven't needed to do anything like that although I guess every company is going to be different in terms of what their needs are. But yeah, I'm totally happy to being a little bit more reactionary in terms of looking at the numbers and responding, and then proactive just in terms of the regular communication with people.Corey: I tend to take the perspective that engineers need to know enough about cost to maybe fill an index card at most because you don't want them, I guess, over-fixating on it. Left to my own devices in my personal account, I'll see a $7 a month bill and, “Oh, I'm going to spend two weeks knocking that down to $4.” And of course, I can do it, but is that the best use of my time? Absolutely not.Very often what is a lot of money to an engineer is absolutely not to the business. And vice versa when you bring in a data science team; it's, “Oh, yeah, we need at least four more exabytes of data because we never learned to do a join properly.” Yeah, maybe don't do that. Understanding the difference between those two approaches is key. But I've always been of the mindset that I would rather bias for letting developers build and experiment and have things that catch outsized things quickly, then trying to wind up putting a culture of fear around cost because I'd much rather see whether the thing they're trying to build is possible to build, then go back and optimize it later, once that's proven out. But again, this is a nuanced thing.Everyone seems to think I have this back pocket answer that will apply to all companies. And you've been doing this at Datadog for almost four years with a team of people. I am an outsider; I see the global trend, I see what works in different ways in different companies, but the idea that I can sit down and say, “Oh. Well, clearly the thing you're doing is completely wrong because that's not how I think about it,” is the hallmark of a terrible consultant. There are reasons that things are the way that they are and it's generally not that people are expecting to do a terrible job today. You know, unless they work in the Facebook ethics department, which is neither here nor there.Dann: Yeah, I mean, like I said, the product lifecycle, when you're building something new, you want to go as fast as possible. When you're launching it, you want it to be as reliable as possible. Once you're launched, once you're reliable, then you can start focusing on costs is, kind of like, not the universal rule, but kind of the flow that I tend to see. So, as you're at a company that is regularly innovating, creating new products, going through that cycle, you're going to have these kind of periods.As well as you have the products that have been around. There's a lot of legacy code, there's a lot of stuff going on, that maybe isn't the best, or some efficiency work that has been deprioritized for whatever reason, that maybe it's time to start considering doing this. So, keeping track of all of that. And like I said, if for whatever reason the business wants to focus on cloud cost efficiency, or a team has decided that in a particular quarter or for a particular reason they want to focus on that, being able to assist as much as you can, being able to save all that work so that there's kind of like a queue that you can go to when it is time to focus on cost efficiency stuff.Corey: So, here's a fun one for you. As of the time of this recording, it's a couple weeks old, but if you're anything like what we do here for some of our more sophisticated clients, we do occasionally build out prediction models, models of economics that wind up defining how some architectural patterns should be addressed, et cetera, et cetera. What's always fun is the large clients who have this significant level of spend on an outlier service. Every once in a while—it was great that we got to do a deep dive into the Washington Post's use of Lambda because normally, Lambda is a rounding error on the bill; they had a specific challenge and they did a whole blog post on this for the AWS blog. I believe the Monitoring Tools blog, but don't take that at face value; I never remember which AWS blog is which because AWS doesn't speak with a single voice on anything.But yeah, most of the time is block, tackle, baseline stuff that is the big driver of spend, but a few weeks ago, they change the pricing dimensions for S3 intelligent tiering, where there's no longer a monitoring charge for objects that are smaller than 128 kilobytes, and there's no 30-day minimum. So, the fact that those two things went away removed almost every caveat that I can picture for using S3 intelligent tiering, which means that for most use cases, that should now be the default. I imagine you caught that change as well, since that's one of those wake up and take notice, no matter what time of the world [laugh] it is where you are when that gets dropped. How did that change your modeling? Or did that not significantly shift how you view any of this?Dann: No, I mean, I think part of our role within the organization is to pay attention to stuff like that, and then you just have those conversations with the teams that I know were either exploring intelligent tiering. We do some pricing modeling for different products, S3 storage for different types, so updating those and being like, “Hey, this might be something we want to actually use and explore now.” Similar and I guess, more of something that I actively worked on that I consider in the same category is when Amazon announced savings plans as replacing convertible reservations. Because at first they announced, and being like, “Okay, well, it's going to automatically rebalance between… different instance families across regions, too”—which convertible RIs could never do it—“And it's going to be the exact same price for a compute savings plan as a convertible RI.” And we were kind of like, what's the catch? And we spent a few weeks doing a deep dive working with our data science team, kind of like being, “Where is the catch here?”Corey: Yeah, the real catch is that you can't sell it on the secondary market if it—Dann: Yeah.Corey: —turns out you bought the wrong thing, which if that's your Plan A, then good luck.Dann: Yeah. We definitely don't use that secondary market. I don't have as much experience there, although I'm sure some people can use it to their advantage.Corey: Almost no one does. In fact, the reason that it exists—my pet theory—is that once upon a time, companies would try and classify some of the reserved instance purchases as capital expenditures, which there has since been guidance from regulatory authorities not to do that. But at the time, the fact that you could sell it to a third-party on the secondary market would help shore up that argument. If you're listening to this, and you're classifying some of your RIs as CapEx, please don't do that. Feel free to reach out to me, I can dig out the actual regulation and send it to you. There are two of them. It's a nuanced topic. If you're listening to this and have no idea what I'm talking about, God, do I envy you.Dann: [laugh]. Yeah, definitely don't do that. [laugh].Corey: There was a lot that was interesting about savings plans. When I was read in the month or so in advance of them being announced, it was, “Great. I want to see this and this and these other things, too.” And some of those things came to pass. It was extended to work with Lambda.Now, I don't believe that is financially useful in almost every case, but it doesn't need to be because so much of cloud economics from where I sit is psychological in nature, where, “Oh, we have this workload that lives on EC2 instances and we want to move it to Lambda, but we already bought the reserved instances so we're not going to do it because of sunk cost fallacy.” Which is not much of a fallacy when it's that kind of money, in some cases. Okay, great. Now, if it can migrate to Lambda and still wind up getting the discounts you've paid for it, you have removed an architectural barrier. And that's significant.Now, I want to see that same thing apply to oh if you move from EC2 to RDS, or DynamoDB or anything else, that should be helpful, too. But whatever you do, don't do what SageMaker did and launch their own separate savings plan that is not compatible with the compute savings plans, so effectively, it's great; you're locked-in architecturally to one or the other because machine learning is, once again, a marvelously executed scam to sell pickaxes into a digital gold rush.Dann: I mean, I like savings plans a lot and we've been slowly, as convertible RIs have expired, replacing them with savings plans. And I think that it is pushing the other cloud providers forward—because we're definitely multi-cloud—and so that's really useful and I hope more people will take on the compute savings plan type model, just because it makes our lives so much easier. Or it makes my life so much easier in terms of planning it, selling the commitment internally, just everything about it has made my life easier. So, I mean, how many years later are we? I definitely haven't found any big gotchas, I guess, from the secondary market. But that doesn't really impact me.Corey: Yeah, I spent a lot of time looking forward, too, doing deep analyses of okay, for which instance classes in which regions is there a price discrepancy? And I finally got someone to go semi on record and say, “Yeah. There should not be any please ping us if you find one.” “Oh, okay, great. That is enough for me to work with.”Dann: Exactly, we got that, too. I didn't believe it so we were downloading price sheets and doing comparisons, doing all that stuff.Corey: Oh, trust but verify. And when we're talking this kind of money, I don't trust very far. They make mistakes on billing issues from time to time. And I get it; it's hard, but there are challenges here and there. I am glad you mentioned a minute ago that you are multi-cloud because my position on that has often been misconstrued.I think that designing something from day one to work on multiple cloud providers is generally foolish. I think that unless you have a compelling reason not to go all-in on one cloud provider, that's what you should do. Pick a cloud—I don't care which—and go all-in. Conversely, you have a product like Datadog where your customers are in multiple clouds, and first, no one wants to pay egress to send all the telemetry from where they are into AWS, and secondly, they're not going to put up, in many cases, with their data going to a cloud provider they have explicitly chosen not to work with, so you have to meet your customers where they are. In your case, it is absolutely the right thing to do. And Twitter often gets upset and calls me hypocrite on stuff like this because Twitter believes that two things that take opposite visions cannot possibly both be true, but the world is messy.Dann: Yeah. And I mean, the nice thing about us being in multiple clouds is we are our own biggest user. And that's actually one of the reasons why I love working at Datadog is because I get to use Datadog all the time. And not only that, Datadog is on everything and we have all of our products. I'm very spoiled [laugh] with all of this. But I mean, we are running in these different cloud providers; we are using Datadog in those different cloud providers, and that is just helping everything overall, too. In addition to supporting customers that are in each cloud because that is a huge reason as well.Corey: This episode is sponsored in part by something new. Cloud Academy is a training platform built on two primary goals. Having the highest quality content in tech and cloud skills, and building a good community the is rich and full of IT and engineering professionals. You wouldn't think those things go together, but sometimes they do. Its both useful for individuals and large enterprises, but here's what makes it new. I don't use that term lightly. Cloud Academy invites you to showcase just how good your AWS skills are. For the next four weeks you'll have a chance to prove yourself. Compete in four unique lab challenges, where they'll be awarding more than $2000 in cash and prizes. I'm not kidding, first place is a thousand bucks. Pre-register for the first challenge now, one that I picked out myself on Amazon SNS image resizing, by visiting cloudacademy.com/corey. C-O-R-E-Y. That's cloudacademy.com/corey. We're gonna have some fun with this one!Corey: One of the problems that I keep running into across the board is that with things like Datadog—and again, not to single you out; every monitoring vendor to some extent has aspects of this problem—it's that when I'm a customer and I'm hooking my accounts up to Datadog, I want you to tell me about things that are going on, but the CloudWatch charges can be so egregious on the customer side, where it is bizarre and, frankly, abhorrent to me when I wind up paying more for the CloudWatch charges than I am for Datadog. And let's be clear here; I am, in fact, a Datadog customer. I pay you folks money. Not a lot of money, but I pay you money because I have certain things that I need to know are working for a variety of excellent reasons.And the problem that I keep smacking into on this is—it's not your fault; there's not anything you can do. In fact, you are one of the better providers as far as not only not being egregious with the way that you slam the CloudWatch endpoints, but also in giving guidance to customers on how to tune it further. And I really wish that more folks in your space would do things like that. It always bugs me when I wind up using a tool that tries to save money that in turn winds up costing me more than it saves.Dann: Yeah. Yeah, it's tricky there. I have less experienced myself setting up Datadog and running it in my own infrastructure as I'm more digging deep into the cost stuff and us using the cloud, so I can't speak to that specifically. But yeah, you're not the first person that I've heard have that experience. [laugh].Corey: And again, it's not your fault at all. I've been beating up the CloudWatch team for years on this, and I will continue to do so until I'm safely dead, which—depending on Amazon's level of patience—might be in mere minutes.Dann: In the larger-picture-wise, we have to remember that we're super early in the cloud adoption, even looking at the cloud economics FinOps cloud cost optimization world. I feel like most businesses at this stage in their journey are still in data centers and they're dealing with the problem of how do we move to the cloud and do it cost-efficiently? How do we set everything up? And that's where the world is right now.And I think that dealing with, “Okay, we are one hundred percent running in the cloud. What are the processes that we have in place? How do we think of finance and the finance organization not through the lens of ‘we once had data centers and now we don't,' but how do we look through that in the lens of ‘okay, we are cloud-native from day one? What does the finance department look like?'” And dealing with those problems is really interesting because Datadog has never been in a data center. We are cloud-native from the very beginning, and so it was interesting for me to join the company and build up a lot of these processes because it is different than what a lot of other people were dealing with and doing. And it presents some really interesting problems and questions that I think are going to be the foundation for the next decade of building companies and operating in the cloud.Corey: I always love having conversations with folks who are building out teams to handle these things because usually the folks I keep talking to, or who want to have conversations like this are building tools themselves to solve this problem through the miracle of SaaS, where they will bend over backwards to avoid ever talking to a customer. And we're all dealing with the same AWS APIs; there's not that much of a new spin you can put on most of these things. But understanding what customers are actually trying to do instead of falling down the rabbit hole trap of, “Hey, turn off those idle instances that are all labeled ‘drsite' because you probably don't need them,” is foolish. And after a few foolish recommendations, tooling doesn't get there. I am a big believer that tools can assist the process and narrow down what to look at.I believe they shouldn't have to exist; I think that the billing dashboard should be a hell of a lot better natively than having to pay a third party to make sense of it for me. But by and large, I do believe this is a problem that is best solved from a consultative approach. When I started this place, I was planning to build out some software, tried doing it—called DuckTools—and wound up mothballing the whole thing because what we were building was not what the industry claimed to want and, frankly, educating people into a position where then they see the value and only then will they buy is never been a game that I wanted to play.Dann: Yeah, I really liked that article that you guys published about exploring that product and the reason why you decided not to pursue it. But it's super interesting in terms of where the industry is going and building out those tools because I found that there isn't really any new thing that you can do with the tools. All the tools that exist for looking at your costs are largely the same. The main differences that I've seen is that the UI is slightly different and they have different sales teams. And if the sales teams are better, they're going to get more of the market share. And if the sales teams are not as good, it's going to be a smaller market share. And it's weird, too, be in this industry for as long as we have been, and seeing okay, well, Andreessen Horowitz just funded this new company, and this other company got invited into Y Combinator, or all of these things that are happening, and I'm kind of like, okay, but what is this tool really doing differently? And there are a few of them that are; that are doing something innovative and different, but there's also a few that are just like, this is a space where people are in, there's money here, we're doing the same thing, but we got our sales team, and we'll carve out our little corner, and then we'll get acquired, and that'll be that. Although I guess we're just at that stage of innovation in this space, I guess.Corey: Yeah, I have no earthly idea what the story is around how these companies plan to differentiate because it seems to me that they're directly attempting to compete with Cost Explorer, which—Dann: Yeah.Corey: —it's taken some time for that thing to improve to the point where it is now and it'll take further time for it to improve beyond it, but long-term, I don't think you're going to outrun AWS on a straight line like that.Dann: Yeah, I mean, when you work for one of these third-party cost tooling things, and you're working with one of your customers, and they're like, “How do I view this?” And it's kind of like, that is the easiest thing to find in Cost Explorer as well, it's—I can't imagine being like, “Well, you should pay me thousands, tens of thousands, hundreds of thousands of dollars a month to view it here,” when Cost Explorer is free. And I think Cost Explorer, it doesn't do everything, but it's gotten a lot better at what it does, and it could probably solve 90% of people's problems without using a third-party tool.Corey: You are at significant scale in multiple clouds, so the answer that these companies always give is, “Ah, but we provide a single dashboard so that you can look at costs across multiple providers in one place.” Is that even slightly useful to you?Dann: Man, if you need dashboards, get a dashboard tool. Don't get this crazy cost analysis tool. I mean, there are some great dashboard solutions that you can get where you can connect your detailed billing, cost and usage report—whatever cloud provider is calling it, but, like, that really detailed gigabytes per hour report—and then visualize it, build reports, do all that kind of stuff because that's not something that the tooling does well right now, in terms of building out cost dashboards and stuff. But that's also right now. It could in the future.Corey: Yeah. If you're a BI tool, wind up passing out templates that normalize these things? I am so tired of building it all from scratch in Tableau myself. If you're Tableau, sell me a whole bunch of things that I can use to view this stuff through, so I don't have to wind up continually reinventing that particular wheel.Dann: Yeah.Corey: Oh, I like your approach. I didn't know the answer when I was asking the question. I was about to learn something if you'd gone the other direction, but nope, but it's good to know that my impressions remain intact.Dann: Yeah, I mean, I've used different tools in the past. Again, I hesitate to name any of them, but there's a few in this space that I feel like everybody—if they're in this space, they know which tools I'm talking about—Corey: Yes, we do.Dann: —and… yeah, I've used them. They're okay—a few of them are okay, a few of them are better than others, but I mean, I was trying to evaluate the value-add over me manually setting some things up and having some sort of visualization, and just the value-add in terms of what they were charging, even if it was like a significantly smaller percent of the bill because that alone, like, percent of bill is such a difficult cost model—Corey: Oh—Dann: —to do.Corey: I hate that. Pricing is hard. Let's start there.Dann: Yeah. Yeah. Yeah, yeah.Corey: I hate the percent of bill because then it's, “Let me get this straight. I'm paying you a percentage of things like data transfer charges that I know are fixed, that I can't optimize? I'm paying you a percentage of my AWS enterprise support subscription? I'm paying you a percentage of the marketplace?” And so on and so forth. And it doesn't work. At some point of scale as well it's, I could hire a team of 20 people and save money versus what you're charging me. The other side of it though, “Ah, we'll charge you percentage of savings.” Well, then you wind up with people doing a whole bunch of things like before they bring you in, they'll make a bunch of ill-advised reserved instance purchases or savings plan purchases you have to then unwind after the fact. When I was setting this place up, I looked long and hard at different billing models and the only thing I found that worked is fixed fee. The end. Because at that point, suddenly everyone's on board with, “Hey, let's solve the problem and then get out as soon as possible.” We're not trying to build ourselves a forever job nestled in the heart of your company. And it's the only model I found that removes a whole swath of conflicts of interest. And that's the hard part. We have no partners with anyone in this space—including AWS themselves—just because as soon as we do, it becomes extremely disingenuous when we suggest doing something for your sake that happens to benefit them, such as, “Maybe back that S3 bucket up somewhere.” Well, okay, if we're partnered with them, does that mean we're trying to influence spend in the other direction? And it just becomes a morass that I never found it worth the time to deal with.Dann: Yeah, I—Corey: But that doesn't work for SaaS.Dann: Yeah, that makes a lot of sense. And I haven't actually thought about pricing model for consulting in this space that closely, but I mean, when you're charging a percent of bill or percent of savings, you have the opportunity to screw the customer, right, through all the things that you were saying. If you charge a fixed fee, you have the possibility of undervaluing yourself, which the only one that's screwed in that case is you, potentially, and if you're okay with that risk and you're okay with those dollars, that's great. Because yeah, if you're able to be like, “Okay, here's the services that I do, here's the fixed costs.” “Done.” “Done.” That just sets everybody's expectations for the relationship in a much better way that you're not constantly worried about, like, upsells and other things that might happen along the way that screws the customer.Corey: And that's the hardest part, I think, is that people lose sight of the entire customer obsession piece of it. That's one of the things Amazon gets super right. I wish more companies embrace that. Dann, I want to thank you for taking so much time out of your day to suffer my slings, and arrows, and half-formed opinions. If people want to learn more about who you are and what you're up to, where can they find you?Dann: Yeah, I have a website you guys can go to that links everywhere else. It is dannb.org. And I spell my name with two ns, so D-A-N-N-B dot org. And I have LinkedIn, I have Twitter, I have a monthly newsletter that is not really about FinOps or anything, but I really enjoy it; I've been doing it for a year, now, that you should sign up for.Corey: And links to that will, of course, be in the [show notes 00:36:26]. Dann, thanks again for your time. I really appreciate it.Dann: Yeah. Thanks so much for having me again. It's been a blast.Corey: It really has. Dann Berg, senior CloudOps analyst at Datadog. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with a comment featuring a picture of several corkboards full of post-it notes and string, and a deranged comment telling me that you have in fact finally found the catch in savings plans.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Screaming in the Cloud
Making Multi-Cloud Waves with Betty Junod

Screaming in the Cloud

Play Episode Listen Later Nov 3, 2021 35:13


About Betty Betty Junod is the Senior Director of Multi-Cloud Solutions at VMware helping organizations along their journey to cloud. This is her second time at VMware, having previously led product marketing for end user computing products.  Prior to VMware she held marketing leadership roles at Docker and solo.io in following the evolution of technology abstractions from virtualization, containers, to service mesh. She likes to hang out at the intersection of open source, distributed systems, and enterprise infrastructure software. @bettyjunod  Links: Twitter: https://twitter.com/BettyJunod Vmware.com/cloud: https://vmware.com/cloud TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: You know how git works right?Announcer: Sorta, kinda, not really Please ask someone else!Corey: Thats all of us. Git is how we build things, and Netlify is one of the best way I've found to build those things quickly for the web. Netlify's git based workflows mean you don't have to play slap and tickle with integrating arcane non-sense and web hooks, which are themselves about as well understood as git. Give them a try and see what folks ranging from my fake Twitter for pets startup, to global fortune 2000 companies are raving about. If you end up talking to them, because you don't have to, they get why self service is important—but if you do, be sure to tell them that I sent you and watch all of the blood drain from their faces instantly. You can find them in the AWS marketplace or at www.netlify.com. N-E-T-L-I-F-Y.comCorey: This episode is sponsored in part by our friends at Vultr. Spelled V-U-L-T-R because they're all about helping save money, including on things like, you know, vowels. So, what they do is they are a cloud provider that provides surprisingly high performance cloud compute at a price that—while sure they claim its better than AWS pricing—and when they say that they mean it is less money. Sure, I don't dispute that but what I find interesting is that it's predictable. They tell you in advance on a monthly basis what it's going to going to cost. They have a bunch of advanced networking features. They have nineteen global locations and scale things elastically. Not to be confused with openly, because apparently elastic and open can mean the same thing sometimes. They have had over a million users. Deployments take less that sixty seconds across twelve pre-selected operating systems. Or, if you're one of those nutters like me, you can bring your own ISO and install basically any operating system you want. Starting with pricing as low as $2.50 a month for Vultr cloud compute they have plans for developers and businesses of all sizes, except maybe Amazon, who stubbornly insists on having something to scale all on their own. Try Vultr today for free by visiting: vultr.com/screaming, and you'll receive a $100 in credit. Thats v-u-l-t-r.com slash screaming.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Periodically, I like to poke fun at a variety of different things, and that can range from technologies or approaches like multi-cloud, and that includes business functions like marketing, and sometimes it extends even to companies like VMware. My guest today is the Senior Director of Multi-Cloud Solutions at VMware, so I'm basically spoilt for choice. Betty Junod, thank you so much for taking the time to speak with me today and tolerate what is no doubt going to be an interesting episode, one way or the other.Betty: Hey, Corey, thanks for having me. I've been a longtime follower, and I'm so happy to be here. And good to know that I'm kind of like the ultimate cross-section of all the things [laugh] that you can get snarky about.Corey: The only thing that's going to make that even better is if you tell me, “Oh, yeah, and I moonlight on a contract gig by naming AWS services.” And then I just won't even know where to go. But I'll assume they have to generate those custom names in-house.Betty: Yes. Yes, I think they do those there. I may comment on it after the fact.Corey: So, periodically I am, let's call it miscategorized, in my position on multi-cloud, which is that it's a worst practice that when you're designing something from scratch, you should almost certainly not be embracing unless you're targeting a very specific corner case. And I stand by that, but what that has been interpreted as by the industry, in many cases because people lack nuance when you express your opinions in tweet-sized format—who knew—as me saying, “Multi-cloud bad.” Maybe, maybe not. I'm not interested in assigning value judgment to it, but the reality is that there are an awful lot of multi-cloud deployments out there. And yes, some of them started off as, “We're going to migrate from one to the other,” and then people gave up and called it multi-cloud, but it is nuanced. VMware is a company that's been around for a long time. It has reinvented itself in a few different ways at different periods of its evolution, and it's still highly relevant. What is the Multi-Cloud Solutions group over at VMware? What do you folks do exactly?Betty: Yeah. And so I will start by multi-cloud; we're really taking it from a position of meeting the customer where they are. So, we know that if anything, the only thing that's a given in our industry is that there will be something new in the next six months, next year, and the whole idea of multi-cloud, from our perspective, is giving customers the optionality, so don't make it so that it's a closed thing for them. But if they decide—it's not that they're going to start, “Hey, I'm going to go to cloud, so day one, I'm going to go all-in on every cloud out there.” That doesn't make sense, right, as—Corey: But they all gave me such generous free credit offers when I founded my startup; I feel obligated to at this point.Betty: I mean, you can definitely create your account, log in, play around, get familiar with the console, but going from zero to being fully operationalized team to run production workloads with the same kind of SLAs you had before, across all three clouds—what—within a week is not feasible for people getting trained up and actually doing that. Our position is that meeting customers where they are and knowing that they may change their mind, or something new will come up—a new service—and they really want to use a new service from let's say GCP or AWS, they want to bring that with an application they already have or build a new app somewhere, we want to help enable that choice. And whether that choice applies to taking an existing app that's been running in their data center—probably on vSphere—to a new place, or building new stuff with containers, Kubernetes, serverless, whatever. So, it's all just about helping them actually take advantage of those technologies.Corey: So, it's interesting to me about your multi-cloud group, for lack of a better term, is there a bunch of things fall under its umbrella? I believe Bitnami does—or as I insist on calling it, ‘bitten-A-M-I'—I believe that SaltStack—which I wrote a little bit of once upon a time, which tells me you folks did no due diligence whatsoever because everything I've ever written is molten garbage—Betty: Not [unintelligible 00:04:33].Corey: And—so to be clear, SaltStack is good; just the parts that I wrote are almost certainly terrible because have you met me?Betty: I'll make a note. [laugh].Corey: You have Wavefront, you have CloudHealth, you have a bunch of other things in the portfolio, and yeah, all those things do work across multiple clouds, but there's nothing that makes using any of those things a particularly bad idea even if you're all-in on one cloud provider, too. So, it's a portfolio that applies to a whole bunch have different places from your perspective, but it can be used regardless of where folks stand ideologically.Betty: Yes. So, this goes back to the whole idea that we meet the customers where they are and help them do what they want to do. So, with that, making sure these technologies that we have work on all the clouds, whether that be in the data center or the different vendors, so that if a customer wants to just use one, or two, or three, it's fine. That part's up to them.Corey: The challenge I've run into is that—and maybe this is a ‘Twitter Bubble' problem, but unfortunately, having talked to a whole bunch of folks in different contexts, I know it isn't—there's almost this idea that you have to be incredibly dogmatic about a particular technology that you're into. I joke periodically about the Rust Evangelism Strikeforce where their entire job is talking about using Rust; their primary IDE is PowerPoint because they're giving talks all the time about it rather than writing code. And great, that's a bit of an exaggeration, but there are the idea of a technology purist who is taking, “Things must be this way,” well past a point of being reasonable, and disregarding the reality that, yeah, the world is messy in a way that architectural diagrams never are.Betty: Yeah. The architectural diagrams are always 2D, right? Back to that PowerPoint slide: how can I make pretty boxes? And then I just redraw a line because something new came out. But you and I have been in this industry for a long time, there's always something new.And I think that's where the dogmatism gets problematic because if you say we're only going to do containers this way—you know, I could see Swarm and Kubernetes, or all-in on AWS and we're going to use all the things from AWS and there's only this way. Things are generational and so the idea that you want to face the reality and say that there is a little bit of everything. And then it's kind of like, how do you help them with a part of that? As a vendor, it could be like, “I'm going to help us with a part of it, or I'm going to help address certain eras of it.” That's where I think it gets really bad to be super dogmatic because it closes you off to possibly something new and amazing, new thinking, different ways to solve the same problem.Corey: That's the problem is left to our own devices, most of us who are building things, especially for random ideas, yeah, there's a whole modern paradigm of how I can build these things, but I'm going to shortcut to the thing I know best, which may very well the architectures that I was using 15 years ago, maybe tools that I was using 15 years ago. There's a reason that Vim is still as popular as it is. Would I recommend it to someone who's a new user? Absolutely not; it's user-hostile, but back in my days of being a grumpy sysadmin, you learned vi because it was on everything you could get into, and you never knew in what environment you were going to be encountering stuff. These days, you aren't logging in to remote systems to manage them, in most cases, and when it happens, it's a rarity and a bug.The world changes; different approaches change, but you have to almost reinvent your entire philosophy on how things work and what your career trajectory looks like. And you have to give up aspects of what you've considered to be part of your identity and embrace something new. It was hard for me to accept that, for example, Docker and the wave of containerization that was rolling out was effectively displacing the world that I was deep in of configuration management with Puppet and with Salt. And the world changes; I said, “Okay, now I'll work on cloud.” And if something else happens, and mainframes are coming back again, instead, well, I'm probably not going to sit here railing against the tide. It would be ridiculous to do that from my perspective. But I definitely understand the temptation to fight against it.Betty: Mm-hm. You know, we spend so much time learning parts of our craft, so it's hard to say, “I'm now not going to be an expert in my thing,” and I have to admit that something else might be better and I have to be a newbie again. That can be scary for someone who's spent a lot of time to be really well-versed in a specific technology. It's funny that you bring up the whole Docker and Puppet config management; I just had a healthy discussion over Slack with some friends. Some people that we know and comment about some of the newer areas of config management, and the whole idea is like, is it a new category or an evolution of? And I went back to the point that I made earlier is like, it's generations. We continually find new ways to solve a problem, and one thing now is it [sigh] it just all goes so much faster, now. There's a new thing every week. [laugh] it seems sometimes.Corey: It is, and this is the joy of having been in this industry for a while—toxic and broken in many ways though it is—is that you go through enough cycles of seeing today's shiny, new, amazing thing become tomorrow's legacy garbage that we're stuck supporting, which means that—at least from my perspective—I tend to be fairly conservative with adopting new technologies with respect to things that matter. That means that I'm unlikely to wind up looking at the front page of Hacker News to pick a framework to build a banking system in, and I'm unlikely to be the first kid on my block to update to a new file system or database, just because, yeah, if I break a web server, we all laugh, we make fun of the fact that it throws an error for ten minutes, and then things are back up and running. If I break the database, there's a terrific chance that we don't have a company anymore. So, it's the ‘mistakes will show' area and understanding when to be aggressive and when to hold back as far as jumping into new technologies is always a nuanced decision. And let's be clear as well, an awful lot of VMware's customers are large companies that were founded, somehow—this is possible—before 2010. Imagine that. Did people—Betty: [laugh]. I know, right?Corey: —even have businesses or lives back then? I thought we all used horse-driven carriages and whatnot. And they did not build on cloud—not because of any perception of distrust; because it functionally did not exist at the time that they were building these things. And, “Oh, come out into the cloud. It's fine now.” It… yeah, that application is generating hundreds of millions in revenue every quarter. Maybe we treat that with a little bit of respect, rather than YOLO-ing it into some Lambda-driven monster that's constructed—Betty: One hundred—Corey: —out of popsicle sticks and glue.Betty: —percent. Yes. I think people forget that. And it's not that these companies don't want to go to cloud. It's like, “I can't break this thing. That could be, like, millions of dollars lost, a second.”Corey: I write my weekly newsletters in a custom monstrosity of a system that has something like 30-some-odd Lambda functions, a bunch of API gateways that are tied together with things, and periodically there are challenges with it that break as the system continues to evolve. And that's fine. And I'm okay with using something like that as a part of my workflow because absolute worst case, I can go back to the way that my newsletter was originally written: in Google Docs, and it doesn't look anywhere near the same way, and it goes back to just a text email that starts off with, “I have messed up.” And that would be a better story than most of the stuff I put out as a common basis. Similarly, yeah, durability is important.If this were a serious life-critical app, it would not just be hanging out in a single region of a single provider; it would probably be on one provider, as I've talked about, but going multi-region and having backups to a different cloud provider. But if AWS takes a significant enough outage to us-west-2 in Oregon, to the point where my ridiculous system cannot function to write the newsletter, that too, is a different handwritten email that goes out that week because there's no announcement they've made that anyone's going to give the slightest toss about, given the fact that it's basically Cloud Armageddon. So, we'll see. It's about understanding the blast radius and understanding your use case.Betty: Yep. A hundred percent.Corey: So, you've spent a fair bit of time doing interesting things in your career. This is your second outing at VMware, and in the interim, you were at solo.io for a bit, and before that you were in a marketing leadership role at Docker. Let's dive in, if you will. Given that you are no longer working at Docker, they recently made an announcement about a pricing model change, whereas it is free to use Docker Desktop for anyone's personal projects, and for small companies.But if you're a large company, which they define is ten million in revenue a year or 250 employees—those two things don't go alike, but okay—then you have to wind up having a paid plan. And I will say it's a novel approach, but I'm curious to hear what you have to say about it.Betty: Well, I'd say that I saw that there was a lot of flutter about that news, and it's kind of a, it doesn't matter where you draw the line in the sand for the tier, there's always going to be some pushback on it. So, you have to draw a line somewhere. I haven't kept up with the details around the pricing models that they've implemented since I left Docker a few years ago, but monetization is a really important part for a startup. You do have to make money because there are people that you have to pay, and eventually, you want to get off of raising money from VCs all the time. Docker Desktop has been something that has been a real gem from a local developer experience, right, giving the—so that has been well-received by the community.I think there was an enterprise application for it, but when I saw that, I was like, yeah, okay, cool. They need to do something with that. And then it's always hard to see the blowback. I think sometimes with the years that we've had with Docker, it's kind of like no matter what they do, the Twitterverse and Hacker News is going to just give them a hard time. I mean, that is my honest opinion on that. If they didn't do it, and then, say, they didn't make the kind of revenue they needed, people would—that would become another Twitter thread and Hacker News blow up, and if they do it, you'll still have that same reaction.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of "Hello, World" demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking databases, observability, management, and security.And - let me be clear here - it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself all while gaining the networking load, balancing and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build.With Always Free you can do things like run small scale applications, or do proof of concept testing without spending a dime. You know that I always like to put asterisks next to the word free. This is actually free. No asterisk. Start now. Visit https://snark.cloud/oci-free that's https://snark.cloud/oci-free.Corey: It seems to be that Docker has been trying to figure out how to monetize for a very long time because let's be clear here; I think it is difficult to overstate just how impactful and transformative Docker was to the industry. I gave a talk “Heresy in the Church of Docker” that listed a bunch of things that didn't get solved with Docker, and I expected to be torn to pieces for it, and instead I was invited to give it at ContainerCon one year. And in time, a lot of those things stopped being issues because the industry found answers to it. Now, unfortunately, some of those answers look like Kubernetes, but that's neither here nor there. But now it's, okay, so giving everything that you do that is core and central away for free is absolutely part of what drove the adoption that it saw, but goodwill from developers is not the sort of thing that generally tends to lead to interesting revenue streams.So, they had to do something. And they've tried a few different things that haven't seemed to really pan out. Then they spun off that pesky part of their business that made money selling support contracts, over to Mirantis, which was apparently looking for something now that OpenStack was no longer going to be a thing, and Kubernetes is okay, “Well, we'll take Docker enterprise stuff.” Great. What do they do, as far as turning this into a revenue model?There's a lot of the, I guess, noise that I tend to ignore when it comes to things like this because angry people on Twitter, or on Hacker News, or other terrible cesspools on the internet, are not where this is going to be decided. What I'm interested in is what the actual large companies are going to say about it. My problem with looking at it from the outside is that it feels as if there's significant ambiguity across the board. And if there's one thing that I know about large company procurement departments, it's that they do not like ambiguity. This change takes effect in three or four months, which is underwear-outside-the-pants-superhero-style speed for a lot of those companies, and suddenly, for a lot of developers, they're so far removed from the procurement side of the house that they are never going to have a hope of getting that approved on a career-wide timespan.And suddenly, for a lot of those companies, installing and running Docker Desktop just became a fireable offense because from the company's perspective, the sheer liability side of it, if they were getting subject to audit, is going to be a problem. I don't believe that Docker is going to start pulling Oracle-like audit tactics, but no procurement or risk management group in the world is going to take that on faith. So, the problem is not that it's expensive because that can be worked around; it's not that there's anything inherently wrong with their costing model. The problem is the ambiguity of people who just don't know, “Does this apply to me or doesn't this apply to me?” And that is the thing that is the difficult, painful part.And now, as a result, the [unintelligible 00:17:28] groups and their champions of Docker Desktop are having to spend a lot more time, energy, and thought on this than it would simply be for cutting a check because now it's a risk org-wide, and how do we audit to figure out who's installed this previously free open-source thing? Now what?Betty: Yeah, I'll agree with you on that because once you start making it into corporate-issued software that you have to install on the desktop, that gets a lot harder. And how do you know who's downloaded it? Like my own experience, right? I have a locked-down laptop; I can't just install whatever I want. We have a software portal, which lets me download the approved things.So, it's that same kind of model. I'd be curious because once you start looking at from a large enterprise perspective, your developers are working on IP, so you don't want that on something that they've downloaded using their personal account because now it sits—that code is sitting with their personal account that's using this tool that's super productive for them, and that transition to then go to an enterprise, large enterprise and going through a procurement cycle, getting a master services agreement, that's no small feat. That's a whole motion that is different than someone swiping a credit card or just downloading something and logging in. It's similar to what you see sometimes with the—how many people have signed up for and paid 99 bucks for Dropbox, and then now all of a sudden, it's like, “Wow, we have all of megacorp [laugh] signed up, and then now someone has to sell them a plan to actually manage it and make sure it's not just sitting on all these personal drives.”Corey: Well, that's what AWS's original sales motion looked a lot like they would come in and talk to the CTO or whatnot at giant companies. And the CTO would say, “Great, why should we pick AWS for our cloud needs?” And the answer is, “Oh, I'm sorry. You have 87 distinct accounts within your organization that we've [unintelligible 00:19:12] up for you. We're just trying to offer you some management answers and unify the billing and this, and probably give you a discount as well because there is price breaks available at certain sizing.” It was a different conversation. It's like, “I'm not here to sell you anything. We're already there. We're just trying to formalize the relationship.” And that is a challenge.Again, I'm not trying to cast aspersions on procurement groups. I mean, I do sell enterprise consulting here at The Duckbill Group; we deal with an awful lot of procurement groups who have processes and procedures that don't often align to the way that we do things as a ten-person, fully remote company. We do not have commercial vehicle insurance, for example, because we do not have a commercial vehicle and that is a prerequisite to getting the insurance, for one. We're unlikely to buy one to wind up satisfying some contractual requirements, so we have to go back and forth and get things like that removed. And that is the nature of the beast.And we can say yes, we can say no on a lot of those questionnaires, but, “It depends,” or, “I don't know,” is the sort of thing that's going to cause giant red flags and derail everything. But that is exactly what Docker is doing. Now, it's the well, we have a sort of sloppy, weird set of habits with some of our engineers around the bring your own device to work thing. So, that's the enterprise thing. Let me be very clear, here at The Duckbill Group, we have a policy of issuing people company machines, we manage them very lightly just to make sure the drives are encrypted, so they—and that the screensaver comes out with a password, so if someone loses a laptop, it's just, “Replace the hardware,” not, “We have a data breach.”Let's be clear here; we are responsible about these things. But beyond that, it's oh, you want to have some personal thing installed on your machine or do some work on that stuff? Fine. By all means. It's a situation of we have no policy against it; we understand this is how work happens, and we trust people to effectively be grownups.There are some things I would strongly suggest that any employee—ours or anyone else—not cross the streams on for obvious IP ownership rights and the rest, we have those conversations with our team for a reason. It's, understand the nuances of what you're doing, and we're always willing to throw hardware at people to solve these problems. Not every company is like that. And ten million in revenue is not necessarily a very large company. I was doing the math out for ten million in revenue or 250 employees; assuming that there's no outside investment—which with VC is always a weird thing—it's possible—barely—to have a $10 million in revenue company that has 250 employees, but if they're full time they are damn close to a $15 an hour minimum wage. So, who does it apply to? More people than you might believe.Betty: Yeah, I'm really curious to how they're going to like—like you say, if it takes place in three or four months, roll that out, and how would you actually track it and true that up for people? So.Corey: Yeah. And there are tools and processes to do this, but it's also not in anyone's roadmap because people are not sitting here on their annual planning periods—which is always aspirational—but no one's planning for, “Oh, yeah, Q3, one of our software suppliers is going to throw a real procurement wrench at us that we have to devote time, energy, resources, and budget to figure out.” And then you have a problem. And by resources, I do mean resources of basically assigning work and tooling and whatnot and energy, not people. People are humans, they are not resources; I will die on that hill.Betty: Well, you know, actually resource-wise, the thing that's interesting is when you say supplier, if it's something that people have been able to download for free so far, it's not considered a supplier. So, it's—now they're going to go from just a thing I can use and maybe you've let your developers use to now it has to be something that goes through the official internal vetting as being a supplier. So, that's just—it's a whole different ball game entirely.Corey: My last job before I started this place, was a highly regulated financial institution, and even grabbing things were available for free, “Well, hang on a minute because what license is it using and how is it going to potentially be incorporated?” And this stuff makes sense, and it's important. Now, admittedly, I have the advantage of a number of my engineering peers in that I've been married to a corporate attorney for 11 years and have insight into that side of the world, which to be clear, is all about risk mitigation which is helpful. It is a nuanced and difficult field to—as are most things once you get into them—and it's just the uncertainty that befuddles me a bit. I wish them well with it, truly I do. I think the world is better with an independent Docker in it, but I question whether this is going to find success. That said, it doesn't matter what I think; what matters is what customers say and do, and I'm really looking forward to seeing how it plays out.Betty: A hundred percent; same here. As someone who spent a good chunk of my life there, their mark on the industry is not to be ignored, like you said, with what happened with containers. But I do wish them well. There's lot of good people over there, it's some really cool tech, and I want to see a future for them.Corey: One last topic I want to get into before we wind up wrapping this episode is that you are someone who was nominated to come on the show by a couple of folks, which is always great. I'm always looking for recommendations on this. But what's odd is that you are—if we look at it and dig a little bit beneath the titles and whatnot, you even self-describe as your history is marketing leadership positions. It is uncommon for engineering-types to recommend that I talk to marketing folks.s personally I think that is a mistake; I consider myself more of a marketer than not in some respects, but it is uncommon, which means I have to ask you, what is your philosophy of marketing because it very clearly is differentiated in the public eye.Betty: I'm flattered. I will say that—and this goes to how I hire people and how I coach teams—it's you have to be super curious because there's a ton of bad marketing out there, where it's just kind of like, “Hey, we do these five things and we always do these five things: blah, blah, blah, blah, blah.” But I think it's really being curious about what is the thing that you're marketing? There are people who are just focused on the function of marketing and not the thing. Because you're doing your marketing job in the service of a thing, this new widget, this new whatever, and you got to be super curious about it.And I'll tell you that, for me, it's really hard for me to market something if I'm not excited about it. I have to personally be super excited about the tech or something happening in the industry, and it's, kind of like, an all-in thing for me. And so in that sense, I do spend a ton of time with engineers and end-users, and I really try to understand what's going on. I want to understand how the thing works, and I always ask them, “Well”—so I'll ask the engineers, like, “So… okay, this sounds really cool. You just described this new feature and you're super excited about it because you wrote it, but how is your end-user, the person you're building this for, how did they do this before? Help me understand. How did they do this before and why is this better?”Just really dig into it because for me, I want to understand it deeply before I talk about it. I think the thing is, it shows a tremendous amount of respect for the builder, and then to try to really be empathetic, to understand what they're doing and then partner with them—I mean, this sounds so business-y the way I'm talking about this—but really be a partner with them and just help them make their thing really successful. I'm like the other end; you're going to build this great thing and now I'm going to make it sound like it's the best thing that's ever happened. But to do that, I really need to deeply understand what it is, and I have to care about it, too. I have to care about it in the way that you care about it.Corey: I cannot effectively market or sell something that I don't believe in, personally. I also, to be clear because you are a marketing professional—or at least far more of one than I ever was—I do not view what I do is marketing; I view it as spectacle. And it's about telling stories to people, it's about learning what the market thinks about it, and that informs product design in many respects. It's about understanding the product itself. It's about being able to use the product.And if people are listening to this and think, “Wait a minute, that sounds more like DevRel.” I have news for you. DevRel is marketing, they're just scared to tell you that. And I know people are going to disagree with me on that. You're wrong. But that's okay; reasonable people can disagree.And that's how I see it is that, okay, I'll talk to people building the service, I'll talk to people using the service, but then I'm going to build something with the service myself because until then, it's all a game of who sounds the most convincing in the stories that they tell. But okay, you can tell an amazing story about something, but if it falls over when I tried to use it, well, I'm sorry, you're not being accurate in your descriptions of it.Betty: A hundred percent. I hate to say, like, you're storytellers, but that's a big part of it, but it's kind of like you want to tell the story, so you do something to that people believe a certain thing. But that's part of a curated experience because you want them to try this thing in a certain way. Because you've designed it for something. “I built a spoon. I want you to use that to eat your soup because you can't eat soup with a fork.”So, then you'll have this amazing soup-eating experience, but if I build you a spoon and then not give you any directions and you start throwing it at cars, you're going to be like, “This thing sucks.” So, I kind of think of it in that way. To your point of it has to actually work, it's like, but they also need to know, “What am I supposed to use it for?”Corey: The problem I've always had on some visceral level with formal marketing departments for companies is that they can say that a product that they sell is good, they can say that the product is great, or they can choose to say nothing at all about that product, but when there's a product in the market that is clearly a turd, a marketing department is never going to be able to say that, which I think erodes its authenticity in many respects. I understand the constraints behind, that truly I do, but it's the one superpower I think that I bring to the table where even when I do sponsorship stuff it's, you can buy my attention but not my opinion. Because the authenticity of me being trusted to call them like I see them, for lack of a better term, to my mind at least outweighs any short-term benefit from saying good things about a product that doesn't deserve them. Now, I've been wrong about things, sure. I have also been misinformed in both directions, thinking something is great when it's not, or terrible when it isn't or not understanding the use case, and I am thrilled to engage in those debates. “But this is really expensive when you run for this use case,” and the answer can be, “Well, it's not designed for that use case.” But the answer should not be, “No it's not.” I promise you, expensive is in the eye of the customer not the person building the thing.Betty: Yes. This goes back to I have to believe in the thing. And I do agree it's, like not [sigh]—it's not a panacea. You're not going to make Product A and it's going to solve everything. But being super clear and focused on what it is good for, and then please just try it in this way because that's what we built it for.Corey: I want to thank you for taking the time to have a what for some people is no doubt going to be perceived as a surprisingly civil conversation about things that I have loud, heated opinions about. If people want to learn more, where can they find you?Betty: Well, they can follow me on Twitter. But um, I'd say go to vmware.com/cloud for our work thing.Corey: Exactly. VM where? That's right. VM there. And we will, of course, put links to that in the [show notes 00:30:07].Betty: [laugh].Corey: Thank you so much for taking the time to speak with me. I appreciate it.Betty: Thanks, Corey.Corey: Betty Junod, Senior Director of Multi-Cloud Solutions at VMware. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with a loud, ranting comment at the end. Then, if you work for a company that is larger than 250 people or $10 million in revenue, please also Venmo me $5.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Underscore_
Zemmour ou Mélenchon : que vous recommande Youtube ? - avec Cyrus North & Matthieu Lambda

Underscore_

Play Episode Listen Later Oct 30, 2021 104:29


Dans cet épisode, en plus de recevoir l'incroyable Cyrus North, Nicolas Bouchaïb, expert en Data Visualisation et cartographe de l'Internet est également sur notre plateau pour une chronique sur les recommandations YouTube et les bulles de filtres. Cybernétique, déconnexion, art assisté par des IA sont également au programme, avant de finir par une review des outils qu'on kiff, ici à la rédaction. Enjoy ! Voir Acast.com/privacy pour les informations sur la vie privée et l'opt-out.

Screaming in the Cloud
Teasing Out the Titular Titles with Chris Williams

Screaming in the Cloud

Play Episode Listen Later Oct 27, 2021 39:59


About ChrisChris Williams is a Enterprise Architect for World Wide Technology — a technology solution and service provider. There he helps customers design the next generation of public, private, and hybrid cloud solutions, specializing in AWS and VMware. His first computer was a Commodore 64, and he's been playing video games ever since.Chris blogs about virtualization, technology, and design at Mistwire. He is an active community leader, co-organizing the AWS Portsmouth User Group, and both hosts and presents on vBrownBag. He is also an active mentor, helping students at the University of New Hampshire through Diversify Thinking—an initiative focused on empowering girls and women to pursue education and careers in STEM.Chris is a certified AWS Hero as well as a VMware vExpert. Fun fact that Chris doesn't want you to know: he has a degree in psychology so you can totally talk to him about your feelings.Links: WWT: https://www.wwt.com/ Twitter: https://twitter.com/mistwire Personal site: https://mistwire.com vBrownBag: https://vbrownbag.com/team/chris-williams/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate: is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards, while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other, which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at Honeycomb.io/screaminginthecloud. Observability, it's more than just hipster monitoring.Corey: This episode is sponsored in part by our friends at Vultr. Spelled V-U-L-T-R because they're all about helping save money, including on things like, you know, vowels. So, what they do is they are a cloud provider that provides surprisingly high performance cloud compute at a price that—while sure they claim its better than AWS pricing—and when they say that they mean it is less money. Sure, I don't dispute that but what I find interesting is that it's predictable. They tell you in advance on a monthly basis what it's going to going to cost. They have a bunch of advanced networking features. They have nineteen global locations and scale things elastically. Not to be confused with openly, because apparently elastic and open can mean the same thing sometimes. They have had over a million users. Deployments take less that sixty seconds across twelve pre-selected operating systems. Or, if you're one of those nutters like me, you can bring your own ISO and install basically any operating system you want. Starting with pricing as low as $2.50 a month for Vultr cloud compute they have plans for developers and businesses of all sizes, except maybe Amazon, who stubbornly insists on having something to scale all on their own. Try Vultr today for free by visiting: vultr.com/screaming, and you'll receive a $100 in credit. Thats v-u-l-t-r.com slash screaming.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. One of the things I miss the most from the pre-pandemic times is meeting people at conferences or at various business meetings, not because I like people—far from it—but because we go through a ritual that I am a huge fan of, which is the exchange of business cards. Now, it's not because I'm a collector or anything here, but because I like seeing what people's actual titles are instead of diving into the morass of what we call ourselves on Twitter and whatnot. Today, I have just one of those folks with me. My guest is Chris Williams, who works at WWT, and his business card title is Enterprise Architect, comma AWS Cloud. Chris, welcome.Chris: Hi. Thanks for having me on the show, Corey.Corey: No, thank you for taking the time to speak with me. I have to imagine that the next line in your business card is, “No, I don't work for AWS,” because you know a company has succeeded when they get their name into people's job titles who don't work there.Chris: So, I have a running joke where the next line should actually be cloud therapist. And my degree is actually in psychology, so I was striving to get cloud therapist in there, but they still don't want to let me have it.Corey: Former guest Bobby Allen is now a cloud therapist over at Google Cloud, which is just phenomenal. I don't know what they're doing in a marketing context over there; I just know that they're just blasting them out of the park on a consistent, ongoing basis. It's really nice to see. It's forcing me to up my game a little bit. So, one of the challenges I've always had is, I don't like putting other companies' names into the title.Now, I run the Last Week in AWS newsletter, so yeah, okay, great, there's a little bit of ‘do as I say, not as I do' going on here. Because it feels, on some level, like doing unpaid volunteer work for a $2 trillion company. Speaking of, you are an AWS Community Hero, where you do volunteer work for a $2 trillion company. How'd that come about? What did you do that made you rise to their notice?Chris: That was a brilliant segue. Um—[laugh]—Corey: I do my best.Chris: So I, actually prior to becoming an AWS Community Hero, I do a lot of community work. So, I have run and helped to run four different community-led organizations: the Virtualization Technology User Group of New England; the AWS Portsmouth User Group, now the AWS Boston User Group; I'm a co-host and presenter for vBrownBag; I also do the New England AWS Community Day, which is a conglomeration of all the different user groups in one setting; and various and sundry other things, as well, along the way. Having done all of that, and having had a lot of the SAs and team members come and do speaking presentations for these various and sundry things, I was nominated internally by AWS to become one of their Community Heroes. Like you said, it's basically unpaid volunteer work where I go out and tout the services. I love talking about nerd stuff, so when I started working on AWS technologies, I really enjoyed it, and I just, kind of like, glommed on with other people that did it as well. I'm also a VMware vExpert, which basically use the exact same accolade for VMware. I have not been doing as much VMware stuff in the recent past, but that's kind of how I got into this gig.Corey: One of the things that strikes me as being the right move with respect to these, effectively, community voice accolades is Microsoft got something very right—they've been doing this a long time—they have their MVP program, but they have to re-invite people who have to requalify for it by whatever criteria they are, every year. AWS does not do this with their Heroes program. If you look at their Heroes page, there's a number of folks up there who have been doing interesting things in the cloud years ago, but then fell off the radar for a variety of reasons. In fact, the only way that I'm aware that you can lose Hero status is via getting a job at AWS or one of AWS competitors.Now, the hard part, of course, is well, who is Amazon's competitors? Basically everyone, but it mostly distills down to Microsoft, Google, and Oracle, as best I can tell, for Hero status. How does VMware fall on that spectrum? To be more specific, how does VMware fall on the spectrum of their community engagement program and having to renew, not, “Are they AWS's competitor?” To which the answer is, “Of course.”Chris: So, the renewal process for the VMware vExpert program is an annual re-up process where you fill out the form, list your contribution of the year, what you've done over the previous year, and then put it in for submission to the board of VMware vExperts who then give you the thumbs up or thumbs down. Much like Nero, you know, pass or fail, live or die. And I've been fortunate enough, so my vBrownBag contributions are every week; we have a show that happens every week. It can be either VMware stuff, or cloud in general stuff, or developer-related stuff. We cover the gamut; you know, people that want to come on and talk about whatever they want to talk about, they come on. And by virtue of that, we've had a lot of VMware speakers, we've had a lot of AWS speakers, we've had a lot of Azure speakers. So, I've been fortunate enough to be able to qualify each year with those contributions.Corey: I think that's the right way to go, from my perspective at least. But I want to get into this a little bit because you are an enterprise architect, which is always one of those terms that is super easy to make fun of in a variety of different ways. Your IDE is probably a whiteboard, and at some point when you have to write code, I thought you had a team of people who would be able to do that all for you because your job is to cogitate, and your artifacts are documentation, and the entire value of what you do can only be measured in the grand sweep of time, et cetera, et cetera, et cetera.Chris: [laugh].Corey: But you don't generally get to be a Community Hero for stuff like that, and you don't usually get to be a vExpert on the VMware side, by not having at least technical chops that make people take a second look. What is it you'd say it is you do hear for, lack of a better term?Chris: “What would you say ya, do you here, Bob?” So, I'm not being facetious when I say cloud therapist. There is a lot of working at the eighth layer of the OSI model, the political layer. There's a lot of taking the requirements from the customer and sending them to the engineer. I'm a people person.The easy answer is to say, I do all the things from the TOGAF certification manual: the requirements, risks, assumptions, and constraints; the logical, conceptual, and physical diagrams; the harder answer is the soft skill side of that, is actually being able to communicate with the various levels of the industry, figuring out what the business really wants to do and how to technically solution that and figure out how to talk to the engineers to make that happen. You're right EAs get made fun of all the time, almost as much as consultants get made fun of. And it's a very squishy layer that, you know, depending upon your personality and the personality of the customer that you're dealing with, it can work wonderfully well or it can crash and burn immediately. I know from personal experience that I don't mesh well with financials, but I'm really, really good with, like, medical industry stuff, just the way that the brain works. But ironically, right now I'm working with a financial and we're getting along like a house on fire.Corey: Oh, yeah. I've been saying for a while now that when it comes to cloud, cost and architecture are the same things, and I think that ties back to a lot of different areas. But I want to be very clear here that we talk about, I'm not super deep into the financials, that does not mean you're bad at architecture because working on finance means different things to different folks. I don't think that it is possibly a good architect in the cloud environment and not have a conception of, “Huh, that thing seems really expensive if I do it that way.” That is very different than having the skill of reading a profit and loss statement or understanding various implications of the time value of money calculation that a company uses, or how things get amortized.There are nuances piled on top of nuances in finance, and it's easy to sit here and think that oh, I'm not great at finance means I don't know how money works. That is very rarely true. If you really don't know how money works, you'll go start a cryptocurrency startup.Chris: [laugh]. So, I plugged back to you; I was listening to one of your old shows and I cribbed one of your ideas and totally went with it. So, I just said that there's the logical, conceptual, and physical diagrams of an environment; on one of your shows, you had mentioned a financial diagram for an environment, and I was like, “That's brilliant.” So, now when I go into a customer, I actually do that, too. I take my physical diagram, I strip out all of the IP addresses, and our names, and everything like that, and I plot down how much it's going to cost, like, “This is the value of the EC2 instance,” or, “This is how much this pipe is going to cost if you run this over it.” And they go bananas over it. So, thanks for providing that idea that I mercilessly stole.Corey: Kind of fun on a lot of levels. Part of the challenge is as things get cloudier and it moves away from EC2 instances, ideally the lie we would like to tell ourselves that everything's in an auto-scaling group. Great—Chris: Right.Corey: —stepping beyond that when you start getting into something that's even more intricately tied to a specific user, we're talking about effectively trying to get unit economic measures of every user, every thousand users is going to cost me X dollars to service them on average, on top of a baseline of steady-state spend that is going to increase differently. At that point, talking to finance about predictive models turn into, “Well, this comes down to a question of business modeling.” But conversely, for engineering minds that is exactly what finance is used to figuring out. The problem they have is, “Well, every time we hire a new engineer, we wind up seeing our AWS bill increase.” Funny how that works. Yeah, how do you map that to something that the business understands? That is part of what they do. But it does, I admit, make it much more challenging from a financial map of an environment.Chris: Yeah, especially when the customer or the company is—you know, they've been around for a while, and they're used to just like that large bolus of money at the very beginning of a data center, and they buy the switches, and they buy the servers, and they virtualize them, and they have that set cost that they knew that they had to plunk down at the beginning. And it's a mindset shift. And they're coming around to it, some faster than others. Oddly enough, the startups nowadays are catching on very quickly. I don't deal with a lot of startups, so it takes some finesse.Corey: An interesting inflection that I've seen is that there's an awful lot of enterprises out there that say, “Oh, we're like a startup.” Great. You mean with weird cultural inflections that often distill down to cult of personality, the constant worry about whether you're going to wind up running out of runway before finding product-market fit? And the rooms filled with—Chris: The eighty-hour work weeks? The—[laugh]—Corey: And they're like, “No, no, no, it's like the good parts.” “Oh, so you mean out the upside.” But you don't hear it the other way around where you have a startup that you're interviewing with, “Ha-ha, we're like an enterprise. We have a six-month interview process that takes 18 different stages,” and so on and so forth. However, we do see startups having to mature rapidly, and move up the compliance path as they're dealing with regulated entities and the rest, and wanting to deal with serious customers who have no sense of humor about, “Yeah, we'll figure that part out later as part of an audit document.”So, what we also see, though, is that enterprises are doing things that look a lot more startup-y. If I take a look at the common development environments and tools and techniques that big enterprises use, it looks an awful lot like how startups were doing it five or ten years ago. That is the slow and steady evolution of time. And what startups are doing today becomes enterprise tomorrow, and I can't shake the feeling that there's a sea of vendors out there who, in the event that winds up happening are eventually going to find themselves without a market at all. My model has been that if I go and found a Twitter for Pets style startup tomorrow and in ten years, it has grown to become an S&P 500 component—which is still easier to take seriously than most of what Tesla says—great.During that journey, at what point do I become a given company's customer because if there is no onboarding story for me to become your customer, you're in a long-tail decline phase. That's been my philosophy, but you are a—trademarked term—Enterprise Architect, so please feel free to tell me if I'm missing any of the nuances there, which I'm sure I am because let's face it, nuance is hard; sweeping statements are easy.Chris: As an architect, [laugh] it would be a disservice to not say my favorite catchphrase, it depends. There are so many dependencies to those kinds of sweeping statements. I mean, there's a lot of enterprises that have good process; there are a lot of enterprises that have bad process. And going back to your previous statement of the startup inside the enterprise, I'm hearing a lot of companies nowadays saying, “Oh, well, we've now got this brand new incubator system that we're currently running our little startup inside of. It's got the best of both worlds.”And I'm not going to go through the litany of bad things that you just said about startups, but they'll try to encapsulate that shift that you're talking about where the cheese is moving so quickly now that it's very hard for these companies to know the customer well enough to continue to stay salient and continue to be able to look into that crystal ball to stay relevant in the future. My job as an EA is to try to capture that point in time where what are the requirements today and what are the known detriments that you're going to see in your future that you need to protect against? So, that's kind of my job—other than being a cloud therapist—in a nutshell.Corey: I love the approach. My line has been that I do a lot of marriage counseling between engineering and finance, which is a fun term that also just so happens to be completely accurate.Chris: Absolutely. [laugh]. I'm currently being a marriage counselor right now.Corey: It's an interesting time. So, you had a viral tweet recently that honestly, I'm a bit jealous about. I have had a lot of tweets that have done reasonably well, but I haven't ever had anything go super-viral, where it was just a screenshot of a conversation you had with an AWS recruiter. Now, before we go into this, I want to make a couple of disclaimers here. Before I entered tech myself, I was a technical recruiter, and I can say that these people have hard jobs.There is a constant pressure to perform, it is a sales job that is unlike most others. If you sell someone a pen, great, you can wrap your head around what that's like. But you don't have to worry about the pen deciding it doesn't want to go home with the buyer. So, it becomes a double sale in a lot of weird ways, and there's a constant race to the bottom and there's a lot of competition in the space. It's a numbers game and a lot of folks get in and wash out who have terrible behaviors and terrible patterns, so the whole industry gets tainted—in some respects—like that. A great example of someone who historically has been a terrific example of recruiting done right has been Jill Wohlner. And she's one of the shining beacons of the industry as far as how to do these things in the right way—Chris: Yes.Corey: —but the fact that she is as exceptional as she is is in no small part because there's a lot of random folks coming by. All which is to say that our conversation going forward is not and should not be aimed at smacking around individual recruiters or recruiting as a whole because that is unfair. Now, that disclaimer has been given. Great, what happened?Chris: So, first off, shout out to Jill; she actually used to be a host on vBrownBag. So, hey girl. [laugh]. What happened was—and I have the utmost empathy and sympathy for recruiting; I actually used to have a side gig where I would go around to the local recruiting places around my area here and teach them how to read a cloud resume and how to read a req and try to separate the wheat from the chaff, and to actually have good conversations. This was back when cloud wasn't—this was, like, three or four years ago.And I would go in there and say, “This is how you recruit a cloud person nowadays.” So, I love good recruiters. This one was a weird experience in that—so when a recruiter reaches out to me, what I do is I take an assessment of my current situation: “Am I happy where I'm at right now?” The answer is, “Yes.” And if they ping me, I'll say, “Hey, I'm happy right now, but if you have something that is, you know, a million dollars an hour, taste-testing margaritas on St. John island in the sand, I'm all ears. I'm listening. Conversely, I also am a Community Hero, so I know a ton of people out in the industry. Maybe I can help you out with landing that next person.”Corey: I just want to say for the record, that is absolutely the right answer. And something like that is exactly what I would give, historically. I can't do it now because let's be clear here. I have a number of employees and, “Hey, Corey's out there doing job interviews,” sends a message that isn't good when it comes to how is that company doing anyway. I miss it because I enjoyed the process and I enjoyed the fun, but even when I was perfectly happy, it's, “Well, I'm not actively on the market, but I am interested to have a conversation if you've got something interesting.”Because let's face it, I want to hear what's going on in the market, and if I'm starting to hear a lot of questions about a technology I have been dismissive of, okay, maybe it's time to pay more attention. I have repeatedly been able to hire the people interviewing me in some cases, and sometimes I've gone on interviews just to keep my interview skills sharp and then wound up accepting the job because it turned out they did have something interesting that was compelling to me even though I was reasonably happy at the time. I will always take the meeting; I will always at least have a chat about what they're doing, and I think that doing otherwise is doing yourself a disservice in the long arc of your career.Chris: Right. And that's basically the approach that I take, too. I want to hear what's out there. I am very happy at World Wide right now, so I'm not interested, interested. But again, if they come up with an amazing opportunity, things could happen. So, I implied that in my response to him.I said, “I'm happy right now, thanks for asking, but let's set up the meeting and we can have a chat.” The response was unexpected. [laugh]. The response was basically, “If you're not ready to leave right now, it makes no sense for me to talk to you.” And it was a funny… interaction.I was like, “Huh. That's funny.” I'm going to tweet about that because I thought it was funny—I'm not a jerk, so I'm going to block out all of the names and all of the identifying information and everything—and I threw it up. And the commiseration was so impressive. Not impressive in a good way; impressive in a bad way.Every person that responded was like, “Yes. This has happened to me. Yes, this is”—and honestly, I got a lot of directors from AWS reaching out to me trying to figure out who that person was, apologizing saying that's not our way. And I responded to each and every single one of them. And I was like, “Somebody has already found that person; somebody has already spoken to that person. That being said, look at all of the responses in the timeline. When you tell me personally, that's not the way you do things, I believe that you believe that.”Corey: Yeah, I believe you're being sincere when you say this, however the reality of what the data shows and people's lived experience in the form of anecdotes are worlds apart.Chris: Yeah. And I'm an AWS Hero. [laugh]. That's how I got treated. Not to blow my own horn or anything like that, but if that's happening to me, either A, he didn't look me up and just cold-called me—which is probably the case—and b, if he treats me like that, imagine how he's treating everybody else?Corey: This episode is sponsored in part by something new. Cloud Academy is a training platform built on two primary goals. Having the highest quality content in tech and cloud skills, and building a good community the is rich and full of IT and engineering professionals. You wouldn't think those things go together, but sometimes they do. Its both useful for individuals and large enterprises, but here's what makes it new. I don't use that term lightly. Cloud Academy invites you to showcase just how good your AWS skills are. For the next four weeks you'll have a chance to prove yourself. Compete in four unique lab challenges, where they'll be awarding more than $2000 in cash and prizes. I'm not kidding, first place is a thousand bucks. Pre-register for the first challenge now, one that I picked out myself on Amazon SNS image resizing, by visiting cloudacademy.com/corey. C-O-R-E-Y. That's cloudacademy.com/corey. We're gonna have some fun with this one!Corey: Every once in a while I get some of their sourcers doing outreach to see folks who are somewhat aligned on them via LinkedIn or other things, and, “Oh, okay, yeah; if you look at the things I talked about in various places, I can understand how I might look like a potentially interesting hire.” And they send outreach emails to me, they're always formulaic, and once in a while, I'll tweet a screenshot of them where I redact the person's name, and it was—and there's a comment, like, “Should I tell them?” Because it's fun; it's hilarious. But I want to be clear because that often gets misconstrued; they have done absolutely nothing wrong. You've got to cast a wide net to find talent.I'm surprised I get as few incidents of recruiter outreach as I do. I am not hireable and that's okay, but I don't begrudge people reaching out. I either respond with a, “No thanks,” if it's a particularly good email, or I just hit the archive button and never think about it again. And that's fine, too. But I don't make people feel like a jerk for asking, and that is an engineering behavioral pattern that drives me up a wall.It's, “So, I'm thinking about a job here and I'm wondering if you might be a fit,” and your response is just to set them on fire? Well, guess what an awful lot of those people sending out those emails in the sourcing phase of recruiting are early career, and guess what, they tend to get promoted in the fullness of time. Sometimes they're no longer recruiting at all; sometimes they wind up being hiring managers in different ways or trying to figure out what offer they're going to extend to someone. And if you don't think that people in those roles remember when they're treated poorly as a response to their outreach, I have news for you. Don't do it. Your reputation lingers long after you no longer work there.Chris: Just exactly so. And I feel really bad for that guy.Corey: I do hope that he was not reprimanded because he should not be. It is clearly a systemic problem, and the fact that one person happened to do this in a situation where it went viral does not mean that they are any worse than other folks doing it. It is a teachable opportunity. It is, “I know that you have incredible numbers of roles to hire for, all made all the more urgent by the fact that you're having some significant numbers of departures—clearly—in the industry right now.” So, I get it; you have a hard job. I'm not going to waste your time because I don't even respond to them just because, at AWS particularly, they have hard work to do, and just jawboning with me is not going to be useful for them.Chris: [laugh].Corey: I get it.Chris: And you're trying to hire the same talent too. So.Corey: Exactly. One of the most egregious things I've seen in the course of my career was when that whole multiple accounts opened for Wells Fargo's customers and they wound up firing 3500 people. Yeah, that's not individual tellers doing something unethical. That is a systemic problem, and you clean house at the top because you're not going to convince me that you're hiring that many people who are unethical and setting out to do these things as a matter of course. It means that the incentives are wrong, it means that the way you're measuring things are wrong, and people tend to do things out of fear or because there's now a culture of it. And if you fire individuals for that, you're wrong.Chris: And that was the message that I conveyed to the people that reached out to me and spoke to me. I was like, there is a misaligned KPI, or OKR, or whatever acronym you want to use, that is forcing them to do this churn-and-burn mentality instead of active, compassionate recruiting. I don't know what that term is; I'm very far removed from the recruiting world. But that person isn't doing that because they're a jerk. They're doing that because they have numbers to hit and they've got to grind out as many as humanly possible. And you're going to get bad employees when you do that. That's not a long-term sustainable path. So, that was the conversation that I had with them. Hopefully, it resonated and hits home.Corey: I still remember from ten years ago—and I don't always tell the story, but I absolutely will now—I went up to San Francisco when I lived in Los Angeles; I interviewed with Yammer. I went through the entire process—this was not too long before they got acquired by Microsoft so that gives you some time basis—and I got a job offer. And it was a not ridiculous offer. I was going to think about it, and I [unintelligible 00:24:19], “Great. Thank you. Let me sleep on this for a day or two and I'll get back to you definitely before the end of the week.”Within an hour, I got a response rescinding the offer claiming it had been sent by mistake. Now, I believe that that is true and that they are being sincere with this. I don't know that if it was the wrong person; I don't know if that suddenly they didn't have the req or they had another candidate that suddenly liked better that said no and then came back and said yes, but it's been over a decade now and every time I talk to someone who's considering something in that group, I tell this story. That's the sort of thing that leaves a mark because I have a certain philosophy of I don't ever resign from a job before I wind up making sure everything is solid—things are signed, good to go, the background check clears, et cetera—because I don't want to find myself suddenly without income or employment, especially in that era. And that was fine, but a lot of people don't do that.As soon as the offer comes in, they're like, “I'm going to go take a crap on my boss's desk,” which, let's be clear, I don't recommend. You should write a polite and formulaic resignation letter and then you should email it to your boss, you should not carve it into their door. Do this in a responsible way, and remember that you're going to encounter these people again throughout your career. But if I had done that, I would have had serious problems. And so that points to something systemically awful at a company.I have never in my career as a hiring manager extended an offer and then rescinded it for anything other than we can't come to an agreement on this. To be clear, this is also something I wonder about in the space, when people tell stories about how they get a job offer, they attempt to negotiate the offer, and then it gets withdrawn. There are two ways that goes. One is, “Well if you're not happy with this offer, get out of here.” Yeah, that is a crappy company, but there's also the story of people who don't know how to negotiate effectively, and in turn, they come back with indications that you do not know how to write a business email, you do not know how negotiations work, and suddenly, you're giving them a last-minute opportunity to get out before they hire someone who is going to be something of a wrecking ball in the company, and, “Whew, dodged a bullet on that.”I haven't encountered that scenario myself, but I've seen it from other folks and emails that have been passed around in various channels. So, my position on this is everyone should negotiate offers, but visit fearlesssalarynegotiation.com, it's run by my friend, Josh; he has a whole bunch of free content on his site. Look at it. Read it. It is how to handle this stuff effectively and why things are the way that they are. Follow his advice, and you won't go too far wrong. Again, I have no financial relationship, I just like what he's done a lot and I've been talking to him for years.Chris: Nice. I'll definitely check that out. [laugh].Corey: Another example is developher—that's develop H-E-R dot com. Someone else I've been speaking to who's great at this takes a different perspective on it, and that's fine. There's a lot of advice out there. Just make sure that whoever it is you're talking to about this is in a position to know what they're talking about because there's crap advice that's free. Yeah. How do you figure out the good advice and the bad advice? I'm worried someone out there is actually running Route 53 is a database for God's sake.Chris: That's crazy talk. Who would do that? That's madness.Corey: I can't imagine it.Chris: We're actually in the process of trying to figure out how to do a panel chat on exactly that, like, do a vBrownBag on salary negotiations, get some really good people in the room that can have a conversation around some of the tough questions that come around salary negotiation, what's too much to ask for? What kind of attitude should you go into it with? What kind of process should you have mentally? Is it scrawling in crayon, “No. More money,” and then hitting send? Or is it something a little bit more advanced?Corey: I also want to be clear that as you're building panels and stuff like that—because I got this wrong early on in my public speaking career, to be clear—I built talks aligned with this based on what worked for me—make sure that there are folks on the panel who are not painfully over-represented as you and I are because what works for us and we're considered oh, savvy business people who are great negotiators comes across as entitled, or demanding, or ooh, maybe we shouldn't hire her—and yes, I'm talking about her in a lot of these scenarios—make sure you have a diverse group of folks who can share lived experience and strategies that work because what works for you and me is not universal, I promise.Chris: So, the only requirement to set this panel is that you have to be a not-white guy; not-old-white guy. That's literally the one rule. [laugh].Corey: I like the approach. It's a good way to do it. I don't do manels.Chris: Yes. And it's tough because I'm not going to get into it, but the mental space that you have to be in to be a woman in tech, it's a delicate balance because when I'm approaching somebody, I don't want to slide into their DMs. It's like this, “Hey, I know this other person and they recommended you and I am not a weirdo.” [laugh]. As an old white guy, I have to be very not a weirdo when I'm talking to folks that I'm desperate to get on the show.Because I love having that diverse aspect, just different people from different backgrounds. Which is why we did the entire career series on vBrownBag. We did data science with Ayodele; we did how to get into cybersecurity with Christoph. It was a fantastic series of how to get into IT. This was at the beginning of the pandemic.We wanted to do a series on, okay, there's a lot of people out there that are furloughed right now. How do we get some people on the show that can talk to how to get into a part of IT that they're passionate about? We did a triple series on how to get into game development with Dennis Diack, the founder of Apocalypse Studios. We had a bunch of the other AWS Heroes from serverless, and Lambda, and AI on the show to talk, and it was really fantastic and I think it resonated well with the community.Corey: It takes work to have a group of guests on things like podcasts like this. You've been running vBrownBag for longer than I've been running this, and—Chris: 13 years now.Corey: Yeah. This is I think, coming up on what, four years-ish, maybe three, in that range? The passing of time, especially in a pandemic era, is challenging. And there's always a difference. If I invite a white dude to come on the podcast, the answer is yes before I get the word podcast fully out of my mouth, whereas folks who are not over-represented, they're a little more cautious. First, there's the question of, “Am I a trash bag?” And the answer is, “No.” Well, no, not in the way that you're concerned about other ways—Chris: [laugh]. That you're aware of. [laugh].Corey: Oh, God, yes, but—yeah. And then—and that's part of it, and then very often, there's a second one of, “Well, I don't think I have anything, really, to talk about,” is often a common objection here. And it's, yeah, if I'm inviting you on this show, I promise that's not true. Don't worry about that piece of it. And then it's the standard stuff that just comes with being me, of, “Yeah, I've read your Twitter feed; you got to insult me here?” It's, “No, no, not really the same tone. But great question; throw the”—it goes down to process. But it takes constant work, you can't just put an open call out for guest nominations, and expect that to wind up being representative of our industry. It is representative of our biases, in many respects.Chris: It's a tough needle to thread. Because the show has been around for a long time, it's easier for me now, because the show has been around for 13 years. We actually just recorded our two thousandth and sixtieth episode the other night. And even with that, getting that kind of outreach, [#techtwitter 00:31:32] is wonderful for making new recommendations of people. So, that's been really fun. The rest of Twitter is a hot trash fire, but that's beside the point. So yeah, I don't have a good solution for it. There's no easy answer for it other than to just be empathic, and communicative, and reach people on their level, and have a good show.Corey: And sometimes that's all it takes. The idea behind doing a podcast—despite my constant jokes—it's not out of a love affair of the sound of my own voice. It's about for better or worse, for reasons I don't fully understand, I have a platform. People listen to the show and they care what people have to say. So, my question is, how can I wind up using that platform to tell stories that lift up narratives that are helpful for folks that they can use as inspiration—in my case, as critical warnings of what to avoid—and effectively showcasing some of the best our industry has to offer, in many respects.So, if the guest has a good time and the audience can learn something, and I'm not accidentally perpetuating horrifying things, that's really more than I have any right to ask from a show like this. The fact that it's succeeded is due in no small part to not just an amazing audience, but also guests like you. So, thank you.Chris: Oh no, Thank you. And it is. It's… these kinds of shows are super fun. If it wasn't fun, I wouldn't have done it for as long as I have. I still enjoy chatting with folks and getting new voices.I love that first-time presenter who was, like, super nervous and I spend 15 minutes with them ahead of the show, I say, “Okay, relax. It's just going to be me and you facing each other. We're going to have a good time. You're going to talk about something that you love talking about, and we're going to be nerds and do nerd stuff. This is me and you in front of a water cooler with a whiteboard just being geeks and talking about cool stuff. We're also going to record it and some amount of people is going to see it afterwards.” [laugh].And yeah, that's the part that I love. And then watching somebody like that turn into the keynote speaker at a conference ten years down the road. And I get to say, “Oh, I knew that person when.”Corey: I just want to be remembered by folks who look back fondly at some of the things that we talk about here. I don't even need credit, just yeah. People who see that they've learned things and carry them forward and spread to others, there's so many favors that people have done for us that we can only ever pay forward.Chris: Yeah, exactly. So—and that's actually how I got into vBrownBag. I came to them saying, “Hey, I love the things that you guys have done. I actually passed my VCIX because of watching vBrownBags. What can I do to help contribute back to the community?” And Alistair said, “Funny you should mention that.” [laugh]. And here we are seven years later.Corey: Well, to that end, if people are inspired by what you're saying and they want to hear more about what you have to say or, heaven forbid, follow in your footsteps, where can they find you?Chris: So, you can find me on Twitter; I am at mistwire.com—M-I-S-T-W-I-R-E; if you Google ‘mistwire,' I am the first three pages of hits; so I have a blog; you can find me on vBrownBag. I'm hard to miss on Twitter [laugh] I discourage you from following me there. But yeah, you can hit me up on all of the formats. And if you want to present, I'd love to get you on the show. If you want to learn more about what it takes to become an AWS Hero or if you want to get into that line of work, I highly discourage it. It's a long slog but it's a—yeah, I'd love to talk to you.Corey: And we of course put links to that in the [show notes 00:35:01]. Thank you so much for taking the time to speak with me, Chris. I really appreciate it.Chris: Thank you, Corey. Thanks for having me on.Corey: Chris Williams, Enterprise Architect, comma AWS Cloud at WWT. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with a comment telling me that while you didn't actively enjoy this episode, you are at least open to enjoying future episodes if I have one that might potentially be exciting.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Serverless Chats
Episode #116: Infinite Serverless Workflows with Sam Dengler and Justin Callison

Serverless Chats

Play Episode Listen Later Oct 25, 2021 52:12


Sam Dengler is a Principal Solutions Architect and Justin Callison is an Engineering manager of Workflow service (including Step Functions) at Amazon Web Services. Twitter: Justin Callison @justincallison, Sam Dengler @samdengler Step Functions: https://aws.amazon.com/step-functions Share Your Integration Innovations in the Flex Your Skills Contest on AWS More Resources:  AWS Step Functions integrates with over 200 AWS SERVICES (Marcia Villalba) Serverless Office Hours: AWS Step Functions - AWS SDK Service Integrations Taco Bell: This is My Architecture video (we'll link to this)

Reversim Podcast
424 Melio's payment processor

Reversim Podcast

Play Episode Listen Later Oct 25, 2021


[קישור לקובץ mp3] שלום וברוכים הבאים לפודקאסט מספר 424 של רברס עם פלטפורמה - יצא מספר פלידנרומי, איזה מגניב! . . . - (אורי) 424?! . . . - (רן) 424 . . . התאריך היום הוא 17 באוקטובר, השעה היא 21:30 עוד מעט והשנה היא 2021 - (אורי) והטמפרטורות התחילו לרדת היום, גם היה גשם . . . - (רן) היה גשם היום, נכון, סוף סוף . . . היום אנחנו מתכבדים לארח את אילן ואת אור מחברת Melio - זה Mi-lio או Me-lio? . . .(אילן) האמת שזו שאלה מאוד טובה, כי כשהקמנו את החברה אז קראנו לה באמת Me-lio, אבל כשהתחלנו לדבר עם אנשים מארה”ב, אמרו לנו שיש משהו שנקרא Long e ו- Short e, שזה משהו שלא הכרנו . . . אז חלקם הוגים “Mi-lio” וחלקם הוגים “Me-lio” . . . מבחינתנו זה “Mi-lio”.(אורי) זה בטח ה-Domain שהיה פנוי . . . (אילן) האמת שה-Domain שהיה פנוי היה Mi-lio(paymnets.com), אבל ככל שהצלחנו לגדול והחלטנו שהשם זה ממש משהו שאנחנו שלמים איתו - כי היה גם שם תהליך, אבל זה סיפור לפודקאסט אחר - אז קנינו את melio.com, שהיה קצת יקר אבל הצלחנו להשיג, ארבע אותיות וגם com., אירוע קצת . . . (אורי) טוב, אז זה היה אילן . . .(רן) כן, אנחנו נעשה היום פודקאסט הפוך - נתחיל מהסוף . . . .כן - אז אנחנו שמחים ומתכבדים לארח פה את אילן ואור מחברת Melio - אנחנו נדבר על Melio ועל פלטפורמת התשלומים והטכנלוגיה שפיתחתם כדי באמת לממש את כל הסיפור הזה.אבל לפני זה - בואו נכיר אתכם: אילן - בבקשה:(אילן) תודה רבה שאתם מארחים אותנו - כבוד גדול.אני מכיר את אורי ורן עוד מלפני מספר שנים, כבוד הוא לנו לבוא לפודקאסט אני אילן, אחד ה-Co-Founders וה-CTO של Melio - הקמנו את החברה לפני כשלוש-וחצי שנים רשמית - קצת לפני עבדנו עוד בגראז', להבין מה אנחנו רוצים לעשותאני אספר על זה קצת עוד מעטשנים יחסית אינטנסטיביות בשנים האחרונותלפני כן הייתי ה-VP Engineering בחברה בשם Winward, ולפני זה עבדתי ב-Outbrain כשנתיים + . . .איתי פה נמצא אור . . .(רן) אור - ברוך הבא!(אור) תודה רבה - אני היום ב-Melio ה-Principal Engineer, הצטרפתי יחסית ממש בהתחלה. זה היה . . .לפני כן הייתי Co-Founder בסטארטאפ אחר, ולפני כן הייתי יועץ באיזושהי חברת נקרא לזה “בוטיק-DevOps” קטן שנקרא FewBytes ואחרי שעזבתי את ה-Startup שלי בתור Co-Founder - הייתי Co-Founder ממש לא טוב - מישהו שידך ביני לבין אילן והאמת שממש בשיחות הראשונות עם הפאונדרים של Melio זה פשוט . . . אני יכול להגיד באופן אישי שזה היה מעיין “אהבה ממבט ראשון”, ממש “עפתי עליהם” עד הסוף ואמרתי “אני רוצה לעבוד פה” וכל השאר פחות או יותר היסטוריה . . .(רן) אז Melio - אני מניח שיש כאלה שכבר שמעו את השם, אבל למי שעוד לא שמע: מה עושה Melio?(אילן) אנחנו פיתחנו ומפתחים פלטפורמה לעסקים קטנים, להעברות תשלומים.ככל שזה יהיה אולי מופלא ואולי לא לחלק מהמאזינים או למי שמקשיב, תשלומים, בארה”ב בעיקר, עדיין רובם ככולם מועברים על גבי פיסות נייר - שהם שיקים . . .סדר גודל של 18 טריליון דולר נעים בארה”ב בין עסקים קטנים בכל שנהסדר גודל של כחמישה מיליארד שיקים נכתבים בין עסקיםכשאנחנו . . . זה סדר גודל שראינו לפני ארבע שנים, ואמרנו “רגע - זה לא הגיוני”בעולם שה-Digital Payments קורים בין חברים, כלומר - היום להעביר כסף בין Friends & Family קורה בצורה מאוד פשוטה, יש הרבה מאוד אפליקציות שאתה יכול באמצעותן להעביר כסף בצורה סופר-קלה.אם אתה עכשיו בתור Consumer שרוצה לעשות Check-out ב-Online, התהליך הוא מאוד מאוד מתקדם, כל עולם ה-eCommerce.אתה יכול לעשות Check Out עם Stripe או עם כרטיסי אשראי או Affirm או עם Klarna או עם כל שיטת Check out אחרת.עדיין, תשלומים לספקים, רובם ככולם, מועברים בעצם על פני פיסות נייר - שיקים, העברות בנקאיות - דרך כלים שהם מחוץ . . . בעצם כלי מערכת, שהם בעיקר כלים של הבנקים.(רן) אפילו בקרנות הון סיכון אומרים “I'll write you a check” . . . (אילן) I'll write you a check”, Yes” . . . וגם אנחנו היום, כ-Melio - אנחנו כותבים שיקים . . .בעצם, יש לנו ספקים שרוצים לקבל רק שיקים . . .והבעיה הזו נראתה לנו די מעניינת ומאוד מאתגרת - אמרנו “איך זה יכול להיות, בעולם ש-Payments עוברים ו-Shifting ל-Digital בצורה מאוד מאסיבית, עדיין עולם ה-Supplier Payments נמצא על גבי פיסות נייר” . . .(רן) על אילו סוגי עסקים אנחנו מדברים? מספרות, וטרינרים, . . . ?(אילן) אז אנחנו מדברים כמעט על כל סוגי העסקים - זה יכול להיות כמו שאמרת - מספרות וטרינרים, מסעדות, Doctor Offices למינהם, Professional Services, צלמים . . . (אורי) גולדמן-סאקס?(אילן) גולדמן-סאקס . . . גם כאלה, הגדולים . . . ובאמת לחברות כמו Nike או Fortune-500's יש כלים, היום, לעשות גם Procurement וגם Paymentsאבל כשאתה הולך לעסק הקטן - מה שנקרא Owner-Operated Business - לבעל העסק כיום אין כלי מתאים כדי לנהל את תשלומי הספקים שלו.ומה שמאפיין בעצם את אותם עסקים זה שאין להם היום איזה Bookkeeper או איזשהו Accounts-Payables Expert שעושה עבורם את ה-Paymentsאנחנו היום ב-Melio, או אצלכם ב-Outbrain - יש בעצם Finance Department, שמתעסקים ב-Accounts-Payablesאבל אם אני עסק קטן, אם אני עכשיו בעל מסעדה ויש לי חמישה-עשרה עובדים - בדרך כלל מי שמטפלים בזה זה או אני או מישהו שהוא Trusted Employee.והיום עסק קטן ממוצע - העסקים שאנחנו מטרגטים (Targeting), של 5-10 עובדים, סדר גודל של 1-2 מיליון דולר Revenue בשנה - מוציאים סדר גודל של 50-60 Payments בחודש.וה”אירוע” הזה הוא בדרך כלל Heavy . . . בדרך כלל נעשה ידנית . . .(רן) אנחנו תיכף נצלול לסיפור הטכנולוגי שם, אבל קצת בכל אופן כדי להבין את הרקע - פתאום קם אדם בבוקר ומרגיש שהוא חייב לעשות מערכת Payments? זאת אומרת - איך קורה שילד-טוב-ירושלים, אילן, אחד מהפאונדרים של החברה, מחליט שבא לו להרים מערכת Payments לעסקים בארה”ב?(אילן) אז מה שבעיקר משך אותנו זה גודל ההזדמנות - באנו ואמרנו רגע, עסקים קטנים - סליחה על הקלישאה אבל זה The Backbone of the economy . . . בסופו של יום, הדרך שהם מתנהלים - גם ברמת האופרציה של להוציא את התשלומים וגם האופרציה גם גוררת . . . אופרציה לא יעילה גורמת לניהול תזרימים מאוד לא טוב עבור העסק.עסקים קטנים - אם אתה מסתכל על הסיבות שעסקים נסגרים לרוב - אז חלק נותנים שירות לא טוב או מוצר לא טוב, אבל בהרבה מאוד פעמים זה נובע מכך שהם לא יודעים לנהל נכון את “האירוע התזרימי”, אתה-Cash Flow.והרבה פעמים זה קורה בגלל היעדר יכולת אופרטיבית והבנה של מה בעצם צריך להוציא היום ומה אפשר להוציא מחר ומה אפשר לנהל בצורה יותר חכמה.באמת, מה שהדליק אותנו, מה שבעצם גרם לנו להגיד זה איך אנחנו יכולים לעזור לעסקים קטנים? - על ידי זה שנוכל בעצם לקחת את עולם ה-Payments שלהם לעולם ה-Digital, ולנהל בעיקר את ה-Cash Flow.(רן) אוקיי, אז אני לא מבין גדול בעולם ה-Finance, אבל אני כן יודע שיש כמה חברות וכמה ספקי תשלומים גדולים - הזכרתם אני חושב את Stripe ויש עוד כל מיני גדולים אחרים . . . (אורי) . . . האם בין ה-CRM לניהול הכספי - CRM זה יותר לצד הלקוחות . . . (רן) . . . כן, נשים לרגע את הסיפור העסקי בצד - אני מניח שיש סיבה למה Stripe לא מתאים להם, אבל אתם גם החלטתם לייצר מערכת תשלומים פנימית, זאת אומרת - לנהל את הכל אצלכם. למה לעשות את זה ולמה לא להשתמש באיזשהו צד שלישי - איזשהו בנק, ב-Stripe או כל דבר אחר כזה?(אילן) לפני שאני אענה על השאלה הזאת, אני אקח לרגע צעד אחורה - הסיבה בעצם כיום לכך שעסקים בעיקר מתנהלים - לתשלומי ספקים - בעיקר עם שיקים, זה בגלל חוסר ההסכמה, לרוב, הבסיסי בין איך שצד אחד רוצה לשלם לאיך שהצד השני בעצם רוצה לקבל את הכסף.היום, כשאני הולך ועושה Check out online, ויש שם איזשהו Check out עם Stripe - אז אני יכול לשלם בכרטיס אשראי, והצד השני יקבל את זה לחשבון הבנק שלו, בעצם.יש איזשהו “נדל”ן”, שזה ה-Point-of-Sale, שיכול לסלוק את כרטיס האשראי שלי - והצד השני יקבל את הכסף.בעולם ה-B2B, לרוב הטרנזקציות (Transactions), החלק הארי של הטרנזקציות קורה OTC - Over the Counter.אין בעצם היום איזשהו Point of Sale - לא לרכישה ולא לתשלום - וה-Point of Sale שבעצם קיים זה ה-Invoice.כשאני מזמין, לדוגמא, מהספק דגים שלי עשרה ק”ג סלמון למסעדה - יחד עם הדגים אני מקבל בעצם Invoice, ושם אני אמור לשלם את התשלום עבור הדגים באיזה Net Terms.עכשיו - אני ספק דגים שכבר קיים בשוק 20 שנה, ואני עכשיו מקבל ממאות לקוחות כסף - ובאיזשהו מקום אני לא בהכרח רוצה לתמוך בעוד שיטת תשלוםכי כל תהליך ה-Finance שבניתי או כל תהליך ה-Reconciliation שבניתי בעצם בנוי מעל שיקים, שמגיעים אליאני יודע איך הכסף מגיע ואיך לקשור אותו לחשבונית המתאימה.אבל אותה מסעדה שנפתחה עכשיו, מסעדה חדשה שלא בהכרח רוצה לשלם בשיקים - רוצה לשלם בכרטיס אשראי, רוצה לשלם בהעברה בנקאית . . . .שני הצדדים לא מסכימים על אמצעי התשלום.(אורי) . . . ואז יורדים למכנה המשותף הכי נמוך - שזה השיקים . . .(אילן) בול . . . ולכן מגיעים בדיוק למכנה המשותף הנמוך ביותר שזה השיקים - שיק - ברור שהוא מתקבל בכל מקום, ברור שהוא “ג'וקר”, ואתה יכול בעצם לתת אותו, ובעצם זה סוג של סטנדרט . . . (אורי) זה נייר - אפשר לעטוף איתו דגים . . .(רן) יש יותר נמוך - יש Cash . . . אבל לשם עוד לא ירדנו . . . יש מטבעות זהב . . .(אורי) נייר . . .(רן) אז בעצם אתם החלטתם שאתם בונים איזשהו Transpiler - משהו שמתרגם דיגיטלי לנייר, נייר לדיגיטלי או כל מיני תרגומים אחרים שקיימים . . . (אילן) בדיוק, ולשאלתך של למה בעצם בנינו Payment Infrastructure - כדי להגיע למצב שאנחנו בעצם נוכל לבוא ולשרת את אותם עסקים, הרי היינו צריכים לייצר איזושהי “חוסר תלות” בין הצדדים - Decupling בין המשלם למקבלובנינו בעצם Payments Infrastructure חדש, היום כבר מעל שלושה בנקים - Evolve Bank & Trust ו-Silicon Valley Bank ו-JPMorgan Chaseבעצם בנינו יכולת לבוא ולסלוק כסף מהמשלם בכל דרך שנרצה - זה יכול להיות כרטיס אשראי, זה יכול להיות Debit Card, זה יכול להיות בנק, זה יכול להיות PayPal, זה יכול להיות Apple Pay . . . אנחנו יכולים לסלוק כסף בכל דרך אפשרית - ולהוציא אותו מהצד השני בכל דרך שהצד השני יחפוץ בהבעצם יצרנו ניתוק בין שני הצדדים - מה שנותן לנו היום המון כוח לבוא לעסק - לבוא למסעדה או לאיזשהו צלם או מספרה או כל מקום אחר - ולהגיד “אוקיי, לא משנה עכשיו, אתה לא צריך לשכנע את הצד השני איך לקבל את הכסף, תן להם באיזו דרך שהם יחפצו, ואתה תשלם איך שאתה רוצה”.(אורי) יש גם, כאילו את “הדרך של Melio”, את ה . . . לא יודע, כרטיס או סוג של bit כזה . . . אפליקציה שהיא אפליקציית-סליקה, שאם היא מתאימה לשני הצדדים אז מה טוב, אבל אתה יכול גם דרכה לקבל ול . . .?(אילן) אז הדרך שאנחנו היום . . . בדוגמא שנתתי, נניח שאני מסעדה, אז אני יכול לסלוק, אני יכול עכשיו לבחור לשלם באשראי, יכול לבחור לשלם בבנק - ב-Bank Transfer - ואתה תקבל איך שתחפוץ, נניח שיקים או העברה בנקאית או כל דרך אחרת.אנחנו כן מייצרים . . . אנחנו נייצר בעצם סוג של . . . אם אני מבין נכון את השאלה שלך, מעיין Wallet, כך שאפשר בעצם, ברגע ששני הצדדים ב-Network, אז בעצם נוכל להעביר כסף - שהוא בעצם Wallet, שכל אחד יוכל להשתמש בו ב-Network עצמו.עכשיו, אחד הדברים הנוספים שיצרנו ב-Payments Infrastructure הזה זה בעצם, שלהבדיל ממערכות כמו bit או Pepper, או בארה”ב Venmo או PayPal, ששני הצדדים צריכים להיות ב-Network על מנת שצד אחד יוכל לשלם לצד השני - אנחנו בעצם יצרנו יכולת של מה שאנחנו קוראים לו Open Network - רק צד אחד צריך להיות ברשת על מנת שהצד השני יקבל את הכסף.על ידי כך, בעצם הורדנו את העומס ממי שכרגע משתמש בנו, כדי לשכנע שהצד השני יכנס.(רן) כן, אז החלטתם והבנתם שאתם רוצים להציע מערכת תשלומים נורא גמישה שהיא Open ואתה יכול לשלם איך שאתה רוצה ואתה יכול לקבל את הכסף איך שאתה רוצה - אתה בא לאור, “המתכנת המסכן”, אומר לו: “אור, בוא תבנה לי כזה!” . . . איך מתחילים? מה האתגרים פה? איך בכלל מתחילים לבנות מערכת Payments כזו מאפס?(אורי) אז אור מוציא לו חשבונית . . . (אור) אז באמת, Melio זה קצת יותר ממערכת תשלומים, מן הסתם - חלק גדול מאוד מהמערכת מבוסס על Interface ממש נוח - בגלל שזה Small Business, בגלל שאין להם כל הרבה זמן להתעסק עכשיו עם איזושהי מערכת Business-ית מורכבת, שבדרך כלל פונה לעסקים, אז בגדול, מה שאני הצעתי לאילן כשהתחלנו היה שאמרתי “אילן, תשמע - אנחנו נמצאים עכשיו On the verge of Serverless”, יש לנו הזדמנות לא לתחזק שרתים! יש לנו הזדמנות להינות מהיתרונות . . . “(רן) . . . ואומר את זה אחד שתחזק כבר הרבה שרתים, אמרת ב-Intro . . . (אור) בדיוק - מה שאצלי בראש היה זה שאני לא רוצה להגדיר יותר NTP בחיים, לעולם . . . אז אמרתי לאילן “בוא נעשה Serverless! בוא נלך על זה ובוא נראה אם זה עובד לנו”.[משלנו!]ואילן זרם איתי . . . עשה לי בהתחלה פרצוף של “אתה חושב? אתה בטוח?”, אבל אמרנו “יאללה, בוא נלך על זה” . . .(רן) “זה לא Hype, זה לא כמו GraphQL שיעבור עוד מעט? . . . .”(אור) בדיוק . . . באמת, היו לו ספקות קצת בהתחלה, ואמרתי לו “שמע - עלי! מה שלא יעבוד, אנחנו נסדר”.ואז באמת בנינו את המערכת - ה-Payments Processing שלנו בעצם רץ Serverless.האמת שחלק גדול מאוד מתשתית של Melio רץ רק על Serverless, רק על Lambda, ספציפית על Lambda.(רן) והמוטיבציה היא באמת “אני לא רוצה את כאב הראש הזה של NTP”, או שיש גם סיבות ארכיטקטוניות אחרות?(אור) זה מאוד . . . .(אורי) . . . זה מאוד Stream-oriented, נכון? זה Processing של Streams של Data, וזה נשמע מתאים . . . (אילן) אז זו נקודה מאוד חשובה, מה שאמרת עכשיו - בסופו של יום, תשלומים - רובם יוצאים או בהעברות בנקאיות מצד אחד, או בשיקים . . . עדיין אנחנו מוציאים שיקים - Melio מוציאה היום סדר גודל של מאות אלפי שיקים כל חודש, כי עדיין הספקים רוצים לקבל שיקים . . .תהליך גביית התשלום הוא באמת Stream-oriented, כלומר - אני יכול להיכנס למערכת ולקבוע תשלום.אני יכול לקבוע אותו לעכשיו, אני יוכל לקבוע אותו להיום או למחר לעוד חודש - אבל בסופו של יום, כל או רוב התשלומים מתמקדים בעצם בנקודת זמן אחת.בסופו של יום, כדי להעביר כסף בהעברה בנקאית או בשיק - זה דווקא Batch-oriented, כלומר הכל מתרכז בנקודה אחת, כי הבנקים בסוף עובדים ב-Cut-off-ים . . .זאת אומרת שכשאני רוצה להעביר כסף מנקודה A ל-B, בעצם יש Cut-off של הבנקה-Cut-off של הבנק הוא ב-2300 או 2400 Central Time בארה”ב, ואז בעצם בנקודה הזאת אנחנו לוקחים את כל השלבים שנקבעו להיום, או שנקבעו למועד שאנחנו רוצים - ובעצם מוציאים אותם.מה שאומר שהמערכת מקבלת Event-ים, מקבלת פקודות, ב-Stream - אבל בסופו של יום, היא מתנקזת לנקודה אחת, שבה צריכים להעלות את אותו קובץ, אותו Ledger, לבנק, כדי לבצע את התשלומים השונים - או להוציא שיקים או . . .(אורי) הבנתי שאופי הטרנזקציות האלה זה אופי שלא מצריך State, כמעט . . . (אור) נכון - אז באמת, אני מוכרח להודות שבהתחלה המוטיבציה הייתה מאוד “לנהל כמה שפחות” ולאט לאט, עם הזמן - האמת שדי מהר - ראינו שזה משחק לטובתינו בעוד מקרים, כי יש לנו את הצד . . .צריך להבין שהשוק הזה הוא נורא נוח, כי . . . במובנים מסויימים הוא מאוד נוח ונקרא לזה “פריוויבלגי” לנו, כ-Business - כי מדובר בעסקים, אז הם עובדים 0900-1700, זה רוב העומס שיש לנו במערכתהם לא עובדים בשבת, הם לא עובדים בראשוןהבנקים לא עובדים בשבת ולא עובדים בראשון - אז אנחנו לא עושים Processing בימים האלו.יש לנו פריווילגיה מאוד גדולה להפעיל את המערכת רק בזמנים מסויימים,וגם בתוך אותם ימים - רק בשעות מסויימות(רן) אילו זה רק היה בשעון ישראל אז זה היה אידיאלי . . . (אור) כן, זה היה מושלם . . . אז במובן הזה, Serverless מאוד עזר לנו, כי אם ניקח לרגע רק את ה-Payments Processing -אז 90% מהיום זה 0, לא קורה שום דבר . . .אולי יש כל מיני Management ו-Logistic tasks וכאלו שרצים ברקע, אבל חוץ מזה - כלום.ואז, ב-Trigger מסויים ביום, במערכת מתחילה לעבוד, עושה את כל ה-Processing שהיא צריכה לעשות - וחוזרת לישון.(אורי) זה מזכיר לי קצת ב”רמזור” כשהוא מלמד ריקוד במשרד רואה חשבון - “מה קורה כל החודש? כלום-כלום-כלום . . . 15 לחודש?! אוו . . . .”.(רן) אז אתה אומר שהיכולת היפה של פונקציות Lambda לעשות Scale-up באופן מיייד ואחר כך לכבות לכמעט אפס - זה יתרון ארכיטקטוני אחד . . . דרך אגב, לגבי ה-State שהזכרתם פה, אז לפחות בדרך שבה אני מדמיין, דווקא ב-Payments אני מדמיין שקיים הרבה מאוד State, רק שהוא תמיד צריך להיות Persistent, את אומרת - הוא אף פעם לא In-Memory, כי אסור לאבד אותו . . . אז אולי זה לא נכון להגיד ש”לא קיים State”, אבל ה-State תמיד חייב להיות Persistent . . . .(אור) נכון - ה-State, במקרה שלנו, בוא נגיד . . . . אנחנו לא “Serverless קלאסי”, נקרא לזהה-State שלנו יושב על Database טרנזקציוני (Transactional), הטרנזקציות שלנו הן בתוך ה-Lambda, מן הסתם גם חלק מה-Processing של מה שה-Lambda עושהחלק ממה שאנחנו עושים בעבודה מול הבנקים זה בעצם חלק מהטרנזקציה שקוראת מול ה-Database, זאת אומרת - אנחנו מתייחסים ל-Lambda כאל Volatile לחלוטין - שאם היא תיפול, לא יקרה כלום מבחינת “לא יזוז כסף לשום מקום”.וה-State עצמו באמת נשאר ב-Database.(רן) איך נראים ה-API-ים מול אותם בנקים? אני זוכר מהפעם האחרונה שעשיתי איזשהו Payment, זה היה איזשהו CORBA זוועתי עם Perl וכאלה דברים . . . .מה המצב היום?(אור) אז באמת, תשלומים מול בנקים זה סיפור שלם לגמרי, שאפשר לספר עליו . . . אני אתן לרגע את הראשי תיבות ACH - זה Automated Clearing House, שזה בעצם אוטומציה למשהו ענתיקה שנקרא Clearing House . . .(רן) . . . ושום דבר שם לא אוטומטי . . . .(אור) . . . והאוטומציה . . . אני אתן שתי אנקדוטות, אבל בגדול זה קובץ עם המון Records בפניםאתה שם אותו באיזשהו Server בצד השני של העולם - וזה “חור שחור” . . . .אין שום דבר - לא מודל של Request Response . . . יש Response מסויים, אבל זה לא בדיוק אומר לך “אה, כן - אנחנו בדיוק העברנו את הכל!” - אתה יודע רק אחרי כמה ימים אילו מה-Records נכשלו.מה שלא נכשל - הצליח . . . זה בערך המודל לפרוטוקול של הדבר הזה.(רן) כנראה . . . (אור) “כנראה” . . . בדיוק.עכשיו, תוך כדי שאנחנו עובדים אתה אומר לעצמך אוקיי, זה מודל ש . . . יש שם איזשהו מחשב שעובר על הרשומות אחת-אחת, ה-Processing מעביר אותן הלאה ומחזיר אלינו מה שנכשל ומה שלא נכשל.ואז גילינו שאחת הטרנזקציות שחזרה ונכשלה - אנחנו ראינו איזשהו Meta-data בפנים שאנחנו שומרים כדי למפות את זה אחר כך לטרנזקציות פנימיות שלנו וכו' - ואצלנו זה התחיל נגיד עם “t” קטנה ומספר מאוד ארוךוחזרה אלינו טרזקציה שאנחנו לא מזהים - זיהינו אותה, כי שהסתכלנו בעין זה היה “T” גדולה ומספר מאוד ארוך . . . .ואז הבנו שאיפשהו ב-Chain של הבנקים, יש פשוט איזשהו בנאדם שפשוט הקליד “T” . . . . המחשב לא טועה בין “t” ל-”T”, זה שני דברים שונים לגמרי, אבל בנאדם שמקליד T באיזשהו אקסל או email או משהו - כנראה התחלף לו פעם אחת ל-T גדולה ומשם זה נשאר גדול וחזר אלינו בחזרה עם אות גדולה . . .(רן) זה היום שנשפך לו הקפה על ה-Shift . . . .(אור) משהו כזה, בדיוק . . . אז המערכת הזאת היא כאילו סמי-אוטומטית, כי הדברים הם Triggered בצורה אוטומטית - אבל יש שם הרבה מאוד עבודה אנושיתוגם השגיאות שחוזרות הרבה פעמים זו עבודה אנושית, כל מיני מיפויים שמסתכלים על ה-Owner של החשבון בנק - הרבה פעמים זה שם . . . .הם אשכרה ממפים את זה לשם, והרבה פעמים הם לא מוצאים את המספר . . . יש ממש הרבה מאוד תהליכים, ואני רוצה להגיד אולי - אם המערכת היא סוג של . . . ה-Input-ים שהיא מקבלת מהמכונה - אנחנו מתייחסים אליהם גם כאל Input-ים אנושיים, כדי לוודא שבאמת לא נפלנו גם במקרה הזה.(אורי) אבל רגע - קודם, אילן דיבר על זה שאתם מוציאים המון שיקים. זה כאילו . . . אשכרה יש מדפסות שמדפיסות נייר? Serverless מפה ועד להודעה חדשה, אבל מדפסות . . .(אילן) חבל על הזמן . . . (רן) בטח יש שירות של אמאזון שמדפיס שיקים, לא? . . . .(אילן) א - נכון, יש שירות של Amazon שמדפיס שיקים [?], אבל אנחנו משתמשים בשירותים של הבנקים שמדפיסים שיקיםאור דיבר בעצם על קובץ ACH, שזה קובץ מקודד, ששולחים אותו כדי לבצע העברות בנקאיותיש קובץ עם פורמט אחר, קצת יותר מתקדם, ב-JSON, שמעלים לבנקים והם מוציאים שיקים.בעצם, אנחנו נותנים פקודה לבנק - אתה צריך להעביר את זה עד שעה מסויימת, את הקובץ עצמוכשאתה אומר להם “הנה הפרטים” - ומהצד השני יש מדפסות, ומוציאים בעצם שיקים . . . עכשיו, הם עוברים, נכנסים למעטפות, עוברים ל-USPS - ומגיעים ליעד שלהם . . .(אורי) אני חייב להגיד שכאילו . . . נגיד ב-Outbrain, כשאנחנו עושים תשלומים לספקים - וזה הרבה מאוד ספקים, פעמיים בחודש - אנחנו עובדים עם איזושהי מערכת נוראית שנקראית מס”ב, מכירים? (אילן) [מהנהן ביאוש כנראה](רן) ישראלית?(אורי) כן, “מרכז סליקה בנקאי” או משהו כזה . . . (אור) זה די מזכיר את המבנה של ה-ACH, באיזשהו מקום - מאנקדוטות ששמעתי . . . (אורי) אני, אישית, מעדיף לחזור לשיקים, אחרי החווייה עם המס”ב הזה . . . כאילו, מעלים שם איזשהו קובץ Excel, זה אותו דבר כנראה . . . נורא.(רן) יותר בטוח מלשלם ביטוח לעובדים, שגם זה בדרך כלל לא מגיע, אבל לא משנה . . . .(אורי) נכון . . . .אבל יש שם גם . . . לפעמים מתחלפות להם . . . השמות מתחלפים בצדדים כי הכל בעברית, ואתה צריך לקרוא ביוונית, וזה . . .(רן) אז היום אתה מומחה ל-Payments, אור? את היום והלילה שלך אתה מבלה בפיענוח של קבצים כאלה?(אור) אז אני, בוא נגיד במרכאות “למזלי”, יש צוות הרבה יותר גדול שמתעסק בזה.אני עשיתי את זה תקופה יחסית ארוכה, אני . . . זה כמו במטריקס, שהוא רואה את הקוד ויודע מה מופיע מאחורי זה בלי להסתכל על התמונות? אז זה אותו דבר - אני מסתכל על הקובץ ACH ואני יודע - זה המספר של הזה, המבנה הזה זה שם . . .(רן) זה “T” גדולה אז היום Rachel עבדה, זה “t” אז . . .(אור) כן . . . גם בשיקים, אגב, זה מאוד . . . שוב, שיקים זה תהליך אנושי - זה נשלח בדואר אז זה הולך לאיבודיש גם דברים . . . לדוגמא, כשהתחלנו שלחנו מעטפות בצבע הלא נכון . . . שלחנו מעטפות סגולות של שיקים, של Melio, סגול . . . וגילינו שיש אנשים שפשוט מניחים את השיק בצד ולא עושים איתו כלום, כי הם חושבים שזה פרסומות . . . אז שינינו את זה ללבן - ופתאום אנשים כן הפקידו את השיקים . . . .יש כל מיני דברים . . . זה באמת, הערבוב הזה של תהליך אנושי ותהליך שאנחנו מייצרים דברים אוטומטית, שמים ב-API באיזשהו מקום איזו JSON או לא JSON - אנשים בצד השני בסוף צריכים לעשות פעולה, וזה הופך את הכל להרבה יותר מורכב.(אורי) יש לי משהו שמעניין אותי - דיברת על זה שבאים ועושים תשלום, ומקבלים מצד אחד ומשלמים מצד שני - ואתה רגיל שהטרנזקציה נסגרת, נכון?עכשיו, “נסגרת” זה אומר “הכסף עבר”, אני יודע, אבל זה לא בדיוק ככה - אתה . . . הכסף לא עבר, אתה רק העברת את הקובץ ל-Processing של מישהו אחר או שהשיק בדואר, זה . . . ואין היזון-חוזר.(אילן) אין היזון חוזר, זה נכון, וגם במקרים מסויימים, כמו שאור אמר - ב-Bank Transfer, ב-ACH, הפרוטוקול עובד בזה שהוא אומר “כל עוד לא חזרתי אליך אז הכל בסדר”, ואם חזרתי אליך עם שגיאה אז הנה הדברים שנכשלו”אבל כן בנינו מערכת - בנינו מערכת, בסופו של דבר אנחנו מעבירים היום בקצבים של עשרות מיליארדים של דולרים בשנה, יש לנו עשרות אלפי לקוחות ואנחנו חייבים שהכל יהיה מאוזן.ה-SLA הוא מאוד מאוד חשוב - בסופו של יום, אנחנו חייבים . . . לא יכולים להפסיד שדולר לא יגיע לצד אחד או תשלום או שניים יפלו, כי בסופו של דבר מדובר על עסקים שהכסף שלהם לא הגיע לספקים, וזה המון המון Relations שבין העסק לספק.ולכן בנינו מערכות שיושבות בעצם מחוץ ל-Payment Processing, שבעצם בודקות שהספרים מאוזניםנכנסות לבנקים, לוקחות קבצים שאנחנו . . . שמחוץ ל-Transactions, שהם קבצים שמגיעים אלינו - כדי לאזן את הספריםכדי לראות בעצם שכל מה שאנחנו ייצאנו - אנחנו אחרי זה מתשאלים את הבנק, אז אנחנו מבינים . . .כל מה שאנחנו שלחנו לבנק כהוראה, כשאנחנו אחרי זה מתשאלים את הבנק, אנחנו מוודאים שהבנק באמת הוציא את זה.כל המערכות של ה-FinOps שאנחנו . . . .(רן) אני מניח אגב, שזה ערך מוסף משמעותי, מעבר ליכולת הטכנית להעביר תשלום - לוודא שזה מאוזן, לוודא שהדברים עברו, אני מנחש שזה ערך מוסף . . . אני יכול להגיד, שוב - אם נחזור לאנקדוטה של החברת ביטוח - אני זוכר פגישה עם סוכן ביטוח שהבטיח לי ש”פה יש מחשב שבודק!”, אז שאלתי אותו “מה, לפעמים אין מחשב שבודק?”, אז הוא אמר לי “לא . . . בחברות זה אנשים, אצלי זה מחשב!”. אז ברוך הבא למאה העשרים . . . (אורי) אבל אתה אומר “אני מבצע את הטרנזקציות, ואחרי זה יש לי Sweeper כזה שעובר ובודק שבאמת כל הטרנזקציות - "באמת הבנק שילם את זה”, זה מה שתכל'ס סוגר את הטרנזקציה.(אילן) זה יוצא אצלנו כדוח במערכתאנחנו עושים בדיקות אצלנו, עוד במהלך העלאת הקבצים - גם שם יכולות להיות נפילות שלנויש הרבה Lambda-ות שרצות, יש הרבה קבצים, אנחנו עושים תהליך של MapReduce, שעוברים בעצם שורה-שורה בקבצים ופותחים אותם ב-Lambda-ות שיש לנובסופו של דבר אנחנו צריכים להבין שכל מה שקראנו מה-Database עולה לתוך הבנקים - עוד לפני בכלל שיכולים לסגור את הטרנזקציה.אז גם שם פיתחנו יכולת שבאנו ואמרנו שאנחנו לא מחכים - בגלל שאין שגיאות ואין היזון חוזר . . .(אורי) זה לא שאין שגיאות - אין הודעות שגיאה . . . (אילן) אין הודעות שגיאה, בדיוק - אז אנחנו, בתהליך העלאת הקבצים, אנחנו כל הזמן בודקים מה העלינו לעומת מה שהיה כתוב ב-Database - כי התהליך בעצם חיצוני ונפרד - כדי להקפיד שהדברים מאוזנים.רק בשלב שלאחר מכן, יש תהליך שבעצם מתשאל את הבנקים ובודק מה בעצם אנחנו העלינו, ואז רואה שהכל מאוזן.(אורי) עד כדי “t” קטנה ו-”T” גדולה . . . .(אילן) . . .שרק אור תופס, כן . . . (אור) יש פה באמת . . .אפשר להגיד שאנחנו עושים Reconciliation בכמה רמות שונות, מכמה Check-Points שונים בתוך התהליךגם מיד אחרי שאנחנו מעבירים את הכסף, גם כמה ימים אחר כך, גם כשמהבנק מודיעים לנו, בדיעבד, מה הצליח ומה לא הצליח, גם אחר כך במאזן של של הבנק, הסופי, שאנחנו רואים . . . .אנחנו מנסים באמת לקבל את התמונה השלמה, כי שוב, כמו שאילן אמר - אנחנו לא יכולים להרשות שבגללנו ה-Customer שלנו לא ישלם חשבון אחר, כי אז הוא, שוב, בבעיה מול הספק שלויש לו עכשיו Cash-flow problems . . . בשבילו זה 100% - תשלום אחד בשבילו . . .אצלנו תשלום אחד זה פרומיל-של-הפרומיל - אצלו זה 100% מהדברים שהוא מתעסק בהם.(אורי) זה גם פוגע לו לפעמים בדירוגי אשראי או כאילו . . . Credit Score.(אור) יכול . . .(אילן) זה יחס עם הספק . . . זה יחס עם הספק, שהוא אומר לו “The Cheque is on its way” - והוא לא באמת on its way, ואז היחסים ביניהם עשויים להיפגע.(רן) איך עוד נראה הסיפור הטכנולוגי? זאת אומרת - האם עצם זה שאתם עוסקים ב-Domain הזה, של פיננסים, יש לזה השלכות טכנולוגיות, לצורך העניין - באילו שפות אתם כותבים? איזה Security זה אומר מבחינתכם? השלכות אחרות, טכנולוגיות שקיימות?(אור) מבחינת שפות, אנחנו די “סטנדרטיים”, נקרא לזה ככה, לפחות בתעשייה היום.אנחנו עובדים ב-JavaScript, גם קצת Python בכל מיני מקומות בתוך המערכת - אבל בגדול רוב המערכת כתובה ב-Node.זה מאפשר לנו, פשוט בגלל ש-Lambda ו-Node זה מאוד . . . נקרא לזה “Native” ב- Runtime.לא ניסינו יותר מדי להתחכם שם - אנחנו בודקים את עצמנו כמה שיותר.מבחינת Security, גם - Lambda משחק יחסית לידיים שלנו במקרה הזה: אין Server . . . אין Port לפרוץ אליו אפילוזה לא קיים, כקונספט . . . גם ל-Compliance, אגב - גם מאוד עזר לנו, כל מה שקשור ל-Serverless.כשעברנו Compliance - עברנו כבר שני תהליכים - ופשוט, יש חברות . . .(אורי) מי הגוף שמבקש מכם את ה-Compliance? זה Compliance עם מי?(אור) Compliance ISO 27001 . . . (אורי) שהוא יותר פיננסי או . . .(אור) זה של אירופה יותר, אם אני לא טועה . . . (אילן) האירופאי זה בעיקר Security, ועכשיו אנחנו בעצם בתהליך, מסיימים אותו, של SOC 2 Type 2מי שדורש מאיתנו את הרגולציות האלה זה (א) השותפים שלנו, זה הבנקים שאיתם אנחנו עובדים - זה ה-Rails שאיתם אנחנו מעבירים את הכסף ושותפים - Melio בסוף . . . עוד לא נגענו בזה, אבל נחזור רגע לחלק הטכנולוגי - ל-Melio יש שני קווי מוצר עיקריים - הקו הראשון זה Stand-alone Experienceהקו השני בעצם זה ה-Platform - “היכולת לאמבד (To Embed) את ה-Experience בנדל”ן של מישהו אחר”השותף הכי גדול שלנו היום זה Intuit, ב-QuickBooks - בעצם שמו את היכולות שלנו בתוך QuickBooksושותף שמקבל שירות פיננסי רוצה לדעת שאנחנו Well-Secured.(רן) אז אמרת . . . למה אתה מוריד את ה-Attack-Surface? . . .(אור) דילגנו על זה . . . גם בתוך ה-Compliance יש סעיפים שלמים של Port management וכל מיני דברים כאלו, ברמת המכונות וה-Server-ים, שזה פשוט לדלג עליהם . . . לקצר מאוד את הזמן של ה-Compliance, באופן מפתיע . . . זה מפתיע את הצד השני, שעושה לנו את ה-Review - כמה חתכנו.זה היה מאוד נוח בהקשר הזה.(רן) למרות שאתה יודע - אני מניח שה-Compliance הזה יזוז עם הזמן ויתרגל, ויגלה שגם לצורך העניין, ב-Serverless צריך פשוט לבדוק דברים אחרים . . . אין יותר Port-ים פתוחים, אוקיי . . . אין יותר File Descriptors, אבל כן יש דברים אחרים . . .(אור) יש Dependencies, יש Static code analysis . . . עדיין יש הרבה API-ים שחשופים החוצה לעולם, מן הסתם . . .(רן) אני מבין שה-Compliance עוד לא הגיע לשם . . . .(אור) אנחנו מנסים כמה שיותר לדאוג בעצמנו, כי שוב - ה-Compliance חשוב לנו בגלל שזה חשוב לפרטנרים שלנו, זה חשוב לנראותחשוב לנו שלא יקרה לנו שום דבר, לשמור בעצם על כל הלקוחות שלנו, אז יש כאן את האספקט של האם אנחנו מרגישים מספיק אחריות בשביל לעשות את זה.כן . . . .אז בהקשר הזה, השימוש ב-Lambda ובאופן כללי ב-Serverless - אני רוצה רגע להגיד מילה על Serverless - אני תמיד שומע “Serverless, Serverless” . . . כשהתחלנו להתעסק עם זה, אני פחות התעניינתי בזה שזה Serverless, אפילו קראתי לזה הרבה פעמים Management-less . . . .יש Server, הוא קיים - יש Lambda, זה Server, יש Instance, יש לנו Connection ל-Database שאנחנו עושים לו Re-use, יש RAM ואנחנו מחזיקים שם כל מיני דברים, יש CPU . . . . יש הכל.זה מבחינתינו כאילו מתנהג קצת כמו Server שמריץ קוד ב-Check-point-ים - רק שאנחנו כאילו לא מנהלים אותו.אז במדרג, אנחנו כן מסתכלים על זה כי Lambda זה ה-Holy Grail מהבחינה הזו של Management-lessמתחת לזה יש לנו FarGate, יש לנו זה . . . אז אנחנו לא Pure-Serverless - אנחנו משתמשים במה שמתאים לנו באותה נקודת זמן.(רן) איך זה משפיע על חווית הפיתוח? זאת אומרת - אם אני עכשיו בא ומתקן איזשהו Bug ב-Service, שהוא כנראה חלק מ-70 רכיבים אחרים - איך אני מפתח אותו? איך אני בודק אותו?(אור) יפה, אז זה אחד הדברים הראשונים שגם אני חשבתי עליהם כשאמרנו “בואו נעשה Serverless” . . .אז יש לנו כרגע שתי גישות - אחת שהיא קצת יותר Legacy בתוך החברה ואחת שהיא יותר חדשה, שאנחנו ככה מתחילים לעשות לה סוג של Imploy מבפניםהגישה הראשונה, שהיא עדיין עובדת בחלק גדול מה-Service-ים - מה שעשינו איתה בעצם . . . ה-Service-ים עצמם, היה להם מבנה פנימי מאוד ספציפי, הם היו נראים כמו איזשהו Web Application, והייתה איזושהי מעטפת קטנה שסידרה בעצם את כל התשתית מסביב, שהיא כאילו תיקרא ל-Routing בתוך ה-Web Applicationאם זה Event מ-SQS אז הוא מול איזשהו Route עם Fake payload, שזה בעצם ה-Payload מ-SQS, ועוד כל מיני דברים בסגנון הזהאם יש S3 אז הוא מול איזשהו Payload מ-S3ואז זה מאפשר לנו בעצם להריץ את הדבר הזה בתוך Lambda כרגיל, עם Event-ים ו-Listeners והכל . . . (רן) וב-Commit אתה מייצר איזשהו Container שעוטף את זה . . .(אור) אפילו לא Container - הלכנו ממש npm-start . . . פשוט, מה שהיה . . . היו פשוט, בכל פרויקט, היו שני סקריפטיםאחד שמתאים ל-Lambda והשני שהוא Server עם איזושהי מעטפת.כשמפתחים עבדו לוקאלית, אז בעצם הם . . . ה-Service שלהם דיבר ישירות עם ה-Cloud, לא עבדנו עם RabbitMQ לוקאלית ו-SQS ב-Cloud, עם DynamoDB ב-Cloud ועם Redis לוקאליתפשוט הכל - לכל Developer יש תשתית שלמה - “שלד” כזה של התשתית - בלי ה-Computeהוא פשוט בוחר איזשהו Service שהוא רוצה, npm-start - וזה מתחיל “לנגן” מול התשתית, מול ה-SQS הרלוונטי, מול ה-DynamoDB הרלוונטיה-RDS, במקרה הזה MySQL, עדיין לוקאליתזאת הייתה הגישה הראשונה - זה עבד יחסית טוב, רצנו עם זה יחסית הרבה זמן.עכשיו אנחנו נהיינו קצת יותר Powerhouse של -Lambda, ואנחנו עובדים לגישה שהיא קצת שונה - אנחנו עובדים עם SAM היום - SAM זה המתחרה-Serverlss, זה “ה-Serverless.com של AWS“. . . הרעיון זה שהוא מייצר לנו CloudFormation templates, אנחנו עושים לזה Deployments כחבילה שלמה, כ-Stack שלםואז, ברגע שיש לך Stack כזה, של . . . בגדול, לכל מפתחת אצלנו נגיד יש חשבון AWS פרטי, זה כרגע . . . עדיין אנחנו בסוג של נקרא-לזה-POC כדי לבדוק שזה . . . שההתיכנות של זה היא ממש בסדר.לכל מפתחת יש חשבון AWS - בפנים יש בעצם את המיני-Production של Melio - איזה שירות שהיא רוצה להריץ שם, את ה-email Service שלה, גם את ה-Payments Processing, הכל . . . ואז, אם היא רוצה לפתח Lambda מסויימת, אז כתבנו איזשהו כלי משלנו, שבעצם משתלט על ה-Lambda הזאת, ומעביר את ה-Compute אליה למחשבואז היא יכולה לעשות Break-points, לוקאלית - זה רץ ממש על המחשב . . .(רן) כמו Telepresence בעולם של Kubernetes . . . .(אור) בדיוק - רק עם פחות משחקיםפחות משחקים עם Port-ים, פחות משחקים עם Networking - רק לקחת את ה-Message, לשלוח אותו למחשב, לעשות את ה-Compute . . .כי ה-Resources של AWS בכל מקרה זמינים - SQS זמין ב-API Call ו-SNS זמין ב-API Call, אז ה-Compute שרץ לוקאלית על המחשב “מדבר עם ה-Cloud כאילו הוא ב-Cloud”אז ה-Telepresence במובן הזה זה רק להעביר את ה-Messaging למקום הנכון ב . . . נקרא לזה “ב-Network הגלובאלי העולמי”, למחשב הספציפי שבו זה נמצא כרגע.(רן) אז מפתח חדש שמצטרף אליכם - אנחנו כבר לקראת סיום, וזו שאלה אחרונה אולי - מפתח חדש שמצטרף אליכם, שמעולם לא חווה Serverless ולא חווה את ה-Concept - עד כמה, להערכתם, קל או קשה לו להכנס ל-Mindset הנכון, של Serverless, של Stateless, וכו'?(אור) אז אני מודה שזה אתגר . . . אנחנו, ככה, מנסים בתקופת ה-Onboarding של המפתחים והמפתחות, אנחנו מנסים להכניס את זה מעיין ל-Mindset של “אנחנו חיים על Lambda”, עם האתגרים - מה שיבוא, אנחנו נתמודד איתו.בגדול, הגענו למצב שיש כבר הרבה מאוד Engineers שכבר עובדים עם זה, אז ברגע שמישהו מצטרף, יש את ה . . . נקרא לזה תמיכה, ה-Ecosystem הפנימי של החברה שיודע לעזור.אני יכול להגיד שהחבר'ה של ה-Payments Processing מדהימים בקטע הזה - ממש אימצו את זה לגמרי והם הולכים עם זה עד הסוף.גם עם ה-Pitfalls ועם ה-Challenges שיש לזה - הם הולכים עם זה ורצים עם זה קדימה ממש יפה.רציתי לגעת דווקא בנקודה, בהקשר של Serverless, אם יש לנו זמן - בהקשר של Pricing . . .יש איזושהי מנטרה כזאת, ש-”Serverless הרבה יותר יקר” [תלוי . . . 412 Serverless at Via], בגלל שזה בעצם שירות Premium כדי להריץ פונקציה אחת בודדתאנחנו, מה שנקרא, מוצאים - בהשאלה מאנגלית [we find it] - אנחנו מוצאים את זה יחסית - אם לא יותר זול אז מקביל לדברים אחרים.יש לזה כמה סיבות - מן הסתם, אחת הסיבות העיקריות זה שאם לא הרצנו אז אנחנו לא משלמים, אבל באיזשהו מקום . . .(אורי) אין דבר כזה “להשאיר Instance באוויר” . . .(אור) בדיוק - אין Instance באוויר . . . כשהוא כן באוויר זה יקר יותר, אבל רוב הזמן אצלנו הוא לא באוויר.בוא נגיד לא “רוב הזמן”, אני מגזים - אבל חלק גדול מהחודש הוא לא באוויר.ויש פיצ'ר מאוד נחמד, בהקשר הזה, שיחסית מאוד קל לנו לעשות לו מה שנקרא Unit economicsכי בעצם כל Processing אצלנו - אנחנו יודעים כמה הוא עולה, אנחנו יכולים לעשות איזשהו חישוב גס ולדעת כמה בעצם לתרגם- ממש לחשבונית AWS - לתרגם כמה עולה הפעילות העסקית, ואפילו לתת תחזיות על סמך זה.וזה יתרון מאוד גדול בשבילנו.(אורי) זה מחזיר אותי לשאלה שמחכה מההתחלה . . . מה המודל העסקי? זאת אומרת - אתם פר-טרנזקציה? אתם . . .(רן) עושים פרסומות! מה בעיה? . . . (אילן) Recommendations, כן . . .במערכת שלנו, בסופו של דבר, יש שני סוגים של Transactions - יש את מה שאנחנו קוראים לו Basic Transactions, ה-Fundamental - להעביר ACH ל-ACH או ACH לשיק - התשלומים האלה הם חינם, בעצם Engagement Flywheel עבור העסק ועבורנו בעצם - שהעסק ישתמש בנו.הסוג השני של התשלומים זה בעצם Premium Payments - אם עכשיו עסק רוצה להשתמש בכרטיס אשראי - אז לא מתאפשר לו כרטיס אשראי, כי רוב הספקים לא מקבלים אשראי בעולמות ה-B2Bאנחנו, בזכות ה-Decupling, מאפשרים לעסק בעצם לשלם בכרטיס אשראי - והצד השני יקבל שיק.וע”י כך, בעצם לעזור לעסק ולדחות תשלום בעוד 30 או 45 יום ל-Billing cycle הבא שלך, של כרטיסי האשראיהדבר הזה יעלה למשלם 2.9% . . . (רן) “אשראי ישראלי” - שוטף פלוס . . .(אורי) “השיק בדואר” . . . .(אילן) ותשלומים אחרים שהם Premium Services זה אם אני עכשיו בתור . . . אם אני רוצה . . . ACH, לוקח לו שלושה ימים להגיע בין צד אחד לצד אחר, זו המערכת הבנקאית בארה”ב [גם בארץ…]אם עכשיו רוצים שהתשלום יגיע באותו יום, או Instant - אז בעצם זו עלות שאחד הצדדים יכול לספוג בינתיים - מי שרוצה להאיץ את התשלום או לקבל יותר מהר את התשלום.(רן) דרך אגב, גם במערכת הבנקאית - אני מניח שאתה מכיר את זה - יש גם אפשרות לזרז את התשלום תמורת “תשלום סמלי” . . .(אילן) בדיוק - International payouts -אנחנו היום נכנסים לתשלומים בינלאומיים - ותשלומים כאלה עולים כסף, Domestic wire.אז אנחנו נותנים את התשלומים, את ה-Fundamental payments, בחינם - אבל התשלומים היותר Premium הם בעצם עולים, לאחד הצדדים, תלוי למי אתה מוכר אותו.משם מגיעים ה-Unit Economics שלנו.(אורי) אבל יש, נקרא לזה “הלימה”, בין כמות הטרנזקציות שאתם תבצעו - תכל'ס תשמשו ב-Lambda-ות, נכון? - לבין כמה כסף שתרוויחו, זאת אומרת - זה יחס ישר, מסויים, אבל . . .(אילן) זה לגמרי ככה . . . בסופו של יום, כשאנחנו בעצם מודדים, אנחנו מסתכלים בעצם על סך כל ה-Volume ש-Melio הוציאה באותו יום או באותו חודש - וכמה מה-Volume הזה הוא בעצם volume ש-Melio קיבלה עליו Revenueויש לנו איזשהו יעד שאנחנו באים ואומרים - “רגע, מה היחס?”אם מסתכלים, נגיד, על Check out באונליין, בוא נניח על Check out ב-Stripe - בסופו של יום, כש-Stripe מסתכלת על 100% מהטרנזקציות, הם מרוויחים רווח כזה או אחר, 2.5% או Whatever.אז ב-Melio זה עובד קצת אחרת, בגלל שיש Blend - יש Blend של תשלומים שהם בחינם ותשלומים שהם עולים, ש-Melio בעצם מקבלת עליהם Revenue.כשמסתכלים על הכל, אז יש לנו איזשהו יעד של כמה “Bips-ים” בעצם מסך כל ה-TPV הוא בעצם רווח או Revenue ל-Melio(רן) תרגם שנייה . . . Bips זה?(אילן) זה בעצם האחוזים שבעצם עליהם אנחנו . . .(אורי) זה רווח . . .(אילן) זה הרווח . . . זה ה-Revenue[בערך . . Basis points (BPS) refers to a common unit of measure for interest rates and other percentages in finance. One basis point is equal to 1/100th of 1%]וזה בעצם יעד שאנחנו מסתכלים עליו כל הזמןויש הלימה, בדיוק כמו שאמרת, אורי - בעצם, זה שאנחנו רואים שעסק משתמש בנו יותר, או מבצע יותר תשלומים, אז כמות ה-Premium Payments היחסית שקוראת שם בעצם עולה.ולכן אנחנו באים, וזה עדיין כלכלי עבורנו לבוא ולהגיע למצב שאנחנו רוצים שה-Engegement יעלה - כי אנחנו יודעים שאפשר אחרי זה To derive more revenue.(רן) הנושא הזה, של Unit Economy, אני לגמרי מזדהה איתו - אני נמצא גם במקום שמאוד קשה להבין כמה דברים עולים ואני יודע שזה משמעותי - אבל אני תוהה עד כמה זה בכלל זה משמעותי, עלות מרכיב הענן אצלכם היום - זה בכלל משהו משמעותי? אתם בכלל שמים לב אליו בשלב הזה של הגדילה?(אורי) . . . כאחוז מה-Revenue, ה-Cost of Sales . . .(אילן) בוא נגיד ככה, אם אני יכול ככה “לשתף ולא לשתף”, מה שנקרא . . .(רן) אם המשקיעים לא מקשיבים . . . [אבל אולי קוראים?](אילן) יש לנו עלויות עסקיות, שהן לא עלויות של הענן, בעצם - העלויות מול הבנקים, מול השותפים “הטבעיים”, נקרא לזהוכשמסתכלים על התמונה הכוללת, כשכוללים בפנים את העלות של הענן - אז זה לא כל כך מפחיד.(רן) זה בסדר, ואני חושב שהרבה חברות נמצאות במקום כזה, בעיקר בשלב של גדילה, שבו יש עלויות הרבה הרבה יותר משמעותיות - והן במכוון “שופכות כסף” על הענן, נקרא לזה.הבעיה שהן אחר כך מגיעות לנקודה שממנה מאוד קשה לחזור, של “אוקיי, עכשיו אני רוצה לצמצם את עלויות הענן - אבל עכשיו זה כבר ממש ממש קשה”.[השלמות למיטבי שמע - 421 The Cost of Cloud, a Trillion Dollar Paradox with Martin Casado ו - 418 Carboretor 31 Cost of cloud paradox](אילן) אז אני אהיה איתך כנה - זה שיקול מאוד . . . זה שיקול שעובר לנו גם.בסופו של יום, כשאמרנו שאנחנו רוצים להיות Management-less, אנחנו מעדיפים להתרכז ב-Core Businessכי Melio זו חברה שגדלה - גדלה וגדלה מאוד מהר - בשנה האחרונה הגדלנו את נפח הפעילות ב-5000% אחוז . . . ה-Covid, הקורונה, נתנה Boost מאוד גדול לעסקים להיפטר ממשהו פיזי או לפגוש אחד את השני כדי לבצע תשלומים ולעבור לתשלומים Online.דרך אגב - ה-Serverless או ה-Lambda-ות עזרו לנו To scale out בצורה מאוד טובה - מראש בנינו את המערכת שנוכל To Scale out בצורה טובה, וזה עזר לנו בגדילה הבאמת מאוד מהירה שקרתה לנו.אבל לנקודה שלך - כן, אנחנו הרבה יותר מפוקסים ביכולת שלנו להגדיל את ה-Business מאשר ללכת ולהבין איך אנחנו נחסוך בעלויות עיבוד.(רן) אבל עושים הכנה למזגן? זאת אומרת - מתישהו תתקינו את המזגן הזה . . . .(אילן) לגמרי . . . בחברות Payments זה נהוג להבין בעצם “כמה עולה תשלום”כשאני מסתכל שנייה רגע על . . . Melio בעצם ביצעה מיליונים של טרנזקציות - מה העלות הכוללת שלי, מתהליך העיבוד, עלויות שותפים - Per-Transactionהיכולת לחשב את זה היא יכולת מאוד חשובה כדי להביא את ה-Business to Scale(רן) אז הזכרנו שאתם גדלים - לא אמרתם איפה אתם גרים . . . איפה המשרד?(אילן) המשרד שלנו נמצא בתל אביב, ברחוב הארבעה, מגדלי הארבעהמאוד נגיש מבחינת “קרוב לרכבת” - מאוד נגיש למי שנמצא מחוץ לתל אביב, מאוד נגיש למי שבתוך תל אביבמשרדים יפים, חדשים, שתי קומות - וגדלים . . . (רן) מה אתם מחפשים היום?(אילן) היום ה-Engineering ב-Melio הוא כ-80 אנשים, שנמצאים בארבע קבוצות - אנחנו רוצים להכפיל את גודל הקבוצה, את קבוצת ה-Engineering בשנה הקרובה . . .מחפשים קצת “הכל מהכל” - מחפשים Full-Stack Engineers, יותר לצוותים שהם Product-facing, שמתעסקים בחווייה - לא דיברנו על זה הרבה היום, אבל יש חווייה - אחד הדברים, ואור הזכיר את זה קצת, דיברנו בעיקר על ה-Payments Processing, אבל בסופו של יום אנחנו מוכרים חווייה - חווייה שתיהיה מאוד מאוד נוחה ופשוטה לבעל עסק קטן כדי לנהל את התשלומים שלואז יש צוותים שהם Product-facing שהם בעיקר Full-Stack Engineers.מחפשים Data Science - כי -Melio עושה את כל ה-Risk של ה-Payments, כי Risk “לא קיים” בכל עולמות ה-B2B כמשהו שהוא off the shelf, אז היינו צריכים לפתח את כל המודלים בעצמנואז גם Big Data Engineers וגם Data Science לקבוצות של ה-Risk וה-Data.ו-Backend engineers ל-Payment Processing, שדיברנו עליו עכשיו . . . (רן) יופי - אז שיהיה בהצלחה, תודה רבה על הביקור, השיק בדואר, להתראות! האזנה נעימה ותודה רבה לעופר פורר על התמלול!

Screaming in the Cloud
Works Well with Others with Abby Kearns

Screaming in the Cloud

Play Episode Listen Later Oct 19, 2021 39:53


About AbbyWith over twenty years in the tech world, Abby Kearns is a true veteran of the technology industry. Her lengthy career has spanned product marketing, product management and consulting across Fortune 500 companies and startups alike. At Puppet, she leads the vision and direction of the current and future enterprise product portfolio. Prior to joining Puppet, Abby was the CEO of the Cloud Foundry Foundation where she focused on driving the vision for the Foundation as well as  growing the open source project and ecosystem. Her background also includes product management at companies such as Pivotal and Verizon, as well as infrastructure operations spanning companies such as Totality, EDS, and Sabre.Links: Cloud Foundry Foundation: https://www.cloudfoundry.org Puppet: https://puppet.com Twitter: https://twitter.com/ab415 TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Liquibase. If you're anything like me, you've screwed up the database part of a deployment so severely that you've been banned from touching every anything that remotely sounds like SQL, at at least three different companies. We've mostly got code deployments solved for, but when it comes to databases we basically rely on desperate hope, with a roll back plan of keeping our resumes up to date. It doesn't have to be that way. Meet Liquibase. It is both an open source project and a commercial offering. Liquibase lets you track, modify, and automate database schema changes across almost any database, with guardrails to ensure you'll still have a company left after you deploy the change. No matter where your database lives, Liquibase can help you solve your database deployment issues. Check them out today at liquibase.com. Offer does not apply to Route 53.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate: is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards, while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other, which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at Honeycomb.io/screaminginthecloud. Observability, it's more than just hipster monitoring.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Once upon a time, I was deep into the weeds of configuration management, which explains a lot, such as why it seems I don't know happiness in any meaningful sense. Then I wound up progressing into other areas of exploration, like the cloud, and now we know for a fact why happiness isn't a thing for me. My guest today is the former CEO of the Cloud Foundry Foundation and today is the CTO over at a company called Puppet, which we've talked about here from time to time. Abby Kearns, thank you for joining me. I appreciate your taking the time out of your day to suffer my slings and arrows.Abby: Thank you for having me. I have been looking forward to this for weeks.Corey: My stars, it seems like things are slow over there, and I kind of envy you for that. So, help me understand something; you went from this world of cloud-native everything, which is the joy of working with Cloud Foundry, to now working with configuration management. How is that not effectively Benjamin Button-ing your career. It feels like the opposite direction that most quote-unquote, “Digital transformations” like to play with. But I have a sneaking suspicion, there's more to it than I might guess from just looking at the label on the tin.Abby: Beyond I just love enterprise infrastructure? I mean, come on, who doesn't?Corey: Oh, yeah. Everyone loves to talk about digital transformation, reading about books like a Head in the Cloud to my children used to be a fun nightly activity before it was formally classified as child abuse. So yeah, I hear you, but it turns out the rest of the world doesn't necessarily agree with us.Abby: I do not understand it. I have been in enterprise infrastructure my entire career, which has been a really, really long time, back when Unix and Sun machines were still a thing. And I'll be a little biased here; I think that enterprise infrastructure is actually the most fascinating part of technology right now. And why is that? Well, we're in the process of actively rewritten everything that got us here.And we talk about infrastructure and everyone's like, “Yeah, sure, whatever,” but at the end of the day, it's the foundation that everything that you think is cool about technology is built on. And for those of us that really enjoy this space, having a front-row seat at that evolution and the innovation that's happening is really, really exciting and it creates a lot of interesting conversation, debate, evolution of technologies, and innovation. And are they all going to be on the money five, ten years from now? Maybe not, but they're creating an interesting space and discussion and just the work ahead for all of us across the board. And I'm kind of bucketing this pretty broadly, intentionally so because I think at the end of the day, all of us play a role in a bigger piece of pie, and it's so interesting to see how these things start to fit together.Corey: One of the things that I've noticed is that the things that get attention on the keynote stage of, “This is this far future, serverless, machine-learning Kubernetes, dingus nonsense,” great is—Abby: You forgot blockchain. [laugh].Corey: Oh, yeah. Oh, yeah blockchain as well. Like, what other things can we wind up putting into the buzzword thing to wind up guaranteeing that your seed round is at least $200 million? Great. There's that.But when you look at the actual AWS bill—my specialty, of course—and seeing where the money is actually going, it doesn't really look that different, as far as percentages go—even though the numbers are higher—than it did ten years ago, at least in the enterprise world. You're still buying a bunch of EC2 instances, you're still potentially modernizing to some of the managed services like RDS—which is Amazon's reimagining of what a database could be if you still had to manage the finicky bits, but had no control over when and how they worked—and of course, data transfer and disk. These are the basic building blocks of everything in cloud. And despite how much we talk about the super neat stuff, what we're doing is not reflected on the conference stage. So, I tend to view the idea of aspirational architecture as its own little world.There are still seasoned companies out there that are migrating from where they are today into this idea of, well, virtualization, we've just finally got our heads around that. Now, let's talk about this cloud thing; seems like a fad—in 2021. And people take longer to get to where they think they're going or where they intend to go than they plan for, and they get stuck somewhere and instead of a cloud migration, they're now hybrid because they can redefine things and declare victory when they plant that flag, and here we are. I'm not here to make fun of these companies because they're doing important work and these are super hard problems. But increasingly, it seems that the technology is not the thing that's holding them back or even responsible for their outcome so much as it is people.The more I work with tech, the more I realized that everything that's hard becomes people issues. Curious to get your take on that, given your somewhat privileged perspective as having a foot standing very deeply in each world.Abby: Yeah, and that's a super great point. And I also realized I didn't fully answer the first question either. So, I'll tie those two things together.Corey: That's okay, we're going to keep circling around until you get there. It's fine.Abby: It's been a long week, and it's only Wednesday.Corey: All day long, as it turns out.Abby: I have a whole soapbox that I drag around behind me about people and process, and how that's your biggest problem, not technology, and if you don't solve for the people in the process, I don't care what technology you choose to use, isn't going to fix your problem. On the other hand, if you get your people and process right, you can borderline use crayons and paper and get [laugh] really close to what you need to solve for.Corey: I have it on good authority that's known as IBM Cloud. Please continue.Abby: [laugh]. And so I think people and process are at the heart of everything. They're our biggest accelerators with technology and they're our biggest limitation. And you can cloud-native serverless your way into it, but if you do not actually do continuous delivery, if you did not actually automate your responses, if you do not actually set up the cross-functional teams—or sometimes fondly referred to as two-pizza teams—if you don't have those things set up, there isn't any technology that's going to make you deliver software better, faster, cheaper. And so I think I care a lot about the focus on that because I do think it is so important, but it's also—the reason a lot of people don't like to talk about it and deal with it because it's also the hardest.People, culture change, digital transformation, whatever you want to call it, is hard work. There's a reason so many books are written around DevOps. And you mentioned Gene Kim earlier, there's a reason he wrote The Phoenix Project; it's the people-process part is the hardest. And I do think technology should be an enabler and an accelerator, but it really has to pair up nicely with the people part. And you asked your earlier question about my move to Puppet.One of the things that I've learned a lot in running the Cloud Foundry Foundation, running an open-source software foundation, is you could a real good crash course in how teams can collaborate effectively, how teams work together, how decisions get made, the need for that process and that practice. And there was a lot of great context because I had access to so much interesting information. I got to see what all of these large enterprises were doing across the board. And I got to have a literal seat at the table for how a lot of the decisions are getting made around not only the open-source technologies that are going into building the future of our enterprise infrastructure but how a lot of these companies are using and leveraging those technologies. And having that visibility was amazing and transformational for myself.It gave me so much richness and context, which is why I have firmly believed that the people and process part were so crucial for many years. And I decided to go to a company that sold products. [laugh]. You're like, “What? What is she talking about now? Where is this going?”And I say that because running an open-source software foundation is great and it gives you so much information and so much context, but you have no access to customers and no access to products. You have no influence over that. And so when I thought about what I wanted to do next, it's like, I really want to be close to customers, I really want to be close to product, and I really want to be part of something that's solving what I look at over the next five to ten years, our biggest problem area, which is that tweener phase that we're going to be in for many years, which we were just talking about, which is, “I have some stuff on-prem and I have some stuff in a cloud—usually more than one cloud—and I got to figure out how to manage all of that.” And that is a really, really, really hard problem. And so when I looked at what Puppet was trying to do, and the opportunity that existed with a lot of the fantastic work that Puppet has done over the last 12 years around Desired State Configuration management, I'm like, “Okay, there's something here.”Because clearly, that problem doesn't go away because I'm running some stuff in the cloud. So, how do we start to think about this more broadly and expansively across the hybrid estate that is all of these different environments? And who is the most well-positioned to actually drive an innovative product that addresses that? So, that's my long way of addressing both of those things.Corey: No, it's a fair question. Friend of the show, Matt Stratton, is famous for saying that, “You cannot buy DevOps, but I sure would like to sell it to you,” and if you're looking at it from that perspective, Puppet is not far from what that product store look like in some ways. My first encounter with Puppet was back around 2009, 2010 or so, and I was using it in an environment I was working within and thought, “Okay, this is terrible, and it's crap, and obviously, I know what I'm doing far better than this, and the problem is the Puppet's a bad product.” So, I was one of the early developers behind SaltStack, which was a terrific, great way of approaching the problem from a novel perspective, and it wasn't crap; it was awesome. Right up until I saw the first time a customer deployed it and looked at their environment, and it wasn't crap, it was worse because it turns out that you can build a super finely crafted precision instrument that makes a fairly bad hammer, but that's how customers are going to use it anyway.Abby: Well, I mean, [sigh] look, you actually hit something that I think we don't actually talk about, which is how hard all of this shit really is. Automation is hard. Automation for distributed systems at scale is super duper hard. There isn't an easy way to solve that problem. And I feel like I learned a lot working with Cloud Foundry.Cloud Foundry is a Platform as a Service and it sits a layer up, but it had the same challenges in that solving the ability to run cloud-native applications and cloud-native workloads at scale and have that ephemerality to it and that resilience to it, and the things everyone wants but don't recognize how difficult it is, actually, to do that well. And I think the same—you know, that really set me up for the way that I think about the problem, even the layer down which is, running and managing desired state, which at the end of the day is a really fancy way of saying, “Does your environment look like the way you think it should? And if it doesn't, what are you going to do about it?” And it seems like, in this year of—what year are we again? 2021, maybe? I don't know. It feels like the last two years of, sort of, munged together?Corey: Yeah, the passing of time is something it's very hard for me to wrap my head around.Abby: But it feels like, I know some people, particularly those of us that have been in tech a long time are probably like, “Why are we still talking about that? Why is that a thing?” But that is still an incredibly hard problem for most organizations, large and small. So, I tend to spend a lot of time thinking about large enterprises, but in the day, you've got more than 20 servers, you're probably sitting around thinking, “Does my environment actually look the way I think it does? There's a new CVE that just came out. Am I able to address that?”And I think at the end of the day, figuring out how you can solve for that on-prem has been one of the things that Puppet has worked for, and done really, really well the last 12 years. Now, I think the next challenge is okay, how do you extend that out across your now bananas complex estate that is—I got a huge data estate, maybe one or two data centers, I got some stuff in AWS, I got some stuff in GCP, oh yeah, got a little thing over here and Azure, and oh, some guy spun up something on OCI. So, we got a little bit of everything. And oh, my God, the SolarWinds breach happened. Are we impacted? I don't know. What does that mean? [laugh].And I think you start to unravel the little pieces of that and it gets more and more complex. And so I think the problems that I was solving in the early aughts with servers seems trite now because you're like, I can see all of my servers; there's eight of them. Things seem fine. To now, you've got hundreds of thousands of applications and workloads, and some of them are serverless, and they're all over the place. And who has what, and where does it sit?And does it look like the way that I think it needs to so that I can run my business effectively? And I think that's really the power of it, but it's also one of those things that I don't feel like a lot of people like to acknowledge the complexity and the hardness of that because it's not just the technology problem—going back to your other question, how do we work? How do we communicate? What are our processes around dealing with this? And I think there's so much wrapped up in that it becomes almost like, how do you eat an elephant story, right? Yes, one bite at a time, but when you first look at the elephant, you're like, “Holy shit. This is big. What do I need to do?” And that I think is not something we all collectively spend enough time talking about is how hard this stuff is.Corey: One of the biggest challenges I see across the board is this idea of conference-ware style architecture; the greatest lie you ever see is someone talking about their infrastructure in public because peel it back a little bit and everything's messy, everything's disastrous, and everything's a tire fire. And we have this cult in tech—Abby: [laugh].Corey: —it's almost a cult where we have this idea that anything that isn't rewritten completely within the last six months based upon whatever is the hot framework now that is designed to run only in Google Chrome running on the latest generation MacBook Pro on a gigabit internet connection is somehow less than. It's like, “So, what does that piece of crap do?” And the answer is, “Well, a few $100 million a quarter in revenue, so how about you watch your mouth?” Moving those things is delicate; moving those things is fraught, and there are a lot of different stakeholders to the point where one of the lessons I keep learning is, people love to ask me, “What is Amazon's opinion of you?” Turns out that there's no Ted Amazon who works over there who forms a single entity's opinion. It's a bunch of small teams. Some of them like me, some of them can't stand me, far and away the majority don't know who I am. And that is okay. In theory; in practice, I find it completely unforgivable because how dare you? But I understand it's—Abby: You write a memo, right now. [laugh].Corey: Exactly. Companies are people and people are messy, and for better or worse, it is impossible to patch them. So, you have to almost route around them. And that was something that I found that Puppet did very well, coming from the olden days of sysadmin work where we spend time doing management [bump 00:15:53] the systems by hand. Like, oh, I'm going to do a for loop. Once I learned how to script. Before that, I use Cluster SSH and inadvertently blew away a University's entire config file what starts up on boot across their entire FreeBSD server fleet.Abby: You only did it once, so it's fine.Corey: Oh, yeah. I'm never going to screw up again. Well, not like that. In other ways. Absolutely, but at least my errors will be novel.Abby: Yeah. It's learning. We all learn. If you haven't taken something down in production in real-time, you have not lived. And also you [laugh] haven't done tech. [laugh].Corey: Oh, yeah, you either haven't been allowed close enough to anything that's important enough to be able to take down, you're lying to me, or thirdly—and this is possible, too—you're not yet at a point in your career where you're allowed to have access to the breaky parts. And that's fine. I mean, my argument has always been about why I'd be a terrible employee at Google, for example, is if I went in maliciously on day one, I would be hard-pressed to take down google.com for one hour. If I can't have that much impact intentionally going in as a bad actor, it feels like there'd be how much possible upside, positive impact can I have what everyone's ostensibly aligned around the same thing?It's the challenge of big companies. It's gaining buy-in, it's gaining investment in the idea and the direction you're going in. Things always take longer, you have to wind up getting multiple stakeholders on board. My consulting practice is entirely around helping save money on the AWS bill. You'd think it would be the easiest thing in the world to sell, but talking to big companies means a series of different sales conversations with different folks, getting them all on the same page. What we do functionally isn't so much look at the computer parts as it is marriage counseling between engineering and finance. Different languages, different ways of thinking about things, ostensibly the same goals.Abby: I mean, I don't think that's a big company problem. I think that's an every company problem if you have more than, like, five people in your company.Corey: The first few years here, it was just me and I had none of those problems. I had very different problems, but you know—and then we started bringing other people in, it's like, “Oh, yeah, things were great until we hired people. Ugh, mistake. Never do that.” And yeah, it turns out that's not particularly sustainable.Abby: Stakeholder management is hard. And you mentioned something about routing around. Well, you can't actually route around people, unfortunately. You have to get people to buy in, you have to bring people along on the journey. And not everybody is at the same place in the way they think about the work you're doing.And that's true at any company, big or small. I think it just gets harder and more complex as the company gets bigger because it's harder to make the changes you need to make fast enough, but I'd say even at a company the size of Puppet, we have the exact same challenges. You know, are the teams aligned? Are we aligned on the right things? Are we focusing on the right things?Or, do we have the right priorities in our backlog? How are we doing the work that we do? And if you're trying to drive innovation, how fast are we innovating? Are we innovating fast enough? How tight are our feedback loops?It's one of those things where the conversations that you and I have had externally with customers are the same conversations I have internally all the time, too. Let's talk about innovators' dilemma. [laugh]. Let's talk about feedback loop. Let's talk about what does it mean to get tighter feedback loops from customers and the field?And how do you align those things to the priorities in your backlog? And it's one of those never-ending challenges that's messy and complicated. And technology can enable it, but the technology is also messy and hard. And I do love going to conferences and seeing how pretty and easy things could look, and it's definitely a great aspiration for us to all shoot for, but at the end of the day, I think we all have to recognize there's a ton of messiness that goes on behind to make that a reality and to make that really a product and a technology that we can sell and get behind, but also one that we buy in, too, and are able to use. So, I think we as a technology industry, and particularly those of us in the Bay Area, we do a disservice by talking about how easy things are and why—you know, I remember a conversation I had in 2014 where someone asked me if Docker was already passe because everybody was doing containerized applications, and I was like, “Are they? Really? Is that an everyone thing? Or is that just an ‘us' thing?” [laugh].Corey: Well, they talk about it on the conference stages an awful lot, but yeah. New problems that continue to arise. I mean, I look back at my early formative years as someone who could theoretically be brought out in public and it was through a consulting project, where I was a traveling trainer for Puppet back in 2014, 2015, and teaching people who hadn't had exposure before what Puppet was about. And there was a definite experience in some of the people attending class where they were very opposed to the idea. And dig down a little bit, it's not that they had a problem with the software, it's not that they had a problem with any of the technical bits.It's that they made the mistake that so many technologists made—I know I have, repeatedly—of identifying themselves with the technology that they work on. And well, in some cases, yeah, the answer was that they ran a particular script a bunch of times and if you can automate that through something like Puppet or something else, well, what does that mean for them? We see it much larger-scale now with people who are, okay, I'm in the data center working on the storage arrays. When that becomes just an API call or—let's be serious, despite what we see in conference stages—when it becomes clicking buttons in the AWS console, then what does that mean for the future of their career? The tide is rising.And I can't blame them too much for this; you've been doing this for 25 years, you don't necessarily want to throw all that away and start over with a whole new set of concepts and the rest because unlike what Twitter believes, there are a bunch of legitimate paths in this industry that do treat it as a job rather than an all-consuming passion. And I have no negative judgment toward folks who walk down that direction.Abby: Most people do. And I think we have to be realistic. It's not just some. A lot of people do. A lot of people, “This is my nine-to-five job, Monday through Friday, and I'm going to go home and I'm going to spend time with my family.”Or I'm going to dare I say—quietly—have a life outside of technology. You know, but this is my job. And I think we have done a disservice to a lot of those individuals who for better or for worse, they just want to go in and do a job. They want to get their job done to the best of their abilities, and don't necessarily have the time—or if you're a single parent, have the flexibility in your day to go home and spend another five, six hours learning the latest technology, the latest programming language, set up your own demo environment at home, play around with AWS, all of these things that you may not have the opportunity to do. And I think we as an industry have done a disservice to both those individuals, as well in putting up really imaginary gates on who can actually be a technologist, too.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of "Hello, World" demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking databases, observability, management, and security.And - let me be clear here - it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself all while gaining the networking load, balancing and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build.With Always Free you can do things like run small scale applications, or do proof of concept testing without spending a dime. You know that I always like to put asterisks next to the word free. This is actually free. No asterisk. Start now. Visit https://snark.cloud/oci-free that's https://snark.cloud/oci-free.Corey: Gatekeeping, on some level, is just—it's a horrible thing. Something I found relatively early on is that I didn't enjoy communities where that was a thing in a big way. In minor ways, sure, absolutely. I wound up gravitating toward Ubuntu rather than Debian because it turned out that being actively insulted when I asked how to do something wasn't exactly the most welcoming, constructive experience, where they, “Read the manual.” “Yeah, I did that and it was incomplete and contradictory, and that's why I'm here asking you that question, but please continue to be a condescending jackwagon. I appreciate that. It really just reminds me that I'm making good choices with my life.”Abby: Hashtag-RTFM. [laugh].Corey: Exactly. In my case, fine, its water off a duck's back. I can certainly take it given the way that I dish it out, but by the same token, not everyone has a quote-unquote, thick skin, and I further posit that not everyone should have to have one. You should not get used to personal attacks as a prerequisite for working in this space. And I'm very sensitive to the idea that people who are just now exploring the cloud somehow feel that they've missed out on their career, and that so there's somehow not appropriate for this field, or that it's not for them.And no, are you kidding me? You know that overwhelming sense of confusion you get when you look at the AWS console and try and understand what all those services do? Yeah, I had the same impression the first time I saw it and there were 12 services; there's over 200 now. Guess what? I've still got it.And if I am overwhelmed by it, I promise there's no shame in anyone else being overwhelmed by it, too. We're long since past the point where I can talk incredibly convincingly about AWS services that don't exist to AWS employees and not get called out on it because who in the world has that entire Rolodex of services shoved into their heads who isn't me?Abby: I'd say you should put out… a call for anyone that does because I certainly do not memorize the services that are available. I don't know that anyone does. And I think even more broadly, is, remember when the landscape diagram came out from the CNCF a couple of years ago, which it's now, like… it's like a NASCAR logo of every logo known to man—Corey: Oh today, there's over 400 icons on it the last time I saw—I saw that thing come out and I realized, “Wow, I thought I was going to shit-posting,” but no, this thing is incredible. It's, “This is great.” My personal favorite was zooming all the way in finding a couple of logos on in the same box three times, which is just… spot on. I was told later, it's like, “Oh, those represent different projects.” I'm like, “Oh, yeah, must have missed that in the legend somewhere.” [laugh]. It's this monstrous, overdone thing.Abby: But the whole point of it was just, if I am running an IT department, and I'm like, “Here you go. Here's a menu of things to choose,” you're just like, “What do I do with this information? Do I choose one of each? All the above? Where do I go? And then, frankly, how do I make them all work together in my environment?” Because they all serve very different problems and they're tackling different aspects of that problem.And I think I get really annoyed with myself as an industry—like, ourselves as an industry because it's like, “What are we doing here?” We're trying to make it harder for people, not only to use the technology, to be part of it. And I think any efforts we can make to make it easier and more simple or clear, we owe it to ourselves to be able to tell that story. Which now the flip side of that is describing cloud-native in the cloud, and infrastructure and automation is really, really hard to do [laugh] in a way that doesn't use any of those words. And I'm just as guilty of this, of describing things we do and using the same language, and all of a sudden you're looking at it this says the same thing is 7500 other websites. [laugh]. So.Corey: Yep. I joke at RSA's Expo Hall is basically about twelve companies selling different things. Sure, each one has a whole bunch of booths with different logos and different marketing copy, but it's the same fundamental product. Same challenge here. And this is, to me, the future of cloud, this is where it's going, where I want something that will—in my case, I built a custom URL shortener out of DynamoDB, API Gateway, Lambda, et cetera, and I built this thing largely as a proof of concept because I wanted to have experience playing with these tools.And that was great, not but if I'm doing something like that in production, I'm going with Bitly or one of the other services that provide this where someone is going to maintain it full time. Unless it is the core of what I'm doing, I don't want to build it myself from popsicle sticks. And moving up the stack to a world of folks who are trying to solve a business problem and they don't want to deal with the ten prerequisite services to understand the cloud, and then a whole bunch of other things tied together, and the billing, and the flow becomes incredibly problematic to understand—not to mention insecure: because we don't understand it, you don't know what your risk exposure is—people don't want that. They—Abby: Or to manage it.Corey: Yeah.Abby: Just the day-to-day management. Care and feeding, beyond security. [laugh].Corey: People's time is free. So, yeah. For example, do I write my own payroll system? Absolutely not. I have the good sense to pay a turnkey company to handle that for me because mistakes will show.I started my career running email systems. I pay for Google workspaces—or GSuite, or Gmail, or whatever the hell they're calling it this week—because it's not core and central to my business. I want a thing that winds up solving a business problem, and I will pay commensurately to the value that thing delivers, not the individual constituent costs of the components that build it together. Because until you're significantly scaled out and it is the core of what you do, you're spending more on people to run the monstrous thing than you are for the thing itself. That's always the way it works.So, put your innovation where it matters for your business. I posit the for an awful lot of the things we're building, in order to achieve those outcomes, this isn't it.Abby: Agreed. And I am a big believer in if I can use off-the-shelf software, I will because I don't believe in reinventing everything. Now, having said that, and coming off my soapbox for just a hot minute, I will say that a lot of what's happening, and going back to where I started around the enterprise infrastructure, we're reinventing so many things that there is a lot of new things coming up. We've talked about containers, we've talked about Kubernetes, around container scheduling, container orchestration, we haven't even mentioned service mesh, and sidecars, and all of the new ways we're approaching solving some of these older problems. So, there is the need for a broad proliferation of technology until the contraction phase, where it all starts to fundamentally clicks together.And that's really where the interesting parts happen, but it's also where the confusion happens because, “Okay, what do I use? How do I use it? How do these pieces fit together? What happens when this changes? What does this mean?”And by the way, if I'm an enterprise company, I'm a payroll company, what's the one thing I care about? My payroll software. [laugh]. And that's the problem I'm solving for. So, I take a little umbrage sometimes with the frame that every company is a software company because every company is not a software company.Every company can use technology in ways to further their business and more and more frequently, that is delivering their business value through software, but if I'm a payroll company, I care about delivering that payroll capabilities to my customer, and I want to do it as quickly as possible, and I want to leverage technology to help me do that. But my endgame is not that technology; my endgame is delivering value to my customers in real and meaningful ways. And I worry, sometimes, that those two things get conflated together. And one is an enabler of the other; the technology is not the outcome.Corey: And that is borderline heresy for an awful lot of folks out there in the space, I wish that people would wake up a little bit more and realize that you have to build a thing that solves customer pain, ideally, an expensive customer pain, and then they will basically rush to hurl money at you. Now, there are challenges and inflections as you go, and there's a whole bunch of nuances that can span entire fields of endeavor that I am hand-waving over here, and that's fine, but this is the direction I think we're going and this is the dawning awareness that I hope and trust we'll see start to take root in this industry.Abby: I mean, I hope so. I do take comfort in the fact that a lot of the industry leaders I'm starting to see, kind of, equate those two things more closely in the top [track 00:31:20]. Because it's a good forcing function for those of us that are technologists. At the end of the day, what am I doing? I am a product company, I am selling software to someone.So clearly, obviously, I have a vested interest in building the best software out there, but at the end of the day, for me, it's, “Okay, how do I make that truly impactful for customers, and how do I help them solve a problem?” And for me, I'm hyper-focused on automation because I honestly feel like that is the biggest challenge for most companies; it's the hardest thing to solve. It's like getting into your auto-driving car for the first time and letting go the steering wheel and praying to the software gods that that software is actually going to work. But it's the same thing with automation; it's like, “Okay, I have to trust that this is going to manage my environment and manage my infrastructure in a factual way and not put me on CNN because I just shut down entire customer environment,” or if I'm an airline and I've just had a really bad week because I've had technology problems. [laugh]. And so I think we have to really take into consideration that there are real customer problems on the other end of that we have to help solve for.Corey: My biggest problem is the failure mode of this is not when people watch the conference-ware presentations is that they're not going to sit there and think, “Oh, yeah, they're just talking about a nuanced thing that doesn't apply to our constraints, and they're hand-waving over a lot of stuff,” it's that, “Wow, we suck.” And that's not the takeaway anyone should ever have. Even Netflix doesn't operate the way that Netflix says that they do in their conference talks. It's always fun sitting next to someone from the company that's currently presenting and saying something to them, like, “Wow, I wish we did things that way.” And they said, “Yeah, I wish we did, too.”And it's always the case because it's very hard to get on stage and talk for 45 minutes about here's what we completely screwed up on, especially at the large publicly traded companies where it's, “Wait, why did our stock price just dive five perce—oh, my God, what did you say on stage?” People care [laugh] about those things, and I get it; there's a risk factor that I don't have to deal with here.Abby: I wish people would though. It would be so refreshing to hear someone like, “You know what? Ohh, we really messed this up, and let me walk you through what we did.” [laugh]. I think that would be nice.Corey: On some level, giving that talk in enough detail becomes indistinguishable from rage-quitting in public.Abby: [laugh].Corey: I mean, I'm there for it. Don't get me wrong. But I would love to see it.Abby: I don't think it has to be rage-quitting. One of the things that I talk to my team a lot about is the safety to fail. You can't take risk if you're too afraid to fail, right? And I think you can frame failure in a way of, “Hey, this didn't work, but let me walk you through all the amazing things we learned from this. And here's how we used that to take this and make this thing better.”And I think there's a positive way to frame it that's not rage-quitting, but I do think we as an industry gloss over those learnings that you absolutely have to do. You fail; everything does not work the first time perfectly. It is not brilliant out the gate. If you've done an MVP and it's perfect and every customer loves it, well then, you sat on that for way too long. [laugh]. And I think it's just really getting comfortable with this didn't work the first time or the fourth, but look, at time seven, this is where we got and this is what we've learned.Corey: I want to thank you for taking so much time out of your day to wind up speaking to me about things that in many cases are challenging to talk about because it's the things people don't talk about in the real world. If people want to learn more about what you're up to, who you are, et cetera, where can they find you?Abby: They can find me on the Twitters at @ab415. I think that's the best way to start, although I will say that I am not as prolific as you are on Twitter.Corey: That's a good thing.Abby: I'm a half-assed Tweeter. [laugh]. I will own it.Corey: Oh, I put my full ass into it every time, in every way.Abby: [laugh]. I do skim it a lot. I get a lot of my tech news from there. Like, “What are people mad about today?” And—Corey: The daily outrage. Oh, yeah.Abby: The daily outrage. “What's Corey ranting about today? Let's see.” [laugh].Corey: We will, of course, put a link to your Twitter profile in the [show notes 00:35:39]. Thank you so much for taking the time to speak with me. I appreciate it.Abby: Hey, it was my pleasure.Corey: Abby Kearns, CTO at Puppet. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with a comment telling me about the amazing podcast content you create, start to finish, at Netflix.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Lavando de Noche
NODAL SALVA A FAN - A LA SPANIC NO LA QUIEREN NI SUS AMIGOS - LAMBDA LLORA DESCONSOLADO | LDN

Lavando de Noche

Play Episode Listen Later Oct 19, 2021 65:16


Suscríbanse al canal ⬇️ https://www.youtube.com/hildaisasalas Síguenos en Instagram ⬇️ https://www.instagram.com/lavandodenoche Síguenos en Twitter ⬇️ https://twitter.com/lavandodenoche Escucha nuestro Podcast en Spotify ⬇️ https://open.spotify.com/show/5qZvo25texwnKRqq0WaDWD #HildaIsaSalas #HugoAlexanderMaldonado #ComandanteMaganda #YouTube #Facebook #FamososEnVivo #ChismeEnVivo #EspectáculosEnVivo #LavandoDeNoche #LDN --- This episode is sponsored by · Anchor: The easiest way to make a podcast. https://anchor.fm/app --- Send in a voice message: https://anchor.fm/lavando-de-noche/message Support this podcast: https://anchor.fm/lavando-de-noche/support

Serverless Chats
Episode #115: Serverless Complexity with Ant Stanley

Serverless Chats

Play Episode Listen Later Oct 18, 2021 67:38


Ant is a consultant, community organizer, and co-founder of Homeschool from Senzo. He also founded and currently runs the Serverless User Group in London, is part of the ServerlessDays London organizing team and the global ServerlessDays leadership team. Previously Ant was a co-founder of A Cloud Guru, and was responsible for organizing the first ServerlessConf event in New York in May 2016. Living in London since 2009, Ant's background before Serverless is primarily as a Solution Architect at various organisations, from managed service providers to Tier 1 telecommunications providers. He started his career in 1999 doing Y2K upgrades in his native South Africa, and then spent 5 years being paid to write VB6. His current focus is Serverless, GraphQL and Node.js. Twitter: @IamStan Homeschool from Senzo: https://homeschool.dev ServerlessDays: serverlessdays.io For organizer information: organise@serverlessdays.io

The Swyx Mixtape
[Weekend Drop] Abhi Aiyer & Ward Peeters: Gatsby 4 and the Jamstack Endgame

The Swyx Mixtape

Play Episode Listen Later Oct 17, 2021 51:01


The following is my conversation with Abhi Aiyer and Ward Peeters, two lead engineers behind Gatsby Cloud and the recently announced Gatsby v4, which is at the forefront of what I think is the most significant change in the Jamstack landscape in the past 2 years.Watch the video version here. Links: Gatsby 4 Netlify DPR My blogpost on Smart Clients vs Smart Servers Timestamps:  [00:00:00] Cold Open  [00:00:28] Swyx Intro  [00:01:59] Call Start  [00:03:07] Gatsby v4  [00:06:23] Incremental Builds  [00:07:16] Cache Invalidation  [00:09:03] Gatsby DSG vs Netlify DPR  [00:09:35] Abandoning Redux for LMDB  [00:11:50] Parallel Queries (PQR)  [00:13:32] Gatsby DSG  [00:15:24] Netlify DPR vs Gatsby DSG  [00:19:19] The End of Jamstack  [00:22:12] Tradeoffs and Performance  [00:24:34] Image Processing  [00:27:25] Automatic DSG  [00:29:33] Gatsby Cloud vs Netlify  [00:33:34] Gatsby vs Next.js  [00:35:41] Gatsby and the Content Mesh  [00:37:19] React 18 and Gatsby  [00:39:45] Custom rendering page fragments with React 18  [00:42:10] Server Components in Limbo  [00:43:33] Smart Servers vs Smart Clients  [00:45:21] Apollo and Open Source Startup Strategy  [00:47:06] TMA: Too Many Acronyms  [00:49:16] Gatsby for Docs  Transcript [00:00:00] Cold Open  [00:00:00] Abhi Aiyer: And so with LMDB in place, right? We have workers that can read and write to LMDB, which allows us to run parallel queries. So PQR was a huge advancement for us. I think we saw up to like 40% reduction in query running time. And build times went down. We had a goal, I think it was like, we'd try to look for at least 20% reduction in build times and I think we hit 26%, so all cool wins, you know? [00:00:28] Swyx Intro  [00:00:28] swyx: The following is my conversation with Abhi Aiyer, and Ward Peeters, two lead engineers behind Gatsby Cloud, and the recently announced Gatsby V4, which is at the forefront of what I think is the most significant change in the JAMstack landscape in the past two years. We discussed how parallel query writing PQR and deferred static generation DSG are achieving 40% faster queries and 300% faster overall builds. [00:00:53] And they did a wonderful job handling the most impolite questions I could think of, including whether it Gatsby Cloud is a Netlify clone or the Gatsby should just be a data layer on top of Next.js and how they're dealing with TMA too many acronyms in web development. This conversation should be viewed together with my past discussions, with Sunil Pai and Misko Hevery in considering the cutting-edge of web development today. Online discussions often present a binary split in that your technical choices either have to optimize for developer experience or user experience. [00:01:25] But I find that it is builders like Abhi and Ward and Misko and Sunil who are constantly trying to improve the experience of developers in building great user experiences by default. I hope you enjoy these long form conversations I'm trying to produce with amazing developers. I still don't have a name for it. [00:01:41] And I still don't know what the plan is. I just know that I really enjoy it. And the feedback from you guys have been really great. So if you like this, share with a friend, if you have other requests for guests, tag them on social media, I basically like to make this a space where passionate builders and doers can talk about their craft and where things are going. [00:01:58] So here's the interview.  [00:01:59] Call Start  [00:01:59] Abhi Aiyer: I'm Abhi Aiyer. I'm a principal engineer at Gatsby. Thanks for having us.  [00:02:05] Ward Peeters: My name is Ward Peeters. I'm a staff software engineer at Gatsby and I'm from Belgium. And I've been working mostly on the open source side.  [00:02:15] Abhi Aiyer: I forgot to say where I'm from. I'm from Los Angeles, you know, Hollywood,  [00:02:21] swyx: I'm actually heading down to LA,  [00:02:22] Abhi Aiyer: in a couple of weeks, there's,  [00:02:24] swyx: I'm going to Kubecon, which is like a very interesting thing for a front end engineer to end up at. But that's where my career has taken me.  [00:02:34] So this conversation started because I had a chat with Sunil, on this podcast that I accidentally launched. I don't think we did Gatsby much, a good favor. [00:02:45] Like we both saw the new updates and I didn't get to say the nice things that I thought about Gatsby. I should also say that I used to have my blog on Gatsby and I no longer do. I used to work at Netlify and I no longer do. There's a lot of history here for me with Gatsby. It's been a while since I caught up, and I'm curious to see or get the latest. [00:03:07] Gatsby v4  [00:03:07] swyx: Maybe we should start off with like a quick summary of what's new with Gatsby with Gatsby V4, right?  [00:03:13] Abhi Aiyer: Is that a good place to start? Yeah, I think so. [00:03:17] swyx: So first of all, I think the marketing was really nice. Gatsby camp, it seems like a really big push and qualitatively very different from Gatsby 3. Tell me about what the behind the scenes was like. [00:03:30] Abhi Aiyer: Yeah, it was, we're getting better at the marketing side of what we're doing these days and Gatsby 4 was a big push. It really changed how we approach the framework as a whole.  [00:03:43] For those who don't know, traditionally Gatsby was a static site generator, purely static. We hold ourselves high on our connections to a content management system. [00:03:55] And we provide a really good data layer there, that takes all those requests that you would normally make to a content manager system, turns them into a, like a store of data that you can then use and query from graph QL. And the big thing that we were hitting before gas before was. Company was growing. [00:04:17] And as more customers were using Gatsby cloud, we started realizing that we couldn't scale to really large sites and large sites is like a misnomer. Like you could be, you could be a 50,000 page site and be considered large given the data that you may have. But we're talking like hundreds of thousands of pages. [00:04:38] And the thing that we kind of realized is not all pages are created equal on your site. Especially the ones from like 20, 15, 20 14, where, you know, no one's looking at that people, those pieces of content, if you're a site with a huge archive of content yeah, you should probably go check Google analytics to see how, you know, how, how many people are actually viewing your pages. [00:05:02] And the way gets me. And we'll get into this a little bit later, but today Gatsby isn't as smart as it should be in what pages should be rebuilt. For example, we're looking at the gatsbyjs.com page right here, but there are other data dependencies. This pure content. Like if you look at the nav bar, there's a whole bunch of items there. [00:05:22] And we have this navbar bar on all of our pages, cause that's what a website has, but the problem with Gatsby today and will be changed in the future is. If I change anything about this nav bar, any page, that depends on the nav bar now has a dependency that needs to be invalidated. [00:05:39] And so now I have a hundred thousand pages with this navbar I have 200,000 pages of this nav bar and I spelled Gastby instead of Gatsby or something, the navbar I made a typo and now I'm paying for. A hundred thousand pages of reload to rebuild. And so we just saw that and that this happens a lot, right? [00:05:57] This is a very small example, but this happens a lot to our customers and building a hundred thousand pages is not necessarily easy thing to do. There's memory requirements that come in. There is, what kind of machine are you doing this thing on? And so we had to figure out a way to scale Gatsby and do things differently. [00:06:15] We were traditionally static and now we're trying to be somewhere in between, you can go dynamic or you could go static and it's up to you.  [00:06:23] Incremental Builds  [00:06:23] swyx: So the new rendering options are SSG, DSG and SSR. Is ISR gone?  [00:06:32] Ward Peeters: Well, that's what Next.js has is doing. And I'm like, wait,  [00:06:36] swyx: we never have guessed. [00:06:38] We had an incremental mode.  [00:06:41] Abhi Aiyer: What do you  [00:06:41] Ward Peeters: call it? Yes. And that's still all statically. So when we do it, we have it in open source where we in V3 we enabled it where we only build HTML what's necessary on cloud. We have a more pumped up version of it where When you get the data change, we only update that page more rapidly than in open source, but still when you change your navbar with what Abhi said, you still have to rebuild everything because all the pages get invalidated. [00:07:09] So incremental builds works for data updates, but not so much for code changes.  [00:07:16] Cache Invalidation  [00:07:16] swyx: Right. Okay. Well, I guess, how do you solve cache invalidation? [00:07:26] Ward Peeters: Well, the thing is that because Gatsby owns the data, like the heads of data layer and a source plugins like WordPress, when we source data and to give us a webhook or, we just go to Wordpress and say like, Hey, what has changed? [00:07:40] Data. I was like, okay, these nodes have changed. Or these pieces, like a poster page has been changed. It gets me knows which node is used where, like, for example, this post is used on this section page. It's used in this article and that's all happening already behind the scenes because graph QL shenanigans. [00:07:59] And that's how we can build incremental builds. So we know, okay. Only these spaces need to be built. And that's also where DSG comes in because as a user, you don't have to care about cache invalidation anymore. Cause it's using the same engine as you were used to with like incremental builds. [00:08:15] When you use SSG and I think that's a major benefit of it, where you as a user, don't really have to care about cache control, because it is difficult to manage on a large scale. Like a lot of corporations just say like every 10 minutes we'll remove the cache because it is difficult to get through when change. [00:08:37] Yeah.   [00:08:39] swyx: That's pretty funny. At Netlify, one of the reasons that we constantly talk about for CDN level caching, like people say like, you know, why don't you just enable CDN level caching and then just have a regular server render. One of Matt Billman points that he always makes is that people always turn it off the moment there's a bug, it's like, oh, schedule, call, and turn it off. [00:09:02] And then don't turn it back on again. [00:09:03] Gatsby DSG vs Netlify DPR  [00:09:03] swyx: Okay. So let's, let's talk about like, DSG. That's the fancy new one and DPR, right? So maybe we should, is there. Yeah, there's a graphic. Okay. All right. This is new. So first of all, like what was the inspiration? What's the backstory I'm always interested in how these things come about. [00:09:21] Abhi Aiyer: I think we were inspired by DPR a lot, you know? But where we saw the benefit of our approach is our data layer, you know, and it took those many steps even before getting to DSG.  [00:09:35] Abandoning Redux for LMDB  [00:09:35] Abhi Aiyer: So it started actually in like Gatsby 3.10. We had to redo Gatsby's node store. [00:09:42] So traditionally we were using Redux to persist all these the data that we get from content management systems. And we had a particular customer who could not even persist the cache, like a Gatsby cache between builds, because they had so much data that it would OOM when they try to persist the cache. Right. [00:10:03] So for them, they were running cold builds for every build. Even if you had like a warm cache or you had your pods, you know, we use Kubernetes. So like, if you have your pods up, you're doing like an hour and a half cold build for everything. You could like change the typo and it'd be an hour and a half. [00:10:19] And so from there we were like, We need to reduce peak memory utilization and Redux is not going to help us there. And so we started looking into alternatives. We looked at SQL Lite, we looked at Reddis and we landed on LMDB, which is, Lightning memory, mapped database manager. [00:10:39] It's like a file system DB, which is really cool for us because one, it's pretty fast. It allows you to, to have like a query interface, which is good. You can store more data than available RAM. So for a site like this customer who pretty much is blowing up this pod on every warm build. To try to even have a warm build, we could actually store their data now, which then unlocked warm builds for them. [00:11:05] So an hour and a half, that went to 25 minutes, which is pretty good for them. now we have this thing, now we call it Gatsby DB internally. And so now Gatsby is node store is in LMDB. And the cool thing about LMDB is it's just comprised of a bunch of files. You have a lock file and database files. [00:11:26] And if you have files, that means you can move files around. They don't have to be in one place, right. They could be in storage, they can be in a serverless function. They could be anywhere you, you really want. Right. And so that was step one was we needed to redo the node store. And we did that and memory utilization from a lot of customers went down. Enough to unlock a simple thing as a warm build.  [00:11:50] Parallel Queries (PQR)  [00:11:50] Abhi Aiyer: So then the second thing that this, these other customers were having was like, wow, it takes so long to query, to run queries. Right. And people have like 25,000, 50,000 queries. And I don't know if they should have those that much, but they do. [00:12:05] Right. They do have that much. And it's a huge part of the build time. Right. A lot of people complained that. You know, Gatsby builds are sometimes slow for large sites and we agree. That's totally true. And so our next foray into like improvement was this thing called parallel queries. Which would allow Gatsby to run chunks of queries at a given time and what PQR in his pool, a diagram of it, you know, query running does take a huge percentage of your builds. [00:12:39] But now we can parallelize that across workers in the Gatsby process. But if you were to do this naively with Redux, like a child process can't write to a JavaScript object in the main process. Right. It's hard to maintain state. There's no easy way to communicate between workers to write state and read it. [00:12:59] And so with LMDB in place, we have workers that can read and write to LMDB, which allows us to run parallel queries. Right. We don't need to do things serially, anymore. So PQR was a huge advancement for us. I think we saw up to like 40% reduction in query running time. And build times went down or we had like a goal, like I think it was like, we'd try to look for at least 20% reduction in build times. [00:13:27] And I think we hit 26%, so all cool wins, you know?  [00:13:32] Gatsby DSG  [00:13:32] Abhi Aiyer: And so then Ward and I, and the team were all just like thinking like, okay, we have LMDB. We've got PQR. Alright, well really we have a Gatsby data layer that can be accessed from anywhere, right? Cause if you can access it from a worker, you can access it in a serverless function cloud run, you know, on your somewhere, anywhere you spin up your own machine and your own office, if you want it well  [00:13:56] swyx: steady coast. [00:13:57] How about that? Like an S3  [00:14:00] Abhi Aiyer: bucket, you put it in an S3 bucket, for sure. You know, like you could put those files there and then retrieve them from wherever you want. And so that's when we started thinking like, okay, we have this information now, what can we do to improve the, the life of our users even more. [00:14:19] And then we started thinking about DPR and like, we saw the approach and we were like, wow, this is exactly what we need, but we have Gatsby's data layer that kind of complicates things, but it's not complicated anymore because we can essentially use the data layer wherever we wants. So I'll let ward kind of go from there on like how DSG came about after these like fundamental pieces. [00:14:42] Ward Peeters: Yeah. So then we looked at like ISR DPR and like what's the difference in both of them. And then we saw like ISR that's where you have a refresh timeout and an hour with, in the latest next, you can also being an endpoint to they're getting validated cache, but it's all manual work. And there were many complaints about it's an index. [00:15:02] It's nothing in Gatsby and they complained about stale data, because what Next.js does is you visit the page and then the next time it will update. So I think it's a refresh or something. Yeah. [00:15:15] swyx: Alright. Alright. We don't have to dig through issues on, on the, on our call, but I just wanted to illustrate the problem.  [00:15:24] Ward Peeters: Yeah.   [00:15:24] Netlify DPR vs Gatsby DSG  [00:15:24] Ward Peeters: And then that's where we took it away and then say, okay, DPR. And then I looked at the spec of DPR, like, okay. Can we use the same name or not? And the problem with DPR was they had Atomic deploys. So every change means blow the whole cache away and do everything new and we were like, what do we have incremental builds from there? We don't want to like invalidate the whole cache. We just want to invalidate the pages that got removed. And there's like a GitHub discussion about it, where I commented as well. [00:15:55] And it felt like they didn't want to change the name. Yep. There you go.  [00:16:04] swyx: So you said to me, DPR, doesn't need to be opinionated about if the file is part of the atomic deploy. Can you reiterate why?  [00:16:13] Ward Peeters: Yeah, the thing is basically because they mentioned like everyday glory needs to blow the cache away and needs to be fresh. [00:16:20] And for me, like it shouldn't be in a spec like DPR should just say you built pieces at build-time and you build those pieces at runtime. That's basically what I was trying to say. And then because we have incremental builds, we only want to invalidate like five pages, even if you use SSG or DSG, we still want to say if you only changed five pages for evil dates to cache for five pages, I couldn't get that from the spec. [00:16:46] I think that's also because Netlify does it their way, which is totally fine, but then that's why we created a new acronym called DPR. And I think it's also probably explains. What we offer as well, a little bit better too, because it's Deferred Static Generation. It's like lazy SSG, something like that, because that's what we do. [00:17:08] Like you can mark a page as defer and that just means we don't do it at build time, but the first time you hit a request. We rebuild it in like a Lambda, I could use Cloud Run, we build it and then we give the response to a user and then also we save it to disk. So from there on, the second request, it's technically an SSG page. [00:17:29] We store it like you have the CDN cache, but we also have it inside our bucket. Like, your S3 buckets or whatever you want to call it.  [00:17:37] Abhi Aiyer: Yeah. We're caching responses, but we're also making that file exist as if it existed at build time. And that's a big distinction for us because what that allows us to do in the future would be like, if nothing changed about the data for the given page, then you don't need to DSG again. [00:17:56] Right. Like if nothing changes for, let's say like there's five builds and build a. Something changed in your data dependencies. So now you have a DSG page and then nothing changed for the next five builds, but a user comes and actually visits that page generates the files. It gets cacheed in our data layer or our files storage layer and on build five because nothing changed. [00:18:24] You're not DSGing. Right. You're not going to go through this process again. And so that's we think is the big thing about DSG.  [00:18:31] Yeah. And then I think the extra piece of it is because the date, like you can say it it's a benefit or or a negative point of Gatsby, like we source all the data at the build time. [00:18:41] So even if your APIs go down, even with DSG, you still go to our local database. So debts will never go down. Cause if like your site is down, your database will be down as well, but you, you're not dependent of other API. So let's say GitHub goes down or X go down and you need to get that data. We have it locally in our database, so you're still good to go through, still keep that resilience. [00:19:06] And the security even that you, you used to have with Gatsby, and I think that's a main benefit of the whole datalayer piece of Gatsby and DSG.  [00:19:17] Yeah.   [00:19:18] swyx: Yeah. Perfect.  [00:19:19] The End of Jamstack  [00:19:19] swyx: So something I always wonder about like, is this basically the last stage of JAMstack like, I feel like we have explored all possible varieties of rendering. [00:19:30] And this is like the end. This is like, this is it right? Like we have all the options.  [00:19:34] Ward Peeters: And now it's mixing them together. It's the next step having been static and on bits of your thesis, SSR. Uh,  [00:19:43] swyx: okay. I'll put it this way. Do you think that JAMstack at the end of the day after this, maybe like five-year journey of like, Hey, like a WordPress sucks. [00:19:53] That's everyone moves to static. Right. And then, and then we found like, oh yeah, static. Doesn't scale, big surprise. We were telling you that from the beginning. And now okay. Right. Hybrid. Is that it, like, it was that the Jamstack movement in like a five year period? [00:20:10] Abhi Aiyer: I think it's a yes or no. Like evolution is like, I think we're, you know, we're all coming full circle and I think in engineering, particularly we do the same thing all the time, every 10 years or something. Right. But where DSG came into play is for use cases that we saw, you know, and our customers still prefer static. [00:20:31] So I know we're talking about DSG. Like it's like a great thing and it is, but a lot of our customers prefer static and it's really up to their use case. If you're a small site out of a bunch of top of funnel page, any lag in anything, right? Cause DST is not like instant, right? Like you're doing a runtime build essentially. [00:20:51] Right? So in some cases it could be, you know, it could, it could be a longer response time than what the standards should be. And we have customers that won't DSG anything because they have essentially, most pages are top of funnel or high traffic that they would rather just wait.They don't mind waiting for the performance that they would want. [00:21:11] But we also have customers that have hundreds of thousands of pages, like there's one customer that has like a company handbook or something where like, you can see every employee. And like, if they like dogs and like, you know what I'm saying? Like, Bio's and stuff. And they have a lot of employees worldwide, and there, they can only like before DSG, they can only build their site once a week. [00:21:33] Cause it takes like 24 hours to build. What, and now with DSG, they don't really care about someone who no, one's going to view their profile. No offense to that person, but no one's viewing the non CEO's profile. So then how they can, like, you know, and there are other people that are important too. I'm sure, but like now they can actually, you know, make changes to their site. [00:21:55] You know, we actually had to work with them to make sure that, you know, they can build. I mean, previous to DSG, they can build like, at some cadence that we don't necessarily support, but we help support that. So, so just looking static is still king when it makes sense. For sure.  [00:22:12] Tradeoffs and Performance  [00:22:12] swyx: I feel like it's a bit scary when you tell people like, okay, you're deferring the build. [00:22:16] And then on the first request, someone's going to build that. It's not going to take that long. Yeah. Right. It's not like it's that bad. I think bottom line is, I think people are very scared whenever you say, like, okay, there's a trade off, but you don't quantify the trade-offs. And then they're like, oh, it's bigger in their mind than it really is. [00:22:37] Ward Peeters: Yeah, I think a big problem with the plugin ecosystem is that it's difficult to, to quantify like what's slow and what's not slow. For example, generating an MDX page is more time-consuming because it has to like get some dependencies make sure that they have bundled together, then use react to render and then render again because it's how the Gatsby plugin, is built right now that takes more time than a simple React renderToString with something. [00:23:07] And I think that's the difficult thing to say like, okay, it's some pages will be instant. Some pages might take a second to build or we'll half a second.  [00:23:18] swyx: Yeah. The important thing is that there are not data dependencies that you're waiting on. Right. That's usually the slowest part fetch all the data upfront and then you store it in a LMDB cache. [00:23:28] And that's written to a serverless function or written to I guess your build process or whatever. And then people can render that whenever which I think is great. Like, it should be fairly fast, like we're talking tens of milliseconds difference between like for first render, right? [00:23:44] Like something like that. Like I think, I think when you quantify, like, okay, we're talking tens of milliseconds, not hundreds of milliseconds and not thousands of seconds that really helps me with. Put these things in perspective.  [00:23:56] Abhi Aiyer: Yeah. But then, you know, people always find a way to screw it up. So say that like, of  [00:24:01] swyx: course. [00:24:01] Yeah. But, but you give a realistic benchmark and then you go like, yeah, for these benchmarks, we tested it like a hundred times or something. The median was this, the P 95 was that. That's it like, I mean, people can't really fault you for not accounting for every use case because no one, no one can, but at least you can give a reasonable basis and say like,  [00:24:22] Abhi Aiyer: there's,  [00:24:23] swyx: there's an up, there's an upper bound to you know, how bad, how the, the, the trade-off like, you know, when, whenever you miss channels, I like to quantify it basically. [00:24:32] Um, that's a good, that's a good idea.  [00:24:34] Image Processing  [00:24:34] Abhi Aiyer: And like, one thing to know for DSG is like, your data may be like available and that's cool that that may not be the long pole, but let's say you have a portfolio site that generates 20 different types of images for every image. Now you're getting into image processing at runtime, you know? [00:24:54] And so there, there are ways to kind of not do this properly. Right. And or like, for example, let's say your homepage, I love this example. Your homepage has links like to every other page on your site,and it's all DSG, right? So you load the homepage and because Gatsby does prefetch for link tags are doing Gatsby link to other pages. [00:25:17] We go and prefetch every page on your site. And essentially you're doing your build at runtime. So we're going to try to prevent these cases from happening, but just natively going through DSG everything is not my recommendation. That's for sure. [00:25:32] Not today. At least not today.  [00:25:35] swyx: so a couple of things on that. So, this Gatsby image point is very interesting. So how does Gatsby image work with DSG? [00:25:42] Abhi Aiyer: So yeah it works how it does it in Gatsby build. currently today Gatsby uses Gatsby-plugin-sharp and the sharp transformers to take one image, turn it into another. [00:25:54] And even in Gatsby cloud, before we implemented parallel image processing, images were like the slowest part of the Gatsby build because a lot of time, a lot of memory, et cetera. And so we solved that problem. And so in the DSG runtime, we do image processing there for a particular page. [00:26:15] So you will have to wait for image processing. If you're image heavy on a DSG page.  [00:26:21] swyx: Which I mean, does that mean that you cannot do a DSG in a serverless function?  [00:26:26] Abhi Aiyer: In a total? We do. We actually do DSG in serverless. And that's totally fine. Like you can do image processing, you know? But like, I would say your mileage may vary given what kind of transformations you have going on, how many images you have, right. [00:26:42] But like you said, there's, trade-offs right. If the page makes sense for it, you know, we have a bunch of examples that do have images and they work great, you know? But I don't know if I go full on portfolio with like a, you know, like a masonry thing where like, there's like tons of images and they have sub images and you have to go, like, I'll be like a carousel of images and stuff that may not be good for your. [00:27:06] I don't know, but the choices, the users, that's, what we're trying to get at is like, we're trying to give as many options. We're going to give guidance and like we're having our own opinions, but you, you can choose to listen or not, or, you know, do your own thing and we should try to support you as much as we can. [00:27:25] Automatic DSG  [00:27:25] swyx: Yeah, you give me some thought about like, having sort of like a browsers list type of API where you can say like, okay, top 100 most visited pages. No, this is not it. You know what I mean? Like, as a handholding for what should be DSG and what should be statically generated you know, plug into my Google analytics, tell me like top hundred pages statically render those, everything else, DSG. [00:27:48] I'm sure you've thought about it. And I think like maybe four years ago, Gatsby and Guess.js had someone in collaboration, which I assume went nowhere. But let me know if there's.  [00:27:59] Ward Peeters: Uh, okay.  [00:28:02] For now. Yeah, because there is a new way to do it because now greet guests, it stored everything in one file. So we have to like sometimes download a five megabyte Jason file to make guess.js work. Mondays switching around that you could make, get smarter to say like a guess for this route. You only need the bit of the JSON file. But we never implemented it. So,  [00:28:26] Abhi Aiyer: yeah. And we have this, so I'm speaking from the Gatsby cloud perspective, but like you're right, Shawn. Like, if you can hook into Google analytics, you'll get the same thing. [00:28:36] But if you host with Gatsby cloud, we know what, what routes coming through our hosting layer. We know what routes for your site. Are the most hit, you know, we know the requests per route. I mean, how much bandwidth you're using, like per route. And so we could be smarter and tell people exactly how. How to DSG, right? How should you DSG and get it done that way, for sure.  [00:29:04] swyx: Okay. So like a, to be, to be complete, uh, typical to be  [00:29:08] Abhi Aiyer: complete, you know, we're still in beta forgets before, so I would say like, maybe like after we launched for, for sure, we'll start adding some sugar on.  [00:29:17] swyx: Got it. So first of all I did, so this was my first time trying out Gatsby Cloud. I, I think it was behind like a signup wall, like a private beta in the past. And I never really gave it a spin, but again, you know, the V4 announcement really got me going and And yeah. I mean, I'm pretty impressed.  [00:29:33] Gatsby Cloud vs Netlify  [00:29:33] swyx: So how much of this, you know, the hard question, right? How much of this is a Netlify clone, what are you doing differently? [00:29:40] Abhi Aiyer: Let's talking about that. How much does like DSG is  [00:29:45] swyx: how much of Gatsby Cloud? Isn't it  [00:29:48] Abhi Aiyer: like? 0%. Ooh, okay. Yeah. Probably 0% of it is a Netlify clone. [00:29:56] swyx: I do like when you provision it, it gives me like a really good set of options. Uh, let's see, uh, you know, connect CMS guests. Netlify does not have that. [00:30:07] Abhi Aiyer: Yeah. I mean, I would, yeah. We're far from an elephant clone Mo multiple weeks. We've built our whole system based on the needs of Gatsby. The way our cloud front end and our back ends talk to our customers, Gatsby Sites is a novel way of doing it. We've exposed that in open source and I think Netlify actually did implement something for external jobs or something with Google pub sub I, I saw that, but everything we do in Gatsby cloud is for Gatsby. We have no other framework that we need to maintain nor care about, sorry. Luke's or whatever. Like we don't care about that. On Gatsby cloud, we've optimized our hosting layer with Fastly to be part of the data. And so if Gatsby changes, Gatsby cloud changes, and if we need to get to be framework to change, it will for Gatsby cloud. So, and we use Google cloud, so we're not on AWS.  [00:31:09] I would say we have the similar features though, and that's a valid point to bring out. [00:31:13] We have, we have functions, right. [00:31:15] We have domains and we don't have a purchasing domains or anything yet, but you know, we have the whole hosting product and everything like that. Yeah.  [00:31:27] swyx: Is that, is that what you would need for Gatsby Cloud to come out of beta? Like. Domains or like what, what, what are you waiting  [00:31:35] Abhi Aiyer: for essentially? Well, Gatsby cloud isn't in beta. [00:31:38] It's like a  [00:31:38] Oh Gatsby v4  [00:31:40] swyx: is in beta [00:31:41] Abhi Aiyer: yeah. V4 it's in beta. Yeah. Sorry. Yeah. Yeah, domain like domain registry and all that stuff is more sugar on top that we'll be probably releasing mid next year. But we're just trying to be I mean, Gatsby cloud, from that perspective, we just want to be at the table with all the other JAMstack providers. [00:31:59] But our edge is if you want to build a Gatsby site the best way, you know, you have our support team, that'll help you. Right. As a customer of ours, you're like our family. The Gatsby family, you know, we're, we'll help. We help our customers. We have great support and everything we do on the platform is for Gatsby and making Gatsby better. [00:32:18] So there's like so many things in the framework that we've improved by having Gatsby cloud. Cause we didn't know all the sites that could exist and not do things nicely or have problems or, you know, because of Gatsby cloud that the framework is getting so much better because we have real users feedback and they have a lot of demands and we like to, you know, fulfill them. Yeah.  [00:32:41] swyx: Okay. Actually I should probably clarify this earlier. How much of what we just talked about is Gatsby Cloud-only? [00:32:48] Abhi Aiyer: Pretty much all of it, DSG, SSR, they're all capable, you know, you can run it locally and stuff. And I know Netlify has a Gatsby plugin as well that will allow you to run DSG and SSR as well. [00:33:03] For those who are not using those platforms, it's like maybe you're using Amplify or whatever. You're going to have to implement this yourself. I don't recommend it though, because it was a pain in the ass to put it together. But yeah, it should work the best on Gatsby cloud.  [00:33:19] Ward Peeters: So technically all of that we building with v4 is in open source. [00:33:22] So you could wire it up all yourself, but why bother if you can use, like Gatsby Cloud. Yeah, you don't have to care about it.  [00:33:34] Gatsby vs Next.js  [00:33:34] swyx: That's true. Okay. So, just on the rendering side of things, right? I made this observation that, Gatsby now has SSR, now has serverless rendering. All the different rendering modes, like this looks very similar to next JS. [00:33:48] Is it possible to basically say like, okay, Gatsby is the data layer and is this the best data layer and most advanced or whatever, because this is basically what Next.js does, right? Like it's a very, very constrained rendering layer. Why can't you, I mean, you know, sunk costs aside. Why can't you be a next JS layer? [00:34:08] Ward Peeters: Well, technically we could now, because they like implemented everything too, like they have SSG, they have ISR, they have SSR and we could technically move the data layer out of, and use it with next. That could be a possibility, but. We've been, we've come so far and I think do already have built this. [00:34:31] And then now they're also parity. I think having two separate ones and having different dev experience, and maybe Next.js is winning now and Gatsby will a win in, in two months or vice versa. I think it's just a healthy balance. Like it's and I think it's the same thing as a browser wars, like everyone is going to Chrome or chromium and then there is still like, Firefox and iOS, but how long will they survive? [00:34:58] And I think just the competition is what you need. And I think that's why a good reason why we keep separate. And also, I don't think that Next.js is for like, merging with Gatsby or like having the like the same.  [00:35:13] swyx: Oh, I think I know Next.js, it be super happy about it, because then they, when they, when the server for reacts you know, role, and then you focus on the data role, right? [00:35:22] Like, uh, Makes sense to me, obviously I'm brushing over a lot of the plugins actually have a rendering portion as well. So how much can you separate those things  [00:35:33] Abhi Aiyer: if in the next. No, this is possible. I don't, I mean, we're not going to like say that it's happening or anything.  [00:35:41] Gatsby and the Content Mesh  [00:35:41] Abhi Aiyer: Like if we look at Gatsby's like, this is how it's set up. [00:35:45] It's, it's what we call the content mesh. You have all these different data warehouses that exist. WordPress Drupal, et cetera, can even be a freaking Google Sheets. You know, like whatever, and we assemble this data layer at build time. And in doing DSG and SSR, we build something called the query engine that allows you to query this LMD B store that has like the manifested data in there. [00:36:13] So. It really opens up the gate for yeah. If you want to use our data layer in a Next.js app, like, I mean, go ahead. Like once we expose this API to our customers then you can essentially have Gatsby data in an iOS app or an Android app react native. Like, it's just an API call at that point. And you know, Gatsby cloud hosts, like a graphical API for you that you can just query your data. [00:36:38] I don't know if any data scientists would care for that. They could add that into Looker or something. You know, like I remember they want to do it like that stuff would be available and it's almost like a content data lake versus, you know, traditional data lake I guess. It's purely for content and you would have the benefits of Gatsby because we normalize and we create structures and you like, the user can customize a schema, however you want. [00:37:05] And then now you can use it on multiple platforms, right? It's not an immediate goal for us to do so. It's a logical next step. Yeah. Yeah.  [00:37:15] swyx: Awesome. Awesome. Cool. Yeah, I, I feel like that's a really good and in depth coverage. [00:37:19] React 18 and Gatsby  [00:37:19] swyx: Maybe let's end off with talking about the future of React 18 and your plans there. First of all, what's happening in react 18. Is it out? Like the plan for the react 18 and published in June? Okay. All right. Let's talk about it. What's what's going on? [00:37:35] Ward Peeters: So, yeah, so we are working closely with the React team and we also in the working group to figure out like, okay, how can we help the team, make it more stable and give it in user hands. [00:37:46] So I think from may or something, we have introduced React 18 as part of Gatsby. So you can now install React 18 alpha. And we just moved to the new rendering mode. So the async mode suspense and all those things were. Like what, what we're planning on, at least when you use Gatsby, like we have page queries and we have static queries and there's a big pain point is static queries, cause it's a graph QL query, but you cannot have any variables, which means you're kind of limited to the unit. And then you have to move everything to page queries going to have to know all the content up front and wait the new async rendering bits of React to get into like a useQuery, because you can yield the rendering of React at any time. [00:38:34] Cause async doesn't mean you have to go like, uh, use Apollo Server to get server data tree or something or other pieces, or you kind of have two have React async mode or React Suspense in SSR and we can all move it to the page components or the components of your reactor. So basically look that you're just recreating an react application and then every async bit like using react-fetch or a useQuery, it all just works. [00:39:02] I think that's where, where we activate in benefits a lot where it's. It just removes a lot of cruft or that you have to do now. It gets you where you have to be in the Gatsby mindset when you're developing and, and you basically go to a, creating a react app and you have a data layer, but I think React 18 opens so many doors with the new cache APIs. It just becomes way smarter and when you look at it from a performance perspective with the whole concurrent mode where inputs gets priority over rendering, it's just going to be way smoother than what they had so far. [00:39:39] Abhi Aiyer: And hopefully people stop complaining about lighthouse scores and stuff. That'll be great.  [00:39:45] Custom rendering page fragments with React 18  [00:39:45] Abhi Aiyer: Another cool thing that React 18 kind of unlocked for Gatsby in particular is a concept of fragments. And so we were talking about that nav bar example earlier with the a hundred thousand pages. And we want to leverage react 18 with like custom renderers so that we can essentially create fragments of a page that had beta dependent. [00:40:07] Because there's no page query or static query anymore. That's just a query. Your navbar component has a query and essentially Gatsby can make that nap bar a navbar fragment and your body has a fragment, or maybe your footer has a fragment. Your sidebar has a fragment. And as data changes incrementally, we only rebuild fragments and our hosting layer, stitches, fragments together. This is an old concept called ESI includes like if everyone did PHP back in the day, like, you know, very familiar with this stuff, like I said, every 10 years, things has come back around and we're going to try to do that. We're going to try to build fragments of pages, stitch them together. So a navbar change doesn't break the bank, you know? But we can only do that once react 18. It's like, you know, fully there. I mean, we could do it now, but like why, when we should just like work off the, the, the work of others.  [00:41:02] swyx: So when you say fragments, are you referring to GraphQL Fragments or, or like  [00:41:06] Abhi Aiyer: Asian fragment might be a, maybe we call it like, you know, today, like an HTML page that has specific. [00:41:13] You know, I like to call him like the rectangles that we all draw around are our websites. Right. They all have independent data isolation. Right. And so these are like what maybe a Gatsby slice of a page or a fragment or some type of include, you know, like in the templating days. Right. And that's what I kind of mean there. [00:41:31] So these includes or templates or whatever you want to call them would be independently built. And then independently stitched at the cache layer. And then, you know, the data dependencies don't cross, and now I'm not building a hundred thousand pages because I misspelled Gasby and it should've been, you know,  [00:41:51] swyx: sounds like it happens a lot,  [00:41:54] Abhi Aiyer: but definitely those,  [00:41:56] Ward Peeters: and it looks a lot like donut caching. [00:41:58] If you're more familiar with that piece, like you have a page where I said parks has a different. Limit and another one. So that's more or less the technical piece out of  [00:42:10] Server Components in Limbo  [00:42:10] swyx: a server components. Anything on any implications on that forgets me?  [00:42:15] Ward Peeters: Not yet. I would say because they're not going to ship it with react 18. [00:42:19] We've been talking about it, but it's still very fresh or very new, like even the React team hasn't, hasn't worked more on it, so they did their demo, but then it's got like a little bit  [00:42:31] swyx: stagnated. Oh my God. [00:42:37] Ward Peeters: All the pieces. Like they need to build all the pieces underneath it to make it work. [00:42:45] swyx: They jumped, they jumped the gun, maybe in announcing I got so excited. I was like, wow. Okay. I can cut my Javascript bundle by 40% and run backend functions in my react component. And then nothing, nothing for 10 months,  [00:43:01] Ward Peeters: because we are super excited about it too. Because when you look at especially marketing sites, like marketing pages or blogs, there's only a small piece of JavaScript that you actually need. [00:43:13] Like maybe you need a bit for your newsletter button or you like something like that. And why. 200 kilobytes of JavaScript could bring technically only need maybe 10, 20 kilobytes. So I think it's static or with like marketing pages. Uh, [00:43:33] Smart Servers vs Smart Clients  [00:43:33] Abhi Aiyer: yeah, so the world was server rendered. Then we went client side rendered. Then we went static rendered. Now we're DSG rendered, and then we're going to go back to server run. So, you know, time just keeps spinning. Partially server.  [00:43:47] swyx: I called it smart server versus smart clients is my term for it. So this is the, I think maybe my, my most recent posts, because I have been trying to write more, but then I keep have having real life get in the way. [00:44:01] But why is traditional, which is server rendered, different from the new server rendered. We have essentially is essentially exactly the same, but there's a thin runtime, which I'll ship the stuff that we send over the wires changes. And we actually doing rendering in the browser, but like partial rendering, maybe I should say. [00:44:20] And yeah. I dunno. I think, I think this is a very interesting exploration. Phoenix live view is also the other one that, that gets a lot of love for this. And then rails is also adopting Hotwire. So, I don't know where this goes. I mean, I, I it's, it seems like we fully explored the smart client space and the smart server revolution is just kind of get, getting going. [00:44:41] Ward Peeters: We're going back to Meteor.  [00:44:44] swyx: Back to meteor, but not so opinionated, I think, you know, I was very excited about meteor. Like when I, when I first started as a web dev, I was like, oh yeah. Okay. Everything is in there. I actually mentioned Meteor here because it had the mini Mongo data store, which was I thought it was just such a great experience. [00:44:59] Did you use.  [00:45:02] Abhi Aiyer: Oh, both my last company, we used meteor for our backend, and then we had to kind of migrate slowly off of it. Cause they were just ahead of their time. You know, now all those concepts. Those are like, those are the concepts of today. Right. And that's the beautiful thing they were  [00:45:19] swyx: just ahead of their time. [00:45:21] Apollo and Open Source Startup Strategy  [00:45:21] swyx: I mean, you know, what they did was they became Apollo. They were just like, oh no, one's no, one's handling all the hard parts of GraphQL. Well,  [00:45:29] Abhi Aiyer: okay. We'll do it. Yeah, good job of that too,  [00:45:33] swyx: which is by the way, like in terms of just honestly, I'm interested in startups, entrepreneurship, uh, you know, we worked so hard in web dev stuff. [00:45:41] A lot of this, we never charge a cent for and something I would like to make money on the smart things that we do in tech.  [00:45:47] Taking an under specified spec, which most of the times is intentionally under specified, and then building all the hard parts around it, is a very interesting formula for success. [00:45:58] So essentially React and under specified framework and Next.js came in and went like, oh, okay, well, we'll build the get initial props that you guys forgot. And great, very successful Gatsby, same thing. And then Apollo and Relay by the way, but, but relay was not a serious company, a company effort. [00:46:19] I mean, Relay is a serious effort. It's not a startup that was like existentially relying on like, uh unsuccess. Whereas was Apollo was like, okay, GraphQL was under specified. There's a reference JS implementation, but no one's building the production quality standard. We'll do it. And then, and yeah, like it's really interesting. Cause as the spec grows or as adoption of the thing grows, you're you grow with it and, you serve the audience and you also capture a lot of the value and you essentially have Facebook working for you in the sense of like, oh, there's the spec maintainers, you know, whatever, whatever the spec is, they're working for you because every time they contribute to the spec, you. [00:47:06] TMA: Too Many Acronyms  [00:47:06] Abhi Aiyer: Yeah, maybe that's what the what's going to happen with DPR. Right?  [00:47:10] swyx: The naming socks, too many, three letter acronyms. I'm sure. Like, look like you and I, and everyone in like the WebDev, like Twitter sphere or whatever, we don't mind new things and like understanding the differences in nuances, but anyone who is like just a regular web dev or just like not web dev, but talking to web devs, they think we're crazy. [00:47:36] This is actually bad. Like it, we look like the nerds, uh, who. Talking about all these minor differences and inventing new acronyms for them. I don't know how to fix it. Jargon is important for specialists to understand in a very short amount of time, the differences between what we referring to. Jargon is important, but we don't do ourselves, our industry a favor when we have all these acronyms and then people just throw them on onto a page or a blogpost or a slide deck. [00:48:05] And then. People would just go like, okay. Yeah, the JS ecosystem  [00:48:09] Abhi Aiyer: is crazy. And you ended up explaining the same thing all the time. Right? Cause you use some acronym. It was funny, like on the way to Gatsby camp, like we had, like all of our release had all of the releases and gas before had the acronym. Yeah, like PQR parallel query, running DSE, SSR, SSG, man. [00:48:26] We were like trying to figure it out. How many more acronyms can we fit to, to get like the, the acronym count up, but it's a serious problem for us too, because our, some of our customers have never used Gatsby before they're coming from a WordPress full on WordPress background and our sales team marketing, we all need to be able to convey like, yeah, this is what it really is. [00:48:45] And this is what it means. And maybe. The acronym sticks after they understand it, but that's a really uphill battle to explain right on the way. So I would love if a community we all got together and like, kind of just understood it. You know, it's kind of like the GraphQL spec have a formal definition for what this is. [00:49:02] Don't be too heavy handed on approach, let people implement however they want to. And then there's just a concept that has different flavors. Yeah. Oh, it's different  [00:49:14] swyx: flavors. Okay. That'd be interesting.  [00:49:16] Gatsby for Docs  [00:49:16] swyx: Is there anything else that we haven't covered that you wanted to shout out?  [00:49:21] Abhi Aiyer: This is fun. I really enjoyed talking to you too.  [00:49:24] swyx: Yeah, I love, uh, I love catching up. Um, uh, Fun fact, we're actually at my workplace. We use Docusaurus right now for our docs. We're actually considering moving to Gatsby. [00:49:35] Nice. Not something I thought I would do this year, but we're, we're running into enough limitations to Docusaurus that we're essentially customizing so much that we don't get much benefit anymore. So maybe a good standard docs implementation. It would be interesting for you guys actually, because a lot of the reason that people pick Docusaurus is basically it has docs in the name and it's got a lot of good defaults for docs, right? [00:50:04] And Gatsby, maybe it doesn't have such a developed theme for docs.  [00:50:07] Ward Peeters: We've mostly pushed people to the Apollo team. Like they have a great, like the whole Apolo site is, or docs site is built with Gatsby and a open source. The building blocks up there. So, or you could start from there and then, oh  [00:50:20] Abhi Aiyer: yeah.  [00:50:23] New Relic is with Gatsby and they're working on something similar too.  [00:50:30] swyx: Awesome. Awesome. Yeah. All right. Cool. Well thanks for those pointers. I'm actually going to go explore them. [00:50:38] Abhi Aiyer: Yeah. If you need any help. Yeah, we'll do.  [00:50:41] swyx: And there's no reason why we shouldn't move to Gatsby cloud, if that makes sense for us as well. Okay. Okay.  [00:50:47] Ward and Abhi,thanks so much, and this is really great chatting, thanks for reaching out. And, yeah, I hope  [00:50:52] Abhi Aiyer: people would try out Gatsby. [00:50:54] Thanks for having us.

Screaming in the Cloud
Keeping the Cloudwatch with Ewere Diagboya

Screaming in the Cloud

Play Episode Listen Later Oct 14, 2021 32:21


About EwereCloud, DevOps Engineer, Blogger and AuthorLinks: Infrastructure Monitoring with Amazon CloudWatch: https://www.amazon.com/Infrastructure-Monitoring-Amazon-CloudWatch-infrastructure-ebook/dp/B08YS2PYKJ LinkedIn: https://www.linkedin.com/in/ewere/ Twitter: https://twitter.com/nimboya Medium: https://medium.com/@nimboya My Cloud Series: https://mycloudseries.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate: is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards, while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other, which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at Honeycomb.io/screaminginthecloud. Observability, it's more than just hipster monitoring.Corey: This episode is sponsored in part by Liquibase. If you're anything like me, you've screwed up the database part of a deployment so severely that you've been banned from touching every anything that remotely sounds like SQL, at at least three different companies. We've mostly got code deployments solved for, but when it comes to databases we basically rely on desperate hope, with a roll back plan of keeping our resumes up to date. It doesn't have to be that way. Meet Liquibase. It is both an open source project and a commercial offering. Liquibase lets you track, modify, and automate database schema changes across almost any database, with guardrails to ensure you'll still have a company left after you deploy the change. No matter where your database lives, Liquibase can help you solve your database deployment issues. Check them out today at liquibase.com. Offer does not apply to Route 53.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I periodically make observations that monitoring cloud resources has changed somewhat since I first got started in the world of monitoring. My experience goes back to the original Call of Duty. That's right: Nagios.When you set instances up, it would theoretically tell you when they were unreachable or certain thresholds didn't work. It was janky but it kind of worked, and that was sort of the best we have. The world has progressed as cloud has become more complicated, as technologies have become more sophisticated, and here today to talk about this is the first AWS Hero from Africa and author of a brand new book, Ewere Diagboya. Thank you for joining me.Ewere: Thanks for the opportunity.Corey: So, you recently published a book on CloudWatch. To my understanding, it is the first such book that goes in-depth with not just how to wind up using it, but how to contextualize it as well. How did it come to be, I guess is my first question?Ewere: Yes, thanks a lot, Corey. The name of the book is Infrastructure Monitoring with Amazon CloudWatch, and the book came to be from the concept of looking at the ecosystem of AWS cloud computing and we saw that a lot of the things around cloud—I mostly talked about—most of this is [unintelligible 00:01:49] compute part of AWS, which is EC2, the containers, and all that, you find books on all those topics. They are all proliferated all over the internet, you know, and videos and all that.But there is a core behind each of these services that no one actually talks about and amplifies, which is the monitoring part, which helps you to understand what is going on with the system. I mean, knowing what is going on with the system helps you to understand failures, helps you to predict issues, helps you to also envisage when a failure is going to happen so that you can remedy it and also [unintelligible 00:02:19], and in some cases, even give you a historical view of the system to help you understand how a system has behaved over a period of time.Corey: One of the articles that I put out that first really put me on AWS's radar, for better or worse, was something that I was commissioned to write for Linux Journal, back when that was a print publication. And I accidentally wound up getting the cover of it with my article, “CloudWatch is of the devil, but I must use it.” And it was a painful problem that people generally found resonated with them because no one felt they really understood CloudWatch; it was incredibly expensive; it didn't really seem like it was at all intuitive, or that there was any good way to opt out of it, it was just simply there, and if you were going to be monitoring your system in a cloud environment—which of course you should be—it was just sort of the cost of doing business that you then have to pay for a third-party tool to wind up using the CloudWatch metrics that it was gathering, and it was just expensive and unpleasant all around. Now, a lot of the criticisms I put about CloudWatch's limitations in those days, about four years ago, have largely been resolved or at least mitigated in different ways. But is CloudWatch still crappy, I guess, is my question?Ewere: Um, yeah. So, at the moment, I think, like you said, CloudWatch has really evolved over time. I personally also had that issue with CloudWatch when I started using CloudWatch; I had the challenge of usability, I had the challenge of proper integration, and I will talk about my first experience with CloudWatch here. So, when I started my infrastructure work, one of the things I was doing a lot was EC2, basically. I mean, everyone always starts with EC2 at the first time.And then we had a downtime. And then my CTO says, “Okay, [Ewere 00:04:00], check what's going on.” And I'm like, “How do I check?” [laugh]. I mean, I had no idea of what to do.And he says, “Okay, there's a tool called CloudWatch. You should be able to monitor.” And I'm like, “Okay.” I dive into CloudWatch, and boom, I'm confused again. And you look at the console, you see, it shows you certain metrics, and yet [people 00:04:18] don't understand what CPU metric talks about, what does network bandwidth talks about?And here I am trying to dig, and dig, and dig deeper, and I still don't get [laugh] a sense of what is actually going on. But what I needed to find out was, I mean, what was wrong with the memory of the system, so I delved into trying to install the CloudWatch agent, get metrics and all that. But the truth of the matter was that I couldn't really solve my problem very well, but I had [unintelligible 00:04:43] of knowing that I don't have memory out of the box; it's something that has to set up differently. And trust me, after then I didn't touch CloudWatch [laugh] again. Because, like you said, it was a problem, it was a bit difficult to work with.But fast forward a couple of years later, I could actually see someone use CloudWatch for a lot of beautiful stuff, you know? It creates beautiful dashboards, creates some very well-aggregated metrics. And also with the aggregated alarms that CloudWatch comes with, [unintelligible 00:05:12] easy for you to avoid what to call incident fatigue. And then also, the dashboards. I mean, there are so many dashboards that simplified to work with, and it makes it easy and straightforward to configure.So, the bootstrapping and the changes and the improvements on CloudWatch over time has made CloudWatch a go-to tool, and most especially the integration with containers and Kubernetes. I mean, CloudWatch is one of the easiest tools to integrate with EKS, Kubernetes, or other container services that run in AWS; it's just, more or less, one or two lines of setup, and here you go with a lot of beautiful, interesting, and insightful metrics that you will not get out of the box, and if you look at other monitoring tools, it takes a lot of time for you to set up, for you to configure, for you to consistently maintain and to give you those consistent metrics you need to know what's going on with your system from time to time.Corey: The problem I always ran into was that the traditional tools that I was used to using in data centers worked pretty well because you didn't have a whole lot of variability on an hour-to-hour basis. Sure, when you installed new servers or brought up new virtual machines, you had to update the monitoring system. But then you started getting into this world of ephemerality with auto-scaling originally, and later containers, and—God help us all—Lambda now, where it becomes this very strange back-and-forth story of, you need to be able to build something that, I guess, is responsive to that. And there's no good way to get access to some of the things that CloudWatch provides, just because we didn't have access into AWS's systems the way that they do. The inverse, though, is that they don't have access into things running inside of the hypervisor; a classic example has always been memory: memory usage is an example of something that hasn't been able to be displayed traditionally without installing some sort of agent inside of it. Is that still the case? Are there better ways of addressing those things now?Ewere: So, that's still the case, I mean, for EC2 instances. So before, now, we had an agent called a CloudWatch agent. Now, there's a new agent called Unified Cloudwatch Agent which is, I mean, a top-notch from CloudWatch agent. So, at the moment, basically, that's what happens on the EC2 layer. But the good thing is when you're working with containers, or more or less Kubernetes kind of applications or systems, everything comes out of the box.So, with containers, we're talking about a [laugh] lot of moving parts. The container themselves with their own CPU, memory, disk, all the metrics, and then the nodes—or the EC2 instance of the virtual machines running behind them—also having their own unique metrics. So, within the container world, these things are just a click of a button. Everything happens at the same time as a single entity, but within the EC2 instance and ecosystem, you still find this there, although the setup process has been a bit easier and much faster. But in the container world, that problem has totally been eliminated.Corey: When you take a look at someone who's just starting to get a glimmer of awareness around what CloudWatch is and how to contextualize it, what are the most common mistakes people make early on?Ewere: I also talked about this in my book, and one of the mistakes people make in terms of CloudWatch, and monitoring in generalities: “What am I trying to figure out?” [laugh]. If you don't have that answer clearly stated, you're going to run into a lot of problems. You need to answer that question of, “What am I trying to figure out?” I mean, monitoring is so broad, monitoring is so large that if you do not have the answer to that question, you're going to get yourself into a lot of trouble, you're going to get yourself into a lot of confusion, and like I said, if you don't understand what you're trying to figure out in the first place, then you're going to get a lot of data, you're going to get a lot of information, and that can get you confused.And I also talked about what I call alarm fatigues or incident fatigues. This happens when you configure so many alarms, so many metrics, and you're getting a lot of alarms hitting and notification services—whether it's Slack, whether it's an email—and it causes fatigue. What happens here is the person who should know what is going on with the system gets a ton of messages and in that scenario can miss something very important because there's so many messages coming in, so many integrations coming in. So, you should be able to optimize appropriately, to be able to, like you said, conceptualize what you're trying to figure out, what problems are you trying to solve? Most times you really don't figure this out for a start, but there are certain bare minimums you need to know about, and that's part of what I talked about in the book.One of the things that I highlighted in the book when I talked about monitoring of different layers is, when you're talking about monitoring of infrastructure, say compute services, such as virtual machines, or EC2 instances, the certain baseline and metrics you need to take note of that are core to the reliability, the scalability, and the efficiency of your system. And if you focus on these things, you can have a baseline starting point before you start going deeper into things like observability and knowing what's going on entirely with your system. So, baseline understanding of—baseline metrics, and baseline of what you need to check in terms of different kinds of services you're trying to monitor is your starting point. And the mistake people make is that they don't have a baseline. So, we do not have a baseline; they just install a monitoring tool, configure a CloudWatch, and they don't know the problem they're trying to solve [laugh] and that can lead to a lot of confusion.Corey: So, what inspired you from, I guess, kicking the tires on CloudWatch—the way that we all do—and being frustrated and confused by it, all the way to the other side of writing a book on it? What was it that got you to that point? Were you an expert on CloudWatch before you started writing the book, or was it, “Well, by the time this book is done, I will certainly know [laugh] more about the service than I did when I started.”Ewere: Yeah, I think it's a double-edged sword. [laugh]. So, it's a combination of the things you just said. So, first of all, I have experienced with other monitoring tools; I have love for reliability and scalability of a system. I started Kubernetes at some of the early times Kubernetes came out, when it was very difficult to deploy, when it was very difficult to set up.Because I'm looking at how I can make systems a little bit more efficient, a little bit more reliable than having to handle a lot of things like auto-scaling, having to go through the process of understanding how to scale. I mean, that's a school of its own that you need to prepare yourself for. So, first of all, I have a love for making sure systems are reliable and efficient, and second of all, I also want to make sure that I know what is going on with my system per time, as much as possible. The level of visibility of a system gives you the level of control and understanding of what your system is doing per time. So, those two things are very core to me.And then thirdly, I had a plan of a streak of books I want to write based on AWS, and just like monitoring is something that is just new. I mean, if you go to the package website, this is the first book on infrastructure monitoring AWS with CloudWatch; it's not a very common topic to talk about. And I have other topics in my head, and I really want to talk about things like networking, and other topics that you really need to go deep inside to be able to appreciate the value of what you see in there with all those scenarios because in this book, every chapter, I created a scenario of what a real-life monitoring system or what you need to do looks like. So, being that I have those premonitions, I know that whenever it came to, you know, to share with the world what I know in monitoring, what I've learned in monitoring, I took a [unintelligible 00:12:26]. And then secondly, as this opportunity for me to start telling the world about the things I learned, and then I also learned while writing the book because there are certain topics in the book that I'm not so much of an expert in things, like big data and all that.I had to also learn; I had to take some time to do more research, to do more understanding. So, I use CloudWatch, okay? I'm kind of good in CloudWatch, and also, I also had to do more learning to be able to disseminate this information. And also, hopefully, X-Ray some parts of monitoring and different services that people do not really pay so much attention into.Corey: What do you find that is still the most, I guess, confusing to you as you take a look across the ecosystem of the entire CloudWatch space? I mean, every time I play with it, I take a look, and I get lost in, “Oh, they have contributor analyses, and logs, and metrics.” And it's confusing, and every time I wind up, I guess, spiraling out of control. What do you find that, after all of this, is a lot easier for you, and what do you find that's a lot more understandable?Ewere: I'm still going to go back to the containers part. I'm sorry, I'm in love containers. [laugh].Corey: No, no, it's fair. Containers are very popular. Everyone loves them. I'm just basically anti-container based upon no better reason than I'm just stubborn and bloody-minded most of the time.Ewere: [laugh]. So, pretty much like I said, I kind of had experience with other monitoring tools. Trust me, if you want to configure proper container monitoring for other tools, trust me, it's going to take you at least a week or two to get it properly, from the dashboards, to the login configurations, to the piping of the data to the proper storage engine. These are things I talked about in the book because I took monitoring from the ground up. I mean, if you've never done monitoring before, when you take my book, you will understand the basic principles of monitoring.And [funny 00:14:15], you know, monitoring has some big data process, like an ETL process: extraction, transformation, and writing of data into an analytic system. So, first of all, you have to battle that. You have to talk about the availability of your storage engine. What are you using? An Elasticsearch? Are you using an InfluxDB? Where do you want to store your data? And then you have to answer the question of how do I visualize the data? What method do I realize this data? What kind of dashboards do I want to use? What methods of representation do I need to represent this data so that it makes sense to whoever I'm sharing this data with. Because in monitoring, you definitely have to share data with either yourself or with someone else, so the way you present the data needs to make sense. I've seen graphs that do not make sense. So, it requires some level of skill. Like I said, I've [unintelligible 00:15:01] where I spent a week or two having to set up dashboards. And then after setting up the dashboard, someone was like, “I don't understand, and we just need, like, two.” And I'm like, “Really?” [laugh]. You know? Because you spend so much time. And secondly, you discover that repeatability of that process is a problem. Because some of these tools are click and drag; some of them don't have JSON configuration. Some do, some don't. So, you discover that scalability of this kind of system becomes a problem. You can't repeat the dashboards: if you make a change to the system, you need to go back to your dashboard, you need to make some changes, you need to update your login, too, you need to make some changes across the layer. So, all these things is a lot of overhead [laugh] that you can cut off when you use things like Container Insights in CloudWatch—which is a feature of CloudWatch. So, for me, that's a part that you can really, really suck out so much juice from in a very short time, quickly and very efficiently. On the flip side, when you talk about monitoring for big data services, and monitoring for a little bit of serverless, there might be a little steepness in the flow of the learning curve there because if you do not have a good foundation in serverless, when you get into [laugh] Lambda Insights in CloudWatch, trust me, you're going to be put off by that; you're going to get a little bit confused. And then there's also multifunction insights at the moment. So, you need to have some very good, solid foundation in some of those topics before you can get in there and understand some of the data and the metrics that CloudWatch is presenting to you. And then lastly, things like big data, too, there are things that monitoring is still being properly fleshed out. Which I think that in the coming months and years to come, they will become more proper and they will become more presentable than they are at the moment.Corey: This episode is sponsored by our friends at Oracle HeatWave is a new high-performance accelerator for the Oracle MySQL Database Service. Although I insist on calling it “my squirrel.” While MySQL has long been the worlds most popular open source database, shifting from transacting to analytics required way too much overhead and, ya know, work. With HeatWave you can run your OLTP and OLAP, don't ask me to ever say those acronyms again, workloads directly from your MySQL database and eliminate the time consuming data movement and integration work, while also performing 1100X faster than Amazon Aurora, and 2.5X faster than Amazon Redshift, at a third of the cost. My thanks again to Oracle Cloud for sponsoring this ridiculous nonsense.Corey: The problem I've always had with dashboards is it seems like managers always want them—“More dashboards, more dashboards”—then you check the usage statistics of who's actually been viewing the dashboards and the answer is, no one since you demoed it to the execs eight months ago. But they always claim to want more. How do you square that?I guess, slicing between what people asked for and what they actually use.Ewere: [laugh]. So yeah, one of the interesting things about dashboards in terms of most especially infrastructure monitoring, is the dashboards people really want is a revenue dashboards. Trust me, that's what they want to see; they want to see the money going up, up, up, [laugh] you know? So, when it comes to—Corey: Oh, yes. Up and to the right, then everyone's happy. But CloudWatch tends to give you just very, very granular, low-level metrics of thing—it's hard to turn that into something executives care about.Ewere: Yeah, what people really care about. But my own take on that is, the dashboards are actually for you and your team to watch, to know what's going on from time to time. But what is key is setting up events across very specific and sensitive data. For example, when any kind of sensitive data is flowing across your system and you need to check that out, then you tie a metric to that, and in turn alarm to it. That is actually the most important thing for anybody.I mean, for the dashboards, it's just for you and your team, like I said, for your personal consumption. “Oh, I can see all the RDS connections are getting too high, we need to upgrade.” Oh, we can see that all, the memory, there was a memory spike in the last two hours. I know that's for you and your team to consume; not for the executive team. But what is really good is being able to do things like aggregate data that you can share.I think that is what the executive team would love to see. When you go back to the core principles of DevOps in terms of the DevOps Handbook, you see things like a mean time to recover, and change failure rate, and all that. The most interesting thing is that all these metrics can be measured only by monitoring. You cannot change failure rates if you don't have a monitoring system that tells you when there was a failure. You cannot know your release frequency when you don't have a metric that measures number of deployments you have and is audited in a particular metric or a particular aggregator system.So, we discovered that the four major things you measure in DevOps are all tied back to monitoring and metrics, at minimum, to understand your system from time to time. So, what the executive team actually needs is to get a summary of what's going on. And one of the things I usually do for almost any company I work for is to share some kind of uptime system with them. And that's where CloudWatch Synthetics Canary come in. So, Synthetic Canary is a service that helps you calculate that helps you check for uptime of the system.So, it's a very simple service. It does a ping, but it is so efficient, and it is so powerful. How is it powerful? It does a ping to a system and it gets a feedback. Now, if the status code of your service, it's not 200 or not 300, it considers it downtime.Now, when you aggregate this data within a period of time, say a month or two, you can actually use that data to calculate the uptime of your system. And that uptime [unintelligible 00:19:50] is something you can actually share to your customers and say, “Okay, we have an SLA of 99.9%. We have an SLA of 99.8%.” That data should not be doctored data; it should not be a data you just cook out of your head; it should be based on your system that you have used, worked with, monitored over a period of time so that the information you share with your customers are genuine, they are truthful, and they are something that they can also see for themselves.Hence companies are using [unintelligible 00:20:19] like status page to know what's going on from time to time whenever there is an incident and report back to their customers. So, these are things that executives will be more interested in than just dashboards, [laugh] dashboards, and more dashboards. So, it's more or less not about what they really ask for, but what you know and what you believe you are going to draw value from. I mean, an executive in a meeting with a client and says, “Hey, we got a system that has 99.9% uptime.”He opens the dashboard or he opens the uptime system and say, “You see our uptime? For the past three months, this has been our metric.” Boom. [snaps fingers]. That's it. That's value, instantly. I'm not showing [laugh] the clients and point of graphs, you know? “Can you explain the memory metric?” That's not going to pass the message, send the message forward.Corey: Since your book came out, I believe, if not, certainly by the time it was finished being written and it was in review phase, they came out with Managed Prometheus and Managed Grafana. It looks almost like they're almost trying to do a completely separate standalone monitoring stack of AWS tooling. Is that a misunderstanding of what the tools look like, or is there something to that?Ewere: Yeah. So, I mean by the time those announced at re:Invent, I'm like, “Oh, snap.” I almost told my publisher, “You know what? We need to add three more chapters.” [laugh]. But unfortunately, we're still in review, in preview.I mean, as a Hero, I kind of have some privilege to be able to—a request for that, but I'm like, okay, I think it's going to change the narrative of what the book is talking about. I think I'm going to pause on that and make sure this finishes with the [unintelligible 00:21:52], and then maybe a second edition, I can always attach that. But hey, I think there's trying to be a galvanization between Prometheus, Grafana, and what CloudWatch stands for. Because at the moment, I think it's currently on pre-release, it's not fully GA at the moment, so you can actually use it. So, if you go to Container Insights, you can see that you can still get how Prometheus and Grafana is presenting the data.So, it's more or less a different view of what you're trying to see. It's trying to give you another perspective of how your data is presented. So, you're going to have CloudWatch: it's going to have CloudWatch dashboards, it's going to have CloudWatch metrics, but hey, this different tools, Prometheus, Grafana, and all that, they all have their unique ways of presenting the data. And part of the reason I believe AWS has Prometheus and Grafana there is, I mean, Prometheus is a huge cloud-native open-source monitoring, presentation, analytics tool; it packs a lot of heat, and a lot of people are so used to it. Everybody like, “Why can't I have Prometheus in CloudWatch?”I mean—so instead of CloudWatch just being a simple monitoring tool, [unintelligible 00:22:54] CloudWatch has become an ecosystem of monitoring tool. So, we got—we're not going to see cloud [unintelligible 00:23:00], or just [unintelligible 00:23:00] log, analytics, metrics, dashboards, no. We're going to see it as an ecosystem where we can plug in other services, and then integrate and work together to give us better performance options, and also different perspectives to the data that is being collected.Corey: What do you think is next, as you take a look across the ecosystem, as far as how people are thinking about monitoring and observability in a cloud context? What are they missing? Where's the next evolution lead?Ewere: Yeah, I think the biggest problem with monitoring, which is part of the introduction part of the book, where I talked about the basic types of monitoring—which is proactive and reactive monitoring—is how do we make sure we know before things happen? [laugh]. And one of the things that can help with that is machine learning. There is a small ecosystem that is not so popular at the moment, which talks about how we can do a lot of machine learning in DevOps monitoring observability. And that means looking at historic data and being able to predict on the basic level.Looking at history, [then are 00:24:06] being able to predict. At the moment, there are very few tools that have models running at the back of the data being collected for monitoring and metrics, which could actually revolutionize monitoring and observability as we see it right now. I mean, even the topic of observability is still new at the moment. It's still very integrated. Observability just came into Cloud, I think, like, two years ago, so it's still being matured.But one thing that has been missing is seeing the value AI can bring into monitoring. I mean, this much [unintelligible 00:24:40] practically tell us, “Hey, by 9 p.m. I'm going to go down. I think your CPU or memory is going down. I think I'm line 14 of your code [laugh] is a problem causing the bug. Please, you need to fix it by 2 p.m. so that by 6 p.m., things can run perfectly.” That is going to revolutionize monitoring. That's going to revolutionize observability and bring a whole new level to how we understand and monitor the systems.Corey: I hope you're right. If you take a look right now, I guess, the schism between monitoring and observability—which I consider to be hipster monitoring, but they get mad when I say that—is there a difference? Is it just new phrasing to describe the same concepts, or is there something really new here?Ewere: In my book, I said, monitoring is looking at it from the outside in, observability is looking at it from the inside out. So, what monitoring does not see under, basically, observability sees. So, they are children of the same mom. That's how I put it. One actually needs the other and both of them cannot be separated from each other.What we've been working with is just understanding the system from the surface. When there's an issue, we go to the aggregated results that come out of the issue. Very basic example: you're in a Java application, and we all know Java is very memory intensive, on the very basic layer. And there's a memory issue. Most times, infrastructure is the first hit with the resultant of that.But the problem is not the infrastructure, it's maybe the code. Maybe garbage collection was not well managed; maybe they have a lot of variables in the code that is not used, and they're just filling up unnecessary memory locations; maybe there's a loop that's not properly managed and properly optimized; maybe there's a resource on objects that has been initialized that has not been closed, which will cause a heap in the memory. So, those are the things observability can help you track. Those are the things that we can help you see. Because observability runs from within the system and send metrics out, while basic monitoring is about understanding what is going on on the surface of the system: memory, CPU, pushing out logs to know what's going on and all that.So, on the basic level, observability helps gives you, kind of, a deeper insight into what monitoring is actually telling you. It's just like the result of what happened. I mean, we are told that the symptoms of COVID is coughing, sneezing, and all that. That's monitoring. [laugh].But before we know that you actually have COVID, we need to go for a test, and that's observability. Telling us what is causing the sneezing, what is causing the coughing, what is causing the nausea, all the symptoms that come out of what monitoring is saying. Monitoring is saying, “You have a cough, you have a runny nose, you're sneezing.” That is monitoring. Observability says, “There is a COVID virus in the bloodstream. We need to fix it.” So, that's how both of them act.Corey: I think that is probably the most concise and clear definition I've ever gotten on the topic. If people want to learn more about what you're up to, how you view about these things—and of course, if they want to buy your book, we will include a link to that in the [show notes 00:27:40]—where can they find you?Ewere: I'm on LinkedIn; I'm very active on LinkedIn, and I also shared the LinkedIn link. I'm very active on Twitter, too. I tweet once in a while, but definitely, when you send me a message on Twitter, I'm also going to be very active.I also write blogs on Medium, I write a couple of blogs on Medium, and that was part of why AWS recognized me as a Hero because I talk a lot about different services, I help with comparing services for you so you can choose better. I also talk about setting basic concepts, too; if you just want to get your foot wet into some stuff and you need something very summarized, not AWS documentation per se, something that you can just look at and know what you need to do with the service, I talk about them also in my blogs. So yeah, those are the two basic places I'm in: LinkedIn and Twitter.Corey: And we will, of course, put links to that in the [show notes 00:28:27]. Thank you so much for taking the time to speak with me. I appreciate it.Ewere: Thanks a lot.Corey: Ewere Diagboya, head of cloud at My Cloud Series. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you hated this podcast, please leave a five-star review on your podcast platform of choice along with a comment telling me how many more dashboards you would like me to build that you will never look at.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

The Cloud Pod
138: Cloud Pod productivity is way up thanks to the Facebook outage

The Cloud Pod

Play Episode Listen Later Oct 14, 2021 66:32


On The Cloud Pod this week, the team is running at half-duplex without Peter and Ryan. Plus Cloudflare R2 is here, Facebook died for a day, and AWS releases Cloud Control Plane.  A big thanks to this week's sponsors: Foghorn Consulting, which provides full-stack cloud solutions with a focus on strategy, planning and execution for enterprises seeking to take advantage of the transformative capabilities of AWS, Google Cloud and Azure. JumpCloud, which offers a complete platform for identity, access, and device management — no matter where your users and devices are located.  This week's highlights

Serverless Chats
Episode #114: Serverless for Salary Transparency with Kesha Williams

Serverless Chats

Play Episode Listen Later Oct 11, 2021 44:56


Kesha Williams is an award-winning software engineer and technology leader teaching others how to transform their lives through technology. Forbes, Amazon Web Services (AWS), and Oracle have applauded her contributions to the technology community, and she has spoken on the TED stage about the transformative power of artificial intelligence (AI). Amazon recognized her pioneering work in AI with both its AWS Machine Learning Hero and Alexa Champion honors — the first person to receive both. Williams was named Mentor of the Year by Women Tech Network and received the Innovator Award from Hospitality Technology. She has launched several successful startups and appears in the 2020 tech documentary, "Hello World: The Film." Additionally, Williams serves on the Board of Directors for Women in Voice and as a mentor to women in tech.  Twitter: @KeshaWillz Blog: kesha.tech/ LinkedIn: linkedin.com/in/java-rock-star-kesha/ Salary Overflow: salaryoverflow.com

AWS Morning Brief
DNSSEC Inspired Outages

AWS Morning Brief

Play Episode Listen Later Oct 7, 2021 8:16


Links: Let's Encrypt's root certificate has expired, and it might break your devices: https://techcrunch.com/2021/09/21/lets-encrypt-root-expiry/ Slack was bitten by DNSSEC: https://Twitter.com/tqbf/status/1443654964556013569 Prepare For Cybersecurity Assessments From Your Customers: https://www.securitysystemsnews.com/article/prepare-for-cybersecurity-assessments-from-your-customers AWS Lambda now supports triggering Lambda functions from an Amazon SQS queue in a different account: https://aws.amazon.com/about-aws/whats-new/2021/09/aws-lambda-lambda-function-amazon-sqs-queue/ Migrating custom Landing Zone with RAM to AWS Control Tower: https://aws.amazon.com/blogs/mt/migrating-custom-landing-zone-with-ram-to-aws-control-tower/ Introducing the Ransomware Risk Management on AWS Whitepaper: https://aws.amazon.com/blogs/security/introducing-the-ransomware-risk-management-on-aws-whitepaper/ Validate IAM policies in CloudFormation templates using IAM Access Analyzer: https://aws.amazon.com/blogs/security/validate-iam-policies-in-cloudformation-templates-using-iam-access-analyzer/ Pacu: The Open Source AWS Exploitation Framework: https://rhinosecuritylabs.com/aws/pacu-open-source-aws-exploitation-framework/ TranscriptCorey: This is the AWS Morning Brief: Security Edition. AWS is fond of saying security is job zero. That means it's nobody in particular's job, which means it falls to the rest of us. Just the news you need to know, none of the fluff.Corey: This episode is sponsored in part by Thinkst Canary. This might take a little bit to explain, so bear with me. I linked against an early version of their tool, canarytokens.org, in the very early days of my newsletter, and what it does is relatively simple and straightforward. It winds up embedding credentials, files, or anything else like that that you can generate in various parts of your environment, wherever you want them to live. It gives you fake AWS API credentials, for example, and the only thing that these things do is alert you whenever someone attempts to use them. It's an awesome approach to detecting breaches. I've used something similar for years myself before I found them. Check them out. But wait, there's more because they also have an enterprise option that you should be very much aware of: canary.tools. You can take a look at this, but what it does is it provides an enterprise approach to drive these things throughout your entire environment and manage them centrally. You can get a physical device that hangs out on your network and impersonates whatever you want to. When it gets Nmap scanned, or someone attempts to log into it, or access files that it presents on a fake file store, you get instant alerts. It's awesome. If you don't do something like this, instead you're likely to find out that you've gotten breached the very hard way. So, check it out. It's one of those few things that I look at and say, “Wow, that is an amazing idea. I am so glad I found them. I love it.” Again, those URLs are canarytokens.org and canary.tools. And the first one is free because of course it is. The second one is enterprise-y. You'll know which one of those you fall into. Take a look. I'm a big fan. More to come from Thinkst Canary in the weeks ahead.Corey: Somehow we made it through an entire week without a major vendor having a headline-level security breach. You know, I could get used to this; I'll take, “It's harder for me to figure out what to talk about here,” over, “A bunch of customers are scrambling because their providers have failed them,” every time.So, let's see what the community had to say. Last week, as you're probably aware, Let's Encrypt's root certificate expiredwhich caused pain for a bunch of folks. Any device or configuration that hadn't been updated for a few years is potentially going to see things breaking. The lesson here is to be aware that certificates do expire. The antipattern is to do super-long registrations for thing, but that just makes it worse.One of the things Let's Encrypt got very right is forcing 90-day certificate rotations for client certs. When you've got to do that every three months, you know where all of your certificates are. If you've got to replace it once every ten years, you'll have no clue; that was six employees ago.In bad week news, Slack was bitten by DNSSEC when they attempted and failed to roll it out. DNSSEC is a bag of pain it's best not to bother with, as a general rule. DNS is always a bag of pain because of caching and TTL issues. In effect, Slack tried to roll out DNSSEC—probably due to a demand by some big corporate customer—had it fail, panicked and rolled back the change, and was in turn bitten by outages as a bunch of DNS resolvers had the DS key cached, but the authoritative nameservers stopped publishing it. This is a mess and a great warning to those of us who might naively assume that anything like DNSSEC that offers improved security comes without severe tradeoffs. Measure twice, cut once because mistakes are going to show.I also found a somewhat alarmist article talking about cybersecurity assessments from your customers and fine, but it brings up a good point. If you're somehow responsible for security but don't have security in your job title—which, you know, this show is aimed at—you may one day be surprised to have someone from sales pop up and ask you to fill out a form from a prospective customer. Ignore the alarm and the panic but you're going to want to get towards something approaching standardization around how you handle those.The first time you get one of these, it's a novel exercise; by the tenth, you just want to have a prepared statement you can hand them so you can move on with things. Well, those prepared statements are often called things like, “SOC 2 certifications.” There's a spectrum and where you fall on it depends upon who you work for and what you do. So, take them seriously and don't be surprised when you get one.AWS had a few interesting security-related announcements. AWS Lambda now supports triggering Lambda functions from an Amazon SQS queue in a different account. That doesn't sound like a security announcement, so why am I talking about it? Because until recently, it wasn't possible so a lot of folks scoped their IAM policies very broadly; what do you care if any random SQS queue in your own account can invoke a Lambda? With this change, suddenly internet randos can invoke Lambda functions, and you should probably go check production immediately.Announcer: Have you implemented industry best practices for securely accessing SSH servers, databases, or Kubernetes? It takes time and expertise to set up. Teleport makes it easy. It is an identity-aware access proxy that brings automatically expiring credentials for everything you need, including role-based access controls, access requests, and the audit log. It helps prevent data exfiltration and helps implement PCI and FedRAMP compliance. And best of all, Teleport is open-source and a pleasure to use. Download Teleport at goteleport.com. That's goteleport.com.Corey: Migrating custom Landing Zone with RAM to AWS Control Tower. It's worth considering the concept here because, “Using the polished thing” is usually better than building and then maintaining something yourself. You wind up off in the wilderness; then AWS shows up and acts befuddled, “Why on earth would you build things the way that we told you to build them at the time you set up your environment?” It's obnoxious and they need to stop talking and own their mistakes, but keeping things current with the accepted way of doing things is usually worth at least considering.AWS has a whitepaper on Ransomware Risk Management out and I'm honestly conflicted about it. There are gems but it talks about a pile of different services they offer to offset the risk. Some of them—like AWS Backup—are great.Others—“Use Systems Manager State Manager”—present as product pitches for products of varying quality and low adoption. On balance, it's worth reading but retain a healthy skepticism if you do. It should be noted that the points that the address and the framework they lay out is exactly how risk management folks think, and that's helpful.Validate IAM policies in CloudFormation templates using IAM Access Analyzer. I like that one quite a bit. It does what it says on the tin, and applies a bunch of more advanced linting rules than you'd find in something like cfn-lint.Note that this costs nothing for a change, even though it does communicate with AWS to run its analysis. Note that as AWS improves the Access Analyzer, findings will likely change, so be aware that this may well result in a regression should you have it installed as part of a CI/CD pipeline.And as far as tools go, if you're not a security researcher, good; you're in the right place. But that said, if you have a spare afternoon at some point, you may want to check out Pacu—that's P-A-C-U. It's an open-source AWS exploitation framework that lets you see just how insecure your AWS accounts might be. I generally leave playing with those sorts of things to security professionals, but this is a fun way to just take a quick check and see if there's a burning fire that jumps out that might arise for you down the road. And I'll talk to you more about all this stuff next week.Corey: I have been your host, Corey Quinn, and if you remember nothing else, it's that when you don't get what you want, you get experience instead. Let my experience guide you with the things you need to know in the AWS security world, so you can get back to doing your actual job. Thank you for listening to the AWS Morning Brief: Security Edition.Announcer: This has been a HumblePod production. Stay humble.

The Thing About Cars
Celebrating Pride with the Lambda Car Club

The Thing About Cars

Play Episode Listen Later Oct 5, 2021 29:29


Happy Pride! We record this show on the 50th anniversary of the first Pride Parade in Atlanta. We celebrate Pride Dave introduces us to the Lambda Car Club, the largest club for LGBTQ+ car enthusiasts in the country. Ben recounts his visit to the Worship Cruise In where he learned about the Cops-In-Cars Car Show at the Roswell City Hall. Misty recounts a great day for the Dutch in Formula One! What grinds Mickey's gears this week? The fact that all cars should be hybrid by now. And we learn the identity of Volkswagen part number 199398500A. lccimembers.com #lcc-dogwood.org #RoswellPD #Cops-in-Cars-Car-Show We welcome your support via Patreon and your questions and feedback via our website. This podcast is powered by Pinecast.

No Sharding - The Solana Podcast
Jonathan Schemoul - Founder of Aleph.im Ep #48

No Sharding - The Solana Podcast

Play Episode Listen Later Oct 5, 2021 43:36


Anatoly Yakovenko (00:12):Hey folks, this is Anatoly, and you're listening to The Solana Podcast. And today I have Jonathan Schemoul with me, who's the founder of the Aleph.im project. Really awesome to have you.Jonathan Schemoul (00:22):Thank you very much. I'm really happy to be here today.Anatoly Yakovenko (00:25):Cool. We usually start these with a simple question, how did you get into crypto? What's your story? What's the origin story?Jonathan Schemoul (00:36):Well, into crypto it's a long story. I started way back in time, a bit on Bitcoin then I stopped because it was only money back then. And that wasn't the end game for me. Then I came back into crypto in 2015, 2016, and I started doing a bit of development because I saw that I really wanted to be part of Web 3, to do nice things with it. I started developing as an open-source developer for a few projects. One of these is the newest project which is Chinese blockchain layer one. I'm not really involved with it anymore.Jonathan Schemoul (01:16):But working with them as a community open source developer, I saw that there was some missing links somewhere that you couldn't decentralize all the stack with just layer one, it is not the one that they were building back then. So that's how the Aleph.im project is born. For me, besides that, I've been developing for a lot of companies before in the IOT space and also for big banks sometime ago. I've been a developer for a lot of years.Anatoly Yakovenko (01:48):That's great. I mean, that's a great background. The thing that you're focusing on with Aleph is this idea that Web 3 is just a small part of the piece, but you still need UI front-ends, business logic and things sitting on top of the blockchain. How does that work?Jonathan Schemoul (02:09):The idea is that, okay, now you can have smart contracts on Solana, that's great. You can even do way much more on like just money on smart contracts, that's great. Now, you need to have a front-end. So you need to have storage for that front-end. That's not all because a smart contract, a program doesn't have all the data that you need. So you will need some kind of indexing to get history. You will need a back-end for that.Jonathan Schemoul (02:37):Most of the DeFi application that we see have some centralized back-end behind them. They're running on AWS, sometimes on dedicated servers or stuff like that that is still centralized. If a government, and we just saw something about it today, wants to shut down the DeFi protocol that is organized like that, they can. With Aleph.im what we are trying to do is decentralize the last mile, because for that last mile most projects are using AWS, so we need to decentralize AWS.Jonathan Schemoul (03:11):So we provide storage, as in file storage for the front-end files, database storage, because most applications are just databases and also an equivalent to Amazon Lambda, where you start small functions that will be launched on a decentralized cloud, where there is place for them and will get you a return value, and these can be written in any language and connects the web and also a PC from blockchains here at Solana obviously.Anatoly Yakovenko (03:42):Got it. Super Cool. So this is a storage mechanism. Does it guarantee consistency? How's it decentralized? What happens if you nuke it? Yellowstone flows up, the current set of servers from Aleph get destroyed in the volcano. How do I move, switch, what state do I lose? Those are the hard distributed systems question.Jonathan Schemoul (04:08):Yeah. It's a really good question. Aleph.im is not a blockchain at all. We don't have a blockchain. There are enough already. We just accept messages from blockchains. All the supported blockchains are accepted on the network, that means that that message that is signed by a material address is accepted on network, a message that is signed by a certain address is accepted on the network. All our network, hence the name .im, dot instant messaging, the whole system works with messages on the network.Jonathan Schemoul (04:45):Those messages are organized by channels, just like you would go on telegram channels and get the history of them. The network keeps track of those messages and when you start a new node, you get the history of messages, not directly from the other nodes, you will connect two blockchains to specific smart contracts on blockchain. Look at past events, for example, on the Ethereum or on Solana. You look at past events for the synchronization of the network and you look, okay, there has been all these events, okay, let me ask the whole network what those messages were. Then you resync, when there are missing parts you leave them apart and then you get a view on the channels on the messages.Anatoly Yakovenko (05:31):So you write your software, your Lambda hook as if it's a re-entrant, right? So you're kind of recording your progress potentially on Solana as you're processing it.Jonathan Schemoul (05:43):For the Lambda it's a bit different. Here I was explaining how the network works for the messaging on the global state. For the state of pure application, you could either get your state from a blockchain here at Solana. For example, all the indexing effort that we are doing is using Solana as a source of synchronization for these Lambda. But then you can have multiple kind of volumes because since it's Linux Micro VM machine, everything is a volume.Jonathan Schemoul (06:18):So we have local storage volume that is local to the running host. And then the Lambda kind of issue messages on a decentralized database of data and project or under storage, and then raising to the local file system and then issue messaging, et cetera. And we are also working on another kind of phase system that is distributed, where any of them that can write in it on the overall receive the changes, which is kind of tricky.Anatoly Yakovenko (06:48):Is the database, the Aleph database, distributed database? Is that a Byzantine fault-tolerant database? Is it designed with that in mind?Jonathan Schemoul (06:58):Yeah. The idea is that when you send a message on the network, it gets stored by all the over nodes that are interested in your channel. And then there are synchronization node that go and write hashes of the data and signatures inside messages that they push on blockchains. So that when overcome, they can synchronize it and replicate all the data. So that even if one part of the network gets totally disconnected, you can have one part that gets reconnected to the other therefore the peer to peer network for blockchain, for APFS. We have multiple kind of different connectivity solutions so that they can reconnect on resync.Anatoly Yakovenko (07:42):So the Aleph database, if it's Byzantine fault tolerant, I mean, doesn't that make it a blockchain? Is there a token? Is crypto economically like fault tolerant?Jonathan Schemoul (07:56):Yeah. So we have a token, but the token is living on multiple blockchain, Ethereum, Solana, and a few others, but those are the most used today. We have a token, you need a token for your data to stay there. If you don't have any more your data gets garbage collected. But we don't have a blockchain because we go and write on over layer ones. We are technically a layer two database which is computing pre storage.Anatoly Yakovenko (08:23):But the data storage, like the Aleph distributed database, what is that backed by? Or can I pick my own blockchain to use it as a common interface or something like that?Jonathan Schemoul (08:34):Well, currently it writes on Ethereum, we're working on making it write on Solana. For this we need our indexer to be super powerful. So we'll get it writing on Solana very soon. Basically you can write on multiple blockchains and use it as a source of proof.Anatoly Yakovenko (08:53):Got it. That's pretty interesting. So it really doesn't have its own blockchain and you're just using the fault tolerance of the chains you're connected to.Jonathan Schemoul (09:04):Exactly.Anatoly Yakovenko (09:06):Awesome. Yeah, that's really cool. So the other challenge I think is like how do you deal with domains and the web? Where do you run these executed nodes? How do you connect all those pieces?Jonathan Schemoul (09:20):It's a really good question. To connect all the pieces together, we didn't develop some really fancy stuff like proof of space and time and things like that to verify that the data is really stored. We are using something much more low-tech, which is just a quality control. We have core channel nodes, which are the controllers of the network, which needs to keep some Aleph have stakers on such economics. They are verifying that other core channel nodes are behaving well. And that also the resource nodes are behaving well. Then the resource nodes are really doing the work of storing data, providing computing, et cetera. And they're continuously controlled by the core channel nodes.Anatoly Yakovenko (10:09):That's great. So they're basically like a tokenized health check, right?Jonathan Schemoul (10:14):Yeah.Anatoly Yakovenko (10:14):I can spin this up and they can continuously monitor whether this computation is making progress, right?Jonathan Schemoul (10:21):Exactly.Anatoly Yakovenko (10:21):Is that verification, is that programmable? Can me as an app developer, can I kind of code up my own apps, specific health checks or an interface or something like that?Jonathan Schemoul (10:35):It's a really good question. That's what we are working on exactly right now.Anatoly Yakovenko (10:40):I'm leaking all the features. My imagination is going.Jonathan Schemoul (10:44):No, no worry. Well, it's really interesting because to understand if an application behaves well on one host, you need to understand what the application is doing. So yes, we will give some kind of health check, which is kind of a unit test of how the app should work. So you will be able to provide unit tests for your app basically.Anatoly Yakovenko (11:11):That's really Cool. What about domains? Like actual DNS?Jonathan Schemoul (11:17):Yeah.Anatoly Yakovenko (11:20):I'm asking all the hard questions.Jonathan Schemoul (11:22):Yeah. These questions will be answered if I explain how we handle access to this virtual machine. Because for DNS, for just IPFS, there is already quite a few solution, that's not an issue. But then if you want to make a domain point to one micro VM, you want your micro VM to be able to serve your data. How we do first the load balancing because that's the important question. For load balancing we have two ways, one, which is a regular cloud load balancing, which could be blocked by government, could be censored, because that's what can happen when you have centralized point of control.Jonathan Schemoul (12:07):We will run it ourselves and a few of our partners might run some of the cloud load balancers that basically you can just point your domain to the cloud load balancer. And then the cloud load balancer will create certificates and stuff like that. It will work. We will run one instance. Ubisoft will likely run another. And like many of our partners. Well, for Ubisoft it's not sure, just some talks about it. But perhaps over partners could run cloud load balancers that we'd go on point on specific micro VM host to see where your app is running and point it to them, that might work.Jonathan Schemoul (12:48):Now What happens if a government says, "This app shouldn't work, this domain shouldn't work." Then you have two solutions, you either put the front-end inside IPFS, use some IPFS gateways, et cetera. And then the back-end is on the VM network. But then what happens if a government blocks the specific DNS inside the micro VM global.aleph.sh .aleph.cloud Whatever. Then we have a decentralized load balancing that comes into play.Jonathan Schemoul (13:24):The idea of the decentralized load balancing is that your browser will connect to the IPFS network using leap peer to peer, just leap peer to peer, find Pi Aleph nodes running, contact them directly then ask Pi Aleph node, "What micro VM host are running this software?" And then you can contact them directly. We are working on the JavaScript library that will do all this work on the client side so that you can have your front-end in IPFS that will then go and find all the back-end hosts that could answer your request.Anatoly Yakovenko (13:58):That's super cool. You guys are working on some really hard problems. I think it should be fairly easy to kind of have basically a resolver that points to ENS in the system, right. That's fairly straightforward. And basically you should be able to use any kind of like name, system, command any blockchain.Jonathan Schemoul (14:25):Yeah, clearly.Anatoly Yakovenko (14:26):Do you think that this is something that browsers are starting to recognize as standardizable? Is there a future where you think this technology could start percolating to the UI level where the end user can pick like blockchain based DNS resolver that kind of like connects all the pieces, right? From the human to this decentralized one.Jonathan Schemoul (14:51):I think that something that could come, I think that those that could really help in this is Mozilla foundation, I think that they would be the one to talk with. We aren't in talk with them because we don't really take that step right now. We have a lot on our plate. But in the future I'm pretty sure it's the way to go. We will connect to any effort in that area and we will recognize it. I know that for IPFS for example, IPFS, IPNS, there are some efforts on some browser extension that you can install to have it, et cetera.Anatoly Yakovenko (15:29):How does like certificate chaining play with us? What happens if I need to have a cert on my service and things like that.Jonathan Schemoul (15:38):A certificate on your service? Yeah.Anatoly Yakovenko (15:41):Like their sign or whatever.Jonathan Schemoul (15:43):Well, we use the one that everyone uses, which is-Anatoly Yakovenko (15:48):Let's Encrypt. The EFF one.Jonathan Schemoul (15:49):Yeah, exactly. We're using this one, we used the discovery with the content, so that we switch to a specific content when Let's Encrypt connects, then we serve this content, then we get a valid certificate, we can serve the good content.Anatoly Yakovenko (16:07):Can you unpack that a little bit?Jonathan Schemoul (16:10):Yeah. Well, Let's Encrypt has multiple ways to certify that you have a certain domain, for sub domains of .aleph.sh and .aleph.cloud, It's easy, we are using wildcard certificates. For custom domains that you could make point to your content directly, what we do is that you put a key inside your DNS to say, this is the virtual machine that should be mapped to that domain. Then you do a CNAME to our cloud load balancer and then the VM host when they get a request for this one, they go and check the DNS to see what VM they should serve on the generator certificate using Let's Encrypt for that domain and they start serving it.Anatoly Yakovenko (16:59):Oh man, this would be really cool. But if we could have like an ENS where in my ENS registry I set my Let's Encrypt domain, and then I run a local DNS server on my home machine where I run my browser and point that as a resolver, you could kind of tie these knots together and get-Jonathan Schemoul (17:23):Yeah, it could work.Anatoly Yakovenko (17:24):That's really cool. What happens if these instances die, where do you guys get more hardware? How does that process work?Jonathan Schemoul (17:36):Well, an instance can just stop, then the load balancing system will find another instance to run your code. Then what happens when an instance get a request for a code that doesn't have for the micro VM network. I mean, it goes on the network, checks, okay, what is the database entry that is in front? It takes the database entries. Has there been any upgrades to it? Okay. I get the upgrades. I subscribe using web socket to the upgrades of this database entry basically because it's a document about database entry.Jonathan Schemoul (18:14):And then it looks, okay, so this is the root FS that I should load. Do I have it? I have it, could I use it? If not, I download it from the network. I applied that root FS, where is the code? Okay. What volume does it needs and it builds and retransits and gets you the answer. For a cold start with no root FS or whatever, it can take a few seconds. But in general you use the same root FS as others. So you can get the code start. If you don't have the code, it's less than a second. If you already have the code of the application is like 150 millisecond for a cold start.Anatoly Yakovenko (18:53):Got It. And is the coordination to decide where to start this particular instance? Does that occur over the underlying chain, like Solana or Ethereum or whatever?Jonathan Schemoul (19:08):Again, that's something that we're working on. At start it's on the cloud load balancer. So the cloud load balancer are semi centralized for that. The idea is that each micro VM running node that starts running one will register a message, which is a database entry with a reference to say, "I am running this one." And then the cloud load balancer looks at the uptimes of the available micro VMs and say, "Okay, this micro VM has it ready." I'm forwarding it to it.Jonathan Schemoul (19:40):And then if there is none, then it could just route it to like a random one that has a good uptime. And then this one, the next time kind of like be choosing automatically because it is already serving it. If there is a lot of requests, it will provision multiple ones.Anatoly Yakovenko (19:59):Interesting. Got it. And you anticipate that you'll basically be able to move if the underlying chain is cheap and fast enough you should be able to move the coordination and kind of like start this instance, pull this volume. This would be really cool with like Arweave backed storage volumes. Because you could almost then see the lifetime, the life cycle of the application as its business logic is evolving, right? That state is very useful to developers who are being able to go back to a checkpoint effectively at any given time too.Jonathan Schemoul (20:38):Well, right now we are using our own storage engine, which is APFS compatible. But in the future we will allow to choose other storage engine and we will also develop gateways with like Arweave, Filecoin and other.Anatoly Yakovenko (20:53):Super cool. I used to work at Mesosphere so I don't know if you've heard of them, like D2iQ, this was kind of Kubernetes competitor, trying to build this decentralized operating system using Mesos as the jobs kind of Q-engine. There's a lot of similar challenges there, and this is really cool that you guys are building this in a decentralized web application that's kind of hosted in the real cloud, the mythical cloud.Jonathan Schemoul (21:28):Yeah. Well, there's a saying, there is no cloud, it's just other people computers. Here it's really other people computer. So it's pretty good because then you don't trust those computers because you know it's other people computers.Anatoly Yakovenko (21:44):How do you guys ensure the integrity of the computation itself? How do I know that the virtual machine, the execution environment that's running isn't malicious.Jonathan Schemoul (21:54):It's a really good question. There is multiple questions there. How can I ensure that this computation isn't returning a bad result because it knows who is on the other end. The load balancing system ensures that you don't really see who is in the other end, so you don't know who is making the request. So you don't know if it's a quality control call or if it's a real call. It goes back to your question of the testing of the application. And there is another one there which is the question of the secrets, because you might need secrets. If you want to do push notification based on a smart contract event on Solana, let's say, because that's something that we are working on right now, thinking about it.Anatoly Yakovenko (22:48):That's super cool.Jonathan Schemoul (22:48):So you would need secrets. You will need to story a secret to being able to go back to this device and send these device and notification. So you either store secrets in the local storage of the instance, but then if the instance dies, you can get it back or you try to get shared secrets between multiple hosts. We are working on it. We don't have a total answer on that. What we are working on is using free shirt cryptography, so that multiple host defined by the developer come under these secrets. And then you go back to a question of trust, which is problematic.Anatoly Yakovenko (23:30):By the threshold cryptography, is this like an MPC to compute, or are you guys thinking like BLS or like Schnorr aggregation?Jonathan Schemoul (23:42):More like you encrypt something that can be decrypted by multiple private keys.Anatoly Yakovenko (23:47):Got it.Jonathan Schemoul (23:48):And then if they want to send a message, it needs to be signed by at least x of y.Anatoly Yakovenko (23:54):Right. Got it.Jonathan Schemoul (23:57):Because this micro VM I mentioned can also send messages on the network. These messages on the network will be database entries that in the end might end up also on-chain using all records or whatever. Because these micro VM can read from on chain data and the idea is that we are working so that they can also write on chain as well. So then you might need some kind of trust somewhere. So one developer could say, I trust this host this host this host, but they need at least to do that calculation three times, let's say. But it's a bit problematic and we are still working on it. It's not finished yet, so yeah.Anatoly Yakovenko (24:40):That's what I mean, that's a really hard problem.Jonathan Schemoul (24:41):Yeah.Anatoly Yakovenko (24:43):Really cool. Yeah, the secrets thing is really challenging. I guess, what's your vision for this? You guys are tackling on some really hard problems, you get all of them done in the next year.Jonathan Schemoul (25:01):I hope so.Anatoly Yakovenko (25:06):What happens then? What is the vision for Aleph?Jonathan Schemoul (25:08):Well, here we are only speaking about a few crypto issues. We aim at bigger than just the crypto ecosystem. What we really want to do is decentralize the web, so getting bigger, way, way bigger, that's the goal. We are working with a few bigger partners who are part of the Ubisoft entrepreneurial labs, for example. We want to have a lot of hosting partners in the game that start providing resources so that I want it to be as easy as spinning up AWS server or whatever, you would just spin up VMs under the .im network. I want it to be as easy as using Firebase, using Amazon Lambda, et cetera.Jonathan Schemoul (25:51):And we have another big project going on, which is the indexing on Solana, where we are indexing data for a few protocols, currently Raydium, we might have another already soon. Well, I can say the name. We are working a lot on Orca, on port finance right now, and a lot of others actually that I can't really talk yet. But the idea is to have all these data available, have all these data feed coming up so that you can have events based on them, also do off-chain computation and things like that.Jonathan Schemoul (26:29):I really want DeFi to be totally resilient because until it's totally decentralized, you can stop DeFi. When it's totally decentralized, you can't. And if there is only the smart contracts that are decentralized, you can still stop it.Anatoly Yakovenko (26:48):Yeah. That's definitely a fair point. I think the UX issues around building also just like push notifications and all these other things for projects are really hard to overcome if it's a decentralized project, because who's going to host those servers, right, to connect to mobile and everything else. Yeah. You guys have a lot of work set out and it's pretty exciting. What do you think is missing? If you guys had like another, somebody else was building this other piece that you think is missing in the Web 3, what would it be?Jonathan Schemoul (27:26):What is missing today in the Web 3 ease of use for all this. We are trying to tackle this, but we have so much on our end. So this is a big issue, ease of use for developers, ease of use for users. Well, Phantom is already doing a great work on that end on Solana. But yeah, this and also I think that there is some kind of breaks between the ... In DeFi, if you want to move money into the real world, it gets hard really fast because there has been some kind of complications that have been put in place by regulators, by banks, by whatever. If we could just get all these parts simpler, it could be great. Some kind of link between FinTech and crypto that would work everywhere in the world, including Europe, USA, et cetera. It would be great. There are a lot of people working on it, but that's something that is missing as well.Anatoly Yakovenko (28:28):Yeah. Identity and like having those easy ramps is still hard. What about DNS? Just straight up resolving, do you think that's tackleable from a Web 3 perspective.Jonathan Schemoul (28:45):The issue is the way DNS is done. DNS protocol is great, but it implies centralization points, a lot of centralization points, which are problematic. Then you will need another standard on DNS. But if you have another standard on DNS, then you have the issue that the network right now is done, is not done for it and the browser don't understand it, et cetera, and operating system don't understand it. We would need gateways for that. I think it's doable. It's definitely doable, but it's a lot of work. And you would need multiple root servers, even virtual root servers, like what you said, local DNS server that would resolve your request, it could work.Jonathan Schemoul (29:38):If Let's Encrypt could understand it in the same way, it would work. Or we could even have something different than the root certificate that we have today, because with blockchain, we already have private keys. We already have signature. So if you sign your content with your private key, then you can verify it on the other end. And you don't really need all these chains of certificates that are here today. So that could also be another solution, but it would need another way, because right now we have roots certificate, children's certificate, et cetera. And it all goes back to central authority. The whole DNS on certificate system today goes with authority. With blockchain we are trained to remove authorities.Anatoly Yakovenko (30:33):Yeah. Do you guys see this as becoming developer facing, or maybe someday eventually kind of like client facing and want these decentralized applications running for me, kind of my own instances. Or is this always going to be here I am, team Orca, go to this domain as a user.Jonathan Schemoul (30:56):It's a good question as well. It's always the issue between hosted components, locally run components and kind of pragmatic on that. At start I would really like to, everything runs inside my browser, everything works. That's great. In reality, you have mobile phones, you have tablets, you have computers, you have a lot range of devices that can be running all the time. So real peer to peer application can't really work that well, unless you go and say, "Okay. While you are waiting for me, please send it to my friend, that will forward the data for me, et cetera.Jonathan Schemoul (31:40):Blockchains are really helping there is that we have a centralized authority, which is the blockchain that you can trust and that can hold data for you and can even encrypt it for you or store it on aleph.im, whatever, and only you can decrypt it. I think that the mix between the two would be good, like self hosted data and remotely hosted data on the decentralized cloud, a good mix of the two could be good. And the efforts by the leap peer to peer team, with the javascript leap peer to peer. And there are a few of us like that helps, because once you have access to a peer-to-peer network directly from your browser, you can cut middlemen. You can cut central authorities, et cetera, if you're the blockchain that serves as a central authority.Anatoly Yakovenko (32:28):What kind of loads have you guys seen or been able to test this out, in terms of like users request per second, kind of WebSocket connections per second.Jonathan Schemoul (32:39):It depends because when it's per server, that's not that much of an issue because the micro VM supervisor just forwards the request to the underlying software. If you don't choose local persistent volume, the supervisor can run as many instances of your program as needed, then you can spawn multiple one even inside the same supervised cluster. And then the network, if it sees that this one has issues adding the request load you can load new ones.Jonathan Schemoul (33:18):I don't think that there is really a limit on the request per second for that. So it's not really the issue that we have. And then on the database part, same, if you access one API server and you give it 500,000 requests per second, it would go down, because it's a server. If you target multiple API server, you are good. So that's also where the decentralized load balancing helps because if you use a cloud load balancer obviously even this cloud can go down. But if you contact a peer to peer network to know what host can answer, then you can contact multiple host. And all our core channel nodes, we are currently 54 of them are also API servers that users can connect to to get the data, which will be certified by our core channel node.Anatoly Yakovenko (34:10):Cool. As a whole, how many, I guess, do you have an idea of how many users per second or humans per second have you guys served in some peak times?Jonathan Schemoul (34:21):We don't, because we don't store metrics currently, we should. We don't have it because we didn't want to have any kind of log or whatever on the users, but we should add it, that's actually a good point, we will.Anatoly Yakovenko (34:37):Yeah. I mean, I think you got to be really aware of privacy and how that impacts some applications. But really interesting to see how this works. Caching is another one of those things, basically having a distributed cache around the world for often queried data. And this is an issue that I think doesn't have a good solution in Web 3 right now. You do all this work, set up a purely thin client, that's like loads from code, only talks to the chain and then you got to go fetch assets. And if you're using centralized ... Yeah, they can basically inject whatever they want.Jonathan Schemoul (35:25):Yeah, that's the main issue. And the good part is that if you also randomize where the request of the users go, if there is one bad actor, it will only inject bad data once in a while you don't even know where. Once there is a quality control it will detect it, so that can also be a solution. It's not a silver bullet either, but it can definitely help. So like for Solana what we are doing right now, for Raydium for example, is that we have an indexer that talks to multiple RPC of Solana then get the transaction history, store it inside the level DB, inside the micro VM, and then index the data.Jonathan Schemoul (36:09):Then we can get data on the pool's latest trades and stuff like that. The idea is that if there is too much request on one index, it will start another index or another index or another index, or et cetera, so that when you do a request, it reroutes you randomly to multiple hosts that have the same index.Anatoly Yakovenko (36:28):How fast is that?Jonathan Schemoul (36:31):Not fast enough currently. Well, it's fast enough for Raydium.Anatoly Yakovenko (36:35):Okay.Jonathan Schemoul (36:36):It works really well.Anatoly Yakovenko (36:40):Raydium gets a ton of hits. I mean, some of their IDOs have seen half a million requests per second-Jonathan Schemoul (36:48):Yeah. So for the Raydium data, it handles it well, like all the trades, whatever, it handles it pretty well. We don't get behind blocks in the indexing, so it works well. For Serum it's a bit more problematic because you need to watch, event cue all the time. I really hope they will have some kind of flux in the future. I think that they are working on it. So that would really help us either to get history even when we aren't watching their event cue.Anatoly Yakovenko (37:23):Yeah. So not half a million per second, half a million total, which is quite different, but yeah, they see some really good traffic.Jonathan Schemoul (37:30):Yeah.Anatoly Yakovenko (37:32):Cool. I mean, that's really cool. I think really hard part I think in designing these systems, one, is the problem is difficult, but then once you build the first version of it and you start hitting real traffic, there's a lot of parts that fit together that break under load. So what is your debugging like? How do you guys actually monitor like debug, like PagerDuty, what do you guys use as a team?Jonathan Schemoul (38:01):Right now our team is still small. We are growing a lot. Right now we are like 10 developers. A few months ago we were only three. A year ago I was alone. So we are growing really fast and we are putting all these things into place. Right now everyone monitors and checks what happens and it helps. There is Hugo who is on the micro VM side, Ali was mostly on the indexer side, myself we can get everything. But we are putting really real stuff in place right now to have it, because we are a growing startup so it takes time to get everything in place.Anatoly Yakovenko (38:43):Yeah, for sure. Do you envision a PagerDuty team for this?Jonathan Schemoul (38:48):Yes. I think that we will need one. Once we have more application that are using it, we will need one. So yes, if you have advices on that day, I'm really happy to get them.Anatoly Yakovenko (39:00):I mean, it's just part of life. It's not complicated. It's just work. This is I think that like response team I think is a difficult thing to set up in a decentralized community. If you guys are building a decentralized network with providers that are supplying hardware and all this other stuff, those are the folks that we found to be really responsive and have a lot of stake in growing this. How do the economics work for all the people actually supplying the hardware and bandwidth, et cetera?Jonathan Schemoul (39:36):Again, the research and economics aren't live yet. We are working on them. The core channel nodes economics is already there for like a year, now it works well. For the core channel node you need to have 200,000 Aleph to start a node and 500,000 Aleph, staked on a node, so that it can start to run. And then all the node operator get a share of a global envelope daily for all the nodes. All the stakers get a part of the envelope for stakers. The more nodes active, the bigger the envelope for staker is. But then for each node, they will earn a bit less if there are more nodes because it's a global envelope. So it helps stakers grow the number of nodes that are active, so that's for the core channel nodes.Jonathan Schemoul (40:25):For the resource nodes, to get storage or computing on network, there is two ways to get it. One that is already live, which is hold X amount of Aleph and get that amount of storage, hold X amount of Aleph and have the ability to start one VM with X megabyte of RAM, X virtual CPU, et cetera. And then the multiplier, and all that gives you the total count of micro VM I mentioned that can be running on your network based on your balance. The good part with that is that partner project could use a lending protocol to borrow Aleph where depositing their own token to get service. They would get the service for free just paying interest in their token, inside the borrowing protocol.Anatoly Yakovenko (41:14):Got it.Jonathan Schemoul (41:15):So that's a way for protocols to get it, but it's quite expensive because they don't directly pay for it. So for this way of using it, Aleph.im network is paying for them from the incentive pool, which right now it's one fifth of the supply, and we are changing it in the next few months, we'll change a bit of economics. It will be nearly half of the supply that would be dedicated to pay for that. Because since you lock a part of the supply, then you can release a bit inside circulating because of this new use. So that's for the hold X Aleph tokens.Jonathan Schemoul (41:51):And then there is another way that isn't developed yet that we will likely use Solana for, because it's fast enough for micro-payments in that area. It's like pay per action, pay X Aleph per gigabyte per month. You as a provider, you can say, "I am okay to be paid at least that much." And then users will say, "I want my data to be replicated at least four time. And I'm okay to pay at most that much for this." Then you get divided by those who provide service and the payment is done as micro payments. And same for the micro VM you pay per CPU per hour, et cetera.Anatoly Yakovenko (42:32):Got it. That's really cool. Well, this has been awesome to have you on the show. I mean, we got into I think the really deep, deep tells of how Aleph works, so I had a blast because it really reminds me of the spending, working on the stuff for centralized systems. It's really cool to see this kind of built ground up for decentralized ones as well. So appreciate the work you're doing. Thank you, Jonathan.Jonathan Schemoul (43:00):Thank you very much for having that call. It was really great talking with you.Anatoly Yakovenko (43:04):Awesome. And good luck to you guys. I mean, startups are blood, sweat and tears, so just keep working on the vision. You'll get there.Jonathan Schemoul (43:11):Thank you very much.Anatoly Yakovenko (43:13):Cool. Take care.

Serverless Chats
Episode #113: Serverless for Startups with Chris Munns

Serverless Chats

Play Episode Listen Later Oct 4, 2021 57:38


Chris Munns is a Tech Lead/Advisor for Startup Solution Architects at Amazon Web Services based in New York City. Chris spent the last 4.5 years working with AWS's developer customers to understand how serverless technologies can drastically change the way they think about building and running applications at potentially massive scale with minimal administration overhead. Before this, Chris a global Business Development Manager for DevOps at AWS, he spent a few years as a Solutions Architect at AWS, and has held senior operations engineering posts at Etsy, Meetup, and other NYC based startups. Chris has a Bachelor of Science in Applied Networking and System Administration from the Rochester Institute of Technology. Twitter: @chrismunns Email: munns@amazon.com AWS Compute Blog: https://aws.amazon.com/blogs/compute/

Hunger Hunt Feast | Strategic Fitness
81. I Had C-19, EUA and Mandates

Hunger Hunt Feast | Strategic Fitness

Play Episode Listen Later Oct 4, 2021 35:47


Welcome back to Hunger Hunt Feast! Today Zane shares recent updates and everything you need to know about EUA, the current Mandates, and his "experience" with C-19.  LINKS: FDA letter regarding EUA https://www.fda.gov/media/150386/download FDA press release re: EUA of Pfizer booster https://www.fda.gov/news-events/press-announcements/fda-authorizes-booster-dose-pfizer-biontech-covid-19-vaccine-certain-populations Doctor cites early treatment success with 6000 covid patients. https://globalcovidsummit.org/news/doctor-cites-early-treatment-success-with-6000-covid-patients El Salvador, Guatemala and Bolivia offer medicine kits for covid 19 https://trialsitenews.com/el-salvador-guatemala-and-bolivia-offer-medicine-kits-for-covid-19-without-anticipating-adverse-reactions/ Covid 19 vaccine mandates are now pointless by Nina Pierpont, MD, PhD https://theexpose.uk/wp-content/uploads/2021/09/Pierpont-Why-mandated-vaccines-are-pointless-final-1.pdf?utm_campaign=MD-Sun9%2F19Email-CTA%20%28S2a95v%29&utm_medium=campaign-email&utm_source=Engaged%20Segment%20%28from%20Main%20List%29&_kx=9YJ86h4wPgEUc22WlROdG8Gr9lYwm35XJm4GSKAp0ac%3D.WY2S6U Use of Ivermectin Is Associated With Lower Mortality in Hospitalized Patients With Coronavirus Disease 2019: The Ivermectin in COVID Nineteen Study https://pubmed.ncbi.nlm.nih.gov/33065103/ Lambda variant has a better chance than the Delta variant to escape vaccines https://www.biorxiv.org/content/10.1101/2021.08.25.457692v1 Central obesity, smoking habit, and hypertension are associated with lower antibody titres in response to COVID‐19 mRNA vaccine https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8209952/ Vaccine Adverse Event Reporting System https://openvaers.com/index.php Zane's Links: Get organic keto meals delivered to your door!! https://trifectanutrition.llbyf9.net/zane ReLyte Electrolytes by Redmond Real Salt https://shop.redmond.life?afmc=Zane Follow me on Instagram: https://www.instagram.com/zanegriggsfitness QUICK EPISODE SUMMARY This is not a one-size-fits-all conversation The stats behind the pandemic  When Zane tested “positive” Managing symptomS Alternative responses around the world The recent revisions to the RUA How to prove you have antibodies Delta and the jab Is the mandate needed? Support the freedom to choose

Lambda3 Podcast
Lambda3 Podcast 267 – Lambda3, PrograMaria Summit 2021, perguntas recebidas no evento

Lambda3 Podcast

Play Episode Listen Later Oct 1, 2021 75:21


Neste podcast contamos como foi a participação da Lambda no ProgramariaSummit 2021, além de responder algumas perguntas frequentes sobre nosso modelo de trabalho. Neste podcast contamos como foi a participação da Lambda no ProgramariaSummit 2021, além de responder algumas perguntas frequentes sobre nosso modelo de trabalho. Feed do podcast: www.lambda3.com.br/feed/podcast Feed do podcast somente com episódios técnicos: www.lambda3.com.br/feed/podcast-tecnico Feed do podcast somente com episódios não técnicos: www.lambda3.com.br/feed/podcast-nao-tecnico Lambda3 · 267 - Lambda3 no PrograMaria Summit 2021 - Participação de Lambdas e perguntas recebidas no evento Pauta: PrograMaria Summit 2021 O que rolou? Lambda3 como principal patrocinadora Diamante e porque isso se alinha com a visão da empresa O que levamos? Estande virtual Salas de Mentorias Mestres de cerimônia Renata e Luciana Patricia Kost, Olivia Janequine, Sarah Barros, Luiza Prata, Isabela Hinckel, Angélica Canuto, Carolina Felix Speed Hiring Palestras Olivia Janequine: Táticas para lidar com ambientes hostis Mafoane Odara, líder RH LATAM Facebook: Gente é para brilhar! (Keynote) Isabela e Sarah: Descoberta de produto em times multidisciplinares Luiza Prata: Como funciona React Native Kits sorteados exclusivos da Lambda3 Resposta das TOP 10 perguntas enviadas no estande Lambda3 Participantes: Luiza Prata - Luiza Prata Soldi Passos Olivia Janequine - Olivia Janequine Patricia Kost - Patrícia Kost Sarah Barros - Sarah Barros Vanda Ferreira - Vanda Ferreira  Vitor Norton - Vítor Norton Edição: Compasso Coolab Créditos das músicas usadas neste programa: Music by Kevin MacLeod (incompetech.com) licensed under Creative Commons: By Attribution 3.0 - creativecommons.org/licenses/by/3.0

Break Things On Purpose
John Martinez

Break Things On Purpose

Play Episode Listen Later Sep 21, 2021 32:55


In this episode, we cover: 00:00:00 - Introduction  00:03:15 - FinOps Foundation and Multicloud  00:07:00 - Costs  00:10:40 - John's History in Reliability Engineering  00:16:30 - The Actual Cost of an Outages, Security, Etc. 00:21:30 - What John Measures  00:28:00 - What John is Up To/Latinx in Tech Links: Palo Alto Networks: https://www.paloaltonetworks.com/ FinOps Foundation: https://www.finops.org Techqueria.org: https://techqueria.org LinkedIn: https://www.linkedin.com/in/johnmartinez/ TranscriptJohn: I would say a tip for better monitoring, uh, would be to, uh turn it on. [laugh]. [unintelligible 00:00:07] sounds, right?Jason: Welcome to the Break Things on Purpose podcast, a show about chaos engineering and operating reliable systems. In this episode we chat with John Martinez, Director of Cloud R&D at Palo Alto Networks. John's had a long career in tech, and we discuss his new focus on FinOps and how it has been influenced by his past work in security and chaos engineering.  Jason: So, John, welcome to the show. Tell us a little bit about yourself. Who are you? Where do you work? What do you do?John: Yeah. So, John Martinez. I am a director over at Palo Alto Networks. I have been in the cloud security space for the better of, I would say, seven, eight years or so. And currently, am in transition in my role at Palo Alto Networks.So, I'm heading headstrong into the FinOps world. So, turning back into the ops world to a certain degree and looking at what can we do, two things: better manage our cloud spend and gain a lot more optimization out of our usage in the cloud. So, very excited about new role.Jason: That's an interesting new role. I'd imagine that at Palo Alto Networks, you've got quite a bit of infrastructure and that's probably a massive bill.John: It can be. It can be. Yeah, [laugh] absolutely. We definitely have large amount of scale, in multi-cloud, too, so that's the added bonus to it all. FinOps is kind of a new thing for me, so I'm pretty happy to, as I dig back into the operations world, very happy to discover that the FinOps Foundation exists and it kind of—there's a lot of prescribed ways of both looking at FinOps, at optimization—specifically in the cloud, obviously—and as well as there's a whole framework that I can go adopt.So, it's not like I'm inventing the wheel, although having been in the cloud for a long time, and I haven't talked about that part of it but a lot of times, it feels like—in my early days anyway—felt like I was inventing new wheels all the time. As being an engineer, the part that I am very excited about is looking at the optimization opportunities of it. Of course, the goal, from a finance perspective, is to either reduce our spend where we can, but also to take a look at where we're investing in the cloud, and if it takes more of a shift as opposed to a straight-up just cut the bill kind of thing, it's really all about making sure that we're investing in the right places and optimizing in the right places when it comes down to it.Jason: I think one of the interesting perspectives of adopting multi-cloud is that idea of FinOps: let's save money. And the idea, if I wanted to run a serverless function, I could take a look at AWS Lambda, I could take a look at Azure Functions to say, “Which one's going to be cheaper for this particular use case,” and then go with that.John: I really liked how the FinOps Foundation has laid out the approach to the lifecycle of FinOps. So, they basically go from the crawl, walk, run approach which, in a lot of our world, is kind of like that. It's very much about setting yourself up for success. Don't expect to be cutting your bill by hundreds of thousands of dollars at the beginning. It's really all about discovering not just how much we're spending, but where we're spending it.I would categorize the pitting the cloud providers against each other to be more on the run side of things, and that eventually helps, especially in the enterprise space; it helps enterprises to approach the cloud providers with more of a data-driven negotiation, I would say [laugh] to your enterprise spend.Jason: I think that's an excellent point about the idea of that is very much a run. And I don't know any companies within my sphere and folks that I know in the engineering space that are doing that because of that price competition. I think everybody gets into the idea of multi-cloud because of this idea of reliability, and—John: Mm-hm.Jason: One of my clouds may fail. Like, what if Amazon goes down? I'd still need to survive that.John: That's the promise, right? At least that's the promise that I've been operating under for the 11 years or so that I've been in the cloud now. And obviously, in the old days, there wasn't a GCP or an Azure—I think they were in their infancy—there was AWS… and then there was AWS, right? And so I think eventually though you're right, you're absolutely right. Can I increase my availability and my reliability by adopting multiple clouds?As I talk to people, as I see how we're adopting the multiple clouds, I think realistically though what it comes down to is you adopted cloud, or teams adopt a cloud specifically for, I wouldn't say some of the foundational services, but mostly about those higher-level niche services that we like. For example, if you know large-scale data warehousing, a lot of people are adopting BigQuery and GCP because of that. If you like general purpose compute and you love the Lambdas, you're adopting AWS and so on, and so forth. And that's what I see more than anything is, I really like a cloud's particular higher level service and we go and we adopt it, we love it, and then we build our infrastructure around it. From a practical perspective, that's what I see.I'm still hopeful, though, that there is a future somewhere there where we can commoditize even the cloud providers, maybe [laugh]. And really go from Cloud A to Cloud B to Cloud C, and just adopt it based on pricing I get that's cheaper, or more performant, or whatever other dimensions that are important to me. But maybe, maybe. We'll remain hopeful. [laugh].Jason: Yeah, we're still very much in that spot where everybody, despite even the basics of if I want to a virtual machine, those are still so different between all the clouds. And I mean even last week, I was working on some Terraform and the idea of building it modularly, and in my head thinking, “Well, at some point, we might want to use one of the other clouds so let's build this module,” and thinking, “Realistically, that's probably not going to happen.”John: [laugh]. Right. I would say that there's the other hidden cost about this and it's the operational costs. I don't think we spend a whole lot of time talking about operational costs, necessarily, but what is it going to cost to retrain my DevOps team to move from AWS to GCP, as an example? What are the underlying hidden costs that are there?What traps am I going to fall into because of that? It seems cool; Terraform does a great job of getting that pain into the multiple clouds from an operations perspective. Kubernetes does a great job as well to take some of that visibility into the underlying—and I hate to use it this way but ‘hardware' [laugh] virtual hardware—that's like EC2 or Google Compute, for example. And they do great jobs, but at the end of the day we're still spending a lot of time figuring out what the foundational services are. So, what are those hidden costs?Anyway, long story short, as part of my journey into FinOps, I'm looking forward into not just uncovering the basics of FinOps, where is what are we spending? Where are we spending it? What are the optimization opportunities? But also take a look at some of the more hidden types of costs. I'm very interested in that aspect of the FinOps world as well. So, I'm excited.Jason: Those hidden costs are also interesting because I think, given your background in security—John: Mm-hm.Jason: —one of the challenges in multi-cloud is, if I'm an expert in AWS and suddenly we're multi-cloud and I have to support GCP, I don't necessarily know all of those correct settings and how to necessarily harden and build my systems. I know a model and a general framework, but I might be missing something. Talk to me a bit more about that as a security person.John: Yeah.Jason: What does that look like?John: Yeah, yeah. It's very nuanced, for sure. There are definitely some efforts within the industry to help alleviate some of that nuance and some of those hidden settings that I might not think about. For example, CIS Foundations as a community, the foundations of benchmarks that CIS produces can be pretty exhaustive—and there are benchmarks for the major clouds as well—those go a long way to try and describe at least, what are the main things I should look at from a security perspective? But obviously, there are new threats coming along every day.So, if I was advising security teams, security operations team specifically, it would be definitely to keep abreast into what are the latest and go take a look at what some of the exploit kits are looking for or doing and adopting some of those hidden checks into, for example, your security operations center, what you react to, what the incident responses are going to be to some of those emerging threats. For sure it is a challenge, and it's a challenge that the industry faces and one that we go every day. And an exploit that might be available for EC2 may be different on Google Compute or maybe different on Azure Compute.Jason: There's a nice similarity or parallel there to what we often talk about, especially in this podcast, is we talk about chaos engineering and reliability and that idea of let's look at how things fail and take what we know about one system or one service, and how can we apply that to others? From your experience doing a wide breadth of cloud engineering, tell me a bit more about your experience in the reliability space and keeping—all these great companies that you've worked for, keeping their systems up and running.John: I think I have one of the—fortunate to have one of the best experiences ever. So, I'll have to dig way back to 11 years ago, or so [laugh]. My first job in the cloud was at Netflix. I was at Netflix right around the time when we were moving applications out of the data center and into AWS. Again, fortunate; large-scale, at the cusp of everything that was happening in the cloud, back in those days.I had just helped finish—I was a systems engineer; that's where I transitioned from, systems engineering—and just a little bit of a plug there, tomorrow is Sysadmin Day, so I still am an old school sysadmin at heart so I still celebrate Sysadmin Day. [laugh]. But I was doing that transition from systems engineering into cloud engineering at Netflix, just helped move a database application out from the data center into AWS. We were also adopting in those days, very rapidly, a lot of the new services and features that AWS was rolling out. For example, we don't really think about it today anymore, but back then EBS-backed instances was the thing. [laugh].Go forth and every new EC2 instance we create is going to be EBS-backed. Okay, great. March, I believe it was March 2011, one of AWS's very first, and I believe major, EBS outages occurred. [laugh]. Yeah, lots of, lots of failure all over the place.And I believe from that a lot of what—at least in Gremlin—a lot of that Chaos Monkey and a lot of that chaos engineering really was born out of a lot of our experiences back then at Netflix, and the early days of the cloud. And have a lot of the scars still on me. But it was a very valuable lesson that I take now every day, having lived through it. I'm sure you guys at Gremlin see a lot of this with your customers and with yourselves, right, is that the best you can do is test those failure scenarios and hope that you are as resilient as possible. Could we have foreseen that there was going to be a major EBS outage in us-east-1? Probably.I think academically we thought about it, and we were definitely preaching the mantra of architect for failure, but it still bit us because it was a major cascading outage in one entire region in AWS. It started with one AZ and it kept rolling, and it kept rolling. And so I don't know necessarily in that particular scenario that we could have engineered—especially with the technology of the day—we could have engineered full-on failover to another region, but it definitely taught us and me personally a lot of lessons around how to architect for failure and resiliency in the cloud, for sure.Jason: I like that point of it's something that we knew theoretically could maybe happen, but it always seems like the odds of the major catastrophes are so small that we often overlook them and we just think, “Well, it's going to be so rare that it'll never happen, so we don't think about it.” As you've moved forward in your career, moving on from Netflix, how has that shaped how you approach reliability—this idea of we didn't think EBS could ever go down and lead to this—how do you think of catastrophic failures now, and how do you go about testing for them or architecting to withstand them?John: It's definitely stayed with me. Every ops job that I've had since, it's something that I definitely take into account in any of those roles that I have. As the opportunity came up to speak with you guys, wanted to think about reliability and chaos in terms of cloud spend, and how can I marry those two worlds together? Obviously, the security aspect of things, for sure, is there. It's expecting the unexpected and having the right types of security monitoring in place.And I think that's—kind of going back to an earlier comment that I made about these unexpected or hidden costs that are there lying dormant in our cloud adoption, just like I'm thinking about the cost of security incidents, the cost of failure, what does that look like? These are answers I don't have yet but the explorer in me is looking forward to uncovering a lot of what that's going to be. If we talk in a year from now, and I have some of that prescribed, and thought of, and discovered, and I think it'll be awesome to talk about it in a year's time and where we are. It's an area that I definitely take seriously I have applied not just to operational roles, but as I got into more customer-facing roles in the last 11 years, in between advising customers, both as a sales engineer, as head of customer success, and cloud security startup that I worked for, Evident.io, and then eventually moving here to Palo Alto Networks, it's like, how do I best advise and think about—when I talk to customers—about failure scenarios, reliability, chaos engineering? I owe it all to that time that I spent at Netflix and those experiences very early on, for sure.Jason: Coming back to those hidden costs is definitely an important thing. Especially I'm sure that as you interact with folks in the FinOps world, there's always that question of, “Why do I have so much redundancy? Why am I paying for an entire AZs worth of infrastructure that I'm never using?” There's always the comment, “Well, it's like a spare tire; you pay for an extra tire in case you have a flat.” But on some hand, there is this notion of how much are we actually spending versus what does an outage really cost me?John: Right. We thought about that question very early on at another company I worked at after Netflix and before the startup. I was fortunate again to work in another large-scale environment, at Adobe actually, working on the early days of their Creative Cloud implementation. Very different approach to doing the cloud than Netflix in many ways. One of the things that we definitely made a conscious effort to do, and we thought about it in terms of an insurance policy.So, for example, S3 replication—so replicating our data from one region to another—in those days, an expensive proposition but one that we looked at, and we intentionally went in with, “Well, no, this is our customer data. How much is that customer data worth to us?” And so we definitely made the conscious decision to invest. I don't call it ‘cost' at that point; I call that an investment. To invest in the reliability of that data, having that insurance policy there in case something happened.You know, catastrophic failure in one region, especially for a service as reliable and as resilient as S3 is very minuscule, I would say, and in practice, it has been, but we have to think about it in terms of investing. We definitely made the right types of choices, for sure. It's an insurance policy. It's there because we need it to be there because that's our most precious commodity, our customers' data.Jason: Excellent point about that being the most precious commodity. We often feel that our data isn't as valuable as we think it is and that the value for our companies is derived from all of the other things, and the products, and such. But when it comes down to it, it is that data. And it makes me think we're currently in this sort of world where ransomware has become the biggest headline, especially in the security space, and as I've talked with people about reliability, they often ask, “Well, what is Gremlin do security-wise?” And we're not a security product, but it does bring that up of, if your data systems were locked and you couldn't get at your customer information, that's pretty similar to having a catastrophic outage of losing that data store and not having a backup.John: I've thought about this, of course, in the last few weeks, obviously. A very, very public, very telling types of issues with ransomware and the underlying issues of supply chain attacks. What would we do [laugh] if something like that were to happen? Obviously, rhetorically, what would we do? And lots of companies are paying the ransom because they're being held at gunpoint, you know, “We have your data.”So yeah, I mean, a lot of it, in the situation, like the example I gave before, could not just the replication of, for example, my entire S3 bucket where my customer data is thwarted a situation like that? And then you think about, kind of like, okay, let's think about this further. If we do it in the same AWS account, as an example, if the attacker obtained my IAM credentials, then it really comes down to the same thing because, “Oh, look it, there's another bucket in that other region over there. I'm going to go and encrypt all of those objects, too. Why not, right?” [laugh].And so, it also begs the question or the design principles and decisions of, well, okay, maybe do I ship it to a different account where my security context is different, my identity context is different? And so there's a lot of areas to explore there. And it's very good question and one that we definitely do need to think about, in terms of catastrophic failure because that's the way to think about it, for sure.Jason: Yeah. So, many parallels between that security and reliability, and all comes together with that FinOps, and how much are you—how much do we pay for all of this?John: Between the reliability and the security world, there's a lot of parallels because your job is about thinking what are the worst-case scenarios? It's, what could possibly go wrong? And how bad could it be? And in many cases, how bad is it? [laugh].Especially as you uncover a lot of the bad things that do happen in the real world every day: how bad is it? How do I measure this? And so absolutely there's a lot of parallels, and I think it's a very interesting point you make. And so… yeah so, Jason, how can we marry the two worlds of chaos engineering and security together? I think that's another very exciting topic, for sure.Jason: That is, absolutely. You mentioned just briefly in that last statement, how do you measure it?John: Yep.Jason: That comes up to something that we were chatting about earlier is monitoring, and what do you measure, and ensuring that you're measuring the right things. From your experience building secure systems, talk to me about what are some of the things that you like to measure, that you like to get observability on, that maybe some folks are overlooking.John: I think the overlooking part is an interesting angle, but I think it's a little bit more basic than that even. I'll go to my time in the startup—so at Evident.io—mainly because I was in customer success and my job was to talk to our customers every day—I would say that a bunch of our customers—and they varied based on maturity level, but we were working with a lot of customers that were new in the cloud world, and I would say a lot of customers were still getting tripped up by a lot of the basic types of things. For example—what do I mean by that? Some of the basic settings that were incorrect were things just, like, EC2 security groups allowing port 22 in from the world, just the simple things like that. Or publicly accessible S3 buckets.So, I would say that a lot of our customers were still missing a lot of those steps. And I would say, in many of the cases, putting my security hat on, the first thing you go to is, well, there's an external hacker trying to do something bad in your AWS accounts, but really, the majority of the cases were all just mistakes; they were honest. I'm an engineer setting up a dev account and it's easier for me, instead of figuring out what my egress IP is for my company's VPN, it's easier for me just to set port 22 to allow all from the world. A few minutes later, there you go. [laugh]. Exploit taken, right? It's just the simple stuff; we really as an industry do still get tripped up by the simple things.I don't know if this tracks with the reliability world or the chaos engineering world, but I still see that way too much. And that just tells me that even if we are in the cloud—mature company or organization—there's still going to be scenarios where that engineer at two in the morning just decides that it's just easier to open up the firewall on EC2 than it is to do, quote-unquote, “The right thing.” Then we have an issue. So, I really do think that we can't let go of not just monitoring the basics, but also getting better as an industry to alert on the basics and when there are misconfigurations on the basics, and shortening that time to alert because that really is—especially in the security world—that really is very critical to make sure that window between when that configuration setting is made to when that same engineer who made the misconfiguration get alerted to the fact that it is a misconfiguration. So. I'll go to that: it's the basics. [laugh].Jason: I like that idea of moving the alert forward, though. Because I think a lot of times you think of alerts as something bad has happened and so we're waiting for the alert to happen when there's wrongful access to a system, right? Someone breaks in, or we're waiting for that alert to happen when a system goes down. And we're expecting that it's purely a response mechanism, whereas the idea of let's alert on misconfigurations, let's alert on things that could lead to these, or that will likely lead to these wrong outcomes. If we can alert on those, then we can head it off.John: It's all the way. And in the security world, we call it shifting left, shifting security all the way to the left, all the way to the developer. Lots of organizations are making a lot of the right moves in that direction for embedding security well into the development pipeline. So, for example, I'll name two players in the Infrastructure as Code as we call it in the security space. And I'll name the first one just because they're part of Palo Alto Networks now, so Bridgecrew; so very strong, open-source solution in that space, as well as over on the HashiCorp side where Sentinel is another example of a great developer-forward shift-left type of tool that can help thwart a lot of the simple security misconfigurations, right from your CI/CD pipelines, as opposed to the reaction time over here on the right, where you're chasing security misconfigurations.So, there's a lot of opportunity to shorten that alert window. And even, in fact, I've spent a lot of time in the last couple of years—I and my team have spent a lot of time in the last couple of years thinking about what can the bots do for us, as opposed to waiting for an alert to pop up on a Slack message that says, “Hey, engineer. You've got port 22 open to the world. You should maybe think about doing something.” The right thing to do there is for something—could be something as simple as an alert making it to a Lambda function and the Lambda function closing it up for you in the middle of the night when you're not paying attention to Slack, and the bot telling you, “Hey, engineer. By the way, I closed the port up. That's why it's broken this morning for you.” [laugh]. “I broke it intentionally so that we can avoid some security problems.”So, I think there's the full gamut where we can definitely do a lot more. And that's where I believe the new world, especially in the security world, the DevSecOps world, can definitely help embed some of that security mindset with the rest of the cloud and DevOps space. It's certainly a very important function that needs to proliferate throughout our organizations, for sure.Jason: And we're seeing a lot of that in the reliability world as well, as people shift left and developers are starting to become more responsible for the operations and the running of their services and applications, and including being on call. That does bring to mind that idea, though—back to alerting on configurations and really starting to get those alerts earlier, not just saying that, “Hey, devs, you're on call so now you share a pain,” but actually trying to alleviate that pain even further to the left. Well, we're coming up close to time here. So, typically at this point, one thing that I like to do is we like to ask folks if they have anything to plug. Oftentimes that's where people can find you on social media or other things. I know that you're connected with Ana through Latinx in Tech, I would love to share more about that, too. So.John: For sure, yeah. So, my job in terms of my leadership role is definitely to promote a lot of diversity, inclusion, and equity, obviously, within the workspace. Personally, I do also feel very strongly that I should be not just preaching it, but also practicing it. So, I discovered in the last year—in fact, it's going to be about a year since I joined Techqueria—so techqueria.org—and we definitely welcome anybody and everybody.We're very inclusive, all the way from if you're a member of the Latinx community and in technology, definitely join us, and if you're an ally, we definitely welcome you with open arms, as well, to join techqueria.org. It is a very active and very vibrant community on Slack that we have. And as part of that, I and a couple of people in Techqueria are running a couple of what we call cafesitos which is the Spanish word for coffees, coffee meetings.So, it's a social time, and I'm involved in helping lead both the cybersecurity cafecito—we call it Cafecito Cibernético, which happens every other Friday. And it's security-focused, it's security-minded, we go everywhere from being very social and just talking about what's going on with people personally—so we like to celebrate personal wins, especially for those that are joining the job market or just graduating from school, et cetera, and talk about their personal wins, as well as talk about the happenings, like for example, a very popular topic of late has been supply chain attacks and ransomware attacks, so definitely very, very timely there. As well as I'm also involved—being in the cloud security space, I'm bridging, sort of, two worlds between the DevOps world and the security world; more recently, we started up the DevOps Cafecito, which is more focused on the operations side. And that's where, you know, happy to have Ana there as part of that Cafecito and helping out there. Obviously, there, it's a lot of the operations-type topics that we talk about; lots of Kubernetes talk, lots of looking at how the SRE and the DevOps jobs look in different places.And I wouldn't say I'm surprised by it, but it's very nice to see that there is also a big difference with how different organizations think about reliability and operations. And it's varied all over the place and I love it, I love the diversity of it. So anyway, so that's Techqueria, so very happy to be involved with the organization. I also recently took on the role of being the chapter co-director for the San Francisco chapter, so very happy to be involved. As we come out of the pandemic, hopefully, pretty soon here [laugh] right—as we're coming out of the pandemic, I'll say—but looking forward to that in-person connectivity and socializing again in person, so that's Techqueria.So, big plug for Techqueria. As well, I would say for those that are looking at the FinOps world, definitely check out the FinOps Foundation. Very valuable in terms of the folks that are there, the team that leads it, and the resources, if you're looking at getting into FinOps, or at least gaining more control and looking at your spend, not so much like this, but with your eyes wide open. Definitely take a look at a lot of the work that they've done for the FinOps community, and the cloud community in general, on how to take a look at your cloud cost management.Jason: Awesome. Thanks for sharing those. If folks want to follow you on social media, is that something you do?John: Absolutely. Mostly active on LinkedIn at johnmartinez on LinkedIn, so definitely hit me up on LinkedIn.Jason: Well, it's been a pleasure to have you on the show. Thanks for sharing all of your experiences and insight.John: Likewise, Jason. Glad to be here.Jason: For links to all the information mentioned, visit our website at gremlin.com/podcast. If you liked this episode, subscribe to the Break Things on Purpose podcast on Spotify, Apple Podcasts, or your favorite podcast platform. Our theme song is called, “Battle of Pogs” by Komiku, and it's available on loyaltyfreakmusic.com.

American Conservative University
American Thought Leaders- PART 2: Dr. Robert Malone on Ivermectin, Escape Mutants, and the Faulty Logic of Vaccine Mandates. AMERICAN THOUGHT LEADERS

American Conservative University

Play Episode Listen Later Sep 11, 2021 42:59


American Thought Leaders- PART 2: Dr. Robert Malone on Ivermectin, Escape Mutants, and the Faulty Logic of Vaccine Mandates. AMERICAN THOUGHT LEADERS PART 2: Dr. Robert Malone on Ivermectin, Escape Mutants, and the Faulty Logic of Vaccine Mandates In part one of this American Thought Leaders episode, mRNA vaccine inventor Dr. Robert Malone explained the latest research on COVID-19 vaccines, booster shots, and natural immunity. Now in part two, we take a closer look at repurposed drugs like ivermectin and how a universal vaccination policy could actually backfire—and bring about the emergence of vaccine-resistant escape mutants. At their core, vaccine mandates are not just unethical and divisive, but also “impractical and unnecessary,” says Dr. Malone. You can watch the first part of this episode here. Below is a list of references mentioned or related to the discussion in this episode:  “Ivermectin for preventing and treating COVID-19” — The Cochrane Database of Systematic Reviews “Use of Ivermectin Is Associated With Lower Mortality in Hospitalized Patients With Coronavirus Disease 2019” — Chest Journal “Review of the Emerging Evidence Demonstrating the Efficacy of Ivermectin in the Prophylaxis and Treatment of COVID-19” — American Journal of Therapeutics “Effects of Ivermectin in Patients With COVID-19: A Multicenter, Double-Blind, Randomized, Controlled Clinical Trial” — Clinical Therapeutics “Dexamethasone in Hospitalized Patients with Covid-19” — The New England Journal of Medicine “ACTIV-6: COVID-19 Study of Repurposed Medications” — NIH “Convergent antibody responses to the SARS-CoV-2 spike protein in convalescent and vaccinated individuals” — Cell Reports “Reduced sensitivity of SARS-CoV-2 variant Delta to antibody neutralization” — Nature The SARS-CoV-2 Delta variant is poised to acquire complete resistance to wild-type spike vaccines (Note: This is a preprint) “Mutation rate of COVID-19 virus is at least 50 percent higher than previously thought” — Phys.org “Infection and Vaccine-Induced Neutralizing-Antibody Responses to the SARS-CoV-2 B.1.617 Variants” — The New England Journal of Medicine “Why is the ongoing mass vaccination experiment driving a rapid evolutionary response of SARS-CoV-2?” — Trial Site News “The emergence and ongoing convergent evolution of the N501Y lineages coincides with a major global shift in the SARS-CoV-2 selective landscape” (Note: This is a preprint) “The Lambda variant of SARS-CoV-2 has a better chance than the Delta variant to escape vaccines” (Note: This is a preprint) “Imperfect Vaccination Can Enhance the Transmission of Highly Virulent Pathogens” — PLOS Biology “Vaccinated and unvaccinated individuals have similar viral loads in communities with a high prevalence of the SARS-CoV-2 delta variant” (Note: This is a preprint). “Fauci: Amount of virus in breakthrough delta cases ‘almost identical' to unvaccinated” — The Hill CDC: “Outbreak of SARS-CoV-2 Infections, Including COVID-19 Vaccine Breakthrough Infections, Associated with Large Public Gatherings — Barnstable County, Massachusetts, July 2021” “Predominance of antibody-resistant SARS-CoV-2 variants in vaccine breakthrough cases from the San Francisco Bay Area, California” (Note: This is a preprint) “New delta variant studies show the pandemic is far from over” — ScienceNews “Read: Internal CDC document on breakthrough infections” — The Washington Post “New UCSF study: Vaccine-resistant viruses are driving ‘breakthrough' COVID infections” — The Mercury News “Comparing SARS-CoV-2 natural immunity to vaccine-induced immunity: reinfections versus breakthrough infections” (Note: This is a preprint) “Having SARS-CoV-2 once confers much greater immunity than a vaccine—but vaccination remains vital” — Science “Differential effects of the second SARS-CoV-2 mRNA vaccine dose on T cell immunity in naïve and COVID-19 recovered individuals” (Note: This is a preprint) “SARS-CoV-2 variants of concern and variants under investigation in England” — Public Health England “Safety of the BNT162b2 mRNA Covid-19 Vaccine in a Nationwide Setting” — The New England Journal of Medicine “Real-World Study Captures Risk of Myocarditis With Pfizer Vax” — MedPage Today CDC: “Effectiveness of COVID-19 Vaccines in Preventing SARS-CoV-2 Infection Among Frontline Workers Before and During B.1.617.2 (Delta) Variant Predominance — Eight U.S. Locations, December 2020—August 2021” “CDC: Covid-19 Vaccine Effectiveness Fell From 91% To 66% With Delta Variant“ — Forbes “SARS-CoV-2 infection induces long-lived bone marrow plasma cells in humans” — Nature “Causes and consequences of purifying selection on SARS-CoV-2” — Genome Biology and Evolution “The reproductive number of the Delta variant of SARS-CoV-2 is far higher compared to the ancestral SARS-CoV-2 virus” — Journal of Travel Medicine Subscribe to the American Thought Leaders newsletter so you never miss an episode. You can also follow American Thought Leaders on Parler, Facebook, or YouTube. If you'd like to donate to support our work, you can do so here. Follow Epoch TV on Facebook and Twitter. 

American Conservative University
American Thought Leaders- PART 2: Dr. Robert Malone on Ivermectin, Escape Mutants, and the Faulty Logic of Vaccine Mandates. AMERICAN THOUGHT LEADERS

American Conservative University

Play Episode Listen Later Sep 11, 2021 42:59


American Thought Leaders- PART 2: Dr. Robert Malone on Ivermectin, Escape Mutants, and the Faulty Logic of Vaccine Mandates. AMERICAN THOUGHT LEADERS PART 2: Dr. Robert Malone on Ivermectin, Escape Mutants, and the Faulty Logic of Vaccine Mandates In part one of this American Thought Leaders episode, mRNA vaccine inventor Dr. Robert Malone explained the latest research on COVID-19 vaccines, booster shots, and natural immunity. Now in part two, we take a closer look at repurposed drugs like ivermectin and how a universal vaccination policy could actually backfire—and bring about the emergence of vaccine-resistant escape mutants. At their core, vaccine mandates are not just unethical and divisive, but also “impractical and unnecessary,” says Dr. Malone. You can watch the first part of this episode here. Below is a list of references mentioned or related to the discussion in this episode:  “Ivermectin for preventing and treating COVID-19” — The Cochrane Database of Systematic Reviews “Use of Ivermectin Is Associated With Lower Mortality in Hospitalized Patients With Coronavirus Disease 2019” — Chest Journal “Review of the Emerging Evidence Demonstrating the Efficacy of Ivermectin in the Prophylaxis and Treatment of COVID-19” — American Journal of Therapeutics “Effects of Ivermectin in Patients With COVID-19: A Multicenter, Double-Blind, Randomized, Controlled Clinical Trial” — Clinical Therapeutics “Dexamethasone in Hospitalized Patients with Covid-19” — The New England Journal of Medicine “ACTIV-6: COVID-19 Study of Repurposed Medications” — NIH “Convergent antibody responses to the SARS-CoV-2 spike protein in convalescent and vaccinated individuals” — Cell Reports “Reduced sensitivity of SARS-CoV-2 variant Delta to antibody neutralization” — Nature The SARS-CoV-2 Delta variant is poised to acquire complete resistance to wild-type spike vaccines (Note: This is a preprint) “Mutation rate of COVID-19 virus is at least 50 percent higher than previously thought” — Phys.org “Infection and Vaccine-Induced Neutralizing-Antibody Responses to the SARS-CoV-2 B.1.617 Variants” — The New England Journal of Medicine “Why is the ongoing mass vaccination experiment driving a rapid evolutionary response of SARS-CoV-2?” — Trial Site News “The emergence and ongoing convergent evolution of the N501Y lineages coincides with a major global shift in the SARS-CoV-2 selective landscape” (Note: This is a preprint) “The Lambda variant of SARS-CoV-2 has a better chance than the Delta variant to escape vaccines” (Note: This is a preprint) “Imperfect Vaccination Can Enhance the Transmission of Highly Virulent Pathogens” — PLOS Biology “Vaccinated and unvaccinated individuals have similar viral loads in communities with a high prevalence of the SARS-CoV-2 delta variant” (Note: This is a preprint). “Fauci: Amount of virus in breakthrough delta cases ‘almost identical' to unvaccinated” — The Hill CDC: “Outbreak of SARS-CoV-2 Infections, Including COVID-19 Vaccine Breakthrough Infections, Associated with Large Public Gatherings — Barnstable County, Massachusetts, July 2021” “Predominance of antibody-resistant SARS-CoV-2 variants in vaccine breakthrough cases from the San Francisco Bay Area, California” (Note: This is a preprint) “New delta variant studies show the pandemic is far from over” — ScienceNews “Read: Internal CDC document on breakthrough infections” — The Washington Post “New UCSF study: Vaccine-resistant viruses are driving ‘breakthrough' COVID infections” — The Mercury News “Comparing SARS-CoV-2 natural immunity to vaccine-induced immunity: reinfections versus breakthrough infections” (Note: This is a preprint) “Having SARS-CoV-2 once confers much greater immunity than a vaccine—but vaccination remains vital” — Science “Differential effects of the second SARS-CoV-2 mRNA vaccine dose on T cell immunity in naïve and COVID-19 recovered individuals” (Note: This is a preprint) “SARS-CoV-2 variants of concern and variants under investigation in England” — Public Health England “Safety of the BNT162b2 mRNA Covid-19 Vaccine in a Nationwide Setting” — The New England Journal of Medicine “Real-World Study Captures Risk of Myocarditis With Pfizer Vax” — MedPage Today CDC: “Effectiveness of COVID-19 Vaccines in Preventing SARS-CoV-2 Infection Among Frontline Workers Before and During B.1.617.2 (Delta) Variant Predominance — Eight U.S. Locations, December 2020—August 2021” “CDC: Covid-19 Vaccine Effectiveness Fell From 91% To 66% With Delta Variant“ — Forbes “SARS-CoV-2 infection induces long-lived bone marrow plasma cells in humans” — Nature “Causes and consequences of purifying selection on SARS-CoV-2” — Genome Biology and Evolution “The reproductive number of the Delta variant of SARS-CoV-2 is far higher compared to the ancestral SARS-CoV-2 virus” — Journal of Travel Medicine Subscribe to the American Thought Leaders newsletter so you never miss an episode. You can also follow American Thought Leaders on Parler, Facebook, or YouTube. If you'd like to donate to support our work, you can do so here. Follow Epoch TV on Facebook and Twitter. 

Intelligent Design the Future
Why This Virus is No Threat to Intelligent Design

Intelligent Design the Future

Play Episode Listen Later Sep 10, 2021 15:01


On this episode of ID the Future from the vault, host and biologist Ray Bohlin interviews biophysicist Cornelius Hunter, author of Darwin's God, about an article in the journal Science concerning a virus invasion of E. coli bacteria. The article subtitle announces “Natural Selection Caught in the Act,” and suggests that an impressive instance of unguided evolution has been directly witnessed. Not so fast, Hunter says. The results were intelligently designed (by the lab scientists), he notes, and the changes are less impressive than they may appear at first glance. Hunter also explains protein-protein binding and counters evolutionist Dennis Venema to argue that the way the immune system drives change is not analogous to the evolutionary process of random mutations and natural selection. Read More › Source

Lambda3 Podcast
Lambda3 Podcast 264 – O que é WSL?

Lambda3 Podcast

Play Episode Listen Later Sep 10, 2021 77:56


E já que o Linux completou recentemente 30 anos, neste episódio falamos sobre WSL, o subsistema do Windows para Linux. Neste episódio Lambda debatem sobre WSL, o subsistema do Windows para Linux. Entenda o que é e por que foi criado, arquitetura, desempenho, problemas e desenvolvimento. Feed do podcast: www.lambda3.com.br/feed/podcast Feed do podcast somente com episódios técnicos: www.lambda3.com.br/feed/podcast-tecnico Feed do podcast somente com episódios não técnicos: www.lambda3.com.br/feed/podcast-nao-tecnico Lambda3 · #264 - O que é WSL? Pauta: O que é e por que foi criado? Duas versões: v1 e v2 Arquitetura Desempenho Problemas Uso excessivo de memória Relógio sem sync Sem init system (systemd) Desenvolvimento Com VSCode usando a extensão de Remote Com IDEs da Jetbrains Com Visual Studio WSL e Docker/Kubernetes Suporte a apps com interface gráfica O novo terminal do Windows Links Citados: Palestra sobre WSL do Giovanni WSL com interfaces gráficas Problema do relógio do WSL Solução temporária para o problema do relógio do WSL Desenvolvimento remoto com WSL Windows Terminal WSL Config Participantes: Caio Oliveira - @caiovosilva Giovanni Bassi - @giovannibassi Higor Rozan - @HigorRozan Edição: Compasso Coolab Créditos das músicas usadas neste programa: Music by Kevin MacLeod (incompetech.com) licensed under Creative Commons: By Attribution 3.0 - creativecommons.org/licenses/by/3.0

Screaming in the Cloud
Deftly Building for the Customer with Eric Dynowski

Screaming in the Cloud

Play Episode Listen Later Sep 9, 2021 33:14


About EricEric Dynowski, Managing Partner and Chief Solutions Officer at Deft, has been developing software, designing global infrastructures, and managing large technology installations for over 20 years. His background in complex infrastructure design and integration has helped him reduce customer budgets by millions.Links: Deft: https://www.deft.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: You could build you go ahead and build your own coding and mapping notification system, but it takes time, and it sucks! Alternately, consider Courier, who is sponsoring this episode. They make it easy. You can call a single send API for all of your notifications and channels. You can control the complexity around routing, retries, and deliverability and simplify your notification sequences with automation rules. Visit courier.com today and get started for free. If you wind up talking to them, tell them I sent you and watch them wince—because everyone does when you bring up my name. Thats the glorious part of being me. Once again, you could build your own notification system but why on god's flat earth would you do that? Corey: This episode is sponsored in part by Thinkst. This is going to take a minute to explain, so bear with me. I linked against an early version of their tool, canarytokens.org in the very early days of my newsletter, and what it does is relatively simple and straightforward. It winds up embedding credentials, files, that sort of thing in various parts of your environment, wherever you want to; it gives you fake AWS API credentials, for example. And the only thing that these things do is alert you whenever someone attempts to use those things. It's an awesome approach. I've used something similar for years. Check them out. But wait, there's more. They also have an enterprise option that you should be very much aware of canary.tools. You can take a look at this, but what it does is it provides an enterprise approach to drive these things throughout your entire environment. You can get a physical device that hangs out on your network and impersonates whatever you want to. When it gets Nmap scanned, or someone attempts to log into it, or access files on it, you get instant alerts. It's awesome. If you don't do something like this, you're likely to find out that you've gotten breached, the hard way. Take a look at this. It's one of those few things that I look at and say, “Wow, that is an amazing idea. I love it.” That's canarytokens.org and canary.tools. The first one is free. The second one is enterprise-y. Take a look. I'm a big fan of this. More from them in the coming weeks.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. For a while I've been talking about how I started The Duckbill Group as a highly niche, highly focused consultancy aimed at one very expensive problem—the AWS bill—and that's all we do. We don't do implementation; we simply do the thing that it says on the tin. And it turns out that you can get fairly good at the problem like that in not a tremendous amount of time.And today's promoted episode of Screaming in the Cloud. My guest is Eric Dynowski, Chief Solutions Officer and Partner at Deft, which is a consulting company that is almost inverted, in the sense that they do an awful lot of stuff. And we're going to argue about it now. Eric, thank you for taking the time to speak with me today.Eric: Great to be here, Corey. Can't wait to dig in. [laugh].Corey: So, when I started this place back almost five years ago now, I was coming out of an engineering career that I found deeply unsatisfying. I had a bunch of working with computers skill sets, and I wanted to apply them to something that I could use as an independent consultant—because who would ever build a team or a company out of this?—that I could wind up doing repeatedly, so I could, you know, make money, not have a boss, and ideally work less than 80 hours a week. And fixing the AWS bill was the first one I tried; it turned out that, yeah, there is, in fact, an awful lot of expensive business problem hidden in there. And that's what I've been focusing on ever since. I get the distinct impression that your story doesn't quite sound like that one.Eric: Well, it's a little bit like that one, in the sense that I was once an engineer and a technician doing things, and had the crazy idea to go off and start a company that consulted and helped people with their technology needs. And maybe we're a little bit alike in the sense that we also have a very singular focused offering—I'm going to tease you a little bit here, Corey—is that we do one thing: we provide technology solutions for our customers. But probably what you are getting at is, “Well, that's a really broad statement, Eric, what is a technology solution?”Corey: It is. And the reason I did this is because my marketing budget was 50 bucks. So, I wanted something that the Rolodex effect is what I was after, where when someone says at a party to someone else, “Well, I have a problem: my AWS bill.” I want the answer to be, “Hey, I have someone for you to talk to.” You don't generally tend to see that with more broad statements around positioning most of the time.Eric: Sure. Sure.Corey: I also felt like if I was going to be, all right, I'm the best cloud architect advisor ever. Great, now I'm competing against folks like you and the giant consultancies that are in every country. And honestly, those folks have better airport ads than I'm going to be able to put up, at least at the time. Now, I have a platypus for a mascot. So, one wonders whether that would still hold true. You took a very different approach and have done fantastically—Eric: Yeah.Corey: —well with it.Eric: Yeah, I mean, maybe a little bit of history here is helpful. When I started my company, which was Turing Group back in 2013, we did actually focus pretty tightly just on AWS, and that's all we did. We wanted to help customers get in AWS, fix their problems in AWS, scale in AWS, manage their costs, all these sorts of things. And along the way, we had customers coming back to us saying, “We love you guys. You're doing great in [laugh] AWS, but I have other needs too.”And I started saying, “Well, I know a guy over there that does that.” And, “I've got a good friend over here at this company that does it.” And, you know, we're referring back and forth. And kind of parallel to that, one of my business partners who is running a company called Server Central was having the exact inverse problem. And, you know, they were providing managed services within the data center, managed networking capability, things of that nature, you know, helping customers build out an infrastructure to operate at scale, where it's like, “Oh, we need 1000 servers, and we need them really inexpensive, and we have to be able to manage them at scale.”And they were crushing it, doing a great job except his customer start getting back to him saying, “Love what you guys are doing. We're also using AWS, and we want some help with that.” And so, he would pick up the phone and call me and say, “Eric, can you guys help here?” And in some sense, we were competitors. I wanted to move everybody out of the data center into the cloud.And he wanted to move everyone out of the cloud back into the data center or keep them in the data center. And it was like, “Okay, this is weird, but we both have the same problem.” And so we went out to lunch and started this conversation of like, “Well, what if we weren't competitors?” [laugh].Corey: “Maybe there's alignment here.” Yeah. I think Ben Franklin once said that three moves is as good as a fire when it comes to cleaning out old cruft. And migrations are like that. I do want to call out that since I make a frequent practice of saying that multi-cloud as a best practice is foolish, I want to be clear that is in the absence of other constraints.If you're building something new, you probably should pick a provider and go all-in. I don't care which one you do—you might; I don't—but beyond that, at a significant point of scale, when a company says, “All right, we're in one provider—or a data center—and we're going to move to a cloud or other clouds.” Generally, they're correct. They have context that I don't when I'm speaking in the general case. I am not anti-multi-cloud; I am anti-multi-cloud when it is foolish and when it's badly done. Just to set my bias out there and, ideally, avoid getting some letters.Eric: You know, I tend not to disagree with that in the right context as well. When we were doing just AWS only, I think I would have argued that multi-cloud, yeah, there's no place for it. If you're going multi-cloud, you're giving up on all of the greatness that a single public cloud has to offer. You know, and by the greatness, I mean, the proprietary services they offer, and the APIs, and things like SNS and SQS and Route 53, all of those things that you could build into your application and just start using them without having to build all the infrastructure to run it. And so I would agree with you, I think, in that sense.But I wish the world were that simple and I wish the companies that we worked with operated in a nice, clean, unambiguous context. But the more you dig in, I think you realize that when you start dealing with a company, maybe, that has 20,000 employees and offices all across the country, and 25 years of legacy applications—or maybe a vision for the future that just is so massive that it requires a different point of view—and this is really where we engage. And it's interesting that you mentioned about the context, and usually, if a customer approaches us and says, “We want to go multi-cloud,” the first thing we do is put the brakes on and get away from the discussion around multi-cloud and move straight into, “What are you trying to accomplish here?” And navigate that conversation before it turns into—you know, let's not start with the solution type of discussion.Corey: Part of the problem, it seems, that when you start talking to folks about these things, especially in some vendor corners, an awful lot of self-interest that winds up informing the answers that immediately come from it, where it's, “Oh, yeah, you want to go multi-cloud.” And you scratch underneath the surface and the reason is that if you go all-in on one provider, they have nothing left to sell you, in some cases. Other times, it's by a cloud provider themselves pushing multi-cloud strongly because they know that if you go all-in on one provider, it will absolutely not be them. So, the hard part is finding someone who can serve as a trusted advisor. And I mean that in the actual sense of a person, not the crappy AWS service that tells you everything's fine when it isn't. That's ‘Plausible Advisor' at best. Let's be clear here.Eric: Well, come on. If it wasn't for Trusted Advisor, we wouldn't have a market for all the third-party analysis tools [laugh] and people like you to help us manage our costs.Corey: Believe me, if I thought a tool could solve the problem, I would have built it years ago. Tried, turned out it didn't, and well, here we are. You position what you do at Deft as starting from an advisory perspective. You're not pushing a particular product, you're not pushing any particular vendors that I'm aware of, you have a partner list that is not a single vendor, what a concept. So, it's clear that you're doing something that goes one of two ways.Either it is, yeah, we'll take money from anyone who will pay us, or alternately you're approaching it from a thoughtful perspective of trying to figure out what's going on with the customer. Based upon our conversations, I'm going to go ahead and guess it's that one.Eric: It definitely is the latter, Corey. We've certainly had many customers approach us over the years and ask for things that are a bad fit. And a bad fit might be, they're asking for technology experience we don't have. If someone came to us and said, “We want you guys to be Oracle DBAs because you do technology,” our answer is very likely going to be no. Could we learn to be Oracle DBAs? Yeah, we probably could. Do we want to probably not? Maybe?Corey: There are some very qualified Oracle DBAs out in the world, and it's—Eric: There's—Corey: —great.Eric: —there's places for people to specialize. And I think one of our virtues is to know and recognize where we belong and where we don't belong. And the good thing is, not only have we sort of built up our own partner list in terms of technology partners, strategic partners, but we also have our own internal list of referral partners where we know that something's out of our wheelhouse, and I got another company and another team here that I know can crush it and help them out. There's other areas of work that we just don't get into. If you want to outsource your IT and have a company that's going to help you figure out why your printer is not working, definitely not us.You're going to be wasting your money with a firm like us. Or maybe you want to partner in a way that isn't going to take the best advantage of the capabilities that we have, meaning you just want to take advantage of us in a halfway manner, and you want to keep an internal team and the two teams are up against one another and fighting about stuff constantly. We need to have good strong trust between our two companies and our partnerships. And so if we feel like there isn't an opportunity for that, we might walk away from it as well.So, it's not a case of everything that walks in front of us, here's a proposal. [laugh]. We definitely do some opportunity vetting and analysis. We ask a lot of questions upfront about how our customers work, what their internal teams are like, what their expectations of us are, what they want in that relationship. Is it transactional or is it strategic? We're interested in the strategic partnerships with our customers.Corey: I think this is something that is not well understood by a lot of the fly-by-night folks, for lack of a better term. I don't mean to sound disparaging, but the folks who don't seem to understand that long-term reputation is important. I mean, both of our consulting companies, although radically different in focus, have pages on our site where we list reference customers. In fact, there's some overlap between our customers. And as we look at this, you aren't allowed to put a customer logo up if they're going to take umbrage to you doing it, first off. And secondly, you don't want to put a customer logo up if people are going to ask them about their experience with you, and the response is, “Oh, they were crap.” At some point, no, let's not do that.Eric: [laugh]. That's right.Corey: The only way to get there is to deliver on an engagement in such a way that the response is, “That was great. Would you do it again?” “Can I?” And the idea of excitement of delivering an outcome where people who you've worked with become some of your biggest advocates, that's how I always viewed the proper way of building a business.Eric: Yeah. Now, I'd ask you, in terms of when you guys are providing advice to your customers about AWS spend, or someone approaches you, obviously, you're probably first thinking, “Okay, well, how much AWS spend do you have in a month and is this worth my time?” But there's probably another element of evaluation that you must do in terms of is this a good customer for us, and can we do the right thing for them? What are pieces that you guys think about?Corey: Oh, absolutely. As a general rule, we do a lot of AWS contract negotiation. And that is, if you have an AWS offer in front of you for committing to something, come talk to us; it's fun. That's half of our engagements today. The other half are cost optimization projects, and generally speaking, we aren't going to be able to effectively deliver return on investment for much less than about a million bucks a month in spend.So, I do at some point want to explore how to help people who are not already paying a king's ransom to AWS every month, but that is down the road. The next step is a conversation. It's a, “So great, you want to optimize your AWS bill.” And then my favorite is the—I get the quote-unquote, “Dumb question.” “Why? Why do you care? Why is this an actual problem other than it looks like a phone number and your CFO has some questions, what is the actual concern?”Very often, we'll find that it's not that you're spending too much on cloud, in many cases, it's that it's not understood what it's doing. “Okay, the bill is 20% higher this month. Is that new normal? Is that something that's going to inform our planning and we do adjust our expectations for what this is going to cost to run this? Is it just a mistake that someone left up?” The same questions would have arisen if the bill were 20% lower, except somehow it never is.Eric: Right, right. You know what's interesting about that, Corey is, too, also I think that you tend to tease AWS from time to time. And also I think the work you do would not be in Amazon's interest, right? They want customers to spend more money on their infrastructure and their services and their capabilities, and you're helping customers spend less. But what's interesting about that is that we're one of the few managed service partners in the country.And I don't know how much you know about that program, and what it takes to get into that program and to maintain the certification in that program. It's like a three-day audit; there's 500 control items that we have to go through. In fact, it was that program that took our business to the next level. It was that program and its rigor that took what we were doing and actually matured it and turned it into what I would consider a respectable world-class operation. But one of the interesting aspects of that audit is that there's several control items in there that ask us to show Amazon that we are taking steps to manage our customers' costs on AWS and reduce spend. And it's interesting that Amazon is the one pushing that on us and instilling that requirement as we support our customers in Amazon.Corey: It's counterintuitive, but this is one of those areas where there's no one on the other side of this issue. Of course, Amazon wants customers to spend more money with Amazon—I swear the company spends half its time lying awake at night worrying someone who isn't them is making money somewhere, at least that's how it feels some days. But they want that spend to be intelligent. They don't want the narrative to be that the cloud is just as expensive a bunch of nonsense. If there's a bunch of instances that are sitting there idle, they will advise you—if they're on top of the game—to turn them off because that is the goal. They want it to be—Eric: That's how they deliver on their promise. Right.Corey: Well, yeah. Pandemic aside, with most of our customers, what we notice a year after an engagement is that they're in fact spending more than they were when we started. But it's more efficient; it's growth that's tying into this. It becomes a component of cost of goods sold where, “Yeah, we're doing more business, so it costs us more to fulfill that business; we're perfectly happy,” is generally the response to that. And I think that everyone with a vision that extends beyond this quarter's numbers is likely to start to get into that, on some level.Eric: Right.Corey: One thing I do want to ask you is—relevant to what you just said—one thing that we do at The Duckbill Group explicitly is we have no partners, full stop. And the reason behind that is because with what we're doing around billing, and money, and contract negotiation, and the rest, as soon as we have a partner, it suddenly gives rise to a bunch of real or perceived conflicts of interest. And in this particular niche, it makes an awful lot of sense not to do that. Now, if you're in any other arena, where you're in—“Oh, you're in security, for example. Oh, we have no partners with any vendors,” the answer question becomes, “Well, what's wrong with you? What, there's no one willing to trust you? Do you think somehow you're better than all these other people?” It's the wrong answer. So, my question for you is, how do you evaluate whether you should partner with a particular company or not?Eric: Sure. Great question. Deft's reason for existence—when we think about ourselves, we reframe it as our purpose—is to deliver on the promise of technology. And if you unpack that statement a little bit is like, “Well, what the heck is the promise of technology? What does that even mean?”And what we get down to is that technology itself doesn't make any kind of promises. That router you just bought, it doesn't promise really anything; that EC2 instance you just booted in AWS doesn't really ultimately, at the end of the day, promise anything. It's incapable of making promises, but people are. And what we promise is that we can wrangle that technology, we can configure it, we can set it up in a particular kind of way, we can bring in the right components into the solution, and deliver on a promise of, “Yes, you can scale to 100 million users,” or, “Yes, you can reduce latency and improve the customer experience for your customers.” It's all about the people, and that's what we have the most of.And that's the best thing that we have in our house.s we have an inventory of highly qualified, talented, empathetic, compassionate, excited people. So, when we start thinking about our partners and who we want to partner with, what we take into consideration is what technology, tools, and capabilities do our people need to have in their toolbox, such that when we start working with our customers crafting that promise and that solution, we've got the right things at hand and at the right time. And then the second piece of it is, does the partner align well with us in terms of our vision? And in some sense, keeping us relatively technology-neutral, in the same sense that you're trying to stay neutral from that billing perspective and making sure that you're looking out and advocating for your customers first.So, when we're thinking about our partners as well, it's not that, oh, well, we want to build our whole business on top of AWS, or Azure, or in our data center. And those are the on—you know, we try to remove dogma from the picture in that sense, and try to probably be dogmatic mostly about the customer and what it is that they're trying to achieve, and being honest with them. So, it's more of a, “Hey, let's scan the horizon. Let's listen to our customers, let's understand what problems they're trying to solve, what challenges they have today. Let's evaluate the technology options on the table across the world.” Our partner might be [unintelligible 00:18:36], it might be VM, or it might be Amazon, it might be a small little company somewhere that does a niche service. But our job is to come together with all of those things and present a cohesive solution.Corey: And it's clearly working. You were the Turing Group and you wound up partnering with—you said your business partner—who was over at Server Central, which I'm just guessing from the name and assuming I hadn't paid attention to the industry for a while, sounds like it might not be fully cloud-focused, on some level, given the name. What did they do? And why was merging the right answer?Eric: Yeah. I mean, it's funny that you bring that up. Server Central. Wow, a server; who's talking about servers anymore, right? It's containers and virtualization and—Corey: But they've got to run somewhere.Eric: That's right. Serverless applications. Like, hmm, is this the right name? And I think it speaks to the 20-year heritage that Server Central has had and how they built their business. And they do and did, and we do have a cloud focus that's not related to the public cloud.We have a significant number of customers that operate on private clouds that we've built for them and manage for them, for various reasons. Some are legit and some maybe not so legit and mostly about how they feel about something. And some of them are technically driven. And after we brought the companies together, we realized that hey, you know what, we have a lot of brand equity and history and Server Central and we have to respect that. Turing Group had its own set of brand equity in the market that we had established and promoted a certain kind of ideology and thinking.And so for a short period of time, we were a little bit unsure of how are we going to bring this together in any kind of cohesive fashion? And it actually went out into the world for about a year as Server Central Turing Group. And I think my tongue twisted as I said it, [laugh]. It's a lot of words and it mostly just confuses people and makes them scratch their head. And so we went off on a journey to figure out what our reimagined new company is going to look like with all the combined services and capabilities that we have.And that's how we arrived at Deft and the idea that's how we want to engage with our customers; that's what we want our solutions to be like; that's what we want the experience in working with us to be. And we want to remove the friction and anxiety that technology can bring. It reminds me of when I was starting my first company. I spent a lot of time sort of navel-gazing, saying, “What do I like about this, and why am I doing this?” And went back to my early days as an engineer, and at the core of it was a really simple idea and it was the idea that when I helped somebody with a technology problem, they were elated. They thought it was magic. They thought it was black magic.They didn't understand how I took this goofy, strange, cryptic thing and made it do what they wanted, and I did it quickly and I did it deftly. And there was joy and they were happy and I loved that; I loved that response. I loved knowing that I helped fix this mysterious problem for somebody that just didn't even know where to begin. And I did it time and time again and it helped me grow my career. And when we started the first company, it was sort of like, I want to continue that feeling.I want to create that feeling for our customers where they feel like maybe they're stuck with some crazy complex technology problem, and because I happen to have the innate skill for understanding these things and figuring these things out and I have a team that can do it, we can create that same feeling for our customers. And we want to continue doing that today.Corey: This episode is sponsored by our friends at Oracle HeatWave is a new high-performance accelerator for the Oracle MySQL Database Service. Although I insist on calling it “my squirrel.” While MySQL has long been the worlds most popular open source database, shifting from transacting to analytics required way too much overhead and, ya know, work. With HeatWave you can run your OLTP and OLAP, don't ask me to ever say those acronyms again, workloads directly from your MySQL database and eliminate the time consuming data movement and integration work, while also performing 1100X faster than Amazon Aurora, and 2.5X faster than Amazon Redshift, at a third of the cost. My thanks again to Oracle Cloud for sponsoring this ridiculous nonsense. Corey: I do want to point out that, at least in my mind, there's always a little bit of, I guess, we call it technical elitism, on some level, where, “Oh, someone is working through a partner. They must be a company that's stuck in the past.” But a glance at companies that you're working with, make it very clear that's not the case. I mean, Ars Technica, New Relic, Wildbit. You've got some companies that are very forward-looking, and by no definition are these companies that don't understand cloud or understand how the internet works. It's something that I think is not fully understood among a subset of the industry that, in many cases, having a third-party partner, in many cases winds up helping you go faster, further.Eric: Yeah. It might be cliche to say—oftentimes, in cliches, there's a little bit of truth—which is, focus on what you're good at, and focus on what you're best at, and focus on your core products. And with a lot of these technology companies where you might read on the surface that, “Oh, yeah, they're a smart bunch over there. Why do they need a partner?” Technology is complicated. The stack is deep.Whether you're talking about deployment pipelines, or should I use a fiber connection on this or should I use copper? Or should we have jumbo frames enabled? Or should we be using API Gateway and Lambda functions for this? I just listed a broad range of technologies and things that solve different problems. And these customers have their own products that they have to put out into the world; those products need to be meaningful and thoughtful and aligned with their customers.And because that technology stack is complex and deep, it creates an opportunity for companies like ours—for partners—to step in, and grab a piece of that complexity, and manage it, and handle it, and help a customer with it to create the space for them to create the most excellent product. And so even though they are technology companies because you're managing this big wrangling layered technologies, abstractions, and—well, even when we talk about containerization, right, and running a small application, there's seven layers before you get to the CPU. [laugh]. And within that seven layers, there's I don't know how many lines of code, and there's how many hidden assumptions and configuration files, and you name it. And there's areas of that entire stack that we're really good at and customers derive value from that.Corey: One area that you've been relatively active within is the hybrid universe. My talking point on that has generally looked a lot like the snarky take of, “Well, you have a company in a data center today, and they're going to go all-in on the cloud. And it turns out halfway through that it's hard to move some workloads. There is no AWS/400 and they have a mainframe.” So, what are they going to do? They give up halfway, plant a flag, declare victory, and now we're hybrid as a best practice. That is not entirely accurate, but there's an element of accuracy in some cases to it.Eric: Yeah.Corey: But I don't get the sense that's how you see it. I'm left with a strong impression that it's a very intentional choice for some of your customers, that in some cases, workloads that are live in data centers, were at one point living in a cloud provider. Talk to me about that.Eric: Yeah, I like to think of it not as, like, a binary situation. And something that exists on degrees, and often times has a lot to do with the lifecycle of a product or company and the scale of a company. And we touched on this earlier in our conversation, which is that if someone approached us—maybe a startup or a smaller company—trying to migrate off of half-dozen servers and move into the public cloud, and they approached us and said, “Yeah, we need to be hybrid for this thing,” I would probably question that and I would question it really hard, and say, “Really, what are you going after here? You're going to give up a lot if you choose to go hybrid, and you won't really take advantage of some of the amazing opportunities that a full-on single cloud solution has to offer.” On the flip side, we've seen companies that started like that, were a hundred percent in public cloud on a single provider, everything's working fine.There was no issues whatsoever, except the bill, or maybe a fear of what the bill could be. And this is something that happens at scale. There's just a point where the public cloud just doesn't make sense anymore, even despite those benefits. And for the bottom line and in terms of the margins and your cost of revenue, giving up some of those additional benefits that allow you to grow and scale is worth it. And if you look at the technology landscape, there's a reason Facebook's not in AWS. [laugh].There's a reason a lot of these larger technology companies where we have hundreds of millions of users or bazillions of petabytes of data, that move out and get out of there. I mean, look at Dropbox right? [laugh]. Is Dropbox storing all their data in the public cloud? Not really, and they are a public cloud in a sense on their own, right?Corey: Yeah, they did just launch a 34-petabyte data warehouse for analytics on AWS, and they've made a bunch of big—Eric: Yeah, yeah. I mean—Corey: —deals out of that, but the core storage workload, yeah, that does not economically make sense, given their access patterns and how they have built that offering. So yeah, that is a very well understood, very specific, very niche workload. Yeah, that does not belong in AWS.Eric: We've even gone as far as launching our own multi-petabyte managed object storage solution, it's totally S3 compatible. It works identically to the way S3 works, but we have customers that actually can do better on our platform, either because we can provide lower latency, we can provide custom contracts that aren't just purely pay as you go; there's all kinds of different options that we can give our customers that are more custom and tailored to their needs that you're going to get from, “Here's your API keys have fun.” [laugh]. And so there's still a market for that stuff and there's still a need for that stuff.Corey: There really is.Eric: So, the answer is it depends, Corey. [laugh].Corey: It seems to be the answer to any nuanced question. So, if people want to learn more about what you're doing at Deft, and potentially whether it might help them with some of the challenges they're facing, where can they find you?Eric: Easy. deft.com, D-E-F-T dot com. Great, short four-letter domain name that you wouldn't believe what we had to go through to get. [laugh].Corey: I can only imagine. Thank you so much for taking the time to speak with me today. I really appreciate it.Eric: You're welcome, Corey.Corey: Eric Dynowski, Chief Solutions Officer and Partner at Deft. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you hated this podcast, please leave a five-star review on your podcast platform of choice, along with a comment telling me that no, customers should in fact go all-in on your third-rate cloud.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Revelations Radio Network
LITTLE HORN UNICORN

Revelations Radio Network

Play Episode Listen Later Sep 9, 2021


Canary Cry News Talk ep. 386 - 09.08.2021 - LITTLE HORN UNICORN: Matrix 4, NIH Exposed, New Navy Weapon - CCNT 386 Our LINK TREE: CanaryCry.Party SUBSCRIBE TO US ON: NewPodcastApps.com PAYPAL: https://bit.ly/3v59fkR MEET UPS: https://CanaryCryMeetUps.com Basil's other project: Ravel Podcast   INTRO Matrix 4 Trailer (Variety) El Salvador Bumpy Bitcoin Day and Dip (Reuters)   FLIPPY Chinese robot unicorn (Engadget) Rabbit Trail: Bible study on Unicorns, Lion and Unicorn, Finance, [F]Unicorn   WACCINE/PANDEMIC SPECIAL   Narrative A Fauci says hospitals will have to make tough decisions (Clip) Biden plans to combat Delta, announcement tomorrow (CNN) Reduced to meme tweets (meme 1, meme 2, meme 3) Note: Pinned Twitter C19 Newsfeed all from several months ago   Clash of the Narratives Rolling Stone story about hospitals overwhelmed with overdose of Ivermectin turns out false (Fox)   Narrative B 900 pages released under FOIA, gain of function funded by NIH (The Intercept) -Rutgers Board of Governor, Prof of Chem and BioChem RAILS against Three Amigos (Tweet) Top US Scientists say they knew 2 weeks before breakout of C19 (DailyMail) C19 cases plummet in South America where Mu and Lambda broke out (Newsweek) ICAN vs CDC lawsuit   BREAK (producer party)   GEOENGINEERING/BUILD BACK BETTER Biden doom and gloom on climate change (Clip) Biden says they no longer call it Tornado's (Clip)   BEING WATCHED New Navy Weapon stops you from Talking

Canary Cry News Talk
LITTLE HORN UNICORN

Canary Cry News Talk

Play Episode Listen Later Sep 9, 2021 248:27


Canary Cry News Talk ep. 386 - 09.08.2021 - LITTLE HORN UNICORN: Matrix 4, NIH Exposed, New Navy Weapon - CCNT 386 Our LINK TREE: CanaryCry.Party SUBSCRIBE TO US ON: NewPodcastApps.com PAYPAL: https://bit.ly/3v59fkR MEET UPS: https://CanaryCryMeetUps.com Basil's other project: Ravel Podcast   INTRO Matrix 4 Trailer (Variety) El Salvador Bumpy Bitcoin Day and Dip (Reuters)   FLIPPY Chinese robot unicorn (Engadget) Rabbit Trail: Bible study on Unicorns, Lion and Unicorn, Finance, [F]Unicorn   WACCINE/PANDEMIC SPECIAL   Narrative A Fauci says hospitals will have to make tough decisions (Clip) Biden plans to combat Delta, announcement tomorrow (CNN) Reduced to meme tweets (meme 1, meme 2, meme 3) Note: Pinned Twitter C19 Newsfeed all from several months ago   Clash of the Narratives Rolling Stone story about hospitals overwhelmed with overdose of Ivermectin turns out false (Fox)   Narrative B 900 pages released under FOIA, gain of function funded by NIH (The Intercept) -Rutgers Board of Governor, Prof of Chem and BioChem RAILS against Three Amigos (Tweet) Top US Scientists say they knew 2 weeks before breakout of C19 (DailyMail) C19 cases plummet in South America where Mu and Lambda broke out (Newsweek) ICAN vs CDC lawsuit   BREAK (producer party)   GEOENGINEERING/BUILD BACK BETTER Biden doom and gloom on climate change (Clip) Biden says they no longer call it Tornado's (Clip)   BEING WATCHED New Navy Weapon stops you from Talking (Interesting Engineering) Tsunami of text spam “just the beginning,” how to make it stop (Insider)   BEAST SYSTEM Why won't Christians get jabbed? (Relevant Mag)   ADDITIONAL STORIES VAERS story of mother losing her baby (VAERS) When life begins is a matter of politics, not biology (Salon)   PRODUCERS ep. 385-386: Zachary R, William J, InkSpot, Aaron J, Sir Casey, Sir Sigrah the Beast, JC, Christian N, Juan A, Big Tank, Menace Sensei, Gail M, Volodymyr S, Craig B, Scott K, Veronica D, Shane & Ragen, Brandt W, Runksmash, Daniel S, Ciara, Pizza Rat, Aki K, Mr Green, Marcus   TIMESTAMPS: Jade Bouncerson Christine C   JINGLES: LeirBag3000   ART: Dame Allie of the Skillet Nation Sir Dove, Knight of Rustbeltia Juan A Pizza Rat Alex Guzman

Café Con Nata
Panel Feminista con la dibujante Sol Díaz

Café Con Nata

Play Episode Listen Later Sep 8, 2021 82:10


Mitad de semana en Café Con Nata y hoy, junto a nuestra editora Sol Abarca, te actualizamos de las últimas novedades en relación a las vacunas para niñxs y la circulación de variantes en América Latina. Por ejemplo, ¿sabías que la variante Mu, que fue detectada por primera vez en Colombia, ya supera a Lambda y a Delta, y que ya es la segunda de mayor circulación en Chile? En política nacional, te contamos que ingresaron formalmente el proyecto de reforma constitucional para permitir renuncias de convencionales y parlamentarios. Y desde México, noticias esperanzadoras parala lucha feminista porque la Suprema Corte despenalizó la interrupción voluntaria del embarazo en un fallo histórico para el país. En un nuevo Panel Feminista presentado por Corporación Humanas, recibimos a la dibujante Sol Díaz para hablar de “La Constitución entre líneas”, un proyecto del Instituto Francés de Chile que invita a reflexionar sobre el proceso histórico actual a través de la ilustración.

Screaming in the Cloud
Building and Maintaining Cultures of Innovation with Francessca Vasquez

Screaming in the Cloud

Play Episode Listen Later Sep 8, 2021 44:05


About FrancescaFrancessca is the leader of the AWS Technology Worldwide Commercial Operations organization. She is recognized as a thought leader of business technology cloud transformations and digital innovation, advising thousands of startups, small-midsize businesses, and enterprises. She is also the cofounder of AWS workforce transformation initiatives that inspire inclusion, diversity, and equity to foster more careers in science and technology.Links: Twitter: https://twitter.com/FrancesscaV/ LinkedIn: https://www.linkedin.com/in/francesscavasquez/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by “you”—gabyte. Distributed technologies like Kubernetes are great, citation very much needed, because they make it easier to have resilient, scalable, systems. SQL databases haven't kept pace though, certainly not like no SQL databases have like Route 53, the world's greatest database. We're still, other than that, using legacy monolithic databases that require ever growing instances of compute. Sometimes we'll try and bolt them together to make them more resilient and scalable, but let's be honest it never works out well. Consider Yugabyte DB, its a distributed SQL database that solves basically all of this. It is 100% open source, and there's not asterisk next to the “open” on that one. And its designed to be resilient and scalable out of the box so you don't have to charge yourself to death. It's compatible with PostgreSQL, or “postgresqueal” as I insist on pronouncing it, so you can use it right away without having to learn a new language and refactor everything. And you can distribute it wherever your applications take you, from across availability zones to other regions or even other cloud providers should one of those happen to exist. Go to yugabyte.com, thats Y-U-G-A-B-Y-T-E dot com and try their free beta of Yugabyte Cloud, where they host and manage it for you. Or see what the open source project looks like—its effortless distributed SQL for global apps. My thanks to Yu—gabyte for sponsoring this episode. Corey: And now for something completely different!Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. It's pretty common for me to sit here and make fun of large cloud companies, and there's no cloud company that I make fun of more than AWS, given that that's where my business generally revolves around. I'm joined today by VP of Technology, Francessca Vasquez, who is apparently going to sit and take my slings and arrows in person. Francessca, thank you for joining me.Francessca: Hi, Corey, and thanks for having me. I'm so excited to spend this time with you, snarking away. I'm thrilled.Corey: So, we've met before, and at the time you were the Head of Solutions Architecture and Customer Solutions Management because apparently someone gets paid by every word they wind up shoving into a job title and that's great. And I vaguely sort of understood what you did. But back in March of this year, you were promoted to Vice President of Technology, which is both impressive, and largely non-descriptive when one works for a technology company. What is it you'd say it is you do now? And congratulations, by the way.Francessca: Thank you, I appreciate it. By the way, as a part of that, I also relocated to our second headquarters, so I'm broadcasting with you out of HQ2, or Arlington, Virginia. But my team, essentially, we're a customer-facing organization, Corey. We work with thousands of customers all over the globe, from startups to enterprises, and we ultimately try to ensure that they're making the right technology architecture decisions on AWS. We help them in driving people and culture transformation when they decide to migrate onto the cloud.And the last thing that we try to do is ensure that we're giving them tools so that they can build cultures of innovation within the places that they work. And we do this for customers every day, 365 days a year. And that's what I do. And I've been doing this for over 20 years, so I'm having a blast.Corey: It's interesting because when I talk to customers who are looking at what their cloud story is going to be—not just where it is, but where they're going—there's a shared delusion that they all participate in—and I'm as guilty as anyone. I have this same, I guess, misapprehension as well—that after this next sprint concludes, I'm going to suddenly start making smart decisions; I'm going to pay off all of my technical debt; I'm going to stop doing this silly thing and start doing the smart thing, and so on and so forth. And of course, it's a myth. That technical debt is load-bearing; it's there for a reason. But foundationally, when talking to customers at different points along their paths, I often find that the conversation that I'm having with them is less around what they should be doing differently from a tactical and execution perspective and a lot more about changing the culture.As a consultant, I've never found a way to successfully do that, that sticks. If I could I'd be in a vastly different, vastly more lucrative consulting business. But it seems like culture is one of those things that, in my experience, has to be driven from within. Do you find that there's a different story when you are speaking as AWS where, “Yeah, we're outsiders, but at the same time, you're going to be running production on us, which means you're our partner whether you want to be or not because you can't treat someone who owns production as a vendor anymore.” Does that position you better to shift culture?Francessca: I don't know if it positions us better. But I do think that many organizations, you know, all of them are looking at different business drivers, whether that be they want to move to more digital, especially since we're going through COVID-19 and coming out of it. Many of them are looking at things like cost reduction, some organizations are going through mergers and acquisitions. Right now I can tell you new customer experiences driven by digital is pretty big, and I think what a lot of companies do, some of them want to be the north star; some of them aspire to be like other companies that they may see in or outside the industry. And I think that sometimes we often get a brand as having this culture of innovation, and so organizations very much want to understand what does that look like: what are the ingredients on being able to build cultures of innovation?And sometimes organizations take parts of what we've been able to do here at AWS and sometimes they look at pieces from other companies that they view as north star, and I see this across multiple industries. And I think the one that is the toughest when you're trying to drive big change—even with moving to the cloud—oftentimes it's not the services or the tech. [smile]. It's the culture. It's people. It's the governance. And how do you get rallied around that? So yeah, we do spend some time just trying to offer our perspective. And it doesn't always mean it's the right one, but it certainly has—it's worked for us.Corey: On some level, I've seen cloud adoptions stall, in some scenarios, by vendors being a little too honest with the customer, if that doesn't—Francessca: Mmm. Mm-hm.Corey: —sound ridiculous, where it's—so they take the customer will [unintelligible 00:05:24], reasonable request. “Here's what we built. Here's how we want to migrate to the cloud. How will this work in your environment?” And the overly honest answer from a certain provider—I don't feel the need to name at the moment—is, “Well, great. What you've written is actually really terrible, and if you were to write it better, with smarter engineers, it would run great in the cloud. So, do that then call us.”Surprisingly, that didn't win the deal, though it was, unfortunately, honest. There was a time where AWS offerings were very much aligned with that, and depending on how you wind up viewing what customers should be doing is going to depend on what year it was. In the early days, there was no persistent storage on EC2—Francessca: Mm-hm.Corey: So, if you had a use case that required there had to be a local disk that could survive a reboot, well, that wasn't really the place for you to run. In time, it has changed, and we're still seeing that evolution to the point where there are a bunch of services that come out on a consistent, ongoing basis that the cloud-native set will look at and say, “Oh, that hasn't been written in the last 18 months on the latest MacBook and targeting the developer version of Chrome. Then why would I ever care about that?” Yeah, there's a bigger world than San Francisco. I'm sorry but it's true.And there are solutions that are aimed at customer segments that don't look anything like a San Francisco startup. And it's easy to look at those and say, “Oh, well, why in the world would I wind up needing something like that?” And people point at the mainframe and say, “Because of that thing.” Which, “Well, what does that ancient piece of crap do?” “Oh, billions a year in revenue, so maybe show some respect.” ‘Legacy,' the condescending engineering term for ‘it makes money.'Francessca: [smile]. Yeah, well, first off, I think that our approach today is you have to be able to meet customers where they are. And there are some customers, I think, that are in a position where they've been able to build their business in a far more advanced state cloud-natively, whether that be through tools like serverless, or Lambda, et cetera. And then there are other organizations that it will take a little longer, and the reason for that is everyone has a different starting point. Some of their starting points might be multiple years of on-premise technology.To your point, you talked about tech debt earlier that they've got to look at and in hundreds of applications that oftentimes when you're starting these journeys, you really have to have a good baseline of your application portfolio. One of my favorite stories—hopefully, I can share this customer name, but one of my favorite stories has been our organization working with Nationwide, who sort of started their journey back in 2017 and they had a goal, a pretty aggressive one, but their goal is about 80% of their applications that they wanted to get migrated to the cloud in, like, three to four years. And this was, like, 319 different migrations that we started with them, 80 or so production cut-overs. And to your point, as a result of us doing this application portfolio review, we identified 63 new things that needed to be built. And those new things we were able to develop jointly with them that were more cloud-native. Mainframe is another one that's still around, and there's a lot of customers still working on the mainframe. We work with a very—Corey: There is no AWS/400 yet.Francessca: [smile]. There is no AWS [smile] AS/400. But we do have mainframe migration competency partners to help customers that do want to move into more–I don't really prefer the term modernize, but more of a cloud-native approach. And mostly because they want to deliver new capability, depending on what the industry is. And that normally happens through applications.So yeah, I think we have to meet customers where they are. And that's why we think about our customers in their stage of cloud adoption. Some that are business-to-consumer, more digital native-based, you know, startups, of course; enterprises that tend to be global in nature, multinational; ISVs, independent software vendors. We just think about our customers differently.Corey: Nationwide is such a great customer story. There was a whole press release bonanza late last year about how they selected AWS as their preferred cloud provider. Great. And I like seeing stories like that because it's easy on some level—easy—to wind up having those modernized startups that are pure web properties and nothing more than that—not to besmirch what customers do, but if you're a social media site, or you're a streaming video company, et cetera, it feels differently than it does—oh, yeah, you're a significantly advanced financial services and insurance company where you're part of the Fortune 100. And yeah, when it turns out that the computers that calculate out your amortization tables don't do what you think they're going to do, those are the kinds of mistakes that show. It's a vote of confidence in being able to have a customer testimonial from a quote-unquote, “More serious company.” I wouldn't say it's about modernization; I'd say it's about evolution more than anything else.Francessca: Yeah, I think you're spot on, and I also think we're starting to see more of this. We've done work at places like GE—in Latin America, Itaú is the bank that I was just referring to on their mainframe digital transformation. Capital One, of course, who many of the audience probably knows we've worked with for a long time. And, you know, I think we're going to see more of this it for a variety of reasons, Corey. I think that definitely, the pandemic has played some role in this digital acceleration.I mean, it just has; there's nothing I can say about that. And then there are some other things that we're also starting to see, like sustainability, quite frankly, is becoming of interest for a lot of our customers as well, and as I mentioned earlier, customer experience. So, we often tend to think of these migration cloud journeys as just moving to infrastructure, but in the first part of the pandemic, one of the interesting trends that we also saw was this push around contact centers wanting to differentiate their customer experience, which we saw a huge increase in Amazon Connect adoption as well. So, it's just another way to think about it.Corey: What else have you seen shift during the pandemic now that we're—I guess, you could call it post-pandemic because here in the US, at least at this time of this recording, things are definitely trending in the right direction. And then you take a step back and realize that globally we are nowhere near the end of this thing on a global stage. How have you seen what customers are doing and how customers are thinking about things shift?Francessca: Yeah, it's such a great question. And definitely, so much has changed. And it's bigger than just migrations. The pandemic, as you rightfully stated, we're certainly far more advanced in the US in terms of the vaccine rollout, but if you start looking at some of our other emerging markets in Asia Pacific, Japan, or even AMEA, it's a slower rollout. I'll tell you what we've seen.We've seen that organizations are definitely focused on the shift in their company culture. We've also seen that digital will play a permanent fixture; just, that will be what it is. And we definitely saw a lot of growth in education tech, and collaboration companies like Zoom here in the US. They ended up having to scale from 10 million daily users up to, like, 300. In Singapore, there is an all-in company called Grab; they do a lot of different things, but in their top three delivery offerings—what they call Grabfood, Grabmart, and GrabExpress—they saw, like, an increase of 30% user adoption during that time, too.So, I think we're going to continue to see that. We're also going to continue to see non-technical themes come into play like inclusion, diversity, and equity in talent as people are thinking about how to change and evolve their workforce. I love that term you used; it's about an evolution: workforce and skills is going to be pretty important. And then globally, the need around stronger data privacy and governance, again, is something else that we've started to see in a post-COVID kind of era. So, all industries; there's no one industry doing anything any different than the others, but these are just some observations from the last, you know, 18 months.Corey: In the early days of the pandemic, there was a great meme that was going around of who was the most responsible for your digital transformation: CIO, CTO, or COVID-19?Francessca: [smile].Corey: And, yeah, on some level, it's one of those ‘necessity breeds innovation' type of moments. And we're seeing a bunch of acceleration in the world of digital adoption. And I don't think you get to put the genie back in that particular bottle in a bunch of different respects. One area that we're seeing industry-wide is talent discovering that suddenly you can do a whole bunch of things that don't require you being in the same eight square miles of an earthquake zone in California. And the line that I heard once that really resonated with me was that talent is evenly distributed; opportunity is not. And it seems that when you see a bunch of companies opening up to working in new ways and new places, suddenly it taps a bunch of talent that previously was considered inaccessible.Francessca: That's right. And I think it's one of those things where—[smile] I love the meme—you'll have to send me that meme by the way—that just by necessity, this has been brought to the forefront. And if you just think about the number of countries that, sort of, account for almost half the global population, there's only, like, we'll say eight of them that at least represent close to 60-plus percent. I don't think that there's a company out there today that can really build a comprehensive strategy to drive business agility or to look at cost, or any of those things digitally without having an equally determined workforce strategy. And that workforce strategy, how that shows up with us is through having the right skills to be able to operate in the cloud, looking at the diversity of where your customer base is, and making sure that you're driving a workforce plan that looks at those markets.And then I think the other great thing—and honestly, Corey, maybe why I even got into this business—is looking at, also, untapped talent. You know, technology's so pervasive right now. A lot of it's being designed where it's prescriptive, easier to use, accessible. And so I also think we're tapping into a global workforce that we can reskill, retrain, in all sorts of different facets, which just opens up the labor market even more. And I get really excited about that because we can take what is perceived as, sort of, traditional talent, you know, computer science and we can skill a lot of people who have, again, non-traditional tech backgrounds. I think that's the opportunity.Corey: Early on in my career, I was very interested in opening the door for people who looked a lot like me, in terms of where their experience level was, what they'd done because I'd come from a quote-unquote, non-traditional background; I don't even have a high school diploma at this point. And opening doors for folks and teaching them to come up the way that I did made sense for a while. The problem that I ran into pretty quickly is that the world has moved on. It turns out that if you want to start working in cloud in 2021, the path I walked is closed. You don't get to go be an email systems administrator who's really good at Unix and later Linux as your starting point because those jobs don't exist the way that they once did.Before that, the help desk roles aren't really there the way that they once were either, and they've become much more systematized. You don't have nearly as much opportunity to break the mold because now there is a mold. It used to be that we were all these artisanally crafted, bespoke technologists. And now there are training curriculums for this. So, it leads to a recurring theme on the show of, where does the next generation really wind up coming from?Because trying to tell people to come up the way that I did is increasingly reminiscent of advice of our parents' generation, “Oh, go out and pound the bricks, and have a firm handshake, and hand your resume to the person at the front desk, and you'll get a job today.” Yeah, sure you will. How do you see it?Francessca: You know, I see it where we have an opportunity to drive this talent, long-term, in a variety of different places. First off, I think the personas around IT have shifted quite a bit where, back in the day, you had a storage admin, a sysadmin, maybe you had a Solaris, .NET, Linux developer. But pretty straightforward. I think now we've evolved these roles where the starting point can be in data, the starting point can be in architecture.The personas have shifted from my perspective, and I think you have more starting points. I also think our funnel has also changed. So, for people that are going down the education route—and I'm a big proponent of that—I think we're trying to introduce more programs like AWS Educate, which allows you to go and start helping students in universities really get a handle on cloud, the curriculum, all the components that make up the technology. That's one. I think there are a lot of people that have had career pivots, Corey, where maybe they've taken time out of the workforce.We disproportionately, by the way, see this from our female and women who identify, coming back to the workforce, maybe after caring for parents or having children. So, we've got—there are different programs that we try to leverage for returners. My family and I, we've grown up all around the military veterans as well, and so we also look at when people come out of, perhaps in the US, military status, how do we spend time reskilling those veterans who share some of the same principles around mission, team, the things that are important to us for customers. And then to your point, it's reskill, just, non-traditional backgrounds. I mean, a lot of these technologies, again, they're prescriptive; we're trying to find ways to make them certainly more accessible, right, equitable sort of distribution of how you can get access to them.But, anyone can start programming in things like Python now. So, reskill non-traditional backgrounds; I don't think it's just one funnel, I think you have to tap into all these funnels. And that's why, in addition to being here in AWS, I also try to spend time on supporting and volunteering at nonprofit companies that really drive a focus on underserved-based communities or non-traditional communities as different pathways to tech. So, I think it's all of the above. [smile].Corey: This episode is sponsored in part by CircleCI. CircleCI is the leading platform for software innovation at scale. With intelligent automation and delivery tools, more than 25,000 engineering organizations worldwide—including most of the ones that you've heard of—are using CircleCI to radically reduce the time from idea to execution to—if you were Google—deprecating the entire product. Check out CircleCI and stop trying to build these things yourself from scratch, when people are solving this problem better than you are internally. I promise. To learn more, visit circleci.com.Corey: Yeah, I have no patience left, what little I had at the beginning, for gatekeeping. And so much of technical interviewing seems to be built around that in ways that are the obvious ones that need not even be called out, but then the ones that are a little bit more subtle. For example, the software developer roles that have the algorithm questions on a whiteboard. Well, great. You take a look at the average work of software development style work, you don't see those things coming up in day-to-day. Usually.But, “Implement quicksort.” There's a library for that. Move on. So, it turns out that biases for folks who've recently had either a computer science formal education or computer science formal-like education, and that winds up in many ways, weeding people out have been in the workforce for a while. I take a look at some of the technical interviews I used to pass for grumpy Unix sysadmin jobs; I don't remember half of the terminology.I was looking through some my old question lists of what I used to ask candidates, and I don't remember how 90% of this stuff works. I'd have to sit there and freshen up on it if I were to go and take a job interview. But it doesn't work in the same way. It's more pernicious than that, though, because I look at what I do and how I approach it; the skills you use in a job interview are orthogonal, in many cases, to the skills you'll need in the workforce. How someone performs with their career on the line at a whiteboard in front of a few very judgy, judgy people is not representative of how they're going to perform in a collaborative technical environment, trying to solve an interesting problem, at least in my experience.Francessca: Yeah, it's interesting because in some of our programs, we have this conversation with a lot of the universities, as well, in their curriculums, and I think ultimately, whether you're a software developer, or you're an architect, or just in the field of tech and you're dealing with customers, I think you have to be very good at things like problem-solving, and being able to work in teams. I have a mental model that many of the tech details, you can teach. Those things are teachable.Corey: “Oh, you don't know what port some protocol listens on. Oh, it's a shame you never going to be able to learn that. You didn't know that in the interview off the top of your head and there's no possible way you could learn that. It's an intrinsic piece of knowledge you're born with.” No, it's not.Francessca: [smile]. Yeah, yeah, those are still things every now and then I have to go search for, or I've written myself some nice little Textract. Uh… [smile] [unintelligible 00:22:28] to go and search my handwritten notes for things. But yeah, so problem-solving, being able to effectively communicate. In our case, writing has been a muscle that I've really had to work at hard since joining here.I haven't done that in a while, so that is a skill that's come back. And I think the one that I see around software development is, really, teams. It's interesting because when you're going through some of the curriculums, a lot of the projects that are assigned to you are individual, and what happens when you get into the workplaces, the projects become very team-oriented, and they're more than one people. We're all looking at how we publish code together to create a process, and I think that's one of the biggest surprises making a transition [smile] into the workforce is, you will work in teams. [smile].Corey: Oh, dear Lord. The group project; the things that they do in schools is one of those, great, there's one person who's going to be diligent—which was let's be clear, never me—they're going to do 90% of the work on it and everyone shares credit equally. The real world very rarely works that way with that sense of one person carries the team, at least ideally. But on the other side of it, too, you don't wind up necessarily having to do these things alone, you don't have to wind up with dealing with those weird personal dynamics in small teams, for the most part, and setting people up with the expectation, as students, that this is how the real world works is radically different. One of the things that always surprised me growing up was hearing teachers in middle school and occasionally beyond, say things like, “When you're in the real world”—always ‘the real world' as if education is somehow not the real world—that, “Oh, your boss is never going to be okay with this, or that, or the other thing.”And in hindsight, looking back at that almost 30 years later, it's, “Yeah, how would you know? You've been in academia your entire life.” I'm sorry, but the workplace environment of a public middle school and the workplace environment of a corporate entity are very culturally different. And I feel confident in saying that because my first Unix admin job was at a university. It is a different universe entirely.Francessca: Yeah. It's an area where you have to be able to balance the academia component with practitioner. And by the way, we talk about this in our solutions architecture and our customer solutions team—that's a mouthful—in our organization, that how we like to differentiate our capabilities with customers is that we are users, we are practitioners of the services, we have gone out and obtained certifications. We don't always just speak about it, we'd like to say that we've been in the empty chair with the customer, and we've also done. So yeah, I think it's a huge balance, by the way, and I just hope that over the next several years, Corey, that again, we start really shifting the landscape by tapping into what I think is an incredible global workforce, and of users that we've just not inspired enough to go into these disciplines for STEM, so I hope we do more of that.And I think our customers will benefit better from it because you'll get more diversity in thought, you'll get different types of innovation for your solution set, and you'll maybe mirror the customer segments that you're responsible for serving. So, I'm pretty bullish on this topic. [smile].Corey: I think it's hard not to be because, sure, things are a lot more complex now, technically. It's a broader world, and what's a tech company? Well, every company, unless they are asleep at the wheel, is a tech company. And that that can be awfully discouraging on some level, but the other side of it has really been, as I look at it, is the sheer, I guess, brilliance of the talent that's coming up. I'm not talking the legend of industry that's been in the field for 30 years; I'm talking some of the folks I know who are barely out of high school. I'm talking very early career folks who just have such a drive, and such an appetite for being able to look at how these things can solve problems, the ability to start thinking in innovative ways that I've never considered when I was that age, I look at this. And I think that, yeah, we have massive challenges in front of us as people, as a society, et cetera, but the kids are all right, for lack of a better term.Francessca: [smile].Corey: And I want to be clear as well; when we talk about new to tech, I'm not just talking new grads; I'm talking about people who are career-changing, where they wound up working in healthcare or some other field for the first 10 years of their career—20 years—and they want to move into tech. Great. How do we throw those doors open, not say, “Well, have you considered going back and getting a degree, and then taking a very entry-level job?” No. A lateral move, find the niches between the skill you have and the skill you want to pick up and move into the field in half steps. It takes a little longer, sure, but it also means you're not starting over from square one; you're making a lateral transition which, because it's tech, generally comes with a sizable pay bump, too.Francessca: One of the biggest surprises that I've had since joining the organization, and—you know, we have a very diverse, large global field organization, and if you look at our architecture teams, our customer solution teams, even our product engineering teams, one of the things that might surprise many people is many of them have come from customers; they've not come from what I would consider a traditional, perhaps, sales and marketing background. And that's by design. They give us different perspective, they help us ensure that, again, what we're designing and building is applicable from an end-user perspective, or even an industry, to your point. We have lots of different services now, over a hundred and seventy-five plus. I mean, we've—close to two hundred, now.And there are some customers who want the freedom to be able to build in the various domains, and then we have some customers who need more help and want us to put it together as solutions. And so having that diversity in some of the folks that we've been able to hire from a customer or developer standpoint—or quite frankly, co-founder standpoint—has really been amazing for us. So.Corey: It's always interesting whenever I get the opportunity to talk to folks who don't look like me—and I mean that across every axis you can imagine: people who didn't come up, first off, drowning in the privilege that I did; people who wound up coming at this from different industries; coming at this from different points of education; different career trajectories. And when people say, “Oh, yeah. Well, look at our team page. Everyone looks different from one another.” Great. That is not the entirety what diversity is.Francessca: Right.Corey: “Yeah, but you all went to Stanford together and so let's be very realistic here.” This idea that excellence isn't somehow situational, the story we see about, “Oh, I get this from recruiters constantly,” or people wanting to talk about their companies where, yes, ‘founded by Google graduates' is one of my personal favorites. Google has 140,000 people and they founded a company that currently has five folks, so you're telling me that the things that work at Google somehow magically work at that very small scale? I don't buy that for a second because excellence is always situational. When you have tens of thousands of people building infrastructure for you to work on, back in the early days was always the story that, that empowered folks who worked at places like Google to do amazing things.What AWS built, fundamentally, was the power to have that infrastructure at the click of a button where the only bound—let's be realistic here—is your budget. Suddenly, that same global infrastructure and easy provisioning—‘easy,' quote-unquote—becomes something everyone can appreciate and get access to. But in the early days, that wasn't the thing at all. Watching our technology has evolved the state of the art and opened doors for folks to be just as awesome where they don't need to be in a place like Google to access that, that's the magic of cloud to me.Francessca: Yeah. Well, I'm a huge, just, technology evangelist. I think I just was born with tech. I like breaking things and putting stuff together. I'll tell you just maybe two other things because you talked about excellence and equity.There's two nonprofits that I participate in. One I got introduced through AWS, our current CEO, Andy Jassy, and our Head of Sales and Marketing, Matt Garman. But it's called Rainier Scholars, and it's a 12-year program. They offer a pathway to college graduation for low-income students of color. And really, ultimately, their mission is to answer the question of how do we build a much more equitable society?And for this particular nonprofit, education is that gateway, and so spent some time volunteering there. But then to your point on the opportunity side, there's another organization I just recently became a part of called Year Up. I don't know if you've heard of them or worked with them before—Corey: I was an instructor at Year Up, for their [unintelligible 00:31:19] course.Francessca: Ahh. [smile].Corey: Oh, big fan of those folks.Francessca: So, I just got introduced, and I'm going to be hopefully joining part of their board soon to offer up, again, some guidance and even figuring out how we can help. But so you know, right? They're then focused on serving a student population and decreasing, shrinking the opportunity divide. Again, focused on equitable access. And that is what tech should be about; democratizing technology such that everyone has access. And by the way, it doesn't mean that I don't have favorite services and things like that, but it does mean—[smile] providing [crosstalk 00:31:58]—Corey: They're like my children; I can't stand any of them.Francessca: [smile]. That's right. I do have favorite services, by the way.Corey: Oh, as do we all. It's just rude to name them because everyone else feels left out.Francessca: [smile] that's right. I'll tell you offline. Providing that equitable access, I just think is so key. And we'll be able to tap in, again, to more of this talent. For many of these companies who are trying to transform their business model, and some—like last year, we saw companies just surviving, we saw some companies that were thriving, right, with what was going on.So again, I think you can't really talk about a comprehensive tech strategy that will empower your business strategy without thinking about your workforce plan in the process. I think it would be very naive for many companies to do that.Corey: So, one question that I want to get to here has been that if I take a look at the AWS service landscape, it feels like Perl did back when that was the language that I basically knew the best, which is not saying much.Francessca: You know you're dating yourself now, Corey.Corey: Oh, who else would date me these days?Francessca: [smile].Corey: My God. But, “There's more than one way to do it,” was the language's motto. And I look at AWS environments, and I had a throwaway quip a few weeks back from the time of this recording of, “There are 17 ways to deploy containers on AWS.” And apparently, it turned into an internal meme at AWS, which is just—I love the fact that I can influence company cultures without working there, but I'll take what I can get. But it is a hard problem of, “Great, I want to wind up doing some of these things. What's the right path?” And the answer is always, “It depends.” What are you folks doing to simplify the onboarding journey for customers because, frankly, it is overwhelming and confusing to me, so I can only imagine what someone who is new to the space feels. And from customers, that's no small thing.Francessca: I am so glad that you asked this question. And I think we hear this question from many of our customers. Again, I've mentioned earlier in the show that we have to meet customers where they are, and some customers will be at a stage where they need, maybe, less prescriptive guidance: they just want us to point them to the building blocks, and other customers who need more prescriptive guidance. We have actually taken a combination of our programs and what we call our solutions and we've wrapped that into much stronger prescriptive guidance under our migration and again, our modernization initiative; we have a program around this. What we try to help them do first is assess just where they are on the adoption phase.That tends to drive then how we guide them. And that guidance sometimes could be as simple as a solution deployment where we just kind of give them the scripts, the APIs, a CloudFormation template, and off they go. Sometimes it comes in the form of people and advice, Corey. It really depends on what they want. But we've tried to wrap all of this under our migration acceleration program where we can help them do a fast, sort of, assessment on where they are inclusive of driving, you know, a quick business case; most companies aren't doing anything without that.We then put together a fairly fast mobilization plan. So, how do they get started? Does it mean—can they launch a control foundation, control tower solutions to set up things like accounts, identity and access management, governance. Like, how do you get them doing? And then we have some prescriptive guidance in our program that allows them to look at, again, different solution sets to solve, whether that be data, security. [smile].You mentioned containers. What's the right path? Do I go containers? Do I go serverless? Depending on where they are. Do I go EKS, ECS Anywhere, or Fargate? Yeah. So, we try to provide them, again, with some prescriptive guidance, again, based on where they are. We do that through our migration acceleration initiative. To simplify. So.Corey: Oh, yeah. Absolutely. And I give an awful lot of guidance in public about how X is terrible; B is the better path; never do C. And whenever I talk—for example, I'm famous for saying multi-cloud is the wrong direction. Don't do it.And then I talk to customers who are doing it and they expect me to harangue them, and my response is, “Yeah, you're probably right.” And they're taken aback by this. “Does this mean you're saying things you don't believe?” No, not at all. I'm speaking to the general case, where if, in the absence of external guidance, this is how I would approach things.You are not the general case by definition of having a one-on-one conversation with me. You have almost certainly weighed the trade-offs, looked at the context behind what you're doing and why, and have come to the right decision. I don't pretend to know your business, or your constraints, or your capabilities, so me sitting here with no outside expertise, looking at what you've done, and saying, “Oh, that's not the right way to do it,” is ignorant. Why would anyone do that? People are surprised by that because context matters an awful lot.Francessca: Context does matter, and the reason why we try not to just be overly prescribed, again, is all customers are different. We try to group pattern; so we do see themes with patterns. And then the other thing that we try to do is much of our scale happens through our partner ecosystem, Corey, so we try to make sure that we provide the same frameworks and guidance to our partners with enough flexibility where our partners and their IP can also support that for our customers. We have a pretty robust partner ecosystem and about 150-plus partners that are actually with our migration, you know, modernization competency. So yeah, it's ongoing, and we're going to continue to iterate on it based on customer feedback. And also, again, our portfolio of where customers are: a startup is going to look very different than 100-year-old enterprise, or an independent software vendor, who's moving to SaaS. [smile].Corey: Exactly. And my ridiculous build-out for my newsletter pipeline system leverages something like a dozen different AWS services. Is this the way that I would recommend it for most folks? No, but for what I do, it works for me; it provides a great technology testbed. And I think that people lose sight pretty quickly of the fact that there is in fact, an awful lot of variance out there between use cases' constraints. If I break my newsletter, I have to write it by hand one morning. Oh, heavens, not that. As opposed to, you know, if Capital One goes down and suddenly ATMs starts spitting out the wrong balance, well, there's a slightly different failure domain there.Francessca: [smile].Corey: I'm not saying which is worse, mind you, particularly from my perspective, however, I'm just saying it's different.Francessca: I was going to tell you, your newsletter is important to us, so we want to make sure there's reliability and resiliency baked into that.Corey: But there isn't any because of my code. It's terrible. This—if—like, forget a region outage. It's far more likely I'm going to make a bad push or discover some weird edge case and have to spend an hour or two late at night fixing something, as might have happened the night before this recording. Ahem.Francessca: [smile]. Well, by the way, I'm obligated, as your Chief Solution Architect, to have you look at some form of a prototype or proof of concept for Textract if you're having to handwrite out all the newsletters. You let me know when you'd like me to come in and walk you through how we might be able to streamline that. [smile].Corey: Oh, I want to talk about what I've done. I want to start a new sub-series on your site. You have the This is my Architecture. I want to have something, This is my Nonsense Architecture. In other words, one of these learning by counterexample stories.Francessca: [smile]. Yeah, Matt Yanchyshyn will love that. [smile].Corey: I'm sure he will. Francessca, thank you so much for taking the time to speak with me. If people want to learn more about who you are, what you believe, and what you're up to, where can they find you?Francessca: Well, they can certainly find me out on Twitter at @FrancesscaV. I'm also on LinkedIn. And I also want to thank you, Corey. It's been great just spending this time with you. Keep up the snark, keep giving us feedback, and keep doing the great things you're doing with customers, which is most important.Corey: Excellent. I look forward to hearing more about what you folks have in store. And we'll, of course, put links to that in the [show notes 00:40:01]. Thank you so much for taking the time to speak with me.Francessca: Thank you. Have a good one.Corey: Francessca Vasquez, VP of Technology at AWS. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with a comment telling me why there is in fact an AWS/400 mainframe; I just haven't seen it yet.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

DH Unplugged
DHUnplugged #572: Surprise No Surprise

DH Unplugged

Play Episode Listen Later Sep 8, 2021 60:35


Markets want to keep the summer party rolling…. Delta, Lambda, mu? What is next and how will markets react? An update on the employment report and China’s slowing economy. Better Midler calls for a sex boycott – that is something… PLUS we are now on Spotify and Amazon Music/Podcasts! See this week's stock picks HERE […]

Screaming in the Cloud
The Mythos of Testing with Angie Jones

Screaming in the Cloud

Play Episode Listen Later Sep 7, 2021 36:36


About Angie Angie Jones is a Java Champion and Senior Director who specializes in test automation strategies and techniques. She shares her wealth of knowledge by speaking and teaching at software conferences all over the world, writing tutorials and technical articles on angiejones.tech, and leading the online learning platform, Test Automation University.As a Master Inventor, Angie is known for her innovative and out-of-the-box thinking style  which has resulted in more than 25 patented inventions in the US and China. In her spare time, Angie volunteers with Black Girls Code to teach coding workshops to young girls in an effort to attract more women and minorities to tech.Links: Applitools: https://applitools.com Black Girls Code: https://www.blackgirlscode.com Test Automation University: https://testautomationu.applitools.com Personal website: https://angiejones.tech Twitter: https://twitter.com/techgirl1908 TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by CircleCI. CircleCI is the leading platform for software innovation at scale. With intelligent automation and delivery tools, more than 25,000 engineering organizations worldwide—including most of the ones that you've heard of—are using CircleCI to radically reduce the time from idea to execution to—if you were Google—deprecating the entire product. Check out CircleCI and stop trying to build these things yourself from scratch, when people are solving this problem better than you are internally. I promise. To learn more, visit circleci.com.Corey: This episode is sponsored in part by Thinkst. This is going to take a minute to explain, so bear with me. I linked against an early version of their tool, canarytokens.org in the very early days of my newsletter, and what it does is relatively simple and straightforward. It winds up embedding credentials, files, that sort of thing in various parts of your environment, wherever you want to; it gives you fake AWS API credentials, for example. And the only thing that these things do is alert you whenever someone attempts to use those things. It's an awesome approach. I've used something similar for years. Check them out. But wait, there's more. They also have an enterprise option that you should be very much aware of canary.tools. You can take a look at this, but what it does is it provides an enterprise approach to drive these things throughout your entire environment. You can get a physical device that hangs out on your network and impersonates whatever you want to. When it gets Nmap scanned, or someone attempts to log into it, or access files on it, you get instant alerts. It's awesome. If you don't do something like this, you're likely to find out that you've gotten breached, the hard way. Take a look at this. It's one of those few things that I look at and say, “Wow, that is an amazing idea. I love it.” That's canarytokens.org and canary.tools. The first one is free. The second one is enterprise-y. Take a look. I'm a big fan of this. More from them in the coming weeks.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. If there's one thing that I have never gotten the hang of, its testing. Normally, I just whack the deploy button, throw it out into the general ecosystem, and my monitoring system is usually called ‘customers.' And if I don't want to hear from them, I just stopped answering calls from the support desk. Apparently, that is no longer state of the art because it's been about 15 years. Here to talk about testing from a more responsible direction is Angie Jones, a senior director and developer at Applitools. Thanks for joining me.Angie: Hey, Corey. [laugh]. I am cracking up at your confession there and I appreciate it because you're not unique in that story. I find that a lot of engineers [laugh] follow that same trend.Corey: There are things we talk about and there are the things that we really do instead. We see it all over the place. We talk about infrastructure as code, but everyone clicks around for a few things in the Cloud Console, for example. And so on, and so forth. We all know we should in theory be doing things, but expediency tends to win the day.And for better or worse, talking about testing, in many cases, makes some of us feel better about not actually doing testing. And one of these days, it's one of those, “I really should learn how TDD would work in an approach like this.” But my primary language has always been, well, always been a crappy version of whatever I'm using, but for the last few years, it's been Python. There are whole testing frameworks around all of these things, but I feel like it requires me to actually have good programming practices to begin with which, let's be very clear here, I most assuredly doubt.Angie: [laugh]. That's a fair assessment, but I would also argue, in cases like those, you need testing even more, right? You need something to cover your butt. So, what are you doing? You're just, kind of, living on the edge here?Corey: Sort of. In my case, it's always been that I'll bring in an actual developer who knows what they're doing to—Angie: Ah.Corey: —turn some of my early scripts into actual tools. And the first question is, “Okay, can you explain what this is doing for me?” “Great. So, we're going to throw it away and completely replace it with—so what are the inputs, what are the outputs, and do you want me to preserve the bugs or not?” At which point, it's great.It's more or less like I'm inviting someone to come in and just savage my code, which is apparently also a best practice. But for better or worse, I've never really thought of myself as an engineer, so it's one of those areas where it's it doesn't cut to the core of my identity in any particular way. I do know it would be nice that, oh yeah, when I wind up doing an iterative deployment of a Lambda function or something, if it takes five minutes to get updated, and then I forgot to put a comma in or something ridiculous like that. Yeah. Would have been nice to have something—you know, a pre-commit hook—that caught something like that.Angie: Yeah, yeah. It's interesting. You said, “Well, maybe one of these days, I'll learn.” And that's the issue I find. No matter what route you took to learn how to become—whatever you are, software engineer, whatever—testing likely wasn't part of that curriculum.So, we focus—when teaching—very heavily on teaching you how to code and how to build something, but very little, if any, on how to ensure you built the right thing and that it stands the test of time.Corey: My approach has always been well, time to write some code, and it started off as just, as a grumpy systems administrator, it was always shell scripts, which, okay, great. Instead of doing this thing on 15 machines, run upon a for loop and just iterate through them. And in time, you start inheriting other people's crappy tooling, and well, I could rewrite the entire thing and a week-and-a-half, or I could figure out just enough Perl to change that one line in there, and that's how they get you. You sort of stumbled your way into it in that direction. Naive questions I always like to ask around testing that never really get answers for because I don't think to ask these when other people are in the room and it's not two o'clock in the morning and the power is gone out.You have a basic linter test of, do you have basic syntax errors in the code? Will it run? Seems to be a sort of baseline, easy acceptance test. But then you get into higher-level testing of unit tests, integration tests, and a bunch of others I'm sure I'm glossing over because—to be direct—I tend to conflate all these in my head. What is the hierarchy of testing if there is such a thing?Angie: Yeah, so Mike Cohn actually created a model that is very heavily used within the industry, and it's called the ‘Test Automation Pyramid.' And what this model suggests is that you have your unit tests; you have some kind of, like, integration-type tests in the middle, and then you have these end-to-end tests on top. So, think of a pyramid divided into three sections. But that's not divided equally; the largest part of that pyramid, which is the base, is the unit test. So, this suggests that the bulk of your test suite should comprise of unit tests.The idea here is that these are very small, they're very targeted, meaning they're easier to write, they take less time to run, and if you have an error, it kind of pinpoints exactly what's wrong in the system. So, these are great. The next level would be your integration. So, now how do two units integrate together? So, you can test this layer multiple different ways: it might be with APIs, it might be the business logic itself, you know, calling into functions or something like that.And this one is smaller than the unit test but not as large as the final part, which is the end-to-end test. And that one is your smallest piece, and it doesn't even have to be end-to-end. It could be UI, actually. That's how it's labeled by Mike Cohn in his book: UI tests. So, the UI tests, these are going to be your most fragile tests, these are going to take the most time to write as well as the most time to execute.If something goes wrong, you have to dig down to figure out what exactly broke to make this happen. So, this should be the smallest chunk of your overall testing strategy.Corey: People far smarter than I have said that in many cases—along with access—testing, and monitoring—or observability, which is apparently a term for hipster monitoring—are lying on the same axis. Where in the olden days of systems administration, you can ping the machine and it responds just fine, but the only thing that's left on that crashed machine is just enough of the network stack to return a ping, so everything except the thing that tells you it's fine is in fact broken. So, as you wind up building more and more sophisticated applications, the idea being that the testing and the ‘is everything all right' monitoring ping tends to, more or less, coalesce into the same thing. Is that accurate from your view of the world? Is that something that is an oversimplification of something much more nuanced? Or did I completely misunderstand what they were saying, which is perfectly possible?Angie: You kind of lost me somewhere in the middle. So, I'm just going to nod and say yes. [laugh].Corey: [laugh]. No, no, it—the hard part that I've always found is… I lie to myself, when I'm writing code: “Oh, I don't need to write a unit test for this,” because I'd gotten it working, I tested it with something that I know is good, it returns what I expect; I tested with something bad and well, some undefined behavior happens—because that's a normal thing to happen with code—and great, I don't need to have a test for that because I've already got it working. Problem solved.Angie: Right. Right.Corey: It's a great lie.Angie: Yeah.Corey: And then I make a change later on that, in fact, does break it. It's the, “But I'm writing this code once and why would I ever go back to this code and write it again? It's just a quick-and-dirty patch that only needs to exist for a couple of weeks.” Yeah, the todo: remove this later, and that code segment winds up being load-bearing decades into the future. I'm like, “Yeah, one of these days, someone's going to go back and clean up all of my code for me.” Like, the code fairies are going to come in the middle of the night with the elves, and tidy everything up. I would love to hire those mythical creatures, but can't find them.Angie: This mythical sprint, where it's, “Oh, let's only clean up this entire sprint.” You know, everybody's kind of holding out and waiting for that. But no, you hit the nail on the head with the reason why you need to automate your tests, essentially. So, I find a lot of newer folks to the space, they really don't understand, why on earth would I spend time writing code to represent this test? Just like you said, “I implemented the feature. I tried it out, it worked.” [laugh]. “And hey, I even tried a non-happy path. And when it broke, I had a nice little error message to tell the user what to do.”And they feel really good about that, so they can't understand, “Why would I invest the time—which I don't have—to write some tests?” The reason for that it's just as you said: this is for regression. Unless that's the end of this application and you're not going to touch it ever again for any reason, then you need to write some tests [laugh] because you're going to constantly change the application, whether that be refactoring, whether that be adding new features to it, it's going to change in some way and you cannot be sure that the tests of yesterday still work today because whenever you make the change, you're just going to poke around manually at that little area not realizing there could be some integration things that you totally screwed up here and you miss that until it goes out into prod.Corey: The worst developer I've ever met—hands down—was me, six months before I'm looking at whatever it is that I've written. And given that I do a lot of my stuff in a vacuum and I'm the only person to ever touch these repositories, I could run Git blame, but I already know exactly what it's going to tell me—Angie: “It's me.” [laugh].Corey: —so we're just going to skip that part. Like it's a test. And, “Yeah, we're just going to try and fix that and never speak about it again.” But I can't count the number of times I have looked at code that I've written—and I do mean written; not blindly copy-and-pasted out of Stack Overflow, but actually wrote, and at the time, I understood exactly what it did—and then I look at it, and it is, “What on earth was I thinking? What—what—it technically doesn't even return anything; it can't be doing anything. I can just remove that piece entirely.” And the whole thing breaks.I've out-clevered myself in many respects. And I love the idea, the vision, that testing would catch these things as I'm making those changes, but then I never do it. It's getting started down that path and developing a more nuanced, and dare I say it, formal understanding of the art and science of software development. Always feels like the sort of thing I'll get to one of these days, but never actually got around to. Nowadays, my testing strategy is to just actually deploy things into someone else's account and hope for the best.And, “Oh, good. Well, everyone has a test account; ideally, it's not their own production account.” And then we start to expand on beyond that. You have come to this from a very different direction in a number of different ways. You are—among other things—a Java Champion, which makes it sound like you fought the final boss at the end of the developer internet. And they sound really hard. What is a Java Champion?Angie: Yeah. So, a Java Champion is essentially an influencer in the Java ecosystem. You can't just call yourself this; like you say, you got to fight the guy at the end, you know? But seriously, in order to become one, a current Java Champion has to nominate you, and all of the other Java Champions has to review your package, basically looking at your work. What have you contributed to the developer community, in terms of Java?So, I've done a number of courses that I've taught; I've taught at the university level, as well; I am always talking about testing and using Java to show how to do that, as well as talks and all of this stuff. So apparently, I had enough [laugh] for folks to vote me in. So, it is an organization that's kind of ordained by Oracle, the Gods of Java. So, it's a great accomplishment for me. I'm extremely happy about it. And just so happens to be the first black woman to become a Java Champion. So, the news made a big deal about that. [laugh].Corey: Congratulations. Anytime you wind up getting that level of recognition in any given ecosystem, it's something to stop and take note of. But that's compounded by just the sheer scale and scope of the Java community as a whole. Every big tech company I know has inordinate amounts of Java scattered throughout their infrastructure, a lot of their core services are written in Java, which makes me feel increasingly strange for not really knowing anything about it, other than that, it's big and that there are—this entire ecosystem of IDs, and frameworks, and ways to approach these things that it feels like those of us playing around in crappy bash-scripting-land have the exact opposite experience of, “Oh, I'm just going to fire up an empty page and fill it with a bunch of weird commands and run it, and it fails, and run it again, and it fails. And it finally succeeds when I fixed all the syntax errors, and that's great.” It feels like there is a much more structured approach to writing Java compared to other languages, be they scripts or full-on languages.Angie: Yeah. That's been a gift and a curse of the language. So, as newer frameworks have come out, or even as JavaScript has made its way to the front of the line, people start looking at Java, it's kind of bloated, and all of these rules and structures were in place, but that feels like boilerplate stuff and cumbersome in today's development space. So, fortunately, the powers that be have been doing a lot of changes in Java. We went for quite a while where releases were about, mmm, every three years or so.And now they've committed to releases every six months. So, [laugh] most people are on Java 8 still, but we're actually at, like, Java 16, now. So, now it's kind of hard to keep up but that makes it fun as well. There's all of these newer features and new capabilities, and now you can even do functional programming in Java, so it's pretty nice.Corey: Question I have is, does testing lend itself more easily to Java versus other language? And I promise I'm not trying to start a language war here. I just know that, “Well, how do I effectively test my Python code?” Leads to a whole bunch of? “Well, it depends.”It's like asking an attorney any question on the planet; same story. Like, “Well, it really depends on a whole bunch of things.” Is it a clearer, more structured path in Java, or is it still the same murky there are 15 different ways to do it and whichever one you pick, there's a whole cacophony of folks telling you you've done it wrong?Angie: Yeah, that's a very interesting question. I haven't dug into that deep, but Java is by far the most popular programming language for UI test automation. And I wonder why that is because you don't use Java for building front end. You use Java scripts. I don't know how this ca—I—well, I do know how it came to be.Like, back in the day, when we first started doing test automation, JavaScript was a joke, right? People would laugh at you if you said that you were going to use JavaScript. It's, you know, “I'm going to learn JavaScript and try to enter the workforce.” So, you know, that was a big no-no, and kind of a joke back then. So, Java was what a lot of your developers were using even if they were only using it for the backend, maybe.You didn't really have a [unintelligible 00:16:32] language on the client-side, back then. You had your PHP on the back end, you just did some HTML and some CSS on the front end. So, there wasn't a whole lot of scripting going on back then. So, Java was the language that people chose to use. And so there's a whole community out there for Java and testing.Like, the libraries are very mature, there's open-source products and things like this. So, this is by far the most popular language that people use, no matter what their application is built in.This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of "Hello, World" demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking databases, observability, management, and security.And - let me be clear here - it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself all while gaining the networking load, balancing and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build.With Always Free you can do things like run small scale applications, or do proof of concept testing without spending a dime. You know that I always like to put asterisks next to the word free. This is actually free. No asterisk. Start now. Visit https://snark.cloud/oci-free that's https://snark.cloud/oci-free.Corey: If I were looking to get a job in enterprise these days, it feels like Java is the direction to go in, with the counterpoint that, let's say that I go the path that I went through: I don't have a college degree; I don't have a high school diploma. If I were to start out trying to be a software engineering today, or advising someone to do the same, it feels like the lingua franca of everything today seems to be JavaScript in many different respects. It does front end; it does back end; people love to complain about it, so you know it's valid. To be clear, I find myself befuddled every time I pick it up. I'm not coming at this from a JavaScript fanboy perspective in any respect.The asynchronous execution flow always messes with my head and leaves me with more questions than answers. Is that assessment though—of starting languages—accurate? Are there cases where Java is absolutely the right answer, as far as what to learn first?Angie: Yeah. So, I first started with C++, and then I learned Java. Well, what I find is, Java because it's so strict—it's a statically typed language, and there's lots of rules, and you really need to understand paradigms and stuff like that with this language—it's harder to learn, but once you learn it, it's much easier to pick up other languages, even if they're dynamically typed, you know? So, that's been my experience with this. As far as jobs, so the last time I looked at this, someone did some research and wrote it up—this was 2019—and they looked at the job openings available at the time, and they divided it by language. And Java was at, like, 65,000 jobs open, Python was a close second was 62,000, and JavaScript was third place with 39,000.So, quite a big difference. But if you looked at tech Twitter, you'd think, like, JavaScript is all there is. Most of my followers and folks that I follow are JavaScript folks, front-end folks. So, it is a language I think you definitely need to learn; it's becoming more and more prevalent. If you're going to do any sort of web app, [laugh] you definitely want to know it.So, I'm definitely not saying, “Oh, just learn Java and that's it.” I think there's definitely a need for adding JavaScript to your repertoire. But Java, there does seem to be more jobs, especially the big enterprise-type jobs, in Java.Corey: The reason I ask so much about some of the early-stage stuff is that in your spare time—which it sounds like you have so much of these days—you volunteer with Black Girls Code to help teach coding workshops to young girls in an effort to attract more women and minorities to tech. Which is phenomenal. Few years ago, I was a volunteer instructor for Year Up before people really realized, “Oh, maybe having an instructor who teaches by counterexample isn't necessarily the best approach of teaching folks who are new to the space.”But the curriculum I was given for teaching people how Linux worked and how to build a web servers and the rest, started off with a three-day module on how to use VI, an arcane text editor that no one understands, and the only reason we use it is because we don't know how to quit it.Angie: [laugh].Corey: And that's great and all, but I'm looking at this and my immediate impression was, “We're scrapping that, replacing it with nano,” which is basically what you see is what you get, and something that everyone can understand and appreciate without three days of training. And it felt an awful lot like we're teaching people VI almost as a form of gatekeeping. I'm curious; when you presumably go down the path of teaching people who are brand-new to the space? How do you wind up presenting testing as something that they should start with? Because it feels like a thing you have to know first before you can start building anything at scale, but it resonates, on some level, with feeling like it's, ah, you must be able to learn this religion first; then you'll be able to go and proceed further. How do you square that circle?Angie: Yeah. So, I had the privilege of being an adjunct professor at a college, and I taught Java programming to freshmen. This was really interesting because there's so much to teach, and this is true of all the courses. So, when I say that they don't include it in the curriculum, that's not really that much of a slight on them. Like, it's just so much you have to cover.So I, me, the testing guru, I still couldn't find space to devote an entire sitting, a chapter, or whatever on testing. So, I kind of wove it into my teaching style. So, I would just teach the concept, let's say I'm teaching loops today, and I'll have a little exercise that you do in class. So, we do things together, and then I say, okay, now you try it by yourself. Here's a problem; call me over when you're done.And as they would call me over when they're done, I would break it; I would break their code, right? I'd do some input that they weren't expecting and all of a sudden is broken. And they started expecting me to do this, you know? “She's going to come and she's going to break my stuff.” So, they start thinking themselves, “Let me test it before I give it to my user,” who is Professor Angie, or whatever.So, that's how I taught them that. Same with homework assignments. So, they would submit it, I would treat it like a code review, go through line by line, I didn't have any automated systems to test their homework assignments. I did it like a code review, gave them feedback on how to improve their style, but also I would try to break it and give them, “Here's all the areas that you didn't think of.” So, that was my way of teaching them that quality matters in how to think about beyond the requirement.The requirement is going to say, “Someone needs to be able to log in.” It's not going to give you all of the things that should happen, you know if there's a wrong password, so these are things, as an engineer, you need to think beyond that one line requirement that you've got and realize that this is part of it as well.Corey: So, it's almost a matter of giving people context beyond just the writing of the code, which frankly, seems to be something that's been missing for many aspects of engineering culture for a while, the understanding the people involved, understanding that it is not just you, or your department, or even your company in some cases.Angie: Exactly. And I tried to stress that very heavily in each lecture: who is your end-user? And your end-user cannot see your code, they cannot see your comments in the code that's telling them, “Make sure you input it this way,” or whatever. None of that is seen so you have to be very explicit in your messages, and your intent, and behavior with the end-user.Corey: One last area I wanted to cover with you, when I was doing some research on you before the show, is that you are an IBM Master Inventor, which I had no idea what that was. Is that a term of art? Let me Google it. And it turns out that you have, according to LinkedIn at least, 27 patents in your name. And it's, “Oh.”Yeah, it's one of those areas where you look at something like, what gives someone the hubris to call themselves—or the grounds to call themselves that? And, “Oh, yeah. Oh, they're super accomplished, and they have a demonstrated track record of inventing things that are substantial and meaningful. I guess that would do it.” I'd never heard the term until now. What is that? And how are you that prolific, for lack of a better term?Angie: Yeah, so I used to work at IBM and they're really big on innovation. And I haven't kept track in a while, but for many, many years, they were the number one producer of patents [laugh] of this year or whatever. So, it was kind of in the culture to innovate. Now, I will say, like, a very small percentage of people—employees—there would take it as far as I did to actually go and patent something—[laugh]—Corey: Oh, it's the ‘don't offer if you're not serious,' model.Angie: Yeah. [laugh]. But I mean, it was there; it was a program there where, hey, you got an idea for a software patent? Write it up, we'll have our lawyers, our IP lawyers review it, and then they'll take your little one-page doc and turn it into a twenty-five-page legal document that we submit to the USPTO—United States Patent Trademark Office—who then reviews it and decides if this is novel enough and grants it, or dismisses it. And, “Hey, we'll pay you for these patents. We'll pay for the whole process.” And so I thought, “Heck, why not?”And I kind of got hooked. [laugh]. So, it just so happens that I got a lot of good ideas. And I would collaborate with people from other areas of the business, and it was an excellent way for me to learn about new technologies. If something new was coming out, I would jump on that to explore, play with it, and think about, are there any problems that this technology is not aimed to solve, but if I tweak it in some way, or if I integrate it with some other concept or some other technology, do I get something unique and novel here?And it got to the point where I just started walking through life and as I'm hit with problems—like, I'll give you an example. I'm in the grocery store, right, and this inevitably happens to everyone, what, you choose the wrong line in the grocery store. “This one looks like it's moving, I'm going to go here.” And then the whole time, you're looking to your right, and that line is moving. And you're, like, stuck.Corey: Every single time.Angie: Every time. So, it got—[laugh]—Corey: Toll booths are the same way.Angie: —it got to the point where I started recognizing when I'm frustrated, and say, “This is a problem. How can I use tech to solve this?” And so I, in that problem, I came up with this solution of how I could be able to tell which one of these is the right line to get into. And that consisted of lots of things like scanning the things in everyone's cart. On your cart, you have these smart carts that know what's inside of them, polling the customers' spending or their behavior; so are they going to come up here and send the clerk back to go get cigarettes, or alcohol, or are they going to pull out 50 coupons? Are they going to write a check, which takes longer?So, kind of factoring in all of these habitual behaviors and what's in your cart right now, and determining an overall processing time. And that way, if you display that over each queue, which one would be the fastest to get into. So, things like that is what I started doing and patenting.Corey: Well, my favorite part of that story is that it is clearly a deeply technical insight into this, but you've told the story in a way that someone who is not themselves deeply technical can wrap their heads around. And I just—making sure you're aware of exactly how rare and valuable that particular skill set is. So, often there are people who are so in love with a technology that they cannot explain to another living soul who is not equally in love with that technology. That alone is one of the biggest reasons I wanted to have you on this show was your repeated, demonstrated ability to explain complex things simply in a way that—I know this is anathema for the tech industry—that is not condescending. I come away feeling I understand what you were talking about, now.Angie: Thank you so much. That is one of the skills I pride myself on. When I give talks, I want everyone in that room to understand it, even if they're not technical. And lots of times I've had comments from anyone from, like, the janitor to the folks who are working A/V who, they don't work with computers or anything at all and they've come to me after these talks like, “Okay, I heard a lot of talks in here. Everybody is over my head. I understood everything you said. Thank you.” And yet it's still beneficial to those who are deeply technical as well. Thank you so much for that.Corey: No, it's a very valuable thing and it's what I look for the most. In fact, my last question for you is tying around that exact thing. You have convinced me. I want to learn more about test automation, and learn how this works and with an eye toward possibly one day applying it to some of my crappy nonsense that I'm writing. Other than going on Google and typing in a variety of search terms that will lead me to, probably, a Stack Overflow thread that has been closed as off-topic, but still left up to pollute Google search results, where should I go?Angie: Yeah. So, I've actually started an entire university devoted to testing, and it's called Test Automation Universityand I got my employer, Applitools, to sponsor this, so all of the courses are free.And they are taught by myself as well as other leading experts in the test automation space. So, you know that it's trusted; I vet all of the instructors, I'm very [laugh] involved in going through their material and making sure that it's correct and accurate so the courses are of top quality. We have about a little over 85,000 students at Test Automation University, so you definitely need to become one if you want to learn more about testing. And we cover all of the languages, so Java, JavaScript, Python, Ruby, we have all of the frameworks, we have things around mobile testing, UI testing, unit testing, API testing. So, whatever it is that you need, we got you covered.Corey: You also go further than that; you don't just break it down by language, you break it down by use case. If I—Angie: Yeah.Corey: —look at Python, for example, you've got a Web UI path, you've got an—Angie: Exactly.Corey: API path, you've got a mobile path. It aligns not just with the language but with the use case, in many respects.Angie: Mm-hm.Corey: I'm really glad I asked that question, and we will, of course, include a link to that in the [show notes 00:31:10]. Thank you so much for taking the time to speak with me. If people want to learn more, other than going to Test Automation University, where can they find you?Angie: Mm-hm. So, my website is angiejones.tech—T-E-C-H—and I blog about test automation strategies and techniques there, so lots of good info there. I also keep my calendar of events there, so if you wanted to hear me speak or one of my talks, you can find that information there. And I live on Twitter, so definitely give me a follow. It's @techgirl1908.Corey: And we will, of course, include links to all of that. Thank you so much for being so generous with your time and insight. I really appreciate it.Angie: Yeah, thank you so much for having me. This was fun.Corey: Angie Jones, Java Champion and senior director at Applitools. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you hated this podcast, please leave a five-star review on your podcast platform of choice along with a long, ranting, incoherent comment that fails to save because someone on that platform failed to write a test.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Living Beyond 120
COVID-19 - What You Need To Know - Ep. 104

Living Beyond 120

Play Episode Listen Later Sep 2, 2021 56:04


On this episode of Living Beyond 120, Dr. Jeffrey Gladden from Gladden Longevity and Dr. Mark Young have a discussion with today's guest, Dr. Ernesto Navarro from Gladden Longevity, on the topic of COVID-19. Dr. Gladden, Dr. Young, and Dr. Navarro dive into the current science behind COVID-19, discuss variants, look at the factors that increase the risk of infection, and talk about ways to be proactive about our health, rather than reactive. Here are some highlights: Dr. Mark Young and Dr. Jeffrey Gladden discuss the tip of the day. Can-C is a dipeptide eye drop that helps with cataracts. We meet Dr. Navarro from Gladden Longevity. We review the factors that increase the risk of a COVID-19 infection. We discuss a study that found some of the greatest risk factors include essential hypertension, anxiety/fear, and various lipid disorders. We discuss what a variant is and get an update on the current Delta and Lambda variant. We discuss ways to be proactive about our health, rather than reactive. This includes living an active lifestyle, having a healthy, anti-inflammatory diet like the Mediterranean style diet, and decreasing our stress. Dr. Navarro and Dr. Gladden share a list of different supplements that are used at Gladden Longevity as a proactive approach. We discuss some actionable items on how to be proactive and really step into an overall healthy lifestyle

Screaming in the Cloud
Severless Hero, Got Severs in His Eyes with Ant Stanley

Screaming in the Cloud

Play Episode Listen Later Aug 31, 2021 37:02


About AntAnt Co-founded A Cloud Guru, ServerlessConf, JeffConf, ServerlessDays and now running Senzo/Homeschool, in between other things. He needs to work on his decision making.Links: A Cloud Guru: https://acloudguru.com homeschool.dev: https://homeschool.dev aws.training: https://aws.training learn.microsoft.com: https://learn.microsoft.com Twitter: https://twitter.com/iamstan TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Thinkst. This is going to take a minute to explain, so bear with me. I linked against an early version of their tool, canarytokens.org in the very early days of my newsletter, and what it does is relatively simple and straightforward. It winds up embedding credentials, files, that sort of thing in various parts of your environment, wherever you want to; it gives you fake AWS API credentials, for example. And the only thing that these things do is alert you whenever someone attempts to use those things. It's an awesome approach. I've used something similar for years. Check them out. But wait, there's more. They also have an enterprise option that you should be very much aware of canary.tools. You can take a look at this, but what it does is it provides an enterprise approach to drive these things throughout your entire environment. You can get a physical device that hangs out on your network and impersonates whatever you want to. When it gets Nmap scanned, or someone attempts to log into it, or access files on it, you get instant alerts. It's awesome. If you don't do something like this, you're likely to find out that you've gotten breached, the hard way. Take a look at this. It's one of those few things that I look at and say, “Wow, that is an amazing idea. I love it.” That's canarytokens.org and canary.tools. The first one is free. The second one is enterprise-y. Take a look. I'm a big fan of this. More from them in the coming weeks.Corey: This episode is sponsored in part my Cribl Logstream. Cirbl Logstream is an observability pipeline that lets you collect, reduce, transform, and route machine data from anywhere, to anywhere. Simple right? As a nice bonus it not only helps you improve visibility into what the hell is going on, but also helps you save money almost by accident. Kind of like not putting a whole bunch of vowels and other letters that would be easier to spell in a company name. To learn more visit: cribl.ioCorey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Every once in a while I talk to someone about, “Oh, yeah, remember that time that you appeared on Screaming in the Cloud?” And it turns out that they didn't; it was something of a fever dream. Today is one of those guests that I'm, frankly, astonished I haven't had on before: Ant Stanley. Ant, thank you so much for indulging me and somehow forgiving me for not having you on previously.Ant: Hey, Corey, thanks for that. Yeah, I'm not too sure why I haven't been on previously. You can explain that to me over a beer one day.Corey: Absolutely, and I'm sure I'll be the one that buys it because that is just inexcusable. So, who are you? What do you do? I know that you're a Serverless Hero at AWS, which is probably the most self-aggrandizing thing you can call someone because who in the world in their right mind is going to introduce themselves that way? That's what you have me for. I'll introduce you that way. So, you're an AWS Serverless Hero. What does that mean?Ant: So, the Serverless Hero, effectively I've been recognized for my contribution to the serverless community, what that contribution is potentially dubious. But yeah, I was one of the original co-founders of A Cloud Guru. We were a serverless-first company, way back when. So, from 2015 to 2016, I was with A Cloud Guru with Ryan and Sam, the two other co-founders.I left in 2016 after we'd run ServerlessConf. So, I led and ran the first ServerlessConf. And then for various reasons, I decided, hey, the pressure was too much; I needed a break, and a few other reasons I decided to leave A Cloud Guru. A very amicable split with my former co-founders. And then yeah, I kind of took a break, took some time off, de-stressed, got the serverless user group in London up and running; ran a small conference in London called JeffConf, which was a take on a blog that Paul Johnson, who was one of the folks who ran JeffConf with me, wrote a while ago saying we could have called it serverless—and we might as well have called it Jeff. Could have called it anything; might as well have called it Jeff. So, we had this joke about JeffConf. Not a reference to Mr. Bazos.Corey: No, no. Though they do have an awful lot of Jeffs working over there. But that's neither here nor there. ‘The Land of the Infinite Jeffs' as it were.Ant: Yeah, exactly. There are more Jeffs than women in the exec team if I remember correctly.Corey: I think it's now it's a Dave problem instead.Ant: Yeah, it's a Dave problem. Yeah. [laugh]. It's not a problem either way. Yeah. So, JeffConf morphed into SeverlessDays, which is a group of community events around the world. So, I think AWS said, “Hey, this guy likes running serverless events for some silly reason. Let's make him a Serverless Hero.”Corey: And here we are. Which is interesting because a few directions you can take this in. One of them, most recently, we were having a conversation, and you were opining on your thoughts of the current state of serverless, which can succinctly be distilled down to ‘serverless sucks,' which is not something you'd expect to hear from a Serverless Hero—and I hope you can hear the initial caps when I say ‘Serverless Hero'—or the founder of a serverless conference. So, what's the deal with that? Why does it suck?Ant: So, whole serverless movement started to gather momentum in 2015. The early adopters were all extremely experienced technologists, folks like Ben Kehoe, the chief robotics scientist at iRobot—he's incredibly smart—and folks of that caliber. And those were the kinds of people who spoke at the first serverless conference, spoke at all the first serverless events. And, you know, you'd kind of expect that with a new technology where there's not a lot of body of knowledge, you'd expect these high-level, really advanced folks being the ones putting themselves out there, being the early adopters. The problem is we're in 2021 and that's still the profile of the people who are adopting serverless, you know? It's still not this mass adoption.And part of the reason for me is because of the complexity around it. The user experience for most serverless tools is not great. It's not easy to adopt. The patterns aren't standardized and well known—even though there are a million websites out there saying that there are serverless patterns—and the concepts aren't well explained. I think there's still a fair amount of education that needs to happen.I think folks have focused far too much on the technical aspects of serverless, and what is serverless and not serverless, or how you deploy something, or how you monitor something, observability, instead of going back to basics and first principles of what is this thing? Why should you do it? How do you do it? And how do we make that easy? There's no real focus on user experience and adoption for inexperienced folks.The adoption curve, the learning curve for serverless, no matter what platform you do, if you want to do anything that's beyond a side project it's really difficult because there's no easy path. And I know there's going to be folks that are going to complain about it, but the Serverless Stack just got a million dollars to solve this problem.Corey: I love the Serverless Stack. They had a great way of building things out.Ant: Yeah.Corey: I cribbed a fair bit from what they built when I was building out my own serverless project of the newsletter production pipeline system. And that's awesome. And I built that, and I run it mostly as a technology testbed. But my website, lastweekinaws.com?I pay WP Engine to host it on WordPress and the reason behind that is not that I can't figure out the serverless pieces of it, it's because when I want to hire someone to do something that's a bit off the beaten path on WordPress, I don't have to spend $400 an hour for a consultant to do it because there's more than 20 people in the world who understand how all this stuff fits together and integrates well. There's something to be said for going in the direction the rest of the market is when there's not a lot of reason to differentiate yourselves. Yeah, could I save thousands of dollars a year in infrastructure costs if I'd gone with serverless? Of course, but people's time is worth more than that. It's expensive to have people work on these things.And even on the serverless stuff that I've built, if it's been more than six months since I've touched a component, someone else may have written it; I have to rediscover what the hell I was thinking and what the constraints are, what the constraints I thought existed there in the platform. And every time I deal with Lambda or API Gateway, I come away with a spiraling sense of complexity tied to all of it. And the vision of serverless I believe in, truly, but the execution has lagged from all providers.Ant: Yeah. I agree with that completely. The execution is just not there. I look at the situation—so Datadog had their report, “The State of Serverless Report” that came out about a month or two ago; I think it's the second year they've done it, now, might be the third. And in the report, one of the sections, they talked about tooling.And they said, “What's the most adopted tools?” And they had the Serverless Framework in there, they had SAM in there, they had CloudFormation, I think they had Terraform in there. But basically, Serverless Framework had 70% of the respondents. 70% of folks using Datadog and using serverless tools were using Serverless Framework. But SAM, AWS's preferred solution, was like 12%.It was really tiny and this is the thing that every single AWS demo example uses, that the serverless developer advocates push heavily. And it's the official solution, but the Serverless Application Model is just not being adopted and there are reasons for that, and it's because it's the way they approach the market because it's highly opinionated, and they don't really listen to end-users that much. And their CDK out there. So, that's the other AWS organizational complexity as well, you've got another team within AWS, another product team who've developed this different way—CDK—doing things.Corey: This is all AWS's fault, by the way. For the longest time, I've been complaining about Lambda edge functions because they are not at all transparent; you have to wait for a CloudFront deployment for it to update every time, only to figure out that in my case, I forgot a comma because I've never heard of a linter. And it becomes this awful thing. Only recently did I find out they only run at regional edge caches, not just in all of the CloudFront pop, so I said, “The hell with it,” ripped it out of everything I was using it with, and wound up implementing it in bog-standard Lambda because it was easier. But then rather than fixing that, they've created their—what was it—their CloudFront Workers. Or is it—is it CloudFront Workers, or is it CloudFront Functions?Ant: No, CloudFront Functions.Corey: I don't even remember it because rather than fixing the thing, you just released a different thing that addresses these problems in very different ways that aren't directly compatible. And it's oh, great, awesome. Terrific. As a customer, I want absolutely not this. It's one of these where, honestly, I've left in many cases with the resigned position of, if you're not going to take this seriously, why am I?Ant: Yeah, exactly. And it's bizarre. So, the CloudFront Functions thing, it's based on Nginx's [little 00:08:39] JavaScript engine. So, it's the Nginx team supporting it—the engine—which is really small number of users; it's tiny, there's no foundation behind it. So, you've got these massive companies reliant on some tiny organization to support the runtime of one of their businesses, one of their services.And they expect people to adopt it. And on top of that, that engine supports primary language is JavaScript's ES5 or ES2015, which is the 2015 edition of JavaScript, so it's a six-year-old version of JavaScript. You cannot use one JavaScript with it which also means you can't use any other tools in the JavaScript ecosystem for it. So basically, anything you write for that is going to be vanilla, you're going to write yourself, there's no tooling, no community to really leverage to use that thing. Again, like, why have you even done that? Why if you now gone off and taken an engine no one uses—they will say someone uses it, but basically no one uses—Corey: No one willingly uses or knowingly uses it.Ant: Yeah. No one really uses. And then decided to run that. Why not look at WebAssembly—it's crazy—which has a foundation behind it and they're doing great things, and other providers are using WebAssembly on the edge. I just don't understand the thought process—well, I say I don't understand, but I do understand the thought processes behind Amazon. Every single GM in Amazon is effectively incentivized to release stuff, and build stuff, and to get stuff out the door. That's how they make money. You hear the stories—Corey: Oh, it's been clear for years. They only recently stopped—in their keynotes every year—talking about the number of feature releases that they've had over the past 12 months. And I think they finally had it clued into them by someone snarky on Twitter—ahem—that the only people that feel good about that are people internal to AWS because customers see that and get horrified of, “I haven't kept up with most of those things. How many of those are important? How many of them are nonsense?”And I'm sure somewhere you have released a serverless that will solve my business problem perfectly so I don't have to build it together myself out of Lambda functions, and string, and popsicle sticks, but I'll never hear about it because you're too busy talking about nonsense. And that problem still exists and it's writ large. There's a philosophy around not breaking existing workloads—which I get; that's a hard problem to solve for—but their solution is, rather than fixing existing services will launch a new one that doesn't have those constraints and takes a different approach to it. And it's horrible.Ant: Yeah, exactly. If you compare Amazon to Apple, Apple releases a net-new product once a year, once every two years.Corey: You're talking about new generations of products, that comes out on an annualized basis, but when you're talking about actual new product, not that frequently. The last one—Ant: Yeah.Corey: —I can really think of is probably going to be AirPods, at least of any significance.Ant: AirTags is the new one.Corey: Oh, AirTags. AirTags is recent, which is a neat—but it's an accessory to the rest of those things. It is—Ant: And then there's AirPods. But yeah, it's once—because they—everything works. If you're in that Apple ecosystem, everything works. And everything's back-ported and supported. My four-year-old phone still works and had a five-year-old MacBook before this current one, still worked, you know, not a problem.And those two philosophies—and the Amazon folk are heavily incentivized to release products and to grow the usage of those products. And they're all incentivized within their bubbles. So, that's why you get competing products. That's why Proton exists when CodeBuild and CodePipeline, and all of those things exist, and you have all these competing products. I'm waiting for the container team to fully recreate AWS on top of containers. They're not far away.Corey: They're already in the process of recreating AWS on top of Lightsail. It's more or less the, “Oh, we're making this the simpler version.” Which is great. You know who likes simplicity? Freaking everyone.So, it's the vision of a cloud, we could have had but didn't. “Oh, you want a virtual machine. Spin up a Lightsail instance; you're going to get a fixed amount of compute, disk, RAM, and CPU that you can adjust, and it's going to cost you a flat fee per month until you exceed some fairly high limits.” Why can't everything be like that, on some level? Because in many cases, I don't care about wanting to know exactly to the penny shave things off.I want to spin up a fleet of 20 virtual machines, and if they cost me 20 bucks a pop each a month, I can forecast that, I can budget for that, I can do a lot and I don't actually care in any business context about the money there, but dialing it in and having the variable charges and the rest, and, “Oh, you went through a managed NAT gateway. That's going to double your bandwidth price and it's going to be expensive. Surprise, you should have looked more closely at it,” is sort of the lesson of the original AWS services. At some level, they've deviated away from anything resembling simplicity and increasingly we're seeing a world where in order to do something effectively with cloud, you have to spend 12 weeks going to cloud school first.Ant: Oh, yeah. Completely. See, that's one of the major barriers with serverless. You can't use serverless for any of the major cloud providers until you understand that cloud provider. So yeah, do your 12 weeks of cloud school. And there's more than enough providers.Corey: Whoa, whoa, whoa. Before you spin up a function that runs code, you have to understand the identity and security model, and how the network works, and a bunch of other ancillary nonsense that isn't directly tied to business value.Ant: And all these fun things. How are you're going to test this, and how are you're going to do all that?Corey: How do you write the entry point? Where is it going to enter? What is it expecting? What objects are getting passed in, if any? What format is it going to take?I've spent days, previously, trying to figure out the exact invocation for working with a JSON object in Python, what that's going to show up as, and how specifically to refer to it. And once you've done that a couple of times, great, fine, it's easy. Copy and paste it from the last time you did it. But figuring it out from first principles, particularly in a time when there isn't a lot of good public demonstrations of this—especially early days—it's hard to do.Ant: Yeah. And they just love complexity. Have you looked at the second edition—so the third version of the AWS SDK for JavaScript?Corey: I don't touch JavaScript with my hands most days, just because I'm bad at it and I don't understand the asynchronous model and computers are really not my thing most.Ant: So, unfortunately for my sins, I do use JavaScript a lot. So, version two of the SDK is effectively the single most popular Cloud SDK of any language, anything out there; 20 million downloads a week. It's crazy. It's huge—version two. And JavaScript's a very fast-evolving language, though.Basically, it's a bit like the English language in that it adopts things from other languages through osmosis, and co-opts various other features of other languages. So, JavaScript has—if there's a feature you love in your language, it's going to end up in JavaScript at some point. So, it becomes a very broad Swiss Army knife that can do almost anything. And there's always better ways to do things. So, the problem is, the version two was written in old JavaScript from years twenty fifteen years five years six kind of level.So, from 2015, 2016, I—you know, 2020, 2021, JavaScript has changed. So, they said, “Oh, we're going to rewrite this.” Which good; you should do. But they absolutely broke all compatibility with version two. So, there is no path from version two to version three without rewriting what you've got.So, if you want to take anything you've written—not even serverless—anything in JavaScript you've written and you want to upgrade it to get some of the new features of JavaScript in the SDK, you have to rewrite your code to do that. And some instances, if you're using hexagonal architecture and you're doing all the right things, that's a really small thing to do. But most people aren't doing that.Corey: But let's face it, a lot of things grow organically.Ant: Yeah.Corey: And again, I can sit here and tell you how to build things appropriately and then I look at my own environment and… yeah, pay no attention to that burning dumpster fire behind the camera. And it's awful. You want to make sure that you're doing things the right way but it's hard to do and taking on additional toil because the provider decides the time to focus on this is a problem.Ant: But it's completely not a user-centric way of thinking. You know, they've got all their 14—is it 16 principles now? Did they add two principles, didn't they?Corey: They added two to get up to 16; one less than the numbers of ways to run containers in AWS.Ant: Yeah. They could barely contain themselves. [laugh]. It's just not customer-centric. They've moved themselves away from that customer-centric view of the world because the reality is, they are centered on the goals of the team, the goals of the GM, and the goals of that particular product.That famous drawing of all the different organizational charts, they got the Facebook chart, and the Google Chart, and the Amazon chart has all these little circles, everyone pointing guns at each other. And the more Amazon grows, the more you feel like that's reality. And it's hurting users, it's massively hurting users. And we feel the pain every day, absolutely every day, which is not great. And it's going to hurt Amazon in the long run, but short-term, they're not going to see that pain quarterly, they're not going to see that pain, probably within 12 months.But they will see the pain long run. And if they want to fix it, they probably should have started fixing it two years ago. But it's going to take years to fix because that's a massive cultural shift to say, “Okay, how do we get back to being more customer-focused? How do we stop that organizational targets and goals from getting in the way of delivering value to the customer?”Corey: It's a good question. The hard part is getting customers to understand enough of what you put out there to be able to disambiguate what you've built, and what parts to trust, what parts not the trust, what parts are going to be hard, et cetera, et cetera, et cetera, et cetera. The concern that I've got across the board here is, how do you learn? How do you get started with this? And the way that I came into this was I started off, in the early days of AWS, there were a dozen services, and okay, I could sort of stumble my way through it.And the UI was rough, but it got better with time. So, the answer for a lot of folks these days is training, which makes sense. In the beginning, we learned through things like podcasts. Like there was a company called Jupiter Broadcasting which did a bunch of Linux-oriented podcasts and learned how this stuff works. And then they were acquired by Linux Academy which really focused on training.And then A Cloud Guru acquired Linux Academy. And then Pluralsight acquired A Cloud Guru and is now in the process of itself being acquired by Vista Equity Partners. There's always a bigger fish eating something somewhere. It feels like a tremendous, tremendous consolidation in the training market. Given that you were one of the founders of A Cloud Guru, where do you stand on that?Ant: So, in terms of that actual transaction, I don't know the details because I'm a long time out of A Cloud Guru, but I've stayed within the whole training sphere, and so effectively, the bigger fish scenario, it's making the market smaller in terms of providers are there. You really don't have many providers doing cloud-specific training anymore. On one level you don't, but then another level, you've got lots of independent folks doing tons of stuff. So, you've got this explosion at the bottom end. If you go to Udemy—which is where A Cloud Guru started, on Udemy—you will see tons of folks offering courses at ten bucks a pop.And then there's what I'm doing now on homeschool.dev; there's serverless-focused training on there. But that's really focused on a really small niche. So, there's this explosion at the bottom end of lots of small people doing lots of things, and then you've got this consolidation at the top end, all the big providers buying each other, which leaves a massive gap in the middle.And on top of that, you've got AWS themselves, and all the other cloud providers, offering a lot of their own free training, whether it's on their own platforms—there's aws.training now, and Microsoft have similar as well—I think it's learn.microsoft.com is theirs. And you've got all these different providers doing their own training, so there's lots out there.There's actually probably more training for lower costs than ever before. The problem is, it's like the complexity of too many services, it's the 17 container problem. Which training do you use because the actual cost of the training is your time? It's not the cost of the course. Your time is always going to be more expensive.Corey: Yeah, the course is never going to be anywhere comparable to the time you spend on it. And I've never understood, frankly, why these large companies charge money for training on their own platform and also charge money for certifications because I don't care what you're going to pay for those things, once you know a platform well enough to hit a certification, you're going to use the thing you know, in most cases; it's a great bottom-up adoption story.Ant: Yeah, completely. That was actually one of Amazon's first early problems with their trainings, why A Cloud Guru even exists, and Linux Academy, and Cloud Academy all actually came into being is because Amazon hired a bunch of folks from VMware to set up their training program. And VMware's training, back in the day, was a profit center. So, you'd have a one-and-a-half thousand, two thousand dollar training course you'd go on for three to five days, and then you'd have a couple hundred dollars to do the certification. It was a profit center because VMware didn't really have that much competition. Zen and Microsoft's Hyper V were so late to the market, they basically own the market at the time. So—Corey: Oh, yeah. They still do in some corners.Ant: Yeah. They're still massively doing in this place as they still exist. And so they Amazon hired a bunch of ex-VMware folk, and they said, “We're just going to do what we did at VMware and do it at Amazon,” not realizing Amazon didn't own the market at the time, was still growing, and they tried to make it a profit center, which basically left a huge gap for folks who just did something at a reasonable price, which was basically everyone else. [laugh].This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of "Hello, World" demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking databases, observability, management, and security.And - let me be clear here - it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself all while gaining the networking load, balancing and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build.With Always Free you can do things like run small scale applications, or do proof of concept testing without spending a dime. You know that I always like to put asterisks next to the word free. This is actually free. No asterisk. Start now. Visit https://snark.cloud/oci-free that's https://snark.cloud/oci-free.Corey: The challenge I found with a few of these courses as well, is that they teach you the certification, and the certifications are, in some ways, crap when it comes to things you actually need to know to intelligently use a platform. So, many of them distill down not to the things you need to know, but to the things that are easy to test in a multiple-choice format. So, it devolves inherently into trivia such as, “Which is the right syntax for this thing?” Or, “Which one of these CloudFormations stanzas or functions isn't real?” Things like that where it's, no one in the real world needs to know any of those things.I don't know anyone these days—sensible—who can write CloudFormation from scratch without pulling up some reference somewhere because most people don't have that stuff in their head. And if you do, I'd suggest forgetting it so you can use that space to remember something that's more valuable. It doesn't make sense for how people interact with these things. But I do see the value as well in large companies trying to upskill thousands and thousands of people. You have 5000 people that are trying to come up to speed because you're migrating into cloud. How do you judge people's progress? Well, certifications are an easy answer.Ant: Yeah, massively. Probably the most successful blog post ever written—I don't think it's up anymore, but it was when I was at A Cloud Gurus—like, what's the value of a certification? And ultimately, it came down to, it's a way for companies that are hiring to filter people easily. That's it. That's really it. It's if you've got to hire ten people and you get 1000 CVs or resumes for those ten roles, first thing you do is you filter by who's certified for that role. And then you go through anything else. Does the certification mean you can actually do the job? Not really. There are hundreds of people who are not cer—thousands, millions of people who are not certified to do jobs that they do. But when you're getting hired and there's lots of people applying for the same role, it's literally the first thing they will filter on. And it's—so you want to get certified, it's hard to get through that filter. That's what the certification does, it's how you get through that first filter of whatever the talent tracking system they're using is. That's it. And how to get into the dev lounge at re:Invent.Corey: Oh yeah, that's my reason for getting a certification, originally. And again, for folks who learn effectively that way, I have no problem with people getting certifications. If you're trying to advance in your career, especially early stage, and you need a piece of paper that says you know what you're talking about, a certification is a decent approach. In time, with seniority, that gets replaced by a piece of paper, it's called your resume or your CV, but that is a longer-term more senior-focused approach. I don't begrudge people getting certifications and I don't think that they're foolish for doing it.But in time, it feels like the market for training is simultaneously contracting into only a few players left, and also, I'm curious as to whether or not the large companies out there are increasing their spend with the training providers or not. On the community side, the direct-to-consumer approach, that is exploding, but at the same time, you're then also dealing—forgive me, listeners—with the general public and there is nothing worse than a customer, from a customer service perspective, who was only paying a little money to you. I used to work in a web hosting company that $3,000 a month customers were great to work with. The $2999 a month customers were hell on earth who expected that they were entitled to 80 hours a month of systems engineering time. And you see something similar in the training space. It's always the small individual customers who are spending personal money instead of corporate money that are more difficult to serve. You've been in the space for a while. What do you see around that?Ant: Yeah, I definitely see that. So, the smaller customers, there's a correlation between the amount of money you spend and the amount of hand-holding that someone needs. The more money someone spends, the less hand-holding they need, generally. But the other side of it, what training businesses—particularly for subscription-based business—it's the same model as most gyms. You pay for it and you never use it.And it's not just subscription; like, Udemy is a perfect example of that, you know, people who have hundreds of Udemy courses they've never done, but they spend ten bucks on each. So, there's a lot of that at the lower end, which is why people offer courses at that level. So, there's people who actually do the course who are going to give you a lot of a headache, but then you're going to have a bunch of folk who never do the course and you're just taking their money. Which is also not great, either, but those folks don't feel bad because I only spent 10, 20 bucks on it. It's like, oh, it's their fault for not doing it, and you've made the money.So, that's kind of how a lot of the training works. So, the other problem with training as well is you get the quality is so variable at the bottom end. It's so, so variable. You really struggle to find—there's a lot of people just copying, like, you see instances where folks upload videos to Udemy that are literally they've downloaded someone's, video resized it, cut out a logo or something like that, and re-uploaded it and it's taken a few weeks for them to get caught. But they made money in the meantime.That's how blatant it does get to some level, but there are levels where people will copy someone else's content and just basically make it their own slides, own words, that kind of thing; that happens a lot. At the low end, it's a bit all over the place, but you still have quality, as well, at the low end, where you have these cheapest smaller courses. And how do you find that quality, as well? That's the other side of it. And also people will just trade in their name.That's the other problem you see. Someone has a name for doing X whatever, and they'll go out and bring a course on whatever that is. Doesn't mean they're a good teacher; it means they're good at building a brand.Corey: Oh, teaching is very much its own skill set.Ant: Oh, yeah.Corey: I learned to speak publicly by being a corporate trainer for Puppet and it teaches you an awful lot. But I had the benefit, in that case, of a team of people who spent their entire careers building curricula, so it wasn't just me throwing together some slides; I would teach a well-structured curriculum that was built by someone who knew exactly what they're doing. And yeah, I needed to understand failure modes, and how to get things to work when they weren't working properly, and how to explain it in different ways for folks who learn in different ways—and that is the skill of teaching right there—but curriculum development is usually not the same thing. And when you're bootstrapping, learning—I'm going to build my own training course, you have to do all of those things, and more. And it lends itself to, in many cases, what can come across as relatively low-quality offerings.Ant: Yeah, completely. And it's hard. But one thing you will often see is sometimes you'll see a course that's really high production quality, but actually, the content isn't great because folks have focused on making it look good. That's another common, common problem I see. If you're going to do training out there, just get referrals, get references, find people who've done it.Don't believe the references you see on a website; there's a good chance they might be fake or exaggerated. Put something out on Twitter, put out something on Reddit, whatever communities—and Slack or Discord, whatever groups you're in, ask questions. And folks will recommend. In the world of Google where you could search for anything, [laugh], the only way to really find out if something is any good is to find out if someone else has done it first and get their opinion on it.Corey: That's really the right answer. And frankly, I think that is sort of the network effect that makes a lot of software work for folks. Because you don't want to wind up being the first person on your provider trying to do a certain thing. The right answer is making sure that you are basically 8,000th person to try and do this thing so you can just Google it and there's a bunch of results and you can borrow code on GitHub—which is how we call ‘thought leadership' because plagiarism just doesn't work the same way—and effectively realizing this has been solved before. If you find a brand new cloud that has no customers, you are trailblazing every time you do anything with the platform. And that's personally never where I wanted to spend my innovation points.Ant: We did that at Cloud Guru. I think when we were—in 2015 and we had problems with Lambda and you go to Stack Overflow, and there was no Lambda tag on Stack Overflow, no serverless tag on Stack Overflow, but you asked a question and Tim Wagner would probably be the one answering. And he was the former head of product on Lambda. But it was painful, and in general you don't want to do it. Like [sigh] whenever AWS comes out with a new product, I've done it a few times, I'll go, “I think I might want to use this thing.”AWS Proton is a really good example. It's like, “Hey, this looks awesome. It looks better than CodeBuild and CodePipeline,” the headlines or what I thought it would be. I basically went while the keynote was on, I logged in to our console, had a look at it, and realized it was awful. And then I started tweeting about it as well and then got a lot of feedback [laugh] on my tweets on that.And in general, my attitude from whatever the new shiny thing is if I'm going to try it, it needs to work perfectly and it needs to live up to its billing on day one. Otherwise, I'm not going to touch it. And in general with AWS products now, you announce something, I'm not going to look at it for a year.Corey: And it's to their benefit that you don't look at it for a year because the answer is going to be, ah, if you're going to see that it's terrible, that's going to form your opinion and you won't go back later when it's actually decent and reevaluate your opinion because no one ever does. We're all busy.Ant: Yeah, exactly.Corey: And there's nothing wrong with doing that, but it is obnoxious they're not doing themselves favors here.Ant: Yeah, completely. And I think that's actually a failure of marketing and communication more than anything else. I don't blame the product teams too much there. Don't bill something as a finished glossy product when it's not. Pitch it at where it is.Say, “Hey, we are building”—like, I don't think at the re:Invent stage they should announce anything that's not GA and anything that it does not live up to the billing, the hype they're going to give it to. And they're getting more and more guilty of that the last few re:Invents, of announcing products that do not live up to the hype that they promote it at and that are not GA. Literally, they should just have a straight-up rule, they can announce products, but don't put it on the keynote stage if it's not GA. That's it.Corey: The whole re:Invent release is a whole separate series of arguments.Ant: [laugh]. Yeah, yeah.Corey: There are very few substantial releases throughout the year and then they drop a whole bunch of them at re:Invent, and it doesn't matter what you're talking about, whose problem it solves, how great it is, it gets drowned out in the flood. The only thing more foolish that I see than that is companies that are not AWS releasing things during re:Invent that are not on the re:Invent keynote stage, which in turn means that no one pays attention. The only thing you should be releasing is news about your data breach.Ant: [laugh]. Yeah. That's exactly it.Corey: What do I want to bury? Whenever Adam Selipsky gets on stage and starts talking, great, then it's time to push the button on the, “We regret to inform you,k” dance.Ant: Yeah, exactly. Microsoft will announce yet another print spooler bug malware.Corey: Ugh, don't get me started on that. Thank you so much for taking the time to speak with me today. If people want to hear more about your thoughts and how you view these nonsenses, and of course to send angry emails because they are serverless fans, where can they find you?Ant: Twitter is probably the easiest place to find me, @iamstan—Corey: It is a place for outrage. Yes. Your Twitter user account is?Ant: [laugh], my Twitter user account's all over the place. It's probably about 20% serverless. So, yeah @iamstan. Tweet me; I will probably respond to you… unless you're rude, then I probably won't. If you're rude about something else, I probably will. But if you're rude about me, I won't.And I expect a few DMs from Amazon after this. I'm waiting for you, [unintelligible 00:32:02], as I always do. So yeah, that's probably the easiest place to get hold of me. I check my email once a month. And I'm actually not joking about that; I really do check my email once a month.Corey: Yeah, people really need me then they'll find me. Thank you so much for taking the time to speak with me. I appreciate it.Ant: Yes, Corey. Thank you.Corey: Ant Stanley, AWS Serverless Hero, and oh so much more. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment defending serverless's good name just as soon as you string together the 85 components necessary to submit that comment.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Coronacast
Despite 1000+ cases, a sliver of freedom for NSW

Coronacast

Play Episode Listen Later Aug 26, 2021 9:49


Fully vaccinated NSW residents will be given a very small increase in freedoms, after the state reached more than 6 million vaccination doses earlier this week. The state's Premier Gladys Berejiklian yesterday revealed that from mid-September, people who are fully vaccinated will be able to meet more people or spend more time outdoors - depending on where they live. But it wasn't all good news, with the state revealing more than 1000 new COVID-19 cases - the highest it's been in Australia at any point of the pandemic. So on today's Coronacast, is this a turning point in how lockdowns operate as we move on from trying to eliminate COVID spread? Also on today's show: * How are we supposed to keep following the rules when we get nothing in return? * Please tell us about Lambda and AY3. Should we be concerned?

Black and Highly Dangerous
Episode 190: Delta Lambda Taliban

Black and Highly Dangerous

Play Episode Listen Later Aug 25, 2021 66:48


Tyrell and Daphne discuss the latest news and events following their BhD summer break. The hosts begin by catching up on life (02:08) and sharing “Oh Lawd” news (10:48 ). They also discuss COVID-19 updates (21:30), U.S withdrawal from Afghanistan (42:55), and politics (51:40).  Resources: BhD Patreon - https://www.patreon.com/bhdpodcast  FDA Approves First COVID-19 Vaccine - https://www.fda.gov/news-events/press-announcements/fda-approves-first-covid-19-vaccine  Job openings requiring vaccination are up 90% in August compared to July -  https://www.cnbc.com/2021/08/19/job-openings-requiring-vaccination-are-up-90percent-in-august-compared-to-july.html  MTA Removes Motorman Accused Of Allowing His Girlfriend To Operate D Train In Brooklyn - https://newyork.cbslocal.com/2021/08/16/mta-operator-girlfriend-drives-train-brooklyn-nypd/  T-Mobile Data Hack: What We Know and What You Need to Do - https://www.wsj.com/articles/t-mobile-data-hack-what-we-know-and-what-you-need-to-do-11629404953  What you need to know about the California recall - https://www.cnn.com/2021/08/21/politics/california-recall-election-what-to-know/index.html 

The REAL David Knight Show
Fri 20Aug21 Metaverse — World Elites are Preparing to Imprison YOU In

The REAL David Knight Show

Play Episode Listen Later Aug 20, 2021 178:05


TOPICS by TIMECODE 18:26 It was predictable that the internet would be a dystopian Big Brother.  In fact, this guy predicted 35 yrs ago that no one would care if data were collected by a lot of companies for apparently innocent reasons 26:06 Metaverse VR is the World Elites are Preparing for YOU. No work, no travel, no meat, a micro-apartment.  How do they plan on keeping you from rebelling?  A little bread and a lot of circus via Virtual Reality escapism.  Facebook, Microsoft, Wall Street, Silicon Valley are all working together as one to make the unreal your reality. 44:13 Hackers steal over $611 MILLION in crypto 49:15 Big court win in 5G health issues as even more “Havana Syndrome” cases appear. 59:36 UPDATE: HIIDE biometric devices left behind by US military can have 1,000's of people's data on them, available w/o having to access central database and it gets worse… 1:02:43  As schools are closed, requiring masks & jabs — it's time to rethink the institutions; grades were a problem, so Oregon is abandoning “proficiency” in reading, writing, math altogether and focusing on what they consider to be “essential” instead 1:21:26    Former homosexuals are angry about Netflix “documentary” attacking homosexual therapy and Christian conversion — “Pray Away” 1:25:36 Listener letters. Pastors who support themselves with other jobs and kept churches going throughout the entire lockdown.  Things are getting tougher now. 1:35:45 Poland may lose billions in EU carrots/sticks but will not push LGBT agenda on children 1:40:13 I thought this was a parody, but unfortunately, it's real. Here's an example of the kind of sex-obsessed person that could be teaching your child 1:46:12  Afghanistan: Christians & Americans LAST Concern of US Military (if they even get out). Meanwhile, Biden says no Americans will be left behind yet they don't know how many they've flown out, how many remain and they've said they won't go out and help them get to the airport.  And some Americans have been charged as much as $2,000 to return.  What group has been totally abandoned? Afghan Christians. 1:59:00   Trump Mocks Booster as “Money Making Operation”. What about Operation Warp Speed and $20+ BILLION he gave these same “greedy” companies? What about the 2 shots that they've made $20-$30 BILLION EACH? Biden, CDC, FDA do a reversal on boosters but the US Surgeon General wasn't informed apparently. 2:27:24   Unvaccinated Nurses Forced Out, Causing “Breaking Point” at Hospitals. Firsthand reports of why hospitals are “over capacity” — nurses who have worked for the past 18 months as “heroes”, now kicked to the curb because they do not consent to taking an experimental injection. 2:16:46   Listeners' letters.  A letter from Cali-Ban, a father's suicide from lockdown 2:39:09 This is not about controlling a virus to “zero” cases.  It's about making sure your life is “zero”.  Technocrats are cheered and the more draconian the measure, the more the press fawns over it. 2:46:11  Vax or Die.  Organ transplants denied unless you take the jab 2:53:02   Lambda? Delta? Remember when it was only Greek fraternities? But now the graduates of Animal House are running the country and the colleges as novel punishments are being dished out by colleges to unvaxed students. Find out more about the show and where you can watch it at TheDavidKnightShow.com If you would like to support the show and our family please consider subscribing monthly here: SubscribeStar https://www.subscribestar.com/the-david-knight-show Or you can send a donation through Zelle: @DavidKnightShow@protonmail.com Cash App at:  $davidknightshow BTC to:  bc1qkuec29hkuye4xse9unh7nptvu3y9qmv24vanh7 Mail:          David Knight          POB 1323          Elgin, TX 78621  

The David Knight Show
Fri 20Aug21 Metaverse — World Elites are Preparing to Imprison YOU In

The David Knight Show

Play Episode Listen Later Aug 20, 2021 178:06


TOPICS by TIMECODE18:26 It was predictable that the internet would be a dystopian Big Brother.  In fact, this guy predicted 35 yrs ago that no one would care if data were collected by a lot of companies for apparently innocent reasons26:06 Metaverse VR is the World Elites are Preparing for YOU. No work, no travel, no meat, a micro-apartment.  How do they plan on keeping you from rebelling?  A little bread and a lot of circus via Virtual Reality escapism.  Facebook, Microsoft, Wall Street, Silicon Valley are all working together as one to make the unreal your reality.44:13 Hackers steal over $611 MILLION in crypto49:15 Big court win in 5G health issues as even more “Havana Syndrome” cases appear.59:36 UPDATE: HIIDE biometric devices left behind by US military can have 1,000's of people's data on them, available w/o having to access central database and it gets worse…1:02:43  As schools are closed, requiring masks & jabs — it's time to rethink the institutions; grades were a problem, so Oregon is abandoning “proficiency” in reading, writing, math altogether and focusing on what they consider to be “essential” instead1:21:26    Former homosexuals are angry about Netflix “documentary” attacking homosexual therapy and Christian conversion — “Pray Away”1:25:36 Listener letters. Pastors who support themselves with other jobs and kept churches going throughout the entire lockdown.  Things are getting tougher now.1:35:45 Poland may lose billions in EU carrots/sticks but will not push LGBT agenda on children1:40:13 I thought this was a parody, but unfortunately, it's real. Here's an example of the kind of sex-obsessed person that could be teaching your child1:46:12  Afghanistan: Christians & Americans LAST Concern of US Military (if they even get out). Meanwhile, Biden says no Americans will be left behind yet they don't know how many they've flown out, how many remain and they've said they won't go out and help them get to the airport.  And some Americans have been charged as much as $2,000 to return.  What group has been totally abandoned? Afghan Christians.1:59:00   Trump Mocks Booster as “Money Making Operation”. What about Operation Warp Speed and $20+ BILLION he gave these same “greedy” companies? What about the 2 shots that they've made $20-$30 BILLION EACH? Biden, CDC, FDA do a reversal on boosters but the US Surgeon General wasn't informed apparently.2:27:24   Unvaccinated Nurses Forced Out, Causing “Breaking Point” at Hospitals. Firsthand reports of why hospitals are “over capacity” — nurses who have worked for the past 18 months as “heroes”, now kicked to the curb because they do not consent to taking an experimental injection.2:16:46   Listeners' letters.  A letter from Cali-Ban, a father's suicide from lockdown2:39:09 This is not about controlling a virus to “zero” cases.  It's about making sure your life is “zero”.  Technocrats are cheered and the more draconian the measure, the more the press fawns over it.2:46:11  Vax or Die.  Organ transplants denied unless you take the jab2:53:02   Lambda? Delta? Remember when it was only Greek fraternities? But now the graduates of Animal House are running the country and the colleges as novel punishments are being dished out by colleges to unvaxed students.Find out more about the show and where you can watch it at TheDavidKnightShow.comIf you would like to support the show and our family please consider subscribing monthly here: SubscribeStar https://www.subscribestar.com/the-david-knight-showOr you can send a donation throughZelle: @DavidKnightShow@protonmail.comCash App at:  $davidknightshowBTC to:  bc1qkuec29hkuye4xse9unh7nptvu3y9qmv24vanh7Mail:         David Knight         POB 1323         Elgin, TX 78621

Chewing the Fat with Jeff Fisher
Ep 688 | Boosters Unlimited

Chewing the Fat with Jeff Fisher

Play Episode Listen Later Aug 17, 2021 49:07


Shrimp recall… Food prices and shortages / Water / Coffee / Wheat / Beer / Beef… Fast Food changing things up / Taco Bell / Wendy's… Subscribe to the YouTube Channel… Email to Chewingthefat@theblaze.com Subscribe www.blazetv.com/jeffy Promo code jeffy… Yik Yak Back… Tik Tok posts on dishwashers / Tik Tok masked vigilante... Dylan sued for abusing 12 year old girl… Afghan President fled with cash… Taliban leader once in Gitmo… Prez calls Dallas ISD… Seattle police fired for being at the capitol on one six… Booster shots help says Pfizer… Booster shots now advised… Lambda is vaccine resistant, maybe… Pre Pandemic normal is not coming back… Dog muzzle claims were false… Learn more about your ad choices. Visit megaphone.fm/adchoices

DevDiscuss
S6:E2 - Lambda, Fargate, EC2, Oh My! An AWS Service Deep Dive

DevDiscuss

Play Episode Listen Later Aug 17, 2021 56:49


In this episode, we talk about solving problems via Amazon Web Services with Ken Collins, AWS Serverless Hero and staff engineer at Custom Ink, and Vlad Ionescu, AWS Container Hero and DevOps consultant. Show Notes Scout APM (DevDiscuss) (sponsor) Cockroach Labs (DevDiscuss) (sponsor) Amazon Web Services Custom Ink AWS Lambda Amazon Elastic Container Service AWS Fargate DigitalOcean Terraform Kubernetes SaaS AWS SaaS Factory Heroku Sidekiq

The Bike Shed
305: Burnout & Bugs

The Bike Shed

Play Episode Listen Later Aug 17, 2021 50:02


This week Chris talks about Bifunctor optics and introduces an app he's been liking recently called CleanShot X, which is a replacement for the built-in screenshot utilities on OSX. Steph talks about her experience using New Relic Browser Stats to troubleshoot a slow page and burnout. Who's feeling it? (Raise your hand.) How do we identify it? What do we do about it? Svelte Is Beloved! - Stack Overflow Survey (https://twitter.com/sveltesociety/status/1422372693827985409?s=21) Bifunctors (https://stackoverflow.com/questions/41073862/what-are-bifunctors/41075765#41075765) CleanShot X (https://cleanshot.com/) Next.js Image (https://nextjs.org/docs/api-reference/next/image) Transcript: CHRIS: Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. And we're off the rails already, everybody. It's going to be a good one. Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Chris Toomey. STEPH: And I'm Steph Viccari. CHRIS: And together, we're here to share a bit of what we've learned along the way. So, Steph, how's your week going? STEPH: Hey, Chris, it's going really well. We talked recently that I have a new laptop. So I have been migrating the things that I'm accustomed to over to my new laptop, but I also love that clean, fresh start. So as part of that fresh start, I was like, what if I use Safari? What if I just switch? I'm a Chrome user, for the record. I'm pretty sure you know that, but just to share that. I was like, well, what if I just switched and I try out Safari for a while? So that was the thing. CHRIS: So I heard the words try and the phrase that was the thing, but I'm going to probe a little deeper. How'd that go? Was it good, great, not so great? STEPH: Honestly, it was fine. I did enjoy being in a new environment to see how Safari handles bookmarks and then also the inspector. So it was novel to be in a different browser where I really don't spend much time in a different browser other than when I need to test this specific UI bug or things like that. But the reason that I ended up migrating back to Chrome was frankly for Chrome profiles because I really like that I can have this clear separation now between my work life and my personal life, and then it also keeps me signed in. So my personal email versus my thoughtbot one versus before the Chrome profiles, which I'm not sure how recent of an addition that is where Chrome introduced that feature. But before, I just always had to be signed into both, and it was just all together in one spot. But now I really like that I can separate. And it's more intentional where I'm like, oh, I'm going into work mode, so I just want that profile versus I do need to hop over to my personal side for a while. So that was the thing that brought me back. CHRIS: Interesting. I don't take advantage of that at all. I know of the feature, but it's never really called to me. And if anything, I do the opposite. So specifically, this doesn't work in the browser, but on my phone, I use the iOS Gmail client, and I use the unified inbox. So I just have everything come together. And I subscribe to the idea of, I don't know, it's all work and stuff. And hopefully, people aren't sending me a lot on the weekends, and I will defer and snooze and all of that. But that holistic view pulls me in. And so it's interesting that you're just on the other side of that. It totally makes sense. I actually think I'm wrong here. I think I'm doing the wrong, bad thing. But it's interesting just the way we're on the two sides of that. STEPH: I can see the merits for your approach where it all goes to one place. So you have one place to go and triage, and I think that makes total sense. I haven't triaged my personal life well enough that I want it to come into my work life. And that's the one that needs the more immediate response typically. So I want to prioritize all of my work emails and focus on that and then have my personal ones more like, okay, I've got some time, and I want to check on this. But I don't want to blend those together. Because frankly, I need to do some more triage on the personal side if I'm just going to bring it all into one space. CHRIS: Interesting. Yeah, I would almost view it from the other point of view of I want to protect my personal space, and this is obviously not what I do based on what I just said but protect my personal time so that when it's evenings or weekends or whatever, that I'm not seeing work emails in there. I do my best to snooze them and get them out of the way. But if they are coming in, maybe there's something I need to respond to or, I don't know, maybe it's FOMO in a certain way, FOMO but professional FOMO. I don't really know. It's interesting that that's the feature that brought you back. But overall, how was your experience using Safari? I have heard loosely that now most of the browsers are evergreen; even Edge has really caught up and is now implementing features in a similar way. And so Chrome, Firefox, and Edge are very similar. And my understanding is Safari is the one that actually lags behind or even holds back web standards and implementations and things like that. So, did you find any rough edges of that sort, or was it otherwise just fine, and it was mostly the profile stuff that made you switch? STEPH: It was honestly just fine. I also may have not used it long enough to run into any of those rough edges. But overall, it was just fine. It worked. I just got to that point where I've run into this situation before where I'm signed into my personal email, and then I'm signed into my work email. And then I'm going to Google Hangouts, and Google Hangouts gets confused, and it's like, which person are you? And then I have that moment of where I have to sign out of one, or sometimes it just gets complicated. And what I found with profiles is that that's just never an issue. I don't have to worry about it anymore. So yeah, overall, Safari was fine. I wouldn't mind going back to using it. I just really like the profile feature. This was one of those moments where it helped me notice how much I really liked that feature because I had just opted into it a while back. But this was that moment where I was like, oh yeah, I really miss that. So I'm going to go back to it. CHRIS: The thing you said a minute ago about which person am I? [chuckles] There's a deep philosophical underpinning there, but I've definitely struggled with that and trying to trick the browser into it. So Trello is the one that I'm struggling with that right now. I am one human in the world, I assure you. But GitHub gets this; GitHub plays the game correctly where I'm one human, but I have multiple internet identities. I am the work person. I am the open-source person. And I'm able to route notifications and things to the different inboxes based on the organization that they're part of, et cetera. And I really liked that, that GitHub seems to understand that I am a human that is multifaceted, whereas Trello does not. And so I use Trello in a professional context a lot, but it's my personal...like, I would have to create a distinct account on Trello. And I'm like, Trello, that's not true; I'm still one person. Just understand this Trello organization is a separate facet of my existence. Got on my soapbox on that part. The other thing I want to say is I do feel bad about the fact that I'm just on Chrome. Because increasingly, Chrome has got so much of the market share, and it's becoming the new deeply dominant thing. And so I want to be the agent of change or like, no, we should use different browsers, and we should support them and make sure that we're testing against them and all of that. And then I'm just on Chrome all the time, and I feel bad about it. But it's one of those like; I have so much muscle memory and built-up knowledge around how to use Chrome. And I've just used it for so long now that the switching cost would be pretty high, I assume. I actually haven't even really tried, but I feel bad about it. I'm now saying two things which are I feel bad about it, and I've never even really tried. So I don't feel great in this moment, but these are my truths. STEPH: [laughs] Well, I can plus-one your truths. Those resonate with me. Well, there's always stuff that I am trying that's new all the time. So I feel like I need some constant in my life until I'm ready for other things to be the constant in my life. And then I can muck around with which browser I'm using and change other things. And you have to be in that moment. You have to be ready for it. So going back to a thing that you said a minute ago about separating your work life and your personal life, I very much like that framing, and that's a nice segue, frankly. And it's something that I've been thinking about where you and I often start with technical topics. But I have a very people-centric topic that I'd love to chat with you about today, and it is emphasis on burnout. And who's feeling it? How do we identify it? What do we do about it? And that's been very much on my mind because I have noticed a lot of people around me, including myself; I just feel like we are more inclined to experience burnout right now or are going through it actively. And it feels even more important to have those conversations with each other and with ourselves to talk about what does it look like when we're burned out? How do we recognize when we're there? Because often, when we are burned out, it's not something that happens gradually, or at least it's not something that we notice happening gradually. It's like, okay, I'm fine. And then suddenly, okay, I'm burned out. And I'm at a place where then I can't really focus. I feel overwhelmed. I'm drained. For anyone that is less familiar with burnout, one, hooray, and then two, burnout is a state of emotional, physical, and mental exhaustion that can be caused by excessive and prolonged stress. So then that's what leads to us feeling very overwhelmed, emotionally drained, unable to meet constant demands, or those are the things that are causing our burnout. So I have been doing what I typically do when I'm thinking about a particular topic, and then I'm looking to gather knowledge and information is I try to go pretty wide where I start looking for podcasts, books, just people that are having similar conversations and trying to synthesize a lot of the information that they're sharing. And I have found some really good stuff. The question is now, what does one do once one has that content, and then how do you bring it back and help apply that content to yourself and then people that are around you? So I have some thoughts there, but before I go further, I'm curious, what's your experience with burnout? CHRIS: I think for me, I've been somewhat lucky in that I don't think I've ever really got into an acute period of burnout, but I've definitely had periods where I just felt weighed down. And there are ebbs and flows of life, and of work, all of those things. There is something that I've been thinking about recently, which is the inherent nature of the work that we do where consistently we're working on something where we don't quite know how to do it, and we're struggling, and we're struggling. And finally, we figure out how to do the thing, how to trick the computer into doing what we want. And then the minute we do that, in trying to encode, ideally, we're automating that away. And we take that solved problem, and we just ship it off. And then, we pick up the next unsolved problem from the list. This isn't entirely true, but it feels like sometimes unspecificity in sort of I'm just working on things that are underdefined, underspecified, and I'm trying to solve little puzzles constantly. And, I don't know, I was feeling that recently of the nature of the work was burning me out. And I think earlier on in my career; I definitely experienced this in a certain way. And then, in the middle of my career, there was this perfect inflection point of my skills and the level of tasks that I was going for. But then, the further I got into my career, the more I tend to take the weird underspecified stuff and anything that's relatively clear I'm giving to other folks on the team. I'm like, "Oh, here's this well-defined piece of work here. Can you go implement this admin page?" Whereas I am doing the investigate integrating with third-party platform XYZ that uses a SOAP API that isn't documented. I'm like, okay, cool. Let me roll up my sleeves and figure out what that means. And I noticed in my work that that was starting to weigh on me, and I was able to shake that off and shift around some of the tasks that I was doing. But that was a particular form where the work itself was weighing on me. And I took a step back, and I was like, why do we do the things that we do as developers? Because there's something just fundamental like, you have to enjoy that nature of challenge and constantly escalating challenge to a certain degree, I think, to really like this work, but it can be a lot sometimes. So I feel like maybe that's a slight digression from the topic, but it was a thing that I was feeling in this space. And that's a little bit of my story. STEPH: Well, the beautiful thing about that is it highlights everybody experiences burnout differently. So that could really be how someone is experiencing burnout where they're taking on all these very complicated, different tasks, and they're feeling just worn down by that and that they're not able to meet demand. And they get to that point where maybe they lose their interest in tech and coding because they have pressed too hard in one direction. And so then they need to take a step back. As for me, I've been thinking back over the last couple of months because there was once or twice where you and I had, I think a conversation here on The Bike Shed where I shared that things were okay, but I didn't feel like my normal self. I was losing some of my interest and energy for technology and coding. And I'm very fortunate; I love what I do. So the fact that I wasn't feeling that interest was a really big sign to me that something's different; something feels off. And it does vary depending on the client that I'm working with. And I think feeling that burnout then was a mix of some of those client pressures that I was feeling and that I was working perhaps too many hours as I was very interested in that client's success. And then the other stuff was more personal because we only have, to borrow from the spoon theory, we only have so many spoons to give. And so if you have a lot going on in your personal life as well, that's going to detract from the energy that you also have to give to work. Are you familiar with the spoon theory? CHRIS: I am not. STEPH: I recently heard about it, and I can't stop using it now because I really like it. But it essentially...and there's a really great article that we can link to so others can read about it because I'm not going to remember exactly who came up with this theory. But the idea is that each spoon represents a unit of energy. And let's say if you start each day with only ten units of energy and you use spoons to represent that, as someone needs energy from you, maybe it's work, maybe it's a personal commitment, maybe you're dealing with a chronic illness, then you are giving a spoon away to each of those. So at some point, you're going to run out of spoons. And you want to also be mindful of who you're giving these spoons to because you are giving that energy away. CHRIS: I definitely liked the idea of we start each day with a certain amount of energy, and different things can pull from that pool and whatnot. I'm intrigued by spoons as the unit. It just feels like a weird...I got this little bag of spoons that I walk around with, [chuckles], and I give them out throughout the day. I guess it could be anything in there, you know, objects. But, I don't know, spoons are interesting to me. STEPH: I think it's because this person who came up with the idea was literally having a conversation with their friend in a cafe. And so that was just something that was in front of them. And they're like, oh, I can use spoons to represent. Well, we'll have to double-check the article to make sure, but I think that's why spoons became the representation. So circling back to once you're in burnout, what do you do with it? And that is one of my questions right now. And that's what I'm trying to synthesize a lot of information around. Because once you're in that state, I don't know of a lot of great ways to help other than take time off because, at that point, you're in a crisis state. And you need to step away, and you need to find out how you can recover from having entered this state of crisis. So that feels really important to identify ways that once someone is in that state, that then we can help them. And that feels good. We can advise someone to take PTO. I still don't feel great about it in terms that then, as a manager myself, I don't really know of other helpful ways to then help someone through that period. So then I really started thinking about the fact that once someone is in that burnout stage, frankly, it's too late. We have let someone get to that point that now they are in that crisis instead of addressing it early on. So that is the other thing that's on my mind is one, how do we help people that are already in that crisis state? But then two, how do we start identifying that someone is starting to go in that direction? And then how do we help them tell us? How do we then triage those situations? How do we prevent them from getting to that burnout state? And that's where I've also found some really good content. And specifically, there is a podcast that I've started listening to called The Burnout Show. They essentially share their experience with burnout, and what they did about it, how they recovered from it, and then how they continue to fight it because a lot of people then still go back to the workforce. So then, once you do find a way to recover, then how do you go back to work? And there have been some really great episodes. And I'll be sure to include a link for it in the show notes. There's one particular episode with Grant Gurewitz, who is a guest on the show. And he speaks specifically to the strategy of Three Good Pockets. And this speaks to the idea that there are many things that we can't control in our day. It could be work, family, other commitments, but we can strive for Three Good Pockets of time where we focus on something that's just for us. This is time that's reserved for you and any activity that you find restorative or joyful. And each pocket can vary in size. So perhaps that first pocket is spent just reading a few pages from a book that you're enjoying, and then the next pocket of time is spent outside or calling a friend. And Grant also has a great suggestion around if you're worried that you'll get sidetracked and not actually step away, which I felt called out for that one because that one's definitely me. I will have good intentions, but then I won't actually take the break that I set for myself. So Grant recommends creating a list of restorative activities so that way when it is time for that break when your calendar is reminding you, then you have a list of these activities to choose from. So it makes it easier to say, okay, then I can do this for a couple of minutes, and I can truly step away from work and step away from my screen. But especially now, when so many of us when we're sharing our workspace with our restorative space, for everybody who is still working from home or working remotely, then creating those daily breaks are incredibly important to our wellbeing. And so, it has me thinking about what restorative activities can I add to my day? How can I encourage other people to add more restorative activities to their day? So I really appreciated that advice. And I have noticed that the idea of burnout, but not so much burnout specifically, I've been thinking of it as recovery and balance is a theme for me. And it is something that I am purposely choosing as a theme right now where I want to research and understand more of how we handle these situations and continue to make progress not just for myself but also for my team. CHRIS: I think finding that right cadence and structure and way to reinvest in yourself and ideally gain more spoons if that is at all possible or at least defend the spoons that you have, those all feel very meaningful. I do have a question. I'm interested in your thoughts on this. I feel like we hear about burnout a lot in our industry. I get the sense maybe that it is a more common thing. Like, I hear so many developers talking about how their dream is just to give up tech and go get a cabin and just farm in the woods or something like that. And I wonder, is it a more pervasive thing in our industry? So that's one question. Another is just an observation that we actually do work in a wonderfully...it's an amazing industry where being a developer, there are so many jobs out there. And I don't want to discredit anyone's efforts if they're earlier on and struggling with that. But broadly speaking, it is a developer's market trying to go out there and get jobs and extremely well-compensated, as a general rule. But does that come with this inherent burnout? And if so, which I'm not sure is true, I wonder if maybe we're just more vocal and maybe we actually share more in public. We have more blogs and podcasts and things like that. And that's just a common thing for developers, and so we hear the stories more often, whereas maybe in other industries, it is actually very common, but people are suffering in silence. But also I do wonder, our industry is still so young. The work that we're doing is changing constantly, and that churn and that working in the unknown maybe there is an inherent nature. So that's a bunch of pontifications off the top of my head. And I have no idea what the answer to any of them is. But I am intrigued because it does feel like the shape of burnout as a concept in the developer world is perhaps a little overrepresented, or maybe it shows up more than I would expect. And, I don't know, is the work that hard? I don't know. But then I hear these stories constantly, and I definitely have felt it myself, so maybe. STEPH: Yeah, maybe. Yes, I do think the work is that hard for the record. It's challenging work. I enjoy it, but it is challenging work between figuring out the tech but then also everything else that comes with that. I don't have anything to back this up, but I suspect that a lot of other industries are also experiencing burnout. And I just happen to be more aware of it right now because I'm hearing it more from my friends and the people that I work with. And I suspect that's more directly related to we all just went through 2020, and probably a number of us were trying to forge ahead and get through that time. And so there may be a lot of us that are just now dealing with those consequences of where we just pushed ourselves through a very hard time. And now a lot of that is manifesting and surfacing around really identifying the damage that we may have done to ourselves by just prioritizing work and trying to put our head down and get the work done even though there was so much happening around us. And I suspect that may be a contributing factor is that now people are really starting to recognize, like, oh, I feel this way. And maybe there's time for me to address it. Or frankly, it may not even be that there's time, but your body is just like, okay, I'm done. I made it through the past year or however many months, and I'm going to start shutting down on you. I've given you all the warning signs, but now we're here. We're at a breaking point. So I don't know about the other industries, but I do know the reason that it's more on my mind is because I'm just hearing it more from people, and they're just expressing it. And so, it has become more of a focal point for me, and I've experienced it myself more recently. I'm sure I experienced this back early on in my career, but I took a strategy of well, I'm just a junior, and I just have to get through this. And I have to build experience. For the record, that is not a healthy mentality. I'm just being honest about where I was in my life. And so, I didn't really stop to think about it, but perhaps it is becoming more normalized where people are having more open, honest discussions about where they're at. And if other industries aren't talking about this, I would love for them to. So to round that out a bit, this is something that is just very interesting to me. It's very top of mind. So I suspect I will be sharing a lot more content in future episodes that are just around this. How do we recover? And then how do we balance? How do we work hard without burning out? CHRIS: Work hard, play hard; those are the two placards that you have. Well, I look forward to continued conversations on all of those topics because they are sort of that's the story that underpins all of the work that we do. So I'm very interested to chat more about that. STEPH: Thanks. So what's going on in your world? How's your week been? CHRIS: Oh, my week has been fine. My topics are going to be way more mundane and tech-focused. But let's see, a couple of things, so one is that Stack Overflow has their I think it's Annual Developer Survey. And this year, the results came out, and there was an interesting standout, which was that Svelte was the most beloved framework, which was very exciting to see. Granted, you always have to take these sorts of stats with a grain of salt. But Svelte was 71% loved and 23% dreaded, which they give it as a ratio of how many people really love this thing versus how many people really hate this thing. And so Svelte, 23% of people who have used it are like, I hate that, but 71% loved, so that's a 48% net approval rating. Versus React which was 69% loved, 31% hated or dreaded as the word would be, so that's a 38% net approval. And then Vue, interestingly, was 64% loved, 36% dreaded for a 28% net approval rating. So, yeah, Svelte was decidedly winning in that. But again, the big grain of salt there is looking at the usage stats. React has 40% usage. So of all the respondents, 40% of the people responding to the survey were like, yeah, I've done React professionally, which is a wildly high number for a JavaScript framework. Vue was at 19%, so roughly half of React's usage, which I'm actually impressed that Vue is that high. And Svelte came in at 3%, so it's definitely still in the early adopter strong fan phase. So it makes sense that they would have this outsized high rating. I'm actually surprised that Vue wasn't higher than React, given that. Because I feel like more people are cajoled into React versus Vue can be more of a choice. And I would have expected this to shape out a little bit differently, but yeah, that's the story. STEPH: That's really cool. I liked how you described that as in the very early adopters' strong fan base stage. CHRIS: But nonetheless, the people that are using Svelte do seem to really like it; that's coming through in these numbers. And that definitely is my experience. I love Svelte and would love to continue using it for as long as possible. But really, I want a lot of other people to start using it. I want to really grow the usage base so that there are more libraries, and frameworks, and blog posts, and just mindshare in that space because I really do believe there are some wonderful ideas in Svelte. And it's just so straightforward to implement things that I just want more people hanging out. So that's one quick thing. Another quick thing is, I've been using a utility lately or a program called CleanShot X, which is a replacement for the built-in screenshot utilities on OSX, and it is just fantastic. So I can capture a screenshot. I can capture a window. You can capture a GIF or a video. And then you can do little trims and annotations. And then it has this really nice feature where after you take a screenshot, it just hovers in the bottom corner of your screen and is easily accessible. So if you take a video, and then you want to upload it to a Trello card, it's just floating there waiting for you. You can actually dismiss it and push it down, but it's still peeking up from the bottom of your screen, and you can pull it back up, and you can have a couple of them. But it just really makes the whole workflow of grabbing screenshots or videos so easy. And I cared deeply about that because now that I have this tool, I'm all the more inclined to grab a screenshot or a video with just about every piece of work that I do. So it's going into pull requests; it's going into Trello cards. And it's so nice to have a utility that just really makes that as easy as possible. STEPH: I really liked how you mentioned that you can annotate because I often...I'm laughing as I'm thinking about this. When I am taking a video of something that I'm going to share with someone, I will use my mouse to indicate, oh, this is important. And so I circle around it and do silly things with my mouse to try to indicate but being able to annotate would be so much nicer. I know there is another tool that you're really excited about that I can't remember off the top of my head right now. Do you know the name of the tool I'm thinking of? CHRIS: Was it Loom? STEPH: Yes, Loom, because I also used that for a little while, and I've really enjoyed it. So I'm curious, how does Loom and CleanShot X stack up? Is one replacing the other, or are they complementary tools? CHRIS: Mostly complimentary. Loom is great because it hosts the videos, and you can also do audio capture, although I wonder if CleanShot has that as well. CleanShot also, I think, has a hosting thing. So I think there's a strong overlap in their functionality, but right now, I'm using both. And definitely for screenshots and things, CleanShot owns that end of it. And I think it's more likely that I could have CleanShot as the entire tool that I'm using. But I'm still using Loom for this is a walkthrough where I'm going to talk to you about a thing. I want to make it available at a URL that everyone can see rather than actually getting a GIF or MOV artifact file on my computer. So ever so slightly different, but I think of them, CleanShot X is probably the ideal one. But yeah, I'm still reaching for both. So the one other thing I did want to talk about is I have been expanding our use of the dry-monads within the project that I'm working on. And I've done some things. I did some stuff, Steph, and I think it's good. STEPH: Shtuff with Shteph. [chuckles] CHRIS: Shtuff with Shteph, yeah. I'm definitely pushing the envelope of how much we're leaning on these concepts within the app, and I continue to question it. I'm really intrigued to see what happens when other folks come into the project, and they're like, "Why can't I just get the value? It should be a string. Why isn't it a string? Why is it a string that I have to do a ceremony and a dance to get at?" And I'm like, "Well, because everything can fail, you know, like life." But what I have done here so dry-monads is the project that we're using, particularly their result type. So the result represents something that can either succeed or fail. And so we either have a success, which is this wrapper around the value that's successfully executed. So say we make an API call, we get back a response. If we get a 200 or maybe even a 300, then we get the data, and that's a success, or we get a failure and the error message. But fundamentally, we're modeling that in our system in a way that downstream from that, we have to basically determine if it was success or failure. So we're really encoding into the system; listen, pretty much everything can fail, so let's be careful with that. Let's be intentional and purposeful with it. But there is an interesting thing where these objects have fmap as a method on them. So fmap is a way to transform that wrapped value, but fmap works specifically on the success case. So if you make an API request, you get back the data. Everything's great. You can call fmap, and it will yield into a block that data. You can transform that data in some way, and then it will rewrap it up as a success object. So you can operate on this thing as if it has been successful. But in the case that it's a failure, it will just ignore that transformation because you don't want to transform the failure. It's going to be a totally different shape of data. So you want to separate those. We're getting into functors and monads here. So I'm going to handwave a bunch. But fundamentally, that's the thing that we're going for here. But we found ourselves really wanting to work with both sides. So we make this API request, and in the case that it succeeds, we actually want to transform and actually slice out a piece of data from the nested object that we get back. So that's one transformation that we want to apply on the success portion of the aisle. But then, we also want to transform the failure message. It turns out this backend is giving us very unfriendly error messages. So we want to take those and transform them into friendlier user-facing error messages. So it turns out we want to map both sides. And so I went to dry-monads, and I was like, what do you got? I want to know about this in the world. And it turns out they did not have anything. So I started looking into it, and it turns out this is a concept in the world of functors, specifically. Or, more specifically, I reached out to a former colleague, Sid Raval, a former thoughtboter as well. And he likes the functional programming stuff, so I knew he was the right person to ask about this. And he pointed me at bifunctors. So I found myself in a new space and category theory which I never thought I would explore a category theory in this way, but here we were. So a bifunctor basically is exactly what I was talking about where there are these two branches. In our case, it's either success or failure, but it allows you to operate on both sides, both branches. So the method or the function that gets applied there is bimap. So it's fmap which I don't know why it's f why that's typically what it's called. Success map would be a really great word in this context in my mind. So success map only deals with the success side, but bimap takes two different transformations, one for the successful outcome and one for the failure outcome. And it allows you to very directly talk about what you want to do with that. To be clear, dry-monads has a function called Either or Either, depending on how you want to pronounce it. And that takes two Lambda proc-type things because it's Ruby, and functions are kind of weird in Ruby. But it yields you either the successful value or the failure value, but then it doesn't rewrap them. So it's meant to be the terminal. You use that in a controller when you're either redirect or render or whatever it is you want to do. What I wanted was something for mapping, so staying in the success object or the failure object but yeah, bimaps. So I introduced my own extra wrapping layer. This is where things go off the rails, I think. We now have our own internal result objects. I thought about monkey patching for a while. I convinced myself monkey patching was a bad idea. Now that I've implemented as an extra layer of wrapping and I got the wrapping wrong like four times, or I kept recursively wrapping and re-wrapping, and there's a reason people aren't supposed to write these things themselves. But I think monkey patching may have been a better idea here, or maybe I should have never done any of this. We ended up with a stable working implementation and a nice test suite that covers it. But I introduced bimap and failmap as two different methods on our success object. And I did it by doubly wrapping the result objects. So we have our internal result, which wraps the dry-monad result. And I'm worried about that future situation where a junior developer comes on the team and is like, "I don't know what any of this is." STEPH: I love the weekly progression of I've done some things, and let's talk about it. And then seeing this glimpse into your argument with yourself as to yes, but we need it, and we want it, and it's not something that's defined. So let's go ahead and implement it. You ended on a high there where you talked about the fact that it is nice to work with. It does the thing that you'd like it to do, and it's well tested; I love that part. I'm deciding which thread to go with because there were a lot of interesting bits in everything that you shared. And I'm intrigued about the monkey patching as to why you think that could have been a better approach. Could you talk more about that one? CHRIS: Sure. So I ended up having to introduce the secondary object that then wraps the result. And so if you poke at that even a tiny bit, you start to see this like Russian doll nesting of it's our result wrapping a result, wrapping a value. And it's a burrito with three different tortillas wrapped around it. And if you want to add guacamole, you got to unwrap all three burritos. I'm sorry, this is a terrible monad joke here. [laughs] STEPH: For the record, if Taco Bell is not offering that, they should now, now that you've created that. [laughter] CHRIS: There is one, but there's usually cheese in between the layers. They're not just extra layers of tortillas, so that's what I've got here. It's way too much tortilla, and that's sad. You don't want that. And it's confusing. You're like, wait, does this one have two or three layers of tortilla? So when I was working on it and when I was implementing our additional wrapping layer, I tricked myself multiple times. And my test suite was telling me that things were working, but I was testing incorrectly. And I was like, oh man; this is very subtle. And even though I'm deeply immersed in the context here, I'm still struggling with this. So the question is, did I successfully encapsulate all of this? And now anyone downstream just gets to use it,, and everything will be fine. Am I the heroic programmer that made the perfect abstraction that no one's ever going to struggle against, or was that pain that I was feeling representative of the complexity of what I'm trying to do here, and maybe I got it wrong? And so then the monkey patching side is we've got this one layer of wrapping. What if we just monkey patch the result such that it's got these new methods? I'm not adding an additional layer. I don't need to deal with double mapping through multiple layers, and I just get to deal with the context that I have. So I did an initial spike of an implementation that way, but I talked myself out of it because #monkeypatchingisbad, but I don't know. I don't know where I'm ending here. I am happy with where we're at, but I am aware that I may be sad down the road. STEPH: I'm just dying over here [laughs] from everything you're saying. I have this image of you staring out the window thinking, am I the hero, or am I the villain? And figuring out who you are in this scenario with this abstraction that you've created. CHRIS: To be clear, it's raining, and I have a nice rocks glass of scotch in my hand. And I'm just wistfully looking out the window trying to determine what's true. That's pretty accurate, actually. That's pretty much what's going on here. STEPH: I think we're just going to need updates as it progresses along. CHRIS: I will say overall this paradigm...so failures can happen at all levels. These command objects, which are the core of where this is coming in in the application, really represent the workflows of the app in a wonderful, testable, straightforward way. So I love that part. And if I have that, it implies that I have to have this other stuff. I don't think I can get away from that. The other thing is I've always loved Gary Bernhardt's Functional Core, Imperative Shell, which is a conference talk that he gave a while back. I talked to Gary about it actually when he was on the show, and we can link to both that and the conference talk because they are fantastic. And I just love the way he thinks about software. But that was always a little bit abstract for me. Like, what does that actually look like, though, in say, a Rails architecture? And I didn't have a great answer. And now this thing that I'm doing is the closest I think to that where the innards of the system are almost functional even though it's Ruby. We're leaning into that. And so we have these command objects that take in some data, and then they operate on it, and they yield out these results. Did it go well, or did it not? We've got the railway-oriented stuff, which again I'll link to. I link to now every third episode, apparently, but here we are. And that just models the reality of these programs in a really great way. And then we've been trying to introduce some, not rules per se, but guidelines as to how we interact with these things. So inside of the core of the application, we're trying to be as functional as possible. We do transformations on these result objects, but that's it. And then it's only really in the controllers or the mailers that we are doing unwrap or deal with the actual nested value, but everything else is working on conceptual values or whatever it is…these result objects. And that's actually been really nice, and it's allowed us to have really nice error handling within the app. The logging is very straightforward. A lot of apps that I've worked on in the past, I've just silently thrown away many error cases or edge cases. And this is a really great way to sequence the work that needs to be done but never throw away data that you would want. And so far, I'm finding it to be really great. And I'm seeing very obvious ways to hook into it like, oh, this is where we need to find the user-facing error message versus this is where we figure out what we want to log. And so, in some ways, it's great, but I am still open to the idea that this was a terrible idea. I always remain open to that idea to be clear. [chuckles] STEPH: I love how that's the feedback that you're always open to it. You're always trying something new, and then you're constantly going back to revisit; was this a good idea or a bad idea? To go back just a little bit, I do absolutely love the priority and focus that you're giving to the failure state because I feel like that is an area that we, as developers we're very skittish of that failure state. And I realize I'm projecting here. So if you're listening, feel free to take that however you like; maybe it doesn't apply to you, maybe it does. But the failure state is like you said, it's life, it's important, and it's something that is going to happen. And it is something that we should make accommodations for. And I find that we're often very hand-wavy with a failure state. So I love, love how much you always prioritize the failure state and make that something that people can work with and understand. Versus then when something goes wrong, then that's when we have to start to understand the failure state. I recognize there's a balance there because you're not going to know the failure state until you encounter it, but there are ways that we can still optimize to have observability into that failure state for when we do encounter that failure. CHRIS: I've definitely seen that as an evolution in my own thinking, how much am I focused on how easily can I do the core thing, the happy path versus how robustly can I do all of the variations of what this app needs to do? What if the network's down? What do we do there? I do occasionally worry that I've overcorrected on that. And it's like, you know what? This thing that I'm worried about that I'm protecting against in the application is a 0.01% edge case. It's going to affect almost no users, and we're both putting time into trying to avoid it. But also, there's code complexity that comes from trying to handle all the different variants. And so there's definitely an optimization, and I feel like, at different points in my life, I've been undercorrected or overcorrected on that. But I think if I were to describe the arc of my career, it is desperately searching for that optimal path and trying to find exactly the right amount of error handling to apply, and then yeah, then I'll be happy, then it'll be great. But it is like when I look at DHH's classic 15-minute I'm making a blog, look how much I'm not doing, I'm like, sure, sure, sure. Show me that in three years, though. What does the blog look like? How easily can I add a new feature? What happens when there's a bug in production, and a user reports it? Can I chase it down? Can I figure it? Can I fix it? These are the questions that I care about now, almost to the exclusion of what's the first run experience like? I almost don't care about that at this point. Because I spend my time…six months and on that's where the hard work is. And so the first couple of months where you're figuring things out, that's not the hard work of this thing. And so, I'm very strongly focused on those later periods of time. But again, I'm open to the idea that maybe I am overcorrected there. STEPH: I think it does highlight more of a shift in our career. We're still building, but we have experienced the maintenance side as well, and we felt that pain. And so that has led to perhaps overcorrecting, or maybe it's the correct amount of correction. But I do like how you highlighted there is always a cost to each side and those are usually the questions that I'm asking myself when I'm thinking about the failure mode and how much I want to optimize for the failure mode is how much does it cost when this fails? Who's it going to impact? And how much does it cost for me to make this more observable or to address this failure state? And then I try to find the balance between those two. Because you're right, it's not free to address that failure state. And so I may not want to fully optimize to handle that if it's going to be a very small percentage of users that actually are impacted by this failure state, or it seems very rare this is going to happen. But then still finding ways to know that if it does fail, then I can say, "Okay, I'll come back, and now it's worth the investment to improve this." CHRIS: When you said earlier that this is really hard work that we do, I don't know that I believed you. What you just described sounds super easy. You just handle all the stuff, and you dynamically optimize for the needs at the point in time. And that seems easy. [laughs] STEPH: Super easy, yeah. [laughs] Way to bring it back. Well, speaking about observability and failure states, that does lead nicely into a bug that I was working on this past week where there was a particular page that was loading very slowly. And it was something that we'd heard from users that then they let us know that this page was either taking a very long time to load or, frankly, it was just crashing. And then they were never getting to that page. So I happened to be the one that then picked up that ticket. And I went to reproduce the issue, and sure enough, when I clicked on this particular link and then started counting, it took about 14 seconds for that page to load, which is a very long time. And then also sometimes it was just crashing. So the first place that I went was to our error tracking. So I went to New Relic to then look to see okay; maybe there's a slow query. There's something here that's creating this performance issue, but I couldn't find anything. And New Relic does a great job of breaking down all the different response times so I can see how long Postgres is taking, Redis, and Ruby. All of those looked very normal. I couldn't find anything that seemed alarming that was indicating that the page was struggling to load even though I could reproduce the problem. Because I was clicking on it several times thinking, okay, well, if I just do this a couple of times, New Relic's going to notice, and then I'll get to see something, a little breadcrumb that's going to lead me in the right direction. And while I was waiting for New Relic to surface something helpful to me, I mentioned to another developer the issue that I was triaging. And they said, "Yeah, that page has been getting progressively slower, and we don't know why." And I thought, ooh, okay, I'm intrigued even more now as this is something that has been escalating over time, and now we've hit this threshold that we're working on it. And I discovered that in New Relic, I can look specifically at Postgres, Redis, Ruby, all those different response times. But there's a browser monitoring tool that I had not used before. And it showed a lot of helpful information around first paint, First Contentful Paint, window load, all of those areas. So I started diving in and found session tracing, and it was there that then I saw New Relic was telling me, "Hey, you have a page that's taking about 14 to 15 seconds to load." And I thought, okay, I feel validated now that at least New Relic is recognizing this issue. I have seen this issue, but I still didn't know why it's occurring. So the next tool that I used that I don't know if I've used before or it's just been a very long time; it felt fresh in the moment, but it's the Chrome DevTools, the performance tool. And so you can open that up in your inspector, and then you can go to the page that you want to track the performance. And then you can essentially say, "Hey, go ahead and start profiling and reload this page." And it has so many stats when it finally does load. It has CPU flames charts, which essentially it visualizes a collection of all the stack traces. It has a film strip, so you can actually see the rendering progress of your website along different time points. So if you wanted to go back to a specific time, you can see what did the webpage look like at this point? And then if you go a little further, okay, how much was loaded at this point? So there's a lot of interesting and a little overwhelming information that's there. But the thing that did catch my attention is there's a chart. I don't actually know what this chart is called. It's not a pie chart because there's no center to it. So it looks like a donut chart, and it's broken down. And it shows you the loading times, scripting, rendering, painting, all of those different values. And the rendering time was taking 35 seconds. And I was like, ooh, okay, that is meaningful right there. So then further investigation, now that I knew what I was looking for, I wasn't looking for something more on the back end. I was looking for something more on the front end. And I didn't think it was necessarily JavaScript because we also have JavaScript on this page. So at least this was helping me get a little bit closer before then I went into the codebase to start seeing what's happening. So once I knew it was a rendering issue, I went to look specifically at that view, and we have a form on that page that was generating an empty HTML select option for every record that's in the system. So let's say that you're ordering from a restaurant. On this page, there was a form where it had a list of all the restaurants, and, in our particular case, we had about 17,000 restaurants. So there were 17,000 empty HTML selection options, which could have some significant impact on the DOM and page load time. And that was the piece that was really leading to the performance, is the fact that we were rendering that empty select option. So from there, it was then just triaging okay; we don't really need to render all of these restaurants. There are ways we can scope this down. And that way, we're only showing a little bit at a time versus creating all of these empty options. I should clarify they're empty because part of this form is you select from the first dropdown, and then it populates the other one, and it gives you more information. But the way that this form was implemented, it was actually trying to show all of them at once. But it didn't actually have the data yet, but it was doing like a restaurant.all type of count. And so then that's how we were getting that many empty options. So it was a very interesting journey. It was very helpful to learn that New Relic has this browser monitoring tool. And I really appreciated their performance tool. And circling back to Chrome, Safari may have something similar. But I found Chrome's performance tool very helpful because then it helped me realize that it was the rendering. And so then I could really focus on the markup and the view versus knowing it wasn't more in the database layer. CHRIS: I really love the description, almost like a mystery novel of these bugs when we encounter them. Because if you just get to the end and you're like, oh, I was rendering a select, and it had all of this, that loses so much of this story because again, the coding is not the hard part of the work that we do; it's the figuring out what needs to be done. And in this case, that journey that you went on to find the bug. I really like the point where you said, "And someone mentioned this page has just been getting progressively slower over time." I was like, ooh, that's interesting. Now we got a clue. Now we've got a lead, and we'll chase it down, and then finding the browser tools and all of that. And also, as an aside, browsers are just such an immensely impressive piece of technology, everything that they do. And then you add the DevTools on top of them and magical stuff going on there. But yeah, also probably don't render 17,000 empty selects. [laughs] That seems like it will get you in trouble pretty quickly. But also very easy to get to and especially if there is this incremental, slow creep over time where it's like, oh, that page seems like it's a little slower. It's a little slower still, and it just keeps creeping up over time. But yes, I appreciate you taking us on that journey with you. STEPH: Yeah, it was a fun discovery. And it made me realize that while we have alerting set up for some of our other queries, we don't have anything set up for the browser time. So that would be a good optimization on our side is to start alerting us before a page gets to the point that it's taking that long to load to notify us sooner. So we don't have to wait for a user to reach out to us, but we can triage sooner. CHRIS: I also do love the idea of extending the metrics that we hold ourselves accountable to all the way through to the user, and so the First Contentful Paint and all of that. The one that I really love recently that has captured an idea that I struggled to put words to is the Cumulative Layout Shift. Are you familiar with this piece? STEPH: Uh-uh. CHRIS: So there's like, how quickly does the page render? That's the thing that we want to know. But a lot of applications these days, particularly single-page apps, render pretty quickly, but they render what ends up being a skeleton or a shell of the page. And then behind the scenes, there are like ten different AJAX requests happening. And as the data comes in, suddenly, a part of the screen will render. And they'll render a list of items that they just got back from the back end, but they're still waiting for the information to populate the header. And so if you look at that page, it's constantly shifting as it's loading and just feels, I don't know, flimsy in my mind. But I didn't have a good word, or I didn't have a metric or a number to attach to that. And then I learned about Cumulative Layout Shift, and I was like, oh, that's the one. Now you've mapped the thing that I was feeling. And I like when that happens. STEPH: Is that the difference between the first paint and First Contentful paint? Is that similar? CHRIS: I think it's more than that because there are not just two discrete events in this. It can be multiple. And so it's like, how many different times did the thing that's rendered on the screen move around? And so if images are loading, but you didn't have a proper image height and width set, that's another way that this can happen. Then initially, the browser is not going to reserve any space for that because it's like, I don't know how big this is. And then, when the image shows up, it now knows the intrinsic height and width of the image. So suddenly, your page is going to jump from that. You can get ahead of that by putting the height and width on your image, and that's great to do. And frameworks like Next.js have done some really amazing work of making that a build time step as opposed to something you have to do manually. But then also, more generally, how do we handle this? React is doing some interesting work with Suspense, where you can aggregate together multiple different loading states into one collective thing. It's almost like promise.all, but for your page. I haven't followed that too closely, but I know that that's framework-level work that's happening over there. GraphQL does a really great job of allowing you to group queries together. So there's a solution on that side. But broadly, if you just render some HTML on the server and you send it to the front end, then you don't have this problem because you just have one ball of HTML. The browser is pretty good at rendering that in one pass versus if you have single-page applications that are making a handful of AJAX requests that will resolve in their own timelines and eventually paint to the screen. You get this different shape. And then the worst case of it in my mind is you render half the page. And then suddenly, one of the requests realizes the JWT has expired, and suddenly, you get thrashed over to the login page. Please don't give me that experience, developers, please. Please do something else that isn't that. That makes me sad in my heart. STEPH: Prioritize the failure state. That's what I'm hearing. CHRIS: Callbacks. STEPH: Well, on that wonderful circular reference, shall we wrap up? CHRIS: Wait, I thought circular references were bad...Never mind. Let's wrap up. The show notes for this episode can be found at bikeshed.fm. STEPH: This show is produced and edited by Mandy Moore. CHRIS: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review on iTunes,, as it really helps other folks find the show. STEPH: If you have any feedback for this or any of our other episodes, you can reach us @bikeshed, or reach me on Twitter @SViccari. CHRIS: And I'm @christoomey. STEPH: Or you can reach us at hosts@bikeshed.fm via email. CHRIS: Thanks so much for listening to The Bike Shed, and we'll see you next week. All: Byeeeeeeeeee. Announcer: This podcast was brought to you by thoughtbot. thoughtbot is your expert design and development partner. Let's make your product and team a success.

The New Yorker: Politics and More
Atul Gawande Discusses the COVID-19 Resurgence

The New Yorker: Politics and More

Play Episode Listen Later Aug 9, 2021 14:23


For a few brief moments this summer, in places where the vaccination rate was high, we could imagine life after COVID-19: restaurants and theatres were filling up, gatherings of all kinds were taking place, and many businesses were planning to return to their offices after Labor Day. Then the story changed, as the highly contagious Delta variant began sweeping the nation. Atul Gawande, a professor of medicine and an internationally recognized expert on public health, tells David Remnick that the Delta surge has also caused a vaccination surge, which is promising. They discuss the idea of booster shots and the possibility of a future variant that would resist the vaccine and cause more severe breakthrough infections. The Lambda variant, Gawande says, has already reached the U.S., but little is known yet about how it responds to the vaccines in use here.    (Gawande has been nominated by President Biden to lead global health development, including COVID-19 efforts, for the United States Agency for International Development. The appointment awaits confirmation in the Senate.)

The New Yorker Radio Hour
Atul Gawande on the COVID-19 Resurgence

The New Yorker Radio Hour

Play Episode Listen Later Aug 6, 2021 25:14


For a few brief moments this summer, in places where the vaccination rate was high, we could imagine life after COVID-19: restaurants and theatres were filling up, gatherings of all kinds were taking place, and many businesses were planning to return to their offices after Labor Day. Then the story changed, as the highly contagious Delta variant began sweeping the nation. Atul Gawande, a professor of medicine and an internationally recognized expert on public health, tells David Remnick that the Delta surge has also caused a vaccination surge, which is promising. They discuss the idea of booster shots and the possibility of a future variant that would resist the vaccine and cause more severe breakthrough infections. The Lambda variant, Gawande says, has already reached the U.S., but little is known yet about how it responds to the vaccines in use here. Plus, forget the big white tent and the plate of rubber chicken: the real New York style is a City Hall wedding, complete with metal detectors. Vinson Cunningham tells us what it's all about.   (Gawande has been nominated by President Biden to lead global health development, including COVID-19 efforts, for the United States Agency for International Development. The appointment awaits confirmation in the Senate.)

Make Me Smart with Kai and Molly
Chill out and wear a mask

Make Me Smart with Kai and Molly

Play Episode Listen Later Aug 6, 2021 15:53


Feeling a little hollowed out this Thursday? Some in the media are downright apocalyptic. On today’s show, we’ll unpack some of the coverage around COVID-19 variants and what you should (and shouldn’t) be worried about. It does get a little dark though. Later on, guest hosts Kimberly Adams and Meghan McCarty Carino cheer each other up with their favorite Olympic moments. Here’s everything we talked about today: “A Doomsday COVID Variant Worse Than Delta and Lambda May Be Coming, Scientists Say” from Newsweek, and a Twitter doctor’s thoughts on that headline “S.Korea on alert for new Delta Plus COVID-19 variant” from Reuters “The Lambda variant: is it more infectious, and can it escape vaccines? A virologist explains” from The Conversation “U.S. plans to give extra COVID-19 shots to at-risk Americans, Fauci says” from Reuters “Knitting diver Tom Daley shows his masterpiece — a Team Great Britain sweater” from ESPN “Qatar’s Mutaz Essa Barshim and Italy’s Gianmarco Tamberi share high jump gold medal” from The Washington Post Finally, this horse just vibing Read the transcript here. Join us on YouTube Fridays at 3:30 p.m. Pacific/6:30 p.m. Eastern for our live happy hour episode! Subscribe to our channel and sign up for notifications so you don't miss it. Our show needs your voice! Tell us what you think of the show or ask a question for our hosts to answer! Send a voice memo or give us a call at 508-82-SMART (508-827-6278).

The Daily Zeitgeist
I Wear My Sunglasses At Zeit 8/4: Lambda Variant, Eviction Moratorium, Rihanna, Meghan Markle, Obama, Washington Guardians

The Daily Zeitgeist

Play Episode Listen Later Aug 4, 2021 22:32


In this edition of I Wear My Sunglasses At Zeit, Miles and Super Producer Anna Hossnieh discuss the Lambda variant, the eviction moratorium being extended, Rihanna becoming a billionaire, Meghan Markle and Obama's Birthdays, and The Washington Guardians telling fans not to show up in culturally insensitive attire Learn more about your ad-choices at https://www.iheartpodcastnetwork.com

The Daily Zeitgeist
Giannis Atrendtokounmpo 7/21: NBA Finals, COVID-19, Sen. McCarthy, Lakers, Climate Change, USWNT

The Daily Zeitgeist

Play Episode Listen Later Jul 22, 2021 20:34


NBA Finals news, the Delta & Lambda variants of COVID-19, Sen. McCarthy deciding Republicans will do their own Jan. 6th Committee, the Laker's search for a veteran point guard, the effects of climate change on Russia's permafrost, and the US Women's National Team's opening match Learn more about your ad-choices at https://www.iheartpodcastnetwork.com