Podcasts about dc all

Season 5 Episode 4-Justin and Zack are dropping big news for the podcast that you do not want to miss, Talking DC Comics Summer of Superman event along with DC All 1st Crossover event coming soon. linktr.ee/TheComiCompodcast https://www.instagram.com/thecomicompodcast/ Email us TheComiCompodcast@gmail.com

Madame Web brings us the newsSaying goodbye a voice from our ChildhoodPopcorn buckets are out of controlI vant to suck your Wallet!It costs quite a few rings … Hopefully he won't be like HomelanderThe Toy R Us games Breaking Bat - The geek Easy: Cobra Kai S6 Part 2 (Netflix), DC Absolute Power & DC All in Special (LCS, DCU),West Coast Avengers (2024) #1 (LCS, Kindle) Milli Vanilli Doco (P+) Nutcrackers (HULU)The Thunderdome: We discuss Wicked part one and lesser known geek musicalsOur Patreon Producers are Sean, Stella and Henry Nyhus, John Sadorf, Brendan Meyers, Phoenix Sisters Cosplay, Corey in HD, Matthew Kiehl, Kurt KrugGet a free one week trial of our Patreon and check out our new member tiers at Secret Friends Unite PatreonUse our special link https://zen.ai/tW9w96GHjJl0oOlORlg-afOO0JOcbUkaBnWlklytL0c to save 30% off your first month of any #Zencastr paid plan.Follow us on Threads, Instagram & BlueSky: @Secret.Friends.Unite, @toxtra, @toddoxtra, @Secretfriendsunite @CeeThreeCarpenterCheck out our LinkTree for all the ways to reach us

Madame Web brings us the newsSaying goodbye a voice from our ChildhoodPopcorn buckets are out of controlI vant to suck your Wallet!It costs quite a few rings … Hopefully he won't be like HomelanderThe Toy R Us games Breaking Bat - The geek Easy: Cobra Kai S6 Part 2 (Netflix), DC Absolute Power & DC All in Special (LCS, DCU),West Coast Avengers (2024) #1 (LCS, Kindle) Milli Vanilli Doco (P+) Nutcrackers (HULU)The Thunderdome: We discuss Wicked part one and lesser known geek musicalsOur Patreon Producers are Sean, Stella and Henry Nyhus, John Sadorf, Brendan Meyers, Phoenix Sisters Cosplay, Corey in HD, Matthew Kiehl, Kurt KrugGet a free one week trial of our Patreon and check out our new member tiers at Secret Friends Unite Patreon Use our special link https://zen.ai/tW9w96GHjJl0oOlORlg-afOO0JOcbUkaBnWlklytL0c to save 30% off your first month of any #Zencastr paid plan.Follow us on Threads, Instagram & BlueSky: @Secret.Friends.Unite, @toxtra, @toddoxtra, @Secretfriendsunite @CeeThreeCarpenterCheck out our LinkTree for all the ways to reach us

Today we hear Advocate Health's Don Calcagno, Senior Vice President and Chief Population Health Officer for Value Operations and Terry Williams, Senior Vice President and Chief Population Health Officer for Partnerships and Strategy who provide insight into Advocate's participation in the Medicare Shared Savings Program and share with CHESS President and host, Dr. Yates Lennon, the successes that have been achieved along with some of the lessons learned.Well, Don, Terry, thank you for joining us on the Move to Value podcast today. Glad to have you. If you don't mind, Don, we will let you start and just take a few minutes to tell our audience a little bit about yourself. And then Terry, you go next and the role you play at Advocate Health.DC: All right, thanks Dr. Lennon. My name is Don Calcagno, I'm currently the Senior Vice President, Chief Population Health Officer for Value Operations for Advocate Health. I also serve as President of Advocate Physician Partners, which is a large, sophisticated, clinically integrated network in the Chicagoland area. Personally, I'm a lab tech by training, completed my schooling in 1992. I have an MBA as well from Northwestern Kellogg and I've been with Advocate for quite some time in various roles from lab tech to others. But I've been a vice President, Operations, Senior Vice President on OPS and I've been either the President or a Senior Vice President of Population Health at Advocate, Advocate Aurora Health since about 2015. So thanks for having me Dr. Lennon. Look forward to the conversation.Glad to have you. Look forward to it. Terry?TW: Hi, I'm Terry Williams, Chief Population Health Officer with focus on partnerships and strategy for Advocate Health. And in terms of background, I was Chief Strategy Officer at a couple of health systems for about a decade as well as started Population Health at one of them that we'll talk about a little later today and I'm also responsible for looking at how we can tie together the academic enterprise and some of the innovations that are happening there into what we're actually doing in population health. So, to give you one example, there was something called the EFI Electronic Frailty Index that was developed in the School of Medicine. It's the single best indicator we have found for predicting future utilization. And so, we use that to we think really do some unique work in our population health work by incorporating that measure.Yeah, familiar with the EFI and I think you just opened the door for a couple more podcasts right there in that one, one statement. So well, one of the things we wanted to do today with you all is to talk a little bit about the MSSP program and Advocate's participation in that. I know we look forward to hearing about some of the successes as well as the challenges that you all have and are facing. It's interesting the program now is what, 11-12 years old and NAACOS just recently at their fall conference released some stats and I'll read some of those to you. So since 2012, ACO's have saved Medicare 21 1/2 billion dollars in gross savings and 8.3 billion in net savings. So that's since the beginning of the program. For '22, It was the sixth straight year that ACO's delivered net savings to Medicare. 84% of ACO's in 2022 saved Medicare money and almost 60% of them were in two-sided risk arrangements. So when you think about where this program started and when it started, it sounds like success right, we're moving in the right direction. With that...

Aug 16-22- DC- “All things shall work together for [my] good.” A commandment with a promise of having the “destroying angel pass over us and not slay us!” DC 89:1-9; 18-21 --- This episode is sponsored by · Anchor: The easiest way to make a podcast. https://anchor.fm/app Support this podcast: https://anchor.fm/rosa-o-garcia/support

Week March 15-21-DC-“All things must be done in Order.” God is not a God of confusion!” “The armor of God will help me stand up against all evil.” The living prophet is God's mouthpiece on the earth for all people.” --- This episode is sponsored by · Anchor: The easiest way to make a podcast. https://anchor.fm/app Support this podcast: https://anchor.fm/rosa-o-garcia/support

Paul and Dan get an in-depth interview with long-time game designer Jeff Grubb and get answer to some questions we’ve been dying to ask for years. What were the AD&D Open tournament events like, such as Jeff designed and ran, in the early 80’s — and how did they effect the design of the AD&D game itself? What’s the inside scoop on his design for the original Marvel Super Heroes RPG (FASERIP), and how the “Universal Table” idea manage to take over all the other games by TSR for a decade? What was his experience in writing the Forgotten Realms comic book for several years at DC? All that and more great stories from Jeff’s illustrious career. Check out Jeff Grubb’s personal blog here Wandering DMs Paul Siegel and Dan “Delta” Collins host thoughtful discussions on D&D and other TTRPGs every week. Comparing the pros and cons of every edition from the 1974 Original D&D little brown books to cutting-edge releases for 5E D&D today, we broadcast live on YouTube and Twitch so we can take viewer questions and comments on the topic of the day. Live every Sunday at 1 PM Eastern time. Subscribe to Wandering DMs for new episodes

The question of diving into Kubernetes is something that faces us all in different ways. Whether you are already on the platform, are considering transitioning, or are thinking about what is best for your team moving forward, the possibilities and the learning-curve make it a somewhat difficult question to answer. In this episode, we discuss the topic and ultimately believe that an individual is the only one who can answer that question well. That being said, the capabilities of Kubernetes can be quite persuasive and if you are tempted then it is most definitely worth considering very seriously, at least. In our discussion, we cover some of the problems that Kubernetes solves, as well as some of the issues that might arise when moving into the Kubernetes space. The panel shares their thoughts on learning a new platform and compare it with other tricky installations and adoption periods. From there, we look at platforms and how Kubernetes fits and does not fit into a traditional definition of what a platform constitutes. The last part of this episode is spent considering the future of Kubernetes and how fast that future just might arrive. So for all this and a bunch more, join us on The Podlets Podcast, today! Follow us: https://twitter.com/thepodlets Website: https://thepodlets.io Feeback: info@thepodlets.io https://github.com/vmware-tanzu/thepodlets/issues Hosts: Carlisia Campos Josh Rosso Duffie Cooley Bryan Liles Key Points From This Episode: The main problems that Kubernetes solves and poses. Why you do not need to understand distributed systems in order to use Kubernetes. How to get around some of the concerns about installing and learning a new platform. The work that goes into readying a Kubernetes production cluster. What constitutes a platform and can we consider Kubernetes to be one? The two ways to approach the apparent value of employing Kubernetes. Making the leap to Kubernetes is a personal question that only you can answer. Looking to the future of Kubernetes and its possible trajectories. The possibility of more visual tools in the UI of Kubernetes. Understanding the concept of conditions in Kubernetes and its objects. Considering appropriate times to introduce a team to Kubernetes. Quotes: “I can use different tools and it might look different and they will have different commands but what I’m actually doing, it doesn’t change and my understanding of what I’m doing doesn’t change.” — @carlisia [0:04:31] “Kubernetes is a distributed system, we need people with expertise across that field, across that whole grouping of technologies.” — @mauilion [0:10:09] “Kubernetes is not just a platform. Kubernetes is a platform for building platforms.” — @bryanl [0:18:12] Links Mentioned in Today’s Episode: Weave — https://www.weave.works/docs/net/latest/overview/ AWS — https://aws.amazon.com/ DigitalOcean — https://www.digitalocean.com/ Heroku — https://www.heroku.com/ Red Hat — https://www.redhat.com/en Debian — https://www.debian.org/ Canonical — https://canonical.com/ Kelsey Hightower — https://github.com/kelseyhightower Joe Beda — https://www.vmware.com/latam/company/leadership/joe-beda.html Azure — https://azure.microsoft.com/en-us/ CloudFoundry — https://www.cloudfoundry.org/ JAY Z — https://lifeandtimes.com/ OpenStack — https://www.openstack.org/ OpenShift — https://www.openshift.com/ KubeVirt — https://kubevirt.io/ VMware — https://www.vmware.com/ Chef and Puppet — https://www.chef.io/puppet/ tgik.io — https://www.youtube.com/playlist?list=PL7bmigfV0EqQzxcNpmcdTJ9eFRPBe-iZa Matthias Endler: Maybe You Don't Need Kubernetes - https://endler.dev/2019/maybe-you-dont-need-kubernetes Martin Tournoij: You (probably) don’t need Kubernetes - https://www.arp242.net/dont-need-k8s.html Scalar Software: Why most companies don't need Kubernetes - https://scalarsoftware.com/blog/why-most-companies-dont-need-kubernetes GitHub: Kubernetes at GitHub - https://github.blog/2017-08-16-kubernetes-at-github Debugging network stalls on Kubernetes - https://github.blog/2019-11-21-debugging-network-stalls-on-kubernetes/ One year using Kubernetes in production: Lessons learned - https://techbeacon.com/devops/one-year-using-kubernetes-production-lessons-learned Kelsey Hightower Tweet: Kubernetes is a platform for building platforms. It's a better place to start; not the endgame - https://twitter.com/kelseyhightower/status/935252923721793536?s=2 Transcript: EPISODE 18 [INTRODUCTION] [0:00:08.7] ANNOUNCER: Welcome to The Podlets Podcast, a weekly show that explores Cloud Native one buzzword at a time. Each week, experts in the field will discuss and contrast distributed systems concepts, practices, tradeoffs and lessons learned to help you on your cloud native journey. This space moves fast and we shouldn’t reinvent the wheel. If you’re an engineer, operator or technically minded decision maker, this podcast is for you. [EPISODE] [0:00:41.9] JR: Hello everyone and welcome to The Podlets Podcast where we are going to be talking about should I Kubernetes? My name is Josh Rosso and I am very pleased to be joined by, Carlisia Campos. [0:00:55.3] CC: Hi everybody. [0:00:56.3] JR: Duffy Cooley. [0:00:57.6] DC: Hey folks. [0:00:58.5] JR: And Brian Lyles. [0:01:00.2] BL: Hi. [0:01:03.1] JR: All right everyone. I’m really excited about this episode because I feel like as Kubernetes has been gaining popularity over time, it’s been getting its fair share of promoters and detractors. That’s fair for any piece of software, right? I’ve pulled up some articles and we put them in the show notes about some of the different perspectives on both success and perhaps failures with Kub. But before we dissect some of those, I was thinking we could open it up more generically and think about based on our experience with Kubernetes, what are some of the most important things that we think Kubernetes solves for? [0:01:44.4] DC: All right, my list is very short and what Kubernetes solves for my point of view is that it allows or it actually presents an interface that knows how to run software and the best part about it is that it doesn’t – the standard interface. I can target Kubernetes rather than targeting the underlying hardware. I know certain things are going to be there, I know certain networking’s going to be there. I know how to control memory and actually, that’s the only reason that I really would give, say for Kubernetes, we need that standardization and you don’t want to set up VM’s, I mean, assuming you already have a cluster. This simplifies so much. [0:02:29.7] BL: For my part, I think it’s life cycle stuff that’s really the biggest driver for my use of it and for my particular fascination with it. I’ve been in roles in the past where I was responsible for ensuring that some magical mold of application on a thousand machines would magically work and I would have all the dependencies necessary and they would all agree on what those dependencies were and it would actually just work and that was really hard. I mean, getting to like a known state in that situation, it’s very difficult. Having something where either both the abstractions of containers and the abstraction of container orchestration, the ability to deploy those applications and all those dependencies together and the ability to change that application and its dependencies, using an API. That’s the killer part for me. [0:03:17.9] CC: For me, from a perspective of a developer is very much what Duffy just said but more so the uniformity that comes with all those bells and whistles that we get by having that API and all of the features of Kubernetes. We get such a uniformity across such a really large surface and so if I’m going to deploy apps, if I’m going to allow containers, what I have to do for one application is the same for another application. If I go work for another company, that uses Kubernetes, it is the same and if that Kubernetes is a hosted Kubernetes or if it’s a self-managed, it will be the same. I love that consistency and that uniformity that even so I can – there are many tools that help, they are customized, there’s help if you installing and composing specific things for your needs. But the understanding of what you were doing is it’s the same, right? I can use different tools and it might look different and they will have different commands but what I’m actually doing, it doesn’t change and my understanding of what I’m doing doesn’t change. I love that. Being able to do my work in the same way, I wish, you know, if that alone for me makes it worthwhile. [0:04:56.0] JR: Yeah, I think like my perspective is pretty much the same as what you all said and I think the one way that I kind of look at it too is Kubernetes does a better job of solving the concerns you just listed, then I would probably be able to build myself or my team would be able to solve for ourselves in a lot of cases. I’m not trying to say that specialization around your business case or your teams isn’t appropriate at times, it’s just at least for me, to your point Carlisia, I love that abstraction that’s consistent across environments. It handles a lot of the things, like Brian was saying, about CPU, memory, resources and thinking through all those different pieces. I wanted to take what we just said and maybe turn it a bit at some of the common things that people run in to with Kubernetes and just to maybe hit on a piece of low hanging fruit that I think is oftentimes a really fair perspective is Kubernetes is really hard to operate. Sure, it gives you all the benefits we just talked about but managing a Kubernetes cluster? That is not a trivial task. And I just wanted to kind of open that perspective up to all of us, you know? What are your thoughts on that? [0:06:01.8] DC: Well, the first thought is it doesn’t have to be that way. I think that’s a fallacy that a lot of people fall into, it’s hard. Guess what? That’s fine, we’re in the sixth year of Kubernetes, we’re not in the sixth year of stability of a stable release. It’s hard to get started with Kubernetes and what happens is we use that as an excuse to say well, you know what? It’s hard to get started with so it’s a failure. You know something else that was hard to get started with? Whenever I started with it in the 90s? Linux. You download it and downloading it on 30 floppy disks. There was the download corruption, real things, Z modem, X modem, Y modem. This is real, a lot of people don’t know about this. And then, you had to find 30 working flopping disk and you had to transfer 30, you know, one and a half megabyte — and it still took a long time to floppy disk and then you had to run the installer. And then most likely, you had to build a kernel. Downloading, transferring, installing, building a kernel, there was four places where just before you didn’t have windows, this was just to get you to a log in prompt, that could fail. With Kubernetes, we had this issue. People were installing Kubernetes, there’s cloud vendors who are installing it and then there’s people who were installing it on who knows what hardware. Guess what? That’s hard and it’s not even now, it’s not even they physical servers that’s networking. Well, how are you going to create a network that works across all your servers, well you’re going to need an overlay, which one are you going to use, Calico? Use Weave? You’re going to need something else that you created or something else if it works. Yeah, just we’re still figuring out where we need to be but these problems are getting solved. This will go away. [0:07:43.7] BL: I’m living that life right now, I just got a new laptop and I’m a Linux desktop kind of guy and so I’m doing it right now. What does it take to actually get a recent enough kernel that the hardware that is shipped with this laptop is supported, you know? It’s like, those problems continue, even though Linux has been around and considered stable and it’s the underpinning of much of what we do on the internet today, we still run into these things, it’s still a very much a thing. [0:08:08.1] CC: I think also, there’s a factor of experience, for example. This is not the first time you have to deal with this problem, right Duffy? Been using Linux on a desktop so this is not the first hardware that you had to setup Linux on. So you know where to go to find that information. Yeah, it’s sort of a pain but it’s manageable. I think a lot of us are suffering from gosh, I’ve never seen Kubernetes before, where do I even start and – or, I learned Kubernetes but it’s quite burdensome to keep up with everything as opposed to let’s say, if 10 years from now, we are still doing Kubernetes. You’ll be like yeah, okay, whatever. This is no big deal. So because we have done these things for a few years that we were not possibly say that it’s hard. I don’t’ think we would describe it that way. [0:09:05.7] DC: I think there will still be some difficulty to it but to your point, it’s interesting, if I look back like, five years ago, I was telling all of my friends. Look, if you’re a system’s administrator, go learn how to do other things, go learn how to be, go learn an API centric model, go play with AWS, go play with tools like this, right? If you’re a network administrator, learn to be a system’s administrator but you got to branch out. You got to figure out how to ensure that you’re relevant in the coming time. With all the things that are changing, right? This is true, I was telling my friend this five years ago, 10 years ago, continues, I continue to tell my friends that today. If I look at the Kubernetes platform, the complexity that represents in operating it is almost tailor made to those people though did do that, that decided to actually branch out and to understand why API’s are interesting and to understand, you know, can they have enough of an understanding in a generalist way to become a reasonable systems administrator and a network administrator and you know, start actually understanding the paradigms around distributed systems because those people are what we need to operate this stuff right now, we’re building – I mean, Kubernetes is a distributed system, we need people with expertise across that field, across that whole grouping of technologies. [0:10:17.0] BL: Or, don’t. Don’t do any of that. [0:10:19.8] CC: Brian, let me follow up on that because I think it’s great that you pointed that out Duffy. I was thinking precisely in terms of being a generalist and understanding how Kubernetes works and being able to do most of it but it is so true that some parts of it will always be very complex and it will require expertise. For example, security. Dealing with certificates and making sure that that’s working, if you want to – if you have particular needs for networking, but, understanding the whole idea of this systems, as it sits on top of Kubernetes, grasping that I think is going to – have years of experience under their belt. Become relatively simple, sorry Brian that I cut you off. [0:11:10.3] BL: That’s fine but now you gave me something else to say in addition to what I was going to say before. Here’s the killer. You don’t need to know distributed systems to use Kubernetes. Not at all. You can use a deployment, you can use a [inaudible] set, you can run a job, you can get workloads up on Kubernetes without having to understand that. But, Kubernetes also gives you some good constructs either in the Kubernetes API's itself or in its client libraries where you could build distributed systems in easier way but what I was going to say before that though is I can’t build a cluster. Well don’t. You know what you should do? Use a cloud vendor, use AWS, use Google, use Microsoft or no, I mean, did I say Microsoft? Google and Microsoft. Use Digital Ocean. There’s other people out there that do it as well, they can take care of all the hard things for you and three, four minutes or 10 minutes if you’re on certain clouds, you can have Kubernetes up and running and you don’t even have to think about a lot of these networking concerns to get started. I think that’s a little bit of the thud that we hear, "It’s hard to install." Well, don’t install it, you install it whenever you have to manage your own data centers. Guess what? When you have to manage your own data centers and you’re managing networking and storage, there’s a set of expertise that you already have on staff and maybe they don’t want to learn a new thing, that’s a personal problem, that’s not really a Kubernetes problem. Let’s separate those concerns and not use our lack or not wanting to, to stop us from actually moving forward. [0:12:39.2] DC: Yeah. Maybe even taking that example step forward. I think where this problem compounds or this perspective sometimes compounds about Kubernetes being hard to operate is coming from of some shops who have the perspective of are operational concerns today, aren’t that complex. Why are we introducing this overhead, this thing that we maybe don’t need and you know, to your point Brian, I wonder if we’d all entertain the idea, I’m sure we would that maybe even, speaking to the cloud vendors, maybe even just a Heroku or something. Something that doesn’t even concern itself with Kube but can get your workload up and running and successful as quickly as possible. Especially if you’re like, maybe a small startup type persona, even that’s adequate, right? It could have been not a failure of Kubernetes but more so choosing the wrong tool for the job, does that resonate with you all as well, does that make sense? [0:13:32.9 DC: Yeah, you know, you can’t build a house with a screwdriver. I mean, you probably could, it would hurt and it would take a long time. That’s what we’re running into. What you’re really feeling is that operationally, you cannot bridge the gap between running your application and running your application in Kubernetes and I think that’s fair, that’s actually a great thing, we prove that the foundations are stable enough that now, we can actually do research and figure out the best ways to run things because guess what? RPM’s from Red Hat and then you have devs from the Debian project, different ways of getting things, you have Snap from Canonical, it works and sometimes it doesn’t, we need to actually figure out those constructs in Kubernetes, they’re not free. These things did not exist because someone says, "Hey, I think we should do this." Many years. I was using RPM in the 90s and we need to remember that. [0:14:25.8] JR: On that front, I want to maybe point a question to you Duffy, if you don’t mind. Another big concern that I know you deal with a lot is that Kubernetes is great. Maybe I can get it up no problem. But to make it a viable deployment target at my organization, there’s a lot of work that goes into it to make a Kubernetes cluster production ready, right? That could be involving how you integrate storage and networking and security and on and on. I feel like we end up at this tradeoff of it’s so great that Kubernetes is super extensible and customizable but there is a certain amount of work that that kind of comes with, right? I’m curious Duff, what’s your perspective on that? [0:15:07.3] DC: I want to make a point that bring back to something Brian mentioned earlier, real quick, before I go on to that one. The point is that, I completely agree that yo do not have to actually be a distributed systems person to understand how to use Kubernetes and if that were a bar, we would have set that bar and incredibly, the inappropriate place. But from the operational perspective, that’s what we were referring to. I completely also agree that especially when we think about productionalizing clusters, if you’re just getting into this Kubernetes thing, it may be that you want to actually farm that out to another entity to create and productionalize those clusters, right? You have a choice to make just like you had a choice to make what when AWS came along. Just like you had a choice to make — we’re thinking of virtual machines, right? You have a choice and you continue to have a choice about how far down that rabbit hole as an engineering team of an engineering effort your company wants to go, right? Do you want farm everything out to the cloud and not have to deal with the operations, the day to day operations of those virtual machines and take the constraints that have been defined by that platformer, or do you want to operate that stuff locally, are you required by the law to operate locally? What does production really mean to you and like, what are the constraints that you actually have to satisfy, right? I think that given that choice, when we think about how to production Alize Kubernetes, it comes down to exactly that same set of things, right? Frequently, productionalizing – I’ve seen a number of different takes on this and it’s interesting because I think it’s actually going to move on to our next topic in line here. Frequently I see that productionizing or productionalizing Kubernetes means to provide some set of constraints around the consumption of the platform such that your developers or the focus that are consuming that platform have to operate within those rails, right? They could only define deployments and they can only define deployments that look like this. We’re going to ask them a varied subset of questions and then fill out all the rest of it for them on top of Kubernetes. The entry point might be CICD, it might be a repository, it might be code repository, very similar to a Heroku, right? The entry point could be anywhere along that thing and I’ve seen a number of different enterprises explore different ways to implement that. [0:17:17.8] JR: Cool. Another concept that I wanted to maybe have us define and think about, because I’ve heard the term platform quite a bit, right? I was thinking a little bit about you know, what the term platform means exactly? Then eventually, whether Kubernetes itself should be considered a platform. Backing u, maybe we could just start with a simple question, for all of us, what makes something a platform exactly? [0:17:46.8] BL: Well, a platform is something that provides something. That is a Brian Lyles exclusive. But really, what it is, what is a platform, a platform provides some kind of service that can be used to accomplish some task and Kubernetes is a platform and that thing, it provides constructs through its API to allow you to perform tasks. But, Kubernetes is not just a platform. Kubernetes is a platform for building platforms. The things that Kubernetes provides, the workload API, the networking API, the configuration and storage API’s. What they provide is a facility for you to build higher level constructs that control how you want to run the code and then how you want to connect the applications. Yeah, Kubernetes is actually a platform for platforms. [0:18:42.4] CC: Wait, just to make sure, Brian. You’re saying, because Kelsey Hightower for example is someone who says Kubernetes is a platform of platforms. Now, is Kubernetes both a platform of platforms, at the same time that it’s also a platform to run apps on? [0:18:59.4] BL: It’s both. Kelsey tweeted that there is some controversy on who said that first, it could have been Joe Beda, it could have been Kelsey. I think it was one of those two so I want to give a shout out to both of those for thinking in the same line and really thinking about this problem. But to go back to what you said, Carlisia, is it a platform for providing platforms and a platform? Yes, I will explain how. If you have Kubernetes running and what you can do is you can actually talk to the API, create a deployment. That is platform for running a workload. But, also what you can do is you can create through Kubernetes API mechanisms, ie. CRD’s, custom resource definitions. You can create custom resources that I want to have something called an application. You can basically extend the Kubernetes API. Not only is Kubernetes allowing you to run your workloads, it’s allowing you to specify, extend the API, which then in turn can be run with another controller that’s running on your platform that then gives you this thing when you cleared an application. Now, it creates deployment which creates a replica set, which creates a pod, which creates containers, which downloads images from a container registry. It actually is both. [0:20:17.8] DC: Yeah, I agree with that. Another quote that I remember being fascinated by which I think kind of also helps define what a platform is Kelsey put on out quote that said, Everybody wants platform at a service with the only requirement being that they’ve built it themselves." Which I think is awesome and it also kind of speaks, in my opinion to what I think the definition of a platform is, right? It’s an interface through which we can define services or applications and that interface typically will have some set of constraints or some set of workflows or some defined user experience on top of it. To Brian's point, I think that Kubernetes is a platform because it provides you a bunch of primitive s on the back end that you can use to express what that user experience might be. As we were talking earlier about what does it take to actually – you might move the entry point into this platform from the API, the Kubernetes API server, back down into CICD, right? Perhaps you're not actually defining us and called it a deployment, you’re just saying, I want so many instances off this, I don’t want it to be able to communicate with this other thing, right? It becomes – so my opinion, the definition about of a platform it is that user experience interface. It’s the constraints that we know things that you're going to put on top of that platform. [0:21:33.9] BL: I like that. I want to throw out a disclaimer right here because we’re here, because we’re talking about platforms. Kubernetes is not a platform, it’s as surface. That is actually, that’s different, a platform as a service is – from the way that we look at it, is basically a platform that can run your code, can actually make your code available to external users, can scale it up, can scale it down and manages all the nuances required for that operation to happen. Kubernetes does not do that out of the box but you can build a platform as a surface on Kubernetes. That’s actually, I think, where we’ll be going next is actually people, stepping out of the onesy-twosy, I can deploy a workload, but let’s actually work on thinking about this level. And I’ll tell you what. DEUS who got bought by Azure a few years ago, they actually did that, they built a pass that looks like Heroku. Microsoft and Azure thought that was a good idea so they purchased them and they’re still over there, thinking about great ideas but I think as we move forward, we will definitely see different types of paths on Kubernetes. The best thing is that I don’t think we’ll see them in the conventional sense of what we think now. We have a Heroku, which is like the git-push Heroku master, we share code through git. And then we have CloudFoundry idea of a paths which is, you can run CFPush and that actually is more of an extension of our old school Java applications, where we could just push [inaudible] here but I think at least I am hoping and this is something that I am actually working on not to toot my own horn too much but actually thinking about how do we actually – can we build a platform as a service toolkit? Can I actually just build something that’s tailing to my operation? And that is something that I think we’ll see a lot more in the next 18 months. At least you will see it from me and people that I am influencing. [0:23:24.4] CC: One thing I wanted to mention before we move onto anything else, in answering “Is Kubernetes right for me?” We are so biased. We need to play devil’s advocate at some point. But in answering that question that is the same as in when we need to answer, “Is technology x right for me?” and I think there is at a higher level there are two camps. One camp is very much of the thinking that, "I need to deliver value. I need to allow my software and if the tools I have are solving my problem I don’t need to use something else. I don’t need to use the fancy, shiny thing that’s the hype and the new thing." And that is so right. You definitely shouldn't be doing that. I am divided on this way of thinking because at the same time at that is so right. You do have to be conscious of how much money you’re spending on things and anyway, you have to be efficient with your resources. But at the same time, I think that a lot of people who don’t fully understand what Kubernetes really can do and if you are listening to this, if you maybe could rewind and listen to what Brian and Duffy were just saying in terms of workflows and the Kubernetes primitives. Because those things they are so powerful. They allow you to be so creative with what you can do, right? With your development process, with your roll out process and maybe you don’t need it now. Because you are not using those things but once you understand what it is, what it can do for your used case, you might start having ideas like, “Wow, that could actually make X, Y and Z better or I could create something else that could use these things and therefore add value to my enterprise and I didn’t even think about this before.” So you know two ways of looking at things. [0:25:40.0] BL: Actually, so the topic of this session was, “Should I Kubernetes” and my answer to that is I don’t know. That is something for you to figure out. If you have to ask somebody else I would probably say no. But on the other side, if you are looking for great networking across a lot of servers. If you are looking for service discovery, if you are looking for a system that can restart workloads when they fail, well now you should probably start thinking about Kubernetes. Because Kubernetes provides all of these things out of the box and are they easy to get started with though? Some of these things are harder. Service discovery is really easy but some of these things are a little bit harder but what Kubernetes does is here comes my hip-hop quote, Jay Z said this, basically he’s talking about difficult things and he basically wants difficult things to take a little bit of time and impossible things or things we thought that were impossible to take a week. So basically making difficult things easy and making things that you could not even imagine doing, attainable. And I think that is what Kubernetes brings to the table then I’ll go back and say this one more time. Should you use Kubernetes? I don’t know that is a personal problem that is something you need to answer but if you’re looking for what Kubernetes provides, yes definitely you should use it. [0:26:58.0] DC: Yeah, I agree with that I think it is a good summary there. But I also think you know coming back to whether you should Kubernetes part, from my perspective the reason that I Kubernetes, if you will, I love that as a verb is that when I look around at the different projects in the infrastructure space, as an operations person, one of the first things I look for is that API that pattern around consumption, what's actually out there and what’s developing that API. Is it a the business that is interested in selling me a new thing or is it an API that’s being developed by people who are actually trying to solve real problems, is there a reasonable way to go about this. I mean when I look at open stack, OpenStack was exactly the same sort of model, right? OpenStack existed as an API to help you consume infrastructure and I look at Kubernetes and I realize, “Wow, okay well now we are developing an API that allows us to think about the life cycle and management of applications." Which moves us up the stack, right? So for my part, the reason I am in this community, the reason I am interested in this product, the reason I am totally Kubernetes-ing is because of that. I realized that fundamentally infrastructure has to change to be able to support the kind of load that we are seeing. So whether you should Kubernetes, is the API valuable to you? Do you see the value in that or is there more value in continuing whatever paradigm you’re in currently, right? And judging that equally I think is important. [0:28:21.2] JR: Two schools of thoughts that I run into a lot on the API side of thing is whether overtime Kubernetes will become this implementation detail, where 99% of users aren’t even aware of the API to any extent. And then another one that kind of talks about the API is consistent abstraction with tons of flexibility and I think companies are going in both directions like OpenShift from Red Hat is perhaps a good example. Maybe that is one of those layer two platforms more so Brian that you were talking about, right? Where Kubernetes is the platform that was used to build it but the average person that interacts with it might not actually be aware of some of the Kubernetes primitives and things like that. So if we could all get out of our crystal balls for a second here, what do you all think in the future? Do you see the Kubernetes API becoming just a more prevalent industry standard or do you see it fading away in favor of some other abstraction that makes it easier? [0:29:18.3] BL: Oh wow, well I already see it as I don’t have to look too far in the future, right? I can see the Kubernetes API being used in ways that we could not imagine. The idea that I will think of is like KubeVirt. KubeVirt allows you to boot basically pods on whatever implements that it looks like a Kubelet. So it looks like something that could run pods. But the neat thing is that you can use something like KubeVirt with a virtual Kubelet and now you can boot them on other things. So ideas in that space, I don’t know VMware is actually going on that, “Wow, what if we can make virtual machines look like pods inside of Kubernetes? Pretty neat." Azure has definitely led work on this as now, we can just bring up either bring up containers, we can bring up VM’s and you don’t actually need a Kube server anymore. Now but the crazy part is that you can still use a workloads API’s, storage API’s with Kubernetes and it does not matter what backs it. And I’ll throw out one more suggestion. So there is also projects like AWS operators in [inaudible] point and what they allow you to do is to use the Kubernetes API or actually in cluster API, I'll use all three. But I use the Kubernetes API to boot things that aren’t even in the cluster and this will be AWS services or this could be databases across multiple clouds or guess what? More Kubernetes services. Yeah, so we are on that path but I just can’t wait to see what people are going to do with that. The power of Kubernetes is this API, it is just so amazing. [0:30:50.8] DC: For my part, I think is that I agree that the API itself is being extended in all kinds of amazing ways but I think that as I look around in the crystal ball, I think that the API will continue to be foundational to what is happening. If I look at the level two or level three platforms that are coming, I think those will continue to be a thing for enterprises because they will continue to innovate in that space and then they will continue to consume the underlying API structure and that portability Kubernetes exposes to define what that platform might look like for their own purpose, right? Giving them the ability to effectively have a platform as a service that they define themselves but using and under – you know, using a foundational layer that it’s like consistent and extensible and extensive I think that that’s where things are headed. [0:31:38.2] CC: And also more visual tools, I think is in our future. Better, actual visual UI's that people can use I think that’s definitely going to be in our future. [0:31:54.0] BL: So can I talk about that for a second? [0:31:55.9] CC: Please, Brian. [0:31:56.8] BL: I am wearing my octant hoodie today, which is a visual tool for Kubernetes and I will talk now as someone who has gone down this path to actually figure this problem out. As a prediction for the future, I think we’ll start creating better API’s in Kubernetes to allow for more visual things and the reason that I say that this is going to happen and it can’t really happen now is because for inside of an octant and whenever creating new eye views, pretty much happened now what that optic is. But what is going to happen and I see the rumblings from the community, I see the rumblings from K-native community as well is that we are going to start standardizing on conditions and using conditions as a way that we can actually say what’s going on. So let me back it up for a second so I can explain to people what conditions are. So Kubernetes, we think of Kubernetes as YAML and in a typical object in Kubernetes, you are going to have your type meta data. What is this, you are going to have your object meta data, what’s name this and then you are going to have a spec, how is this thing configured and then you are going to have a status and the status generally will say, “Well what is the status of this object? Is it deployment? How many references out? If it is a pod, am I ready to go?" But there is also this concept and status called conditions, which are a list of things that say how your thing, how your object is working. And right now, Kubernetes uses them in two ways, they use them in the negative way and the positive way. I think we are actually going to figure out which one we want to use and we are going to see more API’s just say conditions. And now from a UI developer, from my point of view, now I can just say, “I don’t really care what your optic is. You are going to give me conditions in a format that I know and I can just basically report on those in the status and I can tell you if the thing is working or not.” That is going to come too. And that will be neat because that means that we get basically, we can start building UI’s for free because we just have to learn the pattern. [0:33:52.2] CC: Can you talk a little bit more about conditions? Because this is not something I hear frequently and that I might know but then not know what you are talking about by this name. [0:34:01.1] BL: Oh yeah, I will give you the most popular one. So everything in Kubernetes is an object and that even means that the nodes that your workloads run on, are objects. If you run KubeControl, KubeCuddle, Kube whatever, git nodes, it will show you all the nodes in your cluster if you have permission to see that and if you do KubeCTL, gitnode, node name and then you actually have the YAML output what you will see in the bottom is an object called 'conditions'. And inside of there it will be something like is there sufficient memory, is the node – I actually don’t remember all of them but really what it is, they’re line items that say how this particular object is working. So do I have enough memory? Do I have enough storage? Am I out of actual pods that can be launched on me and what conditions are? It is basically saying, “Hey Brian, what is the weather outside?” I could say it's nice. Or I could be like, “Well, it’s 75 degrees, the wind is light but variable. It is not humid and these are what the conditions are.” They allow the object to specify things about itself that might be useful to someone who is consuming it. [0:35:11.1] CC: All right that was useful. I am actually trying to bring one up here. I never paid attention to that. [0:35:18.6] BL: Yeah and you will see it. So the two ones that are most common right now, there is some competition going on in Kubernetes architecture, trying to figure out how they are going to standardize on this but with pods and nodes you will see conditions on there and those are just telling you what is going on but the problem is that a condition is a type, a message, a status and something else but the problem is that the status can be true of false — oh and a reason, the status can be true or false but sometimes the type is a negative type where it would be like “node not ready”. And then it will say false because it is. And now whenever you’re inspecting that with automated code, you really want the positive condition to be true and the negative condition to be false and this is something that the K-native community is really working on now. They have the whole facility of this thing called duck typing. Which they can actually now pattern-match inside of optics to find all of these neat things. It is actually pretty intriguing. [0:36:19.5] CC: All right, it is interesting because I very much status is everything for objects and that is very much a part of my work flow. But I never noticed that there was some of the objects had conditions. I never noticed that and just a plug, we are very much going to have the K-native folks here to talk about duck typing. I am really excited about that. [0:36:39.9] BL: Yeah, they’re on my team. They’ll be happy to come. [0:36:42.2] CC: Oh yes, they are awesome. [0:36:44.5] JR: So I was thinking maybe we could wrap this conversation up and I think we have acknowledged that “Should I Kubernetes?” is a ridiculously hard question for us to answer for you and we should clearly not be the ones answering it for you but I was wondering if we could give some thoughts around — for the Podlet listener who is sitting at their desk right now thinking like, “Is now the right time for my organization to bring this in?” And I will start with some thought and then open it all up to you. So one common thing I think that I run into a lot is you know your current state and you know your desired state to steal a Kubernetes concept for a moment. And the desired state might be more decoupled services that are more scalable and so on and I think oftentimes at orgs we get a little bit too obsessed with the desired state that we forget about how far the gap is between the current state and the desired state. So as an example, you know maybe your shop’s biggest issue is the primary revenue generating application is a massive dot-net framework monolith, which isn’t exactly that easy to just port over into Kubernetes, right? So if a lot of your friction right now is teams collaborating on this tool, updating this tool, scaling this tool, maybe before even thinking about Kubernetes, being honest with the fact that a lot of value can be derived right now from some amount of application architecture changes. Or even sorry to use a buzzword but some amount of modernization of aspects of that application before you even get to the part of introducing Kubernetes. So that is one common one that I run into with orgs. What are some other kind of suggestion you have for people who are thinking about, “Is it the right time to introduce Kube?” [0:38:28.0] BL: So here is my thought, if you work for a small startup and you’re working on shipping value and you have no Kubernetes experience and staff and you don’t want to use for some reason you don’t want to use the cloud, you know go figure out your other problems then come back. But if you are an enterprise and especially if you work in a central enterprise group and you are thinking about “modernization”, I actually do suggest that you look at Kubernetes and here is the reason why. My guess is that if you’re a business of a certain size, you run VMware in your data center. I am just guessing that because I haven’t been to a company that doesn’t. Because we learned a long time ago that using virtual machines in many cases is way more efficient than just running hardware because what happens is we can’t use our compute capacity. So if you are working for a big company or even like a medium sized company, I don’t think – I am not telling you to run for it but I am telling you to at least have someone go look at it and investigate if this could ultimately be something that could make your stack easier to run. [0:39:31.7] DC: I think I am going to take the kind of the operations perspective. I think if you are in the business of coming up with a way to deploy applications on the servers and you are looking at trying to handle the lifecycle of that and you’re pretty fed up with the tooling that is out there and things like Puppet and Chef and tooling like that and you are looking to try and understand is there something in Kubernetes for me? Is there some model that could help me improve the way that I actually handle a lifecycle of those applications, be they databases or monoliths or compostable services? Any which way you want to look at it like are there tools there that can be expressed. Is the API expressive enough to help me solve some of those problems? In my opinion the answer is yes. I look at things like DaemonSet and the things like scheduling [inaudible] that are exposed by Kubernetes. And there is actually quite a lot of power there, quite a lot of capability in just the traditional model of how do I get this set of applications onto that set of servers or some subset they’re in. So I think it is worth evaluating if that is the place you’re in as an organization and if you are looking at fleets of equipment and trying to handle that magical recipe of multiple applications and dependencies and stuff. See what is the water is like on this side, it is not so bad. [0:40:43.1] CC: Yes, I don’t think there is a way to answer this question. It is Kubernetes for me without actually trying it, giving it a try yourself like really running something of maybe low risk. We can read blogposts to the end of the world but until you actually do it and explore the boundaries is what I would say, try to learn what else can you do that maybe you don’t even need but maybe might become useful once you know you can use. Yeah and another thing is maybe if you are a shop that has one or two apps and you don’t need full blown, everything that Kubernetes has to offer and there is a much more scaled down tool that will help you deploy and run your apps, that’s fine. But if you have more, a certain number, I don’t know what that number would be but multiple apps and multiple services just think about having that uniformity across everything. Because for example, I’ve worked in shops where the QA machines were taking care by a group of dev ops people and the production machines, oh my god they were taken care by other groups and now the different group of people and the two sides of these groups used were different and I as a developer, I had to know everything, you know? How to deploy here, how to deploy there and I had to have my little notes and recipes because whenever I did it – First of all I wasn’t doing that multiple times a day. I had to read through the notes to know what to do. I mean just imagine if it was one platform that I was deploying to with the CLI comments there, it is very easy to use like Kubernetes has, gives us with Kubes ETL. You know you have to think outside of the box. Think about these other operations that you have that people in your company are going to have to do. How is this going to be taught in the future? Having someone who knows your stack because your stack is the same that people in your industry are also using. I think about all of these things not just – I think people have to take it across the entire set of problems. [0:43:01.3] BL: I wanted to mention one more thing and this is we are producing lots of content here with The Podlets and with our coworkers. So I want to actually give a shout out to the TGIK. We want to know what you can do in Kubernetes and you want to have your imagination expanded a little bit. Every Friday we make a new video and actually funny enough, three fourths of the people on this call have actually done this. Where, on Friday, we pick a topic and we go in and it might be something that would be interesting to you or it might not and we are all over the place. We are not just doing applications but we are applications low level, mapping applications on Kubernetes, new things that just came out. We have been doing this for a 101 episodes now. Wow. So you can go look at that if you need some examples of what things you could do on Kubernetes. [0:43:51.4] CC: I am so glad to tgik.io maybe somebody, an English speaker should repeat that because of my accent but let me just say I am so glad you mentioned that Brian because I was sitting here as we are talking and thinking there should be a catalog of used cases of what Kubernetes can do not just like the rice and beans but a lot of different used cases, maybe things that are unique that people don’t think about to use because they haven’t run into that need yet. But they could use it as a pause, okay that would enable me to do these thing that I didn’t even think about. That is such a great catalog of used cases. It is probably the best resource. Somebody say the website again? Duffy what is it? [0:44:38.0] DC: tgik.io and it is every Friday at 1 PM Pacific. [0:44:43.2] CC: And it is live. It’s live and it’s recorded, so it is uploaded to the VMware Cloud Native YouTube and everything is going to be on the show notes too. [0:44:52.4] DC: It’s neat, you can come ask us questions there is a live chat inside of that and you can use that live chat. You can ask us questions. You can give us ideas, all kinds of crazy things just like you can with The Podlets. If you have an idea for an episode or something that you want us to cover or if you have something that you are interested in, you can go to thepodlets.io that will link you to our GitHub pages where you can actually open an issue about things you’d love to hear more about. [0:45:15.0] JR: Awesome and then maybe on that note, Podlets, is there anything else you all would like to add on “Should I Kubernetes?” or do you think we’ve – [0:45:22.3] BL: As best as our bias will allow it I would say. [0:45:27.5] JR: As best as we can. [0:45:27.9] CC: We could go another hour. [0:45:29.9] JR: It’s true. [0:45:30.8] CC: Maybe we’ll have “Should I Kubernetes?” Part 2. [0:45:34.9] JR: All right everyone, well that wraps it up for at least Part 1 of “Should I Kubernetes?” and we appreciate you listening. Thanks so much. Be sure to check out the show notes as Duffy mentioned for some of the articles we read preparing for this episode and TGIK links and all that good stuff. So again, I am Josh Russo signing out, with us also Carlisia Campos. [0:45:55.8] CC: Bye everybody, it was great to be here. [0:45:57.7] JR: Duffy Coolie. [0:45:58.5] DC: Thanks you all. [0:45:59.5] JR: And Brian Lyles. [0:46:00.6] BL: Until next time. [0:46:02.1] JR: Bye. [END OF EPISODE] [0:46:03.5] ANNOUNCER: Thank you for listening to The Podlets Cloud Native Podcast. Find us on Twitter at https://twitter.com/ThePodlets and on the http://thepodlets.io/ website, where you'll find transcripts and show notes. We'll be back next week. Stay tuned by subscribing. [END]See omnystudio.com/listener for privacy information.

If you work in Kubernetes, cloud native, or any other fast-moving ecosystem, you might have found that keeping up to date with new developments can be incredibly challenging. We think this as well, and so we decided to make today’s episode a tribute to that challenge, as well as a space for sharing the best resources and practices we can think of to help manage it. Of course, there are audiences in this space who require information at various levels of depth, and fortunately the resources to suit each one exist. We get into the many different places we go in order to receive information at each part of the spectrum, such as SIG meetings on YouTube, our favorite Twitter authorities, the KubeWeekly blog, and the most helpful books out there. Another big talking point is the idea of habits or practices that can be helpful in consuming all this information, whether it be waiting for the release notes of a new version, tapping into different TLDR summaries of a topic, streaming videos, or actively writing posts as a way of clarifying and integrating newly learned concepts. In the end, there is no easy way, and passionate as you may be about staying in tune, burnout is a real possibility. So whether you’re just scratching the cloud native surface or up to your eyeballs in base code, join us for today’s conversation because you’re bound to find some use in the resources we share. Follow us: https://twitter.com/thepodlets Website: https://thepodlets.io Feeback: info@thepodlets.io https://github.com/vmware-tanzu/thepodlets/issues Hosts: Carlisia Campos Josh Rosso Duffie Cooley Olive Power Michael Gasch Key Points From This Episode: Audiences and different levels of depth that our guests/hosts follow Kubernetes at. What ‘keeping up’ means: merely following news, or actually grasping every new concept? The impossibility of truly keeping up with Kubernetes as it becomes ever more complex. Patterns used to keep up with new developments: the TWKD website, release notes, etc. Twitter’s helpful provision of information, from opinions to tech content, all in one place. How helpful Cindy Sridharan is on Twitter in her orientation toward distributed systems. The active side of keeping up such as writing posts and helping newcomers. More helpful Twitter accounts such as InfoSec. How books provide one source of deep information as opposed to the noise on Twitter. Books: Programming Kubernetes; Managing Kubernetes; Kubernetes Best Practices. Another great resource for seeing Kubernetes in action: the KubeWeeky blog. A call to watch the SIG playlists on the Kubernetes YouTube channel. Tooling: tab management and Michael’s self-built Twitter searcher. Live streaming and CTF live code demonstrations as another resource. How to keep a team updated using platforms like Slack and Zoom. The importance of organizing shared content on Slack. Challenges around not knowing the most important thing to focus on. Cognitive divergence and the temptation of escaping the isolation of coding by socializing. The idea that not seeing keeping up to date as being a personal sacrifice is dangerous. Using multiple different TLDR summaries to cement a concept in one’s brain. Incentives for users rather than developers of projects to share their experiences. The importance of showing appreciation for free resources in keeping motivation up. Quotes: “An audience I haven’t mentioned is the audience that basically just throws up their hands and walks away because there’s just too much to keep track of, right?” — @mauilion [0:05:15] “Maybe it’s because I’m lazy, I don’t know? But I wait until 1.17 drops, then I go to the release notes and really kind of ingest it because I’ve just struggled so much to kind of keep up with the day to day, ‘We merged this, we didn’t merge this,’ and so on.” — @joshrosso [0:10:18] “If you find value in being up to date with these things, just figure out – there are so many resources out there that address these different audiences and figure out what the right measure for you is. You don’t have to go deep on the code on everything.” — @mauilion [0:27:57] “Actually putting the right content in the right channel, at least from a higher level, helps me decide whether I want to like look at that channel today, and stuff that should be in the channel is not kind of in a conversation channel.” — @opowero [0:32:21] “When I see something that is going to give me the fundamentals, like I have other priorities now, I sort of always want to consume that to learn the fundamentals, because I think in the long term phase of, but then I neglect physically what I need to know to do in the moment.” — @carlisia [0:33:39] “Just do nothing, because our brain needs that. We need to not be listening, not be reading, just nothing. Just sit and look at the ceiling. Our brain needs that. Ideally, look at nature, like look outside, look at the air, go for a walk. We need that, because that recharges the brain.” — @carlisia [0:42:38] “Just consuming and keeping up, that doesn’t necessarily mean you don’t give back.” — @embano1 [0:49:32] Links Mentioned in Today’s Episode: Chris Short — https://chrisshort.net/ Last Week in Kubernetes Development — http://lwkd.info/ 1.17 Release Notes — https://kubernetes.io/docs/setup/release/notes/ Release Notes Filter Page — https://relnotes.k8s.io/ Cindy Sridharan on Twitter — https://twitter.com/copyconstruct InfoSec on Twitter — https://twitter.com/infosec?lang=en Programming Kubernetes on Amazon —https://www.amazon.com/Programming-Kubernetes-Developing-Cloud-Native-Applications/dp/1492047104 Managing Kubernetes on Amazon — https://www.amazon.com/Managing-Kubernetes-Operating-Clusters-World/dp/149203391X Brendan Burns on Twitter — https://twitter.com/brendandburns Kubernetes Best Practices on Amazon — https://www.amazon.com/Kubernetes-Best-Practices-Blueprints-Applications-ebook/dp/B081J62KLW/ KubeWeekly — https://kubeweekly.io/ Kubernetes SIG playlists on YouTube — https://www.youtube.com/channel/UCZ2bu0qutTOM0tHYa_jkIwg/playlists Twitch — https://www.twitch.tv/ Honeycomb — https://www.honeycomb.io/ KubeKon EU 2019 — https://events19.linuxfoundation.org/events/kubecon-cloudnativecon-europe-2019/ Aaron Crickenberger on LinkedIn — https://www.linkedin.com/in/spiffxp/ Stephen Augustus on LinkedIn — https://www.linkedin.com/in/stephenaugustus Office Hours — https://github.com/kubernetes/community/blob/master/events/office-hours.md Transcript: EPISODE 17[INTRODUCTION][0:00:08.7] ANNOUNCER: Welcome to The Podlets Podcast, a weekly show that explores Cloud Native one buzzword at a time. Each week, experts in the field will discuss and contrast distributed systems concepts, practices, tradeoffs and lessons learned to help you on your cloud native journey. This space moves fast and we shouldn’t reinvent the wheel. If you’re an engineer, operator or technically minded decision maker, this podcast is for you.[EPISODE][0:00:41.5] DC: Good afternoon everybody and welcome to The Podlets. In this episode, we’re going to talk about, you know, one of the more challenging things that we all have to do, just kind of keep up with cloud native and how we each approach that and what we do. Today, I have a number of cohosts with me, I have Olive Power.[0:00:56.6] OP: Hi.[0:00:57.4] DC: Carlisia Campos.[0:00:58.6] CC: Hi everybody.[0:00:59.9] DC: Josh Rosso.[0:01:01.3] JR: Hey all.[0:01:02.8] DC: And Michael.[0:01:01.1] MICHAEL: Hey, hello.[0:01:04.8] DC: This episode, we’re going to do something a little different than we normally do. In most of our episodes, we try to remain somewhat objective around the problem and the potential solutions for it, rather than prescribing a particular solution. In this episode, however, since we’re talking about how we keep up with all of the crazy things that happen in such a fast ecosystem, we’re going to probably provide quite a number of examples or resources that you yourself could use to drive and to try and keep up to date with what’s happening out there.Be sure to check out the notes after the episode is over at thepodlets.io and you will find a link to the episodes up at the top part, click down to this episode, and check out the notes. There will be tons of resources. Let’s get started.One of the things I think about that’s interesting about keeping up with something like, you know, a Kubernetes or a fast-moving project, regardless of what that project is, whether it’s Kubernetes or, you know, for a while, it was the Mesos that I was following or OpenStack or a number have been big infrastructure projects that have been very fast moving over time and I think what’s interesting is I find that there’s multiple audiences that we kind of address when we think about what it means to ‘keep up,’ right?Keeping up with something like a project is interesting because I feel like there’s an audience that it’s actually very interested in what’s happening with the design goals or the code base of the project, and there’s an audience that is very specific to wanting to understand at a high level – like, “Give me the State of the World report like every month or so just so I can understand generally what’s happening with the project, like is it thriving? Is it starting to kind of wane? Are there big projects that it’s taking on?”And then there’s like, then I feel like there’s an audience somewhere in the middle there where they really want to see people using the project and understand, and know how to learn from those people who are using it so that they can elevate their own use of that project. They’re not particularly interested in the codebase per se but they do want to understand, are they exploring this project at a depth that makes sense for themselves? What do you all think about that?[0:03:02.0] CC: I think one thing that I want to mention is that this episode, it’s not so much about on-boarding people onto Kubernetes and the Kubernetes ecosystem. We are going to have an episode soon to talk specifically about that. How you get going, like get started. I think Duffy mentioned this so we’re going to be talking about how we all keep up with things. Definitely, there are different audiences, even when we’re talking about keeping up.[0:03:32.6] JR: Yeah, I think what’s funny about your audience descriptions, Duffy, is I feel like I’ve actually slid between those audiences a bit, right? It’s funny, back in the day, Kubernetes like one-four, one-five days, I feel like I was much more like, “What’s going on in the code?” Like trying to keep track of like how things are progressing.Now my role is a lot more focused with working with customers and standing up cube and like making a production ready. I feel like I’m a lot more, kind of reactive and more interested to see like, what features have become stable and impact me, you know what I mean? I’m far less in the weeds than I used to be. It’s a super interesting thing.[0:04:08.3] OP: Yeah, I tend to – for my role, I tend to definitely fall into the number three first which is the kind of general keeping an eye on things. Like when you see like interesting articles pop up that maybe have been linked internally because somebody said, “Oh, check out this article. It’s really interesting.”Then you find that you kind of click through five or six articles similar but then you can kind of flip to that kind of like, “Oh, I’m kind of learning lots of good stuff generally about things that folks are doing.” To actually kind of having to figure out some particular solution for one of my customers and so having to go quite deep into that particular feature.You kind of go – I kind of found myself going right in and then back out, right in, going back out depending on kind of where I am on a particular day of the week. It’s kind of a bit tricky. My brain sometimes doesn’t kind of deal with that sort of deep concentration into one particular topic and then back out again. It’s not easy.I find it quite tough actually some of the time.[0:05:05.0] DC: Yeah, I think we can all agree on that. Keeping track of everything is – it’s why the episode, right? How do we even approach it? It seems – I feel like, an audience I haven’t mentioned is the audience that basically just throws up their hands and walks away because there’s just too much to keep track of, right? I feel like we are all that at some point, you know?I get that.[0:05:26.4] OP: That’s why we have Christmas holidays, right? To kind of refresh the brain.[0:05:31.4] CC: Yeah, I maybe purposefully or maybe not even – not trying to keep up because it is too much, it is a lot, and what I’m trying to do is, go deeper on the things that I already, like sort of know. And things that I am working with on a day to day basis. I only really need to know, I feel like, I really only need to know – because I’m not working directly with customers.My scope is very well defined and I feel that I really only need to know whenever there’s a new Kubernetes release. I need to know what the release is. We usually – every once in a while, we update our project to the – we bump up the Kubernetes release that we are working against and in general, yeah, it’s like if things come my way, if it’s interesting, I’ll take a look, but mostly, I feel like I work in a spiral.If I’m doing codes related to controllers and there’s a conference talk about controllers then okay, let me take a look at this to maybe learn how to design this thing better, implement in a better way if I know more about it. If I’m doing, looking at CRDs, same thing. I really like conference talks for education but that’s not so much keeping up with what’s new. Are we talking about educating ourselves with things that we don’t know about?Things that we don’t know about. Or are we talking about just news?[0:07:15.6] JR: I think it’s everything. That’s a great question. One of my other questions when we were starting to talk about this was like, what is keeping up even mean, right? I mean, does it mean, where do you find resources that are interesting that keep you interested in the project or are you looking for resources that just kind of keep you up to date with what’s changing? It’s a great question.[0:07:36.2] MICHAEL: Actually, there was some problem that I faced when I edit the links that I wanted to share in the show. I started writing the links and then I realized, “Well, most of the stuff is not keeping up with news, it’s actually understanding the technology,” because I cannot keep up.What does help me in understanding specific areas, when I need to dig into them and I think back five or four years into early days of Kubernetes, it was easy to catch up by the time because it was just about Kubernetes. Later right, it became this platform. We realized that it actually this platform thing. Then we extended Kubernetes and then we realized there are CICD-related stuff and operations and monitoring and so the whole ecosystem grew. The landscape grew so much that today, it’s impossible to keep up, right?I think I’m interested in all those patterns that you have developed over the years that help you to manage this, let’s say complexity or stream of information.[0:08:33.9] DC: Yeah, I agree. This year, I was thinking about putting up a talk with Chris Short, it was actually last year. That was about kind of on the same topic of keeping up with it. In that, I kind of did a little research into how that happens and I feel like some of the interesting stuff that came out of that was that there are certain patterns that a project might take on that make it easier or more approachable to, you know, stay in contact with what’s happening.If we take Kubernetes as an example, there are a number of websites I think that pretty much everybody here kind of follows to some degree, that helps, sort of, kind of, address those different audiences that we were talking about.One of the ones that I’ve actually been really impressed with is LWKD which stands for Last Week in Kubernetes Development, and as you can imagine, this is really kind of focused on, kind of – I wouldn’t say it’s like super deep on the development but it is watching for things that are changing, that are interesting to the people who are curating that particular blog post, right?They’ll have things in there like, you know, code freezes coming up on this date, IPV6, IPV4, duel stack is merging, they’ll have like some of the big mile markers that are happening in a particular release and where they are in time as it relates to that release. I think if that’s a great pattern and I think that – it’s a very narrow audience, right? It would really only be interesting to people who are interested in, or who are caught up in the code base, or just trying to understand like, maybe I want a preview of what the release notes might look like, so I might just like look for like a weekly kind of thing.[0:10:03.4] JR: Yeah, speaking of the release notes, right? It’s funny. I do get to look at Last Week in Kubernetes development every now and then. It’s an awesome resource but I’ve gotten to the point where the release notes are probably my most important thing for staying up to date.Maybe it’s because I’m lazy, I don’t know, but I wait till 1.17 drops, then I go to the release notes and really kind of ingest it because I’ve just struggled so much to kind of keep up with the day to day, “We merged this, we didn’t merge this,” and so on. That has been a huge help for me, you know, day to day, week to week, month to month.[0:10:37.0] MICHAEL: Well, what was also helpful just on the release notes that the new filter webpage that they put out in 1.15, starting 1.15. Have you all seen that?[0:10:44.4] JR: I’ve never heard of it.[0:10:45.4] DC: Rel dot, whatever it is. Rel dot –[0:10:47.7] MICHAEL: Yeah, if you can share it Duffy, that’s super useful. Especially like if you want to compare releases and features added and –[0:10:55.2] DC: I’ll have to dig it up as well. I don’t remember exactly what –[0:10:56.7] CC: I’m sorry, say? Which one is that again?[0:10:59.1] MICHAEL: The real notes. I’ll put it in the hackMD.[0:11:02.8] DC: Yeah relnotes.k8s.io which is an interesting one because it’s sort of like a comparison engine that allows you to kind of compare what it would have featured like how to feature relates to different versions of stuff.[0:11:14.4] CC: That’s great. I cannot encourage enough for the listeners to look at the show notes because we have a little document here that we – can I? The resources are amazing. There are so many things that I have never even heard about and sound great – is – I want to go to this whole entire list. Definitely check it out. We might not have time to mention every single thing. I don’t want people to miss on all the goodness that’s been put together.[0:11:48.7] DC: Agreed, and again, if you’re looking for those notes, you just go to the podlets.io. Click on ‘episodes’ at the right? And then look for this episode and you’ll find that it’s there.[0:11:58.0] CC: I can see that a lot of the content in those notes are like Twitter feeds. Speaking personally, I’m not sure I’m at the stage yet where I learn a lot about Twitter feeds in terms of technical content. Do you guys find that it’s more around people’s thoughts around certain things so thought-provoking things around Kubernetes and the ecosystem rather than actual technical content. I mean, that’s my experience so far.But looking at those Twitter feeds, maybe I guess I might need to follow some of those feeds. What do you all think?[0:12:30.0] MICHAEL: Do you mean the tweets are from those like learn [inaudible 0:12:32] or the person to be tweets?[0:12:35.3] OP: You’ve listed some of there, Michael, and some sort of.[0:12:37.6] MICHAEL: I just wanted to get some clarity. The reason I listed so many Twitter accounts there is because Twitter is my only kind of newsfeed if you will. I used Feedly and RSS and others before and emails and threads. But then I just got overwhelmed and I had this feeling of missing out on all of those times.That’s why I said, “Okay, let’s just use Twitter.” To your question, most of these accounts are people who have been in the Kubernetes space for very long, either running Kubernetes, developing on Kubernetes, having opinions about Kubernetes.Opinions in general on topics related to cloud native because we didn’t want to make the search just about Kubernetes. Most of these people, I really appreciate their thoughts and some of them also just a retweet things that they see which I missed somewhere else and not necessarily just opinions. I think It’s a good mix of these accounts, providing options, some guidance, and also just news that I miss out on because not being on the other channels.[0:13:35.6] OP: Yeah, I agree because sometimes you can kind of read – I tend to require a lot of sort of blog posts and sort of web posts which, you know, without realizing it can be kind of opinionated and then, you know, it’s nice to then see some Twitter feeds that kind of actually just kind of give like a couple of words, a kind of a different view which sometimes makes me think “Okay, I understand that topic from a certain article that I’ve read, it’s just really nice to hear a kind of a different take on it through Twitter.”[0:14:03.0] CC: I think some of the accounts, like fewer of the accounts – and there are a bunch of things that – there are listed accounts here that I didn’t know before so I’ll check them out. I think fewer of the accounts are providing technical content, for example, Cindy Sridharan, not pronouncing it correctly but Cindy is great, she puts out a lot of technical content and a lot of technical opinion and observations that is really good to consume. I wish I had time to just read her blog posts and Twitter alone.She’s very oriented towards distributed systems in general, so she’s not even specific just Kubernetes. Most of the accounts are very opinionated and the benefit for me is that sometimes I catch people talking about something that I didn’t even know was a thing. It’s like, “Oh, this is a thing I should know about for the work that I do,” and like Michael was saying, you know, sometimes I catch retweets that I didn’t catch before and I just – I’m not checking out places, I’m not checking – hash tagging Reddit.I rely on Twitter and the people who I follow to – if there is a blog post that sounds important, I just trust that somebody would, that I’m going to see it multiple times until like, “Okay, this is content that is related to something and I’m working on, that I want to get better at.” Then I’ll go and look at it. My sources are mainly Twitter and YouTube and it’s funny because I love blog posts but it’s like I haven’t been reading them because it takes a long time to read a blogpost.I give preference to video because I can just listen while I’m doing stuff. I sort of stopped reading blog post which is sad. I also want to start writing posts because it’s so helpful for me to engrain the things that I’m learning and hopefully it will be helpful to other people too. But in any case, go Duffy.[0:16:02.8] DC: A number of people that I follow – I have been cultivating my feed pretty carefully, trying to get a broad perspective of technical stuff that’s happening. But also I’ve been trying to develop my persona on Twitter a bit more, right? I’m actually trying to build my audience there. What’s interesting there is I’ve been trying to – to that end, what I’ve been doing is like trying to amplify voices that I think aren’t heard enough out there, right?If I see an article by somebody who is just coming into Kubernetes. or just coming into distributed systems and they’ve taken an effort to really lay out something that they found really interesting about pretty much anything, right? I’m like, “Okay, that’s pretty awesome,” and I’ll try to amplify that, right? Sometimes I even get involved or I’ll, not directly in public on Twitter but I’ll offer to help edit or help provide whatever our guidance I can provide around that sort of stuff.If I see people like having a difficult time with a particular project or something like that, I’ll reach out privately and say, “Hey, can I help you with it so you can go out there and do a great job,” you know? That is something I love to do. I think your point about like not necessarily going at Twitter for the deep knowledge stuff but more just like making sure that you have a broad enough awareness of what’s happening in different ecosystems that you’re not surprised by the things when the things change, right?A couple of other people that I follow are Akira Asuta, I can’t say enough about that person. They are amazing, they have been doing like, incredibly deep security stuff as it relates to containerization and stuff like that for quite a while. I’m always like, learning brand new things to me when following folks like that. I’ve been kind of getting more interested in InfoSec Twitter lately, learning how people kind of approach that problem.Also some of the bias arounds that which has been pretty interesting. Both the bias against people who are in InfoSec which seems weird to me. Also, how InfoSec approaches a problem, like do they put it like a learning experience or they approach it like an attack experience.It’s been kind of fascinating to get in there.[0:18:08.1] OP: You know, I kind of use Twitter as well for some of this stuff but you know, books are kind of a resource as well but in my head, kind of like at the opposite scale. You know, I obviously don’t read as many books as I read twitter feeds, right? It’s just kind of like, with Twitter, you can kind of digest the whole of the stuff and with books, it’s kind of like – I tend to be trying – because I know, I’m only going to read – like I’m only going to read maybe one/two books a year.I’ve kind of like – as I said before, blog posts seem to take up my reading time and books kind of tend to be for like on airplanes and stuff. So if – they’re just kind of two opposite resources for me but I find actually, the content of books are probably stuff that I digest a bit more because you know, it’s kind of like, I don’t know, back to the old days. It’s kind of a physical thing on hand and I can kind of read it and digest it a bit more than the kind of throwaway stuff that kind of keeps on Twitter.Because to be honest, I don’t know what’s on Twitter. Who is kind of a person to listen to or who is not or who is – I just try and form my own opinions and then, again, it kind of gets a bit overwhelming, because it’s a lot of content just streaming through continuously, whereas a book, it’s kind of like just one source of information that is kind of like a bit more personal that I can digest a bit more.[0:19:18.1] JR: Any particular book recommendation in 2019, Olive, that you found particularly interesting?[0:19:23.5] OP: I’m still reading, and it’s on the list for the episode notes actually, Programming Kubernetes. I just want to kind of get into that sort of CRD sort of mindset a bit. I think that’s kind of an area that’s interesting and an area that a lot of people will want to use in their organizations, right, because it’s going to do some of the extensibility to Kubernetes that’s just not there out of the box and everybody wants something that’s not out of the box or always in my experience.[0:19:47.4] MICHAEL: I found the Managing Kubernetes, I think was it, by – from Brendan Burns and some other folks which was just released I think in the end of last year. Super deep and that is kind of the opposite to the Programming Kubernetes, because I like that as well. That is more geared towards understanding architecture and operations.Operational concepts –[0:20:05.0] OP: They’re probably the two books I’ve read.[0:20:08.4] MICHAEL: Okay.[0:20:08.9] OP: One a year, remember?[0:20:11.4] MICHAEL: Yeah.[0:20:14.6] OP: Prolific reading.[0:20:19.6] CC: I think if you know what you need to learn about cloud native or Kubernetes, there’s amazing books out there, and if you are still exploring Kubernetes and trying to learn, I cannot recommend this book enough. If you are watching this on YouTube, you’ll see the cover. It’s called Kubernetes Best Practices because it’s about Kubernetes best practices but what they did simultaneously and maybe they didn’t even realize is just they gave a map for the entire thing.You go, “Oh, these are all the elements in Kubernetes.” Of course, it’s saying, “Okay, this is the best way to go about setting the stuff up,” and this is relatively thin but I just think that going through this book, you get really fast overview of the elements in Kubernetes. Then you can go to other books like Managing Kubernetes to go deep and understand all of the knobs and switches.[0:21:24.6] DC: I want to bring it back to the patterns that we see successful projects. Projects that you think are approachable but, you know, projects that are out there that make it easy for you to kind of stay – or easier at least to stay up to date with them, what some of those patterns are that you think are useful for projects.We’re talking about like having a couple of different entry points from kind of a weekly report mechanism, we’ve talked about the one that LWKD is, I don’t think we got to talk about KubeWeekly which is actually a weekly blog that is actually curated by a lot of the CNCF ambassadors. KubeWeekly is also broken up in different sections, so like sometimes they’ll just talk about – but they’re actually going out actively and trying to find articles of people using Kubernetes and then trying to post those.If you’re interested in understanding how people are actually out there using it, then that’s a great place to go find articles that are kind of related to that. What are some other patterns that we see that are out there that are useful for books?[0:22:27.6] DC: One that I really like. Kubernetes, for everyone listening has this notion of special interest groups, SIGs oftentimes. They’re focused on certain areas of the project. There’s some for networking and storage and life cycles of clusters and what’s amazing, I try to watch them somewhat weekly, I don’t always succeed.They’re all on YouTube and if you go to the Kubernetes project YouTube, there’s playlists for every SIG. A lot of times I’m doing work relating to life cycles of clusters. I’ll open up the cluster life cycle playlist and I’ll just watch the weekly meetings. While it doesn’t always pertain to completely to me, it lets me understand kind of where the developers and contributor’s heads are at and where they’re kind of headed with a lot of different things.There’s a link to that as well if anyone wants to check it out.[0:23:15.9] MICHAEL: Exactly, to add to that. If you don’t have the time to watch the videos, the meeting notes that these gentlemen and women put together are amazing. Usually, I just scroll through and if it’s something to triggers, I go into the episode and watch it.[0:23:28.7] OP: I almost feel like we should talk about tooling to handle all of this stuff, for example, right now, I think I have 200 tabs opened. I just started learning about some chrome extensions to manage tabs. I haven’t started really using them but I need. I don’t have a good system. My system is open a video that I’m pretty sure I want to watch and just get to that tab eventually until something happens in my chrome goes bust and I lose everything.I wanted to mention that when we say watch YouTube, some things you don’t need to sit there and actually watch, you can just listen to it and if you pay for the five bucks for YouTube premium – I don’t get a commission you people, but I’m just saying, for me, it’s so helpful. I can just turn off you know, put my phone on my pocket and keep listening to it without having to have the phone open and on the whole time. It’s very handy.It’s just like listening to a podcast. I also listen to podcasts lots of days.[0:24:35.1] MICHAEL: For tooling, since I’m just mostly on Twitter and by the time I was using or starting to use Twitter, they didn’t have this bookmark function, so I was basically abusing likes or favorites at the time, I think, to bookmark. What I realized later, my bookmarks grew, well, my likes grew.I wanted to go back and find something but that through the Twitter search was just impossible. I blew the tiny little go tool, kind of my first exercise there to just parse my likes and then use JQ because it’s all JSON to query and manipulate the stuff. I almost use it every day because I was like, that was a talk or blog post about scheduling and just correct for scheduling and the likes.I’m sure there’s a better tool or way of doing that but for me, that’s mine too. Because that’s my workflow.[0:25:27.6] DC: Both of the two blogs that you mentioned both KubeWeekly and LWKD, they both have the ability to take – you can submit stories to them. If you come across things that are interesting and you’d like to put that up on an aggregator somewhere, this is one of the ways to kind of solve that problem because at least if it gets cleared up on an aggregator, you know that you go back to the aggregator to see it, so that helps.Some other ones I’ve seen out there, I’ve seen people, I’ve seen a number of interesting startups now, starting to kind of like put out a podcast or – and I have started to see a number of people like you know, engaging with Twitch and also doing things like what we do with TJK.io which is like have sort of some kind of a weekly thing where you are just hacking on stuff live and just exploring it whether that is related to – if you think of about TJK is we’re going to do without being related necessarily to anything that we are doing at VMware just anything to do with the community but obviously if you are working for one of the small companies like Honeycomb or some other company.A smaller kind of startup, you can really just get people more aware of that because for some reason people love to watch others code. They love to understand how people go through that, what are their thought process is and I find it awesome as well. I think it is amazing to me how big a draw that is, you know?[0:26:41.1] OP: And is there lots of them out there Duffy? Is that kind of an easy searchable thing or is it like how do you know those things are going on?[0:26:48.4] DC: Oddly enough Twitter, most of the time, yeah. I mean, most of the time I see that kind of stuff happening on Twitter, like somebody will like – I will scope with this or a number of other people will say, “Hey, I am going to do a live stream during this period of time on this,” and I have actually seen a number of people doing live streams on CTFs, which are capture the flags. That one’s really been fascinating to me because it has been how do people think about approaching the security of an application.Like where do they look for weak spots and how do you determine, how do you approach that kind of a problem, which is fascinating. So yeah, I think it is important to remember that like you know, you are not the only one trying to keep up to date with all of this stuff, right? The one thing we all have said pretty consistently here is that it is a lot, and it is not just Kubernetes, right? Like any fast moving project. It could be your favorite Ruby module that has 200 contributors, right?It doesn’t matter what it is, it is a lot to keep a track of, and it represents some of that cognitive overheads that you have to think about. That is a lot to take on. Even if it is overwhelming, if you find value in being up to date with these things, just figure out – there are so many resources out there that address these different audiences and figure out what the right measure for you is. You don’t have to go deep on the code on everything.Sometimes it might be better to just try and find a source of information that gives you a high enough of a view. Maybe you are looking at the blog posts that come out on Kubernetes.io every release and you are just looking at the release notes and if you just read the release notes every release, that is already miles ahead of what I have seen a lot of folks out there when they are starting to ask me questions about how do you keep up to date.[0:28:35.9] JR: I’m curious, we have been talking a lot about keeping up as an individual. Do you all have strategies for how you help, let’s say your overall team, keep up with all the things that are going on? To give an example, Duffy, Olive and myself, at least at one point, were on the same team and we’d go out to disparate customers and see all of these different new things that they are trying to do or new projects that they are using.So we’d have to think about how do we get together and share that internally to make sure we are bringing the whole team along with what is going on in the ecosystem especially from a customer perspective. I know one of the ways that we do that is having demos and things of that nature that we share weekly. Are there other strategies that you all use with your teams to kind of share interesting information and news?[0:29:25.5] M: So what we do is mostly the way we share in our team, and we are a small team. We use Slack. We pre-filter in terms of like if there is stuff that I think is valuable for me and probably not for the whole team – obviously we are not going to share, but I think if it is related to something that the team has or to come grant and then I will share on Slack but we don’t have any formal way. I know people use some reports, weekly reports, or other platforms to distribute but we just use Slack.[0:29:53.0] DC: I think one of the things – one of the patters that we had at [inaudible 0:29:54] that I thought was actually super helpful was that we would engage a conversation. “I learned a cool new thing about whatever today,” and so we would say, “I am going to – ” and then we would start a Zoom call around that and then people could join if they wanted to, to be a part of the live discussion or not, and if they didn’t, they would still be able to see a recorded Zoom pop up in the channel later on.So even if your time zones don’t line up, like I know it is 2 AM or 3 AM or something like that for Olive right now, you can still go back to those recorded sessions and you’ll just see it on your daily Slack stuff. You would be able to see, “Oh there was a conversation about whether you should deploy Kubernetes crossed availibility zones or not. I would like to go see that,” and see what the inputs were, and so that can be helpful.[0:30:42.5] JR: Yeah, that is a super interesting observation. It is almost like remote-first teams that are used to these processes of recording everything and putting it in a Google doc. They are more equipped for that information sharing perhaps than like the water cooler conversations you’d have in the office.[0:30:58.5] OP: And on the Slack or any of the communication tool, we have different channels because we are all in lots of channels and to have channels dedicated to a particular subject is absolutely the way to go because otherwise in my previous company that seem to be kind of one main channel that all the architect used to discussed everything on and you know sometimes you join and you’re like, “What is everybody talking about?”There would be literally about a hundred messages on some sort of theme that I have never heard of. So you come away from that thinking that, “That is the main channel. Where is the bit – is there messages in the middle that I missed that were just normal discussions as opposed to in around the technical stuff,” and so it made me a bit sad, right? I would be like, “I haven’t understood something and there is a whole load of stuff on this channel that I don’t understand.”But it is the kind of central channel for everyone. So I think you end up then start looking up things that they are discussing and then realizing actually that is not really anything related to what I need to know about today or next week. It might be something for the future but I’ve got other stuff to focus on. So my point is that those communication channels for me sometimes can make me feel a little bit behind the curve or very much sort of reactive in trying to jump on things that are actually not really anything to do with me for me now and wasting my time slightly and kind of messing with my head a little bit in that like, “I really need to try and focus out stuff,” and actually putting the right content in the right channel, at least from a higher level, helps me decide whether I want to like look at that channel today, and stuff that should be in the channel is not kind of in a conversation channel. So organization of where that content is, is important to me.[0:32:37.6] CC: I am so in the same page with you Olive. That is the way my brain works as well. I want to have multiple channels, like if we are talking about Slack or any chat tool, but some people have such aversion to multiple channels. They really have a hard time dealing with too many – like testing their threshold of what they think is too many channels. So I am always mindful too, like it has to work for everybody but if it was up to me, there will be one channel per topic. So I know where to focus on.But you said something that is so interesting. How do we even just – like you were saying in the context of channel, multiple channels, and I go, if I need to pay attention to this this week as oppose to like, I don’t need to look at this until some time in the future. How do we even decide what we focus on that is useful for us in the moment versus it would be good for me to know but I don’t need to know right now.I am super bad at this. When I see something that is going to give me the fundamentals, like I have other priorities now, I sort of always want to consume that to learn the fundamentals because I think in the long term phase of, but then I neglect physically what I need to know to do in the moment and I am trying to sort of fish there and get focused on in the moment things. Anybody else have a hard time?[0:34:04.5] DC: You are not alone on that, yeah.[0:34:06.7] CC: It is terrible.[0:34:08.3] MICHAEL: Something that I wish I would do more often as like being a good citizen is like when you read a lot, probably 90% of my time is not writing but reading, maybe even more and then I share and then on Twitter, the tweet for them the most successful ones in terms of retweets or likes are the ones where I do like TLDR’s or some screen captures like too long to read. Where people don’t have the time, they might want to read the article but they don’t have the time.But if you put in like a TLDR like either a tweet or a thread on it, a lot of people would jump onto it because they can just easily capture it and they can still read the full article if they want but that is something that I learned and it is pretty – what is the right word? Helpful to my followers and the community but I just don’t do it that often unfortunately. If I am writing, summarizing, writing, I kind of remember. That is how the brain works. It is a nice side effect.[0:35:04.9] DC: I was saying, this is definitely one of those things where you can be the change you want to see if you, you know?[0:35:08.6] M: Yeah, I know.[0:35:10.0] DC: This is awesome. I would also say that what you just raised Carlisia is like a super valid point. I mean like not everybody’s brain works the same way, right? There are people who are neuro-divergent. There are people who think very linearly and they are very comfortable with that and there are people who don’t. So it is a struggle I think regardless of how your brain is wired to understand to how to prioritize the attention you will give any given subject.In some cases, your brain is not wired – your brain is almost wired against that whole idea, like you are just not set up for success when it comes to figuring out how to prioritize your attention.[0:35:49.0] CC: You hit the nail on the head. We are so set up for failure in that department because there are so many interesting conversations and you want to hop in and you want to be a part of the conversation and part of the group and socialize. Our work is so isolating to really put our heads down and just work, it can be so isolating. So it is great to participate in conversations out there even if it is for only via Twitter. I mean, obviously we are very biased towards Twitter here in this group.But I am not even this on Twitter so just keep that in mind that we are cognizant of that but in any case, I don’t know what the answer is but what I am trying always to cut down on that, those social activities that seem so appealing. I don’t know how to do that from working out.[0:36:43.9] JR: I am in the same boat. 2020, I am hoping to let more of that go and to your point, it is not that there is no value in it. It is just, I don’t know, I am not deriving the same amount of quality out of it because I am so just multiplexed all over the place, right? So we’ll see how it goes.[0:36:59.9] CC: Oh if any listener has opinions and obviously it seems that all of us are helpless in that department. Share with us, please.[0:37:12.5] DC: It is a tricky one. I think it is also interesting because I find that when we talk about things like work-life balance, we think of the idea of maybe work-life balance is that when you come at the end of the day and you go home and you don’t think about work, right? Sometimes we think that work-life balance means that you have a certain amount of time off that you can actually spend with your family and your friends or your community, what have you, and not be engaging on multiple fronts.Just be that – have that be your focus, but when it comes to things like keeping up, when it comes to things like learning or elevating your education and stuff, it seems like, for the most part, and this is just my own assumption, I am curious how you all feel about this, that we don’t – that that doesn’t enter into it, right? Your personal time is totally on the table when it comes to how do you keep up with these things. We don’t even think about it that way, right?I know I personally don’t. I definitely have to do more and cut back on the amount of time that I spend reading. I am right there with Michael on 90% of my time when my eyes are open, they are either reading or staring up on the sky while I try to think about what I am going to write next. You know one way or the other it is like that is what I am doing.[0:38:24.0] CC: Yeah.[0:38:25.1] MICHAEL: I noticed last year on my Twitter feed, more people than the years before will complain about like personal burn out. I saw a pattern, like reading those people’s tweets, I saw a pattern there. It wasn’t really like a spiral and then they realized and they shot down like deleted Twitter from their phones or any messaging and other stuff, and I think I am at the point where I also need to do that when it comes to vacation PDO, or whatever.Because I am just like, as you said Duffy, my free time is on the table when it comes to Twitter and catching up and keeping up because work-life balance in my mind is not work but what is not work for like – Kubernetes is exciting, adding in all the space, like what is not work there? I need to really get better at that because I think I might end in the same spiral of just soaking in more until I just –[0:39:17.7] CC: Yeah and like Josh said, it is not that there isn’t a value. Obviously we derive a huge value, that is why we’re on it, but you have to weigh things and what are your goals and is that the best way to your goals from where you are right now, and maybe you know, Twitter you use for a while, ramp up your knowledge, ramp up the connections because it is great for making connections, and then you step back and focus on something else, then to go on a cycle.This is how I am thinking now. It is just like what Olive was saying, you know, books are great, blog posts are great, and I absolutely agree with that. It is just that I don’t have even the time and when I have the time, I would be reading code and I would be reading things all day long, it is just really tiring for me at the end of the day to sit down and read more. I want to invest in learning how to speed read to solve that problem because I read a lot of books and blog posts. So something on my list.[0:40:22.8] DC: One of the biggest tips on speed reading I ever learned is that frequently when you read you think of saying the word and if you can get out of that habit, if you get out of the habit of saying the word even with your mouth or you just get out of that habit that will already increase the quickness of what you read.[0:40:39.5] CC: That is so interesting.[0:40:41.4] DC: Yeah, that is a trippy one.[0:40:43.1] CC: Because I think being bilingual, I totally like – that really helps me understand things, by saying the words.[0:40:52.9] DC: I think the point that we are all working around here is, there is a great panel that came out at KubeCon EU in 2019 was put on by Aaron Crickenberger, Esther McNaMara, Steven Augustus, these folks are all very high output people. I mean, they do a lot of stuff especially with regard to community and so they put on a panel that was talking about burn out and self-care and I think that it is definitely worth checking that one out.And actually also thinking about what keeping up means to you and making sure that you are measuring that against your ability to sustain, is incredibly important, right? I feel like keeping up is one of those subjects where we end up – it is almost insidious in its way to – it is a thing that we can just do all the time. We can just spend all of our time, any free moment that you have, you are sitting on the bus, you are trying to keep up with things.And because that happens so much, I feel like that is sort of one of the ways that we can feel burnt out as you are seeing today. We can feel like we did a lot of things but there was no real result to it and keep in mind that that’s part of it, right? Like when you are thinking about how we are keeping up with it, make sure that the value to your time is still something that you have some cognizance about, that you have some thought about, like is it worth it to me to just spend this six hours reading everything, right?Or would it be better for me to spend some amount of time just not reading, you know? Like doing something else, you know? Like bake a cake for crying out loud, you know?[0:42:29.5] CC: Something that a lot of times we don’t allow ourselves to do and I decided to speak for everybody I am sorry, I just do nothing, because our brain needs that. We need to not be listening, not be reading, just nothing. Just sit and look at the ceiling, our brain needs that. Ideally, look at nature, like look outside, look at the air, go for a walk. We need that, because that recharges the brain. Anyway, one thing also that I want to bring up, maybe we can mention real quick because we are coming up at the top of the hour.How do people, projects, how do we really help the users of those projects to be up to date with what they are doing?[0:43:18.4] DC: Well yeah I mean this is the different patterns that we are talking about. So I think the blog posts help. I like the idea of having blogs that are targeted towards different audiences. I like the idea of having an aggregate here for putting up a big project. I mean obviously Kubernetes is such a huge ecosystem that if you have things like KubeWeekly and I know that there are actually quite a number of things out there that try and do this.But if we can kind of agree on one like KubeWeekly I think is a pretty good one because it is actually run by the CNCF. So it kind of falls within that sort of governance as a model but having an aggregator where you can actually produce content or curate content as it relates to your project that’s helpful, and then office-hours I think is also helpful to Josh’s point. I mean office-hours and SIG hours are very similar things. I mean like office-hours there like how to developers think about what’s happening with the space.This is an opportunity for you as an end user to show up and ask questions, those sorts of patterns I think all are incredibly helpful as a project to figure out there to those things.[0:44:17.8] OP: Yeah, I know summary articles or the sort of TLDRs that Michael mentioned earlier, I think I need more of those things in my life because I do a lot of reading, because I think my brain is a bit weird in that I need to read something about five or six different times from five or six different articles for it to sort of frame in my head.So what I am trying to – like for 2020, I have almost tried to do this, is like if I think somebody knows all about this and it would save me reading those five, six, seven articles and if that person has the time, I try and sort of reach out to them and say, “Listen, have you got 20 minutes or so to explain this topic to me? Can I ask you questions about it?” It just saves me, saves my eyes reading the screen, and it just saves me time. I just need a TLDR summary of a project or a feature or something just so I can know what it is all about in my head and talk fairly sort of confidently about it.If I need to get in front and down under the weeds then there is more reading to kind of do for me maybe the coding on the technical side, but sometimes I can’t figure out what this feature sort of means and what is its use case in the real world and I have to read through lots of articles and sometimes kind of vendor specific ones and they’ve got a different slant than maybe an independent one and trying to marry those bits up my head is a bit hard for me and there is sort of wealth of information.So if you are interested in a topic and there is hundreds of articles and you start reading four or five and they are all slightly different, eventually you figure out that – you are confident and I understand what that product is about but it has taken a long time to get there and it is taken a lot of reading time. So TLDRs is like really work and I think as Josh mentioned before, we have this thing internally where we do bench demos.And that is like a TLDR and a show and tell really quickly, like, “This is what this does and this is why we need to know about it and this is why our customers needs to know about it, the end,” you know? And that’s really, really useful because that just saves a whole bunch of people a whole bunch of time figuring out A, whether they need to know about it and B, actually now understanding that product or feature at the end of the five, 10 minutes which is what they typically are. So they are very useful short snippets of information. Maybe we are back to Twitter.[0:46:37.8] JR: Similar to the idea of giving a demo Olive, you made me think of something and that is that I think one of the ways that I keep up with the space is actually through writing along with reading and I think the notion of like – and this admittedly takes up time and the whole quality of life conversation comes in but using writing to help develop your thoughts and kind of aggregate all of these crazy inputs and try to be somewhat concise, which I know I struggle with, around something I’ve learned.It’s helped me a ton and then that asset kind of becomes reusable to share with other people the thing that you wrote. So for people listening to this I guess maybe a call to action for 2020 if that is your style as well, consider starting to write yourself and becoming a resource, right? Because even if you are new to this space, you’d be amazed at just how writing from your perspective can help other people.[0:47:26.3] DC: I think another one that I actually have been impressed with lately is that a number of consumer companies like people out there like Lyft and companies like that have actually started to surface engineering blogs around how they are using technology and how they are using technology to solve things, which I think, as a service provider, as somebody who is involved in the community of Kubernetes, I find those to be incredibly valuable because I get to actually see how those things are doing.I mean at the same time, I see things like – we talked about KubeCon, which is a convention that they have every year. Obviously the project is large enough to support it but there is actually an incentive if you are a consumer of that project to go and talk about how you are using it, right? It is incentivized in that it is more likely your talk will be accepted if you are a consumer of the product than somebody building it, right? We hear from people building it all the time.I love that idea of incentivizing people who are using this thing get out there and talk about it or share their ideas about it or how they are using it, what problems did it solve for them. That is critical I think.[0:48:31.0] CC: Can I also make a suggestion – is to not so much following on the thread that we are talking about just now but kind of on the general thread of this episode. If you have resources that you do use to keep up with things, stop this recording right now and go and give them a like, give them a follow, give them a thumbs up, show somehow appreciation because what Duffy said just now, he was saying, “Oh it is so helpful when I read a blog post.”But people who are writing, they want to know that. So give them some indication, it counts a lot. It takes a lot of effort to sit down and write something or produce a podcast and if you take any, derive any benefit from it, show appreciation. It motivates people to keep doing it.[0:49:26.4] DC: Yeah, agreed.[0:49:27.9] M: I think that is a great bind maybe to close off this episode because it reiterates that just consuming and keeping up that doesn’t necessarily mean you don’t give back, right? So this is a way of giving back, which is really important to keep that flow and creativeness.[0:49:41.8] CC: I go through a lot of YouTube videos and sometimes I just play one after the other but sometimes, you know, I have been making a point of going back and liking it. Liking the ones that I like – obviously I don’t like everything. I mean things that I don’t like I don’t listen in but you know what I mean? It takes no effort but just so people know, “OK, you did a good job here.” By the way, go to iTunes and rate us. So we will know that you liked it and it will help people find our show, our podcast, and if you are watching us on YouTube, give us a like.[0:50:16.1] DC: All right, well unless anybody has any final thoughts, that is what we wanted to cover this session. So thank you all very, very much and I look forward to seeing you next week.[0:50:25.3] M: Bye-bye.[0:50:26.3] CC: Thank you so much.[0:50:27.4] OP: Bye.[0:50:28.1] JR: Bye.[END OF EPISODE][0:50:28.7] ANNOUNCER: Thank you for listening to The Podlets Cloud Native Podcast. Find us on Twitter at https://twitter.com/ThePodlets and on the http://thepodlets.io/ website, where you'll find transcripts and show notes. We'll be back next week. Stay tuned by subscribing.[END]See omnystudio.com/listener for privacy information.

There are two words that get the blame more often than not when a problem cannot be rooted: the network! Today, along with special guest, Scott Lowe, we try to dig into what the network actually means. We discover, through our discussion that the network is, in fact, a distributed system. This means that each component of the network has a degree of independence and the complexity of them makes it difficult to understand the true state of the network. We also look at some of the fascinating parallels between networks and other systems, such as the configuration patterns for distributed systems. A large portion of the show deals with infrastructure and networks, but we also look at how developers understand networks. In a changing space, despite self-service becoming more common, there is still generally a poor understanding of networks from the developers’ vantage point. We also cover other network-related topics, such as the future of the network engineer’s role, transferability of their skills and other similarities between network problem-solving and development problem-solving. Tune in today! Follow us: https://twitter.com/thepodlets Website: https://thepodlets.io Feeback: info@thepodlets.io https://github.com/vmware-tanzu/thepodlets/issues Hosts: Duffie Cooley Nicholas Lane Josh Rosso Key Points From This Episode: • The network is often confused with the server or other elements when there is a problem.• People forget that the network is a distributed system, which has independent routers.• The distributed pieces that make up a network could be standalone computers.• The parallels between routing protocols and configuration patterns for distributed systems.• There is not a model for eventually achieving consistent networks, particularly if they are old.• Most routing patterns have a time-sensitive mechanism where traffic can be re-dispersed.• Understanding a network is a distributed system gives insights into other ones, like Kubernetes.• Even from a developers’ perspective, there is a limited understanding of the network.• There are many overlaps between developers and infrastructural thinking about systems.• How can network engineers apply their skills across different systems?• As the future changes, understanding the systems and theories is crucial for network engineers.• There is a chasm between networking and development.• The same ‘primitive’ tools are still being used for software application layers.• An explanation of CSMACD, collisions and their applicability. • Examples of cloud native applications where the network does not work at all.• How Spanning Tree works and the problems that it solves.• The relationship between software-defined networking and the adoption of cloud native technologies.• Software-defined networking increases the ability to self-service.• With self-service on-prem solutions, there is still not a great deal of self-service. Quotes: “In reality, what we have are 10 or hundreds of devices with the state of the network as a system, distributed in little bitty pieces across all of these devices.” — @scott_lowe [0:03:11] “If you understand how a network is a distributed system and how these theories apply to a network, then you can extrapolate those concepts and apply them to something like Kubernetes or other distributed systems.” — @scott_lowe [0:14:05] “A lot of these software defined networking concepts are still seeing use in the modern clouds these days” — @scott_lowe [0:44:38] “The problems that we are trying to solve in networking are not different than the problems that you are trying to solve in applications.” — @mauilion [0:51:55] Links Mentioned in Today’s Episode: Scott Lowe on LinkedIn — https://www.linkedin.com/in/scottslowe/ Scott Lowe’s blog — https://blog.scottlowe.org/ Kafka — https://kafka.apache.org/ Redis — https://redis.io/ Raft — https://raft.github.io/ Packet Pushers — https://packetpushers.net/ AWS — https://aws.amazon.com/ Azure — https://azure.microsoft.com/en-us/ Martin Casado — http://yuba.stanford.edu/~casado/ Transcript: EPISODE 15 [INTRODUCTION] [0:00:08.7] ANNOUNCER: Welcome to The Podlets Podcast, a weekly show that explores Cloud Native one buzzword at a time. Each week, experts in the field will discuss and contrast distributed systems concepts, practices, tradeoffs and lessons learned to help you on your cloud native journey. This space moves fast and we shouldn’t reinvent the wheel. If you’re an engineer, operator or technically minded decision maker, this podcast is for you. [EPISODE] [0:00:41.4] DC: Good afternoon everybody. In this episode, we’re going to talk about the network. My name is Duffie Cooley and I’ll be the lead of this episode and with me, I have Nick. [0:00:49.0] NL: Hey, what’s up everyone. [0:00:51.5] DC: And Josh. [0:00:52.5] JS: Hi. [0:00:53.6] DC: And Mr. Scott Lowe joining us as a guest speaker. [0:00:56.2] SL: Hey everyone. [0:00:57.6] DC: Welcome, Scott. [0:00:58.6] SL: Thank you. [0:01:00.5] DC: In this discussion, we’re going to try and stay away, like we do always, we’re going to try and stay away from particular products or solutions that are related to the problem. The goal of it is to really kind of dig in to like what the network means when we refer to it as it relates to like cloud native applications or just application design in general. One of the things that I’ve noticed over time and I’m curious, what you all think but like, one of the things I’ve done over time is that people are kind of the mind that if it can’t root cause a particular issue that they run into, they’re like, “That was the network.” Have you all seen that kind of stuff out there? [0:01:31.4] NL: Yes, absolutely. In my previous life, before being a Kubernetes architect, I actually used my networking and engineering degree to be a network administrator for the Boeing Company, under the Boeing Corporation. Time and time again, someone would come to me and say, “This isn’t working. The network is down.” And I’m like, “Is the network down or is the server down?” Because those are different things. Turns out it was usually the server. [0:01:58.5] SL: I used to tell my kids that they would come to me and they would say, the Internet is down and I would say, “Well, you know. I don’t think the entire Internet is down, I think it’s just our connection to the Internet.” [0:02:10.1] DC: Exactly. [0:02:11.7] JS: Dad, the entire global economy is just taking a total hit. [0:02:15.8] SL: Exactly, right. [0:02:17.2] DC: I frequently tell people that my first distributed system that I ever had a real understanding of was the network, you know? It’s interesting because it kind of like, relies on the premises that I think a good distributed system should in that there is some autonomy to each of the systems, right? They are dependent on each other or even are inter communicate with each other but fundamentally, like when you look at routers and things like that, they are autonomous in their own way. There’s work that they do exclusive to the work that others do and exclusive to their dependencies which I think is very interesting. [0:02:50.6] SL: I think the fact that the network is a distributed system and I’m glad you said that Duffie, I think the fact the network is a distributed system is what most people overlook when they start sort of blaming the network, right? Let’s face it, in the diagrams, right, the network’s always just this blob, right? Here’s the network, right? It’s this thing, this one singular thing. When in reality, what we have are like 10 or hundreds of devices with the state of the network as a system, distributed in little bitty pieces across all of these devices. And no way, aside from logging in to each one of these devices are we able to assemble what the overall state is, right? Even routing protocols mean, their entire purpose is to assemble some sort of common understanding of what the state of the network is. Melding together, not just IP addresses which are these abstract concept but physical addresses and physical connections. And trying to reason to make decisions about them, how we center across and it’s far more complex and a lot of people understand, I think that’s why it’s just like the network is down, right? When reality, it’s probably something else entirely. [0:03:58.1] DC: Yeah, absolutely. Another good point to bring up is that each of these distributed pieces of this distributed system are in themselves like basically like just a computer. A lot of times, I’ve talked to people and they were like, “Well, the router is something special.” And I’m like, “Not really. Technically, a Linux box could just be a router if you have enough ports that you plug into it. Or it could be a switch if you needed to, just plug in ports.” [0:04:24.4] NL: Another good interesting parallel there is like when we talk about like routing protocols which are a way of – a way that allow configuration changes to particular components within that distributed system to be known about by other components within that distributed system. I think there’s an interesting parallel here between the way that works and the way that configuration patterns that we have for distributed systems work, right? If you wanted to make a configuration only change to a set of applications that make up some distributed system, you might go about like leveraging Ansible or one of the many other configuration models for this. I think it’s interesting because it represents sort of an evolution of that same idea in that you’re making it so that each of the components is responsible for informing the other components of the change, rather than taking the outside approach of my job is to actually push a change that should be known about by all of these concepts, down to them. Really, it’s an interesting parallel. What do you all think of that? [0:05:22.2] SL: I don’t know, I’m not sure. I’d have to process that for a bit. But I mean, are you saying like the interesting thought here is that in contrast to typical systems management where we push configuration out to something, using a tool like an Ansible, whatever, these things are talking amongst themselves to determine state? [0:05:41.4] DC: Yeah, it’s like, there are patterns for this like inside of distributed systems today, things like Kafka and you know, Kafka and Gossip protocol, stuff like this actually allows all of the components of a particular distributed system to understand the common state or things that would be shared across them and if you think about them, they’re not all that different from a routing protocol, right? Like the goal being that you give the systems the ability to inform the other systems in some distributed system of the changes that they may have to react to. Another good example of this one, which I think is interesting is like, what they call – when you have a feature behind a flag, right? You might have some distributed configuration model, like a Redis cache or database somewhere that you’ve actually – that you’ve held the running configuration of this distributed system. And when you want to turn on this particular feature flag, you want all of the components that are associated with that feature flag to enable that new capability. Some of the patterns for that are pretty darn close to the way that routing protocol models work. [0:06:44.6] SL: Yeah, I see what you're saying. Actually, that’ makes a lot of sense. I mean, if we think about things like Gossip protocols or even consensus protocols like Raft, right? They are similar to routing protocols in that they are responsible for distributing state and then coming to an agreement on what that state is across the entire system. And we even apply terms like convergence to both environments like we talk about how long it takes routing protocol to converge. And we might also talk about how long it takes for and ETCD cluster to converge after changing the number of members in the cluster of that nature. The point at which everybody in that distributed system, whether it be the network ETCD or some other system comes to the same understanding of what that shared state is. [0:07:33.1] DC: Yeah, I think that’s a perfect breakdown, honestly. Pretty much every routing technology that’s out there. You know, if you’re taking that – the computer of the network, you know, it takes a while but eventually, everyone will reconcile the fact that, “Yeah, that node is gone now.” [0:07:47.5] NL: I think one thing that’s interesting and I don’t know how much of a parallel there is in this one but like as we consider these systems like with modern systems that we’re building at scale, frequently we can make use of things like eventual consistency in which it’s not required per se for a transaction to be persisted across all of the components that it would affect immediately. Just that they eventually converge, right? Whereas with the network, not so much, right? The network needs to be right now and every time and there’s not really a model for eventually consistent networks, right? [0:08:19.9] SL: I don’t know. I would contend that there is a model for eventually consistent networks, right? Certainly not on you know, most organizations, relatively simple, local area networks, right? But even if we were to take it and look at something like a Clos fabric, right, where we have top of rack switches and this is getting too deep for none networking blokes that we know, right? Where you take top of rack switches that are talking layer to the servers below them or the end point below them. And they’re talking layer three across a multi-link piece up to the top, right? To the spine switches, so you have leaf switches, talking up spine switches, they’re going to have multiple uplinks. If one of those uplinks goes down, it doesn’t really matter if the rest off that fabric knows that that link is down because we have the SQL cost multi pathing going across that one, right? In a situation like that, that fabric is eventually consistent in that it’s okay if you know, knee dropping link number one of leaf A up to spine A is down and the rest of the system doesn’t know about that yet. But, on the other hand, if you are looking at network designs where convergence is being handled on active standby links or something of that nature or there aren’t enough paths to get from point A to point B until convergence happens then yes, you’re right. I think it kind of comes down to network design and the underlying architecture and there are so many factors that affect that and so many designs over the years that it’s hard to – I would agree and from the perspective of like if you have an older network and it’s been around for some period of time, right? You probably have one that is not going to be tolerant, a link being down like it will cause problems. [0:09:58.4] NL: Adds another really great parallel in software development, I think. Another great example of that, right? If we consider for a minute like the circuit breaking pattern or even like you know, most load balancer patterns, right? In which you have some way of understanding a list of healthy end points behind the load balancer and were able to react when certain end points are no longer available. I don’t consider that a pattern that I would relate to specifically if they consent to eventual consistency. I feel like that still has to be immediate, right? We have to be able to not send the new transaction to the dead thing. That has to stop immediately, right? It does in most routing patterns that are described by multi path, there is a very time sensitive mechanism that allows for the re-dispersal of that traffic across known paths that are still good. And the work, the amazing amount of work that protocol architects and network engineers go through to understand just exactly how the behavior of those systems will work. Such that we don’t see traffic. Black hole in the network for a period of time, right? If we don’t send traffic to the trash when we know or we have for a period of time, while things converge is really has a lot going for it. [0:11:07.0] SL: Yeah, I would agree. I think the interesting thing about discussing eventual consistency with regards to the networking is that even if we take a relatively simple model like the DOD model where we only have four layers to contend with, right? We don’t have to go all the way to this seven-layer OSI model. But even if we take a simple layer like the DOD four-layer model, we could be talking about the rapid response of a device connected at layer two but the less than rapid response of something operating at layer three or layer four, right? In the case of a network where we have these discreet layers that are intentionally loosely coupled which is another topic, we could talk about from a distribution perspective, right? We have these layers that are intentionally loosely coupled, we might even see consistency and the application of the cap theorem, behave differently at different layers of their model. [0:12:04.4] DC: That’s right. I think it’s fascinating like how much parallel there is here. As you get into like you know, deep architectures around software, you’re thinking of these things as it relates to like these distributed systems, especially as you’re moving toward more cloud native systems in which you start employing things like control theory and thinking about the behaviours of those systems both in aggregate like you know, some component of my application, can I scale this particular component horizontally or can I not, how am I handling state. So many of those things have parallels to the network that I feel like it kind of highlights I’m sure what everybody has heard a million times, you know, that there’s nothing new under the sun. There’s million things that we could learn from things that we’ve done in the past. [0:12:47.0] NL: Yeah, totally agree. I recently have been getting more and more development practice and something that I do sometimes is like draw out like how all of my functions and my methods, and take that in rack with each other across a consisting code base and lo and behold when I draw everything out, it sure does look a lot like a network diagram. All these things have to flow together in a very specific way and you expect the kind of returns that you’re looking for. It looks exactly the same, it’s kind of the – you know, how an atom kind of looks like a galaxy from our diagram? All these things are extrapolated across like – [0:13:23.4] SL: Yeah, totally. [0:13:24.3] NL: Different models. Or an atom looks like a solar system which looks like a galaxy. [0:13:28.8] SL: Nicholas, you said your network administrator at Boeing? [0:13:30.9] NL: I was, I was a network engineer at Boeing. [0:13:34.0] SL: You know, as you were sitting there talking, Duffie, so, I thought back to you Nick, I think all the times, I have a personal passion for helping people continue to grow and evolve in their career and not being stuck. I talk to a lot of networking folks, probably dating because of my involvement, back in the NSX team, right? But folks being like, “I’m just a network engineer, there’s so much for me to learn if I have to go learn Kubernetes, I wouldn’t even know where to start.” This discussion to me underscores the fact that if you understand how a network is a distributed system and how these theories apply to a network, then you can extrapolate those concepts and apply them to something like Kubernetes or other distributed systems, right? Immediately begin to understand, okay. Well, you know, this is how these pieces talk to each other, this is how they come, the consensus, this is where the state is stored, this is how they understand and exchange date, I got this. [0:14:33.9] NL: if you want to go down that that path, the controlled plane of your cluster is just like your central routing back bone and then the kublets themselves are just your edge switches going to each of your individual smaller network and then the pods themselves have been nodes inside of the network, right? You can easily – look at that, holy crap, it looks exactly the same. [0:14:54.5] SL: Yeah, that’s a good point. [0:14:55.1] DC: I mean, another interesting part, when you think about how we characterize systems, like where we learn that, where that skillset comes from. You raise a very good point. I think it’s an easier – maybe slightly easier thing to learn inside of networking, how to characterize that particular distributed system because of the way the components themselves are laid out and in such a common way. Where when we start looking at different applications, we find a myriad of different patterns with particular components that may behave slightly differently depending, right? Like there are different patterns within software like almost on per application bases whereas like with networks, they’re pretty consistently applied, right? Every once in a while, they’ll be kind of like a new pattern that emerges, that it just changes the behavior a little bit, right? Or changes the behavior like a lot but at the same time, consistently across all of those things that we call data center networks or what have you. To learn to troubleshoot though, I think the key part of this is to be able to spend the time and the effort to actually understand that system and you know, whether you light that fire with networking or whether you light that fire with like just understanding how to operationalize applications or even just developing and architecting them, all of those things come into play I think. [0:16:08.2] NL: I agree. I’m actually kind of curious, the three of us have been talking quite a bit about networking from the perspective that we have which is more infrastructure focused. But Josh, you have more of a developer focused background, what’s your interaction and understanding of the network and how it plays? [0:16:24.1] JS: Yeah, I’ve always been a consumer of the network. It’s something that is sat behind an API and some library, right? I call out to something that makes a TCP connection or an http interaction and then things just happen. I think what’s really interesting hearing talk and especially the point about network engineers getting into thee distributed system space is that I really think that as we started to put infrastructure behind API’s and made it more and more accessible to people like myself, app developers and programmers, we started – by we, you know, I’m obviously generalizing here. But we started owning more and more of the infrastructure. When I go into teams that are doing big Kubernetes deployments, it’s pretty rare, that’s the conventional infrastructure and networking teams that are standing up distributed systems, Kubernetes or not, right? It's a lot of times, a bunch of app developers who have maybe what we call dev-ops, whatever that means but they have an application development background, they understand how they interact with API’s, how to write code that respects or interacts with their infrastructure and they’re standing up these systems and I think one of the gaps of that really creates is a lot of people including myself just hearing you all talk, we don’t understand networking at that level. When stuff falls over and it’s either truly the network or it’s getting blamed on the network, it’s often times, just because we truly don’t understand a lot of these things, right? Encapsulation, meshes, whatever it might be, we just don’t understand these concepts at a deep level and I think if we had a lot more people with network engineering backgrounds, shifting into the distributed system space. It would alleviate a bit of that, right? Bringing more understanding into the space that we work in nowadays. [0:18:05.4] DC: I wonder if maybe it also would be a benefit to have like more cross discussions like this one between developers and infrastructure kind of focused people, because we’re starting to see like as we’re crossing boundaries, we see that the same things that we’re doing on the infrastructure side, you’re also doing in the developer side. Like cap theorem as Scott mention which is the idea that you can have two out of three of consistency, availability and partitioning. That also applies to networking in a lot of ways. You can only have a network that is either like consistent or available but it can’t handle partitioning. It can be a consistent to handle partitioning but it’s not always going to be available, that sort of thing. These things that apply in from the software perspective also apply to us but we think about them as being so completely different. [0:18:52.5] JS: Yeah, I totally agree. I really think like on the app side, a couple of years ago, you know, I really just didn’t care anything outside of the JVM like my stuff on the JVM and if it got out to the network layer of the host like just didn’t care, know, need to know about that at all. But ever since cloud computing and distributed systems and everything became more prevalent, the overlap has become extremely obvious, right? In all these different concepts and it’s been really interesting to try to ramp up on that. [0:19:19.6]:19.3] NNL: Yeah, I think you know Scott and I both do this. I think as I imagine, actually, this is true of all four of us to be honest. But I think that it’s really interesting when you are out there talking to people who do feel like they’re stuck in some particular role like they’re specialists in some particular area and we end up having the same discussion with them over and over again. You know, like, “Look, that may pay the bills right now but it’s not going to pay the bills in the future.” And so you know, the question becomes, how can you, as a network engineer take your skills forward and not feel as though you’re just going to have to like learn everything all over again. I think that one of the things that network engineers are pretty decent at is characterizing those systems and being able to troubleshoot them and being able to do it right now and being able to like firefight those capabilities and those skills are incredibly valuable in the software development and in operationalizing applications and in SRE models. I mean, all of those skills transfer, you know? If you’re out there and you’re listening and you feel like I will always be a network engineer, consider that you could actually take those skills forward into some other role if you chose to. [0:20:25.1] JS: Yeah, totally agree. I mean, look at me, the lofty career that I’ve been come to. [0:20:31.4] SL: You know, I would also say that the fascinating thing to me and one of the reasons I launched, I don’t say this to like try and plug it but just as a way of talking about the reason I launched my own podcast which is now part of packet pushers, was exploring this very space and that is like we’ve got folks like Josh who comes from the application development spacing is now being, you know, in a way, forced to own and understand more infrastructure and we’ve got the infrastructure folks who now in a way, whether it be through the rise of cloud computing and abstractions away from visible items are being forced kind of up the stack and so they’re coming together and this idea of what does the future of the folks that are kind of like in our space, what does that look like? How much longer does a network engineer really need to be deeply versed in all the different layers? Because everything’s been abstracted away by some other type of thing whether it’s VPC’s or Azure V Nets or whatever the case is, right? I mean, you’ve got companies bringing the VPC model to on premises networks, right? As API’s become more prevalent, as everything gets sort of abstracted away, what does the future look like, what are the most important skills and it seems to me that it’s these concepts that we’re talking about, right? This idea of distributed systems and how distributed systems behave and how the components react to one another and understanding things like the cap theorem that are going to be most applicable rather than the details of trouble shooting VGP or understanding AWS VPC’s or whatever the case may be. [0:22:08.5] NL: I think there is always going to be a place for the people who know how things are running under the hood from like a physical layer perspective, that sort of thing, there’s always going to be the need for the grave beards, right? Even in software development, we still have the people who are slinging kernel code in C. And you know, they’re the best, we salute you but that is not something that I’m interested in it for sure. We always need someone there to pick up the pieces as it were. I think that yeah, having just being like, I’m a Cisco guy, I’m a Juniper guy, you know? I know how to pawn that or RSH into the switch and execute these commands and suddenly I’ve got this port is now you know, trunk to this V neck crap, I was like, Nick, remember your training, you know? How to issue those commands, I wonder, I think that that isn’t necessarily going away but it will be less in demand in the future. [0:22:08.5] SL: I’m curious to hear Josh’s perspective as like having to own more and more of the infrastructure underneath like what seems to be the right path forward for those folks? [0:23:08.7] JS: Yeah, I mean, unfortunately, I feel like a lot of times, it just ends up being trial by fire and it probably shouldn’t be that. But the amount of times that I have seen a deployment of some technology fall over because we overlapped the site range or something like that is crazy. Because we just didn’t think about it or really understand it that well. You know, like using one protocol, you just described BGP. I never ever dreamt of what BGP was until I started using attributed systems, right? Started using BGP as a way to communicate routes and the amount off times that I’ve messed up that connection because I don’t have a background in how to set that up appropriately, it’s been rough. I guess my perspective is that the technology has gotten better overall and I’m mostly obviously in the Kubernetes space, speaking to the technologies around a lot of the container networking solutions but I’m sure this is true overall. It seems like a lot of the sharp edges have been buffed out quite a bit and I have less of an opportunity to do things terribly wrong. I’ve also noticed for what it’s worth, a lot of folks that have my kind of background or going out to like the AWS is the Azure’s of the world. They’re using all these like, abstracted networking technologies that allow t hem to do really cool stuff without really having to understand how it works and they’re often times going back to their networking team on prem when they have on prem requirements and being like it should be this easy or XY and Z and they’re almost like pushing the networking team to modernize that and make things simpler. Based on experiences they’re having with these cloud providers. [0:24:44.2] DC: Yeah, what do you mean I can’t create a load balancer that crosses between these two disparate data centers as it easily is. Just issuing a single command. Doesn’t this just exist from a networking standpoint? Even just the idea that you can issue an API command and get a load balancer, just that idea alone, the thousands of times I have heard that request in my career. [0:25:08.8] JS: And like the actual work under the hood to get that to work properly is it’s a lot, there’s a lot of stuff going on. [0:25:16.5] SL: Absolutely, yeah, [0:25:17.5] DC: Especially when you’re into plumbing, you know? If you’re going to create a load balancer with API, well then, what API does the load balancer use to understand where to send that traffic when it’s being balanced. How do you handle discovery, how do you hit like – obviously, yeah, there’s no shortage on the amount of work there. [0:25:36.0] JS: Yeah. [0:25:36.3] DC: That’s a really good point, I mean, I think sometimes it’s easy for me to think about some of these API driven networking models and the cost that come with them, the hidden cost that come with them. An example of this is, if you’re in AWS and you have a connectivity between wo availability, actually could be any cloud, it doesn’t have to be an AWS, right? If you have connectivity between two different availability zones and you’re relying on that to be reliable and consistent and definitely not to experience, what tools do you have at your disposal, what guarantees do you have that that network has even operating in a way that is responsive, right? And in a way, this is kind of taking us towards the observability conversation that I think we’ve talked a little bit about the past. Because I think it highlights the same set of problems again, right? You have to understand, you have to be able to provide the consumers of any service, whether that service is plumbing, whether it’s networking, whether it’s your application that you’ve developed that represents a set of micro service. You have to provide everybody a way or you know, have to provide the people who are going to answer the phone at two in the morning. Or even the robots that are going to answer the phone at two in the morning. I have to provide them some mechanism by which to observe those systems as they are in use. [0:26:51.7] JS: I’m not convinced that very many of the cloud providers do that terribly well today, you know? I feel like I’ve been burned in the past without actually having an understanding of the state that we’re in and so it is interesting maybe the software development team can actually start pushing that down toward the networking vendors out there out in the world. [0:27:09.9] NL: Yeah that would be great. I mean I have been recently using a managed Kubernetes service. I have been kicking the tires on it a little bit. And yeah there has been a couple of times where I had just been got by networking issues. I am not going to get into what I have seen in a container network interface or any of the technologies around that. We are going to talk about that another time. But the CNI that I am using in this managed service was just so wonky and weird. And it was failing from a network standpoint. The actual network was failing in a sense because the IP addresses for the nodes themselves or the pods wasn’t being released properly and because of our bag. And so, the rules associated with my account could not remove IP addresses from a node in the network because it wasn’t allowed to and so from a network, I ran out of IP addresses in my very small site there. [0:28:02.1] SL: And this could happen in database, right? This could happen in a cache of information, this could happen in pretty much the same pattern that you are describing is absolutely relevant in both of these fields, right? And that is a fascinating thing about this is that you know we talk about the network generally in these nebulous terms and that it is like a black box and I don’t want them to know anything about it. I want to learn about it, I don’t want to understand it. I just want to be able to consume it via an API and I want to have the expectation that everything will work the way it is supposed to. I think it is fascinating that on the other side of that API are people maybe just like you who are doing their level best to provide, to chase the cap theorum into it’s happy end and figure out how to actually give you what you need out of that service, you know? So, empathy I think is important. [0:28:50.4] NL: Absolutely, to bring that to an interesting thought that I just had where on both sides of this chasm or whatever it is between networking and develop, the same principles exists like we have been saying but just to elicited on it a little bit more, it’s like on one side you have like I need to make sure that these ETCD nodes communicate with each other and that the data is consistent across the other ones. So, we use a protocol called RAFT, right? And so that’s eventually existent tool then that information is sent onto a network, which is probably using OSPF, which is “open shortest path first” routing protocol to become eventually consistent on the data getting from one point to the other by opening the shortest path possible. And so these two things are very similar. They are both these communication protocols, which is I mean that is what protocol means, right? The center for communication but they’re just so many different layers. Obviously of the OSI model but people don’t put them together but they really are and we keep coming back to that where it is all the same thing but we think about it so differently. And I am actually really appreciating this conversation because now I am having a galaxy brain moment like boo. [0:30:01.1] SL: Another really interesting one like another galaxy moment, I think that is interesting is if you think about – so let us break them down like TCP and UTP. These are interesting patterns that actually do totally relate again just in software patterns, right? In TCP the guarantee is that every data gram, if you didn’t get the entire data gram you will understand that you are missing data and you will request a new version of that same packet. And so, you can provide consistency in the form of retries or repeats if things don’t work, right? Not dissimilar from the ability to understand like that whether you chuck some in data across the network or like in a particular data base, if you make a query for a bunch of information you have to have some way of understanding that you got the most recent version of it, right? Or ETCD supports us by using the revision by understanding what revision you received last or whether that is the most recent one. And other software patterns kind of follow the same model and I think that is also kind of interesting. Like we are still using the same primitive tools to solve the same problems whether we are doing it at a software application layer or whether we are doing it down in the plumbing at the network there, these tools are still very similar. Another example is like UTP where it is basically there are no repeats. You either got the packet or you didn’t, which sounds a lot like an event stream to me in some ways, right? Like it is very interesting, you just figured out like I put in on the line, you didn’t get it? It is okay, I will put another line here in a minute you can react to that one, right? It is an interesting overlap. [0:31:30.6] NL: Yeah, totally. [0:31:32.9] JS: Yeah, the comparison to event streams or message queues, right? There is an interesting one that I hadn’t considered before but yeah, there are certainly parallels between saying, “Okay I am going to put this on the message queue,” and wait for the acknowledgement that somebody has taken it and taken ownership of it as oppose to an event stream where it is like this happened. I admit this event. If you get it and you do something with it, great. If you don’t get it then you don’t do something with it, great because another event is going to come along soon. So, there you go. [0:32:02.1] DC: Yep, I am going to go down a weird topic associated with what we are just talking about. But I am going to get a little bit more into the weeds of networking and this is actually directed into us in a way. So, talking about the kind of parallels between networking and development, in networking at least with TCP and networking, there is something called CSMACD, which is “carry your sense multi,” oh I can’t remember what the A stands for and the CD. [0:32:29.2] SL: Access. [0:32:29.8] DC: Multi access and then CD is collision detection and so basically what that means is whenever you sent out a packet on the network, the network device itself is listening on the network for any collisions and if it detects a collision it will refuse to send a packet until a certain period of time and they will do a retry to make sure that these packets are getting sent as efficiently as possible. There is an alternative to that called CMSCA, which was used by Mac before they switched over to using a Linux based operating system. And then putting a fancy UI in front of it, which collision avoidance would listen and try and – I can’t remember exactly, it would time it differently so that it would totally just avoid any chance that there could be collision. It would make sure that no packets were being sent right then and then send it back up. And so I was wondering if something like that exists in the realm between the communication path between applications. [0:33:22.5] JS: Is it collision two of the same packets being sent or what exactly is that? [0:33:26.9] DC: With the packets so basically any data going back and forth. [0:33:29.7] JS: What makes it a collision? [0:33:32.0] SL: It is the idea that you can only transmit one message at a time because if they both populate the same media it is trash, both of them are trash. [0:33:39.2] JS: And how do you qualify that. Do you receive an ac from the system or? [0:33:42.8] NL: No there is just nothing returned essentially so it is like literally like the electrical signals going down the wire. They physically collide with each other and then the signal breaks. [0:33:56.9] JS: Oh, I see, yeah, I am not sure. I think there is some parallels to that maybe with like queuing technologies and things like that but can’t think of anything on like direct app dev side. [0:34:08.6] DC: Okay, anyway sorry for that tangent. I just wanted to go down that little rabbit-hole a little bit. It was like while we are talking about networking, I was like, “Oh yeah, I wanted to see how deep down we can make this parallel going?” so that was the direction I went. [0:34:20.5] SL: Like where is that that CSMACD, a piece is like seriously old school, right? Because it only applied to half duplex Ethernet and as soon as we went to full duplex Ethernet it didn’t matter anymore. [0:34:33.7] DC: That is true. I totally forgot about that. [0:34:33.8] JS: It applied the satellite with all of these as well. [0:34:35.9] DC: Yeah, I totally forgot about that. Yeah and with full duplex, we totally just space on that. This is – damn Scott, way to make me feel old. [0:34:45.9] SL: Well I mean satellite stuff, too, right? I mean it is actually any shared media upon which you have to – where if this stuff goes and overlap there, you are not going to be able to make it work right? And so, I mean it is interesting. It is actually an interesting PNL. I am struggling to think of an example of this as well. I mean my brain is going towards circuit breaking but I don’t think that that is quite the same thing. It is sort the same thing that in a circuit breaking pattern, the application that is making the request has the ability obviously because it is the thing making the request to understand that the target it is trying to connect to is not working correctly. And so, it is able to make an almost instantaneous decision or at least a very shortly, a very timely decision about what to do when it detects that state. And so that’s a little similar and that you can and from the requester side you can do things if you see things going awry. And really and in reality, in the circuit breaking pattern we are making the assumption that only the application making the request will ever get that information fast enough to react to it. [0:35:51.8] JS: Yeah where my head was kind of going with it but I think it is pretty off is like on a low level piece of code like it is maybe something you write in C where you implement your own queue in that area and then multiple threads are firing off the same time and there is no block system or mechanism if two threads contend to put something in the same memory space that that queue represents. That is really going down the rabbit hole. I can’t even speak to what degree that is possible in modern programming but that is where my head was. [0:36:20.3] NL: Yeah that is a good point. [0:36:21.4] SL: Yeah, I think that is actually a pretty good analogy because the key commonality here is some sort of shared access, right? Multiple threads accessing the same stack or memory buffer. The other thing that came to mind to me was like some sort of session multiplexing, right? Where you are running multiple application layer sessions inside a single sort of network connection and those network sessions getting comingled in some fashion. Whether through identifiers or sequence number or something else of that nature and therefore, you know garbling the ultimate communication that is trying to be sent. [0:36:59.2] DC: Yeah, locks are exactly the right direction, I think. [0:37:03.6] NL: That is a very good point. [0:37:05.2] DC: Yeah, I think that makes perfect sense. Good, all right. Yes, we nailed it. [0:37:09.7] SL: Good job. [0:37:10.8] DC: Can anybody here think of a software pattern that maybe doesn’t come across that way? When you are thinking about some of the patterns that you see today in cloud native applications, is there a counter example, something that the network does not do at all? [0:37:24.1] NL: That is interesting. I am trying to think where event streams. No, that is just straight up packets. [0:37:30.7] JS: I feel like we should open up one of those old school Java books of like 9,000 design patterns you need to know and we should go one by one and be like, “What about this” you know? There is probably something I can’t think of it off the top of my head. [0:37:43.6] DC: Yeah me neither. I was trying to think of it. I mean like I can think of a myriad of things that do cross over even the idea of only locally relevant state, right? That is like a cam table on a switch that is only locally relevant because once you get outside of that switching domain it doesn’t matter anymore and it is like there is a ton of those things that totally do relate, you know? But I am really struggling to come up with one that doesn’t – One thing that is actually interesting is I was going to bring up – we mentioned the cap theorem and it is an interesting one that you can only pick like two and three of consistency availability and partition tolerance. And I think you know, when I think about the way that networks solve or try to address this problem, they do it in some pretty interesting way. It’s like if you were to consider like Spanning Tree, right? The idea that there can really only be one path through a series of broadcast domains. Because we have multiple paths then obviously we are going to get duplicity and the things are going to get bad because they are going to have packets that are addressed the same things across and you are going to have all kinds of bad behaviors, switching loops and broadcast storms and all kinds of stuff like that and so Spanning Tree came along and Spanning Tree was invented by an amazing woman engineer who created it to basically ensure that there was only one path through a set of broadcast domains. And in a way, this solved that camp through them because you are getting to the point where you said like since I understand that for availability purpose, I only need one path through the whole thing and so to ensure consistency, I am going to turn off the other paths and to allow for partition tolerance, I am going to enable the system to learn when one of those paths is no longer viable so that it can re-enable one of the other paths. Now the challenge of course is there is a transition period in which we lose traffic because we haven’t been able to open one of those other paths fast enough, right? And so, it is interesting to think about how the network is trying to solve with the part that same set of problems that is described by the cap theorem that we see people trying to solve with software routine. [0:39:44.9] SL: No man I totally agree. In a case like Spanning Tree, you are sacrificing availability essentially for consistency and partition tolerance when the network achieves consistency then availability will be restored and there is other ways to doing that. So as we move into systems like I mentioned clos fabrics earlier, you know a cost fabric is a different way of establishing a solution to that and that is saying I’d later too. I will have multiple connections. I will wait those connections using the higher-level protocol and I will sacrifice consistency in terms of how the routes are exchanged to get across that fabric in exchange for availability and partition columns. So, it is a different way of solving the same problem and using a different set of tools to do that, right? [0:40:34.7] DC: I personally find it funny that in the cap theorem there is at no point do we mention complexity, right? We are just trying to get all three and we don’t care if it’s complex. But at the same time, as a consumer of all of these systems, you care a lot about the complexity. I hear it all the time. Whether that complexity is in a way that the API itself works or whether even in this episode we are talking about like I maybe don’t want to learn how to make the network work. I am busy trying to figure out how to make my application work, right? Like cognitive load is a thing. I can only really focus on so many things at a time where am I going to spend my time? Am I going to spend it learning how to do plumbing or am I going to spend it actually trying the right application that solves my business problem, right? It is an interesting thing. [0:41:17.7] NL: So, with the rise of software defined networking, how did that play into the adoption of cloud native technologies? [0:41:27.9] DC: I think it is actually one of the more interesting overlaps in the space because I think to Josh’s point again. his is where we were taking I mean I work for a company called [inaudible 0:41:37], in which we were virtualizing the network and this is fascinating because effectively we are looking at this as a software service that we had to bring up and build and build reliably and scalable. Reliably and consistently and scalable. We want to create this all while we are solving problems. But we need it to do within an API. It is like we couldn’t make the assumption with the way that networks were being defined today like going to each component and configuring them or using protocols was actually going to work in this new model of software confined networking. And so, we had an incredible amount of engineers who were really focused from a computer science perspective on how to effectively reinvent network as a software solution. And I do think that there is a huge amount of cross over here like this is actually where I think the waters meet between the way the developers think about the problems and the way that network engineers think about the problem but it has been a rough road I will say. I will say that STN I think is actually has definitely thrown a lot of network engineers under their heels because they’re like, “Wait, wait but that is not a network,” you know? Because I can’t actually look at it and characterize it in the way that I am accustomed to looking at characterizing the other networks that I play with. And then from the software side, you’re like, “Well maybe that is okay” right? Maybe that is enough, it is really interesting. [0:42:57.5] SL: You know I don’t know enough about the details of how AWS or Azure or Google are actually doing their networking like and I don’t even know and maybe you guys all do know – but I don’t even know that aside from a few tidbits here and there that AWS is going to even divulge the details of how things work under the covers for VPC’s right? But I can’t imagine that any modern cloud networking solution whether it would be VBPC’s or VNET’s or whatever doesn’t have a significant software to find aspect to it. You know, we don’t need to get into the definitions of what STN is or isn’t. That was a big discussion Duffie and I had six years ago, right? But there has to be some part of it that is taking and using the concepts that are common in STN right? And applying that. Just as the same way as the cloud vendors are using the concepts from compute virtualization to enable what they are doing. I mean like the reality is that you know the work that was done by the Cambridge folks on Zen was a massive enabler trade for AWS, right? The word done on KVM also a massive enabler for lots of people. I think GCP is KBM based and V Sphere where VM Ware data as well. I mean all of this stuff was a massive enablers for what we do with compute virtualization in the cloud. I have to think that whether it is – even if it wasn’t necessarily directly stemming out of Martin Casado’s open flow work at Stanford, right? That a lot of these software define networking concepts are still seeing use in the modern clouds these days and that is what enables us to do things like issue an API call and have an isolated network space with its own address space and its own routing and satiated in some way and managed. [0:44:56.4] JS: Yeah and on that latter point, you know as a consumer of this new software defined nature of networking, it is amazing the amount of I don’t know, I started using like a blanket marketing term here but agility that it is added, right? Because it has turned all of these constructs that I used to file a ticket and follow up with people into self-service things that when I need to poke holes in the network, hopefully the rights are locked down, so I just can’t open it all up. Assuming I know what I am doing and the rights are correct it is totally self-service for me. I go into AWS, I change the security group roll and boom, the ports have changed and it never looked like that prior to this full takeover of what I believe is STN almost end to end in the case of AWS and so on. So, it is really just not only has it made people like myself have to understand more about networking but it has allowed us to self-service a lot of the things. That I would imagine most network engineers were probably tired of doing anyways, right? How many times do you want to go to that firewall and open up that port? Are you really that excited about that? I would imagine not so. [0:45:57.1] NL: Well I can only speak from experience and I think a lot of network engineers kind of get into that field because it really love control. And so, they want to know what these ports are that are opening and it is scary to be like this person has opened up these ports, “Wait what?” Like without them even totally knowing. I mean I was generalizing, I was more so speaking to myself as being self-deprecating. It doesn’t apply to you listener. [0:46:22.9] JS: I mean it is a really interesting point though. I mean do you think it makes the networking people or network engineers maybe a little bit more into the realm of observability and like knowing when to trigger when something has gone wrong? Does it make them more reactive in their role I guess. Or maybe self-service is not as common as I think it is. It is just from my point of view, it seems like with STN’s the ability to modify the network more power has been put into the developers’ hands is how I look at it, you know? [0:46:50.7] DC: I definitely agree with that. It is interesting like if we go back a few years there was a time when all of us in the room here I think are employed by VMware. So, there was a time where VMware’s thing was like the real value or one of the key values that VMware brought to the table was the idea that a developer come and say “Give me 10 servers.” And you could just call an API or make it or you could quickly provision those 10 servers on behalf of that developer and hand them right back. You wouldn’t have to go out and get 10 new machines and put them into a rack, power them and provision them and go through that whole process that you could actually just stamp those things out, right? And that is absolutely parallel to the network piece as well. I mean if there is nothing else that SPN did bring to the fore is that, right? That you can get that same capability of just stamping up virtual machines but with networks that the API is important in almost everything we do. Whether it is a service that you were developing, whether it is a network itself, whether it is the firewall that we need to do these things programmatically. [0:47:53.7] SL: I agree with you Duffie. Although I would contend that the one area that and I will call it on premises STN shall we say right? Which is the people putting on STN solutions. I’d say the one area at least in my observation that they haven’t done well is that self-service model. Like in the cloud, self-service is paramount to Josh’s point. They can go out there, they can create their own BPC’s, create their own sub nets, create their own NAT gateways, Internet gateways to run security groups. Load balancers, blah-blah, all of that right? But it still seems to me that even though we are probably 90, 95% of the way there, maybe farther in terms of on premise STN solutions right that you still typically don’t see self-service being pushed out in the same way you would in the public cloud, right? That is almost the final piece that is needed to bring that cloud experience to the on-premises environment. [0:48:52.6] DC: That is an interesting point. I think from an infrastructure as a service perspective, it falls into that realm. It is a problem to solve in that space, right? So when you look at things like OpenStack and things like AWS and things like JKE or not JKE but GCE and areas like that, it is a requirement that if you are going to provide infrastructure as a service that you provide some capability around networking but at the same time, if we look at some of the platforms that are used for things like cloud native applications. Things like Kubernetes, what is fascinating about that is that we have agreed on a least come – we agreed on abstraction of networking that is maybe I don’t know, maybe a little more precooked you know what I mean? In the assumption within like most of the platforms as a service that I have seen, the assumption is that when I deploy a container or I deploy a pod or I deploy some function as a service or any of these things that the networking is going to be handled for me. I shouldn’t have to think about whether it is being routed to the Internet or not or routed back and forth between these domains. I should if anything only have to actually give you intent, be able to describe to you the intent of what could be connected to this and what ports I am actually going to be exposing and that the platform actually hides all of the complexity of that network away from me, which is an interesting round to strike. [0:50:16.3] SL: So, this is one of my favorite things, one of my favorite distinctions to make, right? And that is this is the two worlds that we have been talking about, applications and infrastructure and the perfect example of these different perspectives and you even said it or you talked there Duffie like from an IS perspective it is considered a given that you have to be able to say I want a network, right? But when you come at this from the application perspective, you don’t care about a network. You just want network connectivity, right? And so, when you look at the abstractions that IS vendors and solutions or products have created then they are IS centric but when you look at the abstractions that have been created in the cloud data space like within Kubernetes, they are application centric, right? And so, we are talking about infrastructure artifacts versus application artifacts and they end up meeting but they are coming at this from two different very different perspectives. [0:51:18.5] DC: Yeah. [0:51:19.4] NL: Yeah, I agree. [0:51:21.2] DC: All right, well that was a great discussion. I imagine that we are probably get into – at least I have a couple of different networking discussions that I wanted to dig into and this conversation I hope that we’ve helped draw some parallels back and forth between the way – I mean there is both some empathy to spend here, right? I mean the people who are providing the service of networking to you in your cloud environments and your data centers are solving almost exactly the same sorts of availability problems and capabilities that you are trying to solve with your own software. And I think in itself is a really interesting takeaway. Another one is that again there is nothing new under the sun. The problems that we are trying to solve in networking are not different than the problems that you are trying to solve in applications. We have far fewer tools and we generally network engineers are focused on specific changes that happen in the industry rather than looking at a breathe of industries like I mean as Josh pointed out, you could break open a Java book. And see 8,000 patterns for how to do Java and this is true, every programming language that I am aware of I mean if you look at Go and see a bunch of different patterns there and we have talked about different patterns for just developing cloud native aware applications as well, right? I mean there is so many options in the software versus what we can do and what are available to us within networks. And so I think I am rambling a little bit but I think that is the takeaway from this session. Is that there is a lot of overlap and there is a lot of really great stuff out there. So, this is Duffie, thank you for tuning in and I look forward to the next episode. [0:52:49.9] NL: Yep and I think we can all agree that Token Ring should have won. [0:52:53.4] DC: Thank you Josh and thank you Scott. [0:52:55.8] JS: Thanks. [0:52:57.0] SL: Thanks guys, this was a blast. [END OF EPISODE] [0:52:59.4] ANNOUNCER: Thank you for listening to The Podlets Cloud Native Podcast. Find us on Twitter at https://twitter.com/ThePodlets and on the http://thepodlets.io/ website, where you'll find transcripts and show notes. We'll be back next week. Stay tuned by subscribing. [END]See omnystudio.com/listener for privacy information.

Today on the show we have esteemed Kubernetes thought-leader, Kelsey Hightower, with us. We did not prepare a topic as we know that Kelsey presents talks and features on podcasts regularly, so we thought it best to pick his brain and see where the conversation takes us. We end up covering a mixed bag of super interesting Kubernetes related topics. Kelsey begins by telling us what he has been doing and shares with us his passion for learning in public and why he has chosen to follow this path. From there, we then talk about the issue of how difficult many people still think Kubernetes is. We discover that while there is no doubting that it is complicated, at one point, Linux was the most complicated thing out there. Now, we install Linux servers without even batting an eyelid and we think we can reach the same place with Kubernetes in the future if we shift our thinking! We also cover other topics such as APIs and the debates around them, common questions Kelsey gets before finally ending with a brief discussion on KubeCon. From the attendance and excitement, we saw that this burgeoning community is simply growing and growing. Kelsey encourages us all to enjoy this spirited community and what the innovation happening in this space before it simply becomes boring again. Tune in today! Follow us: https://twitter.com/thepodlets Website: https://thepodlets.io Feeback: info@thepodlets.io https://github.com/vmware-tanzu/thepodlets/issues Hosts: Carlisia Campos Duffie Cooley Bryan Liles Michael Gasch Key Points From This Episode: Learn more about Kelsey Hightower, his background and why he teaches Kubernetes! The purpose of Kelsey’s course, Kubernetes the Hard Way. Why making the Kubernetes cluster disappear will change the way Kubernetes works. There is a need for more ops-minded thinking for the current Kubernetes problems. Find out why Prometheus is a good example of ops-thinking applied to a system. An overview of the diverse ops skillsets that Kelsey has encountered. Being ops-minded is just an end –you should be thinking about the next big thing! Discover the kinds of questions Kelsey is most often asked and how he responds. Some interesting thinking and developments in the backup space of Kubernetes. Is it better to backup or to have replicas? If the cost of losing data is very high, then backing up cannot be the best solution. Debates around which instances are not the right ones to use Kubernetes in. The Kubernetes API is the part everyone wants to use, but it comes with the cluster. Why the Kubernetes API is only useful when building a platform. Can the Kubernetes control theory be applied to software? Protocols are often forgotten about when thinking about APIs. Some insights into the interesting work Akihiro Suda’s is doing. Learn whether Kubernetes can run on Edge or not. Verizon: how they are changing the Edge game and what the future trajectory is. The interesting dichotomy that Edge presents and what this means. Insights into the way that KubeCon is run and why it’s structured in the way it is. How Spotify can teach us a lesson in learning new skills! Quotes: “The real question to come to mind: there is so much of that work that how are so few of us going to accomplish it unless we radically rethink how it will be done?” — @mauilion [0:06:49] “If ops were to put more skin in the game earlier on, they would definitely be capable of building these systems. And maybe they even end up more mature as more operations people put ops-minded thinking into these problems.” — @kelseyhightower [0:04:37] “If you’re in operations, you should have been trying to abstract away all of this stuff for the last 10 to 15 years.” — @kelseyhightower [0:12:03] “What are you backing up and what do you hope to restore?” — @kelseyhightower [0:20:07] “Istio is a protocol for thinking about service mesh, whereas Kubernetes provides the API for building such a protocol.” — @kelseyhightower [0:41:57] “Go to sessions you know nothing about. Be confused on purpose.” — @kelseyhightower [0:51:58] “Pay attention to the fundamentals. That’s the people stuff. Fundamentally, we’re just some people working on some stuff.” — @kelseyhightower [0:54:49] Links Mentioned in Today’s Episode: The Podlets on Twitter — https://twitter.com/thepodlets Kelsey Hightower — https://twitter.com/kelseyhightower Kelsey Hightower on GitHub — https://github.com/kelseyhightower Interaction Protocols: It's All about Good Manners — https://www.infoq.com/presentations/history-protocols-distributed-systems Akihiro Suda — https://twitter.com/_AkihiroSuda_ Carlisia Campos on LinkedIn — https://www.linkedin.com/in/carlisia/ Kubernetes — https://kubernetes.io/ Duffie Cooley on LinkedIn — https://www.linkedin.com/in/mauilion/ Bryan Liles on LinkedIn — https://www.linkedin.com/in/bryanliles/ KubeCon North America — https://events19.linuxfoundation.org/events/kubecon-cloudnativecon-north-america-2019/ Linux — https://www.linux.org/ Amazon Fargate — https://aws.amazon.com/fargate/ Go — https://golang.org/ Docker — https://www.docker.com/ Vagrant — https://www.vagrantup.com/ Prometheus — https://prometheus.io/ Kafka — https://kafka.apache.org/ OpenStack — https://www.openstack.org/ Verizon — https://www.verizonwireless.com/ Spotify — https://www.spotify.com/ Transcript: EPISODE 7 [INTRODUCTION] [0:00:08.7] ANNOUNCER: Welcome to The Podlets Podcast, a weekly show that explores Cloud Native one buzzword at a time. Each week, experts in the field will discuss and contrast distributed systems concepts, practices, tradeoffs and lessons learned to help you on your cloud native journey. This space moves fast and we shouldn’t reinvent the wheel. If you’re an engineer, operator or technically minded decision maker, this podcast is for you. [INTERVIEW] [00:00:41] CC: Hi, everybody. Welcome back to The Podlets, and today we have a special guest with us, Kelsey Hightower. A lot of people listening to us today will know Kelsey, but as usual, there are a lot of new comers in this space. So Kelsey, please give us an introduction. [00:01:00] KH: Yeah. So I consider myself a minimalist. So I want to keep this short. I work at Google, on Google Cloud stuff. I’ve been involved with the Kubernetes community for what? 3, 4, 5 years ever since it’s been out, and one main goal, learning in public and helping other people do the same. [00:01:16] CC: There you go. You do have a repo on your GitHub that it’s about learning Kubernetes the hard way. Are you still maintaining that? [00:01:26] KH: Yeah. So every six months or so. So Kubernetes is a hard way for those that don’t know. It’s a guide, a tutorial. You can copy and paste. It takes about three hours, and the whole goal of that guide was to teach people how to stand up a Kubernetes cluster from the ground up. So starting from scratch, 6 VMs, you install etcd, all the components, the nodes, and then you run a few test workloads so you can get a feel for Kubernetes. The history behind that was when I first joined Google, we were all concerned about the adaption of such a complex system that Kubernetes is, right? Docker Swarm is out at the time. A lot of people are using Mesos and we’re wondering like a lot of the feedback at that time was Kubernetes is too complex. So Kubernetes the hard way was built as an idea that if people understand how it worked just like they understand how Linux works, because that’s also complex, that if people just saw how the moving pieces fit together, then they would complain less about the complexity and have a way to kind of grasp it. [00:02:30] DC: I’m back. This is Duffie Colley. I’m back this week, and then we also have Michael and Bryan with us. So looking forward to this session talking through this stuff. [00:02:40] CC: Yeah. Thank you for doing that. I totally forgot to introduce who else is in this show, and me, Carlisia. We didn’t plan what the topic is going to be today. I will take a wild guess, and we are going to touch on Kubernetes. I have so many questions for you, Kelsey. But first and foremost, why don’t you tell us what you would love to talk about? One thing that I love about you is that every time I hear an interview of you, you’re always talking about something different, or you’re talking about the same thing in a different way. I love that about the way you speak. I know you offer to be on a lot of podcast shows, which is how we ended up here and I was thinking, “Oh my gosh! We’re going to talk about what everybody is going to talk about, but I know that’s not going to happen.” So feel free to get a conversation started, and we are VMware engineers here. So come at us with questions, but also what you would like to talk about on our show today. [00:03:37] KH: Yeah. I mean, we’re all just coming straight off the hills of KubeCon, right? So this big, 12,000 people getting together. We’re super excited about Kubernetes and the Mister V event, things are wrapping up there as well. When we start to think about Kubernetes and what’s going to happen, and a lot of people saw Amazon jump in with Fargate for EKS, right? So those unfamiliar with that offering, over the years, all the cloud providers have been providing some hosted Kubernetes offering, the ideas that the cloud provider, just like we do with hypervisors and virtual machines, would provide this base infrastructure so you can focus on using Kubernetes. You’ve seen this even flow down on-prem with VMware, right? VMware saying, “Hey, Kubernetes is going to be a part of this control plane that you can use to Kubernetes’ API to manage virtual machines and containers on-prem.” So at some point now, where do we go from here? There’s a big serverless movement, which is trying to eliminate infrastructure for all kinds of components, whether that’s compute, database as a storage. But even in the Kubernetes world, I think there’s an appetite when we saw this with Fargate, that we need to make the Kubernetes cluster disappear, right? If we can make it disappear, then we can focus on building new platforms that extend the API or, hell, just using Kubernetes as is without thinking about managing nodes, operating systems and autoscalers. I think that’s kind of been the topic that I’m pretty interested in talking about, because that feature means lots of things disappear, right? Programming languages and compilers made assembly disappear for a lot of developers. Assembly is still there. I think people get caught up on nothing goes away. They’re right. Nothing goes away, but the number of people who have to interact with that thing is greatly reduced. [00:05:21] BL: You know what, Kelsey? I’m going to have you get out of my brain, because that was the exact example that I was going to use. I was on a bus today and I was thinking about all the hubbub, about the whole Fargate EKS thing, and then I was thinking, “Well, Go, for example, can generate assembler and then it compiles that down.” No one complains about the length of the assembler that Go generates. Who cares? That’s how we should think about this problem. That’s a whole solvable problem. Let’s think about bigger things. [00:05:51] KH: I think it’s because in operations we tend to identify ourselves as the people responsible for running the nodes. We’re the people responsible for tuning the API server. When someone says it’s going to go away, in ops – And you see this in some parts, right? Ops, some people focus a lot more on observability. They can care less about what machine something runs on. They’re still going to try to observe and tune it. You see this in SRE and some various practices. But a lot of people who came up in a world like I have in a traditional ops background, you were the one that pixie-booted the server. You installed that Linux OS. You configured it with Puppet. When someone tells you, “We’re going to move on from that as if it’s a good thing.” You’re going to be like, “Hold up. That’s my job.” [00:06:36] DC: Definitely. We’ve touched this topic through a couple of different times on this show as well, and it definitely comes back to like understanding that, in my opinion, it’s not about whether there will be a worker for people who are in operations, people who want to focus on that. The real question that come to mind is like there is so much of that work that how are so few of us are going to be able to accomplish it unless we radically re-sync how it will be done. We’re vastly outnumbered. The number of people walking into the internet for the first time every day is mind-boggling. [00:07:08] KH: In early days, we have this goal of abstract or automating ourselves out of a job, and anyone that tried that a number of times knows that you’re always going to have something else to do. I think if we carry that to the infrastructure, I want to see the ops folks. I was very surprised that Docker didn’t come from operations folks. It came from the developer folks. Same thing for Vagrant and the same thing from Kubernetes. These are developer-minded folks that want to tackle infrastructure problems. If I think if ops were to put more skin in the game earlier on, definitely capable of building these systems and maybe they even end up more mature as more operations people put ops-minded thinking to these problems. [00:07:48] BL: Well, that’s exactly what we should do. Like you said, Kelsey, we will always have a job. Whenever we solve one problem, we could think about more interesting problems. We don’t think about Linux on servers anymore. We just put Linux on servers and we run it. We don’t think about the 15 years where it was little rocky. That’s gone now. So think about what we did there and let’s do that again with what we’re doing now. [00:08:12] KH: Yeah. I think the Prometheus community is a good example of operations-minded folks producing a system. When you meet the kind of the originators of Prometheus, they took a lot of their operational knowledge and kind of build this metrics and monitoring standard that we all kind of think about now when we talk about some levels of observability, and I think that’s what happens when you have good operations people that take prior experience, the knowledge, and that can happen over code these days. This is the kind of systems they produce, and it’s a very robust and extensible API that I think you start to see a lot of adaption. [00:08:44] BL: One more thing on Prometheus. Prometheus is six-years-old. Just think about that, and that’s not done yet, and it’s just gotten better and better and better. We go to give up our old thing so we can get better and better and better. That’s just what I want to add. [00:08:58] MG: Kelsey, if you look at the – Basically your own history of coming from ops, as I understood your own history, right? Now being kind of one of the poster childs in the Kubernetes world, you see the world changing to serverless, to higher abstractions, more complex systems on one hand, but then on the other side, we have ops. Looking beyond or outside the world of Silicon Valley into the traditional ops, traditional large enterprise, what do you think is the current majority level of these ops people? I don’t want to discriminate anyone here. I’m just basically throwing this out as a question. Where do you think do they need to go in terms of to keep up with this evolving and higher level abstractions where we don’t really care about nitty-gritty details? [00:09:39] KH: Yes. So this is a good, good question. I spent half of my time. So I probably spent time onsite with at least 100 customers a year globally. I fly on a plane and visit them in their home turf, and you definitely meet people at various skill levels and areas of responsibility. I want to make sure that I’m clear about the areas of responsibility. Sometimes you’re hired in an area of responsibility that’s below your skillset. Some people are hired to manage batch jobs or to translate files from XML to JSON. That really doesn’t say a lot about their skillset. It just kind of talks about the area of responsibility. So shout out to all the people that are dealing with main frames and having to deal with that kind of stuff. But when you look at it, you have the opportunity to rise up to whatever level you want to be in in terms of your education. When we talk about this particular question, some people really do see themselves as operators, and there’s nothing wrong with that. Meaning, they could come in. They get a system and they turn the knobs. You gave me a mainfrastructure me, I will tell you how to turn the knobs on that mainframe. You buy me a microwave, I’ll tell you how to pop popcorn. They’re not very interested in building a microwave. Maybe they have other things that are more important to them, and that is totally okay. Then you have people who are always trying to push the boundaries. Before Kubernetes, if I think back to 10 years ago, maybe 8. When I was working in a traditional enterprise, like kind of the ones you’re talking about or hinting at, the goal has always been to abstract away all of these stuff that it means to deploy an application the right way in a specific environment for that particular company. The way I manage to do it was say, “Hey, look. We have a very complex change in management processes.” I work in finance at that time. So everything had to have a ticket no matter how good the automation was. So I decided to make JIRA the ticketing system their front door to do everything. So you go to JIRA. There’ll be a custom field that says, “Hey, here are all the RPMs that have been QA’d by the QA team. Here are all the available environments.” You put those two fields in. That ticket goes to change in management and approval, and then something below the scenes automated everything, in that case it was Puppet, Red Hat and VMware, right? So I think what most people have been doing if you’re in the world of abstracting this stuff away and making it easier for the company to adapt, you’ve already been pushing these ideas that we call serverless now. I think the cloud providers put these labels on platforms to describe the contract between us and the consumer of the APIs that we present. But if you’re in operations, you should have been trying to abstract away all of these stuff for the last 10 or 15 years. [00:12:14] BL: I 100% agree. Then also, think about other verticals. So 23 years ago, I did [inaudible 00:12:22] work. That was my job. But we learned how to program in C and C++ because we were on old Suns, not even Spark machines. We’re on the old Suns, and we wanted to write things in CVE and we wanted to write our own Window managers. That is what we’re doing right now, and that’s why you see like Mitchell Hashimoto with Vagrant and you’re seeing how we’re pushing this thing. We have barely scratched the surface of what we’re trying to do. For a lot of people who are just ops-minded, understand that being ops-minded is just the end. You have to be able to think outside of your boundaries so you can create the next big thing. [00:12:58] KH: Of you may not care about creating the next big thing. There are parts of my life where I just don’t care. For example, I pay Comcast to get internet access, and my ops involvement was going to BestBuy and buying a modem and screwing it into the wall, and I troubleshoot this thing every once in a while when someone in the household complains the internet is down. But that’s just far as I’m ever going to push the internet boundaries, right? I am not really interested in pushing that forward. I’m assuming others will, and I think that’s one thing in our industry where sometimes we believe that we all need to contribute to pushing things forward. Look, there’s a lot of value in being a great operations person. Just be welcomed to saying that what we operate will change overtime. [00:13:45] DC: Yeah, that’s fair. Very fair. For me, personally, I definitely identify as an operations person. I don’t consider it my life’s goal to create new work necessarily, but to expand on the work that has been identified and to help people understand the value of it. I find I sit in between two roles personally. One is to help figure out all of the different edges and pieces and parts of Kubernetes or some other thing in the ecosystem. Second, to educate others on those things, right? Take what I’ve learned and amplify it. Having the amplifying effect. [00:14:17] CC: One thing that I wanted to ask you, Kelsey is – I work on the Valero project, and that does back and recovery of Kubernetes clusters. Some people ask me, “Okay. So tell me about the people who are doing?” I’m like, “I don’t want to talk about that. That’s boring. I wanted to talk about the people who are not doing backups.” “Okay. Let’s talk about why you should be doing maybe thinking about that.” Well, anyway. I wonder if you get a lot of questions in the area of Kubernetes operations or cloud native in general, infrastructure, etc., that in the back of your mind you go, “That’s the wrong question or questions.” Do you get that? [00:14:54] KH: Yeah. So let’s use your backup example. So I think when I hear questions, at least it lets me know what people are thinking and where they’re at, and if I ask enough questions, I can kind of get a pulse in the trend of where the majority of the people are. Let’s take the backups questions. When I hear people say, “I want to back up my Kubernetes cluster.” I rewind the clock in my mind and say, “Wow! I remember when we used to backup Linux servers,” because we didn’t know what config files were on the disk. We didn’t know where processes are running. So we used to do these PS snapshots and we used to pile up the whole file system and store it somewhere so we can recover it. Remember Norton Ghost? You take a machine and ghost it so you can make it again. Then we said, “You know what? That’s a bad idea.” What we should be doing is having a tool that can make any machine look like the way we want it. Config management is boring. So we don’t back those up anymore. So when I hear that question I say, “Hmm, what is happening in the community that’s keeping people to ask these questions?” Because if I hear a bunch of questions that already have good answers, that means those answers aren’t visible enough and not enough people are sharing these ideas. That should be my next key note. Maybe we need to make sure that other people know that that is no longer a boring thing, even though it’s boring to me, it’s not boring to the industry in general. When I hear these question I kind of use it as a keeps me up-to-date, keeps me grounded. I hear stuff like how many Kubernetes clusters should I have? I don’t think there’s a best practice around that answer. It depends on how your company segregates things, or depends on how you understand Kubernetes. It depends on the way you think about things. But I know why they’re asking that question, is because Kubernetes presents itself as a solution to a much broader problem set than it really is. Kubernetes manages a group of machines typically backed by IS APIs. If you have that, that’s what it does. It doesn’t do everything else. It doesn’t tell you exactly how you should run your business. It doesn’t tell you how you should compartmentalize your product teams. Those decisions you have to make independently, and once you do, you can serialize those into Kubernetes. So that’s the way I think about those questions when I hear them, like, “Wow! Yeah, that is a crazy thing that you’re still asking this question six years later. But now I know why you’re asking that question.” [00:17:08] CC: That is such a great take on this, because, yes, it in the area of backup, people who are doing backup in my mind – Yeah, they should be independent of Kubernetes or not. But let’s talk about the people who are not doing backups. What motivates you to not do backups? Obviously, backups can be done in many different ways. But, yes. [00:17:30] BL: So think about it like this way. Some people don’t exercise, because exercise is tough and it’s hard, and it’s easier to sit on the couch and eat a bag of potato chips than exercise. It’s the same thing with backups. Well, backing up my Kubernetes cluster before Valero was so hard that I’d rather just invest brain cycles in figuring out how to make money. So that’s where people come from when it comes to hard things like backups. [00:17:52] KH: There’s a trust element too, right? Because we don’t know if the effort we’re putting in is worth it. When people do unit testing, a lot of times unit testing can be seen as a proactive activity, where you write unit tests to catch bugs in the future. Some people only write unit test when there’s a problem. Meaning, “Wow! There’s an odd things in a database. Maybe we should write a test to prove that our code is putting odd things. Fix the code, and now the test pass.” I think it’s really about trusting that the investment is worth it. I think when you start to think about backups – I’ve seen people back up a lot of stuff, like every day or every couple of hours, they’re backing up their database, but they’d never restored the database. Then when you read their root cause analysis, they’re like, “Everything was going fine until we tried to restore a 2 terabyte database over 100 meg link. Yeah, we never exercised that part.” [00:18:43] CC: That is very true. [00:18:44] DC: Another really fascinating thing to think about the backup piece is that especially like in the Kubernetes with Valero and stuff, we’re so used to having the conversation around stateless applications and being able to ensure that you can redeploy in the case of a failure. You’re not trying to actually get back to a known state the way that like a backup traditionally would. You’re just trying to get back to a running state. So there’s a bit of a dichotomy there I think for most folks. Maybe they’re not conceptualizing the need for having to deal with some of those stateful applications when they start trying to just think about how Valero fits into the puzzle, because they’ve been told over and over again, “This is about immutable infrastructure. This is about getting back to running. This is not about restoring some complex state.” So it’s kind of interesting. [00:19:30] MG: I think part of this is also that for the stateful services that why we do backups actually, things change a lot lately, right? With those new databases, scale out databases, cloud services. Thinking about backup also has changed in the new world of being cloud native, which for most of the people, that’s also a new learning experiment to understand how should I backup Kafka? It’s replicated, but can I backup it? What about etcd and all those things? Little different things than backing up a SQL database like more traditional system. So backup, I think as you become more complex, stays if needed for [inaudible 00:20:06]. [00:20:06] KH: Yeah. The case is what are you backing up and what do you hope to restore? So replication, global replication, like we do with like cloud storage and S3. The goal is to give some people 11 9s of reliability and replicate that data almost as many geographies as you can. So it’s almost like this active backup. You’re always backing up and restoring as a part of the system design versus it being an explicit action. Some people would say the type of replication we do for object stores is much closer to active restoring and backing up on a continuous basis versus a one-time checkpoint. [00:20:41] BL: Yeah. Just a little bit of a note, you can back up two terabytes over 100 meg link in like 44 hours and a half. So just putting out there, it’s possible. Just like two days. But you’re right. When it comes to backups, especially for like – Let’s say you’re doing MySQL or Postgres. These days, is it better to back it up or is it better to have a replica right next to it and then having like a 10 minute delayed replica right next to that and then replicating to Europe or Asia? Then constantly querying the data that you’re replicating. That’s still a backup. What I’m saying here is that we can change the way that we talk about it. Backup started as conventional as they used to be. There are definitely other ways to protect your data. [00:21:25] KH: Yeah. Also, I think the other part too around the backup thing is what is the price of data loss? When you take a backup, you’re saying, “I’m willing to lose this much data between the last backup and the next.” That cost is too high than backing up cannot be your primary mode of operation, because the cost of losing data is way too high, then replication becomes a complementing factor in the whole discussion of backups versus real-time replication and shorter times to recovery. I have a couple of questions. When should people not use Kubernetes? Do you know what I mean? I visit a lot of customers, I work with a lot of eng teams, and I am in the camp of Kubernetes is not for everything, right? That’s a very obvious thing to say. But some people don’t actually practice it that way. They’re trying to jam more and more into Kubernetes. So I love to get your insights on where do you see Kubernetes being like the wrong direction for some folks or workloads. [00:22:23] MG: I’m going to scratch this one from my question list to Kelsey. [00:22:26] KH: I’ll answer it too then. I’ll answer it after you will answer it. [00:22:29] MG: Okay. Who wants to go first? [00:22:30] BL: All right. I’ll go first. There are cases when I’m writing a piece of software where I don’t care about the service discovery. I don’t care about ingress. It’s just software that needs to run. When I’m running it locally, I don’t need it. If it’s simple enough where I could basically throw it into a VM through a CloudNet script, I think that is actually lower friction than Kubernetes if it’s simple. Now, but I’m also a little bit jaded here, because I work for the dude who created Kubernetes, and I’m paid to create solutions for Kubernetes, but I’m also really pragmatic about it as well. It’s all about effort for me. If I can do it faster in CloudNet, I will. [00:23:13] DC: For my part, I think that there’s – I have a couple of – I got follow on questions to this real quick. But I do think that if you’re not actively trying to develop a distributed systems, something where you’re actually making use of the primitives that Kubernetes provides, then that already would kind of be a red flag for me. If you’re building a monolithic application or if you’re in that place where you’re just rapidly iterating on a SaaS product and you’re just trying to like get as many commits on this thing until it works and like just really rapidly prototype or even create this thing. Maybe Kubernetes isn’t the right thing, because although we’ve come a long way in improving the tools that allow for that iteration, I certainly wouldn’t say that we’re like all the way there yet. [00:23:53] BL: I would debate you that, Duffy. [00:23:55] DC: All right. Then the other part of it is Kubernetes aside, I’m curious about the same question as it relates to containerization. Is it containerization the right thing for everyone, or have we made that pronouncement, for example? [00:24:08] KH: I’m going to jump in and answer on this one, because I definitely think we need a way to transport applications in some way, right? We used to do it on floppy disks. We used to do it on [inaudible 00:24:18]. I think the container to me I treat as a glorified [inaudible 00:24:23]. That’s the way I’ve been seeing it for years. Registry store them. They replace [inaudible 00:24:28]. Great. Now we kind of have a more maybe universal packaging format that can handle simple use cases, scratch containers where it’s just your binary, and the more complex use cases where you have to compose multiple layers to get the output, right? I think RPM spec files used to do something very similar when you start to build those thing in [inaudible 00:24:48], “All right. We got that piece.” Do people really need them? The thing I get weary about is when people believe they have to have Kubernetes on their laptop to build an app that will eventually deploy to Kubernetes, right? If we took that thinking about the cloud, then everyone would be trying to install open stack on their laptop just to build an app. Does that even make sense? Does that make sense in that context? Because you don’t need the entire cloud platform on your laptop to build an app that’s going to take a request and respond. I think Kubernetes people, I guess because it’s easier to put your on laptop, people believe that it needs to be there. So I think Kubernetes is overused, because people just don’t quite understand what it does. I think there’s a case where you don’t use Kubernetes, like I need to read a file from a bucket. Someone uploaded an XML file and my app is going to translate it into JSON. That’s it. In that case, this is where I think functions as a service, something like Cloud Run or even Heroku make a lot more sense to me because the operational complexity is kind of hitting within a provider and is linked almost like an SDK to the overall service, which is the object store, right? The compute part, I don’t want to make a big deal about, because it’s only there to process the file that got uploaded, right? It’s almost like a plug-in to an FTP server, if you will. Those are the cases where I start to see Kubernetes become less of a need, because I need a custom platform to do such an obvious operation. [00:26:16] DC: Those applications that require the primitives that Kubernetes provides, service discovery, the ability to define ingress in a normal way. When you’re actually starting to figure out how you’re going to platform that application with regard to those primitives, I do see the argument for having Kubernetes locally, because you’re going to be using those tools locally and remotely. You have some way of defining what that platforming requirement is. [00:26:40] KH: So let me pull on that thread. If you have an app that depends on another app, typically we used to just have a command line flag that says, “This app is over there.” Local host when it’s on my laptop. Some DNS name when it’s in the cluster, or a config file can satisfy that need. So the need for service discovery usually arises where you don’t know where things are. But if you’re literally on your laptop, you know where the things are. You don’t really have that problem. So when you bring that problem space to your laptop, I think you’re actually making things worse. I’ve seen people depend on Kubernetes service discovery for the app to work. Meaning, they just assume they can call a thing by name and they don’t support IPs, and ports. They don’t support anything, because they say, “Oh! No. No. No. You’ll always be running into Kubernetes.” You know what’s going to happen? In 5 or 10 years, we’re going to be talking like, “Oh my God! Do you remember when you used to use Kubernetes? Man! That legacy thing. I built my whole career porting apps away from Kubernetes to the next thing.” The number one thing we’ll talk about is where people lean too hard on service discovery, or people who built apps that taught to config maps directly. Why are you calling the Kubernetes API from your app? That’s not a good design. I think we got to be careful coupling ourselves too much to the infrastructure. [00:27:58] MG: It’s a fair point too. Two answers from my end, to your question. So one is I just build an appliance, which basically priced to bring an AWS Lambda experience to the Vsphere ecosystem. Because we don’t – Or actually my approach is that I don’t want any ops people who needs to do some one-off things, like connect this guy to another guy. I don’t want him to learn Kubernetes for that. It should be as simple as writing a function. So for that appliance, we had to decide how do we build it? Because it should be scalable. We might have some function as a service component running on there. So we looked around and we decided to put it on Kubernetes. So build the appliance as a traditional VM using Kubernetes on top. For me as a developer, it gave me a lot of capabilities, like self-healing, the self-healing capabilities. But it’s also a fair point that you wrote, Kelsey, about how much do we depend or write our applications being depend on those auxiliary features from Kubernetes? Like self-healing, restarts, for example. [00:28:55] KH: Well, in your case, you’re building a platform. I would hate for you to tell me that you rebuilt a Kubernetes-like thing just for that appliance. In your case, it’s a great use case. I think the problem that we have as platform builders is what happens when things start leaking up to the user? You tell a user all they have to care about is functions. Then they get some error saying, “Oh! There’s some Kubernetes security context that doesn’t work.” I’m like, “What the hell is Kubernetes?” That leakage is the problem, and I think that’s the part where we have to be careful, and it will take time, but we don’t start leaking the underlying platform making the original goal untrue. [00:29:31] MG: The point is where I wanted to throw this question back was now these functions being written as simple scripts, whatever, and the operators put in. They run on Kubernetes. Now, the operators don’t know that it runs in Kubernetes. But going back to your question, when should we not use Kubernetes. Is it me writing in a higher level abstraction like a function? Not using Kubernetes in first sense, because I don’t know actually I’m using it. But on the covers, I’m still using it. So it’s kind of an answer and not an answer to your question because – [00:29:58] KH: I’ve seen these single node appliances. There’s only one node, right? They’re only there to provide like email at a grocery store. You don’t have a distributed system. Now, what people want is the Kubernetes API, the way it deploys things, the way it swaps out a running container for the next one. We want that Kubernetes API. Today, the only way to get it is by essentially bringing up a whole Kubernetes cluster. I think the K3S project is trying to simplify that by re-implementing Kubernetes. No etcd, SQLite instead. A single binary that has everything. So I think when we start to say what is Kubernetes, there’s the implementation, which is a big distributed system. Then there’s the API. I think what’s going to happen is if you want the Kubernetes API, you’re going to have so many more choices on the implementation that makes better sense for the target platform. So if you’re building an appliance, you’re going to look at K3S. If you’re a cloud provider, you’re going to probably look something like what we see on GitHub, right? You’re going to modify and integrate it into your cloud platform. [00:31:00] BL: Of maybe what happened with Kubernetes over the next few years is what happened with the Linux API, or the API. Firecracker and gVisor did this, and WSL did this. We can basically swap out Linux from the backend because we can just get on with the calls. Maybe that will happen with Kubernetes as well. So maybe Kubernetes will become a standard where Kubernetes standard and Kubernetes implementation that we have right now. I don’t even know about that one. [00:31:30] KH: We’re starting to see it, right? When you say here is my pod, and we can just look at Fargate for EKS as an example. When you give them a pod, their implementation is definitely different than what most people are thinking about running these days, right? One pod per VM. Not using Virtual Kube. So they’ve taken that pod spec and tried to uphold its means. But the problem with that, you get leaks. For example, they don’t allow you to bind to a host 4. Well, the pod spec says you can bind to a host 4. Their implementation doesn’t allow you to do it, and we see the same problem with gVisor. It doesn’t implement all the system calls. You couldn’t run the Docker daemon on top of gVisor. It wouldn’t work. So I think as long as we don’t leak, because when we leak, then we start breaking stuff. [00:32:17] BL: So we’re doing the same thing with Project Pacific here at VMware, where this concept of a pod is actually a virtual machines that loops in like a tenth of a second. It’s pretty crazy how they’ve been able to figure that out. If we can get this right, that’s huge for us. That means we can move out of our appliance and we can create better things that actually work. I’m VMware specific. I’m on AWS and I want this name space. I can use Fargate and EKS. That’s actually a great idea. [00:32:45] MG: I remember this presentation, Kelsey, that you gave. I think two or three years ago. It might be three years, where you took the Kubernetes architecture and you removed the boxes and the only thing remaining was the API server. This is where it clicked to me as like, “This is right,” because I was focused on the scheduler. I wanted to understand the scheduler. But then you zoomed out or your stripped off all these pieces and the only thing remaining was the API server. This is where it clicked to me. It’s like [inaudible 00:33:09] or like the syscall interface. It’s basically my API to do some crazy things that I would have write on my own and assembly kind of something before I could even get started. As well the breakthrough moment for me, this specific presentation. [00:33:24] KH: I’m working on an analogy to talk about what’s happening with the Kubernetes API, and I haven’t refined it yet. But when the web came out, we had all of these HTTP verbs, put post git. We have a body. We have headers. You can extract that out of the whole web, the web browser plus the web server. If you have tracked out that one piece, the instead of building web package, we can build APIs and GraphQL, because we can reuse many of those mechanisms, and we just call that RESTful interfaces. Kubernetes is going through the same evolution, right? The first thing we built was this container orchestration tool. But if you look at the CRDs, the way we do RBAC, the way we think about the status field in a custom object, if you extract those components out, then you end up with this Kubernetes style APIs where we start to treat infrastructure not as code, but as data. That will be the restful moment for Kubernetes, right? The web, we extracted it out, then we have REST interfaces. In Kubernetes, once we extracted out, we’ll end up with this declarative way of describing maybe any system. But right now, the fine, or the perfect match is infrastructure. Infrastructure as data and using these CRDs to allow us to manipulate that data. So maybe you start with Helm, and then Helm gets piped into something like Customize. That then gets piped into a mission controller. That’s how Kubernetes actually works, and that data model to API development I think is going to be the unique thing that lasts longer then the Kubernetes container platform does. [00:34:56] CC: But if you’re talking about – Correct me if I misinterpret it, platform as data. Data to me is meant to be consumed, and I actually have been thinking since you said, “Oh, developers should not be developing apps that connect directly to Kubernetes,” or I think you said the Kubernetes API. Then I was thinking, “Wait. I’ve heard so many times people saying that that’s one great benefit of Kubernetes, that the apps have that access.” Now, if you see my confusion, please clarify it. [00:35:28] KH: Yeah. Right. I remember early on when we’re doing config maps, and a big debate about how config maps should be consumed by the average application. So one way could be let’s just make a configs map API and tell every developer that they need to import a Kubernetes library to call the API server, right? Now everybody’s app doesn’t work anymore on your laptop. So we were like, “Of course not.” What we should do is have config maps be injected into the file system. So that’s why you can actually describe a config map as a volume and say, “Take these key values from the config map and write them as normal files and inject them into the container so you can just read them from the file system. The other option also was environment variables. You can take a config map and translate them into an environment variables, and lastly, you can take those environment variables and put them into command line flags. So the whole point of that is all three of the most popular ways of configuring an app, environment variables, command line flags and files. Kubernetes molded itself into that world so that developers would never tightly couple themselves to the Kubernetes API. Now, let’s say you’re building a platform, like you’re building a workflow engine like Argo, or you’re building a network control plane like Istio. Of course, you should use a Kubernetes API. You’re building a platform on top of a platform. I would say that’s kind of the exception to the rule if you’re building a platform. But a general application that’s leveraging the platform, I really think you should stay away from the Kubernetes API directly. You shouldn’t be making sys calls directly [inaudible 00:37:04] of your runtime. The unsafe package in Go. Once you start doing that, Go can’t really help you anymore. You start pining yourself to specific threads. You’re going to be in a bad time. [00:37:15] CC: Right. Okay. I think I get it. But you can still use Kubernetes to decouple your app from the machine by using objects to generate those dependencies. [00:37:25] KH: Exactly. That was the whole benefit of Kub, and Docker even, saying, “You know what? Don’t worry too much more about C groups and namespaces. Don’t even try to do that yourself.” Because remember, there was a period of time where people were actually trying to build C groups and network namespaces into the runtime. There’s a bunch of like Ruby and Python projects that they were trying to containerize themselves within the runtime. Whoa! What are we doing? Having that second layer now with Containerd on C, we don’t have to implement that 10,000 times for every programming language. [00:37:56] DC: One of the things I want to come back to is your point that you’d made about the Kubernetes API being like one of the more attractive parts of the projects, and people needing that to kind of move forward in some of these projects, and I wonder if it’s more abstract than that. I wonder if it’s abstract enough to think about in terms of like a level triggered versus edge triggered stuff. Taking control theory, the control theory that basically makes Kubernetes such a stable project and applying that to software architecture rather than necessarily bringing the entire API with you. Perhaps, what you should take from this is the lessons that we’ve learned in developing Kubernetes and apply that to your software. [00:38:33] KH: Yeah. I have the fortunate time to spend some time with Mark Burgess. He came out with the Promise Theory, and the Promise Theory is the underpinnings of Puppet Chef, Ansible, CF Engine, and this idea that we would make promises about something and eventually convergent to that state. The problem was with Puppet Chef and Ansible, we’re basically doing this with shell scripts and Ruby. We were trying to write all of these if, and, else statements. When those didn’t work, what did you do? You made an exec statement at the bottom and then you’re like, “Oh! Just run some batch, and who knows what’s going to happen?” That early implementations of Promise Theory, we didn’t own the resource that we were making promises about. Anyone could go behind this and remove the user, or the user could have a different user ID on different systems but mean the same thing. In the Kubernetes world, we push a lot of that if, else statements into the controller. Now, we force the API not have any code. That’s the big difference. If you look at the Kubernetes API, you can’t do if statements. Terraform, you can do if statements. So you kind of fall into the imperative trap at the worst moments when you’re doing dry runs or something like that. It does a really good of it. Don’t get me wrong. So the Kubernetes API says, “You know what? We’re going to go all-in on this idea.” You have to change the controller first and then update the API. There is no escape patches in the API. So it forces a set of discipline that I think gets us closer to the promises, because we know that the controller owns everything. There’s no way to escape in the API itself. [00:40:07] DC: Exactly. That’s exactly what I was pushing for. [00:40:09] MG: I have a somewhat related question and I’m just not sure how to frame it correctly. So yesterday I saw a good talk by someone talking about protocols, like they somewhat forgotten power of protocols in the world of APIs. We got Swagger. We got API definitions. But he made the very easy point of if I give you an open, a close and a write and read method, or an API, you’d still don’t know how to call them in sequence and which one to call it off. This is same for [inaudible 00:40:36] library if you look at that. So I always have to force myself, “Should I do anything [inaudible 00:40:40] or I’m not leaking some stuff.” So I look it up. Versus on protocols, if you look at the RFC definitions, they are very, very precise and very plainly outlined of what you should do, how you should behave, how you should communicate between these systems. This is more of a communication and less about the actual implementation of an API. I still have to go through that talk again, and I’m going to put it in the show notes. But this kind of opened my mind again a little bit to think more about communication between systems and contracts and promises, as you said, Carlisia. Because we make so many assumptions in our code, especially as we have to write a lot of stuff very quickly, which I think will make things brittle overtime. [00:41:21] KH: So the gift and the curse of Kubernetes that it tries to do both all the time. For some things like a pod or a deployment, we all feel that. If I give any Kubernetes cluster a deployment object, I’m going to get back out running pod. This is what we all believe. But the thing is it may not necessarily run on the same kernel. It may not run on the same OS version. It may not even run on the same type of infrastructure, right? This is where I think Kubernetes ends up leaking some of those protocol promises. A deployment gets you a set of running pods. But then we dropdown to a point where you can actually do your own API and build your own protocol. I think you’re right. Istio is a protocol for thinking about service mesh, whereas Kubernetes provides the API for building such a protocol. [00:42:03] MG: Yeah, good point. [inaudible 00:42:04]. [00:42:04] DC: On the Fargate stuff, I thought was a really interesting article, or actually, an interesting project by [inaudible 00:42:10], and I want to give him a shout out on this, because I thought that was really interesting. He wrote an admission controller that leverages autoscaler, node affinity and pod affinity to effectively do the same thing so that whenever there is a new pod created, it will spin up a new machine and associate only that pod with that machine. I was like, “What a fascinating project.” But also just seeing this come up from like the whole Fargate ECS stuff. I was like – [00:42:34] KH: I think that’s the thread that virtual kubelet is pulling on, right? This idea that you can simplify autoscalling if you remove that layer, right? Because right now we’re trying to do this musical chairs dance, right? Like in a cloud. Imagine if someone gave you the hypervisor and told you you’re responsible for attaching hypervisor workers and the VMs. It would be a nightmare. We’re going to be talking about autoscalling the way we do in the cloud. I think Kubernetes moving into a world where a one pod per resource envelope. Today we call them VMs, but I think at some point we’re going to drop the VM and we would just call it a resource envelope. VMs, this is the way we think about that, Firecrackers. Like, “Hey, does it really need to be a complete VM?” Firecracker is saying, “No. It doesn’t. It just needs to be a resource envelope that allows you to run their particular workload.” [00:43:20] DC: Yeah. Same thing we’re doing here. It’s just enough VM to get you to the point where you can drop those containers on to it. [00:43:25] CC: Kelsey, question. Edge? Kubernetes on edge. Yes or no? [00:43:29] KH: Again, it’s just like compute on edge has been a topic for discussion forever. Problem is when some people say compute on edge, they mean like go buy some servers from Dell and put it in some building somewhere close to your property as you can. But then you have to go build the APIs to deploy it to that edge. What people want, and I don’t know how far off it is, but Kubernetes has set the bar so high that the Kubernetes API comes with a way to low balance, attach storage, all of these things by just writing a few YAML files. What I hear people saying is I want that close to my data center or store as possible. When you say Kubernetes on the edge, that’s what they’re saying, is like, “But we currently have one at edge. It’s not enough.” We’ve been providing edge for a very longtime. Open stack was – Remember open stack? Oh! We’re going to do open stack on the edge. But now you’re a pseudo cloud provider without the APIs. I think what Kubernetes is bringing to the table is that we have to have a default low balancer. We have to have a default block store. We have to have a default everything and on or for to mean Kubernetes like it does today centralized. [00:44:31] BL: Well, Doors have been doing this forever in some form or another. 20 years ago I worked for a Duty Free place, and literally traveled all over the world replacing point of sale. You might think of point of sales as a cash register. There was a computer in the back and it was RS-232 links from the cash register to the computer in the back. Then there was dial-up, or [inaudible 00:44:53] line to our central thing. We’ve been doing edge for a long time, but now we can do edge. The central facility can actually manage the compute infrastructure. All they care about is basically CPU and memory and network storage now, and it’s a lot more flexible. The surety is long, but I think we’re going to do it. It’s going to happen, and I think we’re almost right – People are definitely experimenting. [00:45:16] KH: You know what, Carlisia? You know what’s interesting now though? I was watching the Reinvent announcement. Verizon is starting to allow these edge components to leverage 5G for the last mile, and that’s something game-changer, because most people are very skeptical about 5G being able to provide the same coverage as 4G because of the wavelength and point-to-point, all of these things. But for edge, this thing is a game-changer. Higher bandwidth, but shorter distance. This is exactly what edge want, right? Now you don’t have to dig up the ground and run fiber from point-to-point. So if you could buy in these Kubernetes APIs, plus concepts like 5G, and get in that closer to people, yeah, I think that’s going to change the way we think about regions and zones. That kind of goes away. We’re going to move closer to CDNs, like Cloudflare has been experimenting with their worker technology. [00:46:09] DC: On the edge stuff, I think that there’s also an interesting dichotomy happening, right? There’s a definition of edge that we referred to, which is storage stuff and one that you’re alluding to, which is that there may be like some way of actually having some edge capability and a point of presence in a 5G tower or some point with that. In some cases, edge means data gravity. You’re actually taking a bunch of data from sensors and you’re trying to store it in a place where you don’t have to pay the cost of moving all of the data form one point to another where you can actually centralize compute. So in those edge cases, you’re actually willing to invest in a high-end compute to allow for the manipulation of that data where that data lake is so that you can afford to move it into some centralized location later. But I think that that whole space is so complex right now, because there are so many different definitions and so many different levels of constraints that you have to solve for under one umbrella term, which is the edge. [00:47:04] KH: I think Bryan was pulling on that with the POS stuff, right? Because instead of you going to go buy your own cash registry and gluing everything together, that whole space got so optimized that you can just buy a square terminal. Plug it on some Wi-Fi and then there you go, right? You now have that thing. So once we start to do this for like ML capabilities, security capabilities, I think you’re going to see that POS-like thing expand and that computer get a little bit more robust to do exactly what you’re saying, right? Keep the data local. Maybe you ship models to that thing so that it can get smarter overtime, and then upload the data from various stores overtime. [00:47:40] DC: Yup. [00:47:40] MG: One last question from my end. Switching gears a bit, if allow it. KubeCon. I left KubeCon with some mixed feelings this years. But my perspective is different, because I’m not the typical, one of the 12,000 people, because most of them were new comers actually. So I looked at them and I asked myself, “If I would be new to this huge big world of CNCF and Kubernetes and all these stuff, what would I take from that?” I would be confused. Confused like how from [inaudible 00:48:10] talks, which make it sound like it’s so complex to run all these things through the keynotes, which seems to be like just a lineup of different projects that I all have to get through and install and run. I was missing some perspective and some clarity from KubeCon this year, especially for new comers. Because I’m afraid, if we don’t retain them, attract them, and maybe make them contributors, because that’s another big problem. I’m afraid that we’ll lose our base that is using Kubernetes. [00:48:39] BL: Before Kelsey says anything, and Kelsey was a Kub contrary before I was, but I was a Kub contrary this time, and I can tell you exactly why everything is like it is. Well, fortunately and unfortunately, this cloud native community is huge now. There’s lots of money. There are lots of people. There are lots of interests. If we went back to KubeCon when it was in San Francisco years ago, or even like the first Seattle one, that was a community event. We could make the event for the community. Now, there’s community. The people who are creating the products. There’s the end users, the people who are consuming the products, and there are these big corporations and companies, people who are actually financing this whole entire thing. We actually have to balance all three of those. As a person who just wants to learn, what are you trying to learn from? Are you learning from the consumption piece? Are you learning to be a vendor? Are you learning to be a contributor? We have to think about that. At a certain point, that’s good for Kubernetes. That means that we’ve been able to do the whole chasm thing. We’ve cross over to chasm. This thing is real. It’s big. It’s going to make a lot of people a lot of money one day. But I do see the issue for the person who’s trying to come in and say, “What do I do now?” Well, unfortunately, it’s like anything else. Where do you start? Well, you got to take it all in. So you need to figure out where you want to be. I’m not going to be the person that’s going to tell you, “Well, go do a sig.” That’s not it. What I want to tell you is like anything else that we’d have to learn is real hard, whether it’s a programming language or a new technique. Figure out where you want to be and you’re going to have to do some research. Then hopefully you can contribute. I’m sure Kelsey has opinions on this as well. [00:50:19] KH: I think Brian is right. I mean, I think it’s just like a pyramid happening. A the very bottom, we’re new. We need to get everybody together in one space and it becomes more of a tradeshow, like an introductory, like a tasting, right? When you’re hungry and you go and just taste everything. Then when you figure out what you want, then that will be your focus, and that’s going to change every year for a lot of people. Some people go from consumer to contributor, and they’re going to want something out of the conference. They’re only going to want to go to the contributor day and maybe some of the deep-dive technical tracks. You’re trying to serve everybody in two or three days. So you’re going to start to have like everything pulling for your attention. I think what you got to do is commit. If you go and you’re a contributor, or you’re someone what’s building on top, you may have to find a separate event to kind of go with it, right? Someone told me, “Hey, when you go to all of these conferences, make sure you don’t forget to invest in the one-on-one time.” Me going to Oslo and spending an evening with Mark Burgess and really talk about Promise Theory outside of competing for attention with the rest of the conference. When I go, I’d like to meet new people. Sit down with them. Out of the 12,000 people, I call it a win if I can meet three new people that I’ve never met before. You know what? I’ll do a follow-up hangout with them to go deeper in some areas. So I think it’s more of a catch all. It’s definitely has a tradeshow feel now, because it’s big and there’s a lot of money and opportunity involved. But at the same time, you got to know that, “Hey, you got to go and seek out.” You go to Spotif

Welcome to the fourth episode of The Podlets podcast! Today we speak to the topic of observability: what the term means, how it relates to the process of software development, and the importance of investing in a culture of observability. Each of us has a slightly different take on what exactly observability is, but roughly we agree that it is a set of tools that you can use to observe the interactions and behavior of distributed systems. Kris gives some handy analogies to help understand the growing need for observability due to rising scale and complexity. We then look at the three pillars of observability, and what each of these pillars look like in the process of testing and running a program. We also think more about how observability applies to the external problems that might arise in a system. Next up, we cover how implementing observability in teams is a cultural process, and how it is important to have a culture that accepts the necessity of failure and extensive time spend problem-solving in coding. Finally, the conversation shifts to how having a higher culture of observability can do away with the old problem of calling the dinosaur in a team who knows the code backward every time an error crops up. Note: our show changed name to The Podlets. Follow us: https://twitter.com/thepodlets Website: https://thepodlets.io Feeback: info@thepodlets.io https://github.com/vmware-tanzu/thepodlets/issues Hosts: Carlisia Campos Kris Nóva Duffie Cooley Key Points from This Episode: • Duffy and Kris’s different interpretations of observability.• Why we should bake observability into applications before a catastrophic failure.• Observability is becoming more necessary due to scale and complexity.• New infrastructures require new security systems.• Observability is a term for new ways of observing code to catch failures as they happen.• The three pillars of observability: events, metrics, and traceability.• How events, metrics, and traceability play out in an example of a WordPress blog.• Why metrics and events are necessary for observing patterns in problems.• Measuring time series data and how it is managed in a similar way to git deltas.• Why the ephemerality of events in cloud-native architectures urges a new way of thinking.• Countering exterior application issues such as a hard drive getting bumped.• The role of tracing in correlating internal and external issues with a system.• Tracing is about understanding all the bits that are being touched in a problem.• Kubernetes can be broken down into three things: compute, network and storage.• How human experience is a major factor in good observability.• The fact that embracing observability and chaos engineering is a cultural practice.• Understanding observability and chaos testing through the laser metaphor.• The more valuable the application, the higher the need for observability.• The necessity for a cultural turn toward seeing the importance of observability.• Seeming bad at debugging vs convincing teams to implement observability.• The value of having empathy for how the difficulty of software engineering.• Developing more intuition by spending time debugging.• The way automated observability tools can possibly help with developing intuition.• How observability and having common tools removes or normalizes the problem of ‘the guy’ Quotes: “Building software is very hard and complex, so if you are not making mistakes, you either are not human or you are not making enough changes.” — @carlisia [0:33:37] “Observability is just a fancy word for all of the tools to help us solve a problem.” — @krisnova [0:23:09] “You’ll be a better engineer for distributed systems if you are in a culture that is blameless, that gives you tools to experiment, and gives you tools to validate those experiments and come up with new ones.” – @mauilion [0:36:08] Links Mentioned in Today’s Episode: Velero — https://www.velero.ioCloud Native Infrastructure — https://www.amazon.com/Cloud-Native-Infrastructure-Applications-Environment/dp/1491984309 Distributed Systems Observability — https://www.goodreads.com/book/show/40182805-distributed-systems-observability Transcript: EPISODE 04 [INTRODUCTION] [0:00:08.7] ANNOUNCER: Welcome to The Podlets Podcast, a weekly show that explores Cloud Native one buzzword at a time. Each week, experts in the field will discuss and contrast distributed systems concepts, practices, tradeoffs and lessons learned to help you on your cloud native journey. This space moves fast and we shouldn’t reinvent the wheel. If you’re an engineer, operator or technically minded decision maker, this podcast is for you. [EPISODE] [0:00:40.5] CC: Hi everyone, welcome back to episode four. Today we’re going to talk about observability. I am Carlisia Campos. Today here on the show with me are Duffy Coolie. [0:00:52.7] DC: How are you doing folks? I’m Duffy Coolie, I’m staff field engineer here at VMware and looking forward to this topic. [0:00:58.9] CC: Also with us is Kris Nova. [0:01:03.2] CN: Hey everyone, I’m Kris Nova. I’m a developer advocate. I code a lot. I hang out in Kubernetes. [0:01:09.7] CC: I don’t want to be left out. I’m an engineer in the open source project called Valero that does backup and recovery for your Kubernetes applications. So, observability, why do we care? [0:01:25.6] KN: That’s the million-dollar question right there honestly. [0:01:28.0] DC: It sure is. [0:01:30.3] KN: I don’t know, I have a lot of thoughts on observability. I feel like it’s one of those words that it’s kind of like dev ops. It depends which day of the week you ask a specific person, what observability means that you’ll get a different answer. [0:01:43.3] DC: Yeah, I agree with that. It seems like it’s one of those very hot topics. I mean, it feels like people often conflate the idea of monitoring and logging of an application with the idea of observability and what that means. I’m looking forward to kind of digging into the details of that. [0:01:59.9] KN: What does observability mean to you Duffy? [0:02:04.0] DC: In my take, observability is a set of tools that can be applied to describe the way that data moves through a distributed system. Whether that data is a particular request or a particular transaction, in this way, you can actually understand the way all of these distributed parts of this system that we’re building are actually interacting. As you can imagine, things like monitoring and metrics are a part of it, right? Like being able to actually understand how the code is operating for this particular piece of the system, it’s definitely a key part of understanding how that system is operating but when we think of it as a big distributed system with terrible network demons in between and lots of other kind of stuff in between. I feel like we need kind of a higher level of context for what’s actually happening between all those things and that’s where I feel like the term observability fits. [0:02:55.5] KN: Year, I think I generally agree with that. I’ve got a few nuances that I like to pick out but I have high opinions but yeah, I mean, I hear a lot about it. I have my own ideas on what it means but like, why do we need it? [0:03:06.2] DC: I want to hear your idea to what it is, how would you define it? [0:03:08.8] KN: I mean, we have an hour to listen to me rant about observability. I mean, basically, okay, I’m an infrastructure engineer. I wrote this book Cloud Native Infrastructure, everything to me is some layer of software running on top of it, infrastructure, and observability to me is, it solves this problem of how do I gain visibility into something that I want to learn more about. I think my favorite analogy for observability, have you all ever been to like you know, like a gas station or a convenience store? On the front door, it’s like a height scale chart, it will say like four feet, five feet, six feet, seven feet. I always wondered what that was for and I remember I went home one day and I googled it and it turns out, that’s actually for, if the place ever gets robbed as the person runs out the front door, you get a free height measurement of how tall they are so you can help identify them later. To me, that’s like the perfect description of observability. It’s like cleverly sneaking and things into your system that can help you with a problem later downstream. [0:04:10.0] DC: I like that. [0:04:11.0] KN: Yeah. [0:04:12.5] CC: Observability is sort of a new term because it’s not necessarily something that I, as a developer would jump in and say, “Gee, my project doesn’t do observability. I need it.” I understand metrics and I understand logging, monitoring. Now I hear observability. Of course I read about it to talk about it on the show and I have been running into this word everywhere but why are people talking about observability? That’s my question. [0:04:45.8] KN: Yeah. Well, I think this kind of goes back to the gas station analogy again, right? What do you do when your metaphorical application gets robbed? What happens in the case of a catastrophic problem and how do you go about preparing yourself the best way possible to have an upper hand at solving that problem? [0:05:04.6] DC: Leah. [0:05:05.2] KN: Right? You know, some guy robbed a store and ran out the front door and we realized, “We have no idea how tall he is, he could be four feet tall or he could be six feet tall.” You know, we learn the hard way that maybe we should start putting markers on the door. I feel like observability is the same thing but I feel like people just kind of wake up and say like, “I need observability. I’m going to go and you know, I need all of these bells and whistles because my application of course is going to break,” and I feel like in a weird way that’s almost a cop out. We should be working on application before we work on preparing for catastrophic failure. [0:05:37.6] CC: Why didn’t I hear the word observability 10 years ago or even five years ago? I think it’s about two years ago. [0:05:45.9] DC: I’ll argue that the term observability is coming up more frequently and it’s certainly a hot topic today because of effectively context, still comes back down to context. When you’re in a situation where your application, you built like a cloud native architecture of your application, you got a bunch of different services that are intercommunicating or maybe all communicating to some put together shared resource. And things are misbehaving, you’re going to need to have the context to be able to understand how it’s breaking or at what point it’s breaking or where in the tangled web that we move is the problem actually occurring and can we measure that at that point? Traditionally like in a monolithic architecture, you're not really looking at that, maybe break over the monolith, you set up a couple of set points, you’re looking for the way particular clip pads work or if you’re on top of the game, you might like instrument your code in such a way that will emit events when particular transactions happen or particular things happen. You’re going to be looking at those events and logs and looking at metrics to figure out how this one application is performing or behaving. With observability, we have to solve that problem across many systems. [0:06:54.2] CC: That is why I put on the shownotes that it has to do something with the idea of cattle versus [inaud]. Because, I’m saying this because Duffy was asking me before we started recording why was that on the show notes. Because correct me if I’m wrong, I think you are going the direction of saying you don’t see, you don’t see the relation but the relation that I was thinking about was exactly what you just said. If I have a monolith, I’m looking at one thing, we’re both looking at one log. I can treat that as my little pad as supposed to when I have many micro services interacting. I can’t even treat anything, if I treated them as badly without that, right? Because I can’t. This is too much. The idea of the reason why observability is necessary sounds to me like that is a problem of scale and complexity. [0:07:46.2] KN: Yeah, I think that explains why we’re just now hearing it too, right? I’m trying to think of another metaphor here. I guess today’s going to be a metaphor day for me. Got it, okay. I just got back from London last week. I had gotten of the tube and I remember I came up to the surface and blinding light is in my eyes and all of a sudden it’s all sign for Scotland Yard and I was like “Wow, I remember this from all the detective sleuth stories of my childhood.” It dawned on me that the entire point of this part of London was there to help people recover from disasters. I thought about why we don’t have Scotland Yard type places anymore and it’s because we have security systems and we have like different things in place that we had to kind of learn the hard way we needed and we had to develop technology to help make that easier for us and I feel like we’re just kind of at that cusp of like our first wave of security cameras. Metaphorical security cameras with observability. We’re at that first wave of, we can instrument our code and we can start building our systems out with this idea of, “I want to be able to view it or observe it over time in the case of trying to learn more about it or debugging a problem.” [0:08:52.1] DC: Yeah. [0:08:52.9] CC: How do people handle – I’m asking this question because truly, I have yet to have this problem for my project that I need to do observability in my project. I need to make sure my project is observable. I mean, other than the bread and butter metrics and logging, that’s what we do at Valero. We don’t do anything further than that. But, I don’t know if those are the things are constitute observability but what Nova just said, my question is, when we want to look at this stuff later but we’re also talking about cattle and this things. Supposedly, your servers are ephemeral. They can go away and go back. How do we look at, how do we observe things if they have gone away? [0:09:41.1] KN: Year. That’s where we get into like, this exciting world of like, how long do we persist our data and which data do we track? And there’s, you know, a lot of schools of thought and a lot of different opinions around what the right solution here is but I think it kind of just boils down to every application and set of concerns is going to be unique and you’re just going to have to give it some thought. [0:10:01.9] CC: Should we talk more about that because that sounds very interesting. [0:10:04.4] KN: Yeah. I mean, I guess we should probably just start off with like, given a simple application, concretely, what does it mean to build out ‘observability’ for that application? [0:10:18.3] DC: There’s this idea of in a book called Distributed Systems Observability by Cindy Sridharan. I’m probably slaughtering her name but she wrote that there’s like these three pillars and the three pillars are events, metrics and traceability or tracing. Events, metrics and tracing. These are the three pillars of observability. If we were going to lay out the way that those things might apply to just any old application like a monolith then we might look at how – [0:10:45.3] KN: Can we just use like a WordPress blog, just like for an example. It’s got a data score, it’s got a thin layer of software and an API. [0:10:53.0] DC: Sure, like a WordPress app. The first thing we try to do is actually figure out what events we would like to get from the application and figure out how the instrument our application such that we’re getting useful data back as far as like the event stream. Frequently I think that – or in my experience, the things that you want to instrument in your application or it calls that your application is going to make that might represent a period of time, right? If it’s going to make a call to an external system, that’s something that you would definitely want to emit an event for if you’re trying to understand you know, where the problems are going sideways, like how long it took to actually make a query against the database in the back of a WordPress blog. It’s a great example, right? [0:11:33.5] KN: Question, you said the word instrumentation. My understanding of instrumentation is there’s kind of a bit of an art to it and you’re actually going in and you're adding like lines of code to your application that on line 13, we say ‘starting transaction’, on line 14, we make an https transaction and on the next line, we have, ‘the event is now over’ and we can sort of see that and discover that we made this https transaction and see where it broke, if it broke at all. Am I thinking about that right? [0:12:05.2] DC: I think you are but what’s interesting about that, the reporting on line 14, right? Where you're actually saying the event is over, right? I think that we end up actually measuring this in both in event stream and also in a metric, right? So that we can actually understand, of the last hundred transactions to the database, you know, are we seeing any increase in the amount of time and the process takes, like are we actually, you know, is this something that we can measure with metrics and understand, like, is this value changing over time? And then from the event perspective, that’s where we start tying in things like, contextually, in this transaction, what happened, right? In this particular event, is there some way that we can correlate the event with perhaps a trace and we’ll talk a little bit more about tracing too but like – so that we can understand, “okay, well, we have” – at 2:00 we see that there is like an incredible amount of latency being introduced when my WordPress blog tries to write to the database and it happens, every day at 2:00. I need to figure out what’s happening there. That’s a great – to even get to the point where I understand it’s happening at 2:00, I need things like metrics and need things like the events, specifically to give me that time correlation to understand, it’s at two. [0:13:18.7] KN: This is where we get into what Carlisia just asked about which was how do we solve this problem of what do we do when it goes away? In the case of our 2 PM database latency. For a lack of a better term, let’s just call it the heartbeat, the 2 PM heartbeat. What happens when the server that we’re experiencing, that latency, mysteriously goes away? Where does that data go and then you look at tools like I know Prometheus does this in elastic search, has capability to do this, but you look at how do we start managing time series data and how do we start tracking that and recording it. It’s a fascinating problem because you don’t actually record 2 PM. To this second and this degree of a second, this thing happened. You record how long it’s been since the previous event. You’re just constantly measuring deltas. It’s the same way that git works. Every time you do a get commit, you don’t’ actually write all 1,000 lines of software, you just write the one line that changed. [0:14:15.6] DC: Yeah. I think you highlight a really – I mean, both of the two of you highlighted a really good point around like this little cattle versus pets thing. This actually is something that I spent a little time with in a previous life and the challenge is that especially in systems like Kubernetes and other systems where you have – perhaps your application is running or being scaled out dynamically or scaled down dynamically based on load. You have all of these ephemeral events. You have all of this events that are from pods or from particular instances of your application that are ephemeral, they’re not going to be long lived. This highlights a kind of a new problem that we have to solve, I think, when we start thinking about cloud native architectures in that we have to be able to correlate that particular application with information that gives us the context to understand, like perhaps, this was this version of this application and these events are related to that particular version of the app. When we made a change, we saw a great reduction in the amount of time it takes to make that database call and we can correlate those new metrics based on the new version of the app and because we don’t have this like, as a long term entity that we can measure, like this isn’t like a single IP and a single piece of software that is not changing. This is any number of instances of our application deployed – like it makes you have to think about this problem fundamentally differently and how you store that data. This is where that cardinality problem that you’re highlighting comes in. [0:15:45.2] KN: Yeah. Okay, I have a question. Open question for the group. What is the scope here? I guess, to like kind of like build on our WordPress analogy. Let’s say that every day at 2 PM we notice there’s this latency and we’ve spent the last two weeks just endlessly digging through our logs and trying to come up with some sort of hypothesis of what’s going on here and we just can’t find anything. Everything we’ve talked about so far has been at the application layer of the stack. Instrumenting our application, debugging our application, making it https request. What should we do, or does observability even care if one of our hard drives is failing every day at 2 PM when the cleaning service comes by and accidentally bumps into it or something? How are we going to start learning about these deeper problems that might exist outside of our application layer which in my experience, those are the problems that really stick with you and really cause a lot of trouble. [0:16:40.0] DC: Yeah, agreed. Or somebody has like scheduled a backup of your database every day at two is what locks the database for a period of time of the backup and you're like “Wait, when did that happen? Why did that happen?” [0:16:51.4] KN: Somebody like commented out a line in the chron tab and then the server got reset and there’s like some magical bash grip somewhere on the server that goes and rewrites the chron tab. Who knows? [0:17:00.9] DC: Yeah, these are the needles in the haystack that we’ve all stumbled upon one way or another. [0:17:05.1] KN: Yeah, does observability, like, are we responsible for instrumenting like the operating system layer, the hardware layer? [0:17:14.0] CC: Isn’t that what monitoring is, like, some sort of testing from the outside, like an external testing that – of course, you only – it gives us the information after the fact, right? The server already died. My application’s already not available so now I know. [0:17:31.8] KN: Yeah. [0:17:32.4] CC: But isn’t monitoring what would address a problem like that? [0:17:36.1] DC: I think it definitely helps. I think what you're digging at Kris is correlation. Being able to actually identify at a particular period of time, what’s happening across our infrastructure, not just to our application. Being able to – and the important part is like how you even got to that time of day? Like, how do you know that this is happening, like, when you're looking for those patterns, how did you get to the point where you knew that it was happening at 2:00. If you know that it’s happening at 2:00 because of the event stream per se, right? That actually gives you a time correlation. Now you can look at, “Okay, well now I have a time and now I need to like, scoot back to like a macro level and see” – [0:18:10.7] KN: Crank it up at 2 PM. [0:18:12.9] DC: Yeah. Globally at 2:00, what’s going on in my world, right? Is there, you know – I know that these are the two entities that are responsible. I know that I have a bunch of pods that are running on this cluster, I know that I have a database that may be external to my cluster or maybe on the cluster. I need to really understand what’s happening in the world around those two entities as it correlates to that period of time to give me enough context to even troubleshoot. [0:18:39.7] CC: How do you do it though? Because I’m still going to go back to the monitoring, I mean, if I’m using external service to ping my service and my service is down, yeah, I’m going to get the timing – right, I can go back and look at the information, the log stream. Would I know that it was because of the server? No. But should I be pinging the server too? Should I ping every layer of the infrastructure? How do people do that? [0:19:05.4] KN: Yeah, that’s kind of what I was eluding to is like, where does observability at the application level stop and systems observability across the entire stack start and what tools do we have and where are the boundaries there? [0:19:20.3] DC: I think this is actually where we start talking about the third pillar that we were referring to earlier which was tracing and the ability to understand from the perspective of a particular transaction across the system. What entities that particular transaction will touch and where it spends its time across that entire transaction so my query – what I was trying to do was actually like, you know, submit a comment on a WordPress blog. If I had a way of implementing tracing through that WordPress blog, I might be able to leave myself little breadcrumbs throughout the entire set of systems and understand, “Well, at what point did I – I mean, where in this particular web transaction am I spending time?” I might see that you know, from the load balancer, I begin my trace ID and that load balancer terminates to this pod and inside of that pod, I can see where I’m spending my time. A little bit of time to kind of load the assets and stuff, a little bit of time for pushing my comments to the database and identifying what that database is, is the important part of that trace. If I understand – I need to know where that traffic is going to go next and how much time I spent in that transaction. You know, again, this is like down to that code layer. We should have some way of actually like leaving us – producing an event that may be related to a particular trace ID so that we can correlate the entire life cycle of that transaction. That unique trace ID across the entire process. [0:20:45.2] KN: Interesting. [0:20:45.7] DC: It helps us narrow the field to understand what all the bits are that are actually being touched that are part of the problem. Otherwise, we’re looking at the whole world and like obviously, that’s a much bigger haystack, right? [0:21:00.4] KN: One of the things that I’ve kind of learned about Kubernetes is as I’ve been like working with Kubernetes and explaining it to people and going down the road and talking and doing public speaking. I found that it’s very easy for users to understand Kubernetes if you break it down into three things. Compute, network and storage. What I’m kind of getting that here is like the application layer is probably going to be more relevant to the compute layer. Storage is going to be where – that’s observability. Storage is going to be more monitoring. That’s going to be what is my system doing, where am I storing my data, and then network is kind of related to tracing, what you’re looking at here, and these aren’t like necessarily one to one but it just kind of have – distribution of concerns here. Am I thinking about that? Kind of the same way you are Duffy? [0:21:42.8] DC: I think you are. I think what I’m trying to get to is like, I’m trying to identify the tools that I need to be able to understand what’s happening at 2:00 and all of the players involved in that, right? For that, I’m actually relying on tools that are pretty normal like the ability to actually monitor all the systems and understand and have like real time stamps and stuff that describes you know, that nodule server or what have you that says that you know, my backup for my SQL database started at 2:00 and ends at 2:30. I’m relying on things like an event stream to say you know, get to give me some context of time like when my problem is happening and I’m relying on things like tracing perhaps just should narrow the field so that I can actually understand what’s happening with this particular transaction and what are the systems that I should be looking at, whether that is – there’s a bunch of time being spent on the network, so what’s going on with the network at 2:00. There’s a bunch of times being spent on persisting data to a database, what’s going on with the database? You know, like, this kinda gives me I think enough context to actually get into troubleshooting mode, right? [0:22:50.2] KN: Yeah and I don’t want to take away from this lovely definition you just dropped on us but I want to take a stab at trying to summarize this. So observability, expands the whole stack. So I mean it is like if you look at the OSI reference model it is going to cover every one of those layers and all it really is just a fancy word for all of the tools to help us solve a problem. Yeah, sorry I am not trying to take away from your definition, right? I want to just simplify it so that like I can grapple it a little bit better. [0:23:22.2] CC: How about people? Does culture factor into it or it is just tools? [0:23:26.8] KN: I think culture is a huge part of it. Pesky humans. [0:23:28.0] DC: Yeah it is. [0:23:29.8] CC: Would this culture be tremendously different from what we get now usually at least with modern companies that doing modern software? [0:23:40.5] KN: I mean I definitely think like – [0:23:41.7] CC: Would it look different? [0:23:42.8] KN: Yeah, I definitely think there’s like – you can always tell. Like somebody once asked, “What is the difference between an SRE and a senior SRE?” And they were like, “patience,” And it is like you can always tell folks who have been burned because they take this stuff extremely seriously and I think that culture, like there is commodity there, like people are willing to pay for it if you can actually do a good job at going from chaotic problem, “I have no idea what is going on.” And making sense of that noise and coming up with a concrete tangible output that humans can take action on, I mean that is huge. [0:24:16.8] DC: Yeah it is. I was recently discussing the ability – in another medium. We were having a conversation around doing chaos testing and I think that this relates. And the interesting thing that came out of that for me was the idea that you know – I spent a pretty good portion of my career teaching people to troubleshoot, which is kind of weird. You know like teaching somebody to have an intuition about the way that a system works and giving them a place to even begin to troubleshoot a particularly complex problem, especially as we start building more and more complex systems, is really a weird thing to try and do. And I think that culturally, when you have embraced technologies like observability and embrace technologies like chaos engineering, I think that culturally you are actually not only enabling your developers, your operators, your SRE’s to experiment and understand how the system breaks at any point, but you are also enabling them to better understand how to troubleshoot and characterize these distributed systems that they are building. So I think that – and if that is a part, if that is a cultural norm within your company, I mean think about how many miles ahead you are of like the other people in your industry, right? You have made it through adopting these technologies. You have enabled your engineering teams, whether they’d be the people who are writing the code, whether they’d be the people who are operating the code, or the people who are just trying to keep the whole system up or provide you feedback to experiment and to develop hypothesis around how the system might break at a particular scale and to test that, right? And giving them the tools with which to actually observe this is critical, you know? Like it is amazing. [0:26:03.5] KN: Yeah, in my mind, again on my metaphor kick again, I think of the bank robber movies where they take dust and blow it into the air then all of a sudden you can see the lasers. Yeah, I’m feeling like that is what is happening here, is we’re kind of purpose – like chaos testing would just be the practice of intentionally breaking the lasers to make sure the security system works and observability is the practice of actually doing something to make those lasers visible so we can see what is going on. [0:26:31.0] CC: So because the two of you spend time with customers, or maybe Duffy more so than Nova, but definitely, I spent zero time. I spent zero. I am curious to know if someone, let’s say an SRE, wants to implement a set of practices that comprise what we are talking about and saying it is observability but they need to get a buy out from other people. How do you suggest they go about doing that? Because they might know how to do it or be willing to learn but they might need to get approval or they need to get a buy out – I am sorry, a buy in from their managers, from their colleagues, you know, there is a benefit and there is a cost. How will somebody present that? I mean we just talked about – I am sorry Nova, definitely just give us a laundry list of benefits but how do you articulate that in a way you prove those benefits are worth the cost and what are the costs? What are the tradeoffs? [0:27:36.4] KN: Yeah, I mean I think this is such a great question because in my career, I have worked the world’s most paranoid software as a service shop where I mean everything we did, we baked like emergency disaster recovery into it, every layer of everything we did, and I have also worked at shops that are like, “No, we ain’t got time for that, like hurry up and get your code moved and pushed to production,” and I mean I think there is pros and cons to each. But I think, you know, as you look at the value you have in your application, you are going to come up with some sort of way of concretely measuring that, of saying like, “This is an application that brings in 500 bucks a month,” or whatever, and depending on that cost or how much your application is worth to you is going to depend on I think how seriously you take it. For instance, a WordPress blog is going to probably not have the same level of observability with concerns than like maybe a bank routing system have. So I think as your application gets more and more valuable your need for observability and your need for these tools is going to go up more. [0:28:37.6] DC: I agree. I think from the perspective of like, how do you convince, maybe an existing engineering culture to make this jump, to introduce these ideas? I think that that is a tricky question because effectively what you are trying to do is kind of enable that cultural shift that we were talking about before, about like, what tools would set up the culture to succeed as they build out these applications and distributed systems that are going to make up or that are going to comprise the basis of what your product is, right? What tool? And getting to that, coming at that from a SRE perspective that needs air cover to be able to actually have those tough conversations with your developers and say, “Look, this is why we do it that way and this is something I can help you do but fundamentally, we need to instrument this code in a way that we can actually observe it and to understand how it is actually operating when we start before we can actually open the front door and let some of that crazy – and let the Internet in,” right? We need to be able to understand how and when the doors fall off and if we are not working with our developers who are more focused on understanding, does this function do what it says on the box? Rather than, is this function implemented in a way that might admit events or metrics, right? This is a completely different set of problems from the developer’s perspective. I have seen a couple of different implementations of how to implement this within an organization and one of them is Facebook’s idea of product engineering or I think it is called product engineering or production engineering, one of the two, and so this idea is that you might have somebody who’s similar in some ways to an SRE. Somebody who understands the infrastructure and understands how to build applications that will reside upon it and is actually embedded with your developer team to say, “You know, before we can legit sign off on this thing, here are the things that this application must have to be able to wire into to enable us to operate this app so that we can observe it and monitor it. Do all the things that we need to do.”And the great part about that is that it means that you are teaming with the developer team, you have some engineering piece that is teaming with the developer team and enabling them to understand why these tools are there and what they’re for and really – and promoting that engagement. [0:30:59.9] CC: And getting to that place is an interesting proposition isn’t it? Because, as a developer, even as a developer, I see the world moving more and more towards developer taking on the ship of the apps and knowing more, more layers of this stack, and if I am a developer and I want to implement, incorporate these practices then I need to convince some one, either a developer, or whoever is in charge of monitoring it and making sure the system is up and running right? [0:31:33.0] KN: Yeah. [0:31:34.3] CC: So one way to go about quantifying the need for that is to say, “Well over the last month we spent X amount of hours trying to find a bug in production,” and that X is a huge number. So you can bring that number and say, “This is how much the number costs in engineering hours,” but on the other hand, you don’t want to be the one to say that it takes you a 100 hours to find one little bug in production, do you? [0:32:06.2] KN: Yeah, I mean I feel like this is why agile teams are so successful because baked into how you do your work is sort of this implicit way of tracking your time and your progress. So at the end of the day, if you do spend a 100 hours of work trying to find a bug, it is sort of like, that is the team’s hours that is not your hours and you sort of get this data for free at the end of every sprint. [0:32:28.1] CC: Yeah. [0:32:28.5] DC: What you brought up is actually another cultural piece of that that I think is a problem. It has to be – I think that frequently we assume there are many – let me put this differently. I have seen companies where in the culture is somewhat damming for people who spend a lot of time trying to troubleshoot something that they wrote and that is a terrible pattern because it means that the people who are out there writing the code, who are just trying to get across the finish line with the thing that needs to be in production, right, have now this incredible pressure on them to not make a mistake and that is not okay. We are all here to make mistakes. That is what we do professionally, is make mistakes and the rest is just the gravy, you know what I mean? And so yeah, it makes me nuts that there are organizations that are like that. I feel like we really just in it and what is awesome about this is I see that narrative raising up within the ecosystem that I, you know, around cloud native architectures and other things like that, is that like, you know, you are hired to do a hard job and if we come down on you for thinking that that is a hard job then we are messing up. You are not messing up. [0:33:37.2] CC: Yeah absolutely. Building software is very hard and complex. So if you are not making mistakes, you either are not human or you are not making enough changes and in today’s world, we still have humans making software instead of robots. We are not there yet but it is a very risky proposition not to be making continuous changes because you will be left behind. [0:34:04.7] KN: Yeah, I feel like there is definitely something to be said about empathy for software engineers. It is very easy to be like, “Oh my gosh you spent a 100 hours looking at this one bug to save 20 dollars, how dare you?” but it is also a lot harder to be like, “Oh you poor thing, you had to dig through a 100 million lines of somebody else’s code in order to find this bug and it took you a 100 hours and you did all of that just to fix this one little bug, how awesome are you?” And I feel like that is where we get into the team dynamic of are we a blame-centric team? Do we try to assign blame to a certain person or do we look at this as a team’s responsibility, like this is our code and poor Carlisia over here had to go dig through this code that hasn’t been touched in 10 years,” or whatever. [0:34:54.6] CC: Another layer to that is that in my experience, I have never done anything in software or looked at any codes or brought up any system that as trivial as the end result was and especially in relation with the time spent, it has never happened that it wasn’t a huge amount of education that I got to reuse in future work. So does that make sense? [0:35:20.2] DC: Yeah and that is what I was referring to is around being able to build up the intuition around how these systems operate like if the longer, the more time you spend in the trenches working on those things, right? If you are enabled leveraging technologies like observability and chaos into the grain to troubleshoot, to come up with a hypothesis about how this would break when this happens, and test it and view the result and come up with a new hypothesis and continue down that path, you will automatically, I mean like, by your nature, build a better intuition yourself around how all of these system operate. It doesn’t matter whether it is the application you are working on or some other application, you are going to be able to build up their intuition for how to understand and characterize systems in general. You’ll be a better engineer for distributed systems if you are in a culture that is blameless that gives you tools to experiment and gives you tools to validate those experiments and come up with new ones, you know? [0:36:21.3] CC: I am going to challenge you and then I am going to agree with you so hang on, okay? So I am going to challenge you, so we are saying that observability, which actually boils down to using automated tools to do all of this work for us that we don’t have to dig in manually on a case by case basis, no that’s wrong? [0:36:43.4] DC: No, I am saying observability is a set of tools that you can use to observe the interactions and behavior of distributed systems. [0:36:53.4] CC: Okay but with automated tools right? [0:36:55.2] DC: The automation piece isn’t really I mean do you want to take this one Kris? [0:36:59.5] KN: Yeah, I mean I think like they certainly can be automated. I just don’t think that there is a hard bit of criteria that says every one needs to be automated. Like there ain’t nothing wrong with SSH-ing into a server and running a debug script or something if you are having a really bad day. [0:37:12.3] CC: Okay but let me go with my theory, just pretend it is because it will sound better. All right, so let us say, not to exclude the option to do it manually too if you want, but let’s say we have these wonderful tools that can automate a bunch of this work for us and we get to look at it from a high level. So what I am thinking is whereas before, if we didn’t have or use those tools or we are not using those tools, we have to do a lot of that work manually. We have to look at a lot more different places and I will challenge you that we develop even more intuition that way. So we are decreasing the level of intuition that we develop potentially by using the tools. Now, I am going to agree with you. It was just a rationale that I had to follow. I agree with you, it definitely helps to develop intuition but it is a better quality of intuition because now you can hold these different pieces in your hand because you are looking at it at this higher level. Because when you look at the details, you look at thing at this view – at least I am like that. It is like, “Okay, can I hold this one thing, it is big already in my head,” and then for me when I switch context and go look at something else, you know what I looked at over there, and it is hard to, really hard to keep track and really wasteful for – it is impossible to keep all of it in your mind, right? And let’s say you have to go through the whole debugging process all over again. If I don’t have notes, it will be like just the first time because I can’t possibly remember. I mean I have been in situations of having to debug different systems and okay, now third time around I am taking notes because the fourth time is just going to be so painful. So having tools that lets us look at things at a higher level I think has the additional benefit of helping us understand the system and hold it together in our heads because okay, we definitely don’t know the little details of how these are happening behind the scene. But how useful is that anyway? I’d much rather know how the whole system works together, points of failure like I can visualize, right? [0:39:30.1] DC: Yeah. [0:39:30.7] KN: I have a question for everyone. Following up on Carlisia’s how she challenged you and then agreed with you, I really want to ask this question because I think Carlisia’s answer is going to be different than Duffy’s and I think that is going to say a lot about the different ways that we are thinking about observability here and it is really fascinating if you think about it. So have either of you worked in a shop before where you had ‘the guy’? You know, that one person who just knew the code base inside and out, he had been around for forever, he was a dinosaur and whatever something went wrong you’re like, “We got to get this guy on the phone,” and he would come in and be like, “Oh it is this one line and this one thing that it would take you six months to figure out but let me just fix this really quick,” bam-bam-bam-bam and production is back online. [0:40:14.1] CC: Oh code base guy, the system admins guy, like something that is not my app but the system broke, you get that person who knew every like could take one second to figure out what the problem was. [0:40:28.9] KN: Have you seen that before though? Like that one person who just have so much tribal knowledge. [0:40:33.3] CC: Yeah, absolutely. [0:40:34.3] KN: Yeah, Duffy what about you? [0:40:36.0] DC: Absolutely. I have both been that guy and seen that guy. [0:40:38.8] CC: I have never been that person. [0:40:40.2] DC: In lots of shops. [0:40:42.4] KN: Well what I am kind of digging at here is I think observability, and I mean this in the nicest way possible to all of our folks at home who are actively playing the role of ‘the guy’, I think observability kind of makes that problem go away, right? [0:40:56.8] DC: I think it normalizes it to your point. I think that it basically gives you – I think you’re onto it. I think that I agree with you but I think that fundamentally what happens is through tooling like chaos engineering, through tooling like observability, you are normalizing what it looks like to teach anybody to be like that person, right? But that is the key takeaway is like, to Carlisia’s point she might – actually, you know Kris and I, I promise that we will approach some complex distributed systems problem fundamentally differently, right? If somebody has a broken Kubernetes cluster, Kris and I are both going to approach that same problem and we will likely both be able to solve that problem but we are going to approach it in different ways and I think that the benefit of having common tooling with which to experiment and understand and observe the behavior of these distributed systems means that, you know, we can normalize what it looks like to be a developer and have a theory about how the system is breaking or would break, and having some way of actually validating that through the use of observability and perhaps chaos engineering depending, and that means that we are turning keys over, turning the keys to the castle over. There is no more bust test. You don’t have to worry about what happens to me at the end of the day. We all have this common receptacle. [0:42:16.7] CC: You could go on vacation. [0:42:18.1] DC: Yeah. [0:42:19.0] CC: No but this is the most excellent point, I am glad you brought it up Nova because what both of you said is absolutely true. I mean, give me a better documentation and I don’t need you anymore because I can be self-sufficient. [0:42:33.2] DC: Exactly. [0:42:34.9] KN: Yeah so when we’re – [0:42:35.5] CC: If you told me to observe where things went wrong and again I go back to that what I said, more and more developers are having to take being asked, I mean some developers are proactively taking on the shift and in other cases they’d been asked to take more ownership of the whole stack and then say from the application level down the stack and, but you gave me tools to observe where things went wrong beyond my code as a developer, I am not going to call the guy. [0:43:07.3] KN: Yeah. [0:43:08.4] CC: So the level of self-sufficient – [0:43:10.3] KN: The guy doesn’t want you to call him. [0:43:12.5] CC: So it provides – and then the decision – benefit, we could say, is provide the engineer an additional level of self-sufficiency. [0:43:22.0] KN: Yeah, I mean teach someone to fish, give someone a fish. [0:43:25.1] CC: Yeah. [0:43:25.6] KN: Yeah. [0:43:26.4] DC: Exactly. All right, well that was a great conversation on observability and we talked about a bunch of different topics. This is Duffy and I had a great time in this session and thanks. [0:43:38.1] CC: Yeah, we are super glad to be here today. Thanks for listening. Come back next week. [0:43:43.4] KN: Thanks for joining everyone and I apologize again to all of our ‘guys’ at home listening. Hopefully we can help you with observability along the way to get everybody’s job a little bit easier. [0:43:53.8] CC: And I want to say you know for the girls, we know that you are all there too. That is just a joke. [0:43:59.9] KN: Oh yeah, I was totally at it for a while. Good show everyone. [0:44:05.3] DC: All right, cheers. [0:44:06.8] KN: Cheers. [END OF INTERVIEW] [0:44:08.7] ANNOUNCER: Thank you for listening to The Podlets Cloud Native Podcast. Find us on Twitter https://twitter.com/ThePodlets and on the https://thepodlets.io, where you will find transcripts and show notes. We’ll be back next week. Stay tuned by subscribing. [END]See omnystudio.com/listener for privacy information.

Aquaman, Eliza Dushku,Michael Weatherly, Star Trek Discovery, Wrestling corner, DC All access casting, most binged shows on Netflix, Supernatural, Creed 2, Jordan Peele, Sondra Locke, Rodney Kageyama, Stoney Westmoreland, Black-ish, The Goldbergs, The Kids Are Alright and Single Parents. Black-ish, Spider-Verse, gamers corner, Riverdale, The Final Table, Million dollar menu, The Last OG, Narcos Mexico, Chicago Fire, Chicago PD, Mile 22 You can follow us on twitter : @NothingsOn123 http://www.stitcher.com/ https://taylornetworkofpodcasts.com xtremecomedyct.weebly.com oldmagicgaming.com This podcast is part of the TaylorNetwork which is a home to many great podcasts all available on taylornetwork feed on Spotify, Itunes, stitcher radio and also google play

Aquaman, Eliza Dushku,Michael Weatherly, Star Trek Discovery, Wrestling corner, DC All access casting, most binged shows on Netflix, Supernatural, Creed 2, Jordan Peele, Sondra Locke, Rodney Kageyama, Stoney Westmoreland, Black-ish, The Goldbergs, The Kids Are Alright and Single Parents. Black-ish, Spider-Verse, gamers corner, Riverdale, The Final Table, Million dollar menu, The Last OG, Narcos Mexico, Chicago Fire, Chicago PD, Mile 22 You can follow us on twitter : @NothingsOn123 http://www.stitcher.com/ https://taylornetworkofpodcasts.com xtremecomedyct.weebly.com oldmagicgaming.com This podcast is part of the TaylorNetwork which is a home to many great podcasts all available on taylornetwork feed on Spotify, Itunes, stitcher radio and also google play

Listen in to learn:: ➳ The key differences between working in your own business as The Employee, The Manager, and The CEO (and how not knowing this difference is keeping you stuck in a rut!) ➳ How to make the shift from employee to CEO - even if you're just getting started and don't have a team to support you. ➳ Why the most successful CEOs prioritize stepping out of the day-to-day of their business to dream big and plan the next stage of their business growth I'm so excited that my team is joining me for this live mini-training because they will share the real behind-the-scenes of how it's not always easy for me to stay in the role of CEO! Want to join me for the CEO Retreat in DC? All the details and registration are here: http://www.rachealcook.com/ceo-retreat/ Enjoy! Racheal

It's been a busy week for trailers, folks: Tomb Raider, Jumanji, and The Punisher! But first, we have to talk about Harry Dean Stanton and what a huge impact he had on science fiction and on each of us personally. This week, Aaron Sagers and Dany Roth were joined by SYFY WIRE's newest Editors, Jordan Zakarin and Alexis Loinaz. So, what won our week? The trailers? The return of Jamie Lee Curtis and Linda Hamilton to their respective franchises? And what does Zack Snyder's potential departure from the DCEU mean for him and DC? All this and more, in the latest look at the top stories from the week in geek! NOTE: Due to radio interference some audio tracks sound a little funny. Our apologies.