Podcast appearances and mentions of cindy sridharan

A medida que se acelera la carrera por distribuir aplicaciones, se derrumba el muro entre el desarrollo y las operaciones. Cuando eso pasa, quienes están a ambos lados aprenden a trabajar juntos como nunca antes. Pero ¿qué es DevOps en realidad? Los desarrolladores invitados, que incluyen a Scott Hanselman de Microsoft y Cindy Sridharan, mejor conocida como @copyconstruct, ven a DevOps como una práctica de su lado del muro, mientras que los miembros de varios equipos de operaciones explican lo que se han esforzado por defender. Si bien todavía hay diferencias, DevOps logra que los equipos trabajen mejor que nunca. En este episodio, se analiza por qué esto es importante para los héroes de la línea de comandos del futuro.  

If you work in Kubernetes, cloud native, or any other fast-moving ecosystem, you might have found that keeping up to date with new developments can be incredibly challenging. We think this as well, and so we decided to make today’s episode a tribute to that challenge, as well as a space for sharing the best resources and practices we can think of to help manage it. Of course, there are audiences in this space who require information at various levels of depth, and fortunately the resources to suit each one exist. We get into the many different places we go in order to receive information at each part of the spectrum, such as SIG meetings on YouTube, our favorite Twitter authorities, the KubeWeekly blog, and the most helpful books out there. Another big talking point is the idea of habits or practices that can be helpful in consuming all this information, whether it be waiting for the release notes of a new version, tapping into different TLDR summaries of a topic, streaming videos, or actively writing posts as a way of clarifying and integrating newly learned concepts. In the end, there is no easy way, and passionate as you may be about staying in tune, burnout is a real possibility. So whether you’re just scratching the cloud native surface or up to your eyeballs in base code, join us for today’s conversation because you’re bound to find some use in the resources we share. Follow us: https://twitter.com/thepodlets Website: https://thepodlets.io Feeback: info@thepodlets.io https://github.com/vmware-tanzu/thepodlets/issues Hosts: Carlisia Campos Josh Rosso Duffie Cooley Olive Power Michael Gasch Key Points From This Episode: Audiences and different levels of depth that our guests/hosts follow Kubernetes at. What ‘keeping up’ means: merely following news, or actually grasping every new concept? The impossibility of truly keeping up with Kubernetes as it becomes ever more complex. Patterns used to keep up with new developments: the TWKD website, release notes, etc. Twitter’s helpful provision of information, from opinions to tech content, all in one place. How helpful Cindy Sridharan is on Twitter in her orientation toward distributed systems. The active side of keeping up such as writing posts and helping newcomers. More helpful Twitter accounts such as InfoSec. How books provide one source of deep information as opposed to the noise on Twitter. Books: Programming Kubernetes; Managing Kubernetes; Kubernetes Best Practices. Another great resource for seeing Kubernetes in action: the KubeWeeky blog. A call to watch the SIG playlists on the Kubernetes YouTube channel. Tooling: tab management and Michael’s self-built Twitter searcher. Live streaming and CTF live code demonstrations as another resource. How to keep a team updated using platforms like Slack and Zoom. The importance of organizing shared content on Slack. Challenges around not knowing the most important thing to focus on. Cognitive divergence and the temptation of escaping the isolation of coding by socializing. The idea that not seeing keeping up to date as being a personal sacrifice is dangerous. Using multiple different TLDR summaries to cement a concept in one’s brain. Incentives for users rather than developers of projects to share their experiences. The importance of showing appreciation for free resources in keeping motivation up. Quotes: “An audience I haven’t mentioned is the audience that basically just throws up their hands and walks away because there’s just too much to keep track of, right?” — @mauilion [0:05:15] “Maybe it’s because I’m lazy, I don’t know? But I wait until 1.17 drops, then I go to the release notes and really kind of ingest it because I’ve just struggled so much to kind of keep up with the day to day, ‘We merged this, we didn’t merge this,’ and so on.” — @joshrosso [0:10:18] “If you find value in being up to date with these things, just figure out – there are so many resources out there that address these different audiences and figure out what the right measure for you is. You don’t have to go deep on the code on everything.” — @mauilion [0:27:57] “Actually putting the right content in the right channel, at least from a higher level, helps me decide whether I want to like look at that channel today, and stuff that should be in the channel is not kind of in a conversation channel.” — @opowero [0:32:21] “When I see something that is going to give me the fundamentals, like I have other priorities now, I sort of always want to consume that to learn the fundamentals, because I think in the long term phase of, but then I neglect physically what I need to know to do in the moment.” — @carlisia [0:33:39] “Just do nothing, because our brain needs that. We need to not be listening, not be reading, just nothing. Just sit and look at the ceiling. Our brain needs that. Ideally, look at nature, like look outside, look at the air, go for a walk. We need that, because that recharges the brain.” — @carlisia [0:42:38] “Just consuming and keeping up, that doesn’t necessarily mean you don’t give back.” — @embano1 [0:49:32] Links Mentioned in Today’s Episode: Chris Short — https://chrisshort.net/ Last Week in Kubernetes Development — http://lwkd.info/ 1.17 Release Notes — https://kubernetes.io/docs/setup/release/notes/ Release Notes Filter Page — https://relnotes.k8s.io/ Cindy Sridharan on Twitter — https://twitter.com/copyconstruct InfoSec on Twitter — https://twitter.com/infosec?lang=en Programming Kubernetes on Amazon —https://www.amazon.com/Programming-Kubernetes-Developing-Cloud-Native-Applications/dp/1492047104 Managing Kubernetes on Amazon — https://www.amazon.com/Managing-Kubernetes-Operating-Clusters-World/dp/149203391X Brendan Burns on Twitter — https://twitter.com/brendandburns Kubernetes Best Practices on Amazon — https://www.amazon.com/Kubernetes-Best-Practices-Blueprints-Applications-ebook/dp/B081J62KLW/ KubeWeekly — https://kubeweekly.io/ Kubernetes SIG playlists on YouTube — https://www.youtube.com/channel/UCZ2bu0qutTOM0tHYa_jkIwg/playlists Twitch — https://www.twitch.tv/ Honeycomb — https://www.honeycomb.io/ KubeKon EU 2019 — https://events19.linuxfoundation.org/events/kubecon-cloudnativecon-europe-2019/ Aaron Crickenberger on LinkedIn — https://www.linkedin.com/in/spiffxp/ Stephen Augustus on LinkedIn — https://www.linkedin.com/in/stephenaugustus Office Hours — https://github.com/kubernetes/community/blob/master/events/office-hours.md Transcript: EPISODE 17[INTRODUCTION][0:00:08.7] ANNOUNCER: Welcome to The Podlets Podcast, a weekly show that explores Cloud Native one buzzword at a time. Each week, experts in the field will discuss and contrast distributed systems concepts, practices, tradeoffs and lessons learned to help you on your cloud native journey. This space moves fast and we shouldn’t reinvent the wheel. If you’re an engineer, operator or technically minded decision maker, this podcast is for you.[EPISODE][0:00:41.5] DC: Good afternoon everybody and welcome to The Podlets. In this episode, we’re going to talk about, you know, one of the more challenging things that we all have to do, just kind of keep up with cloud native and how we each approach that and what we do. Today, I have a number of cohosts with me, I have Olive Power.[0:00:56.6] OP: Hi.[0:00:57.4] DC: Carlisia Campos.[0:00:58.6] CC: Hi everybody.[0:00:59.9] DC: Josh Rosso.[0:01:01.3] JR: Hey all.[0:01:02.8] DC: And Michael.[0:01:01.1] MICHAEL: Hey, hello.[0:01:04.8] DC: This episode, we’re going to do something a little different than we normally do. In most of our episodes, we try to remain somewhat objective around the problem and the potential solutions for it, rather than prescribing a particular solution. In this episode, however, since we’re talking about how we keep up with all of the crazy things that happen in such a fast ecosystem, we’re going to probably provide quite a number of examples or resources that you yourself could use to drive and to try and keep up to date with what’s happening out there.Be sure to check out the notes after the episode is over at thepodlets.io and you will find a link to the episodes up at the top part, click down to this episode, and check out the notes. There will be tons of resources. Let’s get started.One of the things I think about that’s interesting about keeping up with something like, you know, a Kubernetes or a fast-moving project, regardless of what that project is, whether it’s Kubernetes or, you know, for a while, it was the Mesos that I was following or OpenStack or a number have been big infrastructure projects that have been very fast moving over time and I think what’s interesting is I find that there’s multiple audiences that we kind of address when we think about what it means to ‘keep up,’ right?Keeping up with something like a project is interesting because I feel like there’s an audience that it’s actually very interested in what’s happening with the design goals or the code base of the project, and there’s an audience that is very specific to wanting to understand at a high level – like, “Give me the State of the World report like every month or so just so I can understand generally what’s happening with the project, like is it thriving? Is it starting to kind of wane? Are there big projects that it’s taking on?”And then there’s like, then I feel like there’s an audience somewhere in the middle there where they really want to see people using the project and understand, and know how to learn from those people who are using it so that they can elevate their own use of that project. They’re not particularly interested in the codebase per se but they do want to understand, are they exploring this project at a depth that makes sense for themselves? What do you all think about that?[0:03:02.0] CC: I think one thing that I want to mention is that this episode, it’s not so much about on-boarding people onto Kubernetes and the Kubernetes ecosystem. We are going to have an episode soon to talk specifically about that. How you get going, like get started. I think Duffy mentioned this so we’re going to be talking about how we all keep up with things. Definitely, there are different audiences, even when we’re talking about keeping up.[0:03:32.6] JR: Yeah, I think what’s funny about your audience descriptions, Duffy, is I feel like I’ve actually slid between those audiences a bit, right? It’s funny, back in the day, Kubernetes like one-four, one-five days, I feel like I was much more like, “What’s going on in the code?” Like trying to keep track of like how things are progressing.Now my role is a lot more focused with working with customers and standing up cube and like making a production ready. I feel like I’m a lot more, kind of reactive and more interested to see like, what features have become stable and impact me, you know what I mean? I’m far less in the weeds than I used to be. It’s a super interesting thing.[0:04:08.3] OP: Yeah, I tend to – for my role, I tend to definitely fall into the number three first which is the kind of general keeping an eye on things. Like when you see like interesting articles pop up that maybe have been linked internally because somebody said, “Oh, check out this article. It’s really interesting.”Then you find that you kind of click through five or six articles similar but then you can kind of flip to that kind of like, “Oh, I’m kind of learning lots of good stuff generally about things that folks are doing.” To actually kind of having to figure out some particular solution for one of my customers and so having to go quite deep into that particular feature.You kind of go – I kind of found myself going right in and then back out, right in, going back out depending on kind of where I am on a particular day of the week. It’s kind of a bit tricky. My brain sometimes doesn’t kind of deal with that sort of deep concentration into one particular topic and then back out again. It’s not easy.I find it quite tough actually some of the time.[0:05:05.0] DC: Yeah, I think we can all agree on that. Keeping track of everything is – it’s why the episode, right? How do we even approach it? It seems – I feel like, an audience I haven’t mentioned is the audience that basically just throws up their hands and walks away because there’s just too much to keep track of, right? I feel like we are all that at some point, you know?I get that.[0:05:26.4] OP: That’s why we have Christmas holidays, right? To kind of refresh the brain.[0:05:31.4] CC: Yeah, I maybe purposefully or maybe not even – not trying to keep up because it is too much, it is a lot, and what I’m trying to do is, go deeper on the things that I already, like sort of know. And things that I am working with on a day to day basis. I only really need to know, I feel like, I really only need to know – because I’m not working directly with customers.My scope is very well defined and I feel that I really only need to know whenever there’s a new Kubernetes release. I need to know what the release is. We usually – every once in a while, we update our project to the – we bump up the Kubernetes release that we are working against and in general, yeah, it’s like if things come my way, if it’s interesting, I’ll take a look, but mostly, I feel like I work in a spiral.If I’m doing codes related to controllers and there’s a conference talk about controllers then okay, let me take a look at this to maybe learn how to design this thing better, implement in a better way if I know more about it. If I’m doing, looking at CRDs, same thing. I really like conference talks for education but that’s not so much keeping up with what’s new. Are we talking about educating ourselves with things that we don’t know about?Things that we don’t know about. Or are we talking about just news?[0:07:15.6] JR: I think it’s everything. That’s a great question. One of my other questions when we were starting to talk about this was like, what is keeping up even mean, right? I mean, does it mean, where do you find resources that are interesting that keep you interested in the project or are you looking for resources that just kind of keep you up to date with what’s changing? It’s a great question.[0:07:36.2] MICHAEL: Actually, there was some problem that I faced when I edit the links that I wanted to share in the show. I started writing the links and then I realized, “Well, most of the stuff is not keeping up with news, it’s actually understanding the technology,” because I cannot keep up.What does help me in understanding specific areas, when I need to dig into them and I think back five or four years into early days of Kubernetes, it was easy to catch up by the time because it was just about Kubernetes. Later right, it became this platform. We realized that it actually this platform thing. Then we extended Kubernetes and then we realized there are CICD-related stuff and operations and monitoring and so the whole ecosystem grew. The landscape grew so much that today, it’s impossible to keep up, right?I think I’m interested in all those patterns that you have developed over the years that help you to manage this, let’s say complexity or stream of information.[0:08:33.9] DC: Yeah, I agree. This year, I was thinking about putting up a talk with Chris Short, it was actually last year. That was about kind of on the same topic of keeping up with it. In that, I kind of did a little research into how that happens and I feel like some of the interesting stuff that came out of that was that there are certain patterns that a project might take on that make it easier or more approachable to, you know, stay in contact with what’s happening.If we take Kubernetes as an example, there are a number of websites I think that pretty much everybody here kind of follows to some degree, that helps, sort of, kind of, address those different audiences that we were talking about.One of the ones that I’ve actually been really impressed with is LWKD which stands for Last Week in Kubernetes Development, and as you can imagine, this is really kind of focused on, kind of – I wouldn’t say it’s like super deep on the development but it is watching for things that are changing, that are interesting to the people who are curating that particular blog post, right?They’ll have things in there like, you know, code freezes coming up on this date, IPV6, IPV4, duel stack is merging, they’ll have like some of the big mile markers that are happening in a particular release and where they are in time as it relates to that release. I think if that’s a great pattern and I think that – it’s a very narrow audience, right? It would really only be interesting to people who are interested in, or who are caught up in the code base, or just trying to understand like, maybe I want a preview of what the release notes might look like, so I might just like look for like a weekly kind of thing.[0:10:03.4] JR: Yeah, speaking of the release notes, right? It’s funny. I do get to look at Last Week in Kubernetes development every now and then. It’s an awesome resource but I’ve gotten to the point where the release notes are probably my most important thing for staying up to date.Maybe it’s because I’m lazy, I don’t know, but I wait till 1.17 drops, then I go to the release notes and really kind of ingest it because I’ve just struggled so much to kind of keep up with the day to day, “We merged this, we didn’t merge this,” and so on. That has been a huge help for me, you know, day to day, week to week, month to month.[0:10:37.0] MICHAEL: Well, what was also helpful just on the release notes that the new filter webpage that they put out in 1.15, starting 1.15. Have you all seen that?[0:10:44.4] JR: I’ve never heard of it.[0:10:45.4] DC: Rel dot, whatever it is. Rel dot –[0:10:47.7] MICHAEL: Yeah, if you can share it Duffy, that’s super useful. Especially like if you want to compare releases and features added and –[0:10:55.2] DC: I’ll have to dig it up as well. I don’t remember exactly what –[0:10:56.7] CC: I’m sorry, say? Which one is that again?[0:10:59.1] MICHAEL: The real notes. I’ll put it in the hackMD.[0:11:02.8] DC: Yeah relnotes.k8s.io which is an interesting one because it’s sort of like a comparison engine that allows you to kind of compare what it would have featured like how to feature relates to different versions of stuff.[0:11:14.4] CC: That’s great. I cannot encourage enough for the listeners to look at the show notes because we have a little document here that we – can I? The resources are amazing. There are so many things that I have never even heard about and sound great – is – I want to go to this whole entire list. Definitely check it out. We might not have time to mention every single thing. I don’t want people to miss on all the goodness that’s been put together.[0:11:48.7] DC: Agreed, and again, if you’re looking for those notes, you just go to the podlets.io. Click on ‘episodes’ at the right? And then look for this episode and you’ll find that it’s there.[0:11:58.0] CC: I can see that a lot of the content in those notes are like Twitter feeds. Speaking personally, I’m not sure I’m at the stage yet where I learn a lot about Twitter feeds in terms of technical content. Do you guys find that it’s more around people’s thoughts around certain things so thought-provoking things around Kubernetes and the ecosystem rather than actual technical content. I mean, that’s my experience so far.But looking at those Twitter feeds, maybe I guess I might need to follow some of those feeds. What do you all think?[0:12:30.0] MICHAEL: Do you mean the tweets are from those like learn [inaudible 0:12:32] or the person to be tweets?[0:12:35.3] OP: You’ve listed some of there, Michael, and some sort of.[0:12:37.6] MICHAEL: I just wanted to get some clarity. The reason I listed so many Twitter accounts there is because Twitter is my only kind of newsfeed if you will. I used Feedly and RSS and others before and emails and threads. But then I just got overwhelmed and I had this feeling of missing out on all of those times.That’s why I said, “Okay, let’s just use Twitter.” To your question, most of these accounts are people who have been in the Kubernetes space for very long, either running Kubernetes, developing on Kubernetes, having opinions about Kubernetes.Opinions in general on topics related to cloud native because we didn’t want to make the search just about Kubernetes. Most of these people, I really appreciate their thoughts and some of them also just a retweet things that they see which I missed somewhere else and not necessarily just opinions. I think It’s a good mix of these accounts, providing options, some guidance, and also just news that I miss out on because not being on the other channels.[0:13:35.6] OP: Yeah, I agree because sometimes you can kind of read – I tend to require a lot of sort of blog posts and sort of web posts which, you know, without realizing it can be kind of opinionated and then, you know, it’s nice to then see some Twitter feeds that kind of actually just kind of give like a couple of words, a kind of a different view which sometimes makes me think “Okay, I understand that topic from a certain article that I’ve read, it’s just really nice to hear a kind of a different take on it through Twitter.”[0:14:03.0] CC: I think some of the accounts, like fewer of the accounts – and there are a bunch of things that – there are listed accounts here that I didn’t know before so I’ll check them out. I think fewer of the accounts are providing technical content, for example, Cindy Sridharan, not pronouncing it correctly but Cindy is great, she puts out a lot of technical content and a lot of technical opinion and observations that is really good to consume. I wish I had time to just read her blog posts and Twitter alone.She’s very oriented towards distributed systems in general, so she’s not even specific just Kubernetes. Most of the accounts are very opinionated and the benefit for me is that sometimes I catch people talking about something that I didn’t even know was a thing. It’s like, “Oh, this is a thing I should know about for the work that I do,” and like Michael was saying, you know, sometimes I catch retweets that I didn’t catch before and I just – I’m not checking out places, I’m not checking – hash tagging Reddit.I rely on Twitter and the people who I follow to – if there is a blog post that sounds important, I just trust that somebody would, that I’m going to see it multiple times until like, “Okay, this is content that is related to something and I’m working on, that I want to get better at.” Then I’ll go and look at it. My sources are mainly Twitter and YouTube and it’s funny because I love blog posts but it’s like I haven’t been reading them because it takes a long time to read a blogpost.I give preference to video because I can just listen while I’m doing stuff. I sort of stopped reading blog post which is sad. I also want to start writing posts because it’s so helpful for me to engrain the things that I’m learning and hopefully it will be helpful to other people too. But in any case, go Duffy.[0:16:02.8] DC: A number of people that I follow – I have been cultivating my feed pretty carefully, trying to get a broad perspective of technical stuff that’s happening. But also I’ve been trying to develop my persona on Twitter a bit more, right? I’m actually trying to build my audience there. What’s interesting there is I’ve been trying to – to that end, what I’ve been doing is like trying to amplify voices that I think aren’t heard enough out there, right?If I see an article by somebody who is just coming into Kubernetes. or just coming into distributed systems and they’ve taken an effort to really lay out something that they found really interesting about pretty much anything, right? I’m like, “Okay, that’s pretty awesome,” and I’ll try to amplify that, right? Sometimes I even get involved or I’ll, not directly in public on Twitter but I’ll offer to help edit or help provide whatever our guidance I can provide around that sort of stuff.If I see people like having a difficult time with a particular project or something like that, I’ll reach out privately and say, “Hey, can I help you with it so you can go out there and do a great job,” you know? That is something I love to do. I think your point about like not necessarily going at Twitter for the deep knowledge stuff but more just like making sure that you have a broad enough awareness of what’s happening in different ecosystems that you’re not surprised by the things when the things change, right?A couple of other people that I follow are Akira Asuta, I can’t say enough about that person. They are amazing, they have been doing like, incredibly deep security stuff as it relates to containerization and stuff like that for quite a while. I’m always like, learning brand new things to me when following folks like that. I’ve been kind of getting more interested in InfoSec Twitter lately, learning how people kind of approach that problem.Also some of the bias arounds that which has been pretty interesting. Both the bias against people who are in InfoSec which seems weird to me. Also, how InfoSec approaches a problem, like do they put it like a learning experience or they approach it like an attack experience.It’s been kind of fascinating to get in there.[0:18:08.1] OP: You know, I kind of use Twitter as well for some of this stuff but you know, books are kind of a resource as well but in my head, kind of like at the opposite scale. You know, I obviously don’t read as many books as I read twitter feeds, right? It’s just kind of like, with Twitter, you can kind of digest the whole of the stuff and with books, it’s kind of like – I tend to be trying – because I know, I’m only going to read – like I’m only going to read maybe one/two books a year.I’ve kind of like – as I said before, blog posts seem to take up my reading time and books kind of tend to be for like on airplanes and stuff. So if – they’re just kind of two opposite resources for me but I find actually, the content of books are probably stuff that I digest a bit more because you know, it’s kind of like, I don’t know, back to the old days. It’s kind of a physical thing on hand and I can kind of read it and digest it a bit more than the kind of throwaway stuff that kind of keeps on Twitter.Because to be honest, I don’t know what’s on Twitter. Who is kind of a person to listen to or who is not or who is – I just try and form my own opinions and then, again, it kind of gets a bit overwhelming, because it’s a lot of content just streaming through continuously, whereas a book, it’s kind of like just one source of information that is kind of like a bit more personal that I can digest a bit more.[0:19:18.1] JR: Any particular book recommendation in 2019, Olive, that you found particularly interesting?[0:19:23.5] OP: I’m still reading, and it’s on the list for the episode notes actually, Programming Kubernetes. I just want to kind of get into that sort of CRD sort of mindset a bit. I think that’s kind of an area that’s interesting and an area that a lot of people will want to use in their organizations, right, because it’s going to do some of the extensibility to Kubernetes that’s just not there out of the box and everybody wants something that’s not out of the box or always in my experience.[0:19:47.4] MICHAEL: I found the Managing Kubernetes, I think was it, by – from Brendan Burns and some other folks which was just released I think in the end of last year. Super deep and that is kind of the opposite to the Programming Kubernetes, because I like that as well. That is more geared towards understanding architecture and operations.Operational concepts –[0:20:05.0] OP: They’re probably the two books I’ve read.[0:20:08.4] MICHAEL: Okay.[0:20:08.9] OP: One a year, remember?[0:20:11.4] MICHAEL: Yeah.[0:20:14.6] OP: Prolific reading.[0:20:19.6] CC: I think if you know what you need to learn about cloud native or Kubernetes, there’s amazing books out there, and if you are still exploring Kubernetes and trying to learn, I cannot recommend this book enough. If you are watching this on YouTube, you’ll see the cover. It’s called Kubernetes Best Practices because it’s about Kubernetes best practices but what they did simultaneously and maybe they didn’t even realize is just they gave a map for the entire thing.You go, “Oh, these are all the elements in Kubernetes.” Of course, it’s saying, “Okay, this is the best way to go about setting the stuff up,” and this is relatively thin but I just think that going through this book, you get really fast overview of the elements in Kubernetes. Then you can go to other books like Managing Kubernetes to go deep and understand all of the knobs and switches.[0:21:24.6] DC: I want to bring it back to the patterns that we see successful projects. Projects that you think are approachable but, you know, projects that are out there that make it easy for you to kind of stay – or easier at least to stay up to date with them, what some of those patterns are that you think are useful for projects.We’re talking about like having a couple of different entry points from kind of a weekly report mechanism, we’ve talked about the one that LWKD is, I don’t think we got to talk about KubeWeekly which is actually a weekly blog that is actually curated by a lot of the CNCF ambassadors. KubeWeekly is also broken up in different sections, so like sometimes they’ll just talk about – but they’re actually going out actively and trying to find articles of people using Kubernetes and then trying to post those.If you’re interested in understanding how people are actually out there using it, then that’s a great place to go find articles that are kind of related to that. What are some other patterns that we see that are out there that are useful for books?[0:22:27.6] DC: One that I really like. Kubernetes, for everyone listening has this notion of special interest groups, SIGs oftentimes. They’re focused on certain areas of the project. There’s some for networking and storage and life cycles of clusters and what’s amazing, I try to watch them somewhat weekly, I don’t always succeed.They’re all on YouTube and if you go to the Kubernetes project YouTube, there’s playlists for every SIG. A lot of times I’m doing work relating to life cycles of clusters. I’ll open up the cluster life cycle playlist and I’ll just watch the weekly meetings. While it doesn’t always pertain to completely to me, it lets me understand kind of where the developers and contributor’s heads are at and where they’re kind of headed with a lot of different things.There’s a link to that as well if anyone wants to check it out.[0:23:15.9] MICHAEL: Exactly, to add to that. If you don’t have the time to watch the videos, the meeting notes that these gentlemen and women put together are amazing. Usually, I just scroll through and if it’s something to triggers, I go into the episode and watch it.[0:23:28.7] OP: I almost feel like we should talk about tooling to handle all of this stuff, for example, right now, I think I have 200 tabs opened. I just started learning about some chrome extensions to manage tabs. I haven’t started really using them but I need. I don’t have a good system. My system is open a video that I’m pretty sure I want to watch and just get to that tab eventually until something happens in my chrome goes bust and I lose everything.I wanted to mention that when we say watch YouTube, some things you don’t need to sit there and actually watch, you can just listen to it and if you pay for the five bucks for YouTube premium – I don’t get a commission you people, but I’m just saying, for me, it’s so helpful. I can just turn off you know, put my phone on my pocket and keep listening to it without having to have the phone open and on the whole time. It’s very handy.It’s just like listening to a podcast. I also listen to podcasts lots of days.[0:24:35.1] MICHAEL: For tooling, since I’m just mostly on Twitter and by the time I was using or starting to use Twitter, they didn’t have this bookmark function, so I was basically abusing likes or favorites at the time, I think, to bookmark. What I realized later, my bookmarks grew, well, my likes grew.I wanted to go back and find something but that through the Twitter search was just impossible. I blew the tiny little go tool, kind of my first exercise there to just parse my likes and then use JQ because it’s all JSON to query and manipulate the stuff. I almost use it every day because I was like, that was a talk or blog post about scheduling and just correct for scheduling and the likes.I’m sure there’s a better tool or way of doing that but for me, that’s mine too. Because that’s my workflow.[0:25:27.6] DC: Both of the two blogs that you mentioned both KubeWeekly and LWKD, they both have the ability to take – you can submit stories to them. If you come across things that are interesting and you’d like to put that up on an aggregator somewhere, this is one of the ways to kind of solve that problem because at least if it gets cleared up on an aggregator, you know that you go back to the aggregator to see it, so that helps.Some other ones I’ve seen out there, I’ve seen people, I’ve seen a number of interesting startups now, starting to kind of like put out a podcast or – and I have started to see a number of people like you know, engaging with Twitch and also doing things like what we do with TJK.io which is like have sort of some kind of a weekly thing where you are just hacking on stuff live and just exploring it whether that is related to – if you think of about TJK is we’re going to do without being related necessarily to anything that we are doing at VMware just anything to do with the community but obviously if you are working for one of the small companies like Honeycomb or some other company.A smaller kind of startup, you can really just get people more aware of that because for some reason people love to watch others code. They love to understand how people go through that, what are their thought process is and I find it awesome as well. I think it is amazing to me how big a draw that is, you know?[0:26:41.1] OP: And is there lots of them out there Duffy? Is that kind of an easy searchable thing or is it like how do you know those things are going on?[0:26:48.4] DC: Oddly enough Twitter, most of the time, yeah. I mean, most of the time I see that kind of stuff happening on Twitter, like somebody will like – I will scope with this or a number of other people will say, “Hey, I am going to do a live stream during this period of time on this,” and I have actually seen a number of people doing live streams on CTFs, which are capture the flags. That one’s really been fascinating to me because it has been how do people think about approaching the security of an application.Like where do they look for weak spots and how do you determine, how do you approach that kind of a problem, which is fascinating. So yeah, I think it is important to remember that like you know, you are not the only one trying to keep up to date with all of this stuff, right? The one thing we all have said pretty consistently here is that it is a lot, and it is not just Kubernetes, right? Like any fast moving project. It could be your favorite Ruby module that has 200 contributors, right?It doesn’t matter what it is, it is a lot to keep a track of, and it represents some of that cognitive overheads that you have to think about. That is a lot to take on. Even if it is overwhelming, if you find value in being up to date with these things, just figure out – there are so many resources out there that address these different audiences and figure out what the right measure for you is. You don’t have to go deep on the code on everything.Sometimes it might be better to just try and find a source of information that gives you a high enough of a view. Maybe you are looking at the blog posts that come out on Kubernetes.io every release and you are just looking at the release notes and if you just read the release notes every release, that is already miles ahead of what I have seen a lot of folks out there when they are starting to ask me questions about how do you keep up to date.[0:28:35.9] JR: I’m curious, we have been talking a lot about keeping up as an individual. Do you all have strategies for how you help, let’s say your overall team, keep up with all the things that are going on? To give an example, Duffy, Olive and myself, at least at one point, were on the same team and we’d go out to disparate customers and see all of these different new things that they are trying to do or new projects that they are using.So we’d have to think about how do we get together and share that internally to make sure we are bringing the whole team along with what is going on in the ecosystem especially from a customer perspective. I know one of the ways that we do that is having demos and things of that nature that we share weekly. Are there other strategies that you all use with your teams to kind of share interesting information and news?[0:29:25.5] M: So what we do is mostly the way we share in our team, and we are a small team. We use Slack. We pre-filter in terms of like if there is stuff that I think is valuable for me and probably not for the whole team – obviously we are not going to share, but I think if it is related to something that the team has or to come grant and then I will share on Slack but we don’t have any formal way. I know people use some reports, weekly reports, or other platforms to distribute but we just use Slack.[0:29:53.0] DC: I think one of the things – one of the patters that we had at [inaudible 0:29:54] that I thought was actually super helpful was that we would engage a conversation. “I learned a cool new thing about whatever today,” and so we would say, “I am going to – ” and then we would start a Zoom call around that and then people could join if they wanted to, to be a part of the live discussion or not, and if they didn’t, they would still be able to see a recorded Zoom pop up in the channel later on.So even if your time zones don’t line up, like I know it is 2 AM or 3 AM or something like that for Olive right now, you can still go back to those recorded sessions and you’ll just see it on your daily Slack stuff. You would be able to see, “Oh there was a conversation about whether you should deploy Kubernetes crossed availibility zones or not. I would like to go see that,” and see what the inputs were, and so that can be helpful.[0:30:42.5] JR: Yeah, that is a super interesting observation. It is almost like remote-first teams that are used to these processes of recording everything and putting it in a Google doc. They are more equipped for that information sharing perhaps than like the water cooler conversations you’d have in the office.[0:30:58.5] OP: And on the Slack or any of the communication tool, we have different channels because we are all in lots of channels and to have channels dedicated to a particular subject is absolutely the way to go because otherwise in my previous company that seem to be kind of one main channel that all the architect used to discussed everything on and you know sometimes you join and you’re like, “What is everybody talking about?”There would be literally about a hundred messages on some sort of theme that I have never heard of. So you come away from that thinking that, “That is the main channel. Where is the bit – is there messages in the middle that I missed that were just normal discussions as opposed to in around the technical stuff,” and so it made me a bit sad, right? I would be like, “I haven’t understood something and there is a whole load of stuff on this channel that I don’t understand.”But it is the kind of central channel for everyone. So I think you end up then start looking up things that they are discussing and then realizing actually that is not really anything related to what I need to know about today or next week. It might be something for the future but I’ve got other stuff to focus on. So my point is that those communication channels for me sometimes can make me feel a little bit behind the curve or very much sort of reactive in trying to jump on things that are actually not really anything to do with me for me now and wasting my time slightly and kind of messing with my head a little bit in that like, “I really need to try and focus out stuff,” and actually putting the right content in the right channel, at least from a higher level, helps me decide whether I want to like look at that channel today, and stuff that should be in the channel is not kind of in a conversation channel. So organization of where that content is, is important to me.[0:32:37.6] CC: I am so in the same page with you Olive. That is the way my brain works as well. I want to have multiple channels, like if we are talking about Slack or any chat tool, but some people have such aversion to multiple channels. They really have a hard time dealing with too many – like testing their threshold of what they think is too many channels. So I am always mindful too, like it has to work for everybody but if it was up to me, there will be one channel per topic. So I know where to focus on.But you said something that is so interesting. How do we even just – like you were saying in the context of channel, multiple channels, and I go, if I need to pay attention to this this week as oppose to like, I don’t need to look at this until some time in the future. How do we even decide what we focus on that is useful for us in the moment versus it would be good for me to know but I don’t need to know right now.I am super bad at this. When I see something that is going to give me the fundamentals, like I have other priorities now, I sort of always want to consume that to learn the fundamentals because I think in the long term phase of, but then I neglect physically what I need to know to do in the moment and I am trying to sort of fish there and get focused on in the moment things. Anybody else have a hard time?[0:34:04.5] DC: You are not alone on that, yeah.[0:34:06.7] CC: It is terrible.[0:34:08.3] MICHAEL: Something that I wish I would do more often as like being a good citizen is like when you read a lot, probably 90% of my time is not writing but reading, maybe even more and then I share and then on Twitter, the tweet for them the most successful ones in terms of retweets or likes are the ones where I do like TLDR’s or some screen captures like too long to read. Where people don’t have the time, they might want to read the article but they don’t have the time.But if you put in like a TLDR like either a tweet or a thread on it, a lot of people would jump onto it because they can just easily capture it and they can still read the full article if they want but that is something that I learned and it is pretty – what is the right word? Helpful to my followers and the community but I just don’t do it that often unfortunately. If I am writing, summarizing, writing, I kind of remember. That is how the brain works. It is a nice side effect.[0:35:04.9] DC: I was saying, this is definitely one of those things where you can be the change you want to see if you, you know?[0:35:08.6] M: Yeah, I know.[0:35:10.0] DC: This is awesome. I would also say that what you just raised Carlisia is like a super valid point. I mean like not everybody’s brain works the same way, right? There are people who are neuro-divergent. There are people who think very linearly and they are very comfortable with that and there are people who don’t. So it is a struggle I think regardless of how your brain is wired to understand to how to prioritize the attention you will give any given subject.In some cases, your brain is not wired – your brain is almost wired against that whole idea, like you are just not set up for success when it comes to figuring out how to prioritize your attention.[0:35:49.0] CC: You hit the nail on the head. We are so set up for failure in that department because there are so many interesting conversations and you want to hop in and you want to be a part of the conversation and part of the group and socialize. Our work is so isolating to really put our heads down and just work, it can be so isolating. So it is great to participate in conversations out there even if it is for only via Twitter. I mean, obviously we are very biased towards Twitter here in this group.But I am not even this on Twitter so just keep that in mind that we are cognizant of that but in any case, I don’t know what the answer is but what I am trying always to cut down on that, those social activities that seem so appealing. I don’t know how to do that from working out.[0:36:43.9] JR: I am in the same boat. 2020, I am hoping to let more of that go and to your point, it is not that there is no value in it. It is just, I don’t know, I am not deriving the same amount of quality out of it because I am so just multiplexed all over the place, right? So we’ll see how it goes.[0:36:59.9] CC: Oh if any listener has opinions and obviously it seems that all of us are helpless in that department. Share with us, please.[0:37:12.5] DC: It is a tricky one. I think it is also interesting because I find that when we talk about things like work-life balance, we think of the idea of maybe work-life balance is that when you come at the end of the day and you go home and you don’t think about work, right? Sometimes we think that work-life balance means that you have a certain amount of time off that you can actually spend with your family and your friends or your community, what have you, and not be engaging on multiple fronts.Just be that – have that be your focus, but when it comes to things like keeping up, when it comes to things like learning or elevating your education and stuff, it seems like, for the most part, and this is just my own assumption, I am curious how you all feel about this, that we don’t – that that doesn’t enter into it, right? Your personal time is totally on the table when it comes to how do you keep up with these things. We don’t even think about it that way, right?I know I personally don’t. I definitely have to do more and cut back on the amount of time that I spend reading. I am right there with Michael on 90% of my time when my eyes are open, they are either reading or staring up on the sky while I try to think about what I am going to write next. You know one way or the other it is like that is what I am doing.[0:38:24.0] CC: Yeah.[0:38:25.1] MICHAEL: I noticed last year on my Twitter feed, more people than the years before will complain about like personal burn out. I saw a pattern, like reading those people’s tweets, I saw a pattern there. It wasn’t really like a spiral and then they realized and they shot down like deleted Twitter from their phones or any messaging and other stuff, and I think I am at the point where I also need to do that when it comes to vacation PDO, or whatever.Because I am just like, as you said Duffy, my free time is on the table when it comes to Twitter and catching up and keeping up because work-life balance in my mind is not work but what is not work for like – Kubernetes is exciting, adding in all the space, like what is not work there? I need to really get better at that because I think I might end in the same spiral of just soaking in more until I just –[0:39:17.7] CC: Yeah and like Josh said, it is not that there isn’t a value. Obviously we derive a huge value, that is why we’re on it, but you have to weigh things and what are your goals and is that the best way to your goals from where you are right now, and maybe you know, Twitter you use for a while, ramp up your knowledge, ramp up the connections because it is great for making connections, and then you step back and focus on something else, then to go on a cycle.This is how I am thinking now. It is just like what Olive was saying, you know, books are great, blog posts are great, and I absolutely agree with that. It is just that I don’t have even the time and when I have the time, I would be reading code and I would be reading things all day long, it is just really tiring for me at the end of the day to sit down and read more. I want to invest in learning how to speed read to solve that problem because I read a lot of books and blog posts. So something on my list.[0:40:22.8] DC: One of the biggest tips on speed reading I ever learned is that frequently when you read you think of saying the word and if you can get out of that habit, if you get out of the habit of saying the word even with your mouth or you just get out of that habit that will already increase the quickness of what you read.[0:40:39.5] CC: That is so interesting.[0:40:41.4] DC: Yeah, that is a trippy one.[0:40:43.1] CC: Because I think being bilingual, I totally like – that really helps me understand things, by saying the words.[0:40:52.9] DC: I think the point that we are all working around here is, there is a great panel that came out at KubeCon EU in 2019 was put on by Aaron Crickenberger, Esther McNaMara, Steven Augustus, these folks are all very high output people. I mean, they do a lot of stuff especially with regard to community and so they put on a panel that was talking about burn out and self-care and I think that it is definitely worth checking that one out.And actually also thinking about what keeping up means to you and making sure that you are measuring that against your ability to sustain, is incredibly important, right? I feel like keeping up is one of those subjects where we end up – it is almost insidious in its way to – it is a thing that we can just do all the time. We can just spend all of our time, any free moment that you have, you are sitting on the bus, you are trying to keep up with things.And because that happens so much, I feel like that is sort of one of the ways that we can feel burnt out as you are seeing today. We can feel like we did a lot of things but there was no real result to it and keep in mind that that’s part of it, right? Like when you are thinking about how we are keeping up with it, make sure that the value to your time is still something that you have some cognizance about, that you have some thought about, like is it worth it to me to just spend this six hours reading everything, right?Or would it be better for me to spend some amount of time just not reading, you know? Like doing something else, you know? Like bake a cake for crying out loud, you know?[0:42:29.5] CC: Something that a lot of times we don’t allow ourselves to do and I decided to speak for everybody I am sorry, I just do nothing, because our brain needs that. We need to not be listening, not be reading, just nothing. Just sit and look at the ceiling, our brain needs that. Ideally, look at nature, like look outside, look at the air, go for a walk. We need that, because that recharges the brain. Anyway, one thing also that I want to bring up, maybe we can mention real quick because we are coming up at the top of the hour.How do people, projects, how do we really help the users of those projects to be up to date with what they are doing?[0:43:18.4] DC: Well yeah I mean this is the different patterns that we are talking about. So I think the blog posts help. I like the idea of having blogs that are targeted towards different audiences. I like the idea of having an aggregate here for putting up a big project. I mean obviously Kubernetes is such a huge ecosystem that if you have things like KubeWeekly and I know that there are actually quite a number of things out there that try and do this.But if we can kind of agree on one like KubeWeekly I think is a pretty good one because it is actually run by the CNCF. So it kind of falls within that sort of governance as a model but having an aggregator where you can actually produce content or curate content as it relates to your project that’s helpful, and then office-hours I think is also helpful to Josh’s point. I mean office-hours and SIG hours are very similar things. I mean like office-hours there like how to developers think about what’s happening with the space.This is an opportunity for you as an end user to show up and ask questions, those sorts of patterns I think all are incredibly helpful as a project to figure out there to those things.[0:44:17.8] OP: Yeah, I know summary articles or the sort of TLDRs that Michael mentioned earlier, I think I need more of those things in my life because I do a lot of reading, because I think my brain is a bit weird in that I need to read something about five or six different times from five or six different articles for it to sort of frame in my head.So what I am trying to – like for 2020, I have almost tried to do this, is like if I think somebody knows all about this and it would save me reading those five, six, seven articles and if that person has the time, I try and sort of reach out to them and say, “Listen, have you got 20 minutes or so to explain this topic to me? Can I ask you questions about it?” It just saves me, saves my eyes reading the screen, and it just saves me time. I just need a TLDR summary of a project or a feature or something just so I can know what it is all about in my head and talk fairly sort of confidently about it.If I need to get in front and down under the weeds then there is more reading to kind of do for me maybe the coding on the technical side, but sometimes I can’t figure out what this feature sort of means and what is its use case in the real world and I have to read through lots of articles and sometimes kind of vendor specific ones and they’ve got a different slant than maybe an independent one and trying to marry those bits up my head is a bit hard for me and there is sort of wealth of information.So if you are interested in a topic and there is hundreds of articles and you start reading four or five and they are all slightly different, eventually you figure out that – you are confident and I understand what that product is about but it has taken a long time to get there and it is taken a lot of reading time. So TLDRs is like really work and I think as Josh mentioned before, we have this thing internally where we do bench demos.And that is like a TLDR and a show and tell really quickly, like, “This is what this does and this is why we need to know about it and this is why our customers needs to know about it, the end,” you know? And that’s really, really useful because that just saves a whole bunch of people a whole bunch of time figuring out A, whether they need to know about it and B, actually now understanding that product or feature at the end of the five, 10 minutes which is what they typically are. So they are very useful short snippets of information. Maybe we are back to Twitter.[0:46:37.8] JR: Similar to the idea of giving a demo Olive, you made me think of something and that is that I think one of the ways that I keep up with the space is actually through writing along with reading and I think the notion of like – and this admittedly takes up time and the whole quality of life conversation comes in but using writing to help develop your thoughts and kind of aggregate all of these crazy inputs and try to be somewhat concise, which I know I struggle with, around something I’ve learned.It’s helped me a ton and then that asset kind of becomes reusable to share with other people the thing that you wrote. So for people listening to this I guess maybe a call to action for 2020 if that is your style as well, consider starting to write yourself and becoming a resource, right? Because even if you are new to this space, you’d be amazed at just how writing from your perspective can help other people.[0:47:26.3] DC: I think another one that I actually have been impressed with lately is that a number of consumer companies like people out there like Lyft and companies like that have actually started to surface engineering blogs around how they are using technology and how they are using technology to solve things, which I think, as a service provider, as somebody who is involved in the community of Kubernetes, I find those to be incredibly valuable because I get to actually see how those things are doing.I mean at the same time, I see things like – we talked about KubeCon, which is a convention that they have every year. Obviously the project is large enough to support it but there is actually an incentive if you are a consumer of that project to go and talk about how you are using it, right? It is incentivized in that it is more likely your talk will be accepted if you are a consumer of the product than somebody building it, right? We hear from people building it all the time.I love that idea of incentivizing people who are using this thing get out there and talk about it or share their ideas about it or how they are using it, what problems did it solve for them. That is critical I think.[0:48:31.0] CC: Can I also make a suggestion – is to not so much following on the thread that we are talking about just now but kind of on the general thread of this episode. If you have resources that you do use to keep up with things, stop this recording right now and go and give them a like, give them a follow, give them a thumbs up, show somehow appreciation because what Duffy said just now, he was saying, “Oh it is so helpful when I read a blog post.”But people who are writing, they want to know that. So give them some indication, it counts a lot. It takes a lot of effort to sit down and write something or produce a podcast and if you take any, derive any benefit from it, show appreciation. It motivates people to keep doing it.[0:49:26.4] DC: Yeah, agreed.[0:49:27.9] M: I think that is a great bind maybe to close off this episode because it reiterates that just consuming and keeping up that doesn’t necessarily mean you don’t give back, right? So this is a way of giving back, which is really important to keep that flow and creativeness.[0:49:41.8] CC: I go through a lot of YouTube videos and sometimes I just play one after the other but sometimes, you know, I have been making a point of going back and liking it. Liking the ones that I like – obviously I don’t like everything. I mean things that I don’t like I don’t listen in but you know what I mean? It takes no effort but just so people know, “OK, you did a good job here.” By the way, go to iTunes and rate us. So we will know that you liked it and it will help people find our show, our podcast, and if you are watching us on YouTube, give us a like.[0:50:16.1] DC: All right, well unless anybody has any final thoughts, that is what we wanted to cover this session. So thank you all very, very much and I look forward to seeing you next week.[0:50:25.3] M: Bye-bye.[0:50:26.3] CC: Thank you so much.[0:50:27.4] OP: Bye.[0:50:28.1] JR: Bye.[END OF EPISODE][0:50:28.7] ANNOUNCER: Thank you for listening to The Podlets Cloud Native Podcast. Find us on Twitter at https://twitter.com/ThePodlets and on the http://thepodlets.io/ website, where you'll find transcripts and show notes. We'll be back next week. Stay tuned by subscribing.[END]See omnystudio.com/listener for privacy information.

Welcome to the fourth episode of The Podlets podcast! Today we speak to the topic of observability: what the term means, how it relates to the process of software development, and the importance of investing in a culture of observability. Each of us has a slightly different take on what exactly observability is, but roughly we agree that it is a set of tools that you can use to observe the interactions and behavior of distributed systems. Kris gives some handy analogies to help understand the growing need for observability due to rising scale and complexity. We then look at the three pillars of observability, and what each of these pillars look like in the process of testing and running a program. We also think more about how observability applies to the external problems that might arise in a system. Next up, we cover how implementing observability in teams is a cultural process, and how it is important to have a culture that accepts the necessity of failure and extensive time spend problem-solving in coding. Finally, the conversation shifts to how having a higher culture of observability can do away with the old problem of calling the dinosaur in a team who knows the code backward every time an error crops up. Note: our show changed name to The Podlets. Follow us: https://twitter.com/thepodlets Website: https://thepodlets.io Feeback: info@thepodlets.io https://github.com/vmware-tanzu/thepodlets/issues Hosts: Carlisia Campos Kris Nóva Duffie Cooley Key Points from This Episode: • Duffy and Kris’s different interpretations of observability.• Why we should bake observability into applications before a catastrophic failure.• Observability is becoming more necessary due to scale and complexity.• New infrastructures require new security systems.• Observability is a term for new ways of observing code to catch failures as they happen.• The three pillars of observability: events, metrics, and traceability.• How events, metrics, and traceability play out in an example of a WordPress blog.• Why metrics and events are necessary for observing patterns in problems.• Measuring time series data and how it is managed in a similar way to git deltas.• Why the ephemerality of events in cloud-native architectures urges a new way of thinking.• Countering exterior application issues such as a hard drive getting bumped.• The role of tracing in correlating internal and external issues with a system.• Tracing is about understanding all the bits that are being touched in a problem.• Kubernetes can be broken down into three things: compute, network and storage.• How human experience is a major factor in good observability.• The fact that embracing observability and chaos engineering is a cultural practice.• Understanding observability and chaos testing through the laser metaphor.• The more valuable the application, the higher the need for observability.• The necessity for a cultural turn toward seeing the importance of observability.• Seeming bad at debugging vs convincing teams to implement observability.• The value of having empathy for how the difficulty of software engineering.• Developing more intuition by spending time debugging.• The way automated observability tools can possibly help with developing intuition.• How observability and having common tools removes or normalizes the problem of ‘the guy’ Quotes: “Building software is very hard and complex, so if you are not making mistakes, you either are not human or you are not making enough changes.” — @carlisia [0:33:37] “Observability is just a fancy word for all of the tools to help us solve a problem.” — @krisnova [0:23:09] “You’ll be a better engineer for distributed systems if you are in a culture that is blameless, that gives you tools to experiment, and gives you tools to validate those experiments and come up with new ones.” – @mauilion [0:36:08] Links Mentioned in Today’s Episode: Velero — https://www.velero.ioCloud Native Infrastructure — https://www.amazon.com/Cloud-Native-Infrastructure-Applications-Environment/dp/1491984309 Distributed Systems Observability — https://www.goodreads.com/book/show/40182805-distributed-systems-observability Transcript: EPISODE 04 [INTRODUCTION] [0:00:08.7] ANNOUNCER: Welcome to The Podlets Podcast, a weekly show that explores Cloud Native one buzzword at a time. Each week, experts in the field will discuss and contrast distributed systems concepts, practices, tradeoffs and lessons learned to help you on your cloud native journey. This space moves fast and we shouldn’t reinvent the wheel. If you’re an engineer, operator or technically minded decision maker, this podcast is for you. [EPISODE] [0:00:40.5] CC: Hi everyone, welcome back to episode four. Today we’re going to talk about observability. I am Carlisia Campos. Today here on the show with me are Duffy Coolie. [0:00:52.7] DC: How are you doing folks? I’m Duffy Coolie, I’m staff field engineer here at VMware and looking forward to this topic. [0:00:58.9] CC: Also with us is Kris Nova. [0:01:03.2] CN: Hey everyone, I’m Kris Nova. I’m a developer advocate. I code a lot. I hang out in Kubernetes. [0:01:09.7] CC: I don’t want to be left out. I’m an engineer in the open source project called Valero that does backup and recovery for your Kubernetes applications. So, observability, why do we care? [0:01:25.6] KN: That’s the million-dollar question right there honestly. [0:01:28.0] DC: It sure is. [0:01:30.3] KN: I don’t know, I have a lot of thoughts on observability. I feel like it’s one of those words that it’s kind of like dev ops. It depends which day of the week you ask a specific person, what observability means that you’ll get a different answer. [0:01:43.3] DC: Yeah, I agree with that. It seems like it’s one of those very hot topics. I mean, it feels like people often conflate the idea of monitoring and logging of an application with the idea of observability and what that means. I’m looking forward to kind of digging into the details of that. [0:01:59.9] KN: What does observability mean to you Duffy? [0:02:04.0] DC: In my take, observability is a set of tools that can be applied to describe the way that data moves through a distributed system. Whether that data is a particular request or a particular transaction, in this way, you can actually understand the way all of these distributed parts of this system that we’re building are actually interacting. As you can imagine, things like monitoring and metrics are a part of it, right? Like being able to actually understand how the code is operating for this particular piece of the system, it’s definitely a key part of understanding how that system is operating but when we think of it as a big distributed system with terrible network demons in between and lots of other kind of stuff in between. I feel like we need kind of a higher level of context for what’s actually happening between all those things and that’s where I feel like the term observability fits. [0:02:55.5] KN: Year, I think I generally agree with that. I’ve got a few nuances that I like to pick out but I have high opinions but yeah, I mean, I hear a lot about it. I have my own ideas on what it means but like, why do we need it? [0:03:06.2] DC: I want to hear your idea to what it is, how would you define it? [0:03:08.8] KN: I mean, we have an hour to listen to me rant about observability. I mean, basically, okay, I’m an infrastructure engineer. I wrote this book Cloud Native Infrastructure, everything to me is some layer of software running on top of it, infrastructure, and observability to me is, it solves this problem of how do I gain visibility into something that I want to learn more about. I think my favorite analogy for observability, have you all ever been to like you know, like a gas station or a convenience store? On the front door, it’s like a height scale chart, it will say like four feet, five feet, six feet, seven feet. I always wondered what that was for and I remember I went home one day and I googled it and it turns out, that’s actually for, if the place ever gets robbed as the person runs out the front door, you get a free height measurement of how tall they are so you can help identify them later. To me, that’s like the perfect description of observability. It’s like cleverly sneaking and things into your system that can help you with a problem later downstream. [0:04:10.0] DC: I like that. [0:04:11.0] KN: Yeah. [0:04:12.5] CC: Observability is sort of a new term because it’s not necessarily something that I, as a developer would jump in and say, “Gee, my project doesn’t do observability. I need it.” I understand metrics and I understand logging, monitoring. Now I hear observability. Of course I read about it to talk about it on the show and I have been running into this word everywhere but why are people talking about observability? That’s my question. [0:04:45.8] KN: Yeah. Well, I think this kind of goes back to the gas station analogy again, right? What do you do when your metaphorical application gets robbed? What happens in the case of a catastrophic problem and how do you go about preparing yourself the best way possible to have an upper hand at solving that problem? [0:05:04.6] DC: Leah. [0:05:05.2] KN: Right? You know, some guy robbed a store and ran out the front door and we realized, “We have no idea how tall he is, he could be four feet tall or he could be six feet tall.” You know, we learn the hard way that maybe we should start putting markers on the door. I feel like observability is the same thing but I feel like people just kind of wake up and say like, “I need observability. I’m going to go and you know, I need all of these bells and whistles because my application of course is going to break,” and I feel like in a weird way that’s almost a cop out. We should be working on application before we work on preparing for catastrophic failure. [0:05:37.6] CC: Why didn’t I hear the word observability 10 years ago or even five years ago? I think it’s about two years ago. [0:05:45.9] DC: I’ll argue that the term observability is coming up more frequently and it’s certainly a hot topic today because of effectively context, still comes back down to context. When you’re in a situation where your application, you built like a cloud native architecture of your application, you got a bunch of different services that are intercommunicating or maybe all communicating to some put together shared resource. And things are misbehaving, you’re going to need to have the context to be able to understand how it’s breaking or at what point it’s breaking or where in the tangled web that we move is the problem actually occurring and can we measure that at that point? Traditionally like in a monolithic architecture, you're not really looking at that, maybe break over the monolith, you set up a couple of set points, you’re looking for the way particular clip pads work or if you’re on top of the game, you might like instrument your code in such a way that will emit events when particular transactions happen or particular things happen. You’re going to be looking at those events and logs and looking at metrics to figure out how this one application is performing or behaving. With observability, we have to solve that problem across many systems. [0:06:54.2] CC: That is why I put on the shownotes that it has to do something with the idea of cattle versus [inaud]. Because, I’m saying this because Duffy was asking me before we started recording why was that on the show notes. Because correct me if I’m wrong, I think you are going the direction of saying you don’t see, you don’t see the relation but the relation that I was thinking about was exactly what you just said. If I have a monolith, I’m looking at one thing, we’re both looking at one log. I can treat that as my little pad as supposed to when I have many micro services interacting. I can’t even treat anything, if I treated them as badly without that, right? Because I can’t. This is too much. The idea of the reason why observability is necessary sounds to me like that is a problem of scale and complexity. [0:07:46.2] KN: Yeah, I think that explains why we’re just now hearing it too, right? I’m trying to think of another metaphor here. I guess today’s going to be a metaphor day for me. Got it, okay. I just got back from London last week. I had gotten of the tube and I remember I came up to the surface and blinding light is in my eyes and all of a sudden it’s all sign for Scotland Yard and I was like “Wow, I remember this from all the detective sleuth stories of my childhood.” It dawned on me that the entire point of this part of London was there to help people recover from disasters. I thought about why we don’t have Scotland Yard type places anymore and it’s because we have security systems and we have like different things in place that we had to kind of learn the hard way we needed and we had to develop technology to help make that easier for us and I feel like we’re just kind of at that cusp of like our first wave of security cameras. Metaphorical security cameras with observability. We’re at that first wave of, we can instrument our code and we can start building our systems out with this idea of, “I want to be able to view it or observe it over time in the case of trying to learn more about it or debugging a problem.” [0:08:52.1] DC: Yeah. [0:08:52.9] CC: How do people handle – I’m asking this question because truly, I have yet to have this problem for my project that I need to do observability in my project. I need to make sure my project is observable. I mean, other than the bread and butter metrics and logging, that’s what we do at Valero. We don’t do anything further than that. But, I don’t know if those are the things are constitute observability but what Nova just said, my question is, when we want to look at this stuff later but we’re also talking about cattle and this things. Supposedly, your servers are ephemeral. They can go away and go back. How do we look at, how do we observe things if they have gone away? [0:09:41.1] KN: Year. That’s where we get into like, this exciting world of like, how long do we persist our data and which data do we track? And there’s, you know, a lot of schools of thought and a lot of different opinions around what the right solution here is but I think it kind of just boils down to every application and set of concerns is going to be unique and you’re just going to have to give it some thought. [0:10:01.9] CC: Should we talk more about that because that sounds very interesting. [0:10:04.4] KN: Yeah. I mean, I guess we should probably just start off with like, given a simple application, concretely, what does it mean to build out ‘observability’ for that application? [0:10:18.3] DC: There’s this idea of in a book called Distributed Systems Observability by Cindy Sridharan. I’m probably slaughtering her name but she wrote that there’s like these three pillars and the three pillars are events, metrics and traceability or tracing. Events, metrics and tracing. These are the three pillars of observability. If we were going to lay out the way that those things might apply to just any old application like a monolith then we might look at how – [0:10:45.3] KN: Can we just use like a WordPress blog, just like for an example. It’s got a data score, it’s got a thin layer of software and an API. [0:10:53.0] DC: Sure, like a WordPress app. The first thing we try to do is actually figure out what events we would like to get from the application and figure out how the instrument our application such that we’re getting useful data back as far as like the event stream. Frequently I think that – or in my experience, the things that you want to instrument in your application or it calls that your application is going to make that might represent a period of time, right? If it’s going to make a call to an external system, that’s something that you would definitely want to emit an event for if you’re trying to understand you know, where the problems are going sideways, like how long it took to actually make a query against the database in the back of a WordPress blog. It’s a great example, right? [0:11:33.5] KN: Question, you said the word instrumentation. My understanding of instrumentation is there’s kind of a bit of an art to it and you’re actually going in and you're adding like lines of code to your application that on line 13, we say ‘starting transaction’, on line 14, we make an https transaction and on the next line, we have, ‘the event is now over’ and we can sort of see that and discover that we made this https transaction and see where it broke, if it broke at all. Am I thinking about that right? [0:12:05.2] DC: I think you are but what’s interesting about that, the reporting on line 14, right? Where you're actually saying the event is over, right? I think that we end up actually measuring this in both in event stream and also in a metric, right? So that we can actually understand, of the last hundred transactions to the database, you know, are we seeing any increase in the amount of time and the process takes, like are we actually, you know, is this something that we can measure with metrics and understand, like, is this value changing over time? And then from the event perspective, that’s where we start tying in things like, contextually, in this transaction, what happened, right? In this particular event, is there some way that we can correlate the event with perhaps a trace and we’ll talk a little bit more about tracing too but like – so that we can understand, “okay, well, we have” – at 2:00 we see that there is like an incredible amount of latency being introduced when my WordPress blog tries to write to the database and it happens, every day at 2:00. I need to figure out what’s happening there. That’s a great – to even get to the point where I understand it’s happening at 2:00, I need things like metrics and need things like the events, specifically to give me that time correlation to understand, it’s at two. [0:13:18.7] KN: This is where we get into what Carlisia just asked about which was how do we solve this problem of what do we do when it goes away? In the case of our 2 PM database latency. For a lack of a better term, let’s just call it the heartbeat, the 2 PM heartbeat. What happens when the server that we’re experiencing, that latency, mysteriously goes away? Where does that data go and then you look at tools like I know Prometheus does this in elastic search, has capability to do this, but you look at how do we start managing time series data and how do we start tracking that and recording it. It’s a fascinating problem because you don’t actually record 2 PM. To this second and this degree of a second, this thing happened. You record how long it’s been since the previous event. You’re just constantly measuring deltas. It’s the same way that git works. Every time you do a get commit, you don’t’ actually write all 1,000 lines of software, you just write the one line that changed. [0:14:15.6] DC: Yeah. I think you highlight a really – I mean, both of the two of you highlighted a really good point around like this little cattle versus pets thing. This actually is something that I spent a little time with in a previous life and the challenge is that especially in systems like Kubernetes and other systems where you have – perhaps your application is running or being scaled out dynamically or scaled down dynamically based on load. You have all of these ephemeral events. You have all of this events that are from pods or from particular instances of your application that are ephemeral, they’re not going to be long lived. This highlights a kind of a new problem that we have to solve, I think, when we start thinking about cloud native architectures in that we have to be able to correlate that particular application with information that gives us the context to understand, like perhaps, this was this version of this application and these events are related to that particular version of the app. When we made a change, we saw a great reduction in the amount of time it takes to make that database call and we can correlate those new metrics based on the new version of the app and because we don’t have this like, as a long term entity that we can measure, like this isn’t like a single IP and a single piece of software that is not changing. This is any number of instances of our application deployed – like it makes you have to think about this problem fundamentally differently and how you store that data. This is where that cardinality problem that you’re highlighting comes in. [0:15:45.2] KN: Yeah. Okay, I have a question. Open question for the group. What is the scope here? I guess, to like kind of like build on our WordPress analogy. Let’s say that every day at 2 PM we notice there’s this latency and we’ve spent the last two weeks just endlessly digging through our logs and trying to come up with some sort of hypothesis of what’s going on here and we just can’t find anything. Everything we’ve talked about so far has been at the application layer of the stack. Instrumenting our application, debugging our application, making it https request. What should we do, or does observability even care if one of our hard drives is failing every day at 2 PM when the cleaning service comes by and accidentally bumps into it or something? How are we going to start learning about these deeper problems that might exist outside of our application layer which in my experience, those are the problems that really stick with you and really cause a lot of trouble. [0:16:40.0] DC: Yeah, agreed. Or somebody has like scheduled a backup of your database every day at two is what locks the database for a period of time of the backup and you're like “Wait, when did that happen? Why did that happen?” [0:16:51.4] KN: Somebody like commented out a line in the chron tab and then the server got reset and there’s like some magical bash grip somewhere on the server that goes and rewrites the chron tab. Who knows? [0:17:00.9] DC: Yeah, these are the needles in the haystack that we’ve all stumbled upon one way or another. [0:17:05.1] KN: Yeah, does observability, like, are we responsible for instrumenting like the operating system layer, the hardware layer? [0:17:14.0] CC: Isn’t that what monitoring is, like, some sort of testing from the outside, like an external testing that – of course, you only – it gives us the information after the fact, right? The server already died. My application’s already not available so now I know. [0:17:31.8] KN: Yeah. [0:17:32.4] CC: But isn’t monitoring what would address a problem like that? [0:17:36.1] DC: I think it definitely helps. I think what you're digging at Kris is correlation. Being able to actually identify at a particular period of time, what’s happening across our infrastructure, not just to our application. Being able to – and the important part is like how you even got to that time of day? Like, how do you know that this is happening, like, when you're looking for those patterns, how did you get to the point where you knew that it was happening at 2:00. If you know that it’s happening at 2:00 because of the event stream per se, right? That actually gives you a time correlation. Now you can look at, “Okay, well now I have a time and now I need to like, scoot back to like a macro level and see” – [0:18:10.7] KN: Crank it up at 2 PM. [0:18:12.9] DC: Yeah. Globally at 2:00, what’s going on in my world, right? Is there, you know – I know that these are the two entities that are responsible. I know that I have a bunch of pods that are running on this cluster, I know that I have a database that may be external to my cluster or maybe on the cluster. I need to really understand what’s happening in the world around those two entities as it correlates to that period of time to give me enough context to even troubleshoot. [0:18:39.7] CC: How do you do it though? Because I’m still going to go back to the monitoring, I mean, if I’m using external service to ping my service and my service is down, yeah, I’m going to get the timing – right, I can go back and look at the information, the log stream. Would I know that it was because of the server? No. But should I be pinging the server too? Should I ping every layer of the infrastructure? How do people do that? [0:19:05.4] KN: Yeah, that’s kind of what I was eluding to is like, where does observability at the application level stop and systems observability across the entire stack start and what tools do we have and where are the boundaries there? [0:19:20.3] DC: I think this is actually where we start talking about the third pillar that we were referring to earlier which was tracing and the ability to understand from the perspective of a particular transaction across the system. What entities that particular transaction will touch and where it spends its time across that entire transaction so my query – what I was trying to do was actually like, you know, submit a comment on a WordPress blog. If I had a way of implementing tracing through that WordPress blog, I might be able to leave myself little breadcrumbs throughout the entire set of systems and understand, “Well, at what point did I – I mean, where in this particular web transaction am I spending time?” I might see that you know, from the load balancer, I begin my trace ID and that load balancer terminates to this pod and inside of that pod, I can see where I’m spending my time. A little bit of time to kind of load the assets and stuff, a little bit of time for pushing my comments to the database and identifying what that database is, is the important part of that trace. If I understand – I need to know where that traffic is going to go next and how much time I spent in that transaction. You know, again, this is like down to that code layer. We should have some way of actually like leaving us – producing an event that may be related to a particular trace ID so that we can correlate the entire life cycle of that transaction. That unique trace ID across the entire process. [0:20:45.2] KN: Interesting. [0:20:45.7] DC: It helps us narrow the field to understand what all the bits are that are actually being touched that are part of the problem. Otherwise, we’re looking at the whole world and like obviously, that’s a much bigger haystack, right? [0:21:00.4] KN: One of the things that I’ve kind of learned about Kubernetes is as I’ve been like working with Kubernetes and explaining it to people and going down the road and talking and doing public speaking. I found that it’s very easy for users to understand Kubernetes if you break it down into three things. Compute, network and storage. What I’m kind of getting that here is like the application layer is probably going to be more relevant to the compute layer. Storage is going to be where – that’s observability. Storage is going to be more monitoring. That’s going to be what is my system doing, where am I storing my data, and then network is kind of related to tracing, what you’re looking at here, and these aren’t like necessarily one to one but it just kind of have – distribution of concerns here. Am I thinking about that? Kind of the same way you are Duffy? [0:21:42.8] DC: I think you are. I think what I’m trying to get to is like, I’m trying to identify the tools that I need to be able to understand what’s happening at 2:00 and all of the players involved in that, right? For that, I’m actually relying on tools that are pretty normal like the ability to actually monitor all the systems and understand and have like real time stamps and stuff that describes you know, that nodule server or what have you that says that you know, my backup for my SQL database started at 2:00 and ends at 2:30. I’m relying on things like an event stream to say you know, get to give me some context of time like when my problem is happening and I’m relying on things like tracing perhaps just should narrow the field so that I can actually understand what’s happening with this particular transaction and what are the systems that I should be looking at, whether that is – there’s a bunch of time being spent on the network, so what’s going on with the network at 2:00. There’s a bunch of times being spent on persisting data to a database, what’s going on with the database? You know, like, this kinda gives me I think enough context to actually get into troubleshooting mode, right? [0:22:50.2] KN: Yeah and I don’t want to take away from this lovely definition you just dropped on us but I want to take a stab at trying to summarize this. So observability, expands the whole stack. So I mean it is like if you look at the OSI reference model it is going to cover every one of those layers and all it really is just a fancy word for all of the tools to help us solve a problem. Yeah, sorry I am not trying to take away from your definition, right? I want to just simplify it so that like I can grapple it a little bit better. [0:23:22.2] CC: How about people? Does culture factor into it or it is just tools? [0:23:26.8] KN: I think culture is a huge part of it. Pesky humans. [0:23:28.0] DC: Yeah it is. [0:23:29.8] CC: Would this culture be tremendously different from what we get now usually at least with modern companies that doing modern software? [0:23:40.5] KN: I mean I definitely think like – [0:23:41.7] CC: Would it look different? [0:23:42.8] KN: Yeah, I definitely think there’s like – you can always tell. Like somebody once asked, “What is the difference between an SRE and a senior SRE?” And they were like, “patience,” And it is like you can always tell folks who have been burned because they take this stuff extremely seriously and I think that culture, like there is commodity there, like people are willing to pay for it if you can actually do a good job at going from chaotic problem, “I have no idea what is going on.” And making sense of that noise and coming up with a concrete tangible output that humans can take action on, I mean that is huge. [0:24:16.8] DC: Yeah it is. I was recently discussing the ability – in another medium. We were having a conversation around doing chaos testing and I think that this relates. And the interesting thing that came out of that for me was the idea that you know – I spent a pretty good portion of my career teaching people to troubleshoot, which is kind of weird. You know like teaching somebody to have an intuition about the way that a system works and giving them a place to even begin to troubleshoot a particularly complex problem, especially as we start building more and more complex systems, is really a weird thing to try and do. And I think that culturally, when you have embraced technologies like observability and embrace technologies like chaos engineering, I think that culturally you are actually not only enabling your developers, your operators, your SRE’s to experiment and understand how the system breaks at any point, but you are also enabling them to better understand how to troubleshoot and characterize these distributed systems that they are building. So I think that – and if that is a part, if that is a cultural norm within your company, I mean think about how many miles ahead you are of like the other people in your industry, right? You have made it through adopting these technologies. You have enabled your engineering teams, whether they’d be the people who are writing the code, whether they’d be the people who are operating the code, or the people who are just trying to keep the whole system up or provide you feedback to experiment and to develop hypothesis around how the system might break at a particular scale and to test that, right? And giving them the tools with which to actually observe this is critical, you know? Like it is amazing. [0:26:03.5] KN: Yeah, in my mind, again on my metaphor kick again, I think of the bank robber movies where they take dust and blow it into the air then all of a sudden you can see the lasers. Yeah, I’m feeling like that is what is happening here, is we’re kind of purpose – like chaos testing would just be the practice of intentionally breaking the lasers to make sure the security system works and observability is the practice of actually doing something to make those lasers visible so we can see what is going on. [0:26:31.0] CC: So because the two of you spend time with customers, or maybe Duffy more so than Nova, but definitely, I spent zero time. I spent zero. I am curious to know if someone, let’s say an SRE, wants to implement a set of practices that comprise what we are talking about and saying it is observability but they need to get a buy out from other people. How do you suggest they go about doing that? Because they might know how to do it or be willing to learn but they might need to get approval or they need to get a buy out – I am sorry, a buy in from their managers, from their colleagues, you know, there is a benefit and there is a cost. How will somebody present that? I mean we just talked about – I am sorry Nova, definitely just give us a laundry list of benefits but how do you articulate that in a way you prove those benefits are worth the cost and what are the costs? What are the tradeoffs? [0:27:36.4] KN: Yeah, I mean I think this is such a great question because in my career, I have worked the world’s most paranoid software as a service shop where I mean everything we did, we baked like emergency disaster recovery into it, every layer of everything we did, and I have also worked at shops that are like, “No, we ain’t got time for that, like hurry up and get your code moved and pushed to production,” and I mean I think there is pros and cons to each. But I think, you know, as you look at the value you have in your application, you are going to come up with some sort of way of concretely measuring that, of saying like, “This is an application that brings in 500 bucks a month,” or whatever, and depending on that cost or how much your application is worth to you is going to depend on I think how seriously you take it. For instance, a WordPress blog is going to probably not have the same level of observability with concerns than like maybe a bank routing system have. So I think as your application gets more and more valuable your need for observability and your need for these tools is going to go up more. [0:28:37.6] DC: I agree. I think from the perspective of like, how do you convince, maybe an existing engineering culture to make this jump, to introduce these ideas? I think that that is a tricky question because effectively what you are trying to do is kind of enable that cultural shift that we were talking about before, about like, what tools would set up the culture to succeed as they build out these applications and distributed systems that are going to make up or that are going to comprise the basis of what your product is, right? What tool? And getting to that, coming at that from a SRE perspective that needs air cover to be able to actually have those tough conversations with your developers and say, “Look, this is why we do it that way and this is something I can help you do but fundamentally, we need to instrument this code in a way that we can actually observe it and to understand how it is actually operating when we start before we can actually open the front door and let some of that crazy – and let the Internet in,” right? We need to be able to understand how and when the doors fall off and if we are not working with our developers who are more focused on understanding, does this function do what it says on the box? Rather than, is this function implemented in a way that might admit events or metrics, right? This is a completely different set of problems from the developer’s perspective. I have seen a couple of different implementations of how to implement this within an organization and one of them is Facebook’s idea of product engineering or I think it is called product engineering or production engineering, one of the two, and so this idea is that you might have somebody who’s similar in some ways to an SRE. Somebody who understands the infrastructure and understands how to build applications that will reside upon it and is actually embedded with your developer team to say, “You know, before we can legit sign off on this thing, here are the things that this application must have to be able to wire into to enable us to operate this app so that we can observe it and monitor it. Do all the things that we need to do.”And the great part about that is that it means that you are teaming with the developer team, you have some engineering piece that is teaming with the developer team and enabling them to understand why these tools are there and what they’re for and really – and promoting that engagement. [0:30:59.9] CC: And getting to that place is an interesting proposition isn’t it? Because, as a developer, even as a developer, I see the world moving more and more towards developer taking on the ship of the apps and knowing more, more layers of this stack, and if I am a developer and I want to implement, incorporate these practices then I need to convince some one, either a developer, or whoever is in charge of monitoring it and making sure the system is up and running right? [0:31:33.0] KN: Yeah. [0:31:34.3] CC: So one way to go about quantifying the need for that is to say, “Well over the last month we spent X amount of hours trying to find a bug in production,” and that X is a huge number. So you can bring that number and say, “This is how much the number costs in engineering hours,” but on the other hand, you don’t want to be the one to say that it takes you a 100 hours to find one little bug in production, do you? [0:32:06.2] KN: Yeah, I mean I feel like this is why agile teams are so successful because baked into how you do your work is sort of this implicit way of tracking your time and your progress. So at the end of the day, if you do spend a 100 hours of work trying to find a bug, it is sort of like, that is the team’s hours that is not your hours and you sort of get this data for free at the end of every sprint. [0:32:28.1] CC: Yeah. [0:32:28.5] DC: What you brought up is actually another cultural piece of that that I think is a problem. It has to be – I think that frequently we assume there are many – let me put this differently. I have seen companies where in the culture is somewhat damming for people who spend a lot of time trying to troubleshoot something that they wrote and that is a terrible pattern because it means that the people who are out there writing the code, who are just trying to get across the finish line with the thing that needs to be in production, right, have now this incredible pressure on them to not make a mistake and that is not okay. We are all here to make mistakes. That is what we do professionally, is make mistakes and the rest is just the gravy, you know what I mean? And so yeah, it makes me nuts that there are organizations that are like that. I feel like we really just in it and what is awesome about this is I see that narrative raising up within the ecosystem that I, you know, around cloud native architectures and other things like that, is that like, you know, you are hired to do a hard job and if we come down on you for thinking that that is a hard job then we are messing up. You are not messing up. [0:33:37.2] CC: Yeah absolutely. Building software is very hard and complex. So if you are not making mistakes, you either are not human or you are not making enough changes and in today’s world, we still have humans making software instead of robots. We are not there yet but it is a very risky proposition not to be making continuous changes because you will be left behind. [0:34:04.7] KN: Yeah, I feel like there is definitely something to be said about empathy for software engineers. It is very easy to be like, “Oh my gosh you spent a 100 hours looking at this one bug to save 20 dollars, how dare you?” but it is also a lot harder to be like, “Oh you poor thing, you had to dig through a 100 million lines of somebody else’s code in order to find this bug and it took you a 100 hours and you did all of that just to fix this one little bug, how awesome are you?” And I feel like that is where we get into the team dynamic of are we a blame-centric team? Do we try to assign blame to a certain person or do we look at this as a team’s responsibility, like this is our code and poor Carlisia over here had to go dig through this code that hasn’t been touched in 10 years,” or whatever. [0:34:54.6] CC: Another layer to that is that in my experience, I have never done anything in software or looked at any codes or brought up any system that as trivial as the end result was and especially in relation with the time spent, it has never happened that it wasn’t a huge amount of education that I got to reuse in future work. So does that make sense? [0:35:20.2] DC: Yeah and that is what I was referring to is around being able to build up the intuition around how these systems operate like if the longer, the more time you spend in the trenches working on those things, right? If you are enabled leveraging technologies like observability and chaos into the grain to troubleshoot, to come up with a hypothesis about how this would break when this happens, and test it and view the result and come up with a new hypothesis and continue down that path, you will automatically, I mean like, by your nature, build a better intuition yourself around how all of these system operate. It doesn’t matter whether it is the application you are working on or some other application, you are going to be able to build up their intuition for how to understand and characterize systems in general. You’ll be a better engineer for distributed systems if you are in a culture that is blameless that gives you tools to experiment and gives you tools to validate those experiments and come up with new ones, you know? [0:36:21.3] CC: I am going to challenge you and then I am going to agree with you so hang on, okay? So I am going to challenge you, so we are saying that observability, which actually boils down to using automated tools to do all of this work for us that we don’t have to dig in manually on a case by case basis, no that’s wrong? [0:36:43.4] DC: No, I am saying observability is a set of tools that you can use to observe the interactions and behavior of distributed systems. [0:36:53.4] CC: Okay but with automated tools right? [0:36:55.2] DC: The automation piece isn’t really I mean do you want to take this one Kris? [0:36:59.5] KN: Yeah, I mean I think like they certainly can be automated. I just don’t think that there is a hard bit of criteria that says every one needs to be automated. Like there ain’t nothing wrong with SSH-ing into a server and running a debug script or something if you are having a really bad day. [0:37:12.3] CC: Okay but let me go with my theory, just pretend it is because it will sound better. All right, so let us say, not to exclude the option to do it manually too if you want, but let’s say we have these wonderful tools that can automate a bunch of this work for us and we get to look at it from a high level. So what I am thinking is whereas before, if we didn’t have or use those tools or we are not using those tools, we have to do a lot of that work manually. We have to look at a lot more different places and I will challenge you that we develop even more intuition that way. So we are decreasing the level of intuition that we develop potentially by using the tools. Now, I am going to agree with you. It was just a rationale that I had to follow. I agree with you, it definitely helps to develop intuition but it is a better quality of intuition because now you can hold these different pieces in your hand because you are looking at it at this higher level. Because when you look at the details, you look at thing at this view – at least I am like that. It is like, “Okay, can I hold this one thing, it is big already in my head,” and then for me when I switch context and go look at something else, you know what I looked at over there, and it is hard to, really hard to keep track and really wasteful for – it is impossible to keep all of it in your mind, right? And let’s say you have to go through the whole debugging process all over again. If I don’t have notes, it will be like just the first time because I can’t possibly remember. I mean I have been in situations of having to debug different systems and okay, now third time around I am taking notes because the fourth time is just going to be so painful. So having tools that lets us look at things at a higher level I think has the additional benefit of helping us understand the system and hold it together in our heads because okay, we definitely don’t know the little details of how these are happening behind the scene. But how useful is that anyway? I’d much rather know how the whole system works together, points of failure like I can visualize, right? [0:39:30.1] DC: Yeah. [0:39:30.7] KN: I have a question for everyone. Following up on Carlisia’s how she challenged you and then agreed with you, I really want to ask this question because I think Carlisia’s answer is going to be different than Duffy’s and I think that is going to say a lot about the different ways that we are thinking about observability here and it is really fascinating if you think about it. So have either of you worked in a shop before where you had ‘the guy’? You know, that one person who just knew the code base inside and out, he had been around for forever, he was a dinosaur and whatever something went wrong you’re like, “We got to get this guy on the phone,” and he would come in and be like, “Oh it is this one line and this one thing that it would take you six months to figure out but let me just fix this really quick,” bam-bam-bam-bam and production is back online. [0:40:14.1] CC: Oh code base guy, the system admins guy, like something that is not my app but the system broke, you get that person who knew every like could take one second to figure out what the problem was. [0:40:28.9] KN: Have you seen that before though? Like that one person who just have so much tribal knowledge. [0:40:33.3] CC: Yeah, absolutely. [0:40:34.3] KN: Yeah, Duffy what about you? [0:40:36.0] DC: Absolutely. I have both been that guy and seen that guy. [0:40:38.8] CC: I have never been that person. [0:40:40.2] DC: In lots of shops. [0:40:42.4] KN: Well what I am kind of digging at here is I think observability, and I mean this in the nicest way possible to all of our folks at home who are actively playing the role of ‘the guy’, I think observability kind of makes that problem go away, right? [0:40:56.8] DC: I think it normalizes it to your point. I think that it basically gives you – I think you’re onto it. I think that I agree with you but I think that fundamentally what happens is through tooling like chaos engineering, through tooling like observability, you are normalizing what it looks like to teach anybody to be like that person, right? But that is the key takeaway is like, to Carlisia’s point she might – actually, you know Kris and I, I promise that we will approach some complex distributed systems problem fundamentally differently, right? If somebody has a broken Kubernetes cluster, Kris and I are both going to approach that same problem and we will likely both be able to solve that problem but we are going to approach it in different ways and I think that the benefit of having common tooling with which to experiment and understand and observe the behavior of these distributed systems means that, you know, we can normalize what it looks like to be a developer and have a theory about how the system is breaking or would break, and having some way of actually validating that through the use of observability and perhaps chaos engineering depending, and that means that we are turning keys over, turning the keys to the castle over. There is no more bust test. You don’t have to worry about what happens to me at the end of the day. We all have this common receptacle. [0:42:16.7] CC: You could go on vacation. [0:42:18.1] DC: Yeah. [0:42:19.0] CC: No but this is the most excellent point, I am glad you brought it up Nova because what both of you said is absolutely true. I mean, give me a better documentation and I don’t need you anymore because I can be self-sufficient. [0:42:33.2] DC: Exactly. [0:42:34.9] KN: Yeah so when we’re – [0:42:35.5] CC: If you told me to observe where things went wrong and again I go back to that what I said, more and more developers are having to take being asked, I mean some developers are proactively taking on the shift and in other cases they’d been asked to take more ownership of the whole stack and then say from the application level down the stack and, but you gave me tools to observe where things went wrong beyond my code as a developer, I am not going to call the guy. [0:43:07.3] KN: Yeah. [0:43:08.4] CC: So the level of self-sufficient – [0:43:10.3] KN: The guy doesn’t want you to call him. [0:43:12.5] CC: So it provides – and then the decision – benefit, we could say, is provide the engineer an additional level of self-sufficiency. [0:43:22.0] KN: Yeah, I mean teach someone to fish, give someone a fish. [0:43:25.1] CC: Yeah. [0:43:25.6] KN: Yeah. [0:43:26.4] DC: Exactly. All right, well that was a great conversation on observability and we talked about a bunch of different topics. This is Duffy and I had a great time in this session and thanks. [0:43:38.1] CC: Yeah, we are super glad to be here today. Thanks for listening. Come back next week. [0:43:43.4] KN: Thanks for joining everyone and I apologize again to all of our ‘guys’ at home listening. Hopefully we can help you with observability along the way to get everybody’s job a little bit easier. [0:43:53.8] CC: And I want to say you know for the girls, we know that you are all there too. That is just a joke. [0:43:59.9] KN: Oh yeah, I was totally at it for a while. Good show everyone. [0:44:05.3] DC: All right, cheers. [0:44:06.8] KN: Cheers. [END OF INTERVIEW] [0:44:08.7] ANNOUNCER: Thank you for listening to The Podlets Cloud Native Podcast. Find us on Twitter https://twitter.com/ThePodlets and on the https://thepodlets.io, where you will find transcripts and show notes. We’ll be back next week. Stay tuned by subscribing. [END]See omnystudio.com/listener for privacy information.

As the race to deliver applications ramps up, the wall between development and operations comes crashing down. When it does, those on both sides learn to work together like never before. But what is DevOps, really? Developer guests, including Microsoft’s Scott Hanselman and Cindy Sridharan (better known as @copyconstruct) think about DevOps as a practice from their side of the wall, while members from various operations teams explain what they’ve been working to defend. Differences remain but with DevOps, teams are working better than ever. And this episode explores why that matters for the command line heroes of tomorrow. Read Cindy Sridharan's attempt to demystify DevOps. And check out Gordon Haff's take on how to improve DevOps here.

James Turnbull ( https://jamesturnbull.net/ ) is an author of 10 books on topics like Docker, Packer, Terraform, Monitoring, … and is currently writing a book on Monitoring with Prometheus https://prometheusbook.com/ . We got to chat about what modern monitoring approaches look like, how to pull in developers to start building monitoring into their systems and how to bridge the gap between monitoring for operations vs monitoring for business. Having a monitoring expert like James that knows many tools in the space was great to validate what we at Dynatrace have been doing to solve modern monitoring problems. We learned a lot about key monitoring capabilities such as capturing data vs capturing information, providing just nice dashboards vs providing answers to known and unknown questions and making monitoring easy accessible so that monitoring can benefit both business, operations and developers.We hope you enjoy the conversation and learn as much as we did. A blog we have been referencing several times during the talk was this one from Cindy Sridharan on Monitoring in the time of Cloud Native: https://medium.com/@copyconstruct/monitoring-in-the-time-of-cloud-native-c87c7a5bfa3e

James Turnbull ( https://jamesturnbull.net/ ) is an author of 10 books on topics like Docker, Packer, Terraform, Monitoring, … and is currently writing a book on Monitoring with Prometheus https://prometheusbook.com/ . We got to chat about what modern monitoring approaches look like, how to pull in developers to start building monitoring into their systems and how to bridge the gap between monitoring for operations vs monitoring for business. Having a monitoring expert like James that knows many tools in the space was great to validate what we at Dynatrace have been doing to solve modern monitoring problems. We learned a lot about key monitoring capabilities such as capturing data vs capturing information, providing just nice dashboards vs providing answers to known and unknown questions and making monitoring easy accessible so that monitoring can benefit both business, operations and developers.We hope you enjoy the conversation and learn as much as we did. A blog we have been referencing several times during the talk was this one from Cindy Sridharan on Monitoring in the time of Cloud Native: https://medium.com/@copyconstruct/monitoring-in-the-time-of-cloud-native-c87c7a5bfa3e

The DevOps kids have decided to come up with a new term “observability.” We get to the bottom of the WTF barrel on what that is - it sounds like a good word-project. Also, there’s a spate of kubernetes news, as always, and some interesting acquisitions. Plus, a micro-iOS 11 review. Meta, follow-up, etc. Patreon (https://www.patreon.com/sdt) - like anyone who starts these things, I have no idea WTF it is, if it’s a good idea, or if I should be ashamed. Need some product/market fit. Check out the Software Defined Talk Members Only White-Paper Exiguous podcast over there. Join us all in the SDT Slack (http://www.softwaredefinedtalk.com/slack). Is “observability” just “instrumentation”? Write-up (https://medium.com/@copyconstruct/monitoring-and-observability-8417d1952e1c) from Cindy Sridharan. This guy (https://medium.com/@steve.mushero/observability-vs-monitoring-is-it-about-active-vs-passive-or-dev-vs-ops-14b24ddf182f): “Thinking directionally, Monitoring is the passive collection of Metrics, logs, etc. about a system, while Observability is the active dissemination of information from the system. Looking at it another way, from the external ‘supervisor’ perspective, I monitor you, but you make yourself Observable.” So, yes: if developers actually make their code monitorable and manageable…easy street! It’s a good detailing of that important part of DevOps. Cloud Native Java (http://amzn.to/2jPJHcv) has a good example with the default “observability” attributes for apps, and then an overview of Zipkin tracing. Weekly k8s News Heptio gets funding (https://www.geekwire.com/2017/heptio-raises-25m-series-b-funding-round-kubernetes-takes-world/), now “has raised $33.5 million in funding to date.” I think we’ll cover this press release in a WP episode. Also, something called “StackPointCloud” now with the Istio (https://thenewstack.io/stackpointcloud-drops-istio-service-mesh-integration/). Mesosphere adding K8s support (https://techcrunch.com/2017/09/06/mesosphere-says-its-not-bowing-to-kubernetes/) - “Guagenti also noted that he believes that Mesosphere is currently a leader in the container space, both in terms of the number of containers its users run in production and in terms of revenue (though the company sadly didn’t share any numbers).” "I think it’s fair to call Kubernetes the de facto standard for how enterprises will do container orchestration,” Derrick Harris (http://news.architecht.io/issues/with-oracle-on-board-kubernetes-has-to-be-the-de-facto-standard-for-container-orchestration-73880). Is Kubernetes Repeating OpenStack’s Mistakes? (https://www.mirantis.com/blog/is-kubernetes-repeating-openstacks-mistakes/) - Boris throwing bombs Meanwhile, an abstract of a containers penetration study (https://redmonk.com/fryan/2017/09/10/cloud-native-technologies-in-the-fortune-100/), from RedMonk: "Docker, is running at 71% across Fortune 100 companies. Kubernetes usage is running in some form at 54%, and Cloud Foundry usage is at 50%” This update from the Cloud Foundry Foundation (https://www.cloudfoundry.org/update-containers-2017-research-shows/) is a little more, er, “responsible” in pointing out flaws. Instead it just says there’s lots of growth and tire-kicking: 2016/2017 y/y shows those evaluating containers went up from 31% to 42%, while “using” ticked up a tad from 22% to 25%, n=540. Oracle’s in the CNCF (https://techcrunch.com/2017/09/13/oracle-joins-the-cloud-native-computing-foundation-as-a-platinum-member/) club! K8s on Oracle Linux, K8s for Oracle Public Cloud. “At this point, there really can’t be any doubt that Kubernetes is winning the container orchestration wars, given that virtually every major player is now backing the project, both financially and with code contributions.” James checks in on Red Hat (http://redmonk.com/jgovernor/2017/09/21/red-hat-is-pretty-good-at-being-red-hat/). (https://techcrunch.com/2017/09/13/oracle-joins-the-cloud-native-computing-foundation-as-a-platinum-member/) Acquisitions & more! Rackspace acquires Datapipe (https://techcrunch.com/2017/09/11/rackspace-acquires-datapipe-as-it-looks-to-expand-its-managed-cloud-business/) “The reason we’re buying them is that we want to extend our leadership in multi-cloud services,” Rackspace chief strategy officer Matt Bradley told me. “It’s a sign and signal that we’re going for it.” Bradley expects that the combined company will make Rackspace the largest private cloud player and the largest managed hosting service. Datadog acquires Logmatic.io to add log management to its cloud monitoring platform (https://techcrunch.com/2017/09/07/datadog-acquires-logmatic-io-to-add-log-management-to-its-cloud-monitoring-platform/) Puppet Acquires Distelli (https://www.geekwire.com/2017/puppet-acquires-distelli-bolster-cloud-computing-automation-platform/), known for their Kubernetes dashboard. Jay Lyman at 451 (https://451research.com/report-short?entityId=93381). Sizing Puppet: “The company has grown to more than 500 employees, and has estimated annual revenue in the $100m range.” Coverage from Susan Hall: “What we haven’t had up to this point is all the requisite automation for moving infrastructure code and application code through any kind of automated delivery lifecycle” and now they gots that. https://thenewstack.io/puppet-will-extend-infrastructure-automation-capabilities-distelli-acquisition/ “In May, the company launched its Kubernetes dashboard K8S. It allows users to connect repositories, build images from source, then deploy them to that Kubernetes cluster. You can also set up automated pipelines to push images from one cluster to another, promote software from test/dev to prod, quickly roll back and do all this in the context of one or more Kubernetes clusters… The Kubernetes service is offered as a hosted service or in an on-prem version. It provides notifications through Slack.” Google pays $1.1 billion for HTC team and non-exclusive IP license (https://www.axios.com/login-2487682498.html?rebelltitem=2&utm_medium=linkshare&utm_campaign=organic#rebelltitem2) Security Corner The Apple Effect? — Why BMW might get rid of car keys (http://www.autonews.com/article/20170915/OEM06/170919789/why-bmw-might-get-rid-of-car-keys) Don’t blame Apache — EQUIFAX OFFICIALLY HAS NO EXCUSE (https://www.wired.com/story/equifax-breach-no-excuse/) Is there anything to do here? Setup layers of credit cards? Require Touch ID (etc.) approval of all financial decisions and transactions in your “account”? Food & Safety like inspectors for security? Hackers respond to Face ID on the iPhone X (http://bgr.com/2017/09/21/iphone-x-release-date-soon-hackers-eye-face-id/) iOS 11 Coté has been running the beta. It seems fine. There’s the usual Re-arrangement of how some gestures work that’s jarring at first, but after using it for awhile, you forget what they even are. The extra control center stuff is nice. The Files.app is interesting, but not too featureful. The new photo formats are annoying because, you know, non-Apple things need to support it (which they seem to?) Bonus Links Coté gives up on defining DevOps, and more Interview about DevOpsDays Auckland (https://www.infoq.com/news/2017/09/michael-cote-devops-days-nz). (https://techcrunch.com/2017/09/13/oracle-joins-the-cloud-native-computing-foundation-as-a-platinum-member/) Is Solaris dead yet? Strongly confirmed rumors that Oracle is shutting it down (http://www.zdnet.com/article/sun-set-oracle-closes-down-last-sun-product-lines/). This guy has written a big Solaris-brain to Linux-brain manifesto/guide (http://www.brendangregg.com/blog/2017-09-05/solaris-to-linux-2017.html), plus: “[n]owadays, Sun is a cobweb-covered sign at the Facebook Menlo Park campus, kept as a warning to the next generation.” SICK BURN! Layoffs and more (http://dtrace.org/blogs/bmc/2017/09/04/the-sudden-death-and-eternal-life-of-solaris/): “In particular, that employees who had given their careers to the company were told of their termination via a pre-recorded call — “robo-RIF’d” in the words of one employee — is both despicable and cowardly.” HPE We Can See The New Hewlett Clearly Now, Says CEO Whitman (http://blogs.barrons.com/techtraderdaily/2017/09/05/hpe-we-can-see-the-new-hewlett-clearly-now-says-ceo-whitman/?mod=BOLBlog) - AI in storage arrays, Docker in OneView. Clearly (http://www.marketwatch.com/story/hp-enterprise-has-yet-another-confusing-plan-to-simplify-itself-2017-09-05)? Making money (https://www.cnbc.com/2017/09/05/hpe-earnings-q3-2017.html). They bought CTP!? (https://www.cloudtp.com/doppler/hewlett-packard-enterprise-to-acquire-cloud-technology-partners/) Selling hardware to cloud providers is rough (https://www.nextplatform.com/2017/09/06/prospects-leaner-meaner-hpe/). Huawei New board (http://talkincloud.com/cloud-services/chinas-huawei-braces-board-revamp-western-markets-beckon). Microsoft app support. We can all agree on food Someone has to pay attention to this real world stuff (http://www.nationalgeographic.com/magazine/2017/09/holland-agriculture-sustainable-farming/). This Tiny Country Feeds the World More on VMware/AWS The possible failures in the partnership (https://www.bloomberg.com/news/articles/2017-09-01/how-vmware-s-partnership-with-amazon-could-end-up-backfiring) - sort of an odd article in that the larger point is “maybe it won’t work.” Meanwhile, Matt Asay does some loopty-loops on it all (https://www.theregister.co.uk/2017/09/11/kubernetes_envy/). JEE Code put in github (http://go.theregister.com/feed/www.theregister.co.uk/2017/09/06/oracle_java_ee_java_se_github/). They’re giving it over to the Eclipse Foundation (https://blogs.oracle.com/theaquarium/opening-up-ee-update). Probably a good idea. VMware’s OpenStack Little report form 451 (https://451research.com/report-short?entityId=93303&type=mis&alertid=693&contactid=0033200001wgKCKAA2&utm_source=sendgrid&utm_medium=email&utm_campaign=market-insight&utm_content=newsletter&utm_term=93303-VMware+sheds+free+version+of+its+OpenStack+distribution). “Going forward, users pay a onetime $995-per-CPU socket license fee, in addition to ongoing support.” Recommendations Brandon: Prophets of Rage (http://prophetsofrage.com/). Matt: American Gods (https://en.wikipedia.org/wiki/American_Gods_(TV_series)), the TV show. Zero History (https://www.amazon.com/Zero-History-Blue-William-Gibson-ebook/dp/B003YL4AGC/): finale(?) to William Gibson’s Blue Ant trilogy LOT (https://www.lot2046.com/): a subscription-based service which distributes a basic set of clothing, footwear, essential self-care products, accessories, and media content. Engineering the End of Fashion (https://www.ssense.com/en-gb/editorial/fashion/engineering-the-end-of-fashion) Coté: Rick & Morty (http://amzn.to/2xrHo3L). These cultural guides (http://www.commisceo-global.com/country-guides) are fucking awesome! See America (http://www.commisceo-global.com/country-guides/usa-guide), Australia (http://www.commisceo-global.com/country-guides/australia-guide), and Latvia (http://www.commisceo-global.com/country-guides/latvia-guide) (no one sang at the meals I was at!). Cardenal Mendoza (https://www.thewhiskyexchange.com/p/996/cardenal-mendoza-brandy-solera-gran-reserva), brandy de jerez (https://en.wikipedia.org/wiki/Brandy_de_Jerez). And, you know, cognac/brandy in general - be a fucking adult already, you damn kids.

Cindy Sridharan joined the show to talk about development and operations as a generalist, leveling up as an engineer (while still providing business value), challenging the status-quo, and other interesting Go projects and news.

Cindy Sridharan joined the show to talk about development and operations as a generalist, leveling up as an engineer (while still providing business value), challenging the status-quo, and other interesting Go projects and news.

Cindy Sridharan joined the show to talk about development and operations as a generalist, leveling up as an engineer (while still providing business value), challenging the status-quo, and other interesting Go projects and news.