It's Steph and Chris' last show. Steph found a game, and if you've been following the journey, all of the Test::Unit test files are now live in RSpec. JWTs really grind Chris' gears. They wrap up with things they've learned, takeaways they've had, and their proudest podcasting moments. They also thank all the folks who've helped make The Bike Shed happen. This episode is brought to you by Airbrake (https://airbrake.io/?utm_campaign=Q3_2022%3A%20Bike%20Shed%20Podcast%20Ad&utm_source=Bike%20Shed&utm_medium=website). Visit Frictionless error monitoring and performance insight for your app stack. Microservices (https://www.youtube.com/watch?v=y8OnoxKotPQ) Transcript: CHRIS: One more round of golden roads, our golden. So here we go. STEPH: Oh, one more round of golden roads. Okay, maybe that's going to get to me today. [laughs] CHRIS: [singing] Golden roads take me home to the place. STEPH: [singing] I belong. CHRIS: Yeah, there you go. Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Chris Toomey. STEPH: And I'm Steph Viccari. CHRIS: And together we're here to share a bit of what we've learned along the way, at least one more time. So with that [chuckles] as an intro, Steph, what would you say is new in your world? STEPH: Hey, Chris. Well, today is the big day. It is the day that you and I are recording our final Bike Shed episode, which we have all the feels about, and we will definitely dive into. But to ignore some of that for now, I have another small fun update I can provide about a new game that I found. So one of the things that's new in my world is I started playing a new board game with Tim; it's called Ticket to Ride. Have you heard of that? CHRIS: I have. I don't know if I've played it. I feel like it's a particularly popular one now. But I don't know if I've ever had the pleasure. STEPH: It's a very cute game, so we have the smaller version of it. For anyone that's not familiar, it's essentially a map. And then there's a bunch of spots where you can build trains and connect them, and then you get tickets. So your goal is that you're going to connect one location to another location. And then you get points and yada yada, but it's so much fun and especially the two-player version. It's like this perfect 20, maybe 30-minute game. I'll be honest; I'm not really a board game person. I always enjoy it. Once I get into it, then I'm like, this is great. I don't know why I was resistant to this. But every time someone's like, "Do you want to play a board game?" I'm like, "Not really." [laughs] I first have to get into it. But I have really enjoyed Ticket to Ride. That's been a really fun game to play. And it's been a nice way to, like, even during the day, we'll break for lunch and squeeze in a game. CHRIS: Well, I love good two-player games. They're hard to find. But when you find a good one, and it's got that easy pickup and play...I believe I'm going to now purchase this. And thank you for the tip. STEPH: Yeah, this is definitely one of those where it's easy to pick up, and then you can get the expanded board. So there's a two-player version, but then yeah, you can get one that's a map of the U.S. or a map of Europe. And I think it accommodates up to five players as the maximum, so not a huge group but definitely more than two. On a slightly more technical note, I have something that I'm very excited to share. It is a journey that you have been on with me, that everybody listening has been on this journey with me. And I'm very excited. I see you nodding your head, so I'm guessing that you're going to know where I'm headed with this. But I'm very excited to announce that all of the Test::Unit test files now live in RSpec. So that is a big win. I'm very, very excited for that to be a previous state of life and not an ongoing state of life. Because I have certainly developed too much niche knowledge around migrating these tests, and that became apparent to me when I was pairing with another developer that works with the client because they had offered...they had some time. They're like, "Hey, do you want help migrating a test file?" And I was like, "Sure." I was like, "But this is wonky enough, like, we should pair and work on this together because I just know some ins and outs. And I don't want you to have to learn a lot of the hard lessons that I've learned." And the test that we happened to pick up was very gnarly. It had a lot of mystery guests. And we spent, I think it was a good two hours. And we only migrated one of the tests, so not even a full file but one of the tests. And at the end of it, I was like, I know way too much about some of the oddities and quirkiness of this. And we got through it, but we decided that wasn't a good use of their time for them to go at this alone. So that's why I'm extra excited and relieved because I didn't want this task to carry on to someone else. So, hooray, we did it. CHRIS: Hooray. Just in time. You're Indiana Jones grabbing your hat right as you roll out and off to [laughs] be away from the project for a bit. So you stuck the landing. Well done, Steph. STEPH: Thank you. Thank you. So that's some great news. And then also, everything else in life is pretty much focused around getting ready for maternity leave. That's about to happen soon, and I am so ready. I have thoroughly enjoyed a lot of the things that I'm doing, [laughs] but goodness, being pregnant is hard. And I am very much ready for that leave. So also, a lot of the things that I'm doing right now are very focused on making sure everything's transitioned and communicated and that I just feel really good about that day of departure. That covers all the newness in my world other than the big thing that we're just not talking about yet. How about you? What's new in your world? CHRIS: Well, continuing to skirt the bigger topic that we will certainly get to in the episode, what is new in my world? I'm actually quite excited workwise right now. We have a much larger body of work that finally we got the clarity. All the pieces fell into place, and now we're sort of everybody rowing in the same direction. There's interesting, I think, really impactful code that we're writing for Sagewell right now. So that's really fantastic. We've got the whole team back together on the engineering side. And so we're, I think, in the strongest and most interesting point that I have experienced thus far. So that's all really fantastic. On a slight technical deep dive, you know what really grinds my gears? It's JWTs. JSON Web Tokens and I have never gotten along. It's never been a match made in heaven. And we have a webhook that comes from Plaid. Plaid is a vendor for connecting bank accounts and whatnot. And they have webhooks like many people do. So they can inform us when things change, lovely feature of how we build web apps these days. But often, there's a signature that says, "This is definitively from us, and you can trust us." And usually, it's some calculated signature, HMAC, or something like that. For some reason, Plaid's uses JWTs, and more than that, they use JWKs. So there's JWT which is the signature. That JWT itself is signed with a JWK. You have to fetch the JWK from their server based on the key ID in the header of the JWT. But how do you know if you can trust the JWT before you've gotten the JWK? All of this broke in a recent upgrade. We went from Heroku-20 to Heroku-22 to the new platform with Heroku, which bumped us to OpenSSL 3.0, and it turns out JWT doesn't work with it. And so that's sad. It's a no. It's going to be a no. It turns out the way that OpenSSL 3.0 works is incompatible with some of the code paths in JWT. And so I was like, wait, we just can't do this? And it's low-level cryptographic primitive stuff that I'm not comfortable messing around with. I'm not going to hop in there and roll up my sleeves. And even just getting to the point that I understood what was broken about this took like an hour and a half just to sort of like, wait, which is okay...so the JWT signs and encodes. And this will be a theme that we come back to later, but I think web development should be simpler. I think we should strive for simplicity. And this is a perfect example where I'm guessing Plaid uses JWTs and that approach to communicating security things often, but I've not seen it used much for signing webhooks. And, oof, it led to a complicated day. And it's unfixable now as far as I can tell. There is a commit on the JWT Ruby repo as of five days ago, but it doesn't build in our system. And it's not released. And it's just a mess. So yeah, engineering is complicated. I'm both wildly excited about what we're doing at Sagewell, and then today was this local minimum of like, oh, JWTs again. Again, we find ourselves battling. And you won today, but hopefully not for too long. STEPH: Oof, how did this manifest that you first noticed? So is it because a webhook suddenly stopped working, and that was like the error that rose up, and that's what helped you dive into it? CHRIS: Yeah, we have a little bit of code in the controller for where Plaid events come in. We calculate and verify the signature of the webhook to make sure that it's valid, and we reject it otherwise. And we alert ourselves via Sentry, and then we also have a Datadog scan that can show what's the status code of the response. Because these are incoming HTTP payloads or requests, and so we can see there were 200 up until this magical day when suddenly everything changed. And that was when we switched Heroku stacks. And then we can see it also in Sentry. So we're able to look at it, and we're like, why are none of the Plaid webhooks able to verify the signature anymore? That seems weird. And so then Datadog confirmed that it consistently was broken from this point in time. And then we were able to track that back. It was also pretty easy to guess because the error was "pkeys are immutable in OpenSSL 3.0," and that was the data. And I was like, oh, cool, that sounds fun. Let me go figure out what that means. STEPH: [laughs] Well, it's a nice use of Datadog. I remember in the past you were talking about adding it. And I was excited because I've never been at that point where a team has just introduced it; either a team doesn't have it, and they wish they had more insights, or they have it and don't use it. And nobody ever checks the board. So that's a nice anecdote for Datadog helping you out. Yeah, I'm not envious of your situation, friend. CHRIS: I do love the cup half full take [laughs] that you have on the overall situation, but that's nice how Datadog worked out for you. And you know what? It was. Thank you, Steph, for once again being that voice of positivity. STEPH: I appreciate that you enjoy it because there are times that when someone points it out to me that I do that, I have to be like, "I'm sorry, I'm not trying to be toxic positivity over here. [chuckles] That's just how my brain works." CHRIS: Oh, you are definitively not toxic positivity. That's a different thing. Because you ended with but also, I feel bad for you, and I'm glad that I'm not in your shoes. So you are the right level of positivity. I don't think I could have talked to you for three and a half years as co-host on a podcast if I didn't appreciate the level of positivity or the general approach that you bring to thinking about stuff. STEPH: Okay. Well, to borrow a phrase from Matt Sumner, who has been a guest on the show, cool, cool, cool, cool. I'm glad my positivity has been well calibrated. And I was about to say I'm interested to hear how this turns out for the team. [laughs] But we're in an awkward spot where I mean, you and I, we can still totally chat. But listeners won't get to hear the rest of that particular saga. I mean, you can share. I mean, you do you. I'm setting all sorts of boundaries for you right now. Okay. And now I'm just rambling, and I'm getting weird with it. Because the truth is that, you know, we won't be back. And this is our final episode together. So I think let's just go ahead and rip off the Band-Aid. Let's dive into it. Let's talk about it. Given that it's our last episode that we are recording, we thought of a couple of things that we'd like to talk about. You brought up a great idea that I'm excited to dive into. Do you want to lead us in? CHRIS: Sure. Well, if we go back all the way to Episode 172, that is the first episode that you came on as a guest. I actually continue to really love the title of that episode, which is What I Believe About Software. And it both captured that conversation really well, but also, more generally, it's actually become the tagline of the show when we do our little introduction. What do we believe about building great software? Et cetera. And I think that's been the throughline of the conversations that we've had is what remains true. What are the themes? Not necessarily the specific technologies, although we certainly talk about that. But what do we believe about building great software? And so today, I thought it would be fun for us to talk about what do we still believe about building great software? It's roughly three and a half years or so that we've been doing this. What's still true? STEPH: Oh, well, I have the first unequivocal one, the thing that I still believe about building great software, and that's you should hire thoughtbot. That's definitely the way to go. We'll help you get it done, not that I'm biased in any way. CHRIS: No. I'd say collectively between us; there's zero bias with regard to thoughtbot or any other web development shop out there. But thoughtbot is the best. STEPH: All right, perfect. So we've got the first one, the clutch one of hire thoughtbot. And then I also really like this topic. And I still think back to that first episode that I recorded with you and how much fun that was and how that really got me to start thinking about this. Because it was something that, at the time, I didn't really reflect on a lot in terms of what does it take to build great software? I was often just doing the day-to-day actions but then not really going high-level think about it. So I'm excited this is one of the topics that we're revisiting. So for the next one, this one is, I don't know, maybe it's a little cutesy, but I was trying to think of an alliteration that I enjoyed. And so this one is be an assumption assassin. So what assumptions are you making? And then how can you validate or disprove them? And that is something that I find myself doing constantly. And it always yields better work, better questions, better software, better code, better code reviews. And that's my first one is be an assumption assassin and identify what assumptions you have. And I had a really good example come up today while I was having a conversation with Joël about something that I was looking to merge. But I was a little hesitant about it because there are some oddities that I won't dig in too deeply. But essentially, there's a test that I migrated that highlights an existing concern in the code. And I was like, should I go ahead and merge this test that documents it, or should I wait to fix that concern and address it? And he brought up a good point. And he's like, "Well, we're assuming it's a bug and an issue, but it may not actually be depending on how the software is being used." And so then he was encouraging me to reevaluate that assumption that I had where I'm like, oh, this is definitely a problem to, like, I don't know, is it a problem? Let's ask somebody. CHRIS: First off, I love that as a theme, as one of the things that you still believe about software. Second, I believe you correctly said that you were looking for an alliteration, but my brain heard acronym. STEPH: [laughs] CHRIS: And so then I was like, B-A-A-A. Is it BAAA? What are you going for there? Oh, you just wanted a bunch of As. Okay, I got it now. Secondly or thirdly, I think I'm on my third now. Apparently, within Sagewell team culture, one of the things that I'm most known for is... there are two phrases: one is just to name it, and the other is to be clear. And these are the two things that I do apparently constantly so much that it's become a meme within the team. It's just like, okay, everybody's been talking. But I just want to make sure we're on the same page here. So just to be clear, or just to name it, here's what I'm seeing. But I agree; I think taking those things...what are the implicit bits? What are the assumptions? And making them more explicit. Our job as developers is just to yell at computers all the time and make them try and do human stuff. And there's so much room for lossy conversions at every point in that conversation chain. And so yeah, being very clear, getting rid of assumptions, love it. It's all great stuff. Actually, in a very related note, the first on my list is that code is for humans to read. This is one of the things that I believe most deeply and most impacts the way that I write software. Any given piece of functionality that we want to author in our code feels like 10, 20, 50, frankly, almost infinite different versions of the code that would produce nearly identical functionality. So at the end of the day, the actual symbols and strings of text that we bring together to write the code is all about other humans, other people on your team, you five months from now, you a week from now, frankly, or me. I'm going to say me, me a week from now. I want to do future me and everyone else on the team a solid and spend that extra 10% of okay, I have something that works now, but let me try and push it around and try and massage it into a shape that is a little more representative of how we're actually thinking about the code, how we talk about it as an organization. Is that the word that we use to describe that domain concept? Maybe we could change that just a little bit. Can I push more of this into the private API? What actually needs to be known here? And I think that's where I'm happiest is in those moments because that's where all of the parts of the job come together, the bit where I trick a computer into doing what I want and simultaneously making it so that that code is revisitable, clear, expressive, all of those things. So yeah, code is for humans. And that's true across every language, and framework, and domain that I have worked in. And I've only believed it more and more so over time. So yeah, that's mine. STEPH: Yeah, I love that one. That's one of the things that comes to mind when people talk about disliking code reviews. And I can imagine there are a number of reasons that people may have had a poor experience with a code review process. But at the end of the day, if you're not getting that feedback or validation from fellow humans, then how do you know that you've been successful, that you've written something that other people can follow up on? Which goes back to the assumptions in terms of like, you're assuming that you have written something that your future self or that other people are going to be able to read and maintain down the road. So yeah, I love that one. One of the other things that I still hold really true to building great software is prioritize early and often. So always be checking in to understand with your users, with your tech concerns, with data that you may have, new insights, and then just confirm that yes, you and the team are constantly working on the thing that has been prioritized and that is the most important. And also, be ready to let go. That can be really hard. I have definitely had those moments in my career where I've spent two weeks working really hard on something. And then we've realized that the thing that we were pursuing isn't that valuable, or it's something that users don't need or actually want. And so it was better to let go of it than to pursue it and ship it anyways. So that's one of my other mantras that I have adopted now is prioritize, prioritize, prioritize. CHRIS: Unsurprisingly, I agree wholeheartedly with all of that. We're still searching for that thing, that core thing that we disagree on other than Pop-Tarts and IPAs. But I don't know that today is the episode that we're actually going to find that. But yeah, prioritizing is such a critical activity. And it is this interesting collaboration point. It gets different groups together. It's this trade-off. It's this balance. And it's a way to focus on and make explicit the choices that we're making. And we're always making choices. We're always making trade-offs. And so being more explicit, being more connected and collaborative around those I believe in so, so, so much. So love that that was something on your list. Let's see, next up on my list is reduce complexity, just sort of as an adage, just always be reducing complexity. It is amazing to me in my time, particularly as a consultant, but even now, this is something that I hold very true is just it's so easy to grow a system in anticipation of future complexity or imagine that the performance concerns that we're going to run into will be so large that we must switch from Postgres and a nice, simple atomic database into a sharded, clustered Kafka queue adventure. And there are absolutely cases that make sense for that sort of thing. But at a minimum, I beg of you, anyone starting a new system, don't start with microservices. Don't start with an event queue-based system. These are wildly complex versions of what often can be done with so much simpler of an application. And this scales through to everything. What's the complexity of an API? Do we need caching in that API layer? Or can we just be a little bit inefficient for a little while and avoid the complexity and the overhead of caching? Turns out caching is a tricky thing to get right, just as an aside. And so the idea of like, oh, let's just sprinkle in a little bit of caching. It'll be easy, and then we'll get better performance, like, yeah, but did you get it right? Or did you introduce a subtle bug into your program that's going to be really hard to debug later? Because do you cache in development? Well, maybe, I'm not sure, could be. So over time, this is something that I've sort of always felt, but I've only ratcheted it up. It's only something that I've come to believe in more and to hold more firmly to. I think earlier in my career, it was something that I felt, but I would more easily be swayed by aspirational ideas of the staggering amounts of traffic that we would be getting soon or the nine different ways that the data model will expand. And so, we should code the current version in anticipation of that. And I have become somewhat the old man on his lawn yelling at the clouds like, "Nah, we don't need it yet. We can grow to that." And there's a certain category of things that are useful to try and get out in front of and don't introduce additional complexity, but they're a tiny, tiny list. And so, for most things, my stance is what's the simplest thing that we can get away with right now, that still provides a meaningful experience to our users, that doesn't compromise on security or robustness or correctness but just solves the problem we have right now? And over and over and over again, that has served me incredibly well. So yeah, keep that complexity at bay. STEPH: That is one that I've definitely struggled with. And frankly, it works in my favor, that idea of keeping things simple. Because I'm terrible when it comes to predicting the future or trying to build things in a way that I just don't have enough information to really drive the architecture or the application that I'm building. So anytime I'm trying to then stretch and reach for the future in those ways unless I really have a concrete understanding of I am building for these particular scenarios, it's really hard to do. So I very much like keeping it simple and not optimizing before you need to. And it reminds me of I think it's Mark Twain, who has a quote, "Worrying is like paying a debt that you don't owe." And that's something that comes to mind for me when also writing code and building features and software is that I tend to be someone who will worry about stuff. And I'm like, oh, is this going to be easy to extend? Is it going to be what it needs to be six months from now if we need to add more features to this and build on top of it? And I have to remind myself it's like, well, let's just wait. Let's wait till we get there and we know more. One of my other ideas that couples nicely with the one that you just shared in regards to keeping things simple and then waiting for those needs to arise is that mistakes are going to happen. They are a part of the process. As we are learning and growing and we're stretching our skills and trying things out, things are going to go wrong. We're going to introduce bugs. And to take those opportunities, that's when we start to use that feedback to then improve things like observability, like capturing logs, and how we handle error reporting or having a plan for emergencies. So maybe that's the part of worrying that can pay off is thinking through, all right, if something does break, or if something gets shipped that shouldn't, then what is our plan in how we handle that? How do we roll back? Or how do we get things back to a stable build? CHRIS: It's funny. I was actually visiting with a friend this past weekend, and we were chatting more generally about life things but the idea of worrying and anticipation and trying to prepare for every bad outcome. And there's the adage of an ounce of prevention is worth a pound of cure. But increasingly, both in life, depending on the context, and in code, I've found that I've shifted to the opposite of it's impossible to stop everything. There are going to be bugs that are going to get out there. There are going to be places where we code things incorrectly. And I would rather...I still want to try as hard as I can to get things right, to be clear. I'm not giving up on trying. But I'm all the more focused on how do we know and how do we recover when those things happen? So it's interesting that you just described exactly that, which, again, is a very human life conversation, and yet it applies to the code. STEPH: I love that rephrasing of it. Instead of the mistakes are going to happen, it's, like, how do we know, and how do we recover? I think that's perfect. I've also found that by answering the how do we know and how do we recover, that really helps you build trust with clients as well. Because again, things are going to happen, things are going to break. And the more prepared you are for that and then the better plan that you have, and then they can watch how you execute that plan, and it's going to establish a lot of deep trust with other engineers and also the team that you're working with, that you have been thoughtful and that you have ideas on how are we going to address this? Instead of waiting for that moment to happen. That's going to happen too. You're going to make decisions in the heat of the moment. But I have found that to be a really useful way to establish yourself with a team in terms of I care about this team and these processes and this application. So how do we handle the bad times, not just the good times? I do want to circle back because you alluded to the fact that you and I, we've tried to find things that we disagree on. And so far, Pop-Tarts and beer have been the two things that we disagree on. But I do have a question for you that maybe I will disagree with you on. But I need to know some more about it first. You have alluded to there's the Brussels snack, (Oh, I'm going to get this wrong.) Brussels sprout snack hour or working lunch, something combination of those words. [laughs] And it's the working lunch that has stuck out to me, and I've wanted to ask you about it. So here I am. I'm asking you about it. What's a working lunch? What's the Brussels snack happy hour, snackariffic working lunch look like? CHRIS: This is fantastic. I love that you waited until the last episode that this was rolling around in the back of your head. And you're like, are you making the team work through lunch? And now, on this final episode, we get to address the controversy that has been brewing in the back of your head. Spoiler alert, no, this is just ridiculous nomenclature. These are two meetings that we have that are more like, let's get the dev team together and talk about stuff that's in our platform sort of developer experience. Or stuff in observability often is talked about in this context because it doesn't quite impact users, but it's how we think about the work. And so there are two different meetings that alternate every other week. So every Friday afternoon, we do this, but it's one of two meetings depending on the day. So there's a crispy Brussels snack hour that was the first one that was named, which was named purely for nonsense reasons because we don't have anything else that's named nonsensically in our organization. And so I was like, oh when we name this meeting, we should make it nonsense because we don't have any other...We don't have, you know, an SOA microservices fleet with Barbie doll and Galactus and all of the other wonderful names. Those are references to the greatest video ever about microservices; if you've not seen it, that will be in the show notes. It's required reading. But anyway, we don't have that. And so we thought, let's be funny with the name of this. So the crispy Brussels snack hour is one, and the crispy Brussels we wanted something that was...the first one is a planning meeting. The second is like, let's actually sort of ensemble program. Let's get the four of us together, and we'll work on some of the stuff that we're talking about here but as a group. And so I wanted the idea of we're working, and so I was like, oh, this will be the crispy Brussels work lunch. But it's purely a name. It's the same time slot. It's 3:00 o'clock on a Friday afternoon. [laughs] So it is not at all us working through lunch. I don't think we should work through lunch. I'm concerned that you thought that for a while, and you were just like, I'm a little worried, but I'm not going to bring it up. But I'm glad we got to cover this before we wrapped up this whole Bike Shed co-hosting adventure together. STEPH: I feel relieved and also a little robbed of an opportunity for us to have something that we disagree on because I thought this might be a thing. [laughs] CHRIS: We can continue searching for that thing. But maybe it's okay that we agreed on most stuff for the run [laughs] of this fun, little show that we did together. STEPH: Yeah, that's gone on quite a time. We've got like three years together that we have managed to really only find two, I mean, very important of course, two things. But yeah, it's been pretty limited to those two areas. And each time that you'd mentioned the work lunch, I was like, huh, I need to ask about that because I have feelings about it. But then, you always would dive into very interesting stories of things that came out of it, and I quickly forgot about it. So this feels good. This feels like very good important closure. I'm glad that this finally surfaced. But circling back, since I took us on a detour for a little bit, what are some other things that you still hold deeply about building great software? CHRIS: I've really got one last thing on the list. It's interesting, there's not a ton technically in this list, which I think represents broadly how I feel about software, and I think how you feel about software. It's like, it's actually mostly about how the people interact at the end of the day. And you can program in any language or framework, and you can get the job done. We certainly have our preferences and things that we enjoy. But the last one really rounds us out, which is think about the users. I always want to be anchoring the conversations that we're having, the approach that we're taking to building the software in what do the users think? Who are our users? What do we know about them? What do they care about? How are they using this technology? How is it impacting their lives? We've talked a number of times about potentially actually watching the sales demo as an engineering team, trying to understand what's the messaging that we're putting out into the world for this piece of software that we're building? Or write along with customer support and understand what are the pain points that people are hitting? And really, like, real humans, what are they experiencing? Potentially with a name attached. And that just changes the way that you think about the software. There's also even the lower-level version of it. As we're building classes or modules, what are the public facets of that, and what are the private API? What's the stuff that we're hiding away? And what's the shape that we are exposing to the outside world for varying definitions of outside? And how can we just bring in a little bit of empathy to try and think about, again, in the case of like the API for a class, it's probably you on the other side of it, but it's future you in a slightly different mindset with a little bit less information and context on the current problem that you're working on. And so, how can we make things easier for ourselves in the code, for our users at the end of the day? How can we deliver real value that is not mired in the minutiae of technical complexity and whatnot but really is trying to help people live better lives? That's a little too fancy as I say it out loud. But it is kind of the core of what I believe, so I'm not going to take it back. STEPH: I love how you've expanded users where more traditionally, it's people that are then using the software. But then you've expanded it to include developers because that is something that is often on my mind and something that I just agree with wholeheartedly in terms of when you're writing software; as you mentioned before, software is for people. And so we want to include others. And it does improve people's lives. People show up to work every day, and if you've been thoughtful if your past you has been thoughtful, it's either going to give you your future self a better day, or it's going to give other people a better day. So I think that's a very fair statement, improving lives by being thoughtful in regards to focusing on the users, people consuming software, and working in the codebase. CHRIS: I know we've talked about this before, but I was having a conversation with one of the developers on the team at Sagewell just last week, and they were mentioning how they really loved working on admin features. And I was like, oh, that's interesting. Let's talk more about that. And it was really it's that same thing that I think you and I have discussed of like there's that immediacy. There's that connection. These are actually colleagues, but you can build software to make their day better. You can understand in detail what the pain points are. What's the workflow that as you watch it, you're like, oh, I could put a button up in the corner of the screen that would automate almost all of this and your day would be that much faster? Oh, let me do that. That's exciting. And so I love that as another variation of it, like, yeah, there's for other developers. There's also for the admin team or other users in the organization of the software. There are so many different versions of users, but I think I think we build a better thing if we think about them more. STEPH: I have definitely worked with teams where I can tell that certain people are demoralized, and it comes down to they feel frustrated and often disconnected from the people that they are building for. And so then you really feel isolated. I'm pushing code around, but I don't really see the benefit or the purpose of it. And I think that's very hard for developers who typically want to build something that's going to be useful and not feel like it's just going to be thrown away. So connecting your team to those users, I certainly understand. Getting to build something for your colleagues and then they get to say how much they like it is an incredible, rewarding experience. You also touched on something that I really appreciate, where you highlighted that a lot of the technical decisions that we make are important, but they're not at the center of the things that we believe when it comes to building great software. And that's something that I will often reflect on. Like, as we were thinking through these particular ideas that we still hold true today, how mine are more people and process-focused and rarely deep in the technical weeds. And there are times that I think, well, shouldn't there be something that's more technical, something that's very concrete? Yes, you should build your code this way or build your application or use a specific technology. But after all the projects and teams that I've been a part of, that's just usually not the most important part. And so I appreciate that you highlighted that because sometimes I have to remind myself that, yes, those things can be challenging, but it's often with people and process. That's where the heart of great software lies. CHRIS: That's a fantastic phrase, I think, that really encapsulates all of the conversations that we're having here. MID-ROLL AD: Debugging errors can be a developer's worst nightmare...but it doesn't have to be. Airbrake is an award-winning error monitoring, performance, and deployment tracking tool created by developers for developers that can actually help you cut your debugging time in half. So why do developers love Airbrake? Well, it has all of the information that web developers need to monitor their application - including error management, performance insights, and deploy tracking! Airbrake's debugging tool catches all your project errors, intelligently groups them, and points you to the issue in the code so you can quickly fix the bug before customers are impacted. In addition to stellar error monitoring, Airbrake's lightweight APM enables developers to track the performance and availability of their application through metrics like HTTP requests, response times, error occurrences, and user satisfaction. Finally, Airbrake Deploy Tracking helps developers track trends, fix bad deploys, and improve code quality. Since 2008, Airbrake has been a staple in the Ruby community and has grown to cover all major programming languages. Airbrake seamlessly integrates with your favorite apps and includes modern features like single sign-on and SDK-based installation. From testing to production, Airbrake notifiers have your back. Your time is valuable, so why waste it combing through logs, waiting for user reports, or retrofitting other tools to monitor your application? You literally have nothing to lose. So head on over to airbrake.io/try/bikeshed to create your FREE developer account today! CHRIS: Actually shifting gears a little bit, so we've just talked about what we still believe about building great software. I'm intrigued. We've been chatting for a number of years here on this microphone, these microphones. We have separate ones because we're in different states. But I'm interested; what have we changed our minds about? What have you changed your mind about, Steph? I got a couple of ideas, but I'm intrigued to hear yours. STEPH: Nothing. I've never been wrong. I've stuck to everything that I've ever thought. CHRIS: That must be boring. STEPH: [laughs] Yeah, that's totally not true there. There are definitely things that I've changed my mind about. One of the things that I've changed my mind about is that people who know the most will ask the fewest questions. That's something that I used to consider the trademark of someone who is a more experienced senior developer in terms of you really know what you're doing. And so you typically don't ask for help or need help very often. And so, I'm going way back in terms of things that I have changed my mind about. But I have definitely changed my mind where people who know the most are actually the ones that do a really great job of constantly asking questions and asking for feedback. And I think that is still a misconception that people still carry forward. The idea that if you're asking a lot of questions or asking for help that you are not as skilled in your work, and I view it as quite the opposite, that you are very good at what you do and that you know precisely the value of your time. And then also reaching out to others for help, and then also just getting validation on things that you may have concerns around. So that's one I've changed my mind on is that I think the more experienced you are, the more questions you tend to ask. CHRIS: Oh, I love that one. It's a behavior that I know...I think we've talked about this before. But as consultants, we try and model it just the like; it's totally fine to ask questions. And because we often come in with less context, it makes sense for us to be asking questions, but I will definitely intentionally lean into it in those contexts to be like, everybody keeps throwing around this acronym. I don't actually know what that is. Let me raise my hand. And my favorite moment is when people disagree on what the acronym or what the particular word or what the particular project is. Like, I ask the question, and people are like, "Oh, it's this," and someone across the room is like, "Wait, that's what it means? I thought it was this totally other thing." I'm like, cool, glad that we sorted that out. Glad that we got that one up in the air. But I actually remember many, many, many years ago, at this point, there was a video series of...PeepCode was the company, and there was the Play by Play series. And so there were particular prominent developers, particularly in the Ruby community. And they would come and sort of be interviewed and pair program. And it was amazing getting to watch these big names that you had heard of, like Yehuda Katz is the one that stands out in my mind. He was one of the authors of merb, which was a framework that was merged with Rails, I want to say around the 3.0 time. And just an absolute, very big name in this world and someone that I looked up to and respected. And watching this video, they had to Google for particular API signatures and Rails methods. They were like, "Oh, how does that work? Is it link to and then you pass the name?" I forget what it was specifically. But it was just this very human normalizing moment of this person who has demonstrably done incredible work in our community and produced very complex software still needs to Google for the order of arguments to a particular method within Rails. I was like, oh, okay, that's good to know. And with complete humility in the moment, I was just like, yeah, this is normal. Like, it's impossible to hold all of that in your head. And seeing that early on shook me off the idea that that's the thing to do is just memorize everything. It's like no, no, get good at asking the questions. Get good at debugging. Get good, yeah, asking questions. It's a core skill rather than a thing that you grow out of. But I definitely shared early on I was like, not allowed to ask questions, that'll be scary. STEPH: I love that example. Because counterintuitively, to me, it demonstrates confidence when someone can say, "Oh, I don't remember how this works," or "Let me go look it up." And so I just very much appreciate when I see someone demonstrating that level of confidence of let's keep going. Let's keep making progress. I'm going to ask for help because that is totally fine, and we are in a safe space. Or I'm going to create a safe space for us to do that. One of my favorite versions of this where you shared like if you ask about an acronym and then people disagree, one of my favorite versions is to ask about a particular area of the codebase and be like, what would you say this code is doing here? What do you think users do here? Like, what is the purpose? What's the point of this? [chuckles] And then having people be able to say, "Oh, yeah, this definitively does this thing." Or people are like, "You know, I'm not sure. I don't even know if that code is getting run." That's one of my favorite outcomes of asking questions. How about you? What's something you've changed your mind about? CHRIS: I made a list of a couple of things like remote is on there. I didn't know if I'd like remote. I wanted to try it for a while. Tried it, turns out I like it a lot. It's complex. You got to manage it, whatever. But that I think everybody's talked about that a bunch. I think probably the most interesting one is deadlines. Initially, in my career, I didn't really feel anything about them. And then I experienced the badness of deadlines. Deadlines are bad. Deadlines are things that come down from on high and then you fail to hit them, and then you're sad. And maybe along the way, you're very stressed and work long hours to try and get there. But they're perhaps arbitrary. And what do they even mean? And also, we have this fixed scope, and they're just bad. And then there was a period of my time where, like, deadlines are bad. The only thing that we do is we show up, and we make the software as quickly as we can. But in reality, there are times that we need that constraint. And in fact, I have found a ton of value in deadlines when used intentionally. So we can draw a line in the sand, and we can say, at this point in time, we will have a version of the software. We have a marketing campaign that we need to align with this. So we got to have something at that point. And critically, if you're going to have a deadline, you've now fixed a point in time. You need to flex other things. And critically, I think the thing to flex is the scope. So we need to have team management. We have user accounts right now, but now we need to organize them into teams. That is like a category of functionality. It's not a singular feature. And so yeah, we can ship teams in the next quarter. What exactly that means is up in the air. And as long as we're able to have conversations essentially on a day-to-day at least weekly cadence as to what will make it in by that deadline and what won't, and we're able to have sometimes the hardest conversations but the very necessary conversations of the trade-offs that we have to make as we're building that software, then I find deadlines are absolutely fantastic tools for focusing and for actually reducing scope but in a really useful way. And getting something out there in the hands of users so that you start to get real feedback so that you start to learn, is this useful? What are the ways that people are using this? What should we lean into and do more of? What maybe should we roll back, actually? So yeah, deadlines. First, I didn't know them, then I feared them. Now I love them but only under the right circumstances. It's a double-edged sword, definitely. STEPH: I, too, have felt the terribleness of deadlines and railed against them pretty hard because I had gone through a negative experience with them but have also shifted my feelings about them where they can be incredibly useful. So I really liked that's one of the things that you've changed your mind about. It also reminds me of one of the other things...I'm going to circle back for a moment to one of the things that I believe about creating great software is to not wait for perfection, and deadlines are a really good tool that helps you not wait for perfection. Because I have also seen teams really struggle or sometimes fail because they waited until there was something perfect to present, and then you realize that you've built the wrong thing. So I do want to transition and talk a bit about the show because it's our last episode, and we should talk about it, and the fun adventures that we've had and some of the things that we've learned or things that we're feeling in the moment. So given that it's been a wonderful three years for me, it's been four years for you since you've been a host on the show. How are you feeling? CHRIS: I'm feeling a bunch of different things sort of all at once. I am definitely going to miss this immensely. Particularly, I loved when I started, and I got to interview a bunch of thoughtboters and other people from the community. But frankly, three-plus years of getting to chat with you has been just such a delight. There's been an ease to it. We kind of just show up and talk about what we're doing. And yet there are these themes that have run through it. And it has definitely helped me hone and shape my thinking and my ability to communicate about what I'm thinking. I've learned that you have a literal superpower to remember the last thing that you were talking about. Listeners, you may not know this, but we are not quite the put-together folks that we may sound like in these recordings. We have a wonderful editor, Mandy Moore, who makes us sound so much better than we are. But we'll often pause and stop and then discuss what we want to talk about next. And Steph always knows the exact phrase that she or I left off on. And it has been so valuable to the team. But really, it's been just such a pleasure getting to have these conversations. It's also been something that has just gently been in the back of my mind at all times. And so, I'm observing the work in any given week as I'm doing it. It's almost like meditation in a certain way, whereas I'm working on something, like, oh, this is actually really cool. I want to take a note about this and talk about it on The Bike Shed with Steph. And having this outlet, having this platform to be able to have those conversations and knowing that there are people out there is fantastic, although it's very weird because really, every one of these recordings is just you and I on a video call. And so there is an audience, I'm pretty sure. I think people listen to the show; I don't know, occasionally they write in, so it seems like they do. But at the end of the day, this really just feels like a conversation with a friend, and that has been so valuable to have. And yeah, I'm definitely going to miss that. It's been a wonderful run, you know, four years is a long time. It's about as long as I've done most things in my career. And so I'm very happy with what we have done here. And there's a trite saying that isn't...yeah, whatever; I'll just say it, which is, "Don't be sad that it's over. Be glad that it happened." And I guess I'm still going to be sad that it's over. But I am so glad that I got the opportunity to do this, that you joined in this adventure and that we got to chat each week. It's been really delightful. STEPH: I really liked how you refer to this as being a meditative state. And that is something that I have certainly picked up from you and thoroughly enjoyed that I have this space that I get to show up and bring these ideas and topics and then get to talk them out with you. And that has been such a nice way to either end the week or start a week. I mean, it doesn't matter. Anytime that we record, it's this very nice moment of the week where we get to come together and talk through some of the difficulties and share our stories. And that's been one of my favorite moments is because you and I get to show up and share everything that's going on. But then when someone writes into the show or if they send a tweet or something and they share their story or their version of something that happened, or if they said that we made them laugh, that was one of my favorite accomplishments is the idea that something that we have done was silly enough or fun enough that it has brought them joy and made them laugh. So I, too, I'm very, very much going to miss this. It has been a wonderful adventure. And I thank you for encouraging me to come on this adventure because I was quite nervous in the beginning. And this has definitely been an aspect of my life that started out as something that was very challenging and stretching my limits, and now it has become this very fun aspect and something that I get to show up and do and then get to share with everyone. And I do feel very proud of it, very much in part to Thom Obarski, who was our initial producer and helped us have that safe space to chat about things. And now Mandy, who keeps the show running smoothly and helps us sound our best week to week. So it's been a wonderful adventure. This is going to be hard to let go. And I think it's going to hit me most. Like, this was one of those things as we're talking about it, it's, like, I'll see you next week. This will be fine. But I think it's going to hit me when there's something that I want to talk about where I'm like, oh, this would be great to talk about, and I'll add it to The Bike Shed Trello board. And I'll be like, oh yeah, that's not a thing anymore, at least not quite in the same way that it was. CHRIS: So what I'm taking away from this is that you're immediately going to delete my phone number the minute we hang up this call and stop recording. [laughs] STEPH: Oh yeah. I preemptively deleted. So that's already done. Friendship is over at this point. CHRIS: That's smart. Yeah, because you might forget otherwise in the heat of the moment as we're wrapping this whole thing up. STEPH: [laughs] CHRIS: But actually, on that note, in a slightly more serious vein, again, there's this weird aspect where the audience is out there. But we're very sort of disconnected, particularly at the moment in time where we're recording. But it has been so wonderful getting various notes from listeners, listener questions, but also just commentary and the occasional thanks and focusing; oh, you pointed me in the direction, or you helped me think through a complicated piece of work or process a problem that we were having. And so that has been just so, so rewarding. And one of the facets of this that has been so interesting to me is being able to connect to people and basically put out there what we believe about software, and for the folks that resonate with it and be able to have that connection instantly. And meeting people, and they're like, "Oh, I've listened to The Bike Shed. I like all these things." I'm like, oh, cool, we get to skip way further into the conversation because I've already said a bunch, and you say you like that thing. So, cool, we're halfway through our introductory chat. And I know that we agree about a bunch of things, and that's really wonderful. And frankly, I'm going to miss that immensely. So for anyone out there who's found something valuable in this, who's enjoyed listening week to week, or perhaps even back to Upcase for things, I would love to hear from you. I'd love to connect to folks. Send me an email, Twitter. I'm on all the places. I'm Chris Toomey in various spots or ctoomey.com on the internet. Chris Toomey on GitHub. I'm findable, I think. Chris Toomey developer will probably get you there. But I would really love to hear from folks, to connect to folks, you know, someday down the road; I think I'll be hiring again. And that'll be fun. I would love to work with some of the folks that have listened to this show or meet you at a conference, or if I happen to be traveling to a city or you're traveling to Boston. Really for me, so much of what this show is about is connecting with people around how we think about building great software. And so, I would love to continue that forward into the future. So yeah, say hi, if you're interested. STEPH: I agree. That's been one of the most fun aspects of being co-host of the show. And I'll be honest, you are welcome to contact me, but I am going to be off-grid for probably six months. [laughs] So just know that there will be a bit of a delay before you hear back from me. But I would definitely love to hear from you. I also want to say a very heartfelt thanks to a couple of people, just folks that have made this journey incredible and have made it so much fun. One, in particular, is everyone at thoughtbot for their continuous stream of knowledge. I mean, frankly, my software opinions wouldn't be half as interesting if it wasn't for everyone at thoughtbot constantly sharing their knowledge and being a source of inspiration. So I deeply appreciate everyone that has contributed to topics and ideas and just constantly churning out blog posts because those are phenomenal. And I also want to give a shout-out to my husband, Tim, because he has listened to The Bike Shed for many years and even helped out with a number of show notes when that was something that you and I used to do before Mandy made our life so much easier and took that over for us. And has intervened a number of times when Utah mid-recording would decide it's time to play. So I want to give a very special thank you to him because he has been a very big supporter of the show and frankly helped me manage through a lot of the recordings for when I had an 80-pound dog that was demanding my attention. CHRIS: I think continuing on the note of thanks; similarly, I'm so grateful to thoughtbot as an organization for everything that is represented in my career. It's a decade-plus that I have been following and then listening to the podcasts and then joining the organization, and then getting so many wonderful opportunities to learn about this thing called web development. And then, even after I left the organization, I was able to stay on here on The Bike Shed and hang out and still chat with you, Steph, which has been really wonderful. So thank you, thoughtbot, so much. Thank you to Joël Quenneville, who will be the continuing host of the show. This show is not going anywhere. And, Steph, you and I aren't really going anywhere, but we won't be around anymore. But we are leaving it in the very, very capable hands of Joël, and I'm super excited to hear the direction that he takes it and Joel's incredibly thoughtful and nuanced approach to thinking about programming and communicating. So I think that will be really wonderful. And lastly, I definitely want to thank Derek Prior and Sage Griffin, the two original hosts of this show, who really produced something wonderful, and for many years, I think it was about four years that they hosted together. I was an avid listener despite actually working at the company the whole time and really loved the thing that they produced and was so grateful that they entrusted me with continuing it forward. And hopefully myself and then with the help of you along the way, we've...I think we've done an okay job, but now it is time to pass the torch or the green lantern. That's the adage I've been going with. Gotta pass the lantern, pass the mantle on to the next one. So, Joël, it's going to be in your hands now. STEPH: Yeah, I'm so looking forward to future episodes with Joël Quenneville. They are going to be fabulous. So I've been thinking in terms of this being our finale episode and then a fun ending for it, so there's a thing called the 21-gun salute, which is the military honor that's performed by firing cannons or artillery. Not to be confused with the three-volley salute, which I definitely confused earlier that is reserved and used at funerals, which this is not. So using the 21-gun salute, I was like, hmm, it is The Bike Shed, and we have this cute ring ring that goes. So I think for our finale, we should have a 21-bell salute as we exit the shed and right off into the sunset. CHRIS: I love it. I couldn't imagine a more perfect send-off. So with that, what do you think? Should we wrap up? STEPH: Yes, but I have one more silly thing to add. I've thought of a new software idiom that I'm excited about. And so, this may be my final send-off into glory that I'd like to share with you. And I think that we should make like a shard and split. CHRIS: [laughs] I so appreciate that in this moment, this final moment that we have together, you choose to go with a punny joke. It is so on brand for the show. It is absolutely perfect. And I think with that note, shall we wrap up? STEPH: Let's wrap up. CHRIS: The show notes for this episode can be found at bikeshed.fm. STEPH: This show is produced and edited by Mandy Moore. CHRIS: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review on iTunes, as it really helps other folks find the show. STEPH: If you have any feedback for this or any of our other episodes, you can reach us at @_bikeshed or reach me on Twitter @SViccari. CHRIS: And I'm @christoomey. STEPH: Or you can reach us at firstname.lastname@example.org via email. CHRIS: Thanks so much for listening to The Bike Shed, and we'll see you next week. ALL: Byeeeeeeeeee!!!!!!!! ANNOUNCER: This podcast was brought to you by thoughtbot. thoughtbot is your expert design and development partner. Let's make your product and team a success.
CHRIS NEWBOLD: Hello, wellbeing friends. Welcome to the Path To Well-Being In Law Podcast, an initiative of the Institute for Well-Being in Law. As you know, my name is Chris Newbold. I serve as executive vice president of ALPS Malpractice Insurance. You know, our goal here on the podcast is to introduce you to thought leaders doing meaningful work in the wellbeing space within the legal profession, and in the process, build and nurture a national network of wellbeing advocates intent on creating a culture shift within the profession. As always, I am joined by my co-host, Bree Buchanan. Bree, how are you doing today? BREE BUCHANAN: I'm doing great, Chris. Great to be here. CHRIS: Good, good. As you all know, Bree is the president of the Institute for Well-Being in Law. Bree, we have some really exciting news to share about the institute and the journey that we're on to engineer this culture shift. Would you maybe give us a clue as to the breaking news that I think that we were so excited about? BREE: Nobody could be more excited than me because you said, you know, Bree is the board president. Well, up until this news, I had two jobs. I was the acting executive director, so I am just delighted to let people know we have hired our first full-time staff person and that is our inaugural executive director. Her name is Jennifer DiSanza. She comes to us with a whole host of experience in wellbeing issues and particularly with the law students. For many reasons, we wanted to bring Jennifer on board, but also strategically, we really realized that's where she's coming from is the future of our profession. And also, aside of where we know there's a lot of behavioral health distress and stress on the youngest members of our profession and the law students. So we're just thrilled to have Jennifer on board. CHRIS: Yeah. See, I had the privilege of serving with you Bree on the hiring committee. Boy, we have a dynamic leader now that will be working day-to-day to think about advancing wellbeing in our profession. You know, there's so much work to be done as you well know. We're actually planning on having Jennifer as our next podcast guest, which will be awesome to be able to just talk about the vision, why she's passionate about this work. It will also happen to be after the conclusion of some strategic planning that we as a board will be doing. So things are just really aligning well with both what has transpired, where we're going, and then focusing on what lies ahead in terms of some big issues that we have to tackle as we think about the wellbeing of lawyers and legal professionals in the profession. With that, today we're going to circle back to, we've spent considerable time in the area of diversity, equity, and inclusion. You know, we had anticipated a three part series on this, but sometimes you extend an offer and you get somebody who's so awesome that you sit there and go, we have to expand this even further. Right? BREE: Along came Kori. Yeah. CHRIS: That's right. Along came Kori. And when Kori came along, we're like, okay, we're breaking the rules. We're totally bringing Kori into the mix. And so we were really excited to welcome Kori Carew to the podcast. Bree, would you be so kind to introduce Kori? And again, this is I know a podcast that we've been very excited and looking forward to. BREE: Absolutely. So Kori is a people inclusion strategist, an advocate, a speaker, a writer, a status quo disruptor. Got to love that. Child of God, wife and mother of two curly-haired, wise, energetic, fierce, spitfire daughters. Her family is multi-ethnic, multi-racial, multi-religious and spans multiple nationalities. She brings a fierce love of community and belonging that transcends differences to work, ministry and life. She loves to sing, cook, entertain, dance in the hallways at work, we need a video component of that, and read. Equipping leaders to be inclusive, to interrupt bias and disrupt the status quo. At her day job, she focuses on developing and implementing strategies for individual career and diversity and inclusion success, and helps organizations build bridges across differences and improve inclusion. BREE: When she's not working, she focuses her voice and talent on issues of gender equity and rights, inclusion, and human and civil rights, serving in her church and community, and cherishing her phenomenal tribe and community. She's energized by helping people live their very best lives. Kori was the Director of Strategic Diversity Initiatives for seven years at Shook, Hardy. And then she came over to Seyfarth and is now the Chief Inclusion and Diversity Officer there and oversees their really spectacular wellbeing program, Seyfarth Life, and a whole host of other initiatives we're going to hear about. So Kori, welcome to the podcast. CHRIS: Yay. KORI CAREW: Thank you. I appreciate you inviting me to be on this podcast and also very much the work that you are doing. This conversation of wellbeing for attorneys is such an important conversation. It's one that we probably started having too late, and it's one where diversity and inclusion, there's more work to be done than time. I'm super thankful for all that you do and all that you do to help our profession be better, so thank you very much. BREE: You bet. Kori, I'm going to start off. We ask all of our guests a variation of this question. What experiences in your life are drivers behind your passion for work around diversity, equity, and inclusion and belonging and wellbeing? KORI: Thank you for that question. And of course, you're causing me to go down a bit of memory lane. You would think this is an easy question, but it actually is not. It's not as easy because it forces you to look in the rear view mirror and try to understand where the dots connected to where you are. Before I do that, I do want to make one small correction. Seyfarth Life is an incredible initiative at Seyfarth that I am super proud of and one of the things that energized me about joining the firm. It has a steering committee that leads it. It's four partners at the firm, all of whom have a connection to wellbeing and mindfulness. My department and my role actually does not oversee Seyfarth Life, but we do work very closely with them. Because as one of the founding members, Laura Maechtlen noted from the very beginning, there's that intersection between inclusion and diversity and belonging and wellbeing, and the two work very closely together. But my department does not oversee Seyfarth Life. So just wanted to make sure I give credit to the right people. BREE: Absolutely, give credit where it's due. KORI: You know, because they're awesome and they do great work. In fact, if I may brag on them, out of the steering committee members, one of them is the chair of the largest department in the firm and an executive committee member and co-chair of the national diversity and inclusion action team. Oh, wait a minute. No, that's not right. Three are office managing partners. They're part of this steering committee, this leadership group, because they actually practice wellbeing and mindfulness and meditation in their own personal lives and allow it to influence how they lead. So I know Seyfarth didn't pay me to do a promotion, but I felt like I needed to shout some guys out. BREE: Absolutely. KORI: Our talent team helps them quite a bit in terms of organizing programs and handling the administrative and logistic things. Okay. So to answer your question, what are the experiences? I often say this and it is true that when I look at my life in the rear view mirror, how I ended up where I am makes a lot more sense as I connect the dots in ways that I probably couldn't have foreseen. For example, I never intended to be a diversity and inclusion professional. I actually never intended to go to law school. I started my university career as an electrical engineering major. When I came to the U.S., I wanted to build planes. That was my thing. I wanted to be an aeronautical engineer. I wanted to build planes. I loved science. I could spend hours in the lab. One of the best gifts I ever got was a lab coat. My dad had a custom drawing board built for me when I was a teenager that I carried with me everywhere because technical drawing, engineering drawing was one of my top subjects. KORI: So a lot of things make sense in hindsight. I look at my family composition and my sisters and I were all born in different countries. We have different passports. We grew up in Nigeria, a country with over 300 different ethnic groups with different languages and traditions and customs, so there's that. My family is multi-religious, multi-ethnic, multi-national, multi-racial and there's just so much diversity there. You know, in the family tree, there's a granduncle that's a Methodist church bishop, and one that's an Imam. And my grandfather's father was a teacher, was a teacher of the Quran. And so all of that diversity is there in the family, but it probably influenced how my parents raised my sisters and I and how even through childhood, I was always the person who was connecting the dots between similarities between people. And today we would call that cultural fluency, this ability to recognize cultural differences and not judge them but just adapt to them and be able to say, okay, you know what? KORI: It looks to me like person A is looking through a lens that's different than person B, but they're looking at the same thing. So how can I get these two people to be on the same page? So there's that family dynamic. But another thing that happened when I was growing up that I do think influenced me quite a bit. I grew up in Nigeria. Most of my childhood, we had one military dictator after another. So I grew up with coos happening more often than I would prefer. There were times that things broke out into religious violence. You're talking about incidents where a few people are killed or a lot of people are killed and everything goes to standstill, everybody's on edge. You don't leave your home. When the students go on riots because they're protesting something and things get out of hand, you're turning off the lights in your home and sort of huddled together, trying to make sure that you stay together as a family until everything passes over. So that was also something that I grew up around and experiencing. KORI: And then my parents are from Sierra Leone. Sierra Leone is actually my home country. If you ask me where I'm from, I will tell you I was born in Canada, grew up in Nigeria, but I'm from Sierra Leone. Because in my culture, you're where your father's from. So my entire identity has always been that I am from Sierra Leone. In the '90s, Sierra Leone began to experience a very brutal civil war, which calling it a civil war is actually inaccurate. You have a bunch of people with weapons who terrorize the population for 11 years. And it's been one of the most brutal wars that the world has seen at least in recent times. And that impacted my family in the sense that we lost people, in the sense that I hadn't been back to Sierra Leone for a long time. And it kind of started with my mom not feeling it was safe enough for us to go and visit, with grandparents living on the run and being sick and dying and me not seeing them in a long time because of just this state of chaos. KORI: And all of this fueled how I ended up going to law school, wanting to do human rights work, wanting to be a human rights lawyer, feeling as if I learned so much about the American system and the role that the legal profession played in terms of maintaining democracy and freedom and wanting to multiply that. Right. But then I go to law school. I graduate. I fall in love with a boy who I actually started dating in college, and I ended up in Kansas City because I followed a boy. You know, career took a different turn, ended up being a defense lawyer. And then you fast forward to doing an evaluation and me going through a process of saying, okay, I've done a lot of the things I wanted to do. I've achieved a lot of the things I wanted to achieve. I wanted to try cases. I wanted to build this reputation. I wanted to be successful in A, B, C, D. KORI: And I started taking inventory of the things I was passionate about, the skills I developed, the experiences I had and where I was losing time. You know, where was I given my time in community? What were the things that I could lose myself doing in such deep flow that I don't even recognize that time has gone by? And that journey ended up leading me to inclusion and diversity work and I haven't turned back since. There's some aspects of the legal profession I miss. I miss trying cases. I miss solving problems for clients. It may sound like the weirdest thing, but boy, playing around with evidence, rules, and figuring out how to get things in or keep things out is a nerdy love of mine. And so those are just some of the experiences that I would say led me to this love for helping people build bridges and I'm empower people to succeed despite the challenges, and being able to create just a level of cultural fluency amongst groups of people so that we understand how much better we are together as opposed to isolated from one another. So that's a long answer. BREE: Well, what an amazing life you've had to date and an incredible background that informs your work at a depth that I know Chris and I can't even begin to imagine. CHRIS: For sure. Kori, how long have you been more squarely centered on the inclusion and diversity side of things? KORI: I have been for 11 years now full-time diversity. What I realized, you know, somebody asked me a question similar to this, how long have you been doing diversity work, which is different from what I usually hear. I actually did the inventory and realized that, you know, 29 years ago, when I first came to the U.S., that was when I actually started doing presentations. At the time, we called them multiculturalism. We started doing presentations on bridging differences, on being able to understand different cultures and how you navigate it. And so I've been actually teaching on diversity, inclusion, cultural fluency leadership topics now for 29, 30 years. But it being my full-time job, that happened when I left litigation and moved over to Shook, Hardy & Bacon. CHRIS: Okay. I think a good point to maybe start the conversation is, you know, again, your perspective is so unique and informed. For diverse members of the profession, can you talk to our listeners about some of the more challenging aspects of the last couple of years? KORI: Yeah. So the last couple of years have been tough for everyone. This pandemic, it's been brutal and it's impacted us in so many different ways. We've lost our sense of certainty to the extent that we didn't had any. We've lost our ability to have some kind of predictability, something that is a core need, a core need for many of us. Well, not for many of us, for everyone. It's actually a core human need. And so we've been sort of thrown into this whirlwind of uncertainty with no deadline, right? We went from thinking, well, I'll speak for myself. You know, since I'm not a scientist, I foolishly thought, well, maybe in two weeks I'll go back to the office. And then it was a month. And then I thought six weeks. And then I thought for sure by summer 2020 we'd be able to go out and about and things would be quasi under control. And here we are, you know, some 28, 29 months later and we still have COVID. I'm sick right now recovering from COVID after avoiding it for almost 30 months, I get it. KORI: So you have that benchmark that is impacting everyone and the uncertainty that we've seen with everything going on around us. But as with everything, I think people from historically underrepresented and marginalized groups, what happens is the things that... There's this saying that the things, and I'm going to probably say it wrong. And it may be an African American saying, but it's this thing that what gives some people a cold will give others the flu. And so what you've seen then is populations that have been historically marginalized and underrepresented and haven't had access to full equity, had been impacted very differently by the same storm that we're all in. So we're all in the same storm, but we're not in the same boat. We're experiencing it differently. So communities of color, we know got hit by COVID much harder. KORI: And you have that intersection between race, between housing inequity, between education inequity, between healthcare inequity and healthcare access, all of those things coming together to adversely impact some groups more. So if you are someone who is Brown or Black, or from one of these historically marginalized communities, and you are going to work during the pandemic, or you're working from home, you are more likely to have family members who have been directly impacted by COVID, right? You are more likely to have lost family members. You also, generally speaking are more likely to be in a position where you are in an extended family situation where you are responsible for more people than just yourself. You know, one of the things that we know, for example, that impacts generational wealth is that those of us from communities of color oftentimes are responsible not just for ourselves, but for extended family members. KORI: So you have that dynamic playing, then you have the racial pandemic, which has been going on, but in the last two years have come to fevered pitch. And so the daily trauma of dealing with racism and microaggressions then gets compounded by all the incidents, George Floyd, Charles Cooper, and all the other incidents that have been bombarding us from our television screens, from the news reports, from articles. And so now all of a sudden everything is right in your face and you're dealing with all of it at the same time. And so those are some of the things that are professionals from "diverse communities," from underrepresented marginalized communities have been dealing with. And our reserves have been tapped into and overstretched to where for some of us, it feels like it's been just too much. BREE: Absolutely. It's unimaginable just how much to carry on in that space. All of the things that you just described, this litany of horrors is on top of just the day-to-day difficulty as been expressed to me, and reading in my friends of people of color, just the microaggressions and just how hard it is. Just take away pandemic and everything else and the racial reckoning, how hard it can be just to get through the day. I can't even imagine. It is absolutely just too, too much. Kori, there's so much to unpack here. I wanted to kind of pushing us along here talking about diversity, equity, and inclusion and talking about belonging and overlaying that. I mean, when I started looking in the legal profession, we talk about DEI, it was diversity then DEI, and now we're getting into some of the really, to me, needy and interesting stuff around belonging. I know that you created a belonging project at Seyfarth. Could you talk to us about the importance of that, and also about this project that you got started at Seyfarth? KORI: Sure. Let me separate them out. Belonging is a conversation that more and more of us are having, and it is fairly new to the conversation when you're talking about diversity and inclusion. It started with we talked about diversity, and then we started talking about diversity and inclusion, and now we've included equity and belonging. Belonging goes to that sense, that feeling that each of us have when we belong and we feel like we are part of a group and that we belong to something that is bigger than us. It is also a core human need. Brené Brown has this phrase that she says that we have three irreducible needs, and they are to be loved, to connect, and to belong. What we know from the research is that when we don't have belonging, it impacts us. It is wired into our DNA to belong to something. KORI: So we will either have healthy belonging, or we will seek a belonging that may not be healthy and may not be good. This is where you can queue in hate groups and cult because they will do anything to belong. We will also conform to fit in so that we have a quasi sense of belonging. The problem though is that when we don't have belonging, we actually see physiological, physical, spiritual, mental, psychological impact on our wellbeing. It impacts our sense of health. Forget our sense of health. It actually impacts our health, right? We know that exclusion and the lack of belonging actually results in increased depression, increased high blood pressure, increased diabetes. Incidentally, a lot of the same things that racial trauma and microaggressions also causes on the human body. And so if we don't have that sense of belonging, then we are not able to actually actualize that sense of inclusion where everyone is able to be leveraged and their differences and their strengths leveraged so that they can succeed as they want to succeed. KORI: And without belonging, you don't get wellbeing. But conversely, without wellbeing, you can't cultivate that sense of belonging. And so those two things are intertwined as well as this concept of engagement, which also is in the mix, right? You can't create engagement unless you have social connection and belonging. And so all of these things come together. Unfortunately, in many of our organizations, they're treated as separate, right? In many organizations, you have the wellbeing function being managed in a way that it doesn't speak to diversity, doesn't speak to belonging at all. So imagine now we just talked about COVID and we talked about how COVID has impacted everyone. Then imagine you're developing a wellness initiative or a wellbeing initiative and you're not stopping to think, oh, wait a minute, because of diversity, this pandemic has impacted people in different ways. KORI: And so I can't just trot out a wellbeing program without factoring in diversity and how diversity has resulted in different people experiencing this pandemic differently. Similarly, we fail when we try to, for example, have a wellbeing initiative that doesn't stop and think, oh, wow, we're not talking about racial trauma. We're not talking about microaggressions. We're not talking about the impact of implicit bias and exclusion on the psychological and physical wellbeing of the people in our organization. And so what's happening is these concepts are tied together, but in our organizations and most of our organizations, we're not doing DEI and incorporating wellbeing and we're not doing wellbeing incorporating DEIB. Instead, we're acting as if they're completely separate and they're not. CHRIS: I mean, I think it goes without saying, we, I think as human beings, sometimes we compartmentalize of there's this and then there's that. I think that from the infancy of the institute, I think we've emphasized the importance of diversity, equity, and inclusion as part of, has to flow through everything, every lens that we look at from the wellbeing perspective. But I have to admit, it's been more challenging than I think, than we've appreciated because sometimes we look a little bit myopically at some of these issues without broadening our lens. That's the perspective that I think that you can bring our listeners that, again, this intersection of diversity, equity, inclusion, belonging with wellbeing, I guess I'd be curious on just, how can we merge? Right? Because again, even the fact that there's organizations that work over here and organizations that work over here, and we really should be just the coalition and the umbrella and the totality of how it all works together is something that I don't know that we appreciate the magnitude of. KORI: Well, and the only way we can appreciate the magnitude is if we have these honest conversations. But we also have to have the conversations around the structural and the cultural underpinnings, right? How do we have conversations about wellbeing that take into consideration differences? That take into consideration, okay, we're telling people, hey, we have therapy or we have EAP, or we have whatever the organization offers. But how do you do that and also acknowledge that for some communities that there is a stigma around maybe going to a therapist? How do you have that conversation with those communities? Or that racial bias and racial aggressions are having an impact on people, but you have an entire generation of Black people, for example, who have survived by plowing through all the challenges that the world has put in front of us. And to sit down and talk about the way in which racism has impacted us is asking us to put our shields down, which means opening up ourselves to attack, which means possibly being accused of playing the race card. Right? KORI: All of things that you may have grown up in a time where we just didn't talk about that in mixed company, we only talked about that with each other. And so there are all these layers, all these layers. I recently listened to a friend of mine, Ratu Basin, and she was talking about how it feels for her as someone of Indian heritage to see how much yoga, for example, has been whitewashed. There's so many conversations to be had even in the wellbeing space, even when we're talking to people about things like self-care. Well, what are you recommending? Because some of the things we tell people to do for self-care, go get a massage, who can afford that? What culture support that kind of self-care? And is that really self-care or is that treating a symptom? Should self-care and wellbeing be about a way of life and a way of working such that we don't need these emergency [inaudible 00:32:26] like solutions to fix the symptoms, right? KORI: And that's the big conversation and that's the conversation I'm hearing some lawyers begin to ask where they say, the organization says they care about wellbeing, but we're getting these other messages that say it's productivity and hours and billables that matter, right? How do we shift the culture and how we're embracing these topics in a way that makes it more meaningful? I just realized, I didn't even answer your second question about the belonging project, but yeah, this is the stuff that to me, I see a lot of potential for us to have really good conversations that can lead to solutions that are more inclusive of a diverse profession. BREE: Kori, you're clearly such a thought leader and a visionary in this space. Can you talk a little bit about how do we get change to occur in a profession, the legal profession that is so reluctant to change? Even more so than general society. Where do you see the bright points of really being able to make some change? KORI: Can you repeat that question? BREE: Yeah. Just about how do we get change to occur in the legal profession? You know, this is a profession that is just so stayed and slow and bound up in tradition. This is the way we do it, that sort of thing. And here you are with these fabulous ideas, working with a very large law firm, having come from another very large law firm so you're in this space. What are your ideas for actually getting real change to occur? Where are the pressure points, I guess? KORI: Well, I think some of the pressure points are actually external. You asked me a question earlier about the last two years, something that I didn't mention that has impacted a lot. It's impacting individuals from underrepresented groups, but it's also impacting our organizations. Is this fake cultural war that is also going on, you know, regardless of what political party you're in, I think we can acknowledge that for the last six years, there has been an attack on everything that we are trying to accomplish in diversity and inclusion. White is now Black, Black is now white. And if we are in a state of being, for example, where I'll use Florida as an example where someone can say, we want to ban any training if it makes someone uncomfortable. What you're essentially saying is let's keep the status quo the way it is, even if the status quo supports white supremacy. KORI: Even if the status quo is inequitable. You would rather keep the status quo than have an uncomfortable conversation. When it comes to the legal profession, in particular, law firms, because of how we are constructed. A law firm essentially has multiple owners. It's not like a corporation that has a board of directors and has shareholders. Let's say you have a law firm of a thousand people and 300 of them are partners. You have 300 people running around who think that everybody should have an equal say in every single decision. It's one of the reasons that law firms function so differently from other companies and why decision making is so different. Everything we do is different. You know, we put people in leadership positions not because they're leaders, but because they're great trial attorneys or they're great business generators or whatever, whatever the criteria is, but rarely is it because someone actually is a good leader. KORI: And so we have this culture that we have built that really makes it difficult for us to have real hard conversations on the things that really matter, on the things that really can make change. So imagine that law firm now sitting in the last six years and even more so in the last three years. I can tell you when it comes to diversity, inclusion, many of us are throwing our hands up and saying, so how in the hell are we supposed to have this conversation then? If you're saying, oh, we can't talk about white privilege because someone says, oh, that offends me. Or we can't talk about systemic racism because someone's going to say, oh, wait a minute, if you say systemic racism is real, then that's anti-American. So we are living in a time where the terms racism, the terms CRT have been completely redefined to where they mean nothing that even resembles what they actually mean. KORI: And then we're over here arguing about these fictitious decisions, these fictitious definitions, and we're not actually doing the hard work that needs to be done, right. Because if you won't even acknowledge that systemic racism is real, then how do we evaluate the systems to see where we may be having inequitable results and then changing those systems? Because if you deny a thing exists, then we can't even address it. BREE: Absolutely. KORI: And so that's probably one of the biggest challenges I see, but also the biggest opportunity. And if anything is going to change when it comes to diversity, we have got to get more courageous about having difficult conversations, but conversations that are worthwhile, they are important. Nothing about creating equity is comfortable and cozy and touchy-feely, it's hard work. It requires us to say some things that we maybe may not have faced before, but we don't get to change what we won't face, what we won't acknowledge, and what we won't be honest about. It's like, you can't write a new end into the story if you won't acknowledge the truth of the story. That's the whirlwind that I think we are in now, not just as a profession, but as a country and a society. BREE: Absolutely. What an incredibly difficult place to be? Yeah, go ahead, Chris. CHRIS: Well, I was just going to say, I want to unpack that more. Let's do this. Let's take a quick break and come back because I mean, my burning question and Kori began to sort of thinking about it, which is what's the pathway to better, more productive, honest conversations, right? Because I think that you're right. The question is, how do we create the environments for ultimately that societal discussion to occur in the most productive way? So let's take a quick break and we'll come right back. — ADVERTISEMENT: Meet VERA, your firm's Virtual Ethics Risk Assessment Guide developed by ALPS. VERA's purpose is to help you uncover risk management blind spots from client intake to calendaring, to cybersecurity, and more. VERA: I require only your honest input to my short series of questions. I will offer you a summary of recommendations to provide course corrections if needed, and to keep your firm on the right path. Generous and discreet, VERA is a free and anonymous risk management guide from ALPS to help firms like yours be their best. Visit VERA at alpsinsurance.com/vera. — CHRIS: Okay. We are back with Kori Carew, our esteemed guests and the chief inclusion and diversity officer at Seyfarth Shaw. Kori, we were just getting into the, I think the discussion. I feel like we're going deeper than even I had thought we would in the conversation, which I love. You know, as we think now about we need to have the honest conversations, right. And so I would just be curious on your opinion as what's the pathway to get there. If we appreciate that there's a lot of noise and the volume levels are high, and there's a lot of yelling, frankly, on both sides of the equation. What's the pathway toward problem solving, thoughtful discussion, intentional discussion that ultimately advances the dialogue? KORI: Thank you very much for that question. Honestly, it's one I've been thinking a lot about. You know, I did do a TEDx in 2017 and the impetus for that TED really was that question that you just asked, which was, there's a lot of yelling and not enough dialogue that allows us to move into action. Since I gave that TED, I've sort of watched what's been going on in organizations and in the country. I don't think I would change anything about that TED, except that there are a few more things that I would emphasize. One of the first things that we have to do if we truly want to make progress, and I'm going to steal a Nigerian thing, tell the truth and shame the devil. We are avoiding being honest with ourself about so many things. Whether it is just being honest about the experiences people have in the organization, or being honest about where the gaps are, or being honest about what the failures are, or even individual honesty. KORI: That self-awareness to say, you know Kori, you talk a lot about wellbeing and you talk a lot about leadership, but the reason you talk about those things is because you were searching for something that you did not have in the leaders that you grew up under, right? So you were trying to create something for others that you didn't have, but you are also trying to create it for yourself. And there are many days that you totally suck. There are many days that you are making very bad wellbeing decisions. There are days that you are not as inclusive as you would want to be, but it's okay. And the only way you're going to get better is by acknowledging where you're not doing it right. Now, think about that when we're talking about gender or race or LGBT inclusion or disability inclusion. If we as individuals and we as organizations are not willing to be honest about our history, what has happened and what is happening, then we don't even have a starting point. KORI: And the way that we do that is very, very cliché. Getting comfortable with what is uncomfortable. I remember when I first started saying that, when I was at Shook, Hardy & Bacon and it wasn't even a thing many people were saying, and now people say it so often that it has lost its meaning. But it truly is the beginning point. And in too many of our organizations, we are shutting down any discussion or any movement in the name of trying to get consensus, or in trying to water things so much that they're meaningless, right? Or being so hyperworried about future possible hypothetical litigation that somebody may have over something that they don't like that they heard as opposed to possible litigation over people who do not feel like they are being treated equitably. You know, it's like we have to choose our heart. And so it's either the heart of sitting in the discomfort and learning things we may not want to learn, challenging ourselves, reaching deep to say, you know what? I don't really like that. KORI: When you talk to me about Christian privilege, this is a true story. Okay. True story. A [inaudible 00:46:22] of mine talked about Christian privilege. We're talking about something. She said, "Yeah, but there's also Christian privilege and people never talk about that." And can I admit to you that I was like, "Oh, is she for real? We're talking about racism and she's talking about Christian privilege." That was my initial reaction. But I sat with it. You know what? She was right. Because she was Pagan and I'm Christian. I've never had to use PTO for Christmas. My holidays are respected, they are recognized, they are centered, they are prioritized. But other people in this country who are not Christian do not have those privileges. Now that's a benign example because it's not one that makes people get as upset as some of the other topics. KORI: But the first step has to be a commitment to sit through the discomfort, sit through what may rub you wrong, and acknowledge that just because something is uncomfortable or just because something offends you does not mean the thing is wrong or it is offensive. And in many of our organizations, we haven't even gotten past that first part. Then the next part has to be a commitment to learn more. We have to operationalize being able to say to each other, tell me more, and not just, oh, I didn't like that training, or I didn't like what I was learning. But to say to yourself internally, okay, I didn't like that. But rather than projecting how I'm feeling it in this moment, I'm going to put myself in the position of saying, tell me more, help me understand why that bothered you, help me understand why you feel that way. Because until we're willing to do that, we're not going to learn. KORI: And without knowledge, we have no opportunity for growth. Growth comes with new knowledge. Growth comes with practicing new skill sets. Growth comes with trying things that you haven't done before. But if you're more invested in protecting the status quo than you are fighting for change, then the status quo will always win. And the status quo right now, it's not working for a lot of people from a lot of underrepresented and marginalized communities. Those are some of the things that have to happen. Oh, Chris, something else I want to add. Both sides. We got to talk about this both sides thing. Not every opinion and argument is equal, and that's something else that we're not willing to address head on. We've allowed inclusion to be so redefined that some people think it means anything and everything is of equal footing, right. KORI: But someone saying in the workplace, we need to be more inclusive of people with disabilities is not the same as someone saying, I don't think disabled people should have to work here. And sometimes what is crouching in is people want to hide behind inclusion to spew hate or bigotry or an excuse not to make the change and growth that is consistent with the so-called values of our organizations. I'll pause there because you're about [inaudible 00:50:05]. BREE: Yeah. I just want to comment to our listeners Kori's TED Talk, just in your browser, put in Kori Carew and TED Talk. I really encourage people to check it out. It is powerful and profound. So Kori, I'm going to ask you a question here that we also tend to ask this sometimes near the end, if you could look for, I don't know, five years or even a decade. If we can do a decent job around changing hearts and minds and attitudes around diversity, equity, inclusion, and belonging and wellbeing too, hopefully, how would the profession be different? What do you want to see? KORI: My goodness, my goodness, my goodness. Excuse me. That cough came up. If we could actually accomplish all these things that we've been talking about for 20 years, we would see leadership teams that are more humble in their approach, leadership teams that are people-centric, organizations that are listening to employees and actually care about what employees want. We would no longer be having conversations as if it's either you focus on the bottom line or you focus on employee happiness. Like we will understand that without happy employees who are engaged and doing fulfilling and meaningful work, we actually don't have a great bottom line to talk about. Right? Our organizations would look like inclusion and wellbeing and belonging, it's just part of the business strategy. It's not this separate siloed thing. It's not this thing that we talk about when we are worried about how the woman or the gays may react. Right. KORI: But it's just something that is operationalized into our values, into our competencies, into how we evaluate people, into how we promote people, and that we are constantly in humility, learning from each other. Right. So that even when somebody who's a chief inclusion and diversity officer, here's a phrase and someone says, "Did you realize that that was ableist?" That I would say, "I didn't. Tell me more." And once you tell me more, I changed my language, because we understand that we're always going to be moving. We're always going to be learning something new and there's always an opportunity to be better. And if we do that, we will also see different representation at all levels. We will actually have critical mass of diversity in our organizations. And then I would be unemployed. CHRIS: I was going to wrap up with this though, Kori, like if I was to serve up to you 500 managing partners, that were, again, I think one of the things that you've already mentioned is every individual in an organization is either additive or perhaps distracts from the culture that you're ultimately trying to create. A lot of the wellbeing discussion is about connecting and emphasizing wellbeing with decision makers and those who set the tone of organizations. And so my question to you is this, if I served up 500 managing partners of all sizes of firms around the country and they came and Kori was the keynote, what would be your message to them? KORI: My message to them would be that they are ridiculously in charge, that things happen in their organizations because they allow it, or they create it. And that by choosing to focus a hundred percent on their inclusive leadership skills and up in their ability to interrupt bias, to be culturally fluent, they could transform their organizations because where the leader goes, everyone else follows. BREE: Right. CHRIS: That's great. That's awesome. Well, again, Kori, you have certainly cultivated my curiosity, which I know is one of the things that you strongly advocate for. Couldn't be prouder to have you on the podcast and the sharing of your perspective. We got to get you more platforms for you to be able to shout loudly about these particular issues, because again, we got a lot of work to do, right. We know that there's a lot to be done in terms of realizing the potential of this profession, to realizing the potential of historically underrepresented and marginalized lawyers within our profession. Bree, I think that we all would agree that even as we pursue our wellbeing mission, that so much more has to be done on the diversity, equity, and inclusion perspective that integrates in the intersection there between those two that lanes need to merge in a much more substantive way. KORI: Thank you. CHRIS: Thank you, Kori. KORI: I appreciate it. I appreciate you having me. I appreciate you allowing Justin to come and hold my hand because she's my blinky today. I appreciate you inviting us to talk about what we're doing at Seyfarth and just my perspective as an individual separate from Seyfarth. Again, I've said this before, the work you're doing is so critically important. Thank you. Thank you. Thank you for everything that you do to promote wellbeing in the profession. So important. CHRIS: Awesome. Well, again, thanks for joining us. We will be back with the podcast probably in a couple weeks with our executive director, Jennifer DiSanza, which we are so excited to be having her join us as we talk about the future of where this movement is going. Thanks again, Kori. And to all our friends out there, we will be back in a couple weeks.
Natural disaster movies, anyone? It's what Steph's been into, and Chris has THOUGHTS on the drilling in Armageddon. Additionally, a chat around RuboCop RSpec rules happens, and they answer a listener's question, "how do you get acquainted with a new code base?" This episode is brought to you by BuildPulse (https://buildpulse.io/bikeshed). Start your 14-day free trial of BuildPulse today. Greenland (https://www.imdb.com/title/tt7737786/) Geostorm (https://www.imdb.com/title/tt1981128/) San Andreas (https://www.imdb.com/title/tt2126355/) Armageddon (https://www.imdb.com/title/tt0120591/) This episode is brought to you by Airbrake (https://airbrake.io/?utm_campaign=Q3_2022%3A%20Bike%20Shed%20Podcast%20Ad&utm_source=Bike%20Shed&utm_medium=website). Visit Frictionless error monitoring and performance insight for your app stack. Become a Sponsor (https://thoughtbot.com/sponsorship) of The Bike Shed! Transcript: AD: Flaky tests take the joy out of programming. You push up some code, wait for the tests to run, and the build fails because of a test that has nothing to do with your change. So you click rebuild, and you wait. Again. And you hope you're lucky enough to get a passing build this time. Flaky tests slow everyone down, break your flow, and make things downright miserable. In a perfect world, tests would only break if there's a legitimate problem that would impact production. They'd fail immediately and consistently, not intermittently. But the world's not perfect, and flaky tests will happen, and you don't have time to fix all of them today. So how do you know where to start? BuildPulse automatically detects and tracks your team's flaky tests. Better still, it pinpoints the ones that are disrupting your team the most. With this list of top offenders, you'll know exactly where to focus your effort for maximum impact on making your builds more stable. In fact, the team at Codecademy was able to identify their flakiest tests with BuildPulse in just a few days. By focusing on those tests first, they reduced their flaky builds by more than 68% in less than a month! And you can do the same because BuildPulse integrates with the tools you're already using. It supports all of the major CI systems, including CircleCI, GitHub Actions, Jenkins, and others. And it analyzes test results for all popular test frameworks and programming languages, like RSpec, Jest, Go, pytest, PHPUnit, and more. So stop letting flaky tests slow you down. Start your 14-day free trial of BuildPulse today. To learn more, visit buildpulse.io/bikeshed. That's buildpulse.io/bikeshed. CHRIS: Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Chris Toomey. STEPH: And I'm Steph Viccari. CHRIS: And together, we're here to share a bit of what we've learned along the way. So, Steph, what's new in your world? STEPH: Hey, Chris. So I've been watching more movies lately. So evenings aren't always great; I don't always feel good being around 33 weeks pregnant now. Evenings I can be just kind of exhausted from the day, and I just need to chill and prop my feet up and all that good stuff. And I've been really drawn to natural disaster like end-of-the-world-type movies, and I'm not sure what that says about me. But it's my truth; it's where I'm at. [chuckles] I watched Greenland recently, which I really enjoyed. I feel like they ended it well. I won't share any spoilers, but I feel like they ended it well. And they didn't take an easy shortcut out that I kind of thought that they might do, so that one was enjoyable. Geostorm, I watched that one just last night. San Andreas, I feel like that's one that I also watched recently. So yeah, that's what's new in my world, you know, your typical natural disaster end-of-the-world flicks. That's my new evening hobby. CHRIS: I feel like I haven't heard of any of the three that you just listed, which is wild to me because this is a category that I find enthralling. STEPH: Well, definitely start with Greenland. I feel like that one was the better of the three that I just mentioned. I don't know Geostorm or San Andreas which one you would prefer there. I feel like they're probably on par with each other in terms of like you're there for entertainment. We're not there to judge and be hypercritical of a storyline. You're there purely for the visual effects and for the ride. CHRIS: Gotcha. Interesting. So quick question then, since this seems like the category you're interested in, Armageddon or Deep Impact? STEPH: Ooh, I'm going to have to walk through the differences because I always get those mixed up. Armageddon is where they take Bruce Willis up to an asteroid, and they have to drill and drop a nuke, right? CHRIS: They sure do. STEPH: [laughs] And then what's Deep Impact about? I guess the fact that I know Armageddon better means I'm favoring that one. I can't place what...how does Deep Impact go? CHRIS: Deep Impact is just there's an asteroid coming, and it's the story and what the people do. So it's got less...it doesn't have the same pop. I believe Armageddon was a Michael Bay movie. And so it's got that Michael Bay special bit of something on it. But the interesting thing is they came out the same year; I want to say. It's one of those like Burger King and McDonald's being right next door to each other. It's like, what are you doing there? Why are you...like, asteroid devastation movies two of you at the same time, really? But yeah, Armageddon is the correct answer. Deep Impact is like a fine movie, but Armageddon is like, all right, we're going to have a movie about asteroids. Let's really go for it. Blow it out. Why not? STEPH: Yeah, I'm with you. Armageddon definitely sticks out in my memory, so I'd vote that one. Also, for your other question that you didn't ask, but you kind of implicitly asked, I'm going to go McDonald's because Burger King fries are trash, and also, McDonald's has better ice cream cones. CHRIS: Okay, so McDonald's fries. Oh no, I was thinking Wendy's, get a frosty from there, and then you make that combination because the frostys are great. STEPH: Oh yeah, that's a good combo. CHRIS: And you need the french fries to go with it, but then it's a third option that I'm introducing. Also, this wasn't a question, but I want to loop back briefly to Armageddon because it's an important piece of cinema. There's a really great...like it's DVD commentary, and it's Ben Affleck talking with Michael Bay about, "Hey, so in the movie, the premise is that the only way to possibly get this done is to train a bunch of oil drillers to be astronauts. Did we consider it all just having some astronauts learn to do oil drilling?" And Michael Bay's response is not safe for radio is how I would describe it. But it's very humorous hearing Ben Affleck describe Michael Bay responding to that. STEPH: I think they addressed that in the movie, though. They mentioned like, we're going to train them, but they're like, no, drilling is such an art and a science. There's no way. We don't have time to teach these astronauts how to drill. So instead, it's easier to teach them to be astronauts. CHRIS: Right. That is what they say in the movie. STEPH: [laughs] Okay. CHRIS: But just spending a minute teasing that one apart is like, being an astronaut is easy. You just sit in the spaceship, and it goes, boom. [laughs] It's like; actually, there's a little bit more to being an astronaut. Yes, drilling is very subtle science and art fusion. But the idea that being an astronaut [laughs] is just like, just push the go-to space button, then you go to space. STEPH: The training montage is definitely better if we get to watch people learn how to be astronauts than if we watch people learn how to drill. [laughs] So that might have also played a role. CHRIS: No question, it is the correct cinematic choice. But whether or not it's the true answer...say we were actually faced with this problem, I don't know that this is exactly how it would play out. STEPH: I think we should A/B test it. We'll have one group train to be drill experts and one group train to be astronauts, and we'll send them both up. CHRIS: This is smart. That's the way you got to do it. The one other thing that I'm going to go...you know what really grinds my gears? In the movie Armageddon, they have this robotic vehicle thing, the armadillo; I believe it's called. I know more than I thought I would remember about this movie. [chuckles] Anyway, continuing on, the armadillo, the vehicle that they use to do the drilling, has the drill arm on it that extends out and drills down into the asteroid. And it has gears on the end of it. It has three gears specifically. And the first gear is intermeshed with the second gear, which is intermeshed with the third gear, which is intermeshed with the first gear, so imagine which direction the first gear is turning, then imagine the second gear turning, then imagine the third gear turning. They can't. It's a physically impossible object. One tries to turn clockwise, and the other one is trying to go counterclockwise, and they're intermeshed. So the whole thing would just cease up. It just doesn't work. I've looked at it a bunch of times, and I want to just be wrong about this. I want to be like; I don't know what's going on. But I think the gears on the drilling machine just fundamentally at a very simple mechanical level cannot work. And again, if you're going to do it, really go for it, Michael Bay. I kind of like that, and I really hate it at the same time. STEPH: I have never noticed this. I'm intrigued. You know what? Maybe Armageddon will be the movie of choice tonight. [chuckles] Maybe that's what I'm going to watch. And I'm going to wait for the armadillo to come out so I can evaluate the gears. And I'm highly amused that this is the thing that grinds your gears are the gears on the armadillo. CHRIS: Yeah. I was a young child at the time, and I remember I actually went to Disney World, and I saw they had the prop vehicle there. And I just kind of looked up at it, and I was like, no, that's not how gears work. I may have been naive and wrong as a child, and now I've just anchored this memory deep within me. In a similar way, so I had a moment while traveling; actually, that reminded me of something that I said on a recent podcast episode where I was talking about names and pronunciation. And I was like, yeah, sometimes people ask me how to pronounce my name. And I can't imagine any variation. That was the thing I was just wrong about because 'Toomay' is a perfectly reasonable pronunciation of my name that I didn't even think... I was just so anchored to the one truth that I know in the world that my name is Toomey. And that's the only possible way anyone could pronounce it. Nope, totally wrong. So maybe the gears in Armageddon actually work really, really well, and maybe I'm just wrong. I'm willing to be wrong on the internet, which I believe is the name of the first episode that we recorded with you formally as a co-host. [chuckles] So yeah. STEPH: Yeah, that sounds true. So you're going to change the intro? It's now going to be like, and I'm Chris 'Toomay'. CHRIS: I might change it each time I come up with a new subtle pronunciation. We'll see. So far, I've got two that I know of. I can't imagine a third, but I was wrong about one. So maybe I'm wrong about two. STEPH: It would be fun to see who pays attention. As someone who deeply values pronouncing someone's name correctly, oh my goodness, that would stress me out to hear someone keep pronouncing their name differently. Or I would be like, okay, they're having fun, and they don't mind how it gets pronounced. I can't remember if we've talked about this on air but early on, I pronounced my last name differently for like one of the first episodes that we recorded. So it's 'Vicceri,' but it could also be 'Viccari'. And I've defaulted at times to saying 'Viccari' because people can spell that. It seems more natural. They understand it's V-I-C-C-A-R-I. But if I say 'Vicceri', then people want to add two Rs, or they want a Y. I don't know why it just seems to have a difference. And so then I was like, nope, I said it wrong. I need to say it right. It's 'Vicceri' even if it's more challenging for people. And I think Chad Pytel had just walked in at that moment when I was saying that to you that I had said my name differently. And he's like, "You can't do that." And I'm like, "Well, I did it. It's already out there in the world." [laughs] But also, I'm one of those people that's like, Viccari, 'Vicceri' I will accept either. In a slightly different topic and something that's going on in my world, there was a small win today with a client team that I really appreciated where someone brought up the conversation around the RuboCop RSpec rules and how RuboCop was fussing at them because they had too many lines in their test example. And so they're like, well, they're like, I feel like I'm competing, or I'm working against RuboCop. RuboCop wants me to shorten my test example lines, but yet, I'm not sure what else to do about it. And someone's like, "Well, you could extract more into before blocks and to lets and to helpers or things like that to then shorten the test. They're like, "But that does also work against readability of the test if you do that." So then there was a nice, short conversation around well, then we really need more flexibility. We shouldn't let the RuboCop metrics drive us in this particular decision when we really want to optimize for readability. And so then it was a discussion of okay, well, how much flexibility do we add to it? And I was like, "Well, what if we just got rid of it? Because I don't think there's an ideal length for how long your test should be. And I'd rather empower test authors to use all the space that they need to show their test setup and even lean into duplication before they extract things because this codebase has far more dry tests than they do duplication concerns. So I'd rather lean into the duplication at this point." And the others that happened to be in that conversation were like, "Yep, that sounds good." So then that person issued a PR that then removed the check for that particular; how long are the examples? And it was lovely. It was just like a nice, quick win and a wonderful discussion that someone had brought up. CHRIS: Ooh, I like that. That sounds like a great conversation that hit on why do we have this? What are the trade-offs? Let's actually remove it. And it's also nice that you got to that place. I've seen a lot of folks have a lot of opinions in the past in this space. And opinions can be tricky to work around, and just deeply, deeply entrenched opinions is the thing that I find interesting. And I think I'm increasingly in the space of those sort of, thou shalt not type linter rules are not ideal in my mind. I want true correctness checks that really tell some truth about the codebase. Like, we still don't have RuboCop on our project at Sagewell. I think that's true. Yeah, that's true. We have ESLint, but it's very minimal, what we have configured. And they more are in the what we deem to be true correctness checks, although that is a little bit of a blurry line there. But I really liked that idea. We turn on formatters. They just do the thing. We're not allowed to discuss the formatting, with the exception of that time that everybody snuck in and switched my 80-line length to a 120-line length, but I don't care. I'm obviously not still bitter about it. [chuckles] And then we've got a very minimal linting layer on top of that. But like TypeScript, I care deeply, and I think I've talked in previous episodes where I'm like, dial up the strictness to 14 because TypeScript tends to tell me more truths I find, even though I have to jump through some hoops to be like TypeScript, I know that this is fine, but I can't prove it. And TypeScript makes me prove it, which I appreciate about it. I also really liked the way you referred to RSpec's feedback to you was that RSpec was fussing at you. That was great. I like that. I'm going to internalize that. Whenever a linter or type system or anything like that when they tell me no, I'm going to be like, stop fussing, nope, nope. [chuckles] STEPH: I don't remember saying that, but I'm going to trust you that that's what I said. That's just my true southern self coming through on the mic, fussing, and then go get a biscuit, and it'll just be a delightful day. CHRIS: So if I give RuboCop a biscuit, it will stop fussing at me, potentially? STEPH: No, the biscuit is just for you. You get fussed at; you go get a biscuit. It makes you feel better, and then you deal with the fussing. CHRIS: Sold. STEPH: Fussing and cussing, [laughs] that's most of my work life lately, fussing and cussing. [laughs] CHRIS: And occasional biscuits, I hope. STEPH: And occasional biscuits. You got it. But that's what's new in my world. What's going on in your world? CHRIS: Let's see. In my world, it's a short week so far. So recording on Wednesday, Monday was a holiday. And I was out all last week, which very much enjoyed my vacation. It was lovely. Went over to Europe, hung out there for a bit, some time in Paris, some time in Amsterdam, precious little time on a computer, which is very rare for me. So it was very enjoyable. But yeah, back now trying to just get back into the swing of things. Thankfully, this turned out to be a really great time to step away from the work for a little while because we're still in this calm before the storm but in a good way is how I would describe it. We have a major facet of the Sagewell platform that we are in the planning modes for right now. But we need to get a couple of different considerations, pick a partner vendor, et cetera, that sort of thing. So right now, we're not really in a position to break ground on what we know will be a very large body of work. We're also not taking on anything else too big. We're using this time to shore up a lot of different things. As an example, one of the fun things that we've done in this period of time is we have a lot of webhooks in the app, like a lot of webhooks coming into the app, just due to the fact that we're an integration of a lot of services under the hood. And we have a pattern for how we interact with and process, so we actually persist the webhook data when they come in. And then we have a background job that processes and watch our pattern to make sure we're not losing anything and the ability to verify against our local version, and the remote version, a bunch of different things. Because turns out webhooks are critical to how our app works. And so that's something that we really want to take very seriously and build out how we work with that. I think we have eight different webhook integrations right now; maybe it's more. It's a lot. And with those, we've implemented the same pattern now eight times; I want to say. And in squinting at it from a distance, we're like; it is indeed identically the same pattern in all eight cases or with the tiniest little variation in one of them. And so we've now accepted like, okay, that's true. So the next one of them that we introduced, we opted to do it in a generic way. So we introduced the abstraction with the next iteration of this thing. And now we're in a position...we're very happy with what we ended up with there. It's like the best of all of the other versions of it. And now, the plan will be to slowly migrate each of the existing ones to be no longer a unique special version of webhook processing but use the generic webhook processing pattern that we have in the app. So that's nice. I feel good about how long we waited as well because it's like, we have webhooks. Let's introduce the webhook framework to rule them all within our app. It's like, no, wait until you see. Check and make sure they are, in fact, the same and not just incidental duplication. STEPH: I appreciate that so much. That's awesome. That sounds like a wonderful use of that in-between state that you're in where you still got to make progress but also introduce some refactoring and a new concept. And I also appreciate how long you waited because that's one of those areas where I've just learned, like, just wait. It's not going to hurt you. Just embrace the duplication and then make sure it's the right thing. Because even if you have to go in and update it in a couple of places, okay, sure, that feels a little tedious, but it feels very safe too. If it doesn't feel safe...I could talk myself back and forth on this one. If it doesn't feel safe, that's a different discussion. But if you're going through and you have to update something in a couple of different places, that's quick. And sure, you had to repeat yourself a little bit, but that's fine. Versus if you have two or three of something and you're like, oh, I immediately must extract. That's probably going to cause more pain than it's worth at this point. CHRIS: Yeah, exactly, exactly that. And we did get to that place where we were starting to feel a tiny bit of pain. We had a surprising bit of behavior that when we looked at it, we were like, oh, that's interesting, because of how we implemented the webhook pattern, this is happening. And so then we went to fix it, but we were like, oh, it would actually be really nice to have this fixed across everything. We've had conversations about other refinements, enhancements, et cetera; that we could do in this space. That, again, would be really nice to be able to do holistically across all of the different webhook integration things that we have. And so it feels like we waited the right amount of time. But then we also started to...we're trying to be very responsive to the pressure that the system is pushing back on us. As an aside, the crispy Brussels snack hour and the crispy Brussels work lunch continue to be utterly fantastic ways in which we work. For anyone that is unfamiliar or hasn't listened to episodes where I rambled about those nonsense phrases that I just said, they're basically just structured time where the engineering team at Sagewell looks at and discusses higher-level architecture, refactoring, developer experience, those sort of things that don't really belong on the core product board. So we have a separate place to organize them, to gather them. And then also, we have a session where we vote on them, decide which ones feel important to take on but try and make sure we're being intentional about how much of that work we're taking on relative to how much of core product work and try and keep sort of a good ratio in between the two. And thus far, that's been really fantastic and continues to be, I think, really effective. And also the sort of thing that just keeps the developer team really happy. So it's like, I'm happy to work in this system because we know we have a way to change it and improve it where there's pain. STEPH: I like the idea of this being a game show where it's like refactor island, and everybody gets together and gets to vote which refactor stays or gets booted off the island. I'm also going to go back and qualify something I said a moment ago, where if something feels safe in terms of duplication, where it starts to feel unsafe is if there's like an area that you forgot to update because you didn't realize it's duplicated in several areas and then that causes you pain. Then that's one of those areas where I'll start to say, "Okay, let's rethink the duplication and look to dry this up." CHRIS: Yep, indeed. It's definitely like a correction early on in my career and overcorrection back and trying to find that happy medium place. But as an aside, just throwing this out there, so webhooks are an interesting space. I wish it were a more commoditized offering of platforms. Every vendor that we're integrating with that does webhooks does it slightly differently. It's like, "Oh, do you folks have retries?" They're like, "No." It's like, oh, what do you mean no? I would love it if you had retries because, I don't know, we might have some reason to not receive one of them. And there's polling, and there are lots of different variations. But the one thing that I'm surprised by is that webhook signing I don't feel like people take it serious enough. It is a case where it's not a huge security vulnerability in your app. But I was reading someone who is a security analyst at one point. And they were describing sort of, I've done tons of in-the-code audits of security practices, and here are the things that I see. And so it's the normal like OWASP Top 10 Cross-Site Request Forgery, and SQL injection, and all that kind of stuff. But one of the other ones he highlighted is so often he finds webhooks that are not verified in any way. So it's just like anyone can post data into the system. And if you post it in the right shape, the system's going to do some stuff. And there's no way for the external system to enforce that you properly validate and verify a webhook coming in, verify that payload. It's an extra thing where you do the checksum math and whatnot and take the signature header. I've seen somewhere they just don't provide it. And it's like, what do you mean you don't provide it? You must provide it, please. So it's either have an API key so that we have some way to verify that you are who you say you are or add a signature, and then we'll calculate it. And it's a little bit of a dance, and everybody does it different, but whatever. But the cases where they just don't have it, I'm like, I'm sorry, what now? You're going to say whom? But yeah, then it's our job to definitely implement that. So this is just a notice out there to anyone that's listening. If you got a bunch of webhook handling code in your app, maybe spot-check that you're actually verifying the payloads because it's possible that you're not. And that's a weird, very open hole in the side of your application. STEPH: That's a really great point. I have not worked with webhooks recently. And in the past, I can't recall if that's something that I've really looked at closely. So I'm glad you shared that. CHRIS: It's such an easy thing to skip. Like, it's one of those things that there's no way to enforce it. And so, I'd be interested in a survey that can't be done because this is all proprietary data. But what percentage of webhook integrations are unverified? Is it 50%? Is it 10%? Is it 100%? It's definitely not 100. But it's somewhere in there that I find interesting. It's not a terribly exploitable vulnerability because you have to have deep knowledge of the system. In order to take advantage of it, you need to know what endpoint to hit to, what shape of data to send because otherwise, you're probably just going to cause an error or get a bunch of 404s. But like, it's, I don't know, it's discoverable. And yeah, it's an interesting one. So I will hop off my webhook soapbox now, but that's a thought. MIDROLL AD: Debugging errors can be a developer's worst nightmare...but it doesn't have to be. Airbrake is an award-winning error monitoring, performance, and deployment tracking tool created by developers for developers, that can actually help you cut your debugging time in half. So why do developers love Airbrake? Well, it has all of the information that web developers need to monitor their application - including error management, performance insights, and deploy tracking! Airbrake's debugging tool catches all your project errors, intelligently groups them, and points you to the issue in the code so you can quickly fix the bug before customers are impacted. In addition to stellar error monitoring, Airbrake's lightweight APM enables developers to track the performance and availability of their application through metrics like HTTP requests, response times, error occurrences, and user satisfaction. Finally, Airbrake Deploy Tracking helps developers track trends, fix bad deploys, and improve code quality. Since 2008, Airbrake has been a staple in the Ruby community and has grown to cover all major programming languages. Airbrake seamlessly integrates with your favorite apps and includes modern features like single sign-on and SDK-based installation. From testing to production, Airbrake notifiers have your back. Your time is valuable, so why waste it combing through logs, waiting for user reports, or retrofitting other tools to monitor your application? You literally have nothing to lose. So head on over to airbrake.io/try/bikeshed to create your FREE developer account today! CHRIS: But now that I'm off my soapbox, I believe we have a topic that was suggested. Do you want to provide a little bit of context here, Steph? STEPH: Yeah, I'd love to. So this came up when I was having a conversation with another thoughtboter. And given that we change projects fairly frequently, on the Boost team, we typically change projects around every six months. They asked a really thoughtful question that was "How do you get acquainted with a new codebase? So given that you're changing projects so often, what are some of the tips and tricks for ways that you've learned to then quickly get up to speed with a new codebase?" Because, frankly, that is one of the thoughtbot superpowers is that we are really good at onboarding each other and then also getting up to speed with a new team, and their processes, and their codebase. So I have a couple of ideas, and then I'd love to hear some of your thoughts as well. So I'll dive in with a couple. So the first one, this one's frankly my favorite. Like day one, if there's a team where I'm joining and they have someone that can walk me through the application from the users' perspective, maybe it's someone that's in sales, or maybe it's someone on the product team, maybe it's a recording that they've already done for other people, but that's my first and favorite way to get to know an application. I really want to know what are users experience as they're going through this app? That will help me focus on the more critical areas of the application based on usage. So if that's available, that's fabulous. I'm also going to tailor a lot of this more to like a Rails app since that's typically the type of project that I'm onboarding to. So the other types of questions that I like to find answers to are just like, what's my top-level structure? Like to look through the app and see how are things organized? Chris, you've mentioned in a previous episode where you have your client structure that then highlights all the third-party clients that you're working with. Are we using engines in the app? Is there anything that seems a bit more unique to that application that I'm going to want to brush up on or look into? What's the test coverage like? Do they have something that's already highlighting how much test coverage they have? If not, is there something that then I can run locally that will then show me that test coverage? I also really like to look at the routes file. That's one of my other favorite places because that also is very similar to getting an overview of the product. I get to see more from the user perspective. What are the common resources that people are going to, and what are the domain topics that I'm working with in this new application? I've got a couple more, but I'm going to pause there and see how you get acquainted with a new app. CHRIS: Well, unsurprisingly, I agree with all of those. We're still searching for that dare to disagree beyond Pop-Tarts and IPAs situation. To reiterate or to emphasize some of the points you made, the sales demo thing? I absolutely love that one because, yes, absolutely. What's the most customer-centric point of view that I can have? Can I then login to a staging version of the site so I can poke around and hopefully not break anything or move real money or anything like that? But understanding why is this thing, not in code, but in actual practical, observable, intractable software? Beyond that, your point about the routes, absolutely, that's one of my go-to's, although the routes there often is so much in the routes, and it's like some of those may actually be unused. So a corollary to the routes where available if there's an APM tool like Scout, or New Relic, or something like that, taking a look at that and seeing what are the heavily trafficked endpoints within this app? I like to think about it as the entry points into this codebase. So the routes file enumerates all of them, but some of them matter, and some of them don't. And so, an APM tool can actually tell you which are the ones that are seeing a ton of traffic. That's a really interesting question for me. Similarly, if we're on Heroku, I might look is there a scheduler? And if so, what are the tasks that are running in the background? That's another entry point into the app. And so I like to think about it from that idea of entry points. If it's not on Heroku, and then there's some other system, like, I've used Cronic. I think it's Cronic, Whenever the Cron thing. Whenever, that's what it is, the Whenever gem that allows you to implement that, but it's in a file within the codebase, which as an aside, I really love that that's committed and expressive in the code. Then that's another interesting one to see. If it's more exotic than that, I may have to chase it down or ask someone, but I'll try and find what are all of the entry points and which are the ones that matter the most? I can drill down from there and see, okay, what code then supports these entry points into the application? I want to give an answer that also includes something like, oh, I do fancy static analysis in the codebase, and I do a churn versus complexity graph, and I start to...but I never do that, if we're being honest. The thing that I do is after that initial cursory scan of the landscape, I try and work on something that is relatively through the layers of the app, so not like, oh, I'll fix the text in a button. But like, give me something weird and ideally, let me pair with someone and then try and move through the layers of the app. So okay, here's our UI. We're rendering in this way. The controllers are integrated in this way, et cetera. This is our database. Try and get through all the layers if possible to try and get as holistic of a view of how the application works. The other thing that I think is really interesting about what you just said is you're like, I'm going to give some answers that are somewhat specific to a Rails app. And that totally makes sense to me because I know how to answer this in the context of a Rails app because those organizational patterns are so useful that I can hop into different Rails apps. And I've certainly seen ones that I'm like, this is odd and unfamiliar to me, but most of them are so much more discoverable because of that consistency. Whereas I have worked on a number of React apps, and every single one I come into, I'm like, okay, wait, what are we doing? How are we doing state management? What's the routing like? Are we server-side rendering, are we not? And it is a thing that...I see that community really moving in the direction of finding the meta frameworks that stitch the pieces together and provide more organizational structure and answer more of the questions out of the box. But it continues to be something that I absolutely love about Rails is that Rails answers so many of the questions for me. New people joining the team are like, oh, it's a Rails app, cool. I know how to Rails, and we get to run with that. And so that's more of a pitch for Rails than an answer to the question, but it is a thing that I felt in answering this question. [laughs] But yeah, those are some thoughts. But interested, it sounds like you had some more as well. I would love to hear what else was in your mind when you were thinking about this. STEPH: I do. And I want to highlight you said some really wonderful things. One that really stuck out to me that I had not considered is using Scout APM to look at heavily-trafficked endpoints. I have that on my list in regards as something that I want to know what's my error tracking, observability. Like, if I break something or if you give me a bug ticket to work on, what am I going to use? How am I going to understand what's going wrong? But I hadn't thought of it in terms of seeing which endpoints are heavily used. So I really liked that one. I also liked how you highlighted that you wish you'd do something fancy around doing a churn versus complexity kind of graph because I thought of that too. I was like, oh, that would be such a nice answer. But the truth is I also don't do that. I think it's all those things. I think it would be fun to make it easy. So I do that with new applications. But I agree; I typically more just dive in like, hey, give me a ticket. Let me go from there. I might do some simple command-line checking. So, for example, if I want to look through app models, let's find out which model is the largest. I may look for that to see do we have a God object or something like that? So I may look there. I just want to know how long are some of these files? But I also don't use a particular tool for that churn versus complexity. CHRIS: I think you hit the nail on the head with like, I wish that were easier or more in our toolset. But here on The Bike Shed, we tell the truth. And that is aspirational code flexing that we do not yet have. But I agree, that would be a really nice way to explore exactly what you're describing of, like, who are the God models? I'll definitely do that check, but not some of the more subtle and sophisticated show me the change over time of all these...like nah, that's not what I'm doing, much as I would like to be able to answer that way. STEPH: But it also feels like one of those areas like, it would be nice, but I would be intrigued to see how much I use that. That might be a nice anecdote to have. But I find the diving into the codebase to be more fruitful because I guess it depends on what I'm really looking at. Am I looking to see how complicated of a codebase this is? Because then I need to give more of a high-level review to someone to say how long I think it's going to take for me to work on a particular feature or before I'm joining a team, like, who do I think are good teammates that would then enjoy working on this application? That feels like a very different question to me versus the I'm already part of the team. I'm here. We're going to have complexity and churn. So I can just learn some of that over time. I don't have to know that upfront. Although it may be nice to just know at a high level, say like, okay, if I pick up a ticket, and then I look at that churn and complexity, to be like, okay, my ticket falls right smack-dab in the middle of that. So it's going to be a fun first week. That could be a fun fact. But otherwise, I'm not sure. I mean, yeah, I'd be intrigued to see how much it helps me. One other place that I do browse is I go to the gem file. I'm just always curious, what do people have in their tool bag? I want to see are there any gems that have been pulled in that are helping the team process some deprecated behavior? So something that's been pulled out of Rails but then pulled into a separate gem. So then that way, they don't have to upgrade just yet, or they can upgrade but then still keep some of that existing old deprecated behavior. That kind of stuff is interesting to me. And also, you called it earlier pairing. That's my other favorite way. I want to hear how people talk about the codebase, how they navigate. What are they frustrated by? What brings them joy? All of that is really helpful too. I think that covers all the ways that I immediately will go to when getting acquainted with a new codebase. CHRIS: I think that covers most of what I have in mind, although the question is framed in an interesting way that I think really speaks to the consultant mindset. How do I get acquainted with a new codebase? But if you take the question and flip it around sort of 180 degrees, I think the question can be reframed as how does an organization help people onboard into a codebase? And so everything we just described are like, here's what I do, here's how I would go about it, and pairing starts to get to collaboration. I think we've talked in a number of episodes about our thoughts on onboarding and being intentional with that, pairing people up. A lot of things we described it's like, it's ideal actually if the organization is pushing this. And you and I both worked as consultants for long enough that we're really in the mindset of like, all right, let's assume I'm just showing up. There's no one else there. They give me a laptop and no documentation and no other humans I'm allowed to talk to. How do I figure this out and get the next feature out to production? And ideally, it's something slightly better than that that we experience, but we're ready for whatever it is. Versus, most people are working within the context of an organization for a longer period of time. And most organizations should be thinking about it from the perspective of how do I help the new hires come into this codebase and become effective as quickly as possible? And so I think a lot of what we said can just be flipped around and said from the other way, like, pair them up, put them on a feature early, give them a walkthrough of the codebase, give them a sales-centric demo. Yeah, I feel equally about those things when said from the other side, but I do want to emphasize that this shouldn't be you're out there in the middle of the jungle with only a machete, and you got to figure out this codebase. Ideally, the organization is actually like, no, no, we'll help you. It's ours, so we know it. We can help you find the weird stuff. STEPH: That's a really nice distinction, though, because you're right; I hadn't really thought about this. I was thinking about this from more of the perspective of you're out in the jungle with a machete, minus we did mention pairing in there [laughs] and maybe a demo. I was approaching it more from you're isolated or more solo and then getting accustomed to the codebase versus if you have more people to lean on. But then that also makes me think of all the other processes that I didn't mention that I would include in that onboarding that you're speaking of, of like, how does this team work in terms of where do I push my code? What hooks are going to run? And then what do I wait for? How many people need to review my code? There are all those process-y questions that I think would ideally be included on the onboarding. But that has happened before, I mean, where we've joined projects, and it's been like, okay, good luck. Let us know if you need anything. And so then you do need those machete skills to then start hacking away. [laughs] CHRIS: We've been burned before. STEPH: They come in handy. [laughs] So when you are in that situation, and there's a comet that's coming to destroy earth, and there's a Rails application that is preventing this big doomsday, the question is, do you take astronauts and train them to be Rails experts, or do you take Rails developers and train them to be astronauts? I think that's the big question. CHRIS: What would Michael Bay do? STEPH: On that note, shall we wrap up? CHRIS: Let's wrap up. The show notes for this episode can be found at bikeshed.fm. STEPH: This show is produced and edited by Mandy Moore. CHRIS: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review on iTunes, as it really helps other folks find the show. STEPH: If you have any feedback for this or any of our other episodes, you can reach us at @_bikeshed or reach me on Twitter @SViccari. CHRIS: And I'm @christoomey. STEPH: Or you can reach us at email@example.com via email. CHRIS: Thanks so much for listening to The Bike Shed, and we'll see you next week. ALL: Byeeeeeeeeee!!!!!!!! ANNOUNCER: This podcast was brought to you by thoughtbot. thoughtbot is your expert design and development partner. Let's make your product and team a success.
About ChrisChris Short has been a proponent of open source solutions throughout his over two decades in various IT disciplines, including systems, security, networks, DevOps management, and cloud native advocacy across the public and private sectors. He currently works on the Kubernetes team at Amazon Web Services and is an active Kubernetes contributor and Co-chair of OpenGitOps. Chris is a disabled US Air Force veteran living with his wife and son in Greater Metro Detroit. Chris writes about Cloud Native, DevOps, and other topics at ChrisShort.net. He also runs the Cloud Native, DevOps, GitOps, Open Source, industry news, and culture focused newsletter DevOps'ish.Links Referenced: DevOps'ish: https://devopsish.com/ EKS News: https://eks.news/ Containers from the Couch: https://containersfromthecouch.com opengitops.dev: https://opengitops.dev ChrisShort.net: https://chrisshort.net Twitter: https://twitter.com/ChrisShort TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Coming back to us since episode two—it's always nice to go back and see the where are they now type of approach—I am joined by Senior Developer Advocate at AWS Chris Short. Chris, been a few years. How has it been?Chris: Ha. Corey, we have talked outside of the podcast. But it's been good. For those that have been listening, I think when we recorded I wasn't even—like, when was season two, what year was that? [laugh].Corey: Episode two was first pre-pandemic and the rest. I believe—Chris: Oh. So, yeah. I was at Red Hat, maybe, when I—yeah.Corey: Yeah. You were doing Red Hat stuff, back when you got to work on open-source stuff, as opposed to now, where you're not within 1000 miles of that stuff, right?Chris: Actually well, no. So, to be clear, I'm on the EKS team, the Kubernetes team here at AWS. So, when I joined AWS in October, they were like, “Hey, you do open-source stuff. We like that. Do more.” And I was like, “Oh, wait, do more?” And they were like, “Yes, do more.” “Okay.”So, since joining AWS, I've probably done more open-source work than the three years at Red Hat that I did. So, that's kind of—you know, like, it's an interesting point when I talk to people about it because the first couple months are, like—you know, my friends are like, “So, are you liking it? Are you enjoying it? What's going on?” And—Corey: Do they beat you with reeds? Like, all the questions people have about companies? Because—Chris: Right. Like, I get a lot of random questions about Amazon and AWS that I don't know the answer to.Corey: Oh, when I started telling people, I fixed Amazon bills, I had to quickly pivot that to AWS bills because people started asking me, “Well, can you save me money on underpants?” It's I—Chris: Yeah.Corey: How do you—fine. Get the prime credit card. It docks 5% off the bill, so there you go. But other than that, no, I can't.Chris: No.Corey: It's—Chris: Like, I had to call my bank this morning about a transaction that I didn't recognize, and it was from Amazon. And I was like, that's weird. Why would that—Corey: Money just flows one direction, and that's the wrong direction from my employer.Chris: Yeah. Like, what is going on here? It shouldn't have been on that card kind of thing. And I had to explain to the person on the phone that I do work at Amazon but under the Web Services team. And he was like, “Oh, so you're in IT?”And I'm like, “No.” [laugh]. “It's actually this big company. That—it's a cloud company.” And they're like, “Oh, okay, okay. Yeah. The cloud. Got it.” [laugh]. So, it's interesting talking to people about, “I work at Amazon.” “Oh, my son works at Amazon distribution center,” blah, blah, blah. It's like, cool. “I know about that, but very little. I do this.”Corey: Your son works in Amazon distribution center. Is he a robot? Is normally my next question on that? Yeah. That's neither here nor there.So, you and I started talking a while back. We both write newsletters that go to a somewhat similar audience. You write DevOps'ish. I write Last Week in AWS. And recently, you also have started EKS News because, yeah, the one thing I look at when I'm doing these newsletters every week is, you know what I want to do? That's right. Write more newsletters.Chris: [laugh].Corey: So, you are just a glutton for punishment? And, yeah, welcome to the addiction, I suppose. How's it been going for you?Chris: It's actually been pretty interesting, right? Like, we haven't pushed it very hard. We're now starting to include it in things. Like we did Container Day; we made sure that EKS news was on the landing page for Container Day at KubeCon EU. And you know, it's kind of just grown organically since then.But it was one of those things where it's like, internally—this happened at Red Hat, right—when I started live streaming at Red Hat, the ultimate goal was to do our product management—like, here's what's new in the next version thing—do those live so anybody can see that at any point in time anywhere on Earth, the second it's available. Similar situation to here. This newsletter actually is generated as part of a report my boss puts together to brief our other DAs—or developer advocates—you know, our solutions architects, the whole nine yards about new EKS features. So, I was like, why can't we just flip that into a weekly newsletter, you know? Like, I can pull from the same sources you can.And what's interesting is, he only does the meeting bi-weekly. So, there's some weeks where it's just all me doing it and he ends up just kind of copying and pasting the newsletter into his document, [laugh] and then adds on for the week. But that report meeting for that team is now getting disseminated to essentially anyone that subscribes to eks.news. Just go to the site, there's a subscribe thing right there. And we've gotten 20 issues in and it's gotten rave reviews, right?Corey: I have been a subscriber for a while. I will say that it has less Chris Short personality—Chris: Mm-hm.Corey: —to it than DevOps'ish does, which I have to assume is by design. A lot of The Duckbill Group's marketing these days is no longer in my voice, rather intentionally, because it turns out that being a sarcastic jackass and doing half-billion dollar AWS contracts can not to be the most congruent thing in the world. So okay, we're slowly ameliorating that. It's professional voice versus snarky voice.Chris: Well, and here's the thing, right? Like, I realized this year with DevOps'ish that, like, if I want to take a week off, I have to do, like, what you did when your child was born. You hired folks to like, do the newsletter for you, or I actually don't do the newsletter, right? It's binary: hire someone else to do it, or don't do it. So, the way I structured this newsletter was that any developer advocate on my team could jump in and take over the newsletter so that, you know, if I'm off that week, or whatever may be happening, I, Chris Short, am not the voice. It is now the entire developer advocate team.Corey: I will challenge you on that a bit. Because it's not Chris Short voice, that's for sure, but it's also not official AWS brand voice either.Chris: No.Corey: It is clearly written by a human being who is used to communicating with the audience for whom it is written. And that is no small thing. Normally, when oh, there's a corporate newsletter; that's just a lot of words to say it's bad. This one is good. I want to be very clear on that.Chris: Yeah, I mean, we have just, like, DevOps'ish, we have sections, just like your newsletter, there's certain sections, so any new, what's new announcements, those go in automatically. So, like, that can get delivered to your inbox every Friday. Same thing with new blog posts about anything containers related to EKS, those will be in there, then Containers from the Couch, our streaming platform, essentially, for all things Kubernetes. Those videos go in.And then there's some ecosystem news as well that I collect and put in the newsletter to give people a broader sense of what's going on out there in Kubernetes-land because let's face it, there's upstream and then there's downstream, and sometimes those aren't in sync, and that's normal. That's how Kubernetes kind of works sometimes. If you're running upstream Kubernetes, you are awesome. I appreciate you, but I feel like that would cause more problems and it's worse sometimes.Corey: Thank you for being the trailblazers. The rest of us can learn from your misfortune.Chris: [laugh]. Yeah, exactly. Right? Like, please file your bugs accordingly. [laugh].Corey: EKS is interesting to me because I don't see a lot of it, which is, probably, going to get a whole lot of, “Wait, what?” Moments because wait, don't you deal with very large AWS bills? And I do. But what I mean by that is that EKS, until you're using its Fargate expression, charges for the control plane, which rounds to no money, and the rest is running on EC2 instances running in a company's account. From the billing perspective, there is no difference between, “We're running massive fleets of EKS nodes.” And, “We're managing a whole bunch of EC2 instances by hand.”And that feels like an interesting allegory for how Kubernetes winds up expressing itself to cloud providers. Because from a billing perspective, it just looks like one big single-tenant application that has some really strange behaviors internally. It gets very chatty across AZs when there's no reason to, and whatnot. And it becomes a very interesting study in how to expose aspects of what's going on inside of those containers and inside of the Kubernetes environment to the cloud provider in a way that becomes actionable. There are no good answers for this yet, but it's something I've been seeing a lot of. Like, “Oh, I thought you'd be running Kubernetes. Oh, wait, you are and I just keep forgetting what I'm looking at sometimes.”Chris: So, that's an interesting point. The billing is kind of like, yeah, it's just compute, right? So—Corey: And my insight into AWS and the way I start thinking about it is always from a billing perspective. That's great. It's because that means the more expensive the services, the more I know about it. It's like, “IAM. What is that?” Like, “Oh, I have no idea. It's free. How important could it be?” Professional advice: do not take that philosophy, ever.Chris: [laugh]. No. Ever. No.Corey: Security: it matters. Oh, my God. It's like you're all stars. Your IAM policy should not be. I digress.Chris: Right. Yeah. Anyways, so two points I want to make real quick on that is, one, we've recently released an open-source project called Carpenter, which is really cool in my purview because it looks at your Kubernetes file and says, “Oh, you want this to run on ARM instance.” And you can even go so far as to say, right, here's my limits, and it'll find an instance that fits those limits and add that to your cluster automatically. Run your pod on that compute as long as it needs to run and then if it's done, it'll downsize—eventually, kind of thing—your cluster.So, you can basically just throw a bunch of workloads at it, and it'll auto-detect what kind of compute you will need and then provision it for you, run it, and then be done. So, that is one-way folks are probably starting to save money running EKS is to adopt Carpenter as your autoscaler as opposed to the inbuilt Kubernetes autoscaler. Because this is instance-aware, essentially, so it can say, like, “Oh, your massive ARM application can run here,” because you know, thank you, Graviton. We have those processors in-house. And you know, you can run your ARM64 instances, you can run all the Intel workloads you want, and it'll right size the compute for your workloads.And I'll look at one container or all your containers, however you want to configure it. Secondly, the good folks over at Kubecost have opencost, which is the open-source version of Kubecost, basically. So, they have a service that you can run in your clusters that will help you say, “Hey, maybe this one notes too heavy; maybe this one notes too light,” and you know, give you some insights into Kubernetes spend that are a little bit more granular as far as usage and things like that go. So, those two projects right there, I feel like, will give folks an optimal savings experience when it comes to Kubernetes. But to your point, it's just compute, right? And that's really how we treat it, kind of, here internally is that it's a way to run… compute, Kubernetes, or ECS, or any of those tools.Corey: A fairly expensive one because ignoring entirely for a second the actual raw cost of compute, you also have the other side of it, which is in every environment, unless you are doing something very strange or pre-funding as a one-person startup in your spare time, your payroll costs will it—should—exceed your AWS bill by a fairly healthy amount. And engineering time is always more expensive than services time. So, for example, looking at EKS, I would absolutely recommend people use that rather than rolling their own because—Chris: Rolling their own? Yeah.Corey: —get out of that engineering space where your time is free. I assure you from a business context, it is not. So, there's always that question of what you can do to make things easier for people and do more of the heavy lifting.Chris: Yeah, and to your rather cheeky point that there's 17 ways to run a container on AWS, it is answering that question, right? Like those 17 ways, like, how much of this do you want to run yourself, you could run EKS distro on EC2 instances if you want full control over your environment.Corey: And then run IoT Greengrass core on top within that cluster—Chris: Right.Corey: So, I can run my own Lambda function runtime, so I'm not locked in. Also, DynamoDB local so I'm not locked into AWS. At which point I have gone so far around the bend, no one can help me.Chris: Well—Corey: Pro tip, don't do that. Just don't do that.Chris: But to your point, we have all these options for compute, and specifically containers because there's a lot of people that want to granularly say, “This is where my engineering team gets involved. Everything else you handle.” If I want EKS on Spot Instances only, you can do that. If you want EKS to use Carpenter and say only run ARM workloads, you can do that. If you want to say Fargate and not have anything to manage other than the container file, you can do that.It's how much does your team want to manage? That's the customer obsession part of AWS coming through when it comes to containers is because there's so many different ways to run those workloads, but there's so many different ways to make sure that your team is right-sized, based off the services you're using.Corey: I do want to change gears a bit here because you are mostly known for a couple of things: the DevOps'ish newsletter because that is the oldest and longest thing you've been doing the time that I've known you; EKS, obviously. But when prepping for this show, I discovered you are now co-chair of the OpenGitOps project.Chris: Yes.Corey: So, I have heard of GitOps in the context of, “Oh, it's just basically your CI/CD stuff is triggered by Git events and whatnot.” And I'm sitting here going, “Okay, so from where you're sitting, the two best user interfaces in the world that you have discovered are YAML and Git.” And I just have to start with the question, “Who hurt you?”Chris: [laugh]. Yeah, I share your sentiment when it comes to Git. Not so much with YAML, but I think it's because I'm so used to it. Maybe it's Stockholm Syndrome, maybe the whole YAML thing. I don't know.Corey: Well, it's no XML. We'll put it that way.Chris: Thankfully, yes because if it was, I would have way more, like, just template files laying around to build things. But the—Corey: And rage. Don't forget rage.Chris: And rage, yeah. So, GitOps is a little bit more than just Git in IaC—infrastructure as Code. It's more like Justin Garrison, who's also on my team, he calls it infrastructure software because there's four main principles to GitOps, and if you go to opengitops.dev, you can see them. It's version one.So, we put them on the website, right there on the page. You have to have a declared state and that state has to live somewhere. Now, it's called GitOps because Git is probably the most full-featured thing to put your state in, but you could use an S3 bucket and just version it, for example. And make it private so no one else can get to it.Corey: Or you could use local files: copy-of-copy-of-this-thing-restored-parentheses-use-this-one-dot-final-dot-doc-dot-zip. You know, my preferred naming convention.Chris: Ah, yeah. Wow. Okay. [laugh]. Yeah.Corey: Everything I touch is terrifying.Chris: Yes. Geez, I'm sorry. So first, it's declarative. You declare your state. You store it somewhere. It's versioned and immutable, like I said. And then pulled automatically—don't focus so much on pull—but basically, software agents are applying the desired state from source. So, what does that mean? When it's—you know, the fourth principle is implemented, continuously reconciled. That means those software agents that are checking your desired state are actually putting it back into the desired state if it's out of whack, right? So—Corey: You're talking about agents running it persistently on instances, validating—Chris: Yes.Corey: —a checkpoint on a cron. How is this meaningfully different than a Puppet agent running in years past? Having spent I learned to speak publicly by being a traveling trainer for Puppet; same type of model, and in fact, when I was at Pinterest, we wound up having a fair bit—like, that was their entire model, where they would have—the Puppet's code would live in an S3 bucket that was then copied down, I believe, via Git, and then applied to the instance on a schedule. Like, that sounds like this was sort of a early days GitOps.Chris: Yeah, exactly. Right? Like so it's, I like to think of that as a component of GitOps, right? DevOps, when you talk about DevOps in general, there's a lot of stuff out there. There's a lot of things labeled DevOps that maybe are, or maybe aren't sticking to some of those DevOps core things that make you great.Like the stuff that Nicole Forsgren writes about in books, you know? Accelerate is on my desk for a reason because there's things that good, well-managed DevOps practices do. I see GitOps as an actual implementation of DevOps in an open-source manner because all the tooling for GitOps these days is open-source and it all started as open-source. Now, you can get, like, Flux or Argo—Argo, specifically—there's managed services out there for it, you can have Flux and not maintain it, through an add-on, on EKS for example, and it will reconcile that state for you automatically. And the other thing I like to say about GitOps, specifically, is that it moves at the speed of the Kubernetes Audit Log.If you've ever looked at a Kubernetes audit log, you know it's rather noisy with all these groups and versions and kinds getting thrown out there. So, GitOps will say, “Oh, there's an event for said thing that I'm supposed to be watching. Do I need to change anything? Yes or no? Yes? Okay, go.”And the change gets applied, or, “Hey, there's a new Git thing. Pull it in. A change has happened inGit I need to update it.” You can set it to reconcile on events on time. It's like a cron or it's like an event-driven architecture, but it's combined.Corey: How does it survive the stake through the heart of configuration management? Because before I was doing all this, I wasn't even a T-shaped engineer: you're broad across a bunch of things, but deep in one or two areas, and one of mine was configuration management. I wrote part of SaltStack, once upon a time—Chris: Oh.Corey: —due to a bunch of very strange coincidences all hitting it once, like, I taught people how to use Puppet. But containers ultimately arose and the idea of immutable infrastructure became a thing. And these days when we were doing full-on serverless, well, great, I just wind up deploying a new code bundle to the Lambdas function that I wind up caring about, and that is a immutable version replacement. There is no drift because there is no way to log in and change those things other than through a clear deployment of this as the new version that goes out there. Where does GitOps fit into that imagined pattern?Chris: So, configuration management becomes part of your approval process, right? So, you now are generating an audit log, essentially, of all changes to your system through the approval process that you set up as part of your, how you get things into source and then promote that out to production. That's kind of the beauty of it, right? Like, that's why we suggest using Git because it has functions, like, requests and issues and things like that you can say, “Hey, yes, I approve this,” or, “Hey, no, I don't approve that. We need changes.” So, that's kind of natively happening with Git and, you know, GitLab, GitHub, whatever implementation of Git. There's always, kind of—Corey: Uh, JIF-ub is, I believe, the pronunciation.Chris: JIF-ub? Oh.Corey: Yeah. That's what I'm—Chris: Today, I learned. Okay.Corey: Exactly. And that's one of the things that I do for my lasttweetinaws.com Twitter client that I build—because I needed it, and if other people want to use it, that's great—that is now deployed to 20 different AWS commercial regions, simultaneously. And that is done via—because it turns out that that's a very long to execute for loop if you start down that path—Chris: Well, yeah.Corey: I wound up building out a GitHub Actions matrix—sorry a JIF-ub—actions matrix job that winds up instantiating 20 parallel builds of the CDK deploy that goes out to each region as expected. And because that gets really expensive with native GitHub Actions runners for, like, 36 cents per deploy, and I don't know how to test my own code, so every time I have a typo, that's another quarter in the jar. Cool, but that was annoying for me so I built my own custom runner system that uses Lambda functions as runners running containers pulled from ECR that, oh, it just runs in parallel, less than three minutes. Every time I commit something between I press the push button and it is out and running in the wild across all regions. Which is awesome and also terrifying because, as previously mentioned, I don't know how to test my code.Chris: Yeah. So, you don't know what you're deploying to 20 regions sometime, right?Corey: But it also means I have a pristine, re-composable build environment because I can—Chris: Right.Corey: Just automatically have that go out and the fact that I am making a—either merging a pull request or doing a direct push because I consider main to be my feature branch as whenever something hits that, all the automation kicks off. That was something that I found to be transformative as far as a way of thinking about this because I was very tired of having to tweak my local laptop environment to, “Oh, you didn't assume the proper role and everything failed again and you broke it. Good job.” It wound up being something where I could start developing on more and more disparate platforms. And it finally is what got me away from my old development model of everything I build is on an EC2 instance, and that means that my editor of choice was Vim. I use the VS Code now for these things, and I'm pretty happy with it.Chris: Yeah. So, you know, I'm glad you brought up CDK. CDK gives you a lot of the capabilities to implement GitOps in a way that you could say, like, “Hey, use CDK to declare I need four Amazon EKS clusters with this size, shape, and configuration. Go.” Or even further, connect to these EKS clusters to RDS instances and load balancers and everything else.But you put that state into Git and then you have something that deploys that automatically upon changes. That is infrastructure as code. Now, when you say, “Okay, main is your feature branch,” you know, things happen on main, if this were running in Kubernetes across a fleet of clusters or the globe-wide in 20 regions, something like Flux or Argo would kick in and say, “There's been a change to source, main, and we need to roll this out.” And it'll start applying those changes. Now, what do you get with GitOps that you don't get with your configuration?I mean, can you rollback if you ever have, like, a bad commit that's just awful? I mean, that's really part of the process with GitOps is to make sure that you can, A, roll back to the previous good state, B, roll forward to a known good state, or C, promote that state up through various environments. And then having that all done declaratively, automatically, and immutably, and versioned with an audit log, that I think is the real power of GitOps in the sense that, like, oh, so-and-so approve this change to security policy XYZ on this date at this time. And that to an auditor, you just hand them a log file on, like, “Here's everything we've ever done to our system. Done.” Right?Like, you could get to that state, if you want to, which I think is kind of the idea of DevOps, which says, “Take all these disparate tools and processes and procedures and culture changes”—culture being the hardest part to adopt in DevOps; GitOps kind of forces a culture change where, like, you can't do a CAB with GitOps. Like, those two things don't fly. You don't have a configuration management database unless you absolutely—Corey: Oh, you CAB now but they're all the comments of the pull request.Chris: Right. Exactly. Like, don't push this change out until Thursday after this other thing has happened, kind of thing. Yeah, like, that all happens in GitHub. But it's very democratizing in the sense that people don't have to waste time in an hour-long meeting to get their five minutes in, right?Corey: DoorDash had a problem. As their cloud-native environment scaled and developers delivered new features, their monitoring system kept breaking down. In an organization where data is used to make better decisions about technology and about the business, losing observability means the entire company loses their competitive edge. With Chronosphere, DoorDash is no longer losing visibility into their applications suite. The key? Chronosphere is an open-source compatible, scalable, and reliable observability solution that gives the observability lead at DoorDash business, confidence, and peace of mind. Read the full success story at snark.cloud/chronosphere. That's snark.cloud slash C-H-R-O-N-O-S-P-H-E-R-E.Corey: So, would it be overwhelmingly cynical to suggest that GitOps is the means to implement what we've all been pretending to have implemented for the last decade when giving talks at conferences?Chris: Ehh, I wouldn't go that far. I would say that GitOps is an excellent way to implement the things you've been talking about at all these conferences for all these years. But keep in mind, the technology has changed a lot in the, what 11, 12 years of the existence of DevOps, now. I mean, we've gone from, let's try to manage whole servers immutably to, “Oh, now we just need to maintain an orchestration platform and run containers.” That whole compute interface, you go from SSH to a Docker file, that's a big leap, right?Like, you don't have bespoke sysadmins; you have, like, a platform team. You don't have DevOps engineers; they're part of that platform team, or DevOps teams, right? Like, which was kind of antithetical to the whole idea of DevOps to have a DevOps team. You know, everybody's kind of in the same boat now, where we see skill sets kind of changing. And GitOps and Kubernetes-land is, like, a platform team that manages the cluster, and its state, and health and, you know, production essentially.And then you have your developers deploying what they want to deploy in when whatever namespace they've been given access to and whatever rights they have. So, now you have the potential for one set of people—the platform team—to use one set of GitOps tooling, and your applications teams might not like that, and that's fine. They can have their own namespaces with their own tooling in it. Like, Argo, for example, is preferred by a lot of developers because it has a nice UI with green and red dots and they can show people and it looks nice, Flux, it's command line based. And there are some projects out there that kind of take the UI of Argo and try to run Flux underneath that, and those are cool kind of projects, I think, in my mind, but in general, right, I think GitOps gives you the choice that we missed somewhat in DevOps implementations of the past because it was, “Oh, we need to go get cloud.” “Well, you can only use this cloud.” “Oh, we need to go get this thing.” “Well, you can only use this thing in-house.”And you know, there's a lot of restrictions sometimes placed on what you can use in your environment. Well, if your environment is Kubernetes, how do you restrict what you can run, right? Like you can't have an easily configured say, no open-source policy if you're running Kubernetes. [laugh] so it becomes, you know—Corey: Well, that doesn't stop some companies from trying.Chris: Yeah, that's true. But the idea of, like, enabling your developers to deploy at will and then promote their changes as they see fit is really the dream of DevOps, right? Like, same with production and platform teams, right? I want to push my changes out to a larger system that is across the globe. How do I do that? How do I manage that? How do I make sure everything's consistent?GitOps gives you those ways, with Kubernetes native things like customizations, to make consistent environments that are robust and actually going to be reconciled automatically if someone breaks the glass and says, “Oh, I need to run this container immediately.” Well, that's going to create problems because it's deviated from state and it's just that one region, so we'll put it back into state.Corey: It'll be dueling banjos, at some point. You'll try and doing something manually, it gets reverted automatically. I love that pattern. You'll get bored before the computer does, always.Chris: Yeah. And GitOps is very new, right? When you think about the lifetime of GitOps, I think it was coined in, like, 2018. So, it's only four years old, right? When—Corey: I prefer it to ChatOps, at least, as far as—Chris: Well, I mean—Corey: —implementation and expression of the thing.Chris: —ChatOps was a way to do DevOps. I think GitOps—Corey: Well, ChatOps is also a way to wind up giving whoever gets access to your Slack workspace root in production.Chris: Mmm.Corey: But that's neither here nor there.Chris: Mm-hm.Corey: It's yeah, we all like to pretend that's not a giant security issue in our industry, but that's a topic for another time.Chris: Yeah. And that's why, like, GitOps also depends upon you having good security, you know, and good authorization and approval processes. It enforces that upon—Corey: Yeah, who doesn't have one of those?Chris: Yeah. If it's a sole operation kind of deal, like in your setup, your case, I think you kind of got it doing right, right? Like, as far as GitOps goes—Corey: Oh, to be clear, we are 11 people and we do have dueling pull requests and all the rest.Chris: Right, right, right.Corey: But most of the stuff I talk about publicly is not our production stuff, so it really is just me. Just as a point of clarity there. I've n—the 11 people here do not all—the rest of you don't just sit there and clap as I do all the work.Chris: Right.Corey: Most days.Chris: No, I'm sure they don't. I'm almost certain they don't clap… for you. I mean, they would—Corey: No. No, they try and talk me out of it in almost every case.Chris: Yeah, exactly. So, the setup that you, Corey Quinn, have implemented to deploy these 20 regions is kind of very GitOps-y, in the sense that when main changes, it gets updated. Where it's not GitOps-y is what if the endpoint changes? Does it get reconciled? That's the piece you're probably missing is that continuous reconciliation component, where it's constantly checking and saying, “This thing out there is deployed in the way I want it. You know, the way I declared it to be in my source of truth.”Corey: Yeah, when you start having other people getting involved, there can—yeah, that's where regressions enter. And it's like, “Well, I know where things are so why would I change the endpoint?” Yeah, it turns out, not everyone has the state of the entire application in their head. Ideally it should live in—Chris: Yeah. Right. And, you know—Corey: —you know, Git or S3.Chris: —when I—yeah, exactly. When I think about interactions of the past coming out as a new DevOps engineer to work with developers, it's always been, will developers have access to prod or they don't? And if you're in that environment with—you're trying to run a multi-billion dollar operation, and your devs have direct—or one Dev has direct access to prod because prod is in his brain, that's where it's like, well, now wait a minute. Prod doesn't have to be only in your brain. You can put that in the codebase and now we know what is in your brain, right?Like, you can almost do—if you document your code, well, you can have your full lifecycle right there in one place, including documentation, which I think is the best part, too. So, you know, it encourages approval processes and automation over this one person has an entire state of the system in their head; they have to go in and fix it. And what if they're not on call, or in Jamaica, or on a cruise ship somewhere kind of thing? Things get difficult. Like, for example, I just got back from vacation. We were so far off the grid, we had satellite internet. And let me tell you, it was hard to write an email newsletter where I usually open 50 to 100 tabs.Corey: There's a little bit of internet out Californ-ie way.Chris: [laugh].Corey: Yeah it's… it's always weird going from, like, especially after pandemic; I have gigabit symmetric here and going even to re:Invent where I'm trying to upload a bunch of video and whatnot.Chris: Yeah. Oh wow.Corey: And the conference WiFi was doing its thing, and well, Verizon 5G was there but spotty. And well, yeah. Usual stuff.Chris: Yeah. It's amazing to me how connectivity has become so ubiquitous.Corey: To the point where when it's not there anymore, it's what do I do with myself? Same story about people pushing back against remote development of, “Oh, I'm just going to do it all on my laptop because what happens if I'm on a plane?” It's, yeah, the year before the pandemic, I flew 140,000 miles domestically and I was almost never hamstrung by my ability to do work. And my only local computer is an iPad for those things. So, it turns out that is less of a real world concern for most folks.Chris: Yeah I actually ordered the components to upgrade an old Nook that I have here and turn it into my, like, this is my remote code server, that's going to be all attached to GitHub and everything else. That's where I want to be: have Tailscale and just VPN into this box.Corey: Tailscale is transformative.Chris: Yes. Tailscale will change your life. That's just my personal opinion.Corey: Yep.Chris: That's not an AWS opinion or anything. But yeah, when you start thinking about your network as it could be anywhere, that's where Tailscale, like, really shines. So—Corey: Tailscale makes the internet work like we all wanted to believe that it worked.Chris: Yeah. And Wireguard is an excellent open-source project. And Tailscale consumes that and puts an amazingly easy-to-use UI, and troubleshooting tools, and routing, and all kinds of forwarding capabilities, and makes it kind of easy, which is really, really, really kind of awesome. And Tailscale and Kubernetes—Corey: Yeah, ‘network' and ‘easy' don't belong in the same sentence, but in this case, they do.Chris: Yeah. And trust me, the Kubernetes story in Tailscale, there is a lot of there. I understand you might want to not open ports in your VPC, maybe, but if you use Tailscale, that node is just another thing on your network. You can connect to that and see what's going on. Your management cluster is just another thing on the network where you can watch the state.But it's all—you're connected to it continuously through Tailscale. Or, you know, it's a much lighter weight, kind of meshy VPN, I would say, if I had to sum it up in one sentence. That was not on our agenda to talk about at all. Anyways. [laugh]Corey: No, no. I love how many different topics we talk about on these things. We'll have to have you back soon to talk again. I really want to thank you for being so generous with your time. If people want to learn more about what you're up to and how you view these things, where can they find you?Chris: Go to ChrisShort.net. So, Chris Short—I'm six-four so remember, it's Short—dot net, and you will find all the places that I write, you can go to devopsish.com to subscribe to my newsletter, which goes out every week. This year. Next year, there'll be breaks. And then finally, if you want to follow me on Twitter, Chris Short: at @ChrisShort on Twitter. All one word so you see two s's. Like, it's okay, there's two s's there.Corey: Links to all of that will of course be in the show notes. It's easier for people to do the clicky-clicky thing as a general rule.Chris: Clicky things are easier than the wordy things, yes.Corey: Says the Kubernetes guy.Chris: Yeah. Says the Kubernetes guy. Yeah, you like that, huh? Like I said, Argo gives you a UI. [laugh].Corey: Thank you [laugh] so much for your time. I really do appreciate it.Chris: Thank you. This has been fun. If folks have questions, feel free to reach out. Like, I am not one of those people that hides behind a screen all day and doesn't respond. I will respond to you eventually.Corey: I'm right here, Chris. Come on, come on. You're calling me out in front of myself. My God.Chris: Egh. It might take a day or two, but I will respond. I promise.Corey: Thanks again for your time. This has been Chris Short, senior developer advocate at AWS. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice and if it's YouTube, click the thumbs-up button. Whereas if you've hated this podcast, same thing, smash the buttons five-star review and leave an insulting comment that is written in syntactically correct YAML because it's just so easy to do.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About ChrisChris is the Co-founder and Chief Product Officer at incident.io, where they're building incident management products that people actually want to use. A software engineer by trade, Chris is no stranger to gnarly incidents, having participated (and caused!) them at everything from early stage startups through to enormous IT organizations.Links Referenced: incident.io: https://incident.io Practical Guide to Incident Management: https://incident.io/guide/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: DoorDash had a problem. As their cloud-native environment scaled and developers delivered new features, their monitoring system kept breaking down. In an organization where data is used to make better decisions about technology and about the business, losing observability means the entire company loses their competitive edge. With Chronosphere, DoorDash is no longer losing visibility into their applications suite. The key? Chronosphere is an open-source compatible, scalable, and reliable observability solution that gives the observability lead at DoorDash business, confidence, and peace of mind. Read the full success story at snark.cloud/chronosphere. That's snark.cloud slash C-H-R-O-N-O-S-P-H-E-R-E.Corey: Let's face it, on-call firefighting at 2am is stressful! So there's good news and there's bad news. The bad news is that you probably can't prevent incidents from happening, but the good news is that incident.io makes incidents less stressful and a lot more valuable. incident.io is a Slack-native incident management platform that allows you to automate incident processes, focus on fixing the issues and learn from incident insights to improve site reliability and fix your vulnerabilities. Try incident.io, recover faster and sleep more.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Today's promoted guest is Chris Evans, who's the CPO and co-founder of incident.io. Chris, first, thank you very much for joining me. And I'm going to start with an easy question—well, easy question, hard answer, I think—what is an incident.io exactly?Chris: Incident.io is a software platform that helps entire organizations to respond to recover from and learn from incidents.Corey: When you say incident, that means an awful lot of things. And depending on where you are in the ecosystem in the world, that means different things to different people. For example, oh, incident. Like, “Are you talking about the noodle incident because we had an agreement that we would never speak about that thing again,” style, versus folks who are steeped in DevOps or SRE culture, which is, of course, a fancy way to say those who are sad all the time, usually about computers. What is an incident in the context of what you folks do?Chris: That, I think, is the killer question. I think if you look at organizations in the past, I think incidents were those things that happened once a quarter, maybe once a year, and they were the thing that brought the entirety of your site down because your big central database that was in a data center sort of disappeared. The way that modern companies run means that the definition has to be very, very different. So, most places now rely on distributed systems and there is no, sort of, binary sense of up or down these days. And essentially, in the general case, like, most companies are continually in a sort of state of things being broken all of the time.And so, for us, when we look at what an incident is, it is essentially anything that takes you away from your planned work with a sense of urgency. And that's the sort of the pithy definition that we use there. Generally, that can mean anything—it means different things to different folks, and, like, when we talk to folks, we encourage them to think carefully about what that threshold is, but generally, for us at incident.io, that means basically a single error that is worthwhile investigating that you would stop doing your backlog work for is an incident. And also an entire app being down, that is an incident.So, there's quite a wide range there. But essentially, by sort of having more incidents and lowering that threshold, you suddenly have a heap of benefits, which I can go very deep into and talk for hours about.Corey: It's a deceptively complex question. When I talk to folks about backups, one of the biggest problems in the world of backup and building a DR plan, it's not building the DR plan—though that's no picnic either—it's okay. In the time of cloud, all your planning figures out, okay. Suddenly the site is down, how do we fix it? There are different levels of down and that means different things to different people where, especially the way we build apps today, it's not is the service or site up or down, but with distributed systems, it's how down is it?And oh, we're seeing elevated error rates in us-tire-fire-1 region of AWS. At what point do we begin executing on our disaster plan? Because the worst answer, in some respects is, every time you think you see a problem, you start failing over to other regions and other providers and the rest, and three minutes in, you've irrevocably made the cutover and it's going to take 15 minutes to come back up. And oh, yeah, then your primary site comes back up because whoever unplugged something, plugged it back in and now you've made the wrong choice. Figuring out all the things around the incident, it's not what it once was.When you were running your own blog on a single web server and it's broken, it's pretty easy to say, “Is it up or is it down?” As you scale out, it seems like that gets more and more diffuse. But it feels to me that it's also less of a question of how the technology has scaled, but also how the culture and the people have scaled. When you're the only engineer somewhere, you pretty much have no choice but to have the entire state of your stack shoved into your head. When that becomes 15 or 20 different teams of people, in some cases, it feels like it's almost less than a technology problem than it is a problem of how you communicate and how you get people involved. And the issues in front of the people who are empowered and insightful in a certain area that needs fixing.Chris: A hundred percent. This is, like, a really, really key point, which is that organizations themselves are very complex. And so, you've got this combination of systems getting more and more complicated, more and more sort of things going wrong and perpetually breaking but you've got very, very complicated information structures and communication throughout the whole organization to keep things up and running. The very best orgs are the ones where they can engage the entire, sort of, every corner of the organization when things do go wrong. And lived and breathed this firsthand when various different previous companies, but most recently at Monzo—which is a bank here in the UK—when an incident happened there, like, one of our two physical data center locations went down, the bank wasn't offline. Everything was resilient to that, but that required an immediate response.And that meant that engineers were deployed to go and fix things. But it also meant the customer support folks might be required to get involved because we might be slightly slower processing payments. And it means that risk and compliance folks might need to get involved because they need to be reporting things to regulators. And the list goes on. There's, like, this need for a bunch of different people who almost certainly have never worked together or rarely worked together to come together, land in this sort of like empty space of this incident room or virtual incident room, and figure out how they're going to coordinate their response and get things back on track in the sort of most streamlined way and as quick as possible.Corey: Yeah, when your bank is suddenly offline, that seems like a really inopportune time to be introduced to the database team. It's, “Oh, we have one of those. Wonderful. I feel like you folks are going to come in handy later today.” You want to have those pathways of communication open well in advance of these issues.Chris: A hundred percent. And I think the thing that makes incidents unique is that fact. And I think the solution to that is this sort of consistent, level playing field that you can put everybody on. So, if everybody understands that the way that incidents are dealt with is consistent, we declare it like this, and under these conditions, these things happen. And, you know, if I flag this kind of level of impact, we have to pull in someone else to come and help make a decision.At the core of it, there's this weird kind of duality to incidents where they are both kind of semi-formulaic and that you can basically encode a lot of the processes that happen, but equally, they are incredibly chaotic and require a lot of human impact to be resilient and figure these things out because stuff that you have never seen happen before is happening and failing in ways that you never predicted. And so, this is where incident.io plays into this is that we try to take the first half of that off of your hands, which is, we will help you run your process so that all of the brain capacity you have, it goes on to the bit that humans are uniquely placed to be able to do, which is responding to these very, very chaotic, sort of, surprise events that have happened.Corey: I feel as well—because I played around in this space a bit before I used to run ops teams—and, more or less I really should have had a t-shirt then that said, “I am the root cause,” because yeah, I basically did a lot of self-inflicted outages in various environments because it turns out, I'm not always the best with computers. Imagine that. There are a number of different companies that play in the space that look at some part of the incident lifecycle. And from the outside, first, they all look alike because it's, “Oh, so you're incident.io. I assume you're PagerDuty. You're the thing that calls me at two in the morning to make sure I wake up.”Conversely, for folks who haven't worked deeply in that space, as well, of setting things on fire, what you do sounds like it's highly susceptible to the Hacker News problem. Where, “Wait, so what you do is effectively just getting people to coordinate and talk during an incident? Well, that doesn't sound hard. I could do that in a weekend.” And no, no, you can't.If this were easy, you would not have been in business as long as you have, have the team the size that you do, the customers that you do. But it's one of those things that until you've been in a very specific set of a problem, it doesn't sound like it's a real problem that needs solving.Chris: Yeah, I think that's true. And I think that the Hacker News point is a particularly pertinent one and that someone else, sort of, in an adjacent area launched on Hacker News recently, and the amount of feedback they got around, you know, “You're a Slack bot. How is this a company?” Was kind of staggering. And I think generally where that comes from is—well, first of all that bias that engineers have, which is just everything you look at as an engineer is like, “Yeah, I can build that in a weekend.” I think there's often infinite complexity under the hood that just gets kind of brushed over. But yeah, I think at the core of it, you probably could build a Slack bot in a weekend that creates a channel for you in Slack and allows you to post somewhere that some—Corey: Oh, good. More channels in Slack. Just when everyone wants.Chris: Well, there you go. I mean, that's a particular pertinent one because, like, our tool does do that. And one of the things—so I built at Monzo, a version of incident.io that we used at the company there, and that was something that I built evenings and weekends. And among the many, many things I never got around to building, archiving and cleaning up channels was one of the ones that was always on that list.And so, Monzo did have this problem of littered channels everywhere, I think that sort of like, part of the problem here is, like, it is easy to look at a product like ours and sort of assume it is this sort of friendly Slack bot that helps you orchestrate some very basic commands. And I think when you actually dig into the problems that organizations above a certain size have, they're not solved by Slack bots. They're solved by platforms that help you to encode your processes that otherwise have to live on a Google Doc somewhere which is five pages long and when it's 2 a.m. and everything's on fire, I guarantee you not a single person reads that Google Doc, so your process is as good as not in place at all. That's the beauty of a tool like ours. We have a powerful engine that helps you basically to encode that and take some load off of you.Corey: To be clear, I'm also not coming at this from a position of judging other people. I just look right now at the Slack workspace that we have The Duckbill Group, and we have something like a ten-to-one channel-to-human ratio. And the proliferation of channels is a very real thing. And the problem that I've seen across the board with other things that try to address incident management has always been fanciful at best about what really happens when something breaks. Like, you talk about, oh, here's what happens. Step one: you will pull up the Google Doc, or you will pull up the wiki or the rest, or in some aspirational places, ah, something seems weird, I will go open a ticket in Jira.Meanwhile, here in reality, anyone who's ever worked in these environments knows that step one, “Oh shit, oh shit, oh shit, oh shit, oh shit. What are we going to do?” And all the practices and procedures that often exist, especially in orgs that aren't very practiced at these sorts of things, tend to fly out the window and people are going to do what they're going to do. So, any tool or any platform that winds up addressing that has to accept the reality of meeting people where they are not trying to educate people into different patterns of behavior as such. One of the things I like about your approach is, yeah, it's going to be a lot of conversation in Slack that is a given we can pretend otherwise, but here in reality, that is how work gets communicated, particularly in extremis. And I really appreciate the fact that you are not trying to, like, fight what feels almost like a law of nature at this point.Chris: Yeah, I think there's a few things in that. The first point around the document approach or the clearly defined steps of how an incident works. In my experience, those things have always gone wrong because—Corey: The data center is down, so we're going to the wiki to follow our incident management procedure, which is in the data center just lost power.Chris: Yeah.Corey: There's a dependency problem there, too. [laugh].Chris: Yeah, a hundred percent. [laugh]. A hundred percent. And I think part of the problem that I see there is that very, very often, you've got this situation where the people designing the process are not the people following the process. And so, there's this classic, I've heard it through John Allspaw, but it's a bunch of other folks who talk about the difference between people, you know, at the sharp end or the blunt end of the work.And I think the problem that people are facing the past is you have these people who sit in the, sort of, metaphorical upstairs of the office and think that they make a company safe by defining a process on paper. And they ship the piece of paper and go, “That is a good job for me done. I'm going to leave and know that I've made the bank—the other whatever your organization does—much, much safer.” And I think this is where things fall down because—Corey: I want to ambush some of those people in their performance reviews with, “Cool. Just for fun, all the documentation here, we're going to pull up the analytics to see how often that stuff gets viewed. Oh, nobody ever sees it. Hmm.”Chris: It's frustrating. It's frustrating because that never ever happens, clearly. But the point you made around, like, meeting people where you are, I think that is a huge one, which is incidents are founded on great communication. Like, as I said earlier, this is, like, a form of team with someone you've never ever worked with before and the last thing you want to do is be, like, “Hey, Corey, I've never met you before, but let's jump out onto this other platform somewhere that I've never been or haven't been for weeks and we'll try and figure stuff out over there.” It's like, no, you're going to be communicating—Corey: We use Slack internally, but we have a WhatsApp chat that we wind up using for incident stuff, so go ahead and log into WhatsApp, which you haven't done in 18 months, and join the chat. Yeah, in the dawn of time, in the mists of antiquity, you vaguely remember hearing something about that your first week and then never again. This stuff has to be practiced and it's important to get it right. How do you approach the inherent and often unfortunate reality that incident response and management inherently becomes very different depending upon the specifics of your company or your culture or something like that? In other words, how cookie-cutter is what you have built versus adaptable to different environments it finds itself operating in?Chris: Man, the amount of time we spent as a founding team in the early days deliberating over how opinionated we should be versus how flexible we should be was staggering. The way we like to describe it as we are quite opinionated about how we think incidents should be run, however we let you imprint your own process into that, so putting some color onto that. We expect incidents to have a lead. That is something you cannot get away from. However, you can call the lead whatever makes sense for you at your organization. So, some folks call them an incident commander or a manager or whatever else.Corey: There's overwhelming militarization of these things. Like, oh, yes, we're going to wind up taking a bunch of terms from the military here. It's like, you realize that your entire giant screaming fire is that the lights on the screen are in the wrong pattern. You're trying to make them in the right pattern. No one dies here in most cases, so it feels a little grandiose for some of those terms being tossed around in some cases, but I get it. You've got to make something that is unpleasant and tedious in many respects, a little bit more gripping. I don't envy people. Messaging is hard.Chris: Yeah, it is. And I think if you're overly virtuoustic and inflexible, you're sort of fighting an uphill battle here, right? So, folks are going to want to call things what they want to call things. And you've got people who want to import [ITIL 00:15:04] definitions for severity ease into the platform because that's what they're familiar with. That's fine.What we are opinionated about is that you have some severity levels because absent academic criticism of severity levels, they are a useful mechanism to very coarsely and very quickly assess how bad something is and to take some actions off of it. So yeah, we basically have various points in the product where you can customize and put your own sort of flavor on it, but generally, we have a relatively opinionated end-to-end expectation of how you will run that process.Corey: The thing that I find that annoys me—in some cases—the most is how heavyweight the process is, and it's clearly built by people in an ivory tower somewhere where there's effectively a two-day long postmortem analysis of the incident, and so on and so forth. And okay, great. Your entire site has been blown off the internet, yeah, that probably makes sense. But as soon as you start broadening that to things like okay, an increase in 500 errors on this service for 30 minutes, “Great. Well, we're going to have a two-day postmortem on that.” It's, “Yeah, sure would be nice if we could go two full days without having another incident of that caliber.” So, in other words, whose foot—are we going to hire a new team whose full-time job it is, is to just go ahead and triage and learn from all these incidents? Seems to me like that's sort of throwing wood behind the wrong arrows.Chris: Yeah, I think it's very reductive to suggest that learning only happens in a postmortem process. So, I wrote a blog, actually, not so long ago that is about running postmortems and when it makes sense to do it. And as part of that, I had a sort of a statement that was [laugh] that we haven't run a single postmortem when I wrote this blog at incident.io. Which is probably shocking to many people because we're an incident company, and we talk about this stuff, but we were also a company of five people and when something went wrong, the learning was happening and these things were sort of—we were carving out the time, whether it was called a postmortem, or not to learn and figure out these things. Extrapolating that to bigger companies, there is little value in following processes for the sake of following processes. And so, you could have—Corey: Someone in compliance just wound up spitting their coffee over their desktop as soon as you said that. But I hear you.Chris: Yeah. And it's those same folks who are the ones who care about the document being written, not the process and the learning happening. And I think that's deeply frustrating to me as—Corey: All the plans, of course, assume that people will prioritize the company over their own family for certain kinds of disasters. I love that, too. It's divorced from reality; that's ridiculous, on some level. Speaking of ridiculous things, as you continue to grow and scale, I imagine you integrate with things beyond just Slack. You grab other data sources and over in the fullness of time.For example, I imagine one of your most popular requests from some of your larger customers is to integrate with their HR system in order to figure out who's the last engineer who left, therefore everything immediately their fault because lord knows the best practice is to pillory whoever was the last left because then they're not there to defend themselves anymore and no one's going to get dinged for that irresponsible jackass's decisions, even if they never touched the system at all. I'm being slightly hyperbolic, but only slightly.Chris: Yeah. I think [laugh] that's an interesting point. I am definitely going to raise that feature request for a prefilled root cause category, which is, you know, the value is just that last person who left the organization. That it's a wonderful scapegoat situation there. I like it.To the point around what we do integrate with, I think the thing is actually with incidents that's quite interesting is there is a lot of tooling that exists in this space that does little pockets of useful, valuable things in the shape of incidents. So, you have PagerDuty is this system that does a great job of making people's phone making noise, but that happens, and then you're dropped into this sort of empty void of nothingness and you've got to go and figure out what to do. And then you've got things like Jira where clearly you want to be able to track actions that are coming out of things going wrong in some cases, and that's a great tool for that. And various other things in the middle there. And yeah, our value proposition, if you want to call it that, is to bring those things together in a way that is massively ergonomic during an incident.So, when you're in the middle of an incident, it is really handy to be able to go, “Oh, I have shipped this horrible fix to this thing. It works, but I must remember to undo that.” And we put that at your fingertips in an incident channel from Slack, that you can just log that action, lose that cognitive load that would otherwise be there, move on with fixing the thing. And you have this sort of—I think it's, like, that multiplied by 1000 in incidents that is just what makes it feel delightful. And I cringe a little bit saying that because it's an incident at the end of the day, but genuinely, it feels magical when some things happen that are just like, “Oh, my gosh, you've automatically hooked into my GitHub thing and someone else merged that PR and you've posted that back into the channel for me so I know that that happens. That would otherwise have been a thing where I jump out of the incident to go and figure out what was happening.”Corey: This episode is sponsored in part by our friend EnterpriseDB. EnterpriseDB has been powering enterprise applications with PostgreSQL for 15 years. And now EnterpriseDB has you covered wherever you deploy PostgreSQL on-premises, private cloud, and they just announced a fully-managed service on AWS and Azure called BigAnimal, all one word. Don't leave managing your database to your cloud vendor because they're too busy launching another half-dozen managed databases to focus on any one of them that they didn't build themselves. Instead, work with the experts over at EnterpriseDB. They can save you time and money, they can even help you migrate legacy applications—including Oracle—to the cloud. To learn more, try BigAnimal for free. Go to biganimal.com/snark, and tell them Corey sent you.Corey: The problem with the cloud, too, is the first thing that, when there starts to be an incident happening is the number one decision—almost the number one decision point is this my shitty code, something we have just pushed in our stuff, or is it the underlying provider itself? Which is why the AWS status page being slow to update is so maddening. Because those are two completely different paths to go down and you are having to pursue both of them equally at the same time until one can be ruled out. And that is why time to identify at least what side of the universe it's on is so important. That has always been a bit of a tricky challenge.I want to talk a bit about circular dependencies. You target a certain persona of customer, but I'm going to go out on a limb and assume that one explicit company that you are not going to want to do business with in your current iteration is Slack itself because a tool to manage—okay, so our service is down, so we're going to go to Slack to fix it doesn't work when the service is Slack itself. So, that becomes a significant challenge. As you look at this across the board, are you seeing customers having problems where you have circular dependency issues with this? Easy example: Slack is built on top of AWS.When there's an underlying degradation of, huh, suddenly us-east-1 is not doing what it's supposed to be doing, now, Slack is degraded as well, as well as the customer site, it seems like at that point, you're sort of in a bit of tricky positioning as a customer. Counterpoint, when neither Slack nor your site are working, figuring out what caused that issue doesn't seem like it's the biggest stretch of the imagination at that point.Chris: I've spent a lot of my career working in infrastructure, platform-type teams, and I think you can end up tying yourself in knots if you try and over-optimize for, like, avoiding these dependencies. I think it's one of those, sort of, turtles all the way down situations. So yes, Slack are unlikely to become a customer because they are clearly going to want to use our product when they are down.Corey: They reach out, “We'd like to be your customer.” Your response is, “Please don't be.” None of us are going to be happy with this outcome.Chris: Yeah, I mean, the interesting thing that is that we're friends with some folks at Slack, and they believe it or not, they do use Slack to navigate their incidents. They have an internal tool that they have written. And I think this sort of speaks to the point we made earlier, which is that incidents and things failing or not these sort of big binary events. And so—Corey: All of Slack is down is not the only kind of incident that a company like Slack can experience.Chris: I'd go as far as that it's most commonly not that. It's most commonly that you're navigating incidents where it is a degradation, or some edge case, or something else that's happened. And so, like, the pragmatic solution here is not to avoid the circular dependencies, in my view; it's to accept that they exist and make sure you have sensible escape hatches so that when something does go wrong—so a good example, we use incident.io at incident.io to manage incidents that we're having with incident.io. And 99% of the time, that is absolutely fine because we are having some error in some corner of the product or a particular customer is doing something that is a bit curious.And I could count literally on one hand the number of times that we have not been able to use our products to fix our product. And in those cases, we have a fallback which is jump into—Corey: I assume you put a little thought into what happened. “Well, what if our product is down?” “Oh well, I guess we'll never be able to fix it or communicate about it.” It seems like that's the sort of thing that, given what you do, you might have put more than ten seconds of thought into.Chris: We've put a fair amount of thought into it. But at the end of the day, [laugh] it's like if stuff is down, like, what do you need to do? You need to communicate with people. So, jump on a Google Chat, jump on a Slack huddle, whatever else it is we have various different, like, fallbacks in different order. And at the core of it, I think this is the thing is, like, you cannot be prepared for every single thing going wrong, and so what you can be prepared for is to be unprepared and just accept that humans are incredibly good at being resilient, and therefore, all manner of things are going to happen that you've never seen before and I guarantee you will figure them out and fix them, basically.But yeah, I say this; if my SOC 2 auditor is listening, we also do have a very well-defined, like, backup plan in our SOC 2 [laugh] in our policies and processes that is the thing that we will follow that. But yeah.Corey: The fact that you're saying the magic words of SOC 2, yes, exactly. Being in a responsible adult and living up to some baseline compliance obligations is really the sign of a company that's put a little thought into these things. So, as I pull up incident.io—the website, not the company to be clear—and look through what you've written and how you talk about what you're doing, you've avoided what I would almost certainly have not because your tagline front and center on your landing page is, “Manage incidents at scale without leaving Slack.” If someone were to reach out and say, well, we're down all the time, but we're using Microsoft Teams, so I don't know that we can use you, like, the immediate instinctive response that I would have for that to the point where I would put it in the copy is, “Okay, this piece of advice is free. I would posit that you're down all the time because you're the kind of company to use Microsoft Teams.” But that doesn't tend to win a whole lot of friends in various places. In a slightly less sarcastic bent, do you see people reaching out with, “Well, we want to use you because we love what you're doing, but we don't use Slack.”Chris: Yeah. We do. A lot of folks actually. And we will support Teams one day, I think. There is nothing especially unique about the product that means that we are tied to Slack.It is a great way to distribute our product and it sort of aligns with the companies that think in the way that we do in the general case but, like, at the core of what we're building, it's a platform that augments a communication platform to make it much easier to deal with a high-stress, high-pressure situation. And so, in the future, we will support ways for you to connect Microsoft Teams or if Zoom sought out getting rich app experiences, talk on a Zoom and be able to do various things like logging actions and communicating with other systems and things like that. But yeah, for the time being very, very deliberate focus mechanism for us. We're a small company with, like, 30 people now, and so yeah, focusing on that sort of very slim vertical is working well for us.Corey: And it certainly seems to be working to your benefit. Every person I've talked to who is encountered you folks has nothing but good things to say. We have a bunch of folks in common listed on the wall of logos, the social proof eye chart thing of here's people who are using us. And these are serious companies. I mean, your last job before starting incident.io was at Monzo, as you mentioned.You know what you're doing in a regulated, serious sense. I would be, quite honestly, extraordinarily skeptical if your background were significantly different from this because, “Well, yeah, we worked at Twitter for Pets in our three-person SRE team, we can tell you exactly how to go ahead and handle your incidents.” Yeah, there's a certain level of operational maturity that I kind of just based upon the name of the company there; don't think that Twitter for Pets is going to nail. Monzo is a bank. Guess you know what you're talking about, given that you have not, basically, been shut down by an army of regulators. It really does breed an awful lot of confidence.But what's interesting to me is the number of people that we talk to in common are not themselves banks. Some are and they do very serious things, but others are not these highly regulated, command-and-control, top-down companies. You are nimble enough that you can get embedded at those startup-y of startup companies once they hit a certain point of scale and wind up helping them arrive at a better outcome. It's interesting in that you don't normally see a whole lot of tools that wind up being able to speak to both sides of that very broad spectrum—and most things in between—very effectively. But you've somehow managed to thread that needle. Good work.Chris: Thank you. Yeah. What else can I say other than thank you? I think, like, it's a deliberate product positioning that we've gone down to try and be able to support those different use cases. So, I think, at the core of it, we have always tried to maintain the incident.io should be installable and usable in your very first incident without you having to have a very steep learning curve, but there is depth behind it that allows you to support a much more sophisticated incident setup.So, like, I mean, you mentioned Monzo. Like, I just feel incredibly fortunate to have worked at that company. I joined back in 2017 when they were, I don't know, like, 150,000 customers and it was just getting its banking license. And I was there for four years and was able to then see it scale up to 6 million customers and all of the challenges and pain that goes along with that both from building infrastructure on the technical side of things, but from an organizational side of things. And was, like, front-row seat to being able to work with some incredibly smart people and sort of see all these various different pain points.And honestly, it feels a little bit like being in sort of a cheat mode where we get to this import a lot of that knowledge and pain that we felt at Monzo into the product. And that happens to resonate with a bunch of folks. So yeah, I feel like things are sort of coming out quite well at the moment for folks.Corey: The one thing I will say before we wind up calling this an episode is just how grateful I am that I don't have to think about things like this anymore. There's a reason that the problem that I chose to work on of expensive AWS bills being very much a business-hours only style of problem. We're a services company. We don't have production infrastructure that is externally facing. “Oh, no, one of our data analysis tools isn't working internally.”That's an interesting curiosity, but it's not an emergency in the same way that, “Oh, we're an ad network and people are looking at ads right now because we're broken,” is. So, I am grateful that I don't have to think about these things anymore. And also a little wistful because there's so much that you do it would have made dealing with expensive and dangerous outages back in my production years a lot nicer.Chris: Yep. I think that's what a lot of folks are telling us essentially. There's this curious thing with, like, this product didn't exist however many years ago and I think it's sort of been quite emergent in a lot of companies that, you know, as sort of things have moved on, that something needs to exist in this little pocket of space, dealing with incidents in modern companies. So, I'm very pleased that what we're able to build here is sort of working and filling that for folks.Corey: Yeah. I really want to thank you for taking so much time to go through the ethos of what you do, why you do it, and how you do it. If people want to learn more, where's the best place for them to go? Ideally, not during an incident.Chris: Not during an incident, obviously. Handily, the website is the company name. So, incident.io is a great place to go and find out more. We've literally—literally just today, actually—launched our Practical Guide to Incident Management, which is, like, a really full piece of content which, hopefully, will be useful to a bunch of different folks.Corey: Excellent. We will, of course, put a link to that in the [show notes 00:29:52]. I really want to thank you for being so generous with your time. Really appreciate it.Chris: Thanks so much. It's been an absolute pleasure.Corey: Chris Evans, Chief Product Officer and co-founder of incident.io. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this episode, please leave a five-star review on your podcast platform of choice along with an angry comment telling me why your latest incident is all the intern's fault.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
Chris switched from Trello over to Linear for product management and talks about prioritizing backlogs. Steph shares and discusses a tweet from Curtis Einsmann that super resonated with the work she's doing right now: "In software engineering, rabbit holes are inevitable. You will research libraries and not use them. You'll write code just to delete it. This isn't a waste; sometimes, you need to go down a few wrong paths to get to the right one." This episode is brought to you by BuildPulse (https://buildpulse.io/bikeshed). Start your 14-day free trial of BuildPulse today. Linear (https://linear.app/) Curtis Einsmann Tweet (https://twitter.com/curtiseinsmann/status/1521451508943843329) Louie Bacaj Tweet (https://twitter.com/LBacaj/status/1478241322637033474?s=20) Become a Sponsor (https://thoughtbot.com/sponsorship) of The Bike Shed! Transcript: AD: Flaky tests take the joy out of programming. You push up some code, wait for the tests to run, and the build fails because of a test that has nothing to do with your change. So you click rebuild and you wait. Again. And you hope you're lucky enough to get a passing build this time. Flaky tests slow everyone down, break your flow and make things downright miserable. In a perfect world, tests would only break if there's a legitimate problem that would impact production. They'd fail immediately and consistently, not intermittently. But the world's not perfect, and flaky tests will happen, and you don't have time to fix them all today. So how do you know where to start? BuildPulse automatically detects and tracks your team's flaky tests. Better still, it pinpoints the ones that are disrupting your team the most. With this list of top offenders, you'll know exactly where to focus your effort for maximum impact on making your builds more stable. In fact, the team at Codecademy was able to identify their flakiest tests with BuildPulse in just a few days. By focusing on those tests first, they reduced their flaky builds by more than 68% in less than a month! And you can do the same because BuildPulse integrates with the tools you're already using. It supports all the major CI systems, including CircleCI, GitHub Actions, Jenkins, and others. And it analyzes test results for all popular test frameworks and programming languages, like RSpec, Jest, Go, pytest, PHPUnit, and more. So stop letting flaky tests slow you down. Start your 14-day free trial of BuildPulse today. To learn more, visit buildpulse.io/bikeshed. That's buildpulse.io/bikeshed. CHRIS: Good morning, and welcome to Tech Talk with Steph and Chris. Today at the top of the hour, it's tech traffic hits. STEPH: Ooh, tech traffic. [laughs] I like that statement. CHRIS: Yeah. The Git lanes are clogged up with...I don't know. I got nothing. STEPH: [laughs] Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Steph Viccari. CHRIS: And I'm Chris Toomey. STEPH: And together, we're here to share a bit of what we've learned along the way. So, hey, Chris, what's new in your world? CHRIS: What's new in my world? Actually, I have a specific new thing that I can share, which is, as of the past week, I would say, switched from Trello over to Linear for product management. It's been great. It was a super straightforward transfer. They actually had an importer. We lost some of the comment threads on the Trello cards. But that was easy enough to like each Linear ticket has a link back to Trello. So it's easy enough to keep the continuity. But yeah, we're in a whole new world, a system actually built for maintaining a product backlog, and, man, it shows. Trello was a bunch of lists and cards and stuff that you could link between, which was cool. But Linear is just much more purpose-built and already very, very nice. And we're very happy with the switch. STEPH: I feel like you came in real casual with that news, but that is big news, that you did a switch. [laughter] CHRIS: How are you going to bury the lead like that? You switched project management...[laughter] I actually didn't think it was...I'm excited about it but low-key excited, which is weird because I do like productivity and task management software. So you would think I would be really excited about this. But I've also tried enough of them historically to know that that's never going to be the thing that actually makes or breaks your team's productivity. It can make things worse, but it can't make you great. That's the thing that I believe. And so it's a wonderful piece of software. I'm very excited about it but -- STEPH: Ooh, I like that. It can make you worse, but it doesn't make you great. That's so true, yeah, where it causes pain. Well, and it does make sense. You've been complaining a bit about the whole login with Trello and how that's been frustrating. But I haven't even heard of Linear. That's just...that's, I mean, you're just doing what you do where you bring that new-new. I haven't heard of Linear before. CHRIS: I try to live on the cutting edge. Actually, I deeply try to not live on the cutting edge at this point in my life. That early adopter wave, no, no, no, that's not for me anymore. But I've known a few folks who've moved to Linear. And everyone that I've spoken to who has moved to it has been like, "Yeah, it's been great." I've not heard anything negative. And I've heard or experienced negative things about every other product management tool out there. And so, it seemed like an easy thing. And it was a low-cost enough switch in terms of opportunity costs or the like, it took the effort of someone on our team moving those cards over and setting up the new system and training, but it was relatively straightforward. And yeah, we're super happy with it. And it feels different now. I feel like I can see the work in a different way which is interesting. I think we had brought in a Chrome extension for Trello. I think it's like Hello Epics or something like that that allows...it abuses the card linking functionality in Trello and repurchases that feature as an epic management thing. But it's quarter-baked is how I would describe it, or it's clearly built on top of existing things that were not intended to be used exactly in that way. So it does a great job. Hello Epics does a great job of trying to make something like parent-child list management stuff happen in Trello. But it's always going feel like an afterthought, a secondary feature, something that's bolted on. Whereas in Linear, it's like, no, no, we absolutely have the idea of projects, of course, and you can see burndown charts and things. And the thing that I do want to be careful about is not leaning too much into management. Linear has the idea of cycles or sprints, as many other folks think of them, or iterations or whatever you want to call them. But we've largely not been working in that mode. We've just continued to work through the next up list; that's it. The next up list should be prioritized and well defined at the top and roughly in priority order. So just pick up the next card and work on it. And we just do that every single day. And now we're in a piece of software that has the idea of cycles, and I'm like, oh, this is vaguely interesting. Do we want to do that? Oh, but if you're going to do that, you probably do some estimation, right? And I was like, oh no, now we're into a place that's...okay, I have feelings. I got to decide how to approach that. And so, I am intrigued. And I wonder if we could just say like ten carts that's how many come into a cycle, and that's it. And we use the loosest heuristics possible to define how we structure a cycle so that we don't fall into the trap of, oh, what's our roadmap going to look like six months from now? JK, what's anything going to look like six months from now? That's not a knowable fact. STEPH: I was just thinking where you said that you're moving into sprints or cycles, and then there's that push, well, now you got to estimate. And I just thought, do you? Do you have to estimate? [laughs] CHRIS: We need a burndown chart through 2024, and it must be meticulously accurate down to the hour. STEPH: I think meticulously wrong is how that goes. [laughs] CHRIS: Which is the best kind of wrong. If you're going to be wrong, be meticulous about it. STEPH: Be thorough about it. [laughs] Yeah, the team that I'm on right now, we have our bi-weekly planning, and we go through the board, and we pull stuff in. But there's never a discussion about estimation. And I hadn't really appreciated that until just now. How we don't think about how long is this going to take? We just talked about, well, what's in-flight? And how much work do people still have going on? And then here's the list of things we can pull in. But there's always a list that you can go back to. Like, it's very...we pull in the minimum and knowing that if we run out of work, there's another place to go where there's stuff that's organized. And I just love that cadence, that idea of like, let's not try to make guesses about the future; let's just have it lined up and ready for us to go when we're ready to pull it in. Although I know, that's also coming from a very developer's perspective, and there are product managers who are trying to communicate as to when features are going to get out into the world. So I get that there's a balance, but I still have strong feelings and hesitations around estimating work. CHRIS: Well, I feel like there is a balance there. And so many things in history are like, well, this is an overcorrection against that, and that's an overcorrection against this. And the idea that we can estimate our work that far out into the future that's just obviously false to me based on every project I've ever worked on that has tried to do it. And it has always failed without question. But critically, there is the necessity to sync up work and like, oh, marketing needs to plan the launch of this feature, and it's a critical one. What's it going to look like? When's it going to be ready? You know, we're trying to go for an event, it's not just know...we developers never estimate anything past the immediate moment where like, that's not acceptable. We got to find a middle ground here. But where that middle ground is, is interesting. And so, just operating in the sort of we do work as it comes up is the easiest thing because no one's lying about anything at that point. But sometimes you got to make some guesses and make some estimations. And then it gets into the murky area of I believe with 75% confidence that in three weeks, we will have this feature ready. But to be clear, I said with 75% confidence that means one-quarter of the time; we will not be there at that date. What does that mean? What does that failure mode look like? Let's talk about that. And can you have honest, open, transparent, useful conversations there? That's the space that it becomes more subtle if you need to do that. STEPH: You're reminding me of a conversation that I had with someone where they shared with me some very aggressive team goals. And it was a very friendly conversation. And they're like, "How do you feel about aggressive goals?" And I was like, "Well, it depends. How do you feel about aggressive failure?" Because then once I know where you stand there, then we can talk about aggressive goals. Now, if we're being aggressive, but then we fail to achieve that, and it's one of those, okay, we didn't meet the goal that we'd expected, but everything is fine, and it's not a big deal, then I am okay. Sure, let's shoot for the stars. But if it's one of those, we are communicating these goals to the outside world, and it's going to become incredibly important that we meet these goals, and if we don't, then things are going to go on fire, people are going to be in trouble, and it's just going to be awful, then let's not set aggressive goals. Let's not box ourselves into a space where we are setting ourselves up to fail or feel pain in a meaningful way. I agree that estimations are important, especially in terms of you need to collaborate with other departments, and then also just to provide some sense of where the product is headed and when things may be released. I think estimations then just become problematic when they do become definite, and they're based on so many unknowns, and then when I don't know is not an answer. So if someone asked, "What's your estimate for this?" And the very honest real answer is I don't know, like, we haven't done this type of work before, or these are all the unknowns, and then someone's like, "Well, let's just put an estimation of like two weeks on it," and they just sort of try to force-fit it into being what they want, then that's where it starts to just feel incredibly problematic. CHRIS: Yeah, estimation is a very murky area that we could spend entire episodes talking about, and in fact, I think we have a handful of times. So with that, Linear has been great. We're going to see just how much or how little estimation we actually want to do. But it's been a very nice addition to the toolset. And I'll let you know as we deepen our usage of it what the experience is like, but that's the main thing that's new in my world. What's new in your world? STEPH: Well, before we bounce over to my world, you said something that has intrigued me that has also made me start reflecting on some of the ways that I like to work. And you'd mentioned that you have this prioritized backlog that people are pulling tickets from. And I don't know exactly if there's a planning session or how that looks, but I have recognized that when I am working with a team, and we don't have any planning session, if everybody is just pulling from this backlog, that's being prioritized by someone on the team, that I find that a bit overwhelming. Because the types of work being done can vary so drastically that I find I'm less able to help my colleagues or my teammates because I don't have the context for what they're working on. It surprises me. I'm like, oh, I didn't even know we're working on that feature, or I don't have the context for what's the problem that we're trying to solve here. And it makes it just a lot harder to review and then have conversations with them. And I get overwhelmed in that environment. And I've recognized that about myself based on previous projects that were more similar to that versus if I'm on a project where the team does get together every so often, even if it's high level to be like, hey, here's the theme of the tickets that we're working on, or here's just some of the stuff, then I feel much more prepared for the work that is coming in and to be able to context switch and review. And yeah, so I've kind of learned that about myself. I'm curious, are you similar, or how does that work for you? CHRIS: I'm definitely similar. And I think probably the team is closer to what you're describing. So we do have a planning session every week, just a quick 30-minute scan through the backlog, look at the things that are coming up and also the larger themes. Previously, Epics and Trello now projects and Linear. But talking about what are the bigger pieces of work that we're moving on, and then what are the individual tickets associated with that that we'll be expecting to work on in the next week? And just making sure that everyone has broad clarity around what that feature set is. Also, we're a very small team at this point. Still, we're four people total, but one of the developers is on a break for a couple of weeks this summer. And so there are really only three of us that are driving on the code. And so, with three of us working on the projects, we try very intentionally to have significant overlap between the various...like, we don't want any one person to own any portion of things at this point. And so we're doing a good amount of pairing to cross-pollinate and make sure everyone's...not everyone's aware of everything, but at least one other person is sufficiently aware of everything between the three of us. And I think that's been working well. I don't think we have any major gaps, save for the way that we're doing our mobile architecture that's largely managed by one of the developers on the team and a contractor that we're working with to help do a lot of the implementation. That's a known we chose to silo that thing. We've accepted the cost of that for now. And architecturally, the rest of us are aware of it, but we're not like in the Swift code writing anything because I don't know how to write Swift at this point. I'd love to learn it. I hear good things about the language. [12:26] So yeah, I think conceptually very similar to what you're describing of still want to have people be able to review. Like, I don't want to put up a PR and people be like, I don't know, that looks like code, I guess. I'm not sure what it does. Like, I want it to be very...I want us all to be roughly on the same page, and thus far, we are. As the team grows, that will become trickier to maintain because there are just inherently probably more things that are moving, more different feature areas and surface area that we're tackling in any given week, or there are different ways to approach that. I know you've talked about having a limited number of themes for a given cycle, so that's an idea that can pop up. But that's something that we'll figure out as we get further. I think I'm happy with where we're at right now, so yeah, that's the story there. STEPH: Okay, cool. Yeah, all of that resonates with me, and thinking about it a little more deeply in this moment, I'm realizing I think something you said helped me put this together where when I'm reviewing someone's change, I'm not necessarily just looking to see does your code work? I'm going to trust you that your code works. I may have thoughts about design and other things, but I really want to understand more what's the change to the product that we're making? What's the goal that we're looking to achieve? How are we measuring this? And so if I don't have that context, that's what contributes to that feeling of like, hard context switching of not just context switching, but now I have to level myself up to then understand the problem that's being solved by this. Versus had I known some of the themes going into that particular cycle or sprint, I would have felt far more prepared for that review session versus having to then dig through all the data and/or tickets or talk to someone. Well, switching back to what's going on in my world, I have a particular tweet that I want to share, and it's one that Joël Quenneville brought to my attention. And it just resonates so much with all the type of work that I'm doing right now. So I'm going to read the tweet, and then we'll link to it in the show notes as well. But it's from Curtis Einsmann, and Curtis wrote: "In software engineering, rabbit holes are inevitable. You will research libraries and not use them. You'll write code just to delete it. This isn't a waste; sometimes, you need to go down a few wrong paths to get to the right one." And that describes all the work that I'm doing right now. It's a lot of exploratory, a lot of data-driven work, and finding ways that we can reduce the time that it takes to run RSpec on CI. And it also ties in nicely to one of the things that I think we talked about last week, where we discovered that a number of files have a high runtime variance. And I've really dug into the data there to understand, okay, is it always specific files that have these high runtime variants? Are there any obvious contributions to what's causing this? Are we making real network calls that then could sometimes take a long time to return? And the result is there's nothing obvious. They're giant files. The number of SQL commands that are being run for each file varies drastically. They're all high, but it's still very different. There's no single fact about these files that has really been like, yes, this is what's causing these files to have such a runtime variance. And so while I've been in the data, I'm documenting it, and I'm making a list and putting it all together in a ticket so at least it's there to look at later. But I'm going to move on. It's one of those I would love to know what's causing this. I would love to address it because it would put us in an ideal state for how we're distributing tests, which would have a significant impact on our runtime. But it also feels a little bit like chasing my tail because I'm worried, like with some of the other experiments that we've done in the past where we've addressed tentpoles, that as soon as you address the issue for one or two files, then other files start having the same problem. And you're just going to continue to chase and chase, and I don't want to be in that. So upfront, this was one of those; hey, let's look at the data. If there's something obvious, let's address it; if not, move on. So I'm at that point today where I'm wrapping up all of that data, and then I'm going to move on, move on to the next thing. CHRIS: There's deep truth in that tweet that you shared at the start of this segment. The idea like if we knew the work that we had to do at the front, yeah, we would just do that, but often, we don't. And so, being able to not treat it as a failure when something doesn't work out is, I think, so critical. I think to expand on the idea just a tiny bit, the idea of the scientific method, failure is totally an option and is part of science. I remember watching MythBusters, and Adam Savage is just kind of like, "Failure is always an option," and highlighting that as part of it. Like, it's an outcome. You've learned something. You have a new data point. You can take that and then carry it forward with you. But it's rough in the moment. And so, I do think that this is a worthwhile thing to meditate on. And it's something that I've had to revisit a handful of times in my career of just like, man, I feel like I've just been spinning my tires all week. I'm like, we know what we want to get done, but just each approach I take isn't working for one reason or another. And then, finally, you get to the end. And then you've got this paragraph-long summary of all the things that didn't work in your PR and one-line change sort of thing. And those are painful, but they're part of the game. Like, that is unavoidable. I have not found a way to just know how to do the work upfront always. I would love that. I would sign up for whatever seminar was selling that. I wouldn't. I would know that that seminar is a lie, actually. But broadly, I'm intrigued by the idea if someone were selling that, I'd be like, well, I mean, pitch me on it. Tell me why I should believe you; I don't, just to be clear. But yeah. STEPH: This project has really helped me embrace always setting a goal or a question upfront about what I'm wanting to achieve or what I'm looking to answer because a number of times while Joël and I have been spelunking through data...And then so originally, with the saga, we started out with why doesn't our math match reality? We understand that if these tests are distributed perfectly across the CPUs, then that should cut the runtime in half. But yet, we weren't seeing that even though we had addressed the tentpoles. So we dug into understanding why. And the answer is because they're not perfectly distributed, and it's because of the runtime variance. And that was a critical moment to look back and say, "Did we achieve the goal?" Yes, we identified the problem. But once you see a problem, it's just so easy to dig in and keep going. It's like, well, now I want to know what's causing these files to have a runtime variance. But it's one of those we achieved our goal. We acknowledged upfront that we wanted to at least understand why. Let's make a second decision, do we keep going? And I'm at that point where, frankly, I probably dug in a little more than I should because I'm stubborn. But I'm recognizing that now's the time to back away and then go back and move on to the next high-priority item, which is converting for funsies; I'll share. The next thing is converting Test::Unit test over to RSpec because we have, I think, around 25 tests that are written in Test::Unit. And we want to move them over to RSpec because that particular just step in the build process takes a good three to four minutes. And part of that is just booting up Rails and then running the tests very fast. And we're underutilizing the machine that's running them because it's only 25 tests, but there are 86 CPUs to run it. So we'd really like to combine those 25 tests with the rest of the RSpec suite and drop that step. So then that should add minimal time to the overall build but then should take us down at least a couple of minutes. And then also makes it easier to manage and help folks so that way, there's one consistent testing framework that's in use versus having to manage some of these older tests. CHRIS: It's funny; I think it was just two episodes back where we talked about why RSpec, and I think both of us were just like, well yeah. But I mean, if there are tests and another, like, it's fine, you just leave them with the exception that if there's like 2% of our tests are in Test::Unit, and everything else is in RSpec, yeah, maybe that that conversion efforts seem totally worth it. But again, I think as you're describing that, what I'm hearing is just like the scientific method, just being somewhat structured in the approach to what's the hypothesis? And what's the procedure we're going to use to determine if that hypothesis is true or false? And then what do we do? And then what are the results? And then you just do that on loop. But being not just sort of exploring. Sometimes you have to be on exploratory mode. But I definitely find that that tiny bit of rigor of just like, wait, okay, before I actually do anything, what do I think is going on here? What's my guess? And then, okay, if that guess were true, what would I be able to observe in the world? Okay, here we go. And just that tiny bit of structure is so...it sometimes feels highly formal to go into that mode and be like, no, no, no, let me take a step back. Let me write down my thoughts. I'm going to have a little checklist and do the thing. But I've never regretted doing it. I will say that. I have deeply regretted not doing it. I feel like I should make a list of things that fit that structure. I've never regretted committing in Git ever. That's been great. I've always been able to unwind it, but I've never been able to not unwind it or the opposite. I've regretted not committing. I have not regretted committing. I have regretted not writing out my hypothesis or approach. I have not regretted doing it. And so, yeah, this feels like it falls firmly in that category of like, it's worth just a tiny bit of structure. There's a reason it is the scientific method. STEPH: Yeah, I agree. I've not regretted documenting upfront what it is I look to achieve and how I think I'm going to answer the question. That has been immensely helpful. Because then I also forget, like, two weeks ago, I'll be like, wasn't there a question around why this was happening, and I need to understand? And all of that was so context-heavy that as soon as I'm out of the thick of it, I completely forget it. So if I care about it deeply or if I want to be able to revisit it, then I need to document it for myself. It's given me a lot of empathy for people who do more scientific research around, oh my gosh, like, you have to document everything you do and then still be able to prove it five years from now or however long. I'm just throwing out numbers. And it needs to be organized enough that someone, if they do question your research that, then you have it there. My research documents would not withstand scrutiny at this point because they are still just more personal notes. But yes, it's given me a lot of empathy and respect for people who do run very serious research, experiments, and trials, and then have to be able to prove it to the world. Pivoting just a bit, there's a particular topic that resonated with both you and I; in fact, it's a particular tweet. And, Louie, I do apologize if I mispronounce your last name, but Louie Bacaj. And we'll include a link in the show notes to the tweet, but Louis shared, "I managed multiple engineering teams before quitting tech. Now that I quit, I can speak freely. Here are 12 things your manager may not be telling you, but I know for a fact will help you." So there are a number of interesting discussions and comments that are in this thread. The one thing in particular that really caught my attention is number 10, and it's "Advocate for junior developers." So they said that their friend reminded them that just because you don't have 10-plus years of experience does not mean that they won't be good. Without junior engineers on the team, no one will grow. Help others grow; you'll grow too. And that's the part that I love so much is that without junior engineers on the team, no one will grow because that was very thought-provoking for me. It's something that I find that I agree with deeply, but I hadn't really considered why I agree with that so much. So I'm excited to dive into that topic with you. And then, as a second topic to go along with that is, can juniors start with a remote team? I think that's one of the other questions when you and I were chatting about this. And I'm intrigued to hear your thoughts. CHRIS: A bunch of stuff there. Starting with the tweet, I love elements of this. Some of it feels like it's intentionally somewhat provocative. So like, without junior engineers on the team, no one will grow. That feels maybe a little bit past the bar because I think we can technically grow, and we can build things and whatnot. But I think what feels deeply true to me is truly help others grow; you'll grow too. The act of mentoring, of guiding, of training, of helping someone on their journey will inherently help you grow and, I think, change the way that you think about the work. I think the beginner mind, the earlier in the career folks coming into a codebase, they will see things fundamentally differently in a really useful way. It's possible that along your career, you've just internalized things. You've been like, yeah, no, that was weird. But then I smashed my head against it for a while, and now I understand this thing. And it just makes sense to me. But it's like, that thing actually doesn't make sense. You have warped your mind to match the thing, not, quote, unquote, "come to understand it." This is sounding too judgmental to people who've been in the industry for a while, but I found this of myself. Or I can just take for granted things that took a long time to adapt my head to, and if anything, maybe I should have pushed back a little more. And so, I find that junior engineers can be a really fantastic lens for the complexity of a project. Like, the world is truly a complex place, and that's just true. But our job as software engineers is to tame that complexity and manage it. And so, I love the mindset that can come or the conversations that can come out of that. And it's much like test-driven development is a pressure on the complexity of your code, having junior engineers join the team and needing to explain how all of the different features work, and what the overall architecture is, and the message passing under this and that, it's a really useful conversation to have. And so that "Help others grow; you'll grow too" feels deeply, deeply true to me. STEPH: Yeah, I couldn't agree more in regards to how juniors really help our team and especially, as you mentioned, with complexity and ¬having those conversations. Some of the other things that have come to mind for me as well around the importance of having junior developers on your team...and maybe it's not specifically they're junior developers but that you just have a variety of experience on your team. It's going to help you lean into a culture of learning because you have people that are at different stages of their career. And so you want an environment where people can learn together, that they can fail together, and they can be public about it. And having people that are at different stages of their career will lead, well, at least ideally, it'll lead to more pair programming. It's going to lead to more productive code reviews because then people can ask more questions around why did you choose this, or why are you doing that? Versus if everybody is at the same level, then they may just intuitively have reasons that they think someone did something. But it takes someone that's a bit new to say, "Hey, why did you choose this?" or to bring up some other ideas or ways that they could pursue it. They may bring in new ideas for, like, why has the team always done something this way? Let's think about new ways that we could do this. Or maybe this is really unfriendly, the way that we're doing this, not just for junior people but for people that are new to the team. And then there's typically less knowledge siloing because then you're going to want to pair the newer folks with the more experienced folks. So that way, you don't have this more senior developer who's then off in a corner working by themselves. And it's going to improve your communication skills. There's just...I realized I'm just rambling because I feel like there are so many benefits that go along with having a variety of people on your team, especially in terms of experience. And that just leads to such a better learning environment and, ultimately, better software and better products. And yet, I find that so many companies won't embrace people that are newer to software. They always want the senior developers. They want the 10x-er or whatever those are. They want the people that have many, many years of experience. And there's so much value that comes from mentoring the next group of developers. And it's incredibly frustrating to me that one, companies often aren't open to it. But honestly, more than that, as long as you're upfront and honest about like, hey, this is the team that we need right now to build what we're looking to build, I can get past that; I can understand that. But please don't then mislead people and say that you're a junior-friendly team, and then not be prepared. I feel like some teams will go so far as to say, "Yes, we are junior-friendly," and they may even tweak their interview process to where it is a bit more junior-friendly. But then, by the time that person joins the team, they're really not prepared. They don't have an onboarding plan. They don't have a mentorship plan. And then they fail that person because that person has worked hard to get there. And they've worked hard to bring that person onto the team, but then they don't have a plan from there. And I've seen it too many times. And it just frustrates me so much because it puts that junior person in such a vulnerable state where they really have to be an incredible self-advocate to then overcome those hurdles from a lack of preparation on that company's part. Okay, I think that's my event. I'm sure I could vent about this a lot more, but I will cut it off there. That's the heart of it. CHRIS: I do think, like, with anything else, it's something that we have to be intentional about. And so what you're saying of like, yeah, we're a junior-friendly company, but then you're just hiring folks, trying to find folks that may work at a slightly lower pay grade, and that's what that means to you. So like, no, no, that's not what this is. This needs to be something different. We need to have a structure and an organization that can support folks at different points in their career. But it's interesting to me to think about the sort of why of it. And the earlier part of this conversation, we talked about some of the benefits that can come organizationally from it, and I do sincerely believe in that. But I also believe that it is fundamentally one of the best ways to find really talented people early on in their career and be in a position to hire them where maybe later on in their career, that just wouldn't happen naturally. And I've seen this play out in a number of organizations. I went to Northeastern University for my college, and Northeastern is famous for the co-op program. Northeastern sounds really fancy. Now I learned that they have like a 7% acceptance rate for college applications right now, which is wildly low. When I went to Northeastern, it was not so fancy. So just in case anyone's hearing that and they're like, "Oh, Northeastern, wow." I'm not that fancy. [laughs] But they did have the co-op then, and they still have it now. And the co-op really is a differentiating thing. You do three six-month rotations. And it is this fundamental differentiator in terms of when you're graduating. Particularly, I was in mechanical engineering. I came out, and I actually knew what that meant in the world. And I'd used Outlook, and I knew what a water cooler was and how to talk near one because that's a critical thing to learn in the world. And really transformative experience for me. But also, a thing that I observed was many of my friends ended up working at companies that they had co-opted for. I'm one of those people. I would say more than 50% of my friends ended up with a position at a company that they had done a co-op rotation with. And it really worked out fantastically. That organization and the individual got to try things out, experience. And then, I ended up staying at that company for a number of years, and it was a wonderful experience. But I don't know that I would have ended up there otherwise. That's not necessarily the way that would have played out. And similarly like, thoughtbot has the apprenticeship. And I have seen so many wonderful developers start at that very early point in their career. And there was this wonderful structure around them joining the thoughtbot team, intentional, structured, supported. And then those folks went on to be some of the most talented developers that I've ever worked with at a wonderfully talented organization. And so the story of like, you should do this, organizations. This is a thing that you should invest in for yourself, not just for the individual, like, for both. Everybody wins in this case, in my mind. I will say, though, in terms of transparency, I currently manage a team of three developers. And we hired very intentionally for senior folks this early on in where we're at. And that was an intentional choice because I do believe that if you're going to be hiring more junior developers, that needs to be something that you do very intentionally, that you have a support structure in place, that you're able to invest the time in where they're at and make sure we have sort of... I think a larger team makes more sense to bring juniors into broadly. That's the thing that I'm saying out loud that I'm like, I should push on that a little bit. Is that true? Do I really believe that? But I think so, my actions obviously point to it. But it is an interesting trade-off space of how do you think about that? My hope is that as we grow as an organization, that we would then very intentionally start hiring folks in a more junior, mid-level to junior and be very intentional about how we support them, bring them into the organization, et cetera. I do believe it is a win-win situation for everyone when done with intention and with focus. STEPH: That's such an interesting bit that you just said because I very much appreciate when companies recognize do we have the bandwidth to support someone that's more junior? Because at thoughtbot, we go through periods where we don't have our apprenticeship that's open because we recognize we're not in a place that we can support someone. And we don't want to bring someone in unless we can help them be successful. I very much admire that and appreciate that about companies when they can perform that self-assessment. I am so intrigued. You'd mentioned being a smaller team. So you more intentionally hire senior developers. And I think that also makes sense because then you need to build up who's going to be in that mentorship pool? Because then people could leave, people could take vacations, and so then you need to have that support system in place. But yeah, I don't know what that then perfect balance is. It's like, okay, so then as soon as you have like five people available to mentor or interested in mentorship, it's like, then do you start bringing in the conversation of like, let's bring in someone that we can help build up and help them be successful and join our team? And I don't know what that magical number is. I do think it's important for teams to reflect to say, "Can we take on someone that's junior?" All the benefits of having someone that's junior. And then just being very honest and then having a plan for once that junior person does arrive. What does their career path look like while they've joined that team, and who's going to be that person that's going to help them level up? So not only make that choice upfront of yes, we are bringing someone on but let's also think about like the first six months of their work here at the company and what that's going to look like. It feels like an important step that a lot of companies fail to do. And I think that's why there are so many articles that then are like, hey, if you're a junior dev, here's all the things that you should do to be the best junior dev. That's fabulous. And we're constantly shoring up junior devs to be like, hey, here's all the things that you need to be great at. But we don't have as many conversations around; hey, here's all the things that your manager or the rest of your team should be great at to then support you equally as you are also doing your best to meet them. Like, they need to meet you halfway. And I'm not completely unsympathetic to the plight; I understand. It's often where I've seen with teams the more senior developers that have very strong mentorship communication skills are then also the teammates that get pulled into all the meetings and all the different projects, so then they are less available to be that mentor. And then that's how this often fails. So I don't think anybody is going into this intentionally, but yet, it's what happens for when someone is new and joining a team, and it hasn't been determined the next six months what that person's onboarding and career path looks like. Circling back just a bit, there's the question around, can juniors start with a remote team? I can go first. And I'm going to say unequivocally yes. There's no reason a junior can't start with a remote team. Because all the things that I feel strongly about come down to how is your team going to plan for this person? And how are they going to support this person? And all the benefits that you get from being in an office with a team, I think those do exist. And frankly, for someone like myself, it can be easier to establish a bond with someone that you get to see each day, get to see in person. You can walk up to their desk and can say, "Hey, I've got a question for you." But I think all those benefits just need to be transferred into a remote-friendly way. So I think it does ratchet up how intentional you have to be with your team and then onboarding a junior developer. But I absolutely think it's doable, and we should do it. CHRIS: You went with unequivocally yes as your answer. I'm going to go with a qualified maybe as my answer. I want this to be true, and I think it can be true. But I think it takes all the more intentionality than even what we've been describing. To shift the question around a little bit, what does remote work mean? It doesn't just mean we're doing the work, but we're separate. I think remote work inherently is at its best when we also are largely async first. And so that means more structured writing. The nature of the conversation tends to be more well-formed in each interaction. So it's like I read a big document, and then I pass it over to you. And at your leisure, you respond to it with a bunch of notes, and then it comes back to me. And I think that mode of interaction, while absolutely wonderful and something that I love, I think it fits really well when you're a little bit further on in your career when you understand things a little bit better. And I think the dance of conversation is more useful earlier on and so forth. And so, for someone who's newer to a team, I think having the ability to ask a quick question over and over is really useful to someone who's early on in their career. And remote, again, I think it's at its best when it's async. And those two are sort of at odds. And so it's that mild tension that gives me pause of like, something that I think that makes remote work great I do think is at least a hurdle that you would have to get over in supporting someone who's a little bit newer. Because I want to be deeply present for someone who's newer to their journey so that they can ask a lot of questions so that I am available to be interrupted regularly. I loved at thoughtbot sitting next to someone and being their mentor and being like, yeah, anytime you want, just tap on my desk. If I got my headphones on, that doesn't mean I'm ignoring you; it means I just need to make the sounds go away for a minute because that's the only way my brain will work. But feel free to just tap on my desk or whatever and grab my attention for a moment. And I'm available for that. That's an intentional choice. That's breaking up my continuity of the day, but we're choosing that for a reason. I think that's just a little harder to do in a remote context and all the more so if we're saying, hey, we're going to try this async thing where we write structured documents, and we communicate in these larger, more well-formed, communicates back and forth. But I do believe it can be done. I think it should be done. I just think it's all the harder for all of those reasons. STEPH: I agree that definitely makes it harder. But I'm going to push a little bit and say that when you mentioned being deeply present, I think we can be deeply present with someone and be remote. We can reduce the async requirements. So if you are someone that is more senior or more accustomed to the team, you can fall back to more of those async ways to communicate. But if someone is new, and needs more mentorship, then let's just set up time where we're going to literally hang out for a couple of hours each day or whatever pairing environment works best for them because pairing can also be exhausting. But hey, we're going to have a check-in each day; maybe we close out each day and touchpoint. And feel free to still message me and interrupt me. Like, you're going to just heighten your availability, even though it is remote. And be aware, like, hey, this person could message me at more times, and I'm okay with that. I have opted into this form of communication. So I think we just take that mindset of, hey, there's this person next to me, and I'm their mentor to like, hey, they're not next to me, but I'm still their mentor, and I'm still here for them. So I agree that it's harder. I think it falls on us and the team and the mentors to change ourselves versus saying to juniors, "Hey, sorry, it's remote. That's not going to work for you." It totally works for them. It's us, the mentors, that need to figure out how to make it work. I will say being on that mentor side that then not being able to see someone so if they are next to me, I can pick up on body language and facial expressions, and I can tell when somebody's stuck. And I can see that they're frustrated, or I can see that now's a good time for me to just be like, "Hey, how's it going? What are you working on? Or do you need help with something?" And I don't have that insight when I'm away. So there are real challenges like that that I don't know how to address. I have gone the obnoxious route [laughs] where I just message people, and I'm like, "Hey, how's it going? How's it going? How's it going?" And I try not to do that too much. But I haven't found a better way to manage that other than to constantly check in because I do have less feedback from that person that I'm working with unless they are just incredibly open about sharing when they're stuck. But typically, when you're newer to a team or newer to a career, you're going to be less willing to share when you're stuck. But yeah, there are some real challenges, but I still think it's something for us to figure out. Because otherwise, if we cut off access for remote teams to junior folks, I mean, that's where we're headed. There are so many companies and jobs that are headed remote that not being junior friendly and being remote in my mind is just not an option. It's something that we need to figure out. And it's hard, but we need to figure it out. CHRIS: Yeah, 100% on we need to figure that out and that that's on us as the people managing and structuring and bringing folks into teams. I think my stance would be like, let's just be clear that this is hard. It takes effort to make sure that we've provided a structure in which someone newer to a team can be successful. It takes all the more effort to do so in a remote context, I think. And it's that recognition that I think is critical. Because if we go into this with the wrong mindset, it's like, oh yeah, it's great. We got this new person on the team. And yeah, they should be ready to go in like two weeks, right? It's like, no, no, this is a different thing. We need to be very clear about it. This is going to require that we have someone who is able to work with them and support them in this. And that means that that person's output will likely be a little bit reduced for the period of time that we're talking about. But we're playing a long game here. Let's make sure we're clear on that. This is intentional. And let's be clear, the world of hiring and software right now it's not like super easy. There aren't way more software developers than there are jobs; at least, that's been my experience. So this is something absolutely worth investing in for just core business reasons and also good for people. So hey, it's a win-win. Let's do it. Let's figure it out. But also, let's be clear that it's going to be a little tricky along the way. So, you know, let's be intentional about that. But yeah, obviously do it, got to do it. STEPH: Wait, so I feel like we might have circled back to unequivocally yes. [laughs] Have we gotten there, or are you still on the fence? CHRIS: I was unequivocally yes from the beginning, but I couched it in, but...yeah, I said other things. You're right. I have now come around; let's say to unequivocally yes. STEPH: [laughs] Cool. I don't want to feel like I'm forcing you to agree with me. [laughs] But I mean, we just so rarely disagree. So we've either got to identify this as something that we disagree on, which would be one of those rare occasions like beer and Pop-Tarts. CHRIS: A watershed moment. Beer and Pop-Tarts. STEPH: Yeah, those are the only two so far. [laughter] CHRIS: Not together also. I just want to go on record beer and Pop-Tarts; I don't think would be...anyway. STEPH: Ooh, I don't know. It could work. It could work. CHRIS: Well, there's another thing we disagree on. STEPH: I would not turn it down. If I was eating a Pop-Tart, and you're like, "Hey, you want a beer?" I'd be like, "Sure," vice versa. I'm drinking a beer. "Hey, you want a Pop-Tart?" "Totally." CHRIS: Okay. Well yeah, if I'm making bad decisions, I'm obviously going to chain them together, but that doesn't mean that they're a good decision. It's just a chain of bad decisions. STEPH: I feel like one true thing I know about you is that when you make a decision, you're going to lean into it. So like, this is why you are all about if you're going to have a Pop-Tart, you're going to have the highest sugary junk content Pop-Tart possible. So it makes sense to me. CHRIS: It's the Mountain Dew theorem, yeah. STEPH: I didn't know this had a theorem. The Mountain Dew theorem? CHRIS: No, that's just my name. Well, yeah, if I'm going to drink soda, I'm going to drink Mountain Dew, the nonsense nuclear option of soda. So yeah, I guess you're describing me, although as you say it back to me, I suddenly feel very, like, oh God, is this who I am as a person? [laughs] And I'm not going to say you're wrong. I'm just going to spend a little while thinking about some stuff. STEPH: I mean, you embrace it. I think that's lovely. You know what you want. It's like, all right, let's do this. Let's go all in. CHRIS: Thank you for finding a wonderfully positive way to frame it here at the end. But I think on that note, should we wrap up? STEPH: Let's wrap up. CHRIS: The show notes for this episode can be found at bikeshed.fm. STEPH: This show is produced and edited by Mandy Moore. CHRIS: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review on iTunes, as it really helps other folks find the show. STEPH: If you have any feedback for this or any of our other episodes, you can reach us at @_bikeshed or reach me on Twitter @SViccari. CHRIS: And I'm @christoomey. STEPH: Or you can reach us at firstname.lastname@example.org via email. CHRIS: Thanks so much for listening to The Bike Shed, and we'll see you next week. ALL: Byeeeeeeee!!!!!!!! ANNOUNCER: This podcast was brought to you by thoughtbot. thoughtbot is your expert design and development partner. Let's make your product and team a success.