Podcasts about jason for

As property managers, you know how important communication is. Building solid relationships and creating trust is crucial in the industry, especially when trying to bring on new clients and doors. In this episode of the Property Management Growth Show, property management growth expert Jason Hull sits down with Sam Wakefield from Close it Now to talk about how you can level up your sales game to close more deals at a higher price point. You'll Learn [00:54] Vendor and Property Manager Relationships [09:43] Why You Attract Cheapo Clients [15:33] Building Trust in Sales [21:14] Shifting Perception: It's Not A, It's B [27:43] Learning to Improve Your Sales at DoorGrow Live 2025 Quotables “Truly all that sells is just communication.” “The second you start to develop a trend in your life, look internally because you are attracting exactly who you are.” “If we don't build the right culture, it's on us as a business owner.” “As business owners, we want to not give up big chunks of our life for just money. We want to be able to have something scalable.” Resources DoorGrow and Scale Mastermind DoorGrow Academy DoorGrow on YouTube DoorGrowClub DoorGrowLive Transcript [00:00:00] Sam: A lot of times property management companies think all the companies are the same, so they're looking for maybe cheaper, whoever's cheapest, a cheaper price. [00:00:07] Sam: But then what they get is a company that doesn't communicate and doesn't show up when they say they're going to, and it's really the old adage, you get what you pay for.  [00:00:14] Jason: All right. I am trying a new platform today. This is Jason Hull and I am a property management growth expert. If you're not familiar with me, I help grow and scale property management companies and I am really good at that. And so our company's DoorGrow and we are the world leaders of growing and scaling property management businesses. [00:00:35] Jason: I've helped thousands of property managers do that. And today my guest is Sam Wakefield. Hanging out here with Sam. Sam, welcome to the show.  [00:00:44] Sam: Thanks for having me on, man. I'm glad to be here.  [00:00:46] Jason: Hey, good to have you. So, I'm really excited to get into this. We had some really nice dialogue back and forth. You coach. [00:00:54] Jason: Well, I'll let you tell. What group, category of people do you coach and you help with them with sales and closing more deals, so.  [00:01:01] Sam: Yeah, absolutely. Yeah. So we do sales training and basically sales systems, whole operation systems within companies, but mostly sales focused for home services. So everything from HVAC, plumbing, electrical, and then even outside of that. Garage doors, or you name it. If someone improves a home, then we help the communication side of all of those companies.  [00:01:27] Jason: Got it. So in my industry, property management people would call those vendors. That's usually what they call them. They're like, "these are the vendors." And so we thought it was fun. I went on your podcast, we had this really fun dialogue. [00:01:39] Jason: I highly recommend you go check out Sam's episode with Jason Hull and go check that out. We were going back and forth because we had done a survey each to our audiences, like what's frustrating about HVAC companies and what's frustrating about property management companies. Right. And just seeing the disconnect that existed there. [00:01:56] Jason: Which was interesting. So, before we get into this, I want to read a quick message from our sponsor. This episode's sponsored by KRS SmartBooks. Do you have properties manage, and zero time for bookkeeping headaches? KRS SmartBooks is your secret weapon. They specialize in finances for busy property managers like you, with 15 plus years of real estate knowhow and skills in AppFolio, Yardi, and more imagine monthly reports magically appearing, and zero accounting stress. Sound good? Head to krsbooks.com to book your free discovery call, integrity, quality, and a dash of bookkeeping brilliance, that's KRS SmartBooks, and that's K as in Kansas, R as in Rogers, S as in Sam. Sam. All right, so cool. Now let's get into this. [00:02:45] Jason: So we're going to talk about closing deals, but why don't you give us my audience a little bit of background. How did you get into sales and then starting your own company, helping people with sales, and like, how'd you how did Close it Now come to be?  [00:03:00] Sam: Yeah, for sure. Thanks for that question. So, I've spent almost 20 years now in home services. [00:03:05] Sam: Most of my time has been in HVAC. I've done solar. I've done a lot of different trades over the years and, you know, so I launched the Close it Now company in 2019 because I really just recognized a place where there was not a lot of modern training because truly all that sells is just communication. [00:03:26] Sam: You know, it's how do we communicate clearer and in a way where we can educate so somebody can understand, one, what we're talking about, and two, why they should care and how it's going to make a difference in their life. So at the essence of that, so I was looking for some more modern training for my people at my company that I had at the time, and I didn't find anything out there. [00:03:48] Sam: So I just said, well, now we have a space for, you know, I have communication skills. I can train people. So that's when I launched the company in 2019 and so much of my career built up to that point of, and specifically how it affects here and why I'm here today. You know, I've worked with so many property management companies and individuals across 20 years of doing this. Yeah. So I've definitely learned a lot of best practices and a lot of the things not to do, you know? Got it. I all own my mistakes as well as, you know, coming across maybe property managers that I wouldn't work with again. Right. Yeah. So from all of that experience, you know, I started the training company, so I work with those home service companies to communicate better. [00:04:33] Sam: You know, a lot of it is, you know, of course, working directly with homeowners. But also there's a huge portion of all of those companies that, you know, rely on it and need property management companies to, you know, really help them stay in business and in turn they can turn around and, you know, help those property management companies to efficiently take care of properties. [00:04:58] Sam: But there's always seems to be this kind of struggle of, you know, that back and forth. So that's obviously why we're here today is a big part of that. But that's some of my history. I've been doing it 20 years. I started Close it Now six years or in, coming up on... yeah, April this year, next month is six years anniversary. [00:05:16] Sam: Nice. Of the company. And it's been a fun ride and we've definitely helped lots and lots of organizations to you know, to grow in a way.  [00:05:24] Jason: You're helping them close it now. All right. Yeah. Got it. All right. So you're just, you're helping these vendors close more deals, right? [00:05:31] Jason: So, property managers, I think would love to hear. You're on the other side of this relationship between property managers and vendors. What have you seen and what's the general feedback that you're noticing of the property management industry? What's kind of the vendor's perspective? [00:05:46] Jason: Because I know property managers, they get frustrated with vendors, right? They're like, "oh, the vendors like say you need something when you don't and like they don't like, it's difficult to reach them or this or whatever." Right. What are some of the complaints and gripes about property management companies? [00:06:03] Sam: Yeah. Complaints and gripes about property management companies. One of the big ones is, a lot of it is kind of the same thing is lack of communication. Okay. That's always one of the biggest complaints that comes up is, you know, we will get, you know, say someone, a property manager will call in for us to go evaluate a property. [00:06:21] Sam: We'll take an air conditioning issue or something like that, so we'll show up and then we're trying to call ahead. There's no clear information was given on who to call ahead to. Then we show up to the appointment, maybe the tenant's there, maybe not. A lot of times they're not there. [00:06:36] Sam: Okay. Then we can get ahold of the property manager to even get in the place. So now we're like dancing around in the circle of, okay, who do we contact? You get frustrated, move on to the next call, then the property manager calls and "Well, why'd you leave? Somebody was there." [00:06:50] Sam: Well, nobody was there. And so all of this just seems to happen very often. [00:06:55] Sam: Too often. Yeah. So it creates a stereotype. When the stereotype is created, that means of course there's a reason for it. Yeah. And so this is one of the big ones is the lack of communication. And I know that I've heard that the other direction as well. But so that's one of the things I hear the most. [00:07:11] Jason: Yeah. Got it. Yeah, so I'm sure when a vendor finds a property manager that does communicate effectively that there's clarity in that communication happening, and they've got good systems in place. The tenant's there, the tenant understands what's going on. Everybody's informed. Then those can be really great relationships to have. [00:07:34] Sam: Absolutely. Yeah. Those are, you know, the last the last organization I was at, I was with them, I was a sales manager and trainer for six years there. And I went through about 18 different property management companies to find two to three that were worth working with. Wow. And that was, you know, just sadly. We were always open to when a property management company came to us and we're like, "Hey, we, you know, we need you to do some work. We're looking for a new vendor." We're like, "sure. Absolutely. We'll try you out as well as you're trying us out." Right. But sadly, you know, the two or three that we did find great relationships with. They were fantastic relationships because yeah, we, you know, part of my ethics is our team was like, we will show up on time no matter what. [00:08:19] Sam: Right? We always do what we say. We will never, you know, recommend something that's not verifiable from our, you know, from our testing. We're not going to just guess at this because we're not guessing with anybody's, you know? Yeah. Investment. And at the same time when we, you know, say we're going to do the work, we do the work, and we show up to do the work, we say we're going to. [00:08:43] Sam: So that was my ethics statement I always led with. And then basically I would ask the property management company, can I expect the same thing from you guys? Right? And sure enough, the second that we met in the middle and said, yes, this is how we want to do business, those relationships were always the very best ones because sure, were we a few more dollars than the other contractor down the street? Sure. Yes. But we showed up when we said we were going to and we did the right work right the first time. And so, right. That's a big part of that disconnect, I think, is it seems like so many you know, a lot of times property management companies think all the companies are the same, so they're looking for maybe cheaper, whoever's cheapest, a cheaper price. [00:09:22] Sam: But then what they get is a company that doesn't communicate and doesn't show up when they say they're going to, and. It's really the old adage, you get what you pay for.  [00:09:30] Jason: You know, property managers have the same sort of problem is that a lot of people that are looking for a property manager are just looking for the cheapest price. [00:09:38] Jason: And they hate that. They're like, "we're not all the same." Right. So I, yeah, I think it's really important. I think this is dictated by the morals, the ethics, and the values of the business owner. It's always a top down thing. And so if the business owner is a cheapo, they attract cheapo clients and they deal with vendors through this cheapo lens, and this is where there's going to be a lot of mess and a lot of communication issues, and a lot of times the business owner, and this goes for any business and any industry, has a blind spot to the fact that they're cheap. But they're, you know, you're a cheapo if you're the person that's always looking for the stupid coupon code every time you buy everything online, you're always like hunting for that like. I don't have time to do that. [00:10:21] Jason: Like that's a massive waste of my time to go find, save 10% on some stupid a hundred dollars thing online, right? Right. Like, Ooh, I'm searching around. Right. Oh, I saved $10 even though I could have made a hundred thousand dollars. Like if I just like built something awesome, right? So I think there's a mindset issue is that these property managers or vendor business owners are not valuing their time enough. [00:10:45] Jason: If you value your time, you value other people's time. You then show up on time. You then like try to make sure, like your schedule is tight, you want to make sure your schedule is full. Like you, because you value your time and you feel that it's important. And if you really value your time enough as a person, you get things like assistance. [00:11:03] Jason: You get team members, like you get support because your time is so valuable that you want to go buy other people's time because it's less valuable than your time. Right, and this is how we scale our businesses over time is we are buying other people's time that are like they're willing to trade and give up their life chunks of their life for money. [00:11:24] Jason: And as business owners, we want to not give up big chunks of our life for just money. We want to be able to have something scalable. And so I think there's a mindset thing that we have to not be cheap. We have to operate with integrity, and then our team members need to have these values instilled in them, and if we don't build the right culture, it's on us as a business owner. [00:11:45] Jason: And if we don't build the right culture, we then don't have longevity in our business. We don't get return business, we don't get return clients. We don't get to have that really good vendor to continue to work with. We don't get to have that property owner continue to want to work with us, right? [00:12:00] Jason: Because we have showcased that we are not on top of things, or that we don't have the right values or that we don't have healthy mindset. And so I feel like. At the foundation of everything. It always comes back to mindset. A lot of times  [00:12:13] Sam: I a hundred percent agree with that. It, you know, it's funny that you're kind of started this conversation going down this path. [00:12:19] Sam: This is something that's been a very basically a soapbox for me, a big hot button. Yeah. You know, when I'm coaching... [00:12:26] Sam: jump on that soapbox, Sam. Let's go.  [00:12:27] Sam: Yeah. When I'm coaching and training people lately, especially at this last week especially... yeah. You know, I'm training people with sales and that type of focus, and they, of course, people always come to me, "Hey, how do I overcome these sales objections?" [00:12:43] Sam: You know, somebody says, "I want to get three bids, or somebody says, your price is too high, I want to shop around, or I need to think about it." Yeah. And instead of just going straight to, "well, here's the word track and how to handle these objections." Yeah. We always start with: anytime that you find a trend in your life, [00:13:00] Sam: so if you're getting the same consistent objection, say somebody's getting every single time they get to the end of their appointment and the homeowner or whoever they're talking to says, "I want to think about it." It's like the second you start to develop a trend in your life, look internally because you are attracting exactly who you are. [00:13:17] Sam: I would be willing to bet that person does the same thing when they shop. So then no wonder you're getting every single one of your clients is telling you, "I want to think about it." Or if when you shop, do you ask for say, "oh, I've got to get some three bids on this thing. I got to look around." Yeah. Well, no wonder the people you're selling to always have to get three bids because we attract who we are. Yeah. And it starts right here in the mind. And it's incredible how that works.  [00:13:43] Jason: Yeah. because if we're anxious, if we have that energetic sort of anxiety of that, like things are, it's expensive, and we go into that trying to sell it to somebody. Then they can feel that and we present it differently. And so we're like, "here's the price." And like, yeah, and it's worth it. And they can just, there's so many little subtle clues they pick up on that, Hey, this seems a little high. And because sometimes like if you're presenting to somebody and they're not what I call a cheapo, there's three types of buyers, cheapos, normals, and premiums I call them. [00:14:16] Jason: And normals are like, you typically like 60%. They're like the majority, 61%. The smallest group are usually the premium buyers, supposedly. But the idea is this: if you're a premium buyer and I present a price and I'm not even going to like flinch telling you about it, I'm like, "yeah, we've got this and this is what it costs and this," and they're going to go, "oh, this person feels really confident." [00:14:36] Jason: And it's just energetically how we present it. There's no like, "Hey, I'm trying to prep you for this price, you know, reveal because it's going to hurt a little bit." Right. Or if they just have the confidence and they know they're expensive, they might even just say, "Hey, we're one of the most expensive, but we're also one of the best. Let me tell you about your options." Right? So maybe they start with a pre-frame like that, but either way, they have this confidence that they know they have value and that it's worth it, and then they present it like that, then people would go, oh, okay, but if you have that anxiety deep down related to price and you know, you're this person if you're always looking for the coupon code or the discount code or you're trying to find the cheapest way to do something, then you've got a bit of that going on. [00:15:21] Jason: Because that's your identity. And so I've noticed this. Like in order to get people to be better salespeople, I can't just give them tactics. I have to give them identity. And so, and this is why my greatest sales hack, I call the Golden Bridge Formula. It's like it's the most authentic way to sell, which is your personal why connected to the business why connected to the prospect's why. Because we always trust motives. And the default assumption in sales, if I don't know your motive and you're trying to sell to me, is you want my money. [00:15:54] Sam: Right.  [00:15:54] Jason: And if I think that's your only motive is you want my money and you're willing to do whatever it takes to get that, then you're probably maybe even willing to be unethical in order to get that might be the assumption. [00:16:05] Jason: Right? So that's kind of the default assumption in sales. And so to correct that, if I tell somebody, "Hey. I'm Jason Hull. My personal why is to inspire others to love true principles. And so what that means is I love sharing what works and learning what works and teaching to others. I would do that for free, for fun, and so I created DoorGrow and our why at DoorGrow is to transform property management business owners and their businesses. [00:16:27] Jason: And so if our whole belief system is around helping people transform their businesses. So that allows me to basically feed my addiction to learning, coaches, masterminds, books, whatever, and turn around and be able to share what's working with others. And that's just fun for me. So I have a business that basically fulfills my lifestyle and allows me to have fun and do what I want to do. [00:16:51] Jason: And you, Mr. Property management, business owner, who I'm maybe selling to, want to grow your business. And so our interests are in alignment. My business is the bridge that connects your why to my why. We both get what we want. It's the ultimate win-win, right? Everybody wins. And so I've been able to take really terrible salespeople that are really bad at selling, and I just get them clear on their own identity. [00:17:14] Jason: Mm-hmm. Who they are, why they do what they do, and have them relate that to people and then people trust them. And sales and deals happened at the speed of trust.  [00:17:22] Sam: Oh my gosh, I love this so much. It's insanely powerful too when I'm teaching people how to do just introductions, you know? A super quick formula too for the property managers out there that are listening to that, even if you're property manager, you have to get good at sales. [00:17:38] Sam: Yeah, you have to be good at communication to be able to bring more doors into your portfolio. And so the way you know, a really easy formula for those homeowners when you're having that conversation, first of all, they've got to know who they're talking to. Yeah. You know, this belief, identity, you know, matrix that I actually I love to call, I just did a keynote. [00:17:59] Sam: It's funny for everybody listening. It's almost like Jason and I have read each other's notes, but we haven't. Just did a keynote, well that's maybe a month ago in Minnesota, that the entire talk was your thoughts, create your belief about yourself, your totally belief about yourself creates your identity, and then your identity creates your outcomes. [00:18:16] Sam: Yeah. And, but we have to go back and start with those thoughts. And so, but a simple, easy formula for property managers out there having this conversation is first of all, start asking permission for things. Yes. We can't just tell, right? If we can ask it as a question, ask it as a question. [00:18:36] Sam: So ask permission, like, "Hey, before we get started, do you mind if I take a quick minute and just introduce you to our company and myself."  [00:18:44] Sam: yeah.  [00:18:45] Sam: And so first of all, anytime a conversation starts, there's always this period of icebreaking, right? Yeah. Anytime anything new is introduced in anyone's environment, there's always stiffness until that moment of rapport happens and we relax a little bit. [00:19:00] Sam: Yeah. So taking a couple of minutes to just. "Hey, before we get started, do you mind if I introduce the company and a little bit about myself? Would that be all right?" Yes. So permission to it and then just take a few minutes because I mean, so many times we'll go through this crazy presentation and then we're asking somebody to buy from us and they don't even know who we are. [00:19:21] Sam: We never took the time to even introduce ourselves. Right.  [00:19:24] Jason: Yeah.  [00:19:24] Sam: Or they don't know thing about the company.  [00:19:25] Jason: Trying to immediately shove the product or service down their throat.  [00:19:28] Sam: Yeah. No wonder they need to think about it. They don't even know who you are. And so we introduce that first. [00:19:34] Sam: It's huge. And to just getting into the things. So that's the flow. It's like, okay, now that you know a little bit about us, tell us a little bit about you. What are you looking for? Right. So then you start that discovery process, and I'm sure you trained this but the discovery process is everything. [00:19:51] Sam: We have to understand the motive behind why they want to do things. Somebody just says, "Hey, I'm looking for a property manager." Okay, great. That's one thing. "Why do you would need a property manager? What are you trying to solve? What do we want to accomplish by having a property manager for your property?" [00:20:09] Sam: So we find out, what are the pain points? What are the issues that they're wanting to overcome? And then from there, we can create a, you know, craft a conversation around it. But until we know that, we're just stabbing in the dark and just guessing it. Yeah. Well, hopefully this will work.  [00:20:23] Jason: Right. Yeah. If we just jump right to offering solutions when we don't even ask what they need it's not very effective. [00:20:30] Jason: And then they're going to have a ton of objections.  [00:20:32] Sam: Yeah. Yeah. Absolutely. But yeah, that's the some of the complaints we have are the communication and the other one is just not responding once we find solutions, then give them to the property manager. [00:20:45] Sam: And then it's like ghosting for who knows how long until finally somebody gets back. And so that's the other side of the communication is not getting resolution once we actually, you know, we can do this work, but we're not going to sit around here all day to wait to get it approved. We have other appointments. [00:21:02] Sam: So do we want to reschedule?  [00:21:03] Jason: It's treating the vendor like they're high value, they're going to treat you like you're high value and they're going to prioritize you. And so it really is a mutual respect relationship that needs to be built. So, Sam, I also want to bring up to our audience, you are going to be coming [00:21:19] Jason: to speak at DoorGrow Live. Yeah. And you're going to be teaching some really cool stuff. Could you just touch on real quick what you're going to be sharing at this because I wanted to come bring you to expose my clients and my audience to what you're going to be sharing and maybe you can get some people pumped up for DoorGrow Live, so. [00:21:38] Sam: Absolutely. Yeah. So thank you for the invite as well. I'm super excited to be speaking for DoorGorw Live. It's my passion, in fact to be able to help people in their daily lives, especially in conversations like this, to make it easy. I am such a firm believer that sales should be easy. If it's not easy, we're overcomplicating it. And so what we're going to be talking about at the event is I'm going to give some really simple keys to better communication so people actually not only listen, but they understand what you're saying and, more importantly, why should they care? [00:22:18] Sam: So we're going to talk about something called, the benefit lens. We're going to talk about some easy word substitutions. We're not going to be learning scripts or anything. We're going to be, we're going to show any really easy ways to get immediate buy-in to what our conversation is. Nice. And how to recruit people to be raving fans and be on board. [00:22:38] Sam: And how to ask and get referrals because that's huge in...  [00:22:44] Sam: absolutely.  [00:22:44] Sam: ...something like a property management. If every third door you added also added another one from a referral, what would that do to your business? Yeah, absolutely. So not just asking for referrals, but actually asking in a way where actually get them. [00:22:57] Jason: Right. Yeah. If you're getting enough referrals, one, because you have a good reputation, you're doing a good job, but also because you have an intention and you're asking appropriately, you create this kind of virus of growth in your business where it's multiplying. [00:23:13] Jason: Every client becomes more clients.  [00:23:16] Sam: Yep. Absolutely. In fact, we can do a quick little as an example of some of the things we're going to cover. Are you open to doing a quick little role play with me on...  [00:23:24] Sam: all right. Let's do it.  [00:23:25] Sam: Some of the conversation here. Yeah. I love role play.  [00:23:28] Sam: Let's have fun.  [00:23:29] Sam: Yeah, for sure. [00:23:30] Sam: So I'm property manager. So before we do, give me a quick little context of what is a premium price property manager and what is like a middle range property manager. And so I'll know what I'm working with here. [00:23:44] Jason: Oh yeah. Usually our clients have three different price points for that reason. So, perfect. But let's say like, real typical in the marketplace is 10% is pretty normal. Okay? And this is not what we recommend. because our clients close more deals more easily at a higher price point. [00:23:59] Jason: So we have some special pricing models, but let's say 10%. Premium, maybe 12%, and the lower would maybe be like 8%.  [00:24:08] Sam: Got it. Got it. Perfect. Alright, so I'm the project manager. So I'm going to be a premium 12%. Yeah. So what we're going to do in this conversation, I'm going to ask for the business and you're going to give me a little bit of a price flinch with, "well, the other guy was only 10%." [00:24:23] Sam: Okay. And so we'll show a quick, easy way to handle that. All right. In a way that will make sense for everybody. So, alright, Jason, so, sounds like everything that you've talked about, can you see how all the things we do will take care of the concerns that you have? [00:24:38] Sam: Yeah, absolutely. Sounds great.  [00:24:40] Sam: Awesome. Perfect. So the next steps to get moving is you know, so we're just 12% of the monthly as for us to be able to take care of all of that. And this will just need a quick authorization on this form here and we can get started right away.  [00:24:55] Jason: Ooh, okay. Well, I was expecting, you know, I talked to a company down the street, they were like 10%, which seems to be a bit more normal. [00:25:04] Jason: I don't know.  [00:25:04] Sam: More normal?  [00:25:07] Jason: I've talked to a couple companies and a lot of them all do it at 10%. Could, like, is it possible you could do it at 10%?  [00:25:13] Sam: Oh, gotcha. So listen, I mean, so we were just 12%, but listen, we're not 2% higher or 2% more expensive. We're 2% better. Can I explain to you why that is? [00:25:25] Jason: Sure.  [00:25:26] Sam: Absolutely.  [00:25:27] Sam: So at that point, as a great company, you're going to have a hit list of all of the reasons why you're better than everybody else, and what makes you that premium company. I like it. So the minute we get that permission question in of, "Hey, we're not 2% more expensive, we're 2% higher, we're 2% better." [00:25:43] Sam: Then the permission question is, "can I show you why, or can I show you how?" And they say "Yes." Then we're going to, "okay, so what we do, it's..." never talk bad about the competition. Sure. But it's always with that perspective. "So what we do is this, and what we do is this, and what we do is this. We're always going to have the availability to be in contact, you know, 24/7 or you know, whatever all of the benefits is. [00:26:10] Sam: We're going through this huge benefit list. Yeah. And then when, once we, and it works like magic, once you get to about 10 or 12 things, especially when you know, those first 10 or 12 things are things the other companies don't do. Yeah. So many times that person will go, "you know what? You're right. You know what? You're right. Let's just go ahead and do it." Yeah.  [00:26:31] Jason: I mean, you go through those things you say, "so does that make sense why maybe we're 2% better?" And they're going to be like, "yeah."  [00:26:38] Jason: You've got agreement.  [00:26:39] Sam: Cool. Absolutely. And the other thing to do in this conversation, and this is really powerful too, so, you know, we'll take you know, what's a, what's the average rent that we'd be taking that percentage off of? [00:26:50] Jason: Let's say 2000 bucks.  [00:26:51] Sam: So 2000 bucks. That's what I was going to use. "So we're talking about 2% difference. So we're looking at $40 a month or $10 a week. Is it worth it to you for $10 a week to potentially fight the headache of, you know, your property management company not responding when you need them to respond, your tenants being really unhappy, the tenants turning over and over, for, I mean, $10 a week. Is it worth it to you for that?"  [00:27:22] Jason: Yeah.  [00:27:23] Sam: So if, I mean, if you're willing to roll the dice and take that chance, then of course you could do what you want. But if you want it done right and done once, so you're headache free and you're not going to have to, because the reason you hire a property manager is to be hands off. [00:27:35] Sam: Right? Yeah. Perfect. That's why what, that's what sets us apart. Next to any of the other companies around.  [00:27:43] Jason: Got it. So hypothetical property manager, Sam here, like believes. You can tell by listening to him, he believes in what he is selling. He believes he's worth that 12%. He believes he's worth that value, and I love that reframe. [00:27:58] Jason: One of the NLP hacks I teach clients is, it's not a, it's b, and he's like, "it's not that we're expensive or higher price, it's that we're 2% better." And so you're saying this is how you are looking at it. Here's how I want you to look at it. And that's a really cool correction. I love that right there. [00:28:16] Jason: Very powerful.  [00:28:17] Sam: The other part of that too is when you take, we're not talking about the total monthly, you know, we're talking about what's 12% or 10%? We're talking about 2% difference. Yeah. Is it worth it to you for a 2% difference to take the chance on having to deal with this, having to manage your own projects, having the headache, having the you know, the angry tenants or we don't have that problem. [00:28:42] Sam: And here's proof: review, testimony. Other people in the area, for people that use us just like you guys.  [00:28:49] Jason: Yeah. Awesome. Perfect. And you're going to share some really cool stuff I know at DoorGrow Live. I'm excited, man. Me too.  [00:28:56] Sam: Let's just tip of the iceberg. [00:28:57] Jason: For a salesman to be able to like build a coaching business, teaching sales like these are the best in the world at sales, and so I'm really excited to have you come. I've sold millions and millions of dollars of stuff. I love, I'm always learning more about sales, like this is something you can always continually learn more, so I love that little reframe. [00:29:17] Jason: That's a good one. I'm excited to hear what else you have to share. This is going to be really awesome. And if you're interested, go to doorgrowlive.com and get your tickets. Get your tickets. Our theme this year is innovating the future of property management, and we are bringing future ideas. [00:29:32] Jason: I'm going to be going over hybrid pricing, a new pricing model for property managers. This is the future. We're going to be sharing our DoorGrow hiring system. This is the future of how you're going to need to do hiring, so you're not making mistakes with hires, we're helping a lot of people replace their entire team. [00:29:48] Jason: So anyway, DoorGrow Live is going to be really freaking cool. So, yeah, and it's a holistic conference as well. We're bringing people from outside the industry, people that are related to different things. I've got a biohacking expert. We've got different things just to optimize your life as an entrepreneur and to make you better at what you do. [00:30:05] Jason: So this is going to be really cool. So, well, Sam anything else we should touch on?  [00:30:10] Sam: You know, there's so much we could cover.  [00:30:12] Jason: There's a lot. We'll save it for DoorGrow Live. How can people that, if they're listening, they're like, I'm a vendor, or I've got this, or I could really use Sam's help. [00:30:21] Jason: How can they get ahold of you?  [00:30:23] Sam: Yeah, absolutely. They can go to, of course the website is closeitnow.net. That's NET so closeitnow.net. They can email me directly, sam@closeitnow.net. On an Instagram at @therealcloseitnow. Okay. Or basically search Close it Now anywhere and I pop up all over the place. [00:30:44] Sam: All right. I'm kind of everywhere on social media and on the Googles at this point. All right.  [00:30:50] Jason: All right, well we're going to close this show now, so appreciate you coming on, Sam. It's been great having you. And for those that are watching, listening, if you could use some help from DoorGrow reach out to us. [00:31:00] Jason: You can check us out at doorgrow.com. We are the world leaders at coaching and scaling property management companies. And so if you are dealing with operational challenges, team challenges, hiring challenges, or you just don't know the right strategies for adding doors or business development, we can help you with all of that. [00:31:18] Jason: So reach out to us, check us out at doorgrow.com and until next time, to our mutual growth. Bye everyone. 

The property management industry is no stranger to conferences and in-person events, but have you ever thought about creating an event yourself? In this episode of the #DoorGrowShow, property management growth experts Jason and Sarah Hull discuss the behind-the-scenes of putting on a live event or conference and all the pros and cons of doing so. You'll Learn [04:39] Learning from Past Mistakes and Failures [15:32] Getting Back in the Saddle: DoorGrow Live [21:07] What Goes Into Creating a Conference? [30:31] The Magic of In-Person Events Quotables “I think being able to just connect with people, making sure that people know who you are and what you do, I mean, it's really valuable.” “When you've got a room full of people who are in the same sector, in the same industry, there's so much knowledge in that room.” “When you're connecting with other people that are like you, that are growth minded and you both share an industry and a share a business model, like it really helps you grow.” “Your business is the sum of the five property management business owners you as a business owner are most connected to.” Resources DoorGrow and Scale Mastermind DoorGrow Academy DoorGrow on YouTube DoorGrowClub DoorGrowLive Transcript [00:00:00] Jason: When you're connecting with other people that are like you, that are growth minded and you both share an industry and share a business model, like it really helps you grow. [00:00:08] Jason: Your business is the sum of the five property management business owners you as a business owner are most connected to. [00:00:13] Jason: Welcome DoorGrow property managers to the Property Management Growth Show. If you are a property management entrepreneur that wants to add doors, make a difference, increase revenue, help others, impact lives, and you are interested in growing in business and life. And you're open to doing things a bit differently, then you are a DoorGrow property manager. DoorGrow property managers love the opportunities, daily variety, unique challenges, and freedom that property management brings. Many in real estate think you're crazy for doing it. [00:00:42] Jason: You think they're crazy for not because you realize that property management is the ultimate, high trust gateway to real estate deals, relationships, and residual income. At DoorGrow, we are on a mission to transform property management business owners and their businesses. We want to transform the industry, eliminate the BS, build awareness, change perception, expand the market, and help the best property management entrepreneurs win. [00:01:06] Jason: We're your hosts, property management growth experts, Jason and Sarah Hull, the owners of DoorGrow. Now, let's get into the show. All right.  [00:01:14] Sarah: Woo!  [00:01:15] Jason: So first, you'll have to excuse if I sound a little nasally today, because I have a cold, which doesn't happen often. And I might have given it to Sarah. I don't know. [00:01:25] Sarah: My sinuses just feel weird.  [00:01:27] Jason: So.  [00:01:27] Sarah: So thanks.  [00:01:28] Jason: Yeah.  [00:01:29] Sarah: Thanks for that.  [00:01:30] Jason: Okay, so.  [00:01:31] Sarah: Appreciate it.  [00:01:32] Jason: You keep kissing me. I'm not kissing you. Like I'm not trying to get you sick.  [00:01:35] Sarah: He's not kissing me.  [00:01:36] Sarah: She can't resist.  [00:01:37] Sarah: Does anybody believe that? Nobody believes you. Nobody should.  [00:01:40] Jason: I'm sick. You keep coming up to me. [00:01:42] Jason: I'm like, you want this? Obviously she does, guys. Obviously.  [00:01:46] Sarah: Oh brother.  [00:01:47] Sarah: Alright.  [00:01:48] Sarah: What a great episode. What a great kicker offered.  [00:01:51] Jason: So I might be coughing and I apologize. Alright, so what we're talking about today is we thought we'd give you a little bit of behind the scenes into us creating an event and us doing DoorGrow Live, getting prepped and prepared for this. You know, we put an entire year into getting this thing going and getting this prepared and promoting it, finding speakers. [00:02:15] Jason: And so let's chat a little bit about some of the behind the scenes stuff.  [00:02:19] Sarah: Yeah. So one of the things that I wanted to talk about is kind of everything that really goes into it behind the scenes that when you go attend an event, you just don't notice. You just don't like realize a lot of the times, unless you're used to running events. [00:02:35] Sarah: And once you start running an event, go run one event and then you will attend every other event differently. For example, when we go to, you know, Aaron's events, or Funnel Hacking Live, my brain is constantly going, like, operationally, this must be a nightmare. How on earth are they coordinating all of this? [00:02:56] Sarah: It's just insane. Because I know how crazy it is with our conferences, and we don't yet have thousands of people there. We will, at one point. But, man, there's just so much that goes into it. So, If you're ever considering running events, and I think that for property managers and for anyone who's a real estate agent or investor, I really think events are something that you should at least look into. And it doesn't have to be this big crazy event where, you know, you spend 25- 30 thousand dollars like we do and that's kind of like a low budget, you know. That's like you'll blow through that real quick. It doesn't have to be anything like that and it definitely doesn't have to be this, you know, this big crazy promoted thing you can do your own version of events like in a very different way, back when I was in property management, you know, we would do some little networking events and they were nowhere near the size, but also nowhere near the cost, but they can be really beneficial for you to do. So I think if you haven't experimented with that, then maybe get some tips and pointers and check it out. Like try it, experiment and see what happens. Because for me, it was really great to just be connected. So there's that saying, "your net worth is in your network," and I think being able to just connect with people, make sure that people know who you are and what you do, I mean, it's really valuable. So if you're a property manager and you haven't done a little in person event yet, then perhaps you might want to try. And we're going to talk a little bit about, you know, what goes into like a bigger event the way that we run them. So why don't you give them some background? [00:04:41] Sarah: When was your first? Your first DoorGrow Live was pre-Sarah, the pre-Sarah DoorGrow age, I think it was it 2018?  [00:04:49] Jason: Yeah. 2018. 2018. Yeah. Yeah.  [00:04:51] Sarah: Okay. Can you talk about you know, what was the first DoorGrow Live like?  [00:04:57] Jason: Oh man. Yeah. And if you want to get a visual of this, you can go to, I think it's photos.doorgrow.Com and we have photos of all of our different major events. You can go back to 2018 and there's a nice photo of me and Mike Michalowicz there. And so we brought in some big, you know, for me, they were big speakers. Some people that I really looked up to and that I got a lot of value from. [00:05:22] Jason: So, coach, authors, you know, people that I had worked with. And so, it was a big deal. We spent, I think we spent about $115,000. Putting that event together because I wanted to do it, right. I didn't want my first event to be Mickey Mouse or cheap or you know, whatever So I wanted to do a really good job and I thought well, "and we'll sell tickets to make up for it." We did. We sold about a hundred and fifteen tickets at around, I think $1,000 a pop. [00:05:53] Jason: And I have a whole podcast episode I did on this. I call it my $2 million mistake because we were growing at a pace of, we were doing about a million in revenue a year and we were growing at a pace of about 300% percent at the time we were growing really quickly. We had a lot of momentum, and I decided to do this big conference. It was a little bit of an ego thing. Like it was like kind of a dream that I wanted to feel cool and be on stage and it was super stressful. The event went really well. People liked it, but I was massively stressed during it. And then I didn't do another one for how many years? I don't know.  [00:06:29] Sarah: Yeah, that was his first and only and then like canceled it  [00:06:33] Jason: I was like, "I don't think I'll do that again." [00:06:35] Sarah: Yeah.  [00:06:36] Jason: I mean I didn't realize everything that goes into it. I'm sure people were watching me start my first conference from the sidelines who have done events in the space were like, "good luck, bro," because they know how hard it can be. [00:06:47] Jason: It's like starting a whole nother business but you have to recognize there's like the hotel. It's hard to do an event that's not at a hotel. So you kind of have to do it at hotels and so they have this like, sort of, they're like the mafia. [00:07:01] Jason: They have this control over doing events. Like, and you go to them, you're like, "I want to do an event here." And they're like, "cool." And like finances become a thing and they negotiate a group rate with you, which means you have to book certain number of rooms because they want you to book rooms, and if you don't book out the group rate for the rooms in the room block, then you're responsible to pay for that. [00:07:24] Jason: So we were on the hook for like a lot of money for rooms. I'm like, "well, how many rooms does that mean? And like how many nights?" And all this stuff. So just managing finances for an event is like managing finances for a dangerous business startup is really what it is. Because people have gone bankrupt from doing big events really big events where you have two, three thousand, five thousand. These are millions and millions of dollars in and out. [00:07:48] Sarah: Yeah. [00:07:49] Jason: And if they don't navigate this well, it can bankrupt companies  [00:07:53] Sarah: Russell just said that on stage. He didn't say who, but Russell Brunson said that he knew someone that was running a big event, didn't sell enough rooms in the room block, and he went bankrupt from it because it was such a large event and he was on the hook for so much money and ended up bankrupting the company. [00:08:13] Jason: It's dangerous. And then you got to get people to buy the ticket, book the hotel, like, and then there's marketing to do this. You got to spend a lot of money to get people to do this. And then, you know, in order to attract people, sometimes people will do like big speakers. Like I got some speakers and let me tell you speakers, they're expensive. [00:08:33] Jason: Like usually they, they want thousands and thousands of dollars. Like an  [00:08:37] Sarah: inexpensive speaker just to like put it out there, like an inexpensive speaker is still usually around like 5k  [00:08:44] Jason: Anyone you've probably heard of is that minimum 25 grand.  [00:08:47] Sarah: Well more than that. [00:08:49] Jason: And if they're a big name It's 50k, 100k, it can be really expensive to have them come be in an event. [00:08:58] Jason: So, Yeah, so it can be really challenging. Then there's food and beverage minimums. So the hotel, they're like, "you also have to spend a certain amount on food and beverage while you're here." Yeah, so they're like, "you have to book a certain number of rooms. You have to, like, pay for a certain number of food and beverage, and you're not allowed to bring any other food or beverage into our place." [00:09:19] Jason: Nope.  [00:09:19] Jason: "You have to use our stuff. And our stuff is like going to the movie theater. It's overly priced, like, inflated."  [00:09:26] Sarah: Remember, we did the Game Changer event at the JW Marriott in Austin so I looked at everything afterwards and it was not a huge event. It was not a big event. We had under 20 people there. [00:09:40] Sarah: Yeah. And that included like Jason, myself, DoorGrow staff, speakers, like under 20 people. And one lunch and we had, it was a two day event. So we did like two lunches. So one lunch, I think was somewhere around like two or 3,000 dollars. Yeah, it was insane for lunch.  [00:09:57] Jason: And my first event, we spent eight grand to provide coffee for two days. Eight grand for...  [00:10:03] Sarah: coffee. Yeah.  [00:10:05] Jason: For two days like and you know, and they have all these rules. I think the rules are made to inflate the price, but they have these food and beverage and they charge you sometimes by plate. So that hotel that we were at our first event, we didn't realize this, but they have people to go around and pick up plates. [00:10:22] Jason: And you're paying by the number of plates people use. Like how much food they consume and by plate. So they were picking up plates.  [00:10:29] Sarah: Oh my god.  [00:10:30] Jason: It's a racket. Like if you go into this not knowing what you're doing, some hotels can take gross amounts of money. Wow. They negotiate a terrible group rate, they negotiate a horrible food and beverage minimum is really high for you, and then you go way over that minimum if they have anything to do with it. [00:10:45] Jason: And so you're spending all this money and they're like, "well..."  [00:10:47] Sarah: you'll never have to worry about hitting your minimum in food and beverage, like, never. No, really.  [00:10:51] Jason: I mean, if you want food there, period, like,  [00:10:54] Sarah: you're going to hit it. So, I don't care. I don't even care what my minimum is because it doesn't, honestly, it doesn't even matter.  [00:11:00] Jason: Yeah. So then people think, oh, well, then I'll do the event somewhere else. Well, if you do it somewhere else, then how are they going to get from where they're staying to the venue? And so then there's a logistical challenge. So then like people aren't like coming and it's just like it's so much easier if they walk. [00:11:17] Jason: So everything gets like complicated when you don't do it at the hotel.  [00:11:22] Sarah: Where was your first event? Where was it?  [00:11:24] Jason: It was in St. Louis at an old classic hotel. It was really beautiful.  [00:11:28] Sarah: Okay. Interesting.  [00:11:30] Jason: Yeah, we did in St. Louis. We did it at This hotel and we did it because we thought we'll make it easy because NARPM had an event around the same time. [00:11:41] Jason: So we're like, Oh man, we want to do it at the same time. So let's just do it at the same venue. I think we did it the same venue, but we booked a nicer room on the top floor with lots of windows. It was very cool. And it was on different days. So you could attend both. We thought that would give us some cross pollination and really, it didn't. [00:12:00] Jason: Like there were a few people that went to the NARPM one and came to ours, but yeah, it was like so small. So that didn't even really help. "We're like, yeah, it's so easy to stay a little longer and go to ours." [00:12:08] Sarah: Interesting. Okay. Yeah.  [00:12:10] Jason: Yeah.  [00:12:11] Sarah: So after the first DoorGrow Live, he decided, I think when I came on board, he said, "I'm never doing another event again." [00:12:18] Jason: Yeah, I just didn't want to deal with it. It was so stressful. And your whole team, that's the real part of it, is like your whole team is involved in it in different ways, unless you have someone specifically handling sales, event, marketing, planning, advertising, planning, like every role we had in our business that we needed for our business had to go towards the conference because we were now on the hook for, I can't remember, like 50, 80 grand or something with the hotel. We had to figure out how to get rooms booked. We had to figure out how to pay for speakers. It was a whole thing. It was like starting a whole nother business. And our main thing was no longer the main thing. [00:12:58] Jason: So our business stopped growing. It actually didn't grow for several years after that, like a couple of years after that. And that's why I call it my 3 million or 2 million mistake, but it was probably a bigger multi million dollar mistake than that, because there's a lot of money I could have made over those years extra. [00:13:14] Jason: We're not hurting by any means, but that really slowed things down. And I just chalk that up to being the price of tuition in business. I made a mistake. I didn't know. And I learned from it, right? And I didn't listen to my mentor. Alex was like, "make it a really small event. Make it really small. Do your first one, make it small." I'm like, "no way. I've been to so many events. I'm going to make this awesome. I want this. If I'm going to compete with all the other events that are out there, I want this to be the best." And I really think, like, we had the best food there. We had the best, like, everything was the best. [00:13:46] Jason: We had audio visual team. We had a stage set up, like, we put a lot of money into this and it was pretty awesome. Like, it went pretty well. But I was massively stressed during the whole event. And yeah, but people that went, they gave us good feedback. They had a good experience. So, which I'm glad. Then you got to like ticket sales is hard too. [00:14:06] Jason: That's a tough challenge. How do you get people to give up what they're doing to come do something else? And so, you know, we've created some really strong magic. I think at DoorGrow, like our in person events, there's just something magical about our events. There's more heart, there's more connection. [00:14:20] Jason: It changes lives and that's very different than what has happened in the space. And I think that's more just about who we are and what we bring and the type of speakers that we bring in. It's very different than just property management.  [00:14:34] Sarah: And so that's one of the things I wanted to talk about is, so you did your first event. [00:14:39] Sarah: It went well, but it was pretty crazy. We basically broke even. We're not doing another event. I came on to the business a couple years after this and there's still a lot of like trauma and PTSD associated with it and then we started talking. Well, what if we do another event? And he said "no. No I don't want to do another event," and I said, "well, what if we do it differently?" So we did bring DoorGrow Live back after that first conference that they did and we've done several of them since then. We have another one coming up in May. It's May 16th and 17th. It's a Friday and Saturday at the Kalahari Resorts in the North Austin, Texas area. So if you're watching this and you have not yet registered, then definitely go do that. You can go to doorgrowlive.Com. But we've done several of these events since then, and one of the reasons that we wanted to bring these events back, especially even though for Jason it was just so, so traumatic, we just needed to do them a little differently. [00:15:43] Sarah: So, the reason that we wanted to bring them back though is because everything is just so much different when it's in person. And we know that there's so much magic that can just happen if, you know, if we can get people in a room. It's not just going to another conference. So in the industry, there's a lot of conferences, I mean, there's tech conferences and like all the big you know softwares have their own thing and there's NARPM events and there's all kinds of things like this and DoorGrow Live is just different. It's different than all of those things. We're not trying to focus on hey, you know, what are they doing and let's duplicate it. We're focused on how can we provide like such a great experience and such great value and real connection in a like large group environment? Which is hard. [00:16:38] Sarah: Like that's a challenge. If you're like, okay, we're going to get, you know, 50 to a hundred people in a room and we want them to all be connected. That's hard. That's hard. But I think that our events do actually a really great job at that.  [00:16:49] Jason: Yeah, I think so. Yeah, we get great testimonials. It's going to we have a really cool venue We just decided to keep doing it at this Kalahari resort. [00:16:59] Jason: It's near our house. It's in Round Rock They treat us really well there. It's a big it's like we have endless room to grow there We could have thousands and thousands of people someday if we wanted to. There's plenty of room there  [00:17:12] Sarah: But they're great to work with and the rooms are nice. When you guys book a room, the rooms are nice, everything is right on property, it's very family friendly too, so, you know, if you want to kind of bring your family and usually, I've noticed sometimes people, when they go to the conference, and then their family stays at home, there's a little bit of like, "oh, you're leaving me with the kids, like, what is this? Like, you get to go off to a conference and," well, come, like, come with us and you guys can hang out at, like the water park and the Build A Bear and the restaurants and the like arcade and there's still...  [00:17:48] Jason: America's largest indoor water park. Yeah. Yeah.  [00:17:52] Sarah: And I think when you book a room, they include a ticket. [00:17:53] Sarah: Yeah.  [00:17:54] Jason: You get a ticket to all a bunch of cool stuff. So like you get a, like a wristband. So yeah it's a pretty fun place. Like there's a whole Facebook group just for people looking for deals and discounts to stay at this resort. Yeah. They're like always talking about it in that group. I've joined all the local groups, just see what's going on. [00:18:15] Jason: So, yeah, so it's pretty interesting. So yeah, we've got a really cool venue. And oh, the other things places have charged us for other places we've done some of our events they charge us for electricity, they charge us for, like, just having cords put down.  [00:18:31] Sarah: They charge for internet. [00:18:32] Jason: They find a way to charge you for everything at some venues, and so, not all venues are equal. [00:18:38] Jason: So, yeah, so we've really appreciated the Kalahari Resort in Round Rock. It's a cool resort, and they treat us really well there, so.  [00:18:45] Sarah: Yeah, and it's a great experience for people. Because that's really frustrating when you go into any kind of hotel and you're like, "Oh. Why is this where I'm at? I guess I'll be here because the conference is here, but outside of the conference being here, I would never book here." And this is not that at all. Like people like to book here for sure. I think now let's do our little demo and then we'll get back into it.  [00:19:08] Jason: Got a little sponsor for today's episode, KRS SmartBooks. [00:19:13] Jason: Do you have properties to manage and zero time for bookkeeping headaches? KRS SmartBooks is your secret weapon. They specialize in finances for busy property managers like you with 15 plus years of real estate know how and skills in Appfolio, Yardi, and more. Imagine monthly reports magically appearing and zero accounting stress. [00:19:35] Jason: Sound good? Head to KRS Books. At K as in Kansas, R as in Roger, S as in Sam. Books. Sarah's already dying. She's like, you didn't do the right military phonetically.  [00:19:46] Sarah: I really am dying inside.  [00:19:47] Jason: KRSbooks. com to book your free discovery call. Integrity, quality, and a dash of bookkeeping brilliance. That's KRS Smart Books. [00:19:58] Jason: Alright, how should I phonetically do KRS?  [00:20:00] Sarah: K like Kilo, R like Romeo, S like Sierra.  [00:20:04] Jason: Alright, Sarah, by the way, is Becoming a pilot. She's taking pilot flying lessons.  [00:20:11] Sarah: I've known the military code for years  [00:20:13] Sarah: because I used to work in a casino and that's how they would communicate in slot machines.  [00:20:20] Jason: Yeah, alright. [00:20:21] Sarah: But now it's also handy being a pilot.  [00:20:24] Jason: Okay.  [00:20:24] Sarah: Alright, so if that sounds good, I think it sounds really great. Because I know a lot of property managers struggle with bookkeeping, and that's usually not something that's fun for property managers. It's definitely necessary, but it, oh man, it's not fun, and it's really draining. [00:20:38] Sarah: So if you can find someone that's great at what they do, and you can allow them to handle that, and just kind of check in and make sure things are going well, then, whoo, man, life gets a lot easier.  [00:20:51] Jason: Yeah if you're not paying attention to the finances or the financial health of your business or your accounting You're probably getting stolen from it's just I've seen it happen so many times. [00:21:01] Jason: So get a great bookkeeper. Yeah have people you trust to take care of that. Okay.  [00:21:07] Sarah: So speaking of finances, let's talk a little bit about what kind of goes into an event. So for example, we have our DoorGrow Live coming up in May this year. So we have been working on this event now since, so our last one was in May, and then I think we started working on the new one in like July, June or July. [00:21:31] Sarah: So things that have to kind of happen just to be able to have the space, obviously, you have to look into venues, you have to, you know, look at the space, make sure it's going to work for the size of your group, which means you kind of have to estimate a little bit what it's going to look like, and then make sure that the room can. [00:21:48] Sarah: fit more or less if needed.  [00:21:51] Jason: You've got to negotiate with the hotel.  [00:21:53] Sarah: Yep. You've got to negotiate what the rates would be. You know, am I paying for the space or am I paying for the room block and the food? Because there's different ways to do it. So you've got to figure out, you know, how many rooms in the room block do I need? [00:22:09] Sarah: Because if you overestimate that, if you go, "Hey, I think I'm going to have a thousand people come" and 100 people come, it is not going to be a good time for you because every room in the room block that is not sold, you are financially on the hook for. So you get to pay for that. And it's like, it's a certain number of nights. [00:22:28] Sarah: So it's not even so much how many rooms it's, how many nights someone will book. So you want to track that along the way. And then you want to start looking at a lot of the tactical things that go into it, like, well, who is going to speak at the event? So you want to start looking at speakers and when you're looking at speakers, you start to think about, you know, who would our audience resonate with and what kind of value would they provide? [00:22:55] Sarah: And, you know, is this strategic and tactical stuff or is this like mindset and empowerment stuff? Because you kind of want to get a mix of both at each event because everyone who comes to an event They're looking for a different thing. So it's really impossible to satisfy everybody make sure everybody, you know is super happy with everything sometimes people say, "oh, I wish there was more of this and oh, I wish there was more of that," but you kind of have to do like this balance and mix to make sure that everybody gets something out of it. [00:23:25] Sarah: And that they have a great experience. You also want to build a little bit of fun into it. So that it's not just, "hey, show up to this conference, sit down, learn something, take some notes and walk out of the room." You know, we've been to events like that before. Where it's like, "okay, that was a lot. But also, man, it would have been really cool to like, do something fun and you know connect with people," so you want to you know start to build in some time so that people can connect with other people, you know, so are you going to do a mixer? [00:23:52] Sarah: Are you going to do some sort of networking event? You know, are you going to you know go do kind of some fun event before like the night before? Are you going to, you know, send them off to lunch together? What is that going to look like? So that they can really connect with each other especially when you've got a room full of people who are in the same sector, in the same industry, there's so much knowledge in that room. [00:24:15] Sarah: So just talking to other people in the room is really valuable and making connections. So there's got to be some room for that as well. And then you want to think about well, are we going to have any vendors or sponsors? Yeah, and are those vendors or sponsors people that have services that are valuable and that we trust? Because there have also been times where, you know, someone had wanted to sponsor us and we did not want them to be a sponsor. [00:24:43] Sarah: Because if they don't provide a great service, you know, can you throw some money and be in the room? Yeah, but if it's not the right person to be in the room, then that matters. That matters a lot. So we have turned down money. We've turned down sponsorships. So then you also have to think about all of the tactical things. [00:25:05] Sarah: Well, you know, am I doing round tables? Am I doing classroom style? Are we doing full circles? Are we doing semi circles? Like what is the front of the room? And what's the back of the room? And where are the vendors going to be? And what doors do people walk in and out of? And as soon as they walk in, what is the first thing that they see? [00:25:20] Sarah: In what direction do we want to go in? And are they crossing over our equipment? Is somebody going to trip and fall on all the 10,000 chords that we have like taped down and. Then you have to also think about things like your AV. So does the room have internet? Is there power in the room? And I know that seems like a silly question to ask, but guess what? [00:25:40] Sarah: Sometimes they charge you for power. So you would think, hey, there's power in the room, obviously, because like it's at a hotel. They obviously have electricity. Yeah, but do you have to pay for it?  [00:25:49] Jason: Yeah, AV is expensive. Like we rented it initially and it was so costly.  [00:25:54] Sarah: Yeah.  [00:25:54] Jason: For the price you could rent it for it made sense to just buy it. [00:25:58] Sarah: To buy it.  [00:25:59] Jason: And so we eventually bought all our own equipment, but that means now we have to set it up and we have to figure it out. And so, yeah, so there's always a challenge.  [00:26:08] Sarah: Before the actual conference, like before anybody even steps foot like on property, Jason and I and several members of our team are there setting things up. [00:26:18] Jason: Sometimes my kids. Yeah,  [00:26:19] Sarah: sometimes the kids, sometimes an assistant, sometimes Madi comes on in.  [00:26:22] Jason: We're hooking up lights, we're plugging in audio equipment.  [00:26:25] Sarah: So we like pack everything up in Jason's SUV. We drive it over, we unload it. I'm doing this in stilettos, mind you, because I'm a stubborn  [00:26:33] Jason: You do everything in stilettos. [00:26:33] Sarah: Yeah, that's what I am. Right, so we like, we get there, we unpack it, we have to set it all up. You know, we're making sure that, like, all the lights are working, a sound system has to work, because there's no point in having a microphone if it's not going to work. There's always technical errors, and I'm horrible with technology, so Jason is our tech person, and he is the only tech person that we have. [00:26:54] Sarah: So he gets to figure everything out. And then it's like, you know, is the screen working? And can people see it? And is the laptop connecting to the screen? And is it blurry or is it too big or too far? Like there's always these weird little issues that happen and I don't know how to solve any of them. [00:27:10] Sarah: Yeah, so Jason knows how to do that. And then there's the other things like well. What about swag? And you know, are we doing a registration table and who's going to be there to, you know, check people in and make sure they know what to do and they know where to go? And, you know, is there like just kind of first come first serve seating? [00:27:27] Sarah: Or is there like a separate section for, you know, special clients or VIP clients or speakers or the team? And there's also things like, "Oh, well what about name badges?" You know, are we doing, like, are we doing name badges? Are we, you know, making sure that everybody kind of knows who everybody else is? Is there anything special or is it just like a bunch of people walking into a room and then hopefully they figure out that they're in the right room? Like there's so much that goes into it and then there's the scheduling. So well, you know, who's going to go in what order, what day and time are certain speakers available? Because just because they commit to an event doesn't mean, "oh, I can speak at any point during the event." [00:28:11] Sarah: So, you know, it's putting the agenda together and how long do you give them for lunch and where are they going for lunch? And are we doing lunch? Are we, you know, letting them facilitate it on their own? Are we doing breaks? How do we get them back from breaks? Are we, it's crazy. Like it's so, there's so much. [00:28:28] Jason: If you give people a break at an event, it's like 30 minutes of downtime. Oh yeah. It's really hard to get people to like get to the next thing or come back right away. And they all start talking to each other, which is cool. They want to network. Yeah, so getting people back from lunch.  [00:28:43] Sarah: Yes, absolutely. Yes. [00:28:45] Sarah: And then it's, you know, who kicks off the event? Who opens it? Who closes it? Who's going after lunch? Because we all know most people, what happens to them after lunch? They're tired. I'm fine. But a lot of people, they're tired after lunch. So you can't have a, you know, more mundane or quiet or low energy speaker after lunch. [00:29:06] Sarah: You just can't. Because you'll lose everybody. So there's a lot that goes into the scheduling as well. And then there's things like, you know, who's going to MC it? Who's making announcements? Who's making sure that everybody knows where to be? And what time? And what to do and when to come back? And who's doing the intros for speakers? [00:29:26] Sarah: Are you doing music for every speaker that comes up? If so, like, are they picking it? Are you picking it? What happens? Like there is so so so much that goes into it, and then after you like run the event then you got to break it all down if it's your equipment. Yeah, so then it's like pack it all up and put it away and make sure nothing gets damaged or lost and repack the car and unload it again, and like there is so much that goes Into it. [00:29:53] Sarah: And I would say at this point, it's funny because Jason now can show up to DoorGrow Live and nine out of 10 times, he has no idea what's going to happen or when.  [00:30:05] Sarah: I love it.  [00:30:06] Sarah: I just call him up on stage and he's like, oh, okay, because, and I'm like, my team handle most of it. Talking on this go.  [00:30:12] Jason: Right now. I still just have to make sure the tech stuff all works. [00:30:15] Jason: But yeah, other than that, yeah, I don't. I don't have to do as much which is nice, but because it's stressful enough. It's stressful enough So yeah, so it's a lot. There's a lot that goes into it, but it's been worth it to have you know to see people's lives change to see people impacted. We've noticed there's some sort of magic that happens that when people come to something in person with us even if they've been a client for years, they start to get different results. [00:30:40] Jason: They start to see things differently. They start to absorb all of our content, our information, our training material, our ideas more effectively. Everything just magnifies. There's something about in person. You can't get the same sort of benefit in your business. If you think, "all I need to do is read books and watch videos and show up to zoom calls to grow my business. [00:31:04] Jason: Look, there's a lot of benefits in all of those things. I do all those things, but we still go to in person things. There's something different about in person that I don't know if it's the energy of being in the same space as the people you're learning from. If it's the group energy and that group mind that makes you able to like learn and faster. [00:31:23] Jason: There's, but there's some, I don't know if maybe there's some quantum physical magic, magical stuff, but there's something different about it in person. It's happened too many times for me to like believe otherwise or to dismiss it. I've had too many clients that I've been working with for years, go to their first in person thing with us, and then they have some breakthrough. And I'm like what? And they tell me about it, and I'm like, "I've been teaching you that for years!" Like "I know but like but it's just hit differently." [00:31:51] Jason: Yeah, "I just got it." [00:31:52] Sarah: It hits different. It feels different and you just absorb things. [00:31:57] Jason: And because we've seen this pattern, we've seen this pattern, we now make it part of our onboarding of every new client to come hang out with Sarah and I in person for a one day with usually a small cohort and like, and just get some things figured out and dialed in their business. [00:32:14] Jason: And that's been magic for our business. Like it's been magic for our clients, magic for us. So we give them that in person experience early on. And then DoorGrow Live allows them to connect with others, which is there's just something different about the people at DoorGrow. The property managers at DoorGrow are different. [00:32:30] Jason: I've been to a lot of conferences. A lot. Like in various industries, but especially in property management. And there's something different about the people that we attract and the clients that we attract. They're growth minded, they're positive active in mentalities, which means they're not like the skeptical, negative Nancy's that are grumpy about the industry and the business. [00:32:51] Jason: That there's this positive growth minded, healthier sort of personality that we attract at DoorGrow. And maybe that says a little bit about who we are, because that's what I tried to be. But we attract amazing people and the connections people make, when you're connecting with other people that are like you, that are growth minded and you both share an industry and a share a business model, like it really helps you grow. [00:33:15] Jason: Your business is the sum of the five property management business owners you as a business owner are most connected to or that you're most influenced by. So look at those property managers if you've got coaches or mentors, and they're not people that you really like that maybe you think they're smart, but you don't really want to be more like them, then maybe you're around the wrong people. [00:33:34] Jason: Maybe you have the wrong coach, and I'm not the coach for everybody. Sarah's not the coach for everybody. But you should have a coach. Otherwise, you're selling yourself short if you're not accountable to anybody, you're definitely getting less results than you could or should be so come to DoorGrow Live come check us out. This DoorGrow Live, [00:33:52] Jason: I want to open our playbooks up if Sarah lets me. I want to just reveal some really amazing stuff that only our clients get to see because I want to show anyone that shows up that's not part of our DoorGrow ecosystem. Our clients know the magic's there. We have more case studies or testimonials than anyone else in the industry, but if you're not a DoorGrow client, and you want to come to DoorGrow Live I'm going to give you some gifts for sure, some magic. We're going to make some significant changes in your business. They're going to help you make a lot more money a lot more easily and keep a lot more of your profit and so come hang out with us. [00:34:29] Jason: You're not going to be disappointed for sure So there you go.  [00:34:33] Sarah: Yeah. This event we've got some really awesome things planned. We can't let too much out of the bag at this point. But we always have some really great things planned and every event we do, like we always learn from it. [00:34:46] Sarah: And we always do like a little team meeting afterwards and we get feedback from people. We're always looking to make it better and better. And this year is absolutely no exception to that. So the things that we have planned for this year, like I know that if you come to this event, it will change your business and it will change your life. [00:35:12] Sarah: And I know that's a really bold statement and we're ready to back it.  [00:35:16] Jason: Yeah. And maybe that could be a later podcast episode as we get closer to the event. But we can tell you a little bit more about what's going to be happening there, but hopefully this was interesting to get behind the scenes at all that goes into DoorGrow Live and we meet on this you know, we're talking about it weekly, monthly in our planning meetings, like and quarterly. [00:35:37] Jason: And so, and that's it for today's episode. So if you are interested in that, go check it out at DoorGrowLive.Com and get your tickets and get things booked and get ready to come have an amazing experience in May at DoorGrow Live. So, and until next time to our mutual growth, bye everyone. 

Even with all of the property management software and tools breaking onto the scene lately, it seems that some entrepreneurs are still identifying gaps they could potentially fill… In today's episode of the #DoorGrowShow, property management growth expert Jason Hull sits down with Eric Nelsen of Walkthroo to talk about a new maintenance solution in development for property managers and vendors. You'll Learn [03:36] What is Walkthroo? [08:43] Developing Software and Utilizing AI [16:52] Getting Time Back with User-Friendly Tools [23:02] Get in Touch with Walkthroo Tweetables  ” It's a lot easier to make changes to software when you're smaller and you're getting things started and you're doing it in the right way.” “ Time is probably the biggest benefit we provide.” “ Vendors in a lot of situations end up being the eyes, ears and hands for the property manager.” “ User experience is a big deal when designing software.” Resources DoorGrow and Scale Mastermind DoorGrow Academy DoorGrow on YouTube DoorGrowClub DoorGrowLive TalkRoute Referral Link Transcript [00:00:00] Jason: It's a lot easier to make changes to software when you're smaller and you're getting things started and you're doing it in the right way. Once it turns into a giant beast and it's old, then it's really difficult.  [00:00:11] Welcome DoorGrow property managers to the DoorGrow Show. If you are a property management entrepreneur that wants to add doors, make a difference, increase revenue, help others, impact lives, and you are interested in growing in business and life, and you're open to doing things a bit differently, then you are a DoorGrow property manager. DoorGrow property managers love the opportunities, daily variety, unique challenges, and freedom that property management brings. Many in real estate think you're crazy for doing it. You think they're crazy for not, because you realize that property management is the ultimate high trust gateway to real estate deals, relationships, and residual income. [00:00:52] At DoorGrow, we are on a mission to transform property management business owners and their businesses. We want to transform the industry, eliminate the BS, build awareness, change perception, expand the market, and help the best property management entrepreneurs win. I'm your host, property management growth expert, Jason Hull, the founder and CEO of DoorGrow. [00:01:12] Now let's get into the show. And today I'm hanging out with Eric Nelson of Walkthroo. Eric, welcome to the DoorGrow Show.  [00:01:21] Eric: Thanks, Jason. Glad to be here.  [00:01:23] Jason: So Eric I would love to first get into your background. And my wife's chiming in saying I need to remember to promote DoorGrow live today, so I'll just do that right now real quick, and then we'll get to you, Eric. So if you are a property manager and you're watching this make sure you get tickets to DoorGrow Live like this is the most contribution focused, holistic property management conference in the industry. [00:01:44] We do things very differently. "There's heart" is kind of the feedback we get from others. People cry at our events. Like it's really awesome. It's going to be at the Kalahari resort here in Round Rock, Texas. And get your tickets right now. They go up in price over time. So head on over to DoorGrowLive.Com and get your tickets and be there. We've got sponsors. We've got cool speakers. It's going to be awesome. And DoorGrow magic is there. You're going to learn about growing your business from Sarah and myself and we'll help you out. All right, cool. Shameless plug inserted. [00:02:20] Now, Eric, I would love to get into your background. [00:02:23] You know, we hung out briefly in in Austin you came out and got to know each other a little bit, but I want my audience to get to know you share a little bit about How you kind of got into entrepreneurism, how you got into this. So tell us a little bit about your background.  [00:02:37] Eric: Yeah, sure. Sure. I grew up in Houston, Texas kind of came up through the finance world. So I spent about 10, 15 years in finance, went to grad school at Rice in Houston, and I just couldn't walk down the finance hallway. I saw the entrepreneurial professors down a different hallway, really wanted to kind of do my own thing. [00:02:55] So you know, stayed in finance for a couple more years and got into the pharmacy business. And through that business, I got exposed to IT technology and building software to kind of run our pharmacies and improve our ops and, and run those companies. And then a good friend of mine in Shreveport Springs, Texas was is a general contractor and said he works with these property managers and they, he does a lot of maintenance for rentals. [00:03:20] And he said, "yeah, Eric, I want to take on more business, but I can't keep track. There's so many little jobs. There's so much communication going on, text, emails, phone calls. You've got a software background. Can you help me?" And so that's, what's really exposed me to the property management industry and kind of started me on this path. [00:03:36] Got it. All right. So let's get into talking a little bit about Walkthroo and what it is. And it's, it's "walk T-H-R-O-O. So tell us a little bit about Walkthroo and what is it? What does it do?  [00:03:52] Yeah. So Walkthroo is, it's a really kind of a mindset and approach to the business and the underlying core is as much as accounting and tenant screening and even inspections, that software, those tools have grown, you know, with technological advances and whatnot. [00:04:13] If you really look at what we think is one of the four main pillars of property management is the maintenance, that hasn't grown. I mean, if you look back 10 years ago you really couldn't get multiple bids to do any work. If you look back 10 years ago, you couldn't pull up on your screen and compare two different bids. [00:04:29] 10 years ago, you couldn't split charges on an invoice between a tenant and owner. And you look today, fast forward 10 years, and I would say You know, 90- 95 percent of the platforms, you still cannot do those things. Well, when my partner brought me into this, you know, first he wanted me to help him with his, you know, just his construction company, but we quickly realized the problem wasn't him. [00:04:52] It was the property managers he was working with and the inefficiencies that came with the way they handle maintenance. So right out of the gate within a month. We switched that mantra. We're going to work to help property managers. And so that's really been what Walkthroo's focus has been the last three years. [00:05:09] And we really just, again, within the first three months we can get multiple bidders, we can split charges. And so it just showed me right away that it's not for a lack of technology or, you know, lack of know how even. It's just when you look at these software platforms and these operating systems, they just have bigger fish to fry. [00:05:27] They, you know, they all agree we should be able to hire multiple bidders with a couple clicks, but we're going to spend time doing X. So I can't explain it, but again, within the first six months, we had all these features built. And so now we're coming up on three years. We're really looking to round out the platform and keep growing. [00:05:45] Jason: Okay. So besides doing multiple bids and splitting charges, what would you say Walkthroo is? Like, what is, what does it accomplish?  [00:05:53] Eric: So we're going to be a full operating system for property managers. We started backwards. I spoke with the former CEO of Buildium post sale to real page. [00:06:03] And he told me flat out, "we did a lot of great things." I think they were in 19 countries at the time. He's like, "but I'll be honest here. We never figured out maintenance. And so if that's where you're starting, you know, good on you. Good luck." And so we started with maintenance and we built our platform around maintenance. [00:06:18] We've recently added inspections. And so we'll keep growing. So Walkthroo will be A full suite of operating suite for property managers. Currently, we're not there yet, I'm going to go through a couple of rounds of raising money. Currently, we're a maintenance tool. People can use our platform. And we also provide maintenance services still. [00:06:39] So that's, that's, that's kind of what we do today. And the third leg, which just launched, is, and this is probably the most unique feature of what we're building, every other maintenance tool or platform or operating platform out there has property manager and they invite people in and the people have to learn how to use your system and whatnot. We actually sell our software straight to contractors. [00:07:02] So they're using it independent of property management They're using it to paint houses, do handyman jobs around around their cities, and so we're building this network where property managers will be on Walkthroo, the contractors are on Walkthroo, and it's just a simple connection and you don't have, you know, the training and, you know, as a vendor ourselves the last few years, I've been through some trainings to use different systems and I can imagine. It's can like a painter, you know, in downtown Austin that has two employees trying to figure out all these platforms and how to work with these clients. So we're, our goal is to really simplify all that for all the stakeholders.  [00:07:39] Jason: Got it. So it sounds like Walkthroo, you're building this from the ground up. [00:07:43] You're building it as a tool to support and help based on what business owners actually need in property management. You started with one of the biggest challenges, which is maintenance. You're now adding inspections, you're adding other things. And the goal, the roadmap is to make it a full suite that helps maybe a better property management back office or software solution. [00:08:05] So the next big piece is then I'm sure on the roadmap somewhere is accounting and, tenant portals, owner portals, so they can see statements and submit the maintenance request, maybe like all of this kind of stuff. And so yeah, and I don't, I think that there's, there hasn't been a lot of innovation. [00:08:23] We've seen Rentvine come out recently. And it was born kind of out of a lot of complaints people were having about Appfolio. Appfolio was kind of born out of a lot of complaints people were having about maybe Buildium and Propertyware. Right. Right. And so, you know, when software is born out of complaints, you know, of different tools, yeah, it's going to be better than that tool, but it is interesting to start from the ground up building around the needs of and supporting the property manager and the work that they're doing. It'll be very interesting to see where you guys end up and what's kind of the timeline for all of this? [00:08:55] Eric: Well, you know, it depends on fundraising, right? So it's expensive, especially, you get into the accounting engines and a lot of that. There's a lot of costs involved. So we're hoping in the next You know, 12 to 18 months, we'd have a product out of, you know, for small property managers to run their business off our platform. [00:09:12] Jason: That's pretty fast. That's really the goal right now. Yeah. Okay. Got it. Yeah. And it sounds like you guys move quickly. You know. It's a lot easier to make changes to software when you're smaller and you're getting things started and you're doing it in the right way. Once it turns into a giant beast and it's old, then it's really difficult. [00:09:30] Like some of the older maintenance software companies I'm sure they're toying with the idea. Like, should we just rebuild from scratch or throw all this away? Or do we just work this until this horse dies, you know? And so that's always the challenge with software. [00:09:46] And then adoption is always a big challenge. So getting people to use something new or to change to something else. And a lot of times it's easier to get the smaller guys and the smaller companies to make changes. And the big companies are usually watching the little guys make all the mistakes or test stuff out or see. [00:10:04] And then they stand back to wait to see who the winners are. So...  [00:10:08] Eric: yeah, yeah. And thankfully I've got some experience on our side. My partner, Travis, he before he got into construction, him and his dad ran a small microscope specialized software company they sell it to universities. I don't know the ins and outs of it, but they could like take a laser and look into this, you know, the elemental makeup of a molecule. [00:10:26] It was really, really specialized, but that was exactly where he came from. He's like, yeah, you could go with Hitachi or a big Japanese brand, but you can't get them on the phone. You know, like you said, they've, they've done good. They've built so big, but now that's a hindrance. And we're in the same path. [00:10:40] You know, we didn't have splitting the owner and tenant charges, but you know, after talking to a few clients and a few property managers, that was just a common, very common thing. And I said, "well, let's just build it." Well, we're small or nimble, you know, we can, we can get away with that. [00:10:53] So we're going to take that same approach as we go through the accounting side of things, you know, and just interviewing property managers and listening to the industry and saying, Hey, my background is finance and operations. And so, you know, when I met you, something you brought up a lot was transforming lives and, you know, kind of making people enjoy their work and that's something I don't see. When we launched this tool. We decided to launch it internally two years ago. So we haven't really been selling Walkthroo, we've been using it ourselves. We currently manage Over 250 jobs in nine states. And so I talked to more maintenance coordinators and property managers every day and a lot of them could be happier. [00:11:35] So as we build this out, we want these tools to allow some sort of automation and allow people to focus on growing doors and, you know, and doing other things that are more beneficial versus banging their head against walls.  [00:11:49] Jason: Sure. Yeah. I know property management business owners would much rather spend their time focusing on scaling their business than dealing with all the the nitty gritty day to day challenges and difficulty in all the tools that they're dealing with. [00:12:04] So Eric, we're in the middle of this AI revolution and you're like right in the middle of building this tool as we're coming into this new AI revolution where there's just tons of software just coming out. And people can create tools and software a lot more easily and their AI is helping them. [00:12:22] And then everyone's trying to integrate AI. And then you see all these companies that are dinosaurs. They're trying to strap chat GPT on the side of their crazy rollercoaster. And like, you know, say now we have AI. And so how's AI kind of tie into you guys, you know, getting Walkthroo built out? [00:12:43] Eric: Yeah, great question. We've got a roadmap for it. We don't have anything integrated yet. I think it's, it's too early, but you know, my background is really improving operations efficiencies. And so once we have this tool built out, then we will again, deploy AI where it makes sense. Like you said, it's a buzzword. [00:13:03] People will say everything is aI generated. It's like, no, that's just a search function, but call it AI. And so we, you know, we know most of the data. I'm not well tuned on the accounting yet, but definitely on the maintenance side, we know what data and what decisions are being made every day because again, we've lived that life and we're living it now we're doing jobs. [00:13:24] And so we will bring in AI kind of as we roll out the full suite, you know, I'm not sure to be perfectly honest. I don't know if it's going to be a heavy lift. I mean, again, it really comes down to the operations of the business and work and we see efficiencies and you know, there's some decisions you want eyes on, you know, you want, you want human interaction and others are a little more mundane task. [00:13:45] And so we, we are definitely have that in the playbook but I, at this point, you know, our plan is not to have this fully automated AI, you know, software, it's going to be just a much cleaner, easier tool to use and AI will be obviously just a natural component of that. [00:14:01] Jason: Got it. I mean, I think that makes sense. A lot of people start, you know, thinking, Oh, let's make AI do everything. But I think, I think it probably does make a lot more sense to make sure that the tools and systems are working for humans and they're working the right way first. And then AI create some leverage now that this is working well. [00:14:21] And I think that goes for how business owners should implement technology in general is you first do the process manually, and then you start to look for points of leverage and where can I leverage tech, where could a tool like Walkthroo facilitate what I'm doing now or help move things forward? So who's your current target audience? [00:14:39] Like, who are the people listening to this podcast that you think should reach out to Walkthroo to get an assessment on their current maintenance situation?  [00:14:49] Eric: Yeah. I mean, we've talked to everyone from PMI to sole proprietors to self managers. So I would say our sweet spot is probably property managers with, you know, 200 to 500 doors. [00:15:02] Seems to be small enough where the data is not overwhelming. They're doing a lot of work, I feel from what I've seen personally, and so working with Walkthroo helps some of that. And people can work with us in different ways. We some people just use our software. You know, we, If we can, if we can manage jobs across nine states, truly, you know, we know people can manage jobs in their own town or their own state and some of them just hire us as a, they just have us on their preferred vendor list, you know, we obviously I don't have staff in nine states, so I use my tool to manage jobs and manage vendors and the third way people can access and partner with us Is we come on as your maintenance coordinator, you know, we'll use their vendors, their top vendors, let us manage it. [00:15:43] One question I always ask property managers, not surprisingly, the answer is usually similar is, you know, "have you ever logged in as a vendor to whatever system are you using?" [00:15:51] " Well, why would I do that?" It's like, well, yeah, you probably wouldn't think of it, but I recommend it because you know, it's, it's one of those tasks. It's important, but it's also been done since the dawn of property management, I give someone a job, they go do it. But if you, if you're using tools, I recommend logging in as that contractor and seeing what they're seeing. And, oh, this is why it's hard to communicate because I can't upload anything or I can't text or, you know, whatever, whatever it may be. [00:16:20] So the maintenance coordinator role is something we've been taking on more and more where it's like, yeah, you give us your favorite painters and handyman, and we'll either API into your system, or you just send your tenants our way. You know, we structured any way that works best for our clients and the, let us do the dispatching, you know, all the status checks. [00:16:39] I mean, you know, it's just a constant barrage of phone calls every Monday morning on where we're at. And of course, Sunday night we send out reports so we don't have to get those calls. Those are the three ways that property managers can work with us currently.  [00:16:52] Jason: What, what are the results that people that start working will Walkthroo tend to notice or what sort of the changes that you're creating for these business owners.  [00:17:02] Eric: It's time. Time is probably the biggest benefit we provide. You know most I just mentioned the Monday check ins or daily check ins most maintenance tools that I've seen in, by the way, the other way that we know our, our tool is is well built, it's acting and being a vendor for the last three years. [00:17:21] I've logged into all the other tools. You know, when a property manager sees Walkthroo, yeah, they say Oh, Eric, yeah, we're always looking for a new painter. Here's our login to our system. Great. So immediately we take notes and, and figure out what's, what's wrong, but the time component I would say is probably the, the most we hear back on, on the biggest benefit and then most systems will have status indicators, maybe something's in progress. [00:17:44] We've got over 20 statuses. Are we waiting on the contractor to finish the work? Are we waiting on the tenant to accept the schedule and confirm it? Are we waiting on the after pictures to come in. I mean, there's all these nuanced steps that I think historically again, bigger companies are busy, but coming from at it from fresh from outside the industry, it was like, well, this is important to know if I know that I'm waiting on the tenant to confirm a schedule, I don't need to waste my time calling the contractor, ask what's going on. [00:18:14] And so those, that's a little microcosm of. How we built our system and also just a, again, just the workflow. I mean, I was shocked. None of the systems I've used since I've been in property management, offer me a way to do a change order. Very simple, very common request. And I have to like make a phone call or send an email. [00:18:32] And it's just time, time, time. So we make all that click, click, click.  [00:18:37] Jason: For the listeners. Explain a typical change order sort of situation.  [00:18:41] Eric: Leaky faucet. We've got a leaky faucet. We want somebody to go check it out. Contractor shows up on site, looks at a leaky faucet, and says, yeah, this faucet's leaking here. [00:18:51] I can fix that. But also, it created mold and damage all behind it. All under the counter. We've got to rip all these counters out. Well, that's not what the contractor was there sent to do. It's definitely not approved without, you know, anyone signing off on that. So he's got to communicate back to the property manager, "Hey, there's a much bigger issue here." [00:19:11] And so in the industry, it's, you know, typically referred to as a change order. And so now the contractor usually sits and waits and says, okay, I'll, I'll wait for the property manager to talk to the owner. And see if they want me to rip off this cabinet and do all this extra work. You know, I'm just, you know, I'm just a contractor. [00:19:28] Can I explain what I see? So now we're in a waiting game, right? So a week later, property manager boss comes in and says, "what's going on on one, two, three Smith street?"  [00:19:36] Jason: Yeah.  [00:19:37] Eric: "Oh, well, there was a problem."  [00:19:38] "Okay. What's going on now?" [00:19:40] "I don't know. Oh, it looks like, I think we're waiting for the owner to give us the green light to do the new repairs" [00:19:46] and so you can, you can step back and realize how that can. And you add that times 50 jobs or 100 jobs and it starts, it really adds up. So again, the way we built our system was to really eliminate a lot of that excess time. And where are we in this maintenance process? And just put it on the dashboard. [00:20:03] Just like, you know, many other things in life now. Put it in front of my face, so I know where all my jobs are and all my maintenance tasks are located.  [00:20:11] Jason: Hmm. Yeah. Yeah. Very cool. Yeah, that makes a lot of sense. I'm sure that's a challenge, like people discovering new work when they go out to do work. And there's also the issue a vendor goes out to do work and then they notice other stuff they think the property manager should be aware of. [00:20:25] And yeah, I mean, vendors in a lot of situations end up being the eyes, ears and hands for the property manager, so.  [00:20:32] Eric: Yeah, actually that's, that's why we built our own inspection tool. You know, we see everything else that's out there, but a lot of it's not connected. It's, you know, it's separate tools. So I've got a system that does this and does that. [00:20:45] So we tell our contractors, it's in our app, which I think there might be two or three other maintenance platforms, but not many that actually have an app in the app store for the vendor. So again, I challenged property managers to log into whatever system they're using as a vendor. And you'll probably see it's not the easiest thing to use or communicate with. [00:21:05] Well, we turned that upside down and. We've got an app live in the app store. Contractors can download it. So when they're doing work for us, it's super easy. They're on their phone. So we added an inspection tool and said we're going to require you to do, if it's vacant, to do a full inspection. And we just provide that as a free service, like, hey, in case, in case you or the owner missed something, we happen to notice these other 10 items that you didn't want us to fix, but here's some pictures and a report, and so again, like, just to your point, we know we're the eyes and ears a lot of time, you know, at the property, so anything we can do to capture all that data and get it back to the property manager. [00:21:43] We think so it's a win for everyone.  [00:21:45] Jason: Yeah, I love that So, I mean historically that's been a big complaint about some of the property management maintenance coordination tools out there is that the getting vendors to use it the adoption of vendors has been like real difficult and maybe it's Just your from your experience. [00:22:02] Maybe they're just not very good for the vendors through for their experience. It's just not a great experience. So user experience is a big deal when designing software. And it sounds like you guys have kind of designed this from the ground up to make sure that the vendors are going to have a good experience using it. [00:22:17] Eric: Absolutely. You know, again, we, you know, we're, we're signed on as preferred vendor across, across nine states. And so it's, you know, it's our insurance, our butts on the line if the jobs aren't getting done. So we figured out very quickly, we cannot make this difficult for this contractor in Florida that doesn't know Eric from Dripping Springs, Texas. [00:22:36] So let's make the tool super easy. And that's exactly what we did. And so we've had... oh, I would say over three years, I think maybe three or four times we've had to coach somebody through how to use our maintenance tool.  [00:22:48] Jason: Really? Sometimes vendors are old school. [00:22:49] They're not the most tech savvy. They're, they're using physical tools, you know, but yeah. And so that says a lot that it's pretty intuitive or easy for them to figure out.  [00:22:59] Eric: Yeah, that was a big focus for us right out of the gate.  [00:23:02] Jason: Got it. Okay, cool. Well, for those that are, like, hearing about this, or a little bit interested in this, is there anything else they usually have questions about that we didn't touch on, or that they should know about Walkthroo? [00:23:14] Eric: Let's see, not really. I mean, I think we covered most of it. Again, our goal is to really provide more time. I just, we see so much wasted time, you know, in the maintenance process. Obviously, we're going to carry that on through the rest of the modules and operating software, but our goal is to eliminate that time and give it back to property managers and really allow them to, like you said, I know they'd much rather growing doors and making connections and using their time more wisely. [00:23:39] So, yeah. If we can save them hours a week that's really, really our goal.  [00:23:45] Jason: Got it. Okay. Well, it sounds like you guys focus on simplicity. You focus on making these work. How can people get in touch with Walkthroo?  [00:23:55] Eric: Yeah, you can go to our website. It's www.walkthroo.com . You can also send an email over directly to me or my team. My email is eric@thewalkthroo.com and if you want to just send it to our team, it's work orders@thewalkthroo.com.  [00:24:21] Jason: Got it. So it's 'the Walkthroo' and through is T-H-R-O-O. Okay. All right. Everyone listening, go check that out. [00:24:30] Eric, appreciate you being here on the DoorGrow show and hanging out with us. And I'm looking forward. We'll have to have you come back on once you guys have added some new features and it sounds like you guys are pretty aggressive at doing that.  [00:24:44] Eric: Absolutely. Thanks, Jason. Appreciate the time. Good seeing you.  [00:24:46] Jason: Good seeing you too. [00:24:47] All right. For those of you that are looking to grow your property management business or you're struggling, check us out at doorgrow. com. We would love to help you. We are getting amazing results with our clients. And so if you want to get from 0 to 100 doors, from maybe 100 to 200 doors, or you wanted to go from 200 to 500 doors, Or from 500 doors to a thousand doors, we can help you at each of these stages and each of these sticking points to grow and scale your business rapidly and to get the right stress free ops and systems in place so that you are able to do this without making your life worse personally. [00:25:21] And so check us out at doorgrow. com. And until next time everybody to our mutual growth, bye everybody. [00:25:28] you just listened to the #DoorGrowShow. We are building a community of the savviest property management entrepreneurs on the planet in the DoorGrowClub. Join your fellow DoorGrow Hackers at doorgrowclub.com. Listen, everyone is doing the same stuff. SEO, PPC, pay-per-lead content, social direct mail, and they still struggle to grow!  [00:25:54] At DoorGrow, we solve your biggest challenge: getting deals and growing your business. Find out more at doorgrow.com. Find any show notes or links from today's episode on our blog doorgrow.com, and to get notified of future events and news subscribe to our newsletter at doorgrow.com/subscribe. Until next time, take what you learn and start DoorGrow Hacking your business and your life.

At DoorGrow, we love showing off the awesome entrepreneurial people we get to coach and work with every day. In today's episode, property management growth experts Jason and Sarah Hull sit down with DoorGrow clients Jill Lyons and Alex Platt to talk about their journey in property management and with DoorGrow. You'll Learn [03:00] Starting a journey with coaching [07:26] Finding support as an entrepreneur [12:18] The path to success is hard work [16:54] Getting out of the business [19:28] The importance of good company culture [21:20] The impact of coaching Tweetables “Done is better than perfect.” “The more valuable you are to your business, the less valuable your business is.” “If you don't mind working, you don't set up boundaries.” “Just being open to the thought and the idea is enough to make it work.” Resources DoorGrow and Scale Mastermind DoorGrow Academy DoorGrow on YouTube DoorGrowClub DoorGrowLive TalkRoute Referral Link Transcript [00:00:00] Jason: The more valuable you are to your business, the less valuable your business is. Ooh, like that one.  [00:00:07] Welcome DoorGrow property managers to the #DoorGrowShow. If you are a property management entrepreneur that wants to add doors, make a difference, increase revenue, help others, impact lives, and you're interested in growing in business and life, and you're open to doing things a bit differently, then you are a DoorGrow property manager. DoorGrow property managers love the opportunities, daily variety, unique challenges, and freedom that property management brings. Many in real estate think you're crazy for doing it. You think they're crazy for not because you realize that property management is the ultimate, high trust gateway to real estate deals, relationships, and residual income. [00:00:47] At DoorGrow, we are on a mission to transform property management business owners and their businesses. We want to transform the industry, eliminate the BS, build awareness, change perception, expand the market, and help the best property management entrepreneurs win I'm your host, property management, growth expert Jason Hull, the founder and CEO of DoorGrow along with Sarah Hull, co owner and COO of DoorGrow. Now let's get into the show. [00:01:13] Our guests today... we've got Jill and Alex. Jill Lyons. Alex, what's your last name? Platt. Okay. I just know he's always with Jill, Alex. So we're really glad to have you on the show. And the topic of today's episode is like, we want to talk about your journey with DoorGrow because you've been with us for a little bit. So, why don't you introduce yourself and explain like kind of how you got into property management.  [00:01:39] Jill: Well, I must've taken an insane pill along the way, but I like it. My name is Jill Lyons and I own and I'm broker of Relaxed Realty Group in Sarasota, Florida. Currently we manage about 500 homes. We have like maybe 520 now and our rent roll, we just surpassed 800,000 this month, so I'm stoked and happy and proud. And you know, I love the business. There's never a day that's not that I feel like, "Oh my gosh, it's, you know, Monday." I never feel like that. So it's every day is a joy. Not every instant is a joy, but every day is a joy.  [00:02:12] Jason: So let's Alex, why don't you introduce yourself and tell us what is your role?  [00:02:17] Alex: So, my name is Alex and I've worked with Jill here just over a year and a half, or going on almost two years when I got my real estate license. My wife started with Jill, Miranda, and she's been with Jill for what, 10 years now? Started with a business with her and I do the operations here. So operations and BDM.  [00:02:38] Jason: Awesome. Okay, cool.  [00:02:41] Jill: So he came from a customer service background with T Mobile for the last 10 years. It's great. Corporate's a great, but there's a lot more opportunity here and oh my God, he's great with people. Of course He's not " to brag about himself. So I'll brag about him. So he will put on multiple hats and do everything that whatever needs to be done.  [00:03:00] Jason: Cool. Yeah, you guys make a good team. We've enjoyed having you in the program. So why don't we start with what problem problems were you dealing with when you first came to DoorGrow? Like what challenges were going on?  [00:03:14] Jill: So I would say my strengths are that I love to sell and talk to people and help people. So, you know, that was naturally there and I grew the business with success with growing doors. And I was in a kind of a comfortable, I would say position as. Having a good amount of owners and properties, but I want to start exiting the business and it was just way too 'me centered,' you know, what do we do? What do we do with people coming to me? You know, I don't mind working. Like I say, so unfortunately, if you don't mind working, you don't set up boundaries, you don't set up corporate structures. My flow, there was nothing corporate about me. [00:03:49] If I wanted to step away, which I did this year, hired the operations manager, but I'm like, now what? And now what do you do? I'm an engineer by education. All I know how to do is build a spreadsheet and show people returns. So I was looking for ...I always believed in coaches. I've been coached since day one of my business. [00:04:07] So coaching is definitely something I believe in, but the coaching company I used was really just real estate working with buyers and sellers. So I hadn't ever got the property management business aspect of it and setting up the business and the structure. So when you watched one of your podcasts and listened to your podcast, and I liked what you had to say, so I-- "let's let them get us to that next level." [00:04:32] Jason: Watch the podcast, listen to the podcast, and now you're on the podcast.  [00:04:36] Jill: I know, I'm like, what do I have to offer? That's the first thing, I'm still listening and learning.  [00:04:42] Jason: You know, there's a lot of people listening out there that would dream of having 520 doors, having an amazing operator, having the operations running smoothly and being on your journey, stepping out of the business, like this, that's a dream for a lot of property managers. [00:04:58] They're still in the thick of the mud and wondering if there's a light at the end of the tunnel.  [00:05:03] Jill: So they don't believe that I'm going to step out.  [00:05:05] Alex: She's a workaholic. So, you know, it's a little bit of yin and yang.  [00:05:09] Jason: You know, entrepreneurs, it's a tough thing. I've known a few entrepreneurs that have like exited their business and then they were bored and they started another business. It happens. So entrepreneurs, we want to stay busy and we want to do the things we really enjoy doing. So you just have to find something you maybe enjoy doing more.  [00:05:29] Jill: I don't know. Yeah, no, I'm not closed to what's next, but I don't know. I'm still here.  [00:05:35] Jason: So let's chat about, and maybe this is a question for Alex. So Alex what did you see when you first came into the business? Some of the challenges in how to like support Jill and how to get her out of the operational stuff. And what challenges did you see that DoorGrow so far been able to help with?  [00:05:54] Alex: So luckily with your program we got to revamp everything. I mean, your Rapid Revamp was amazing. I mean, we got to go from rebuilding and rebranding our logo and everything. So I really enjoyed your class, especially with the whole cycle of suck, making sure that you're not holding onto those owners that are sucking up all your time and, you know, using. A lot of your resource when it comes down to it. I would say those were the biggest things and especially your systems that you have. I mean, I think the Flow is going to help a lot for us to map out each and every one of our procedures that we have on an operational standpoint.  [00:06:33] Jason: Okay. So for those listening, DoorGrow Flow, our process software, which is pretty cool. So the Rapid Revamp, I mean, and you guys made a lot of changes. Yes. Changed your pricing.  [00:06:43] Alex: We changed our name.  [00:06:44] Jill: You changed the name. I said I would never, ever do that!  [00:06:49] Sarah: She's like " I'm not rebranding." I'm like, "okay, we don't have to rebrand." And then she's like, "I think I'm going to rebrand." I was like, "wow! All right, let's do it."  [00:06:58] Jason: Everybody says they don't want to do it. But what I love about entrepreneurs is that if you show them how to make more money, they're pretty okay with it. They're pretty okay with making more money. So, and I think the training, we do a good job in converting people into wanting to make more money. "Here's how it'll make you more money if you do the right things with your branding." So website. Did we help with that?  [00:07:23] Alex: We're almost there. We're on the tail end of that portion of it.  [00:07:26] Jason: So for those that have not been exposed to DoorGrow. Maybe they're just listening to this podcast. They're like, "I don't know if these guys are legit. Kind of looks like some sort of one of these Influencer sort of guys," or I don't know what people think before they become a client but what would you say to those that are on the other side of the paywall and maybe struggling?  [00:07:51] Jill: For me, honestly, if I would have found this 10 years ago, it would have happened faster, my growth and where I am now would have happened faster and more organized. I kind of wing it and I'm the type that, you know, I don't want to spend any money unless a bunch of sitting in the bank. And I probably, if I would have opened up the bank and gotten the coaching and the programs from a property management company versus just from, you know, where I got my assistance from, which I had when I did buying and selling, which I hate it. So I kind of kept my things rather than going into property management coaching and training. It would have definitely made it faster and less painful, and I would say that's the biggest thing that I wish I would have found you sooner, but you know, you always find people when you're supposed to find them and entrepreneurs tend not to be, in my opinion, people that go to business school because they just want to do it. They jump in head first. There's no rhyme or reason to how we do it. So the organization is usually where we struggle the most. And just networking and having the beginning, I just went to Google and figured everything out on my own, rather than reaching out to an organization like yours, that's more specific for us and NARPM, which, you know connected me to other property managers and how are they doing it? And why did I have to create the wheel and do it all my way? I didn't even know that there was anything like this.  [00:09:16] Jason: Yeah. And you had been in NARPM for a while before joining DoorGrow.  [00:09:20] Jill: Yeah. I'm heavily involved in NARPM. I'm the president of our local chapter. So that definitely has made helped my business, and the connection and they have a lot of tools that have helped me significantly realize that it is a business and with systems. But but there isn't the sales support, you know, they don't have you, Jason. It's not energetic and make me go, "yes! I'm going to do it!" With you and with everybody around! You know, it's just like the connections.  [00:09:48] Jason: Yeah. I know you have both really enjoyed the operational pieces as well, and you've attended quite a few of our scale calls on Friday that Sarah runs. What what things have you taken away from on the operational side of things? [00:10:04] Jill: So what would you say, because you deal with that more? I kind of say, go do it.  [00:10:07] Alex: So, I take a lot of the way, honestly, you guys definitely on those calls go over a lot of different systems that are in other people's companies, to be honest. And we try to take piece by piece and just kind of make it our own when it comes to this. I think it's developing more of the systems that we have. As far as like a specific system, I think we talked about maintenance heavily. And the processes over how other companies do it and what we do with our maintenance. So it's kind of getting every pieces of everybody's input on that stuff to kind of lay out what maybe we should change, you know? [00:10:45] Jill: I will say that as far as operational, we were in pretty good shape with that. It's not technicalogical. So you have DoorGrow flow. I'm just talking with Errol tomorrow. So it's been on my list of things to do this whole year to set up flow and get that going so that it's more clear how we do things because when we have a new employee, I can't just hand them, "these are our thing," we have to manually tell them or give them a checklist, which doesn't really help. So, I have to hire Errol cause it stays on my list every single month and it hasn't been done. That's what I'm going to pass the buck on versus the website. I'd like to do the marketing. So we need to finish all of this by the end of the year. That's on our list. Does it check the list? We're at the last, getting to the last quarter. So you give us the tools. It's just setting it up. That takes a lot of time and concentration time. And Errol seemed to be I met him at DoorGrow live, you know, in Texas. And yeah, he was talking about processes and creating them. Like I talked about property management, so he's going to be our guy. I'll see how it goes.  [00:11:47] Alex: We have a lot in our heads, obviously. So, that's getting it all down to where if somebody needs to know something, it's much easier.  [00:11:56] Jason: Yeah we're planning on doing some more stuff with Errol Allen, who Jill's speaking with, and he's currently playing around with our DoorGrow flow software and testing it out as well. [00:12:05] So I think it's going to be a game changer for the market. So Sarah's had a lot of interaction, I think, with the two of you. What's been your perception of why they do so well as clients?  [00:12:18] Sarah: Oh, well, so there's a few things that I'd like to kind of. Point out and give you guys like major kudos on. First is, I think you're just open. Sometimes we have people who are very resistant. They're like, " that won't work," and "I'm not going to do it like this," and "I can't do this," and "that's not in my market," right? And I think the difference is just being open to the thought and the idea is enough to make it work because if you go into something and you think, "oh, this won't work," well, you're probably right. Then it's not going to work. But you guys are very open and you also, I love this about you guys, you take action. You just come in and you're like, "this is what we're going to do," and then you take action, you implement and you get it done. I think, to date, they are the fastest people who have completed everything in the Rapid Revamp. Like, they get a medal for that. Like, every time, they're like, "yep, we're done with this," I'm like, "oh, wow, okay!" They just get it done. It's like they just put their heads down. They know what they need to do. They put in the work and they get it done and then they go, "okay, great, we did that. What do we need now? Like what's the next thing that we can do to either like build on top of that or like take us to the next level? And I think you guys are really great at that. And I think you, you work very well together. You know, you balance each other out. You like ping well back and forth, back together, and I think that gives you the ability to move things along so quickly.  [00:13:44] Alex: It's great to have ideas that we can bounce off of each other and make it a solid process and get it out of the way and move on to the next one.  [00:13:52] Jill: Well, and I love a checklist. So you have a checklist. I want to see checks on there. I don't want to see them open. So I think that myself, I can be more reactionary property management. Our phone is always ringing. Things are always happening. You know, I can easily not get anything accomplished in a day and be busy the whole day. So with the Rapid Revamp it has me be on track along with handling the things that come on you know all day but I have to get my things done  [00:14:18] Alex: And the nice thing about your dashboard was the fact that you could assign things, we would take them and split them up and be like, "okay, you're going to do these and they're assigned to you" and then I could assign ones to me so we can you know, handle what we needed to.  [00:14:30] Jason: Cool. [00:14:31] Sarah: Yeah. Yeah. I think that was really awesome just to see you guys because every time I check in with you, you're like, "Oh, yeah, we're done with that already." Like, okay, let's see what's the next thing for you guys? And you already knew! You were never like, "Hey, I don't know what I'm supposed to be doing. Like, you just like stayed the course. And sometimes it's hard for entrepreneurs to do because there's so many shiny objects. There's so many of them, right? Like, "Hey, I'm coming in, I'm doing this one thing and that's it," and then along the way, there's like some other little thing that's like, "Hey, I need your attention." [00:15:04] And it's so tempting to go, "Ooh, but I could focus on that." Like, " let me just go over here for a second," and like, you guys just stayed the course. You like stay on point. And I think that's that's something I really have to give you guys like a huge compliment on because it's hard to do that. It's really difficult to do that. And you guys do it really well.  [00:15:25] Jill: Thank you.  [00:15:26] Jason: Yeah. And so you've interacted with several of our team members, right? It's not just the Jason show or the Jason and Sarah show. And I think that's what a lot of people think. Could you just comment a little bit on DoorGrow's team? You don't have to remember everybody's names, but yeah.  [00:15:43] Jill: Well the two that I've probably enjoyed the most is Clint. He's like the coolest surfer dude in the whole wide world, but he's sharp as a tack. You know, "we're just going to buy a $5 million company." He's the exact person to teach you how to be cool and do acquisitions and whatnot. [00:16:03] And that you can see why he's so successful because he's a joy to listen to.  [00:16:07] Jason: Yeah, he's fun.  [00:16:08] Jill: And ironically considering an acquisition in the middle of all listening to him and he took his time out, sent me a lot of information and questions I should ask and what due diligence I should do. So, I mean, his wealth of all the years that he's done that, enticed in a few documents was, I could have never created that. And then Roya, she's a ball of energy and I'm all into manifesting and all that. So, I mean, not many people you can feel through a computer screen with their energy, you know, that's heard of talent that she has.  [00:16:43] Jason: Yeah, she's our dangerously powerful mindset coach. And teaches the advanced sales stuff. [00:16:51] She's yeah she's had quite an impact. Yeah.  [00:16:54] Jill: Yeah. For sure.  [00:16:56] I went to DoorGrow live, which was fantastic to connect with everybody. But thanks to DoorGrow and Alex being also trained as a DoorGrow. I'm taking my first three week vacation in 10 years.  [00:17:08] Jason: That's amazing. That's awesome. Yeah. Yeah. That's awesome. Your business will be in good hands with Alex and and we've got his back. So. For sure. So awesome. Yep. Property managers, if you're listening to this and you have not taken a significant vacation in the last five years, when's your turn? Maybe it's time to reach out and let us help you take- this is one of the most common things that we hear, especially this summer. [00:17:36] Lots of our clients are taking vacations like for the first time ever, or in the first time in a long time, or it's a longer vacation than they've been able to take.  [00:17:45] Sarah: Brandon and Mark, they took off the majority of July, both of them, took off the majority of July, and they're like, "things were fine, like things were okay," I'm like, "that's great, that's how it should work," and if we set it up that way, then things can work that way.  [00:18:01] Jason: For sure. Yeah, one of our mentors had this quote, I don't know where it came from, but he said, the more valuable you are to your business, the less valuable your business is. Ooh, like that one. So Jill's working on making herself less valuable to the business. I've made DoorGrow less of the Jason show, and we've got all these amazing coaches and yeah, and that's the goal, right? We're able to provide more value and it allows us to be more free as entrepreneurs. To do the things that we really enjoy doing and eventually maybe to do nothing. If that's really the goal. I don't know. Jill, will have to find something to do. She's going to trap the world. She'll think we're not going to do nothing. Exactly. We're not going to do nothing. I don't think Jill knows what to do.  [00:18:43] Jill: We just want freedom to not always to be working.  [00:18:46] Jason: There you go. Yeah.  [00:18:48] Sarah: You can choose the things you do.  [00:18:50] Jill: Yeah.  [00:18:51] Jason: Well, we've really appreciated having you both in the program. You know, the, Sarah mentioned about you, but what I've noticed is Jill, you have this gift of positivity, it seems to rub off on everyone around you. We've really enjoyed having you in the program. Everyone's like, "Oh, we love Jill." All of our coaches and team members love Jill. And you can see Alex has like got a positive, you know, energy going on as well. And so you've created a really good culture on your team and in your business. And I don't know if it's always been that way, but I know that's something that's important to us at DoorGrow is making sure everybody has good culture with their business and with their team. So can you touch on culture just a little bit? [00:19:30] Jill: Well, I think connection and culture is the most important thing. If I don't have it here, how is a client going to want to be attracted to us? You know, how is that going to work? You know, if you don't have a positive look on the industry, the business... I mean, this is anybody that calls us is frustrated with property management and say, "here, we love to do property management." They're like, "I need you!" [00:19:51] you know, tenants and everybody gets to complain to us and we have to listen to them and, you know, do our job, but in these walls of this company, we don't have to do that. We can vent to each other. We can laugh. We don't complain. We more laugh about situations than we do complain. And I think I've been a good leader as far as that goes. But I think that also because I have that energy, I want to attract that energy. And so those people are, who are working here and stay.  [00:20:18] Jason: I love that. I mean, I think having a culture in which complaining is not the norm. I mean, it's easy to complain in property management. Right? And I'm sure there's a lot of you listening that are like, " I complain all the time. I complain every day," like reducing that complaining in the business and creating a culture where the team don't see that it's totally okay to just complain all the time. Because if you're complaining about your clients, they're going to feel that. They're not going to want to work with somebody that's, they know is just going to be complaining about them behind their back. [00:20:47] And so I think that's really powerful. And I think that there's a lot of joking in property management, and I think if you can't laugh about it, then you're just going to be hurt by it, and so...  [00:20:58] Jill: and the only way you make a lot of money is to do the things that nobody wants to do. [00:21:02] Jason: There you go. And they will pay you a pretty penny to do it. [00:21:05] Alex: Yeah, we don't have one person that dreads coming to work every day. That's for sure. Everybody's like, "oh shoot. It's monday. Let's go!"  [00:21:11] Jill: We're a little family.  [00:21:13] Jason: Awesome. Yeah, I love that. You have a good culture. So, cool well, anything else we should chat about? What are the biggest takeaways you feel like you've gotten from being part of working with DoorGrow for those listening? [00:21:28] Jill: I think first of all to make sure that I express my purpose to everybody, you know, start with the person.  [00:21:34] Jason: Has that changed your close rate? Has that changed how clients respond to you?  [00:21:39] Jill: Oh, just overall being brave enough to start with that, you know, I always assume they don't care, you know they're not calling for my me personally, but they are, you know, and some would get to know me on a personal level over time, but I never started the conversation with that. [00:21:54] I always started it with "I love property management" and I think they could feel our energy, but not deep down what my life purpose is. So, and how I could tie that back into having them become our client. But it gets a personal, it makes it a personal fit right away or not.  [00:22:11] Jason: Yeah. They either trust your motives and like them or they don't, but they, at least they know what your motives are. Otherwise they're just going to assume you just want their money.  [00:22:20] Jill: Yeah. The name change was a huge one. And then the third, I think final one for me is. When you did your stack deck and it wasn't like perfectly animated with all these designs and it looked great. And I'm fine with it. I stopped judging my marketing to have to be the caliber of Coca Cola. [00:22:40] I don't have designers out there. I don't want to spend design. So just produce it and get it out there and make it look kind of quirky and we're quirky anyway. So I don't know why I was thinking that we had to be this high level, corporate marketing program in order for it to work.  [00:22:54] Jason: I think done is better than perfect for sure. [00:22:57] That's one of my  [00:22:57] Alex: favorite things is like, no, just get it complete and then we'll move on and we'll get the next thing done.  [00:23:03] Jason: Yeah. Done makes money. And you've made a lot of changes. You've gotten a lot of things done that are going to help shore up leaks that make you a lot more money. And. Yeah. A lot of people get really caught up on things being so perfect. [00:23:14] They don't get as nearly as much done. So kudos to both of you for implementing and taking action. So, well, we appreciate you coming and hanging out with us here on the show. What do you feel like, what are some tangible results besides the brand? Revenue doors, any other shifts that you've seen in the business since joining? [00:23:33] Jill: Well, we've gotten rid of a lot of the properties. I had the guts to say to a couple owners, you know, "You have to either sell this property or find another manager because it's too much of a liability. And I'm scared to because X Y Z and so should you." And obviously it's a great time to sell last year. So this is the time get to get a better asset, 1031 exchange it, or let's you know, we need to drop it by the end of the year. I didn't, you know, say we're going to drop you on 30 days, but they, most of them, most of those as a consulting, they trust us and know us and they sold those properties. We have two that are closing this week, our last two that are closing and we had problems. Yeah, problems. So we've gotten rid of a lot of problems since the beginning and liability issues, you know, you know, liabilities. So that's that's, I think our biggest deal and it's allowed other doors to come in. [00:24:28] It's amazing what you let go just energetically things will fill its place. So door wise, I would say we're at about the same, but revenue has gone up 20%.  [00:24:38] Alex: We've been getting higher-end properties instead of, you know, things that were D class properties that we didn't want.  [00:24:44] Jason: Love it. 20 percent more revenue. Awesome, that does not suck.  [00:24:48] Sarah: And getting rid of the problem, right?  [00:24:55] Jason: Well, we appreciate you being clients and we're super excited to see your progression through the DoorGrow code, and this business I think that could easily be at a thousand doors in the next two to three years. It's totally doable, especially if you start doing some of the acquisition deals, like it's going to be really interesting once you get some of these systems in place, then you're ready to just scale like crazy. So excited to see what you do. All right. Well then we'll go ahead and wrap up. Appreciate you being on the show. [00:25:25] Thanks for hanging out with us, Alex and Jill. Thank you. Great.  [00:25:29] For those listening, if you want to be like Alex and Jill and make good decisions and grow your business in a healthy way, and maybe increase your revenue 20%. aNd clean up your portfolio and optimize your sales pipeline so you make more money, more easily reach out to DoorGrow. [00:25:45] We would love to take a look at your business and see if we can help you. The answer is: we can... most likely and see if you'd be a good fit for our program. You can check us out at doorgrow. com. There's a big pink button on the home page says "I want to grow." click that. Do the three steps there to see if you'd be a good candidate to work with us, and until next time to our mutual growth. Bye everyone  [00:26:08] you just listened to the #DoorGrowShow. We are building a community of the savviest property management entrepreneurs on the planet in the DoorGrowClub. Join your fellow DoorGrow Hackers at doorgrowclub.com. Listen, everyone is doing the same stuff. SEO, PPC, pay-per-lead content, social direct mail, and they still struggle to grow!  [00:26:35] At DoorGrow, we solve your biggest challenge: getting deals and growing your business. Find out more at doorgrow.com. Find any show notes or links from today's episode on our blog doorgrow.com, and to get notified of future events and news subscribe to our newsletter at doorgrow.com/subscribe. Until next time, take what you learn and start DoorGrow Hacking your business and your life.

On today's episode of the "Helping Families Be Happy" podcast, host Adina Oberman, talks to Jason Goldstein about Hanukkah and holiday hosting. Jason Goldstein is a chiropractor by day and a food Blogger by night for his culinary blog, shop Happy. Jason shares his love of easy comfort food recipes, showcasing rich flavors, inventive ideas, and unique cooking tips and advice. He was a finalist on the next Food Network Star season 14 and finished in the top ten in Rachel Raye's cookbook contest. His recipes have been featured on The Chew and the Kitchen, and he has appeared on Good Morning America living in New York City in Hamptons. Jason enjoys testing recipes on his husband Tom's and grabbing French fries by the handful.   Episode Highlights 01:46 Jason emphasizes the importance of minimizing effort in holiday hosting. He recommends a combination of homemade and store-bought appetizers. He shares his love for puff pastry appetizers, highlighting how they can be prepared a day ahead for convenience. 02:39 For the main meal, Jason suggests making it yourself but delegating the responsibility of bringing wine and desserts to guests. This not only shares the load but also helps cut down on expenses and dishwashing. 03:47 Jason's second favorite dish for Hanukkah is brisket made in a slow cooker. This method not only saves oven space but also simplifies the cooking process. The brisket simmers all day, resulting in a tender and delicious main dish that seems effort-intensive but is easy to prepare. 04:29 Jason declares himself to be on team applesauce but acknowledges the strong divide between sour cream and applesauce enthusiasts. He emphasizes the importance of having both condiments available at the table. 04:50 Adina expresses appreciation for the idea of making a sheet pan potato latka instead of individual ones. She comments on the tediousness of shredding potatoes and onions for preparation. 05:18 Jason touches upon dietary preferences, recommending vegan cheese for those following stricter dietary laws. He talks about the simplicity of the dish using shredded potatoes, grated onion, and cheese, emphasizing the ease of preparation and baking. 07:21 Jason describes the brisket's final texture as melt-in-the-mouth, resulting from slow cooking that breaks down the fat and connective tissues. Jason fondly mentions that brisket pairs perfectly with potato pancakes. 08:00 Jason explains the traditional significance of frying foods during Hanukkah, referencing the eight days the oil miraculously burned. 09:33 Adina appreciates Jason's ideas and highlights how Hanukkah is a fun holiday, especially for kids. She asks if he uses a deep fryer for the frying process. 09:44: Jason confirms that one can use a deep fryer or simply a pan filled with oil. He also gives a tip to avoid overcrowding the pan while frying, as it can reduce the oil's temperature, leading to uneven cooking. 11:29 Jason shares a personal anecdote from his wedding where he had a large table dedicated to various versions of "pigs in a blanket" — underscoring his love for the dish and its significance to him. 12:17 Adina talks about attending a party where the pigs in a blanket had everything bagel seasoning on the puff pastry, adding a unique flavor. 14:10 Jason emphasizes the importance of being a part of the party and not getting too stressed out. Advises sticking to familiar recipes and prepping in advance. He also recommends having a variety of dishes like salmon for those who don't eat beef. Mentions the idea of a sheet pan salmon. 15:21 Jason suggests that if you're not into cooking, make one dish and have others bring dishes. Recommends people bring dishes from their childhood for a touch of nostalgia. 16:02 Adina talks about the comfort guests feel with familiar flavors and the importance of childhood flavors during the holidays. 3 Key Points Jason Goldstein emphasizes the importance of minimizing effort in holiday hosting. He shares his love for puff pastry appetizers, highlighting how they can be prepared a day ahead. Jason recommends brisket made in a slow cooker for Hanukkah. The method simplifies the cooking process, resulting in a tender dish. Jason touches upon dietary preferences, recommending vegan cheese. He explains the dish's simplicity using shredded potatoes, grated onion, and cheese. Tweetable Quotes "Minimize holiday hosting effort with a mix of homemade and store-bought appetizers. Puff pastry delights can be prepped a day ahead!" - Jason  "For a melt-in-the-mouth Hanukkah delight, try brisket in a slow cooker. Effortless yet tastes like hours in the kitchen!" - Jason  "Whether you're #TeamApplesauce or #TeamSourCream, ensure both are at the Hanukkah table. Harmony in condiments, harmony in life!" - Jason  "Not into cooking? No worries. Host with one dish and let guests bring nostalgic flavors from their childhood. A holiday feast everyone cherishes!" - Jason    Resources Mentioned Helping Families Be Happy Podcast Apple Podcast Editing

Get ready, as we are joined by Jason Bassler founder of The Free Thought Project and Police the Police for a candid conversation about political corruption, digital activism, and the profound impacts and reach of war. Listen in as Jason shares his transition from libertarian to anarchist, spurred by his experiences during the Occupy Wall Street which also helped catalyze his subsequent dedication to police accountability. He offers an intriguing behind-the-scenes look at the Free Thought Project, its evolution, and its unexpected shutdown on major social media platforms in 2012, and its uphill rebuild. We don't shy away from the hard truths in this episode. We scrutinize the alarming effects of war, corruption, and propaganda on our society. This includes a frank discussion on the startling death toll from the post-9/11 war on terror and the profits made by US contractors. We take a deep look at the Ukrainian War as a potential proxy war, the implications of the military-industrial complex, and the ripple effects of these conflicts across different generations. The power of the internet in how different generations are countering these narratives is a key focus.   Lastly, we expose the uncomfortable realities about the US' militarism, its lack of accountability for war crimes, and the fleeting anti-war stance when candidates ascend to power (and their subsequent escalation of conflicts once they have that power). From Lockheed Martin's sponsorship of a pride parade to the $2300 expat exit fee from a free country, we lay bare the government's blatant hypocrisy. Jason's reflections on the strength of the internet and the value of criticism as a sign of tough love conclude our conversation on a hopeful note. This episode promises to be a thought-provoking, no-holds-barred exploration into our political landscape and the power of digital activism. Don't miss it!   Connect with Jason: Jason's Personal Links Police the Police Links Free Thought Project Links   Starting Points:  02:04 Who is Jason Bassler? 11:00 Power of a Platform  20:30 Would society fall into chaos without the police? 26:29  War as an Export 34:37 Exhaustion of Perpetual Wars on the Public Psyche 41:36 Realities of War and Its Perpetrators  01:01:00 Connect with Jason   For more on this episode and The Bad Roman Project: For Full Show Notes: https://www.thebadroman.com/show-notes/episode-92 Blog submissions: thebadroman.com/contribute-to-the-blog Connect with us on social: thebadroman.com/social-links Want to get more involved? Request to join the private discussion group on Facebook (Bad Romans Only!!) No King but Christ Network: nokingbutchristnetwork.com

Hello Metal Heads. As we roll out a special Crossover episode this week with our friends from the Talk louder Podcast, Let us NOT forget WHY we celebrate Memorial Day. Its for the men and women that have both lost their lives and risked their safety so that metal nerds like us can do podcasts like this today. Thank you for your service! Today we discuss the state of METALLICA in 2023. We welcome "Metal" Dave Glessner and Jason McMaster from the Talk Louder Podcast as we take deep dives into the new CD 72 Seasons, The groundbreaking "NO REPEAT WEEKEND" Tour and the fall POWER TRIP Mega show in California. We introduce our speculation that the end of the band is closer than we think. Are the guys in METALLICA  quietly dropping hints. You be the judge. Dave and Jason share their opinion on such topics as the level of METALLICA in rock history, their views on the Huge POWER TRIP Concert in October and their desire to attend the show. HUGE THANK YOU to both DAVE and JASON For making the time to TALK LOUDER with us Tonight! Another example of a great unique conversation filled with in-depth antidotes and disclosures found only here at Metal Mayhem ROC. Thank you for the support and remember to always KEEP IT HEAVY!! Visit the website, join the Metal Mayhem ROC community by Signing up for our weekly newsletter. This way we can keep you updated on all new podcast episodes reminders for our live Radio show on Monday nights as well as autmatic entry into special prizes and giveaways.  https://metalmayhemroc.com/ https://metaldevastationradio.com/ http://pantheonpodcasts.com/ https://twitter.com/MetalmayhemR https://www.youtube.com/channel/UC1Y8gRcKQODNMWwyLBfIHOA https://www.instagram.com/metalmayhemroc/ https://www.facebook.com/groups/metalmayhemroc TALK LOUDER PODCAST and SOCIAL'S: https://talklouderpodcast.com/ https://www.facebook.com/profile .php?id=100064228684731 https://www.youtube.com/channel/UCuh_3gYykV9PdljEkSWZgQA Learn more about your ad choices. Visit megaphone.fm/adchoices

Jason Pereira talks to Kevin Victor, VP of Sales & Partnerships at Mako Financial Technologies. It is essentially SASS for wealth workflow automation. Episode Highlights02.00: Kevin says that in 2018 it seemed very difficult to have strong remote operations and that in itself was a challenge. More importantly it has always been a headache that private equity firms, VC firms, alternative investment firms, or anyone who ultimately leverages this agreement to complete investor onboarding are an absolute nightmare. 04.30: Kevin talks about the unique type of workflow arrangement. How do you go from your firm's internal documents to the wide variety of account opening forms while going between a household to just let alone individual investor documents, non-Reg, TFSA, RSP?08.01: Most advisors have discretion over how they manage money. If you are dealing with major bank, Jason can pretty much guarantee you that if he goes to 12 different advisors at random, they all have different portfolios and how they match. 09.50: Different firms have different offerings or more focused offerings, and we are simply going to configure and supply that for you, says Kevin.11.05: Kevin talks about digitizing investment selection, digitizing unique firm documents, digitizing standard custodian forms, or any other third party after relying upon to execute client's operations with the capacity to do so. Let's analyze it for the client and give them solutions that make sense.14.26: Mako has the ability of doing three things, collecting data and the data is up to grabs. Populating forms that everybody wants to populate and creating the workflows, says Jason. 20.57: God forbid you are at the 9th inning as an advisor, and you identified a mistake by the custodian. You just have to simply resend the fix, or all interact within the platform, and the client is still going to receive that same white label experience, says Kevin.21.34: You don't have to start from the beginning, so the burden and efficiency are the burden is being resolved. The efficiencies that are being gained by the advisor, same experience for the investor, the turnaround time is important as well if we are reducing the burden of, operational deficiencies, human errors now go up things that are now being corrected on the advisor's behalf. They are going to be able to execute and open up the client's account much faster. They are going to be able to complete that transfer in much faster. 3 Key PointsKevin talks about the genesis of the company and where did the idea of launching it come from?Kevin shares how Mako Financial has created something that adapts to handle high variability; the firm and accounts for a 10/10,000 fund codes that exist in this country and all the ETF. He talks about how his company deals this large degree of complexity. Jason and Kevin talk about advisor and firm complexity issue and also about the complexity in dealing with different custodians.Tweetable Quotes"Digitizing was easier for Robo advisors, because they were a homogeneous uniform group." - Jason"For relationships where you don't have full integration, we also have other options." - Kevin Resources MentionedFacebook – Jason Pereira's FacebookLinkedIn – Jason Pereira's LinkedInWoodgate.com – Sponsor See acast.com/privacy for privacy and opt-out information.

In this episode, we cover: Mauricio talks about his background and his role at Pismo (1:14) Jason and Mauricio discuss tech and reliability with regards to financial institutions (5:59) Mauricio talks about the work he has done in Chaos Engineering with reliability (10:36) Mauricio discusses things he and his team have done to maximize success (19:44) Mauricio talks about new technologies his team has been utilizing (22:59) Links Referenced: Pismo: https://pismo.io/ LinkedIn: https://www.linkedin.com/company/pismo/ TranscriptMauricio: That's why the name Cockroach, I guess, if there's a [laugh] a world nuclear war here, all that will survive would be cockroaches in our client's data. [laugh]. So, I guess that's the gist of it.Jason: Welcome to Break Things on Purpose, a podcast about Chaos Engineering and reliability. In this episode, we chat with Mauricio Galdieri, a staff engineer at Pismo about testing versus exploration, reliability and resiliency, and the challenges of bringing new technologies to the financial sector.Jason: Welcome to the show.Mauricio: Hey, thank you. Welcome. Thanks for having me here, Jason.Jason: Yeah. So, Mauricio, you and I have chatted before in the past. We were at Chaos Conf, and you are part of a panel. So, I'm curious, I guess to kick things off, can you tell folks a little bit more about yourself and what you do at Pismo? And then we can maybe pick up from our conversations previously?Mauricio: Okay, awesome. I work as a staff engineer here at Pismo. I work in a squad called staff engineering squad, so we're a bunch of—five squad engineers there. And we're mostly responsible for coming up with new ways of using the existing technology, new technologies for us to have, and also standardize things like how we use those technologies here? How does it fit the whole processes we have here? And how does it fit in the pipelines we have here, also?And so, we do lots of documentation, lots of POCs, and try different things, and we talk to different people from different companies and see how they're solving problems that we also have. So, this is basically our day-to-day activities here. Before that, well, I have a kind of a different story, I guess. Most people that work in this field, have a degree in something like a technical degree or something like that. But I actually graduated as an architect in urban planning, so I came from a completely different field.But I've always worked as a software developer since a long time ago, more than [laugh] willing to disclose. So, at that time when I started working with software development, I like to say that startups were called dotcoms that back then, so, [laugh] there was a lots of job opportunities back then, so I worked as a software developer at that time. And things evolved. I grew less and less as an architect and more as an engineer, so after I graduated, I started to look for a second degree, but on the more technical college, so I went to an engineering college and graduated as a system analyst.So, from then on, I've always worked as a software developer and never, never have done any house planning or house project or something like that. And I really doubt if I could do that right now [laugh] so I may be a lousy architect [in that sense 00:03:32]. But anyway, I've worked in different companies for both in private and public sectors. And I've worked with consultancy firms and so on. But just before I came to Pismo, I went working with a FinTech.So, this is where I was my first contact with the world of finance in a software context. Since then, I've digged deep into this industry, and here I am now working at Pismo, it's for almost five years now.Jason: Wow. That quite a journey. And although it's a unique journey, it's also one that I feel like a lot of folks in tech come from different backgrounds and maybe haven't gone down the traditional computer science route. With that said, you know, one of the things you mentioned FinTech. Can you give us a little bit of a description of Prismo, just so folks understand the company that you're working at now?Mauricio: Oh, yeah. Well, Pismo, it's a company that has about six years now. And we provide infrastructure for financial services. So, we're not banks ourselves, but we provide the infrastructure for banks to build their financial projects with this. So basically, what we do is we manage accounts, we manage those accounts' balances, we have connections with credit card networks, so we process—we're also a credit card processor.We issue cards, although we're not the issuer in this in the strict sense, but we issue cards here and manage all the lifecycle of those cards. And basically, that's it. But we have a very broad offering of products, from account management to accounting management, and transactions management, and spending control limits and stuff. So, we have a very broad product portfolio. But basically, what we do is provide infrastructure for financial services.Jason: That's fascinating to me. So, if I were to sum that up, would it be accurate to say that you're basically like Software as a Service for financial institutions? You do all the heavy lifting?Mauricio: Yeah, yeah. I could say that, yeah.Jason: It's interesting to me because, you know, traditionally, we always think of banks because they need to be regulated and there needs to be a whole lot more security and reliability around finances, we always think of banks as being very slow when it comes to technology. And so, I think it's interesting that, in essence, what you've said with trying the latest technology and getting to play around with new technology and how it applies, especially within your staff engineering group, it's almost the exact opposite. You're sort of this forefront, this leading edge within the world of finance and technology.Mauricio: Yeah. And that actually is, it's something that—it's the most difficult part to sell banks to sign up with us, you know? Because they have those ancient systems running on-premises and most likely running on top of COBOL programs and so on. But at the same time, it's highly, highly reliable. That they've been running those systems for, like, 40 years, even more than that, so it's a very highly reliable.And as you said, it's a very regulated industry, so it's very hard to sell them this kind of new approach to banking. And actually, we consider this as almost an innovation for them. And it's a little bit strange to talk about innovation in a sense that we're proposing other companies to run in the cloud. This doesn't sound innovating at all nowadays. So, every company runs their systems in the cloud nowadays, so it's difficult to [laugh] realize that this is actually innovation in the banking system because they're not used to running those things.And as you said, they're slow in adopting new technologies because of security concerns, and so on. So, we're trying to bring these new things to the table and prove them. And we had to prove banks and other financial institutions that it is possible to run a banking system a hundred percent in the cloud while maintaining security standards and security compliances and governance compliance and all that stuff. It's very hard to do so and we have a very stringent process to evaluate and assess new technologies because we have to make sure it complies with those standards and all those certifications that we need to have in order to operate in this industry. So, it's very hard, but it doesn't—at that same time, we have lots of new technologies and different ways we can provide the same services to those banks.And then I think the most difficult part in this is to map what traditional banks were doing into this new way of doing things in the cloud. So, this mapping, it's sometimes it gets a little confusing and we have to be very patient and very clear with our clients what they should expect from us and how we will provide the same services they already have now, but using different technologies and different ways. For instance, they are used to these communications with different services, they're used to things like webhooks. But webhooks are not reliable; they can fail and if they fail, you lose that connection, you lose connectivity, and you may lose data and you may have things out of sync using webhooks. So, now we have things like event streaming, or queues and other stuff that you can use to [replay 00:09:47] things and not lose any data.But at the same time, you have to process this, and then offline in an asynchronous manner. So, you have to map those synchronous things that they did before to this asynchronous world and this world where things are—we have an eventual consistency. But it's very difficult but it's also at the same time, it's a very fascinating industry.Jason: Yeah, that is fascinating. But I do love how you mentioned taking the idea of the new technology and what it does, and really trying to map that back to previously—you know, those previous practices that they had. And so, along with that, for folks who are listening again, Mauricio and I had a chat during Chaos Conf a while back, and he was sharing some of the practices that Pisma has done for Chaos Engineering. And I always liken that back to, you know, Chaos Engineering really is very similar to traditional disaster recovery testing, in many ways, other than oftentimes, your disaster recovery would never actually, you know, take things down. Mauricio, I'm curious, can you share a little bit more about what you've been doing with Chaos Engineering and in general, with reliability. Are there any new programs or processes that you've worked on within Prismo around Chaos Engineering and reliability?Mauricio: Well, I think that the first thing to realize, and I think this is the most important point that you need to have very clear in your mind when we're talking about Chaos Engineering is that we're not testing something when we're doing Chaos Engineering; we're experimenting with something. And there's a subtle but very important distinction between those two concepts. When you test for something, you're testing for something that you knew what will happen; you have an idea of how it should behave. You're asserting a certain behavior. You know how the system must behave and you assert that, and it makes sure the system doesn't deviate on that by having an automated test, for instance, a unit or integrated test, or even functional tests and such.But Chaos Engineering is more about experimenting. So, it's designed for the unknowns. You don't know what will happen. You're basically experimenting. It's like a lab, you're working in a laboratory, you're trying different stuff and see what happens, you have an idea of what should happen and we call this a hypothesis, but you're not sure if that is how we will behave.And actually, it doesn't matter if it complies with your expectations. Even if it doesn't behave the way you expect it to behave or the way you want it to behave, you're still gaining knowledge about your system. So, it's much more about experimenting new things instead of actually testing for some something that you know about. And our journey here into Chaos Engineering at Pismo, it all began about a year-and-a-half ago when we got a very huge outage on one of our major cloud providers here. And we went down with them; they were out for about almost an hour.But not only we were affected by it, but other digital banks here in Brazil, but also many other services like Slack, Datadog, other observability tools that were running at that time, using that cloud provider went down, together with them. So, it was a major, major outage here. And then we were actually caught off guard on this because we have lots of different ways to make sure the system doesn't go down if something bad happens. But that was so bad that we went down and we couldn't do anything. We were desperate because we couldn't do anything. And also we can even communicate properly because we use Slack as our communication hub, so Slack was down at that time, also, so we cannot communicate properly with our official channels.Also, Datadog that we were using at a time also went down and we couldn't even see what was happening in the system because we didn't have any observability running at the time. So, that was a major, major outage we had there. So, we started thinking about ways we could experiment with those major outages and see how we could find ways of still operating at least partially and not go down entirely or at least have ways to see what was happening even in the face of a major disaster. And those traditional disaster recovery measures that were valid at the time, even those couldn't cope with the kind of outages we were facing at that time. So, we were trying to look for different ways that we can improve the reliability of our services as a whole.So, I guess that's when we started looking into Chaos Engineering and started looking for different tools to make that work, and different partnerships we could find, and even different ways we could experiment this with our existing technology and platform.Jason: I really love how you characterized that difference between testing and Chaos Engineering. And I think the idea of being more experimental puts you into a mindset of having this concept of, you know, kind of blamelessness, right, around failure. The idea that, like, failure is going to happen and we want to be open to seeing that and to learning from it. More so than a test, right? When we test things, then there's the notion of a pass-fail and fails are bad, whereas with an experiment, that learning is, if it didn't happen the way you expect, there's learning around that and that's a good thing rather than a bad thing, such as failing a test.Mauricio: Yeah, and that works in a higher framework, I guess, which is resilience itself. So, I guess, chaos experiment, chaos engineering, and all that stuff, it's an important part of a bigger whole that we call resilience. And I guess a key to understand resilience is that this point exactly, the systems never work in unexpected ways. They always behave the way it is expected to behave. They're deterministic in nature. So, we're talking about machines here, computers. We told them what we want them to do.And even if we have complexity and randomness involved, say if a network connection goes down, it still will behave the way we programmed them to behave. So, every failure should be expected. What we have here is that sometimes they behave in ways we don't want them to behave. And sometimes they behave in ways we want them to behave. So, it's more of a matter of desire, you know? You want something, you want the system to behave a certain way.So, in that sense, success should be measured as a performance variability, you know? So, sometimes it will work the way you want and sometimes it will work your way in ways that you don't want it to behave. And I guess, realizing that, it's key also to understand another point that is, in that sense, success is the flip side of failure. So, either it works the way you want it or it works the way you don't want it. And what we can do to move the scale towards a more successful operation, the ways you can do this, you must first realize also that—let's go back a little bit then say, if you have a failure and you look at why it happened, almost never it is the result of one single thing.Sometimes it is, but this is very rare. Most of the failures and even mainly when we're talking about major failures, they're most likely the result of a context of things that happened that led to this failure. And you can see that the same thing, it's valid for successes. When you have a success at one point, it's almost never the result of one thing that you did that led to a successful scenario. Most of the time is a context of different things you did that maximizes your chances of success.So, to turn this scale towards success, you should create an environment of several things, of a context of things. And this could be tooling, this could be your organizational culture and stuff, all of those things that you do in your company to maximize their chances of success. It's not, you cannot plan for success in the sense because planning is one thing you can do, and planning doesn't involve strategy, for instance. Because planning should be done thinking about things you can do, tasks you can perform, while strategy, you should be turning tables to [laugh] think in terms of strategy. So, you have to put all of this in the same way in a table and try to organize your company and your culture, your tools and your technology in ways you maximize your chances of success and minimize your chances of failures.Jason: That's such an interesting insight. So, I'm curious, can you dive into some of the things that you and your team have done to maximize your chances of success?Mauricio: Okay. When we started working with Chaos Engineering, it was in this sense of trying to do one more thing to maximize our chances of success. And we partnered up with Gremlin and we saw that working with Chaos Engineering, using Gremlin mainly, it's so easy—that is, it's also easy to lose track of what you're doing. It's easy for you to go just for the fun of it and break things down and have fun with it and stuff. So, we had to come up with a way to bring structure to this process.And by doing so, we should also not be too bureaucratic in the sense of creating a set of steps you should take in order to run a chaos session. So, one way we thought about was to come up with a document. That is the bureaucratic part, so this was a step you should take in order to plan for your chaos session, but there is one part of it—and I think it's one of the most important parts of this chaos session planning—is that you should describe what you're going to test, but more importantly, why you're going to test this. And this is one of the most important questions because this is a fundamental question: why you're doing this kind of experiment. And to answer that, you have to think about all the things in context.What are the technologies you're using? Why it fails in the first place? Do the fails that I expect to see are actually fails or is it just different ways of behaving? And sometimes we consider failure in a business rule that was not complied, that was not met. So, this is an opportunity to think about, are those business rules correct? Should we make it more flexible? Should we change those business logic?So, when you start asking why you're doing something, you're asking fundamental questions, and I think that puts you in context. And this is one of the major starting points to maximize our chances of success because it makes every engineer involved in running a chaos session, think about their role in the whole process and the role of their services in the whole company. So, I think this is one powerful question to ask before starting any chaos session, and I think this contributes a lot to a successful outcome.Jason: Yeah, I think that's a really great perspective on how to approach Chaos Engineering. Beyond the Chaos Engineering, you mentioned that the staff engineering group that you're part of that Prismo is really responsible for seeing new technologies and new trends and really trying to bring those in and see how they can be used and applied within the financial services sector. Are there any new technologies that you've used recently or that you're looking at right now that has really been fruitful or really applied to finding more success as you've mentioned?Mauricio: Yeah, there are some things we're researching. One of those already went past research and we're already using it in production, which is data—cloud-based, multi-region databases and multi-cloud—also—databases. And we're working with CockroachDB as one of our new database technologies we use. And it's a database built from the ground up to be ultra resilient. And that's why the name Cockroach, I guess, if there's a [laugh] a world nuclear war here, all that will survive would be cockroaches in our client's data. [laugh]. So, I guess that's the gist of it.And we have to think about that in different ways of how we approach this because we're talking about multi-cloud data stores and multi-region and how we deal with data in different regions. And should we replicate all the data between regions and how we do partition data. So, we have to think in different ways, how we approach data modeling with those new cloud-based and multi-region and globally distributed databases. Another one that we're—this is more like of a research, is having a sharded processing. And that is, how we can deal with, how we group different parts of the data to be processed separately but using the same logic.And this is a way to scale processing in ways that horizontal scaling in a more traditional way doesn't solve in some instances. Like, when we have—for instance, let me describe one scenario that we have that we're exploring things along those lines. We have a system here called ‘The Ledger,' which keeps track of all of the accounts' balances. And for this system, if we have multiple requests or lots of requests for different accounts, there's no problem because we're updating balances for different accounts, and that works fine. And we can deal with lots and lots of requests. We have a very good performance on that.But when we have lots of requests coming in from one particular accounts, and they're all grouped for this particular account, then we cannot—there's no way around locking at some place. So, you have to lock it either at the database level, or at a distributed locking mechanism level, or at the business logic layer. At some point, you have to lock the access to this account balance. So, this degrades performance because you have to wait for this processing to finish and start another. And how can we deal with that without using locks?And this was the challenge we put that to ourselves. And we're exploring different ways, lots of different ways, and different approaches to that. And we have lots of restrictions on that because this system has to respond quickly, has to respond online, and cannot be in an asynchronous process; it has to be synchronous. So, we have very little space for double-checking it and stuff. So, we're exploring a sharded processing for this one in which we can have a small subset of accounts being routed to one specific consumer to process this transaction, and by doing so, we may have things like a queue of order transactions so we can give up locking at the database and maybe improve on performance. But we're still on the POC on that, so let's see what we come up with [laugh] in the next few months.Jason: I think that's really fascinating. Both from a, you know, having been there, having worked on systems where, you know, very transaction-driven, and having locks be an issue. And so, you know, back in my day of doing this, you know, was traditionally MySQL or Postgres, trying to figure out, like, how do you structure the database. So, I think it's interesting that you're sort of tackling this in two ways, right? You've got CockroachDB, which is more oriented towards reliability, but a lot of the things that you're doing there around, you know, sharding and multi-cloud also have effects for this new work that you're doing on how do you eliminate that locking and try to do sharded processes as well. So, that's all super fascinating to me.Mauricio: Exactly. Yeah, yeah. This is one of the things that makes you do better the end of the day, you know? [laugh].Jason: Yeah, definitely. As an engineer, you know, if anybody's listening and you're thinking of, “Wow, this all sounds fascinating and really cool stuff,” right, “Really cool technologies to be working with and really interesting challenges to solve,” I know, Mauricio, you said that Pismo is hiring. Do you want to share a little bit more about ways that folks can engage with you? Or maybe even join your team?Mauricio: Yeah, sure. We're hiring; we have lots of jobs open for application. You can go to pismo.io and we have a section for that. And also, you can find us on LinkedIn; just search for Pismo and then find us there.And I think if you're an engineer and looking for some cool challenges on that, be sure to check our open positions because we do have lots and lots of cool stuff going on here. And since we're growing global, you have a chance to work from wherever you are. And this also imposes some major challenges for [laugh] for new technologies and making our products, our existing products, work in a globally distributed banking system. So, be sure to check out our channels there.Jason: Fantastic. Before we wrap up, is there anything else that you'd like to promote or share?Mauricio: Oh no, I think those are the main channels. You can find us: LinkedIn and our own website, pismo.io. Also, you can find us in some GopherCon conferences, KubeCon, and other—Money20/20; we're attending all of those conferences, be it in the software industry or in the financial industry. You can find this there with a booth there or just visiting or participating in some conferences and so on. So, be sure to check that out there also. I guess that's it.Jason: Very cool well thanks, Mauricio for joining us. It's been a pleasure to chat with you again.Mauricio: Thank you, Jason. And thanks for having me here.Jason: For links to all the information mentioned, visit our website at gremlin.com/podcast. If you liked this episode, subscribe to the Break Things on Purpose podcast on Spotify, Apple Podcasts, or your favorite podcast platform. Our theme song is called “Battle of Pogs” by Komiku, and it's available on loyaltyfreakmusic.com.

In this episode, we cover: Aaron talks about starting out as a developer and the early stages of cloud development at RBC (1:05) Aaron discusses transitioning to developer advocacy (12:25) Aaron identifies successes he had in his early days of developer advocacy (20:35) Jason asks what it looks like to assist developers in achieving completion with long term maintenance projects, or “sustainable development” (25:40)  Jason and Aaron discuss what “innersource” is and why it's valuable in an organization (29:29) Aaron answers the question “how do you keep skills and knowledge up to date?” (33:55) Aaron talks about job opportunities at RBC (38:55) Links Referenced: Royal Bank of Canada: https://www.rbcroyalbank.com Opportunities at RBC: https://jobs.rbc.com/ca/en TranscriptAaron: And I guess some PM asked my boss, “So, Aaron doesn't come to our platform status meetings, he doesn't really take tickets, and he doesn't take support rotation. What does Aaron do for the Cloud Platform Team?”Jason: [laugh].Jason: Welcome to Break Things on Purpose, a podcast about reliability, learning, and building better systems. In this episode, we talk with Aaron Clark, Director of Developer Advocacy at the Royal Bank of Canada. We chat with him about his journey from developer to advocate, the power of applying open-source principles within organizations—known as innersource—and his advice to keep learning.Jason: Welcome to the show, Aaron.Aaron: Thanks for having me, Jason. My name is Aaron Clark. I'm a developer advocate for cloud at RBC. That is the Royal Bank of Canada. And I've been at the bank for… well, since February 2010.Jason: So, when you first joined the bank, you were not a developer advocate, though?Aaron: Right. So, I have been in my current role since 2019. I've been part of the cloud program since 2017. Way back in 2010, I joined as a Java developer. So, my background in terms of being a developer is pretty much heavy on Java. Java and Spring Boot, now.I joined working on a bunch of Java applications within one of the many functions areas within the Royal Bank. The bank is gigantic. That's kind of one of the things people sometimes struggle to grasp. It's such a large organization. We're something like 100,000… yeah, 100,000 employees, around 10,000 of that is in technology, so developers, developer adjacent roles like business analysts, and QE, and operations and support, and all of those roles.It's a big organization. And that's one of the interesting things to kind of grapple with when you join the organization. So, I joined in a group called Risk IT. We built solely internal-facing applications. I worked on a bunch of stuff in there.I'm kind of a generalist, where I have interest in all the DevOps things. I set up one of the very first Hudson servers in Risk—well, in the bank, but specifically in Risk—and I admin'ed it on the side because nobody else was doing it and it needed doing. After a few years of doing that and working on a bunch of different projects, I was occasionally just, “We need this project to succeed, to have a good foundation at the start, so Aaron, you're on this project for six months and then you're doing something different.” Which was really interesting. At the same time, I always worry about the problem where if you don't stay on something for very long, you never learn the consequences of the poor decisions you may have made because you don't have to deal with it.Jason: [laugh].Aaron: And that was like the flip side of, I hope I'm making good decisions here. It seemed to be pretty good, people seemed happy with it, but I always worry about that. Like, being in a role for a few years where you build something, and then it's in production, and you're running it and you're dealing with, “Oh, I made this decision that seems like a good idea at the time. Turns out that's a bad idea. Don't do that next time.” You never learned that if you don't stay in a role.When I was overall in Risk IT for four, almost five years, so I would work with a bunch of the teams who maybe stayed on this project, they'd come ask me questions. It's like, I'm not gone gone. I'm just not working on that project for the next few months or whatever. And then I moved into another part of the organization, like, a sister group called Finance IT that runs kind of the—builds and runs the general ledger for the bank. Or at least for a part of capital markets.It gets fuzzy as the organization moves around. And groups combine and disperse and things like that. That group, I actually had some interesting stuff that was when I started working on more things like cloud, looking at cloud, the bank was starting to bring in cloud. So, I was still on the application development side, but I was interested in it. I had been to some conferences like OSCON, and started to hear about and learn about things like Docker, things like Kubernetes, things like Spring Boot, and I was like this is some really neat stuff.I was working on a Spark-based ETL system, on one of the early Hadoop clusters at the bank. So, I've been I'm like, super, super lucky that I got to do a lot of this stuff, work on all of these new things when they were really nascent within the organization. I've also had really supportive leadership. So, like, I was doing—that continuous integration server, that was totally on the side; I got involved in a bunch of reuse ideas of, we have this larger group; we're doing a lot of similar things; let's share some of the libraries and things like that. That was before being any, like, developer advocate or anything like that I was working on these.And I was actually funded for a year to promote and work on reuse activities, basically. And that was—I learned a lot, I made a lot of mistakes that I now, like, inform some of the decisions I make in my current role, but I was doing all of this, and I almost described it as I kind of taxed my existing project because I'm working on this team, but I have this side thing that I have to do. And I might need to take a morning and not work on your project because I have to, like, maintain this build machine for somebody. And I had really supportive leadership. They were great.They recognize the value of these activities, and didn't really argue about the fact that I was taking time away from whatever the budget said I was supposed to be doing, which was really good. So, I started doing that, and I was working in finance as the Cloud Team was starting to go through a revamp—the initial nascent Cloud Team at the bank—and I was doing cloud things from the app dev side, but at the same time within my group, anytime something surprising became broken, somebody had some emergency that they needed somebody to drop in and be clever and solve things, that person became me. And I was running into a lot of distractions in that sense. And it's nice to be the person who gets to work on, “Oh, this thing needs rescuing. Help us, Aaron.”That's fantastic; it feels really good, right, up until you're spending a lot of your time doing it and you can't do the things that you're really interested in. So, I actually decided to move over to the Cloud Team and work on kind of defining how we build applications for the cloud, which was really—it was a really good time. It was a really early time in the bank, so nobody really knew how we were going to build applications, how we were going to put them on the cloud, what does that structure look like? I got to do a lot of reading and research and learning from other people. One of the key things about, like, a really large organization that's a little slow-moving like the bank and is a little bit risk-averse in terms of technology choices, people always act like that's always a bad thing.And sometimes it is because we're sometimes not adopting things that we would really get a lot of benefit out of, but the other side of it is, by the time we get to a lot of these technologies and platforms, a bunch of the sharp edges have kind of been sanded off. Like, the Facebooks and the Twitters of the world, they've adopted it and they've discovered all of these problems and been, like, duct-taping them together. And they've kind of found, “Oh, we need to have actual, like, security built into this system,” or things like that, and they've dealt with it. So, by the time we get to it, some of those issues are just not there anymore. We don't have to deal with them.Which is an underrated positive of being in a more conservative organization around that. So, we were figuring there's a lot of things we could learn from. When we were looking at microservices and, kind of, Spring Boot Spring Cloud, the initial cloud parts that had been brought into the organization were mainly around Cloud Foundry. And we were helping some initial app teams build their applications, which we probably over-engineered some of those applications, in the sense that we were proving out patterns that you didn't desperately need for building those applications. Like, you could have probably just done it with a web app and relational database and it would have been fine.But we were proving out some of the patterns of how do you build something for broader scale with microservices and things like that. We learned a bunch about the complexities of doing that too early, but we also learned a bunch about how to do this so we could teach other application teams. And that's kind of the group that I became part of, where I wasn't a platform operator on the cloud, but I was working with dev teams, building things with dev teams to help them learn how to build stuff for cloud. And this was my first real exposure to that scope and scale of the bank. I'd been in the smaller groups and one of the things that you start to encounter when you start to interact with the larger parts of the bank is just, kind of, how many silos there are, how diverse the tech stacks are in an organization of that size.Like, we have areas that do things with Java, we have areas doing things with .NET Framework, we have areas doing lots of Python, we have areas doing lots of Node, especially as the organization started building more web applications. While you're building things with Angular and using npm for the front-end, so you're building stuff on the back-end with Node as well. Whether that is a good technology choice, a lot of the time you're building with what you have. Even within Java, we'd have teams building with Spring Boot, and lots of groups doing that, but someone else is interested in Google Guice, so they're building—instead of Spring, they're using Google Guice as their dependency injection framework.Or they have a… like, there's the mainframe, right? You have this huge technology stack where lots of people are building Java EE applications still and trying to evolve that from the old grungy days of Java EE to the much nicer modern ways of it. And some of the technology conversations are things like, “Well, you can use this other technology; that's fine, but if you're using that, and we're using something else over here, we can't help each other. When I solve a problem, I can't really help solve it for you as well. You have to solve it for yourself with your framework.”I talked to a team once using Vertex in Java, and I asked them, “Why are you using Vertex?” And they said, “Well, that's what our team knew.” I was like, “That's a good technology choice in the sense that we have to deliver. This is what we know, so this is the thing we know we can succeed with rather than actually learning something new on the job while trying to deliver something.” That's often a recipe for challenges if not outright failure.Jason: Yeah. So, it sounds like that's kind of where you come in; if all these teams are doing very disparate things, right—Aaron: Mm-hm.Jason: That's both good and bad, right? That's the whole point of microservices is independent teams, everyone's decoupled, more velocity. But also, there's huge advantages—especially in an org the size of RBC—to leverage some of the learnings from one team to another, and really, like, start to share these best practices. I'm guessing that's where you come into play now in your current role.Aaron: Yeah. And that's the part where how do we have the flexibility for people to make their own choices while standardizing so we don't have this enormous sprawl, so we can build on things? And this is starting to kind of where I started really getting involved in community stuff and doing developer advocacy. And part of how this actually happened—and this is another one of those cases where I've been very fortunate and I've had great leaders—I was working as part of the Cloud Platform Team, the Special Projects group that I was, a couple of people left; I was the last one left. It's like, “Well, you can't be your own department, so you're part of Cloud Platform.” But I'm not an operator. I don't take a support rotation.And I'm ostensibly building tooling, but I'm mostly doing innersource. This is where the innersource community started to spin up at RBC. I was one of the, kind of, founding members of the innersource community and getting that going. We had built a bunch of libraries for cloud, so those were some of the first projects into innersource where I was maintaining the library for Java and Spring using OIDC. And this is kind of predating Spring Security's native support for OIDC—so Open ID Connect—And I was doing a lot of that, I was supporting app teams who were trying to adopt that library, I was involved in some of the other early developer experience things around, you complain this thing is bad as the developer; why do we have to do this? You get invited to one of the VP's regular weekly meetings to discuss, and now you're busy trying to fix, kind of, parts of the developer experience. I was doing this, and I guess some PM asked my boss, “So, Aaron doesn't come to our platform status meetings, he doesn't really take tickets, and he doesn't take support rotation. What does Aaron do for the Cloud Platform Team?”Jason: [laugh].Aaron: And my boss was like, “Well, Aaron's got a lot of these other things that he's involved with that are really valuable.” One of the other things I was doing at this point was I was hosting the Tech Talk speaking series, which is kind of an internal conference-style talks where we get an expert from within the organization and we try to cross those silos where we find someone who's a machine-learning expert; come and explain how TensorFlow works. Come and explain how Spark works, why it's awesome. And we get those experts to come and do presentations internally for RBC-ers. And I was doing that and doing all of the support work for running that event series with the co-organizers that we had.And at the end of the year, when they were starting up a new initiative to really focus on how do we start promoting cloud adoption rather than just people arrive at the platform and start using it and figure it out for themselves—you can only get so far with that—my boss sits me down. He says. “So, we really like all the things that you've been doing, all of these community things and things like that, so we're going to make that your job now.” And this is how I arrived at there. It's not like I applied to be a developer advocate. I was doing all of these things on the side and all of a sudden, 75% of my time was all of these side projects, and that became my job.So, it's not really the most replicable, like, career path, but it is one of those things where, like, getting involved in stuff is a great way to find a niche that is the things that you're passionate about. So, I changed my title. You can do that in some of our systems as long as your manager approves it, so I changed my title from the very generic ‘Senior Technical Systems Analyst—which, who knows what I actually do when that's my title—and I changed that to ‘Developer Advocate.' And that was when I started doing more research learning about what do actual developer advocates do because I want to be a developer advocate. I want to say I'm a developer advocate.For the longest time in the organization, I'm the only person in the company with that title, which is interesting because then nobody knows what to do with me because I'm not like—am I, like—I'm not a director, I'm not a VP. Like… but I'm not just a regular developer, either. Where—I don't fit in the hierarchy. Which is good because then people stop getting worried about what what are titles and things like that, and they just listen to what I say. So, I do, like, design consultations with dev teams, making sure that they knew what they were doing, or were aware of a bunch of the pitfalls when they started to get onto the cloud.I would build a lot of samples, a lot of docs, do a lot of the community engagement, so going to events internally that we'd have, doing a lot of those kinds of things. A lot of the innersource stuff I was already doing—the speaking series—but now it was my job formally, and it helped me cross a lot of those silos and work very horizontally. That's one of the different parts about my job versus a regular developer, is it's my job to cover anything to do with cloud—that at least, that I find interesting, or that my boss tells me I need to work at—and anything anywhere in the organization that touches. So, a dev team doing something with Kubernetes, I can go and talk to them. If they're building something in capital markets that might be useful, I can say, “Hey, can you share this into innersource so that other people can build on this work as well?”And that was really great because I develop all of these relationships with all of these other groups. And that was, to a degree, what the cloud program needed from me as well at that beginning. I explained that this was now my job to one of my friends. And they're like, “That sounds like the perfect job for you because you are technical, but you're really good with people.” I was like, “Am I? I guess I am now that I've been doing it for this amount of time.”And the other part of it as we've gone on more and more is because I talk to all of these development teams, I am not siloed in, I'm not as tunneled on the specific thing I'm working with, and now I can talk to the platform teams and really represent the application developer perspective. Because I'm not building the platform. And they have their priorities, and they have things that they have to worry about; I don't have to deal with that. My job is to bring the perspective of an application developer. That's my background.I'm not an operator; I don't care about the support rotation, I don't care about a bunch of the niggly things and toil of the platform. It's my job, sometimes, to say, hey, this documentation is well-intentioned. I understand how you arrived at this documentation from the perspective of being the platform team and the things that you prioritize and want to explain to people, but as an application developer, none of the information that I need to build something to run on your platform is presented in a manner that I am able to consume. So, I do, like, that side as well of providing customer feedback to the platform saying, “This thing is hard,” or, “This thing that you are asking the application teams to work on, they don't want to care about that. They shouldn't have to care about this thing.” And that sort of stuff.So, I ended up being this human router are sometimes where platform teams will say, “Do you know anybody who's doing this, who's using this thing?” Or finding one app team and say, “You should talk to that group over there because they are also doing the same thing, or they're struggling with the same thing, and you should collaborate.” Or, “They have solved this problem.” Because I don't know every single programming language we use, I don't know all of the frameworks, but I know who I asked for Python questions, and I will send teams to that person. And part of that, then, as I started doing this community work was actually building community.One of the great successes was, we have a Slack channel called ‘Cloud Adoption.' And that was the place where everybody goes to ask their questions about how do I do this thing to put something on Cloud Foundry, put it on Kubernetes? How do I do this? I don't understand. And that was sometimes my whole day was just going onto that Slack channel, answering questions, and being very helpful and trying to document things, trying to get a feel for what people were doing.It was my whole day, sometimes. It took a while to get used to that was actually, like, a successful day coming from a developer background. I'm used to building things, so I feel like success because I built something I can show you, that I did this today. And then I'd have days where I talked to a bunch of people and I don't have anything I can show you. That was, like, the hard part of taking on this role.But one of the big successes was we built this community where it wasn't just me. Other people who wanted to help people, who were just developers on different dev teams, they'd see me ask questions or answer questions, and they would then know the answers and they'd chime in. And as I started being tasked with more and more other activities, I would then get to go—I'd come back to Slack and see oh, there's a bunch of questions. Oh, it turns out, people are able to help themselves. And that was—like that's success from that standpoint of building community.And now that I've done that a couple times with Tech Talks, with some of the developer experience work, some of the cloud adoption work, I get asked internally how do you build community when we're starting up new communities around things like Site Reliability Engineering. How are we going to do that? So, I get—and that feels weird, but that's one of the things that I have been doing now. And as—like, this is a gigantic role because of all of the scope. I can touch anything with anyone in cloud.One of the scope things with the role, but also with the bank is not only do we have all these tech stacks, but we also have this really, really diverse set of technical acumen, where you have people who are experts already on Kubernetes. They will succeed no matter what I do. They'll figure it out because they're that type of personality, they're going to find all the information. If anything, some of the restrictions that we put in place to manage our environments and secure them because of the risk requirements and compliance requirements of being a regulated bank, those will get in the way. Sometimes I'm explaining why those things are there. Sometimes I'm agreeing with people. “Yeah, it sucks. I don't want to have to do this.”But at the same time, you'll have people who they just want to come in, write their code, go home. They don't want to think about technology other than that. They're not going to go and learn things on their own necessarily. And that's not the end of the world. As strange as that sounds to people who are the personality to be constantly learning and constantly getting into everything and tinkering, like, that's me too, but you still need people to keep the lights on, to do all of the other work as well. And people who are happy just doing that, that's also valuable.Because if I was in that role, I would not be happy. And someone who is happy, like, this is good for the overall organization. But the things that they need to learn, the things they need explained to them, the help they need for success is different. So, that's one of the challenges is figuring out how do you address all of those customers? And sometimes even the answer for those customers is—and this is one of the things about my role—it's like the definition is customer success.If the application you're trying to put on cloud should not go on cloud, it is my job to tell you not to put it on cloud. It is not my job to put you on cloud. I want you to succeed, not just to get there. I can get your thing on the cloud in an afternoon, probably, but if I then walk away and it breaks, like, you don't know what to do. So, a lot of the things around how do we teach people to self-serve, how do we make our internal systems more self-serve, those are kind of the things that I look at now.How do I manage my own time because the scope is so big? It's like, I need to figure out where I'm not moving a thousand things forward an inch, but I'm moving things to their completion. And I am learning to, while not managing people, still delegate and work with the community, work with the broader cloud platform group around how do I let go and help other people do things?Jason: So, you mentioned something in there that I think is really interesting, right, the goal of helping people get to completion, right? And I think that's such an interesting thing because I think as—in that advocacy role, there's often a notion of just, like, I'm going to help you get unstuck and then you can keep going, without a clear idea of where they're ultimately heading. And that kind of ties back into something that you said earlier about starting out as a developer where you build things and you kind of just, like, set it free, [laugh] and you don't think about, you know, that day two, sort of, operations, the maintenance, the ongoing kind of stuff. So, I'm curious, as you've progressed in your career, as you've gotten more wisdom from helping people out, what does that look like when you're helping people get to completion, also with the mindset of this is an application that's going to be running for quite some time. Even in the short term, you know, if it's a short-term thing, but I feel like with the bank, most things probably are somewhat long-lived. How do you balance that out? How do you approach that, helping people get to done but also keeping in mind that they have to—this app has to keep living and it has to be maintained?Aaron: Yeah, a lot of it is—like, the term we use is sustainable development. And part of that is kind of removing friction, trying to get the developers to a point where they can focus on, I guess, the term that's often used in the industry is their inner loop. And it should come as no surprise, the bank often has a lot of processes that are high in friction. There's a lot of open a ticket, wait for things. This is the part that I take my conversations with dev teams, and I ask them, “What are the things that are hard? What are the things you don't like? What are the things you wish you didn't have to do or care about?”And some of this is reading between the lines when you talk to them; it's not so much interviewing them. Like, any kind of requirements gathering, usually, it's not what they say, it's what they talk about that then you look at, oh, this is the problem; how do we unstuck that problem so that people can get to where they need to be going? And this kind of informs some of my feedback to the systems we put in place, the processes we put in place around the platform, some of the tooling we look at. I really, really love the philosophy from Docker and Solomon Hykes around, “Batteries included but removable.” I want developers to have a high baseline as a starting point.And this comes partly from my experience with Cloud Foundry. Cloud Foundry has a really great out-of-the-box dev experience for lots of things where, “I just have a web app. Just run it. It's Nginx; it's some HTML pages; I don't need to know all the details. Just make it go and give me the URL.”And I want more of that for app teams where they have a high baseline of things to work with as a starting point. And kind of every organization ends up building this, where they have—like, Netflix: Netflix OSS or Twitter with Finagle—where they have, “Here's the surrounding pieces that I want to plug in that everybody gets as a starting point. And how do we provide security? How do we provide all of these pieces that are major concerns for an app team, that they have to do, we know they have to do?” Some of these are things that only start coming up when they're on the cloud and trying to provide a lot more of that for app teams so they can focus on the business stuff and only get into the weeds when they need to.Jason: As you're talking about these frameworks that, you know, having this high quality or this high baseline of tools that people can just have, right, equipping them with a nice toolbox, I'm guessing that the innersource stuff that you're working on also helps contribute to that.Aaron: Oh, immensely. And as we've gone on and as we've matured, our innersource organization, a huge part of that is other groups as well, where they're finding things that—we need this. And they'll put—it originally it was, “We built this. We'll put it into innersource.” But what you get with that is something that is very targeted and specific to their group and maybe someone else can use it, but they can't use it without bending it a little bit.And I hate bending software to fit it. That's one of the things—it's a very common thing in the corporate environment where we have our existing processes and rather than adopting the standard approach that some tool uses, we need to take it and then bend it until it fits our existing process because we don't want to change our processes. And that gets hard because you run into weird edge cases where this is doing something strange because we bent it. And it's like, well, that's not its fault at that point. As we've started doing more innersource, a lot more things have really become innersource first, where groups realize we need to solve this together.Let's start working on it together and let's design the API as a group. And API design is really, really hard. And how do we do things with shared libraries or services. And working through that as a group, we're seeing more of that, and more commonly things where, “Well, this is a thing we're going to need. We're going to start it in innersource, we'll get some people to use it and they'll be our beta customers. And we'll inform it without really specifically targeting an application and an app team's needs.”Because they're all going to have specific needs. And that's where the, like, ‘included but removable' part comes in. How do we build things extensibly where we have the general solution and you can plug in your specifics? And we're still—like, this is not an easy problem. We're still solving it, we're still working through it, we're getting better at it.A lot of it's just how can we improve day-over-day, year-over-year, to make some of these things better? Even our, like, continuous integration and delivery pipelines to our to clouds, all of these things are in constant flux and constant evolution. We're supporting multiple languages; we're supporting multiple versions of different languages; we're talking about, hey, we need to get started adopting Java 17. None of our libraries or pipelines do that yet, but we should probably get on that since it's been out for—what—almost a year? And really working on kind of decomposing some of these things where we built it for what we needed at the time, but now it feels a bit rigid. How do we pull out the pieces?One of the big pushes in the organization after the log4j CVE and things like that broad impact on the industry is we need to do a much more thorough job around software supply chain, around knowing what we have, making sure we have scans happening and everything. And that's where, like, the pipeline work comes in. I'm consulting on the pipeline stuff where I provide a lot of customer feedback; we have a team that is working on that all full time. But doing a lot of those things and trying to build for what we need, but not cut ourselves off from the broader industry, as well. Like, my nightmare situation, from a tooling standpoint, is that we restrict things, we make decisions around security, or policy or something like that, and we cut ourselves off from the broader CNCF tooling ecosystem, we can't use any of those tools. It's like, well, now we have to build something ourselves, or—which we're never going to do it as well as the external community. Or we're going to just kind of have bad processes and no one's going to be happy so figuring out all of that.Jason: Yeah. One of the things that you mentioned about staying up to speed and having those standards reminds me of, you know, similar to that previous experience that I had was, basically, I was at an org where we said that we'd like to open-source and we used open-source and that basically meant that we forked things and then made our own weird modifications to it. And that meant, like, now, it wasn't really open-source; it was like this weird, hacked thing that you had to keep maintaining and trying to keep it up to date with the latest stuff. Sounds like you're in a better spot, but I am curious, in terms of keeping up with the latest stuff, how do you do that, right? Because you mentioned that the bank, obviously a bit slower, adopting more established software, but then there's you, right, where you're out there at the forefront and you're trying to gather best practices and new technologies that you can use at the bank, how do you do that as someone that's not building with the latest, greatest stuff? How do you keep that skills and that knowledge up to date?Aaron: I try to do reading, I try to set time aside to read things like The New Stack, listen to podcasts about technologies. It's a really broad industry; there's only so much I can keep up with. This was always one of the conversations going way back where I would have the conversation with my boss around the business proposition for me going to conferences, and explaining, like, what's the cost to acquire knowledge in an organization? And while we can bring in consultants, or we can hire people in, like, when you hire new people in, they bring in their pre-existing experiences. So, if someone comes in and they know Hadoop, they can provide information and ideas around is this a good problem to solve with Hadoop? Maybe, maybe not.I don't want to bet a project on that if I don't know anything about Hadoop or Kubernetes or… like, using something like Tilt or Skaffold with my tooling. That's one of the things I got from going to conferences, and I actually need to set more time aside to watch the videos now that everything's virtual. Like, not having that dedicated week is a problem where I'm just disconnected and I'm not dealing with anything. When you're at work, even if KubeCon's going on or Microsoft Build, I'm still doing my day-to-day, I'm getting Slack messages, and I'm not feeling like I can just ignore people. I should probably block out more time, but part of how I stay up to date with it.It's really doing a lot of that reading and research, doing conversations like this, like, the DX Buzz that we invited you to where… I explained that event—it's adjacent to internal speakers—I explained that as I was had a backlog of videos from conferences I was not watching, and secretly if I make everybody else come to lunch with me to watch these videos, I have to watch the video because I'm hosting the session to discuss it, and now I will at least watch one a month. And that's turned out to be a really successful thing internally within the organization to spread knowledge, to have conversations with people. And the other part I do, especially on the tooling side, is I still build stuff. As much as, like, I don't code nearly as much as I used to, I bring an application developer perspective, but I'm not writing code every day anymore.Which I always said was going to be the thing that would make me miserable. It's not. I still think about it, and when I do get to write code, I'm always looking for how can I improve this setup? How can I use this tool? Can I try it out? Is this better? Is this smoother for me so I'm not worrying about this thing?And then spreading that information more broadly within the developer experience group, our DevOps teams, our platform teams, talking to those teams about the things that they use. Like, we use Argo CD within one group and I haven't touched it much, but I know they've got lots of expertise, so talking to them. “How do you use this? How is this good for me? How do I make this work? How can I use it, too?”Jason: I think it's been an incredible, [laugh] as you've been chatting, there are so many different tools and technologies that you've mentioned having used or being used at the bank. Which is both—it's interesting as a, like, there's so much going on in the bank; how do you manage it all? But it's also super interesting, I think, because it shows that there's a lot of interest in just finding the right solutions and finding the right tools, and not really being super-strongly married to one particular tool or one set way to do things, which I think is pretty cool. We're coming up towards the end of our time here, so I did want to ask you, before we sign off, Aaron, do you have anything that you'd like to plug, anything you want to promote?Aaron: Yeah, the Cloud Program is hiring a ton. There's lots of job openings on all of our platform teams. There's probably job openings on my Cloud Adoption Team. So, if you think the bank sounds interesting—the bank is very stable; that's always one of the nice things—but the bank… the thing about the bank, I originally joined the bank saying, “Oh, I'll be here two years, and I'll get bored and I'll leave,” and now it's been 12 years and I'm still at the bank. Because I mentioned, like, that scope and scale of the organization, there's always something interesting happening somewhere.So, if you're interested in cloud platform stuff, we've got a huge cloud platform. If you're in—like, you want to do machine-learning, we've got an entire organization. It should come as no surprise, we have lots of data at a bank, and there's a whole organization for all sorts of different things with machine-learning, deep learning, data analytics, big data, stuff like that. Like, if you think that's interesting, and even if you're not specifically in Toronto, Canada, you can probably find an interesting role within the organization if that's something that turns your crank.Jason: Awesome. We'll post links to everything that we've mentioned, which is a ton. But go check us out, gremlin.com/podcast is where you can find the show note for this episode, and we'll have links to everything. Aaron, thank you so much for joining us. It's been a pleasure to have you.Aaron: Thanks so much for having me, Jason. I'm so happy that we got to do this.Jason: For links to all the information mentioned, visit our website at gremlin.com/podcast. If you liked this episode, subscribe to the Break Things on Purpose podcast on Spotify, Apple Podcasts, or your favorite podcast platform. Our theme song is called, “Battle of Pogs” by Komiku, and it's available on loyaltyfreakmusic.com.

Today we chat with Cisco's head of developer content, community, and events, Michael Chenetz. We discuss everything from KubeCon to kindness and Legos! Michael delves into some of the main themes he heard from creators at KubeCon, and we discuss methods for increasing adoption of new concepts in your organization. We have a conversation about attending live conferences, COVID protocol, and COVID shaming, and then we talk about how Legos can be used in talks to demonstrate concepts. We end the conversation with a discussion about combining passions to practice creativity. We discuss our time at KubeCon in Spain (5:51) Themes Michael heard at KubeCon talking with creators (7:46) Increasing adoption of new concepts (9:27) We talk conferences, COVID shaming, and blamelessness (12:21) Legos and reliability  (18:04) Michael talks about ways to exercise creativity (23:20) Links: KubeCon October 2022: https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/ Nintendo Lego Set: https://www.amazon.com/dp/B08HVXMQ87?ref_=cm_sw_r_cp_ud_dp_ED7NVBWPR8ANGT8WNGS5 Cloud Unfiltered podcast episode featuring Julie and Jason:https://podcasts.apple.com/us/podcast/ep125-chaos-engineering-with-julie-gunderson-and-jason/id1215105578?i=1000562393884 Links Referenced: Cisco: https://www.cisco.com/ Cloud Unfiltered Podcast with Julie and Jason: https://podcasts.apple.com/us/podcast/ep125-chaos-engineering-with-julie-gunderson-and-jason/id1215105578?i=1000562393884 Cloud Unfiltered Podcast: https://www.cisco.com/c/en/us/solutions/cloud/podcasts.html Nintendo Lego: https://www.amazon.com/dp/B08HVXMQ87 TranscriptJulie: And for folks that are interested in, too, what day it is—because I think we're all still a little bit confused—it is Monday, May 24th that we are recording this episode.Jason: Uh, Julie's definitely confused on what day it is because it's actually Tuesday, [laugh] May 24th.Michael: Oh, my God. [laugh]. That's great. I love it.Julie: Welcome to Break Things on Purpose, a podcast about reliability, learning from each other, and blamelessness. In this episode, we talk to Michael Chenetz, head of developer content, community, and events at Cisco, about all of the learnings from KubeCon, the importance of being kind to each other, and of course, how Lego translates into technology.Julie: Today, we are joined by Michael Chenetz. Michael, do you want to tell us a little bit about yourself?Michael: Yeah. [laugh]. Well, first of all, thank you for having me on the show. And I'm really good at breaking things, so I guess that's why I'm asked to be here is because I'm superb at it. What I'm not so good at is, like, putting things back together.Like when I was a kid, I remember taking my dad's stereo apart; wasn't too happy about that. Wasn't very good at putting it back together. But you know, so that's just going back a little ways there. But yeah, so I work for the DevRel at Cisco and my whole responsibility is, you know, to get people to know that know a little bit about us in terms of, you know, all the developer-related topics.Julie: Well, and Jason and I had the awesome opportunity to hang out with you at KubeCon, where we got to join your Cloud Unfiltered podcast. So folks, definitely go check out that episode. We have a lot of fun. We'll put a link in the [show notes 00:02:03]. But yeah, let's talk a little bit about KubeCon. So, as of recording this episode, we all just recently traveled back from Spain, for KubeCon EU, which was… amazing. I really enjoyed being there. My first time in Spain. I got back, I can tell you, less than 24 hours ago. Michael, I think—when did you get back?Michael: So, I got back Saturday night, but my bags have not arrived yet. So, they're still traveling and they're enjoying Europe. And they should be back soon, I guess when they're when they feel like they're—you know, they should be back from vacation.Julie: [laugh].Michael: So. [laugh].Julie: Jason, how about you? When did you get home?Jason: I got home on Sunday night. So, I took the train from Valencia to Barcelona on Saturday evening, and then an early morning flight on Sunday and got home late Sunday night.Julie: And for folks that are interested in, too, what day it is—because I think we're all still a little bit confused—it is Monday, May 24th that we are recording this episode.Jason: Uh, Julie's definitely confused on what day it is because it's actually Tuesday, [laugh] May 24th.Michael: Oh, my God. [laugh]. That's great. I love it. By the way, yesterday was my birthday so I'm going to say—Julie: Happy birthday.Michael: —happy birthday to myself.Julie: Oh, my gosh, happy birthday. [laugh].Michael: Thank you [laugh].Julie: So… what is time anyway?Jason: Yeah.Michael: It's all good. It's all relative. Time is relative.Julie: Time is relative. And so, you know, tell us a little bit about—I'd love to know a little bit about why you want folks to know about, like, what is the message you try to get across?Jason: Oh, that's not the question I thought you were going to ask. I thought you were going to ask, “What's on your Amazon wishlist so people can send you birthday presents?”Julie: Yeah, let's back up. Let's do that. So, let's start with your Amazon wishlist. We know that there might be some Legos involved.Michael: Oh, my God, yeah. I mean, you just told me about a cool one, which was Optimus Prime and I just—I'm already on the website, my credit card is out and I'm ready to buy. So, you know, this is the problem with talking to you guys. [laugh]. It's definitely—you know, that's definitely on my list. So, anything that, anything music-related because obviously behind me is a lot of music equipment—I love music stuff—and anything tech. The combination of tech and music, and if you can combine Legos and that, too, man that would just match all the boxes. [laugh].Julie: Just to let you know, there's a Lego Con. Like, I did not know this until last night, actually. But it is a virtual conference.Michael: Really.Julie: Yeah. But one of the things I was looking at actually on Lego, when you look at their website, like, to request one of their speakers, to request one of their engineers as a speaker, they actually don't do that because they get so many requests for their folks to speak at conferences, they actually have a dedicated part of their website that talks about this. So, I thought that was interesting.Michael: Well listen, just because of that, if they want somebody that's in, you know, cloud computing, I'm not going to go talk for Lego. And I know they really want somebody from cloud computing talking to Lego, so, you know… it's, you know, quid pro quo there, so that's just the way it's going to work. [laugh].Julie: I want to be best friends with Lego people.Michael: [laugh]. I know, me too.Julie: I'm just going to make it a goal in life now to have one of their engineers speak at DevOpsDays Boise. It's like a challenge.Michael: It is. I accept it.Julie: [laugh]. With that, though, just on other Lego news, before we start talking about all the other things that folks may also want to hear about, there is another new Lego, which is the Van Gogh Starry Night that has been newly released by the time this episode comes out.Michael: With a free ear, right?Julie: I mean—[laugh].Michael: Is that what happens?Julie: —well played. Well, played. [laugh]. So, now you really got to spend a lot of time at KubeCon, you were just really recording podcast after podcast.Michael: Oh, my God. Yeah. So, I mean, it was great. I love—because I'm a techie, so I love tech and I love to find out origin stories of stuff. So, I love to, like, talk to these people and like, “Why did that come about? How did—” you know, “What happened in your life that made you want to do this? Who hurt you?” [laugh].And so, that's what I constantly try and figure out is, like, [laugh], “What is that?” So, it was really cool because I had, like, Jimmy Zelinskie who came from CoreOS, and he came from—you know, they create, you know, Quay and some of this other kinds of stuff. And you know, just to talk about, like, some of the operators and how they came about, and like… those were the original operators, so that was pretty cool. Varun from Tetrate was supposed to come on, and he created Istio, you know? So, there were so many of these things that I just geek out knowing about, you know?And then the other thing that was really high on our list, and it's really high from where I am, is API quality, API testing, API—so really, that's why I got in touch with you guys because I was like, “Wow, that fits in really good, you know? You guys are doing stuff that's around chaos, and you know, I think that's amazing.” So, all of this stuff is just so interesting to me. But man, it was just a whirlwind of every day just recording, and by the end that was just like, you know, “I'm so sorry, but I just, I can't talk anymore.” You know, and that was it. [laugh].Jason: I love that chatting with the creators. We had Zack Butcher on who is also from Tetrate and one of the early Istio—Michael: Yeah, yeah.Jason: Contributors. And I find it fascinating because I feel like when you chat with these folks, you start to understand the context of why things were built. And it—Michael: Yes.Jason: —it opens your brain up to, like, cool, there's a software—oh, now I know exactly why it's doing things that way, right? Like, it's just so, so eye-opening. I love it.Julie: With that, though, like, did you see any trends or any themes as you were talking to all these folks?Michael: Yeah, so a few real big trends. One is everybody wants to know about eBPF. That was the biggest thing at KubeCon, by far, was that, “We want to learn how to do this low-level kernel stuff that's really fast, that can give us all the information we need, and we don't have to use sidecars and things like that.” I mean it was—you know, that was the most excitement that I saw. OTel was another one for OpenTelemetry, which was a big one.The other thing was simplification. You know, a lot of people were looking to simplify the Kubernetes ecosystem because there's so much out there, and there's so many things that you have to learn about that it was super hard, you know, for somebody to come into it to say, “Where do I even start?” You know? So, that was a big theme was simplification.I'm trying to think. I think another one is APIs, for sure. You know, because there's this whole thing about API sprawl. And people don't know what their APIs are, people just, like—you know, I always say people can see—like, developers are lazy in a good way, and I consider myself one of them. So, what that means is that when we want to develop something, what we're going to do is we're just going to pull down the nearest API that does what we need, that has the best documentation, that has the best blog, that has the best everything.We don't know what their testing strategy is; we don't know what their security strategy is; we don't know if they use other libraries. And you have to figure that stuff out. And that's the thing that—you know, so everything around APIs is super important. And you really have to test that stuff out. Yes, people, you have to test it [laugh] and know more about it. So, those are those were the big themes, I think. [laugh].Julie: You know, I know that Kerim and I gave a talk on observability where we kind of talked more high-level about some of the overarching concepts, but folks were really excited about that. I think is was because we briefly touched on OpenTelemetry, which we should have gone into a little bit more depth, but there's only so much you can fit into a 30-minute talk, so hopefully we'll be able to talk about that more at a KubeCon in the future, we [crosstalk 00:09:54] to the selection committee.Michael: Hashtag topics?Julie: Uh-huh. [laugh]. You know, that said, though, it really did seem like a huge topic that people just wanted to learn more about. I know, too, at the Gremlin booth, a lot of folks were also interested in talking about, like, how do we just get our organization to adopt some of these concepts that we're hearing about here? And I think that was the thing that surprised me the most is I expected people to be coming up to the booth and deep-diving into very, very deep, technical-level questions, and really, a lot of it was how do we get our organization to do this? How can we increase adoption? So, that was a surprise for me.Michael: Yeah, you know what, and I would say two things to that. One is, when you talk about Chaos Engineering, I think people think it's like rocket science and people are really scared and they don't want to claim to be experts in it, so they're like, “Wow, this is, like, next-level stuff, and you know, we're really scared. You guys are the experts. I don't want to even attempt this.” And the other thing is that organizations are scared because they think that it's going to, like, create mass hysteria throughout their organization.And really, none of this is true in either way. In reality, it's a very, very scripted, very exacting stuff that you're testing, and you throw stuff out there and see what kind of response you get. So, you know, it's not this, like, you know—I think people just have—there needs to be more education around a lot of areas in cloud-native. But you know, that's one of the areas. So, I think it's really interesting there.Julie: I think so too. How about for you, Jason? Like, what was your surprise from the conference or something that maybe—Jason: Yeah, I mean, I think my surprise was mostly around just seeing people coming back, right? Because we're now I would say, six months into conferences being back as a thing, right? Like, we had re:Invent last year in Vegas; we had KubeCon last year in LA, and so, like, those are okay events. They weren't, like, back to normal. And this was, I feel like, one of the first conferences, that it really started to feel back to normal.Like, there was much better attendance, there was much more just buzz and hallway tracking and everything else that we're used to. Like, the whole reason that we go to conferences is getting together with people and hanging out and stuff, and this one has so far felt the most back-to-normal out of any event that I've been to over the past six months.Michael: Can I just talk about one thing that I think, you know, people have to get over is, you know, I see a lot online, I think it was—I forget who it was that was talking about it. But this whole idea of Covid shaming. I mean, we're going to this event, and it's like, yeah, everybody wants to get out, everybody wants to learn things, but don't shame people just because they got Covid, everybody's getting Covid, okay? That's just the point of life at this point. So, let's just, you know, let's just be nice to each other, be friendly to each other, you know? I just have to say that because I think it's a shame that people are getting shamed, you know, just for going to an event. [laugh].Julie: See, and I think that—that's an interesting—there's been a lot of conversation around this. And I don't think anybody should be Covid-shamed. Look, I think that we all took a calculated risk in coming—Michael: Absolutely.Julie: To this event. I personally gave out a lot of hugs. I hugged some of the folks that have mentioned that they have come up positive from Covid, so there's a calculated risk in going. I think there has been a little bit of pushback on maybe how some of the communication has come out around it. That said, as an organizer of a small conference with, like, 400 people, I think that these are very complicated matters. And what I really think is important is to listen to feedback from attendees and to take that.And then we're always looking to improve, right?Michael: Absolutely.Julie: If everything that we did was perfect right out of the gate, then we wouldn't have Chaos Engineering because there'd be nothing [crosstalk 00:13:45] be just perfectly reliable. And so, if we take away anything, let's take away—just like what you said, first of all, Covid, you should never shame somebody for having Covid. Like, that's not cool. It's not somebody's fault that they caught an illness.Michael: Yes.Julie: I mean unless they were licking doorknobs. And that's a whole different—Michael: Yes. [laugh]. That's a whole different thing, right there.Julie: Conversation. But when we talk about just like these questions around cultural adoption, we talk about blamelessness; we talk about learning from failure; we talked about finding ways to improve, and I think all of that can come into play. So, it'll be interesting to see how we learn and grow as we move forward. And like, thank you to re:Invent, thank you to KubeCon, thank you to DevOpsDays Boise. But these conferences that have started going back in-person, at great risk to organizers and the committee because people are going to be mad, one way or the other.Michael: Yeah. And you can see that people want to be back because it was huge, you know?Julie: Yeah.Michael: Maybe you guys, I'm going to put in a feature request for Gremlin to chaos engineer crowds. Can we do that so we can figure out, like, what's going to happen when we have these big events? Can we do that?Julie: I mean, that sounds fun. I think what's going to happen is there's going to be hugs, there's going to be people getting sick, but there's going to be people learning and growing.Michael: Yes.Julie: And ultimately, I just think that we have to remember that just, like, our systems aren't perfect, and neither are people. Like, the fact that we expect people to be perfect, and maybe we should just keep some mask mandates for a little bit longer when we're at conferences with 8000 people.Michael: Sure.Julie: I mean, that's—Michael: That makes sense.Jason: Yeah. I mean, it's all about risk management, right? This is, essentially what we do in SRE is there's always a risk of a massive outage, and so it's that balance of, right, do what you can, but ultimately, that's why we have SLOs and things is, you can never be a hundred percent, so like, where do we draw the line of here are the things that we're going to do to help manage this risk, but you can never shoot for a perfectly, entirely safe space, right? Because then we'd all be having conferences in padded rooms, and not touching each other, and things like that. There's a balance there.And I think we're all just trying to find that, so yeah, as you mentioned, that whole, like, DevOps blamelessness thing, you know, treat each other with the notion that we're all trying to get through this together and do what we think is best. Nobody's just like John Allspaw said, you know, “Nobody goes to work thinking that, like, their intent is to crash everything and destroy the company.” No one's going to KubeCon or any of these conferences thinking, “Yeah, I'm going to be a super-spreader.”Julie: [laugh].Michael: Yeah, that would be [crosstalk 00:16:22].Jason: Like, everyone's trying not to do it. They're doing their best. They're not actively, like, aggressively trying to get you sick or intentionally about it. But you know—so just be kind to one another.Michael: Yeah. And that's the key.Julie: It is.Michael: The key. Be kind to one another, you know? I mean, it's a great community. People are really nice, so, you know, let's keep that up. I think that's something special about the, you know, the community around KubeCon, specifically.Julie: As we can refine this and find ways, I would take all of the hugs over virtual conferences—Michael: Yes.Julie: Any day now. Because, as Jason mentioned, is even just with you, Michael, the time we got to spend with you, or the time I kept going up to Jfrog's booth and Baruch and I would have conversations as he made me a delicious coffee, these hallway tracks, these conversations, that's what no one figured out how to recreate during the virtual events—Michael: Absolutely.Julie: —and it's just not possible, right?Michael: Yeah. I mean, I think it would take a little bit of VR and then maybe some, like, suit that you wear in order to feel the hug. And, you know, so it would take a lot more in order to do that. I mean, I guess it's technologically possible. I don't know if the graphics are there yet, so it might be like a pixelated version, like, you know, like, NES-style, or something like that. But it could look pretty cool. [laugh]. So, we'll have to see, you know?Julie: Everybody listening to this episode, I hope you're getting as much of a kick out of it as we are recording it because I mean, there are so many different topics here. One of the things that Michael and I bonded about years ago, for our listeners that are—not years ago; months ago. Again, what is time?Michael: Yeah. What is time? It's all relative.Julie: It is. It was Lego, though, and so we've been talking about that. But Michael, you asked a great question when we were recording with you, which is, like—Michael: Wow.Julie: Can—just one. Only one great question.Michael: [laugh].Julie: [laugh]. Which was, how would you incorporate Lego into a talk? And, like, when we look at our systems breaking and all of that, I've really been thinking about that and how to make our systems more reliable. And here's one of the things I really wanted to clarify that answer. I kind of went… I went talking about my Lego that I build, like, my Optim—not my Optimus Primes, I don't have it, but my Voltron or my Nintendo Lego. And those are all box sets.Michael: Yep.Julie: But one of the things if you're not playing with a box set with instruction, if you're just playing with just the—or excuse me, architecting with just the Lego blocks because it's not playing because we're adults now, I think.Michael: Yes, now it's architecting. Yes.Julie: Yes, now that we're architecting, like, that's one of the things that I was really thinking about this, and I think that it would make something really fun to talk about is how you're building upon each layer and you're testing out these new connection pieces. And then that really goes into, like, when we get into Technics, into dependencies because if you forget that one little one-inch plastic piece that goes from the one to the other, then your whole Lego can fall apart. So anyway, I just thought that was really interesting, and I'd wondered if you or Jason even gave that any more thought, or if it was just fleeting for you.Michael: It was definitely fleeting for me, but I will give it some more thought, you know? But you know, when—as you're saying that though, I'm thinking these Lego pieces really need names because you're like that little two-inch Lego piece that kind of connects this and this, like, we got to give these all names so that people can know, that's x-54 that's—that you're putting between x-53 and x-52. I don't know but you need some kind of name for these parts now.Julie: There are Lego names. You just Google it. There are actual names for all of the parts but—Michael: Wow. [laugh].Julie: Like, Jason, what do you think? I know you've got [unintelligible 00:19:59].Jason: Yeah, I mean, I think it's interesting because I am one of those, like, freeform folks, right? You know, my standard practice when I was growing up with Legos was you build the thing that you bought once and then you immediately, like, tear it apart, and you build whatever the hell you want.Michael: Absolutely.Jason: So, I think that that's kind of an interesting thing as we think about our systems and stuff, right? Like, part of it is, like, yeah, there's best practices and various companies will publish, like, you know, “Here's how to architect such-and-such system.” And it's interesting because that's just not reality, right? You're not going to go and take, like, the Amazon CloudFormation thing, and like, congrats, you're done. You know, you just implement that and your job's done; you just kick back for the rest of the week.It never works that way, right? You're taking these little bits of, like, cool, I might have, like, set that up once just to see what's happening but then you immediately, like, deconstruct it, and you take the knowledge of what you learned in those building blocks, and you, like, go and remix it to build the thing that you actually need to build.Michael: But yeah, I mean, that's exactly—so you know, Legos is what got me interested in that as a kid, but when you look at, you know, cloud services and things like that, there's so many different ways to combine things and so many different ways to, like—you know, you could use Terraform, you could use Crossplane, you could use, you know, any of the services in the cloud, you could use FaaS, you could use serverless, you could use, you know, all these different kinds of solutions and tie them together. So, there's so much choice, and what Lego teaches you is that, embrace the choice. Figure out and embrace the different pieces, embrace all the different things that you have and what the art of possibility is, and then start to build on that. So, I think it's a really good thing. And that's why there's so much correlation between, like, kind of, art and tech and things like that because that's the kind of mentality that you need in order to be really successful in tech.Jason: And I think the other thing that works really well with what you said is, as you're playing with Legos, you start to learn these hacks, right? Like, I don't have, like, a four-by-one brick, but I know that if I have three four-by-one flats, I can stack those three and it's the same height as a brick, right?Michael: Yep.Jason: And you can start combining things. And I love that engineering mentality of, like, I have this problem that I need to solve, I have a limited toolbox for whatever constraints, right, and understanding those constraints, and then cool, how can I remix what I've got in my toolbox to get this thing done?Michael: And that's a thing that I'm always doing. Like, when I used to do a lot of development, you know, it was always like, what is the right code? Or what is the library that's going to solve my problem? Or what is the API that's going to solve my problem, you know?And there's so many different ways to do it. I mean, so many people are afraid of, like, making the wrong choice, when really in programming, there is no wrong choice. It's all about how you want to do it and what makes sense to you, you know? There might be better options in formatting and in the way that you kind of, you know, format that code together and put them in different libraries and things like that, but making choices on, like, APIs and things like that, that's all up to the artist. I would say that's an artist. [laugh]. So, you know, I think it all stems though, when you go back from, you know, just being creative with things… so creativity is king.Jason: So Michael, how do you exercise your creativity, then? How do you keep up that creativity?Michael: Yeah, so there's multiple ways. And that's a great segment because one of the things that I really enjoy—so you know, I like development, but I'm also a people person. And I like product management, but I also like dealing with people. So really, to me, it's about how do I relate products, how do I relate solutions, how do I talk to people about solutions that people can understand? And that's a creative process.Like, what is the right media? What is the right demos? What is the right—you know, what do people need? And what do people need to, kind of, embrace things? And to me, that's a really creative medium to me, and I love it.So, I love that I can use my technical, I love that I can use my artistic, I love that I can use, you know, all these pieces all at once. And sometimes maybe I'll play guitar and just put it in the intro or something, I don't know. So, that kind of combines that together, too. So, we'll figure that piece out later. Maybe nobody wants to hear me play guitar, that's fine, too. [laugh].But I love to be able to use, you know, both sides of my brain to do these creative aspects. So, that's really what does it. And then sometimes I'll program again and I'll find the need, and I'll say, “Hey, look, you know, I realized there's a need for this,” just like a lot of those creators are. But I haven't created anything cool, but you know, maybe someday I will. I feel like it's just been in between all those different intersections that's really cool.Jason: I love the electric guitar stuff that you mentioned. So, for folks who are listening to this show, during our recording of the Cloud Unfiltered you were talking about bringing that art and technical together with electric guitars, and you've been building electric guitar pickups.Michael: Yes. Yeah. So, I mean, I love anything that can combine my music passion with tech, so I have a CNC machine back here that winds pickups and it does it automatically. So, I can say, “Hey, I need a 57 pickup, you know, whatever it is,” and it'll wind it to that exact spec.But that's not the only thing I do. I mean, I used to design control surfaces for artists that were a big band, and I really can't—a lot of them I can't mention because we're under NDA. But I designed a lot of these big, you know, control surfaces for a lot of the big electronic and rock bands that are out there. I taught people how to use Max for Live, which is an artist's, kind of, programming language that's graphical, so [NMax 00:25:33] and MSP and all that kind of stuff. So, I really, really like to combine that.Nowadays, you know, I'm talking about doing some kind of events that may be combined tech, with art. So, maybe doing things like Algorave, and you know, things that are live-coding music and an art. So, being able to combine all these things together, I love that. That's my ultimate passion.Jason: That is super cool.Julie: I think we have learned quite a bit on this episode of Break Things on Purpose, first of all, from the guy who said he hasn't created much—because you did say that, which I'm going to call you out on that because you just gave a long list of things that you created. And I think we need to remember that we're all creators in our own way, so it's very important to remember that. But I think that right now we've created a couple of options for talks in the future, whether or not it's with Lego, or guitar pickups.Michael: Yeah.Julie: Is that—Michael: Hey—Julie: Because I—Michael: Yeah, why not?Julie: —know you do kind of explain that a little bit to me as well when I was there. So, Michael, this has just been amazing having you. We're going to put a lot of links in the notes for everybody today. So, to Michael's podcast, to some Lego, and to anything else Michael wants to share with us as well. Oh, real quick, is there anything you want to leave our listeners with other than that? You know, are you looking to hire Cisco? Is there anything you wanted to share with us?Michael: Yeah, I mean, we're always looking for great people at Cisco, but the biggest thing I'd say is, just realize that we are doing stuff around cloud-native, we're not just network. And I think that's something to note there. But you know, I just love being on the show with you guys. I love doing anything with you guys. You guys are awesome, you know. So.Julie: You're great too, and I think we'll probably do more stuff, all of us together, in the future. And with that, I just want to thank everybody for joining us today.Michael: Thank you. Thanks so much. Thanks for having me.Jason: For links to all the information mentioned, visit our website at gremlin.com/podcast. If you liked this episode, subscribe to the Break Things on Purpose podcast on Spotify, Apple Podcasts, or your favorite podcast platform. Our theme song is called, “Battle of Pogs” by Komiku, and it's available on loyaltyfreakmusic.com.

At DoorGrow, we have some of the savviest property entrepreneurs on the planet in our DoorGrow and Scale Mastermind. Brannon Potts is a property management business owner in North Texas, who joined DoorGrow with only 71 doors. In only 3 months, Brannon was able to grow his business to over 100 doors with 70 more on the way! Join property management growth expert, Jason Hull, as he interviews Brannon Potts, a DoorGrow client. Brannon shares his experience with DoorGrow and how he has seen it make a beneficial impact on his business. You'll Learn… [01:12] Meet DoorGrow client Brannon Potts [04:42] Investing in Yourself and Your Business with Coaching [07:27] What Makes Jason and DoorGrow Different? [13:09] DoorGrow's Two Key Ideas… [20:07] Finding Fulfillment by Growing and Scaling the Business [22:49] How Brannon used DoorGrow's Script to Add Doors [27:08] How You can Grow and Scale Your Business  Tweetables “We have these moments as coaches where we feel like-- it's similar to as being a dad and seeing your kid get an award or do something.” “As you've been building your business, it could get uglier and more painful, but we always try to make sure that the client understands that's the wrong way to do it.” “Good, coaching or good marketing or good anything that you're going to pay for should give you an ROI, right? That means it's a good investment.” “A lot of people are thinking ‘I'd rather just spend money. I'd rather just spend money because it would save me time,' That's a cost. That's not an investment.” Resources DoorGrow and Scale Mastermind DoorGrow Academy DoorGrow on YouTube DoorGrowClub DoorGrowLive TalkRoute Referral Link Transcript [00:00:00] Jason: For those that are on the fence, thinking about DoorGrow maybe they've heard about DoorGrow, what would you say?  [00:00:05] Brannon: You might not like this, but I think it's so good, sometimes I wouldn't want to tell anybody cause it's so good for people.  [00:00:12] Jason: All right. Welcome, DoorGrow Hackers to the #DoorGrowShow. If you are a property management entrepreneur that wants to add doors, make a difference, increase revenue, help others, impact lives, and you're interested in growing in business and life, And you're open to doing things a bit differently, then you are a DoorGrow Hacker. DoorGrow Hackers love the opportunities, daily variety, unique challenges, and freedom that property management brings. Many in real estate think you're crazy for doing it. You think they're crazy for not because you realize that property management is the ultimate high trust gateway to real estate deals, relationships, and residual income. [00:00:49] At DoorGrow, we are on a mission to transform property management, business owners and their businesses. We want to transform the industry, eliminate the BS, build awareness, change perception, expand the market and help the best property management entrepreneurs win. I'm your host property management growth expert, Jason Hull, the founder and CEO of DoorGrow. Now let's get into the show.  [00:01:12] And my guest today is one of my clients, Brannon Potts. Brannon, welcome to the show.  [00:01:19] Brannon: Thank you. Thank  [00:01:20] Jason: you.  [00:01:21] So it's good to have you. So Brannon, you're a client that I really enjoy working with. 1. Because you just do what I tell you to do, and it works and you're doing the right things. So I appreciate you as a client because that's always fun for me is to have clients that like, believe in what we're doing, and get it. And do it. You know, starting out, why don't you tell everybody a little bit about how you got into property management in the first place?  [00:01:47] Brannon: Sure. I think it was back-- late 2016. We were actually doing well in our sales business but had a friend give us some advice about getting into property management and you say it, even in the intro: at first, I didn't have just a great perception of property management, but I said, okay, I'm going to get into this business, learn it. I didn't know it very much. I didn't know it really much at all. And we began to grow. And over these years, we just, we grew a little bit and happened to see one of your ads and started investigating and just said, you know what? I want to join this coaching. And I did. I just said, "I really don't know very much about the property management business, and you have a background in coaching this. I'm just going to follow what he says, not question it, and just do it."  [00:02:46] And then I was going to hold your feet to the fire because you promised it and committed that if I did, you would refund my money, and I began to do that and I still have so much more to go because in your coaching, the depth of that coaching that you give, I think I'm only maybe touching on 10% of it right now. And I'm looking forward to actually many years to come of getting deeper and deeper in implementing all the things that you provide in coaching because I've been coached before. I've been actually a coach myself for the sales side. And what we've always been taught is when you teach people, typically only 10% implement what you're teaching. And I said, man, "I don't want to be that 90% that doesn't. I want to be the 10% that does. And let the chips fall where they may and begin doing exactly what you said, trying to follow it as closely as I could." [00:03:44] And what do you know? It happens. We grow, we have probably at this point, we're either from the initial investment of the coaching-- I bet you we're-- I'm trying to think-- three, four times, maybe five times now the dollars that we're generating from the coaching. So it was a great investment. [00:04:04] Jason: You mean on a monthly basis?  [00:04:06] On a monthly basis. [00:04:07] Yeah. So you've got way more residual income than what the program costs. So it's a no-brainer. And that's one of our initial goals with clients. Like we want to get them paid and make sure the program's double paid for within hopefully the first 30 to 60 days is the goal so that they can justify the expense and keep going. And then I guess you could say it's paying you now, to be part of the program.  [00:04:27] Brannon: It's-- I've made money off of this coaching and that's what everybody wants.  [00:04:32] Jason: That's what good, coaching or good marketing or good anything that you're going to pay for should give you an ROI, right? That means it's a good investment. So I'm glad that you're getting a good return on your investment. That's our goal. So you brought up something that I think it's interesting that you've worked with a lot of coaches. I've worked with a lot of coaches too. You know, I think one thing that's a little bit different from me than maybe other coaches in the industry, but there's a lot of coaches out there that don't have coaches. And they don't get coached themselves. And I think that's one of my competitive advantages, which is really simple is that I pay for really expensive coaching and masterminds and high ticket things to be involved in so that I can turn around and have value to give to my clients.  [00:05:16] Like I just came back from a mastermind, I pay a lot of money to be in it, and I shared an idea today with the group that you thought was pretty cool. We were talking about not focusing on referrals instead of referring to them, as in asking for– what did I say? I'll let you say it... [00:05:32] Brannon: an introduction. And that is an extra benefit is not just taking the wisdom you already have. You're still pouring into yourself so that you have something to pour out. [00:05:45] Cause I think a lot of people stop getting things poured into them and the great people, great leaders, and great entrepreneurs need something poured into them so that they can pour out to others. We need that relationship to continue. And that's what I appreciate of your coaching. It's your coaching, but you're still getting coached and I'm getting the benefit of your expense of coaching and you're handing extra value to us. [00:06:14] Jason: Yeah. I'm not even going to say how much I spend on coaching a year right now, but it's a lot, it's a lot. I'm in two really high-ticket masterminds, but for me, I love it because I get to hang out with the best. Like I'm talking business owners that are doing millions and millions of dollars. I was hanging out with people that are doing millions a month in business. Some are hitting a million a month or more. And these are the kind of people I get to hang out with. And I love to be able to learn. It's fun for me. And having a program in which I get to share that stuff. That's just even more fun for me. Cause I love to share what I'm learning. That's just fun for me. So...  [00:06:51] Brannon: As soon as you brought that up, I went out and shared that with my team, the ones that were here at the moment, and then I'm going to share that again on another meeting of not asking for the referral, but asking for the introduction. There's just many layers to that, of how good it is.  [00:07:09] Jason: Yeah. Yeah. And we chatted about that on our group coaching call today. And for those that are not in our group, you're missing out. So Brannon what have you noticed since joining the program? I mean, You talked about some of the concerns you had coming in and some of the challenges you were dealing with, and you mentioned that you've made your money back, you've gotten some results. How does this compare or differ or relate to all the other coaching stuff that you've been involved in the past? [00:07:34] Brannon: Jason, I don't want you to get a big head, but it has been the best coaching I've had. And I talk about it all the time. I've been through several different coaches, both in the real estate side, and just some life coaches, and the value you bring is multiple layers and genuinely appreciate that, because that's what you teach us is bringing value to people and you do. And I would share this with anyone. This is not a sales pitch. This is true value. You bring on so many different layers and that's why I've shared in the past, and I've shared it even with my wife. I said, "I see myself. I've only scratched the surface of the value you've already brought. And I see this for many years to come that I plan to be a part of this. Cause it's not a cost, it's been an investment," and yes, we're talking about growing doors, but there's many other layers to the coaching of growing the business and how you do that from operations to people who you hire, what their duties are. This was exactly what I was looking for 'cause I did not have that knowledge. Though I've succeeded it at higher levels in a lot of ways in the real estate industry, these were the parts I didn't know. And I feel like I've still got so much more to learn. [00:09:01] Jason: I really appreciate that. That's-- that means a lot to me. I appreciate it. So for me, it's interesting to me because I've been in this business-- I founded it, and we had some tough times starting this business out and like me building a team I've gone through the entrepreneurial life cycle and journey that I coach clients on. And DoorGrow, our company has made so many changes, even in the last quarter. Like the slew of things that we get done that are on our list for quarterly planning is just amazing to me that we're able to accomplish. DoorGrow's not even the same company it was a year ago. Not even close. And some clients maybe worked with us in the past or knew about us in the past, or maybe we just did a website a long time ago. And DoorGrow is not even close to the same company. Some people are probably hearing you going "operations?" And like "this?" And like "coaching." And they're like, what? And it's funny because people, I think judge me and DoorGrow sometimes by who I was maybe five years ago or two years ago, or even a year ago. [00:10:06] And my personal development aggression, I guess you could say, or my drive and the level of the team members that I have and the drive of the team and how quickly we're able to make changes and implement is I mean, I'm obviously biased, but I think it's pretty amazing. So yeah. And I think people could give us a chance that haven't been with us for a while. Our new mastermind is just really awesome and I think people are really crushing it, which is really fun to see.  [00:10:32] Brannon: And the connections you make in that coaching group. There's several people that I've made connections with that will be valuable for the future and just collaborating or, "Hey, I'm having this struggle. How have you handled it?" There's just, there's so much value and I agree it's even in-- what is it? Five months now I've been in the program it's changed and added more layers to it. But that's because of your growth. You didn't stay stagnant. You're still growing yourself and have something to pour out. [00:11:07] Jason: I'd feel guilty if I took even the majority of the credit, like my operator, Sarah, also my fiance, she's moved the needle significantly in this business, Adam, over fulfillment, Ashlee who's over a client success-- like we've got some amazing people on my team and they're moving most of the objectives forward that we have each quarter, and it's been just awesome to see. And that's part of the DoorGrow OS planning system that we've got and that sort of thing. We've got 90 members in the mastermind, so I appreciate that you brought up community cause that's a big focus of ours moving forward this quarter is we're really trying to focus on improving the community aspect. A lot of people joined for the content in DoorGrow Academy and the material that we have and the ideas, and then the people win because of the coaching. But people generally in a program will stay because of the community and the connection and the benefits of having that comradery, 'cause you know, being an entrepreneur can be a lonely journey without being connected to others, so we're really focused on that, improving that, in fact, we've got about 90 members in the mastermind, 90 businesses. We probably have on average about two people per business. So we've got probably about somewhere close to 200 people in the program I think that are actively involved. [00:12:17] We haven't really grown honestly for the last, maybe two, three months, which is weird, but we've been filtering a lot of people out. We've been really trying to make sure people are active and engaged and shaking the tree, so to speak and some of the people that weren't really engaged or active in our outreach and stuff have dropped off while we've been adding people, but we've cleaned that up. So like the program's really clean. And so I'm really excited about the community aspect because most of the people now are all pretty much engaged. And in at least on one of the calls and doing their check-ins and moving forward. So I think even though the group is still about the same size it's been for a little while, it's a lot mightier, so we're really excited about that. And now that it's cleaned out, now we're going to be adding, I think a lot of people that are going to be staying in the program a lot longer and it'll be, even we're going to be growing for sure. Yeah. So Brannon, what do you feel like are some of the most significant things that you've got out of it so far? Because a lot of people, they hear you probably saying, "Hey, DoorGrow's great. Coaching's great." And you mentioned you're making more money. What are some of the key things that really have stood out to you that you're like, "Hey, like this is different or this is interesting," or that you've really valued? [00:13:29] Brannon: I think a couple of things. There are multiple layers, so I can talk about this for a while, but the key things initially were: how to lead generate that didn't cost money. And sometimes you hear that and you think, "oh, this is just a sales pitch." It was very genuine, and it was very good. And I implemented that. So the only cost was my time and following through with what the coaching did and that added the doors very quickly. The second was helping design a pricing plan and how to put that together. I implemented one of the plans, the hybrid plan that you discuss and implemented that and began to sell that. And through selling it, I've shared this in our mastermind, how I sell that. And I'm seeing how it resonates with owners and just those two things alone, those two changes that we made alone made money. Those are just the two things initially. And then you offer other things that we're beginning to tap into and  [00:14:40] there's so much there, the content, I can only absorb so much at the moment and I'm trying to fully implement those well, but that also gives me a path for several years to come things that I'll be able to dive deeper into different sections of what all you offer and implement those. So I see a path, but those two things alone were the big key movers, which you steer people to. Doing that lead generation first and then begin some pricing and other things. So those two are the big steps that made it an investment and made us money.  [00:15:17] Jason: Yeah, if any of my competitors are listening and they want to figure out how to steal some of the magic from DoorGrow, we focus on two main things with clients and you can probably feel this. I don't think you've heard me mention this Brannon, but one of our big goals within the first 30 to 60 days is we want to make sure clients have really strong clarity on what the future holds for them, like what direction to go in. So we have our clarity assessments we take you through, so you know clearly which path. We have three different paths we take people down depending on which thing is the biggest problem in the business right now. And we focus on pain first. So we get them clarity on what they want and where the pain is and then results. So we want to get them as quickly and as effortlessly as possible to the results. So we're giving you the scripts, the language, the outreach, like all the different things to do. And you mentioned lead gen without spending money, and I know a lot of people are thinking "I'd rather just spend money. I'd rather just spend money because it would save me time," is what they think. What would you say to that?  [00:16:18] Brannon: Yeah, boy, that's a cost. That's not an investment. This is a deeper level of long-term residual lead generation so that what you teach in the coaching pays dividends, not just now, but in the future, it continues to pay residual dividends and you haven't spent any money on it because the big thing in starting a property management company or starting any company is to generate revenue before expenses and it fit with the principles of that is, is generating that revenue before you have any hard costs, which help you get profitable better and faster, then you can have money to do other things to grow it even faster.  [00:17:04] Jason: Yeah there's several things we focus on with clients. We want to decrease the expenses in the business. So we talk about how a lot of property managers, we mentioned this on today's call, right? Like a lot of property managers... it's not about what they need to do more of, or add more of in the business. It's about some of the things they need to eliminate that they are doing. And then we get into, the lead gen piece. A lot of people mistakenly think that they can generate more leads by doing advertising or paid advertising, but that actually are colder leads that take more time. So we've actually decreased your time investment into lead generation and we've zeroed out the costs. [00:17:43] Brannon: And usually it's a better quality person--  [00:17:45] Jason: --and it's an absolutely better quality lead, right? The conversion rate's way higher because we're focusing on warmer lead generation. And the other thing that I think is a secret is that you're creating market share while other property managers are fighting over the small amount of existing market share that exists. They're all in the red water. It's ugly and bloody, and there's a lot of scarcity. And I'm guessing you don't really feel much scarcity in growing your property management business?  [00:18:11] Brannon: Not at all. It's doing so well, there's moments we have to just pause for a few minutes to absorb all the new clients coming on board so that we handle them, you know? It's not from lack of business coming in now from this lead gen source, it's making sure that we handle them effectively. And we talk about this in the coaching too, of how to handle the operations when all this business comes and how to handle it effectively and efficiently. [00:18:38] Jason: That's one of my favorite problems to do is make the growth become so uncomfortable and painful. And then we shift to solving that problem. Everybody wants that problem, but we want to create that problem for our clients that they're having so much growth that it's gotten uncomfortable and they have to start hiring and scaling their systems, so. And then yeah, pricing strategy. We talk about-- like you mentioned the hybrid pricing. Initially, I got the idea from Scott Brady. He's really sharp entrepreneur. And then we've put our own nuance and spin to it to make sure that people do it effectively. So it's psychologically really effective and that's been really really great for our clients that are starting to implement that. [00:19:16] So, yeah, you're right. There's a lot more in the program. I'm excited for you to get into some of the other stuff and get through it. Because I love seeing clients get all these different pieces dialed in because the speed at which the company moves forward is rapid. Now, a lot of people, a lot of property managers are already burnt out. They're already burnt out in their business. They're not enjoying it, which I would normally say, they're just doing it wrong, but that's, I think also one of the key things that we focus on at DoorGrow is not just building a business that just gets more crazy and more hectic and moves fast, that you enjoy less and less, which is typical. Most get to 200 to 400 doors and they're burnt out. They're micromanaging their team. They hate their day-to-day.  [00:19:56] Brannon: You talk about both growing with quantity, but also growing with quality and creating a quality of life too.  [00:20:05] Jason: Yeah, it's a big deal. So our primary focus is on, I call it the four reasons. I've done a previous podcast episode on that. For those listening, you can go back and listen to that. That's our primary goal is to move people towards the four reasons of more fulfillment, more freedom, more contribution, and more support. And as you've been building your business, it could get uglier and more painful, but we always try to make sure that the client understands that's the wrong way to do it. Like we can get you more support and make it more fun and you do less and less in the business.  [00:20:36] Brannon: I like to learn and listen as it scales, how to scale it, and you share this in the coaching, how to scale it properly so that you don't get burnout. So I'm aware of that and want to make sure that happens not just for me, but y'all also share how to do that for your team too. The positions and the different times to hire and how to do that effectively. So it's not just for the owner, but it's how to create quality for everyone.  [00:21:06] Jason: So you've seen some results in the program. What do you feel like your team's perspective of all this movement and change has been, and maybe even your spouse, like how is this kind of rippling out around you? Is this creating some pain and problems for people around you? Or how did they feel about all this?  [00:21:24] Brannon: Jason, you'll appreciate that I use a lot of your quotes at home. But, when you're hearing good things, you want to share it. So I would say we're growing and I think the team, I know the team is all on board and they're excited about the growth, but as any good growth, there is stretching and you have to go through that stretching process that makes you better, but you've provided several good things that help the team that I'm using to help them get through the stretching with the growth that we're having. And we'll take this problem of growth, as we all remember the great recession and we were begging to be busy. I keep mindful of that, of being grateful that we are and would not take the other side of that of not being busy. I keep that in mind and I encourage the team, and they're encouraged by the growth too. They're very excited. Even our sales team notices it and they're like, "man, maybe I should be on that side of the business." They get really excited about it.  [00:22:25] Jason: What's one thing you feel like you could share maybe with the audience, people that aren't in the program that might benefit them, that is maybe something you learned in the program or, maybe just a mindset shift or a takeaway or something that might be helpful to those that are listening? [00:22:41] Brannon: Boy. That's there's so many... [00:22:43] Jason: There are those that are struggling. What feedback or idea would you want to share with them?  [00:22:49] Brannon: I think, you know, looking for referrals from agents that are working in the multi-family or property industry that are selling investments that has helped us quite a bit, but what's been beneficial in the coaching is you've given a great template of a script of how to do it that is genuine, that really flows well and is right in line with building high trust with clients and with agents. That's been the number one benefit of the coaching is not just that idea, but then even giving a practical script that really works. We've been in coaching. We've all been in different programs where we'll see a script that is just not realistic because that person doesn't do it. Your script is genuinely realistic, and it works. I tested it. It worked. I went line by line, even had the script in front of me as I'm going through it, and it really flowed genuine and real and generated referrals that day. [00:23:58] Jason: That was actually one of those moments. We have these moments as coaches where we feel like-- it's similar to as being a dad and seeing your kid get an award or do something, but one of those moments for me was when you sent me your call recording, and you just followed the script. Because I get a lot of call recordings from clients and they don't follow the script. They either don't feel confident doing it that way, or they say it different or they think they're trying to be cute or clever. And then I'm coaching them like, "stop saying 'um' and stop saying 'kinda' and 'maybe' like show confidence." you just followed the script, and it went so beautifully, and that was just really rewarding to me to be able to hear that and go " yes! It worked." And hear that result like real-time is really cool.  [00:24:43] Brannon: I think I came into the coaching with the mindset and I thought of that 90-10 principle, and I said, "I'm going to be the 10%. Sink or swim, I'm going to be the 10% and I'm just going to follow it" and let it go where it went. But the beauty of it is, it went well. And it would for anyone that followed because we all know as we coach or teach, the ones that just say, "Hey, I'm going to be humble and I'm just going to do what you say, and let's see what happens." it generally works.  [00:25:16] Jason: Yeah. Yeah. I'd love to tell clients like, "Hey, it's proven. If Brannon can do it, anybody else can do it too. Brannon's not any smarter or cooler than anybody else in our program, other than the fact that he does the work and he does what we tell them to do. And that makes you, I think, pretty smart and pretty cool. So I appreciate you, Brannon. So, um, Cool. I, appreciate you coming and taking some time out of your day to be here on the #DoorGrowShow and on the podcast. For those that are. On the fence, thinking about DoorGrow maybe they've heard about DoorGrow, a year ago or five years ago or in the past. What would you say to them now? You're on the other side of the paywall. You see what's going on in the community. What would you say?  [00:25:57] Brannon: Well, You might not like this, but I think it's so good, sometimes I wouldn't want to tell anybody 'cause it's so good for people.  [00:26:06] Jason: I've heard that. I've heard that quite a bit, which is really funny. Like "I want everyone to do it except my competition." [00:26:12] Brannon: That's it. [00:26:13] Jason: So which market are you in?  [00:26:16] Brannon: We're in north Texas.  [00:26:17] Jason: All right. So if you're in north Texas, Brannon says, do not do the DoorGrow thing. It's not going to work out for you probably, but everybody else should totally join this program. Does that sound accurate?  [00:26:30] Brannon: North Texas property managers and there's plenty of business for all of us. [00:26:35] Jason: There is. That's something, I think that we're really big on the program. You're not in the red water feeling scarcity fighting with other property managers. There's 70% are self-managing, there's tons of available potential business out there, and you've been able to tap into that tap and you're getting plenty from it and yeah, there's plenty of business out there. Very cool. Brannon appreciate you being a client. Appreciate you taking time out. And anything you wanna say before we wrap this up?  [00:27:03] Brannon: No, I think I've covered quite a bit myself.  [00:27:05] Jason: All right. Awesome. Thanks, Brannon. All right. So for those that have been listening to this and you're curious or interested in DoorGrow, you can reach out to us. And if you want to test the waters a little bit and get familiar with this, because this may be the first time you've heard about us for some reason, join our Facebook group, go to doorgrowclub.com. Videos like this get pushed into the group. I do live streams multiple times a week now. I'm sharing concepts and ideas. My goal and job is to prove to you that we have some value to offer to you. Once you get beyond the paywall, there's even more. And so join the DoorGrowClub. You can go to doorgrowclub.com to get to our Facebook group. The other thing that I would recommend is just go to doorgrow.com. [00:27:52] If you're curious and you want to set up a call and talk to my sales team, they will listen to you. I have great people on the team. They really care about our vision of helping property managers. And so if you're struggling with some issues, some challenges, bring it up to them and talk with them and they will help you see if there's a path in which we can help you deal with those challenges, whether it's you're just not enjoying your day to day, you're struggling with your team and with operations, you're struggling to figure out how to add doors and grow your business, you hate your website, you don't like your brand. Like we can help solve these problems for you and they'll help you see how we can do that. And they'll also give you access to our seven frameworks training so you can see seven different growth frameworks and really shift your mindset out of the idea that you need cold leads and you need to do advertising and you need to spend a bunch of money on marketing. [00:28:41] We'll shift you out of that and help you see why that mindset actually has been hurting your growth and will benefit you and get you moving forward. And that's it for today. So until next time, everybody to our mutual growth. Reach out to DoorGrow, and we'll talk to you soon. Bye, everyone. [00:28:55] You just listened to the #DoorGrowShow. We are building a community of the savviest property management entrepreneurs on the planet in the DoorGrowClub. Join your fellow DoorGrow Hackers at doorgrowclub.com. Listen, everyone is doing the same stuff. SEO, PPC, pay-per-lead content, social direct mail, and they still struggle to grow!  [00:29:22] At DoorGrow, we solve your biggest challenge: getting deals and growing your business. Find out more at doorgrow.com. Find any show notes or links from today's episode on our blog doorgrow.com, and to get notified of future events and news subscribe to our newsletter at doorgrow.com/subscribe. [00:29:43] Until next time, take what you learn and start DoorGrow Hacking your business and your life. 

Sean: When you're looking for farmers or suppliers, what were the KPIs or metrics you were looking for in a supplier to maintain quality control? Jason: For us, it is always relationship-based, are you in it for the long term? Is this the right partner that we should get? Because when you talk to farmers - first of all, they are already decades in terms of trading their fruits and vegetables. So when it comes to us, we're a new entity that's coming in and they will say "who is this new trader? We're tired of them because most of the time they come and go away." So we want to have a partner who will help us grow and would also like to grow together, in terms of how we will be able to tap into the market - just having that collaborative relationship. And so our KPI is actually basically helping each other, its give and take, it should not be 'because you failed to deliver, it will be deducted to your scorecard.' So we want to make sure we know what's your pain point, you know, what's our pain point and we work together on that. Sean: Very nice. So more of relationship-driven rather than giving each other scorecards because you know for farmers it's already tough for them. Jason: Because they will also have the power to not give it to you - the products. They can say that 'I would rather throw my products than give them to you.' They really do have the power to do that. And it is really happening, you will see some trucks full of products and the farmers won't sell them, they will just let them rot because the traders are trying to buy them at a very low price. And remember that these farmers also have pride, and they don't want to sell at below cost, they'll just throw it away rather than you profiting from it. So we always make sure we have the values that are already engraved. - - - Youtube: https://www.youtube.com/leadershipstack Join our community and ask questions here: from.sean.si/discord Facebook: https://www.facebook.com/leadershipstack

It's time to shoot for the stars with Dan Isla, VP of Product at itopia, to talk about everything from astronomical importance of reliability to time zones on Mars. Dan's trajectory has been a propulsion of jobs bordering on the science fiction, with a history at NASA, modernizing cloud computing for them, and loads more. Dan discusses the finite room for risk and failure in space travel with an anecdote from his work on Curiosity. Dan talks about his major take aways from working at Google, his “baby” Selkies, his work at itopia, and the crazy math involved with accounting for time on Mars!In this episode, we cover: Introduction (00:00) Dan's work at JPL (01:58) Razor thin margins for risk (05:40) Transition to Google (09:08)  Selkies and itopia (13:20) Building a reliability community (16:20) What itopia is doing (20:20) Learning, building a “toolbox,” and teams (22:30) Clockdrift (27:36) Links Referenced: itopia: https://itopia.com/ Selkies: https://github.com/danisla/selkies selkies.io: https://selkies.io Twitter: https://twitter.com/danisla LinkedIn: https://www.linkedin.com/in/danisla/ TranscriptDan: I mean, at JPL we had an issue adding a leap second to our system planning software, and that was a fully coordinated, many months of planning, for one second. [laugh]. Because when you're traveling at 15,000 miles per hour, one second off in your guidance algorithms means you missed the planet, right? [laugh]. So, we were very careful. Yeah, our navigation parameters had, like, 15 decimal places, it was crazy.Julie: Welcome to Break Things on Purpose, a podcast about reliability, building things with purpose, and embracing learning. In this episode, we talked to Dan Isla, VP of Product at itopia about the importance of reliability, astronomical units, and time zones on Mars.Jason: Welcome to the show, Dan.Dan: Thanks for having me, Jason and Julie.Jason: Awesome. Also, yeah, Julie is here. [laugh].Julie: Yeah. Hi, Dan.Jason: Julie's having internet latency issues. I swear we are not running a Gremlin latency attack on her. Although she might be running one on herself. Have you checked in in the Gremlin control panel?Julie: You know, let me go ahead and do that while you two talk. [laugh]. But no, hi and I hope it's not too problematic here. But I'm really excited to have Dan with us here today because Dan is a Boise native, which is where I'm from as well. So Dan, thanks for being here and chatting with us today about all the things.Dan: You're very welcome. It's great to be here to chat on the podcast.Jason: So, Dan has mentioned working at a few places and I think they're all fascinating and interesting. But probably the most fascinating—being a science and technology nerd—Dan, you worked at JPL.Dan: I did. I was at the NASA Jet Propulsion Lab in Pasadena, California, right, after graduating from Boise State, from 2009 to around 2017. So, it was a quite the adventure, got work on some, literally, out-of-this-world projects. And it was like drinking from a firehose, being kind of fresh out to some degree. I was an intern before that so I had some experience, but working on a Mars rover mission was kind of my primary task. And the Mars rover Curiosity was what I worked on as a systems engineer and flight software test engineer, doing launch operations, and surface operations, pretty much the whole, like, lifecycle of the spacecraft I got to experience. And had some long days and some problems we had to solve, and it was a lot of fun. I learned a lot at JPL, a lot about how government, like, agencies are run, a lot about how spacecraft are built, and then towards the end a lot about how you can modernize systems with cloud computing. That led to my exit [laugh] from there.Jason: I'm curious if you could dive into that, the modernization, right? Because I think that's fascinating. When I went to college, I initially thought I was going to be an aerospace engineer. And so, because of that, they were like, “By the way, you should learn Fortran because everything's written in Fortran and nothing gets updated.” Which I was a little bit dubious about, so correct folks that are potentially looking into jobs in engineering with NASA. Is it all Fortran, or… what [laugh] what do things look like?Dan: That's an interesting observation. Believe it or not, Fortran is still used. Fortran 77 and Fortran—what is it, 95. But it's mostly in the science community. So, a lot of data processing algorithms and things for actually computing science, written by PhDs and postdocs is still in use today, mostly because those were algorithms that, like, people built their entire dissertation around, and to change them added so much risk to the integrity of the science, even just changing the language where you go to language with different levels of precision or computing repeatability, introduced risk to the integrity of the science. So, we just, like, reused the [laugh] same algorithms for decades. It was pretty amazing yeah.Jason: So, you mentioned modernizing; then how do you modernize with systems like that? You just take that codebase, stuff it in a VM or a container and pretend it's okay?Dan: Yeah, so a lot of it is done very carefully. It goes kind of beyond the language down to even some of the hardware that you run on, you know? Hardware computing has different endianness, which means the order of bits in your data structures, as well as different levels of precision, whether it's a RISC system or an AMD64 system. And so, just putting the software in a container and making it run wasn't enough. You had to actually compute it, compare it against the study that was done and the papers that were written on it to make sure you got the same result. So, it was pretty—we had to be very careful when we were containerizing some of these applications in the software.Julie: You know, Dan, one thing that I remember from one of the very first talks I heard of yours back in, I think, 2015 was you actually talked about how we say within DevOps, embrace failure and embrace risk, but when you're talking about space travel, that becomes something that has a completely different connotation. And I'm kind of curious, like, how do you work around that?Dan: Yeah, so failing fast is not really an option when you only have one thing [laugh] that you have built or can build. And so yeah, there's definitely a lot of adverseness to failing. And what happens is it becomes a focus on testing, stress testing—we call it robustness testing—and being able to observe failures and automate repairs. So, one of the tests programs I was involved with at JPL was, during the descent part of the rover's approach to Mars, there was a power descent phase where the rover actually had a rocket-propelled jetpack and it would descend to the surface autonomously and deliver the rover to the surface. And during that phase it's moving so fast that we couldn't actually remote control it, so it had to do everything by itself.And there were two flight computers that are online, pretty much redundant, everything hardware-wise, and so it's kind of up to the software to recover itself. And so, that's called entry descent and landing, and one of my jobs towards the end of the development phase was to ensure that we tested all of the possible breakage points. So, we would do kind of evil Gremlin-like things. We actually—the people in the testbed, we actually call Gremlins. And [laugh] we would—we—they inject faults during the simulation.So, we had copies of the hardware running on a desk, the software was running, and then we'd have Gremlins go and say like, “Hey, flight computer one just went out. You know, what's going to happen?” And you watch the software, kind of, take over and either do the right thing or simulate a crash landing. And we find bugs in the software this way, we'd find, like, hangs in the control loops for recovery, and we had to fix those before we made it to Mars, just in case that ever happened. So, that was like how we, like, really stressed test the hardware, we did the same thing with situational awareness and operations, we had to simulate things that would happen, like, during launch or during the transit from Earth to Mars, and then see how the team itself reacted to those. You know, do our playbooks work? Can we run these in enough time and recover the spacecraft? So, it was a lot of fun. That's I guess that's about as close to, like, actually breaking something I can claim to. [laugh].Julie: Well, I have to say, you've done a good job because according to Wikipedia—which we all know is a very reliable source—as of May 9th, 2022, Curiosity has been active on Mars for 3468 sols or 3563 days, and is still active. Which is really amazing because I don't—was it ever intended to actually be operational that long?Dan: Not really. [laugh]. The hardware was built to last for a very long time, but you know, as with most missions that are funded, they only have a certain amount of number of years that they can be operated, to fund the team, to fund the development and all that. And so, the prime mission was only, like, two years. And so, it just keeps getting extended. As long as the spacecraft is healthy, and, like, doing science and showing results, we usually extend the missions until they just fall apart or die, or be intentionally decommissioned, kind of like the Cassini project. But yeah.Julie: Well, you've heard it here first, folks. In order to keep funding, you just need to be, quote, “Doing science.” [laugh]. But Dan, after JPL, that's when you went over to Google, right?Dan: Yeah, yeah. So, it was kind of that transition from learning how to modernize with cloud. I'd been doing a lot with data, a lot with Amazon's government cloud, which is the only cloud we could use at JPL, and falling in love with these APIs and ways to work with data that were not possible before, and saw this as a great way to, you know, move the needle forward in terms of modernization. Cloud is a safe place to prototype a safe place to get things done quick. And I always wanted to work for a big tech company as well, so that was always another thing I was itching to scratch.And so Google, I interviewed there and finally made it in. It was not easy. I definitely failed my first interview. [laugh]. But then try it again a few years later, and I came in as a cloud solution architect to help customers adopt cloud more quickly, get through roadblocks.My manager used to say the solution architects were the Navy Seals of cloud, they would drop in, drop a bunch of knowledge bombs, and then, like, get out, [laugh] and go to the next customer. It was a lot of fun. I got to build some cool technology and I learned a lot about what it's like working in a big public company.Julie: Well, one of my favorite resources is the Google SRE book, which, as much as I talk about it, I'm just going to admit it here now, to everybody that I have not read the entire thing.Dan: It's okay.Julie: Okay, thank you.Dan: Most people probably haven't.Julie: I also haven't read all of Lord of the Rings either. But that said, you know, when you talk about the learnings, how much of that did you find that you practiced day-to-day at Google?Dan: In cloud—I've mostly worked in cloud sales, so we were kind of post-sales, the experts from the technology side, kind of a bridge to engineering and sales. So, I didn't get to, like, interact with the SREs directly, but we have been definitely encouraged, I had to learn the principles so that we could share them with our customers. And so, like, everyone wanted to do things like Google did, you know? Oh, these SREs are there, and they're to the rescue, and they have amazing skills. And they did, and they were very special at Google to operate Google's what I would call alien technology.And so, you know, from a principles point of view, it was actually kind of reminded me a lot of what I learned at JPL, you know, from redundant systems and automating everything, having the correct level of monitoring. The tools that I encountered at Google, were incredible. The level of detail you could get very quickly, everything was kind of at your fingertips. So, I saw the SREs being very productive. When there was an outage, things were communicated really well and everyone just kind of knew what they were doing.And that was really inspiring, for one, just to see, like, how everything came together. That's kind of what the best part of working at Google was kind of seeing how the sausage was made, you know? I was like, “Oh, this is kind of interesting.” [laugh]. And still had some of its big company problems; it wasn't all roses. But yeah, it was definitely a very interesting adventure.Jason: So, you went from Google, and did you go directly to the company that you helped start, right now?Dan: I did. I did. I made the jump directly. So, while I was at Google, you know, not only seeing how SRE worked, but seeing how software was built in general and by our customers, and by Google, really inspired me to build a new solution around remote productivity. And I've always been a big fan of containers since the birth of Docker and Kubernetes.And I built the solution that let you run, kind of, per-user workloads on Kubernetes and containers. And this proved to be interesting because you could, you know, stand up your own little data processing system and scale it out to your team, as well as, like, build remote code editors, or remote desktop experiences from containers. And I was very excited about this solution. The customers were really starting to adopt it. And as a solution architect, once the stuff we built, we always open-source it.So, I put it on GitHub as a project called Selkies. And so, Selkies is the Kubernetes components and there's also the high performance streaming to a web browser with WebRTC on GitHub. And a small company, itopia, I met at a Google conference, they saw my talk and they loved the technology. They were looking for something like that, to help some of their product line, and they brought me in as VP of Product.So, they said, “We wanted to productize this.” And I'm like, “Well, you're not doing that without me.” [laugh]. Right? So, through the pandemic and work from home and everything, I was like, you know, now is probably a good time to go try something new.This is going to be—and I get to keep working on my baby, which is Selkies. So yeah, I've been itopia since beginning of 2021, building a remote desktop, really just remote developer environments and other remote productivity tools for itopia.Julie: Well and, Dan, that's pretty exciting because you actually talked a little bit about that at DevOpsDays Boise, which if that video is posted by the time of publication of this podcast, we'll put a link to that in the show notes. But you're also giving a talk about this at SCaLE 19x in July, right?Dan: Yeah, that's right. Yeah, so SCaLE is the Southern California Linux Expo, and it's a conference I really enjoy going to get to see people from Southern California and other out of town, a lot of JPLers usually go as well and present. And so, it's a good time to reconnect with folks. But yeah, so SCaLE, you know, they usually want to talk more about Linux and some of the technologies and open-source. And so yeah, really looking forward to sharing more about selfies and kind of how it came to be, how containers can be used for more than just web servers and microservices, but also, you know, maybe, like, streaming video games that have your container with the GPU attached. The DevOpsDays Boise had a little demo of that, so hopefully, that video gets attached. But yeah, I'm looking forward to that talk at the end of July.Jason: Now, I'm really disappointed that I missed your talk at DevOpsDays Boise. So Julie, since that's your domain, please get those videos online quickly.Julie: I am working on it. But Dan, one of the things that you know you talk about is that you are the primary maintainer on this and that you're looking to grow and improve with input from the community. So, tell us, how can the community get involved with this?Dan: Yeah, so Selkies is on GitHub. You can also get to it from selkies.io. And basically, we're looking for people to try it out, run it, to find problems, you know, battle test it. [laugh]. We've been running it in production at itopia, it's powering the products they're building now.So, we are the primary maintainers. I only have a few others, but, you know, we're just trying to build more of an open-source community and level up the, you know, the number of contributors and folks that are using it and making it better. I think it's an interesting technology that has a lot of potential.Jason: I think as we talk about reliability, one of the things that we haven't covered, and maybe it's time for us to actually dive into that with you is reliability around open-source. And particularly, I think one of the problems that always happens with open-source projects like this is, you're the sole maintainer, right? And how do you actually build a reliable community and start to grow this out? Like, what happens if Dan suddenly just decides to rage quit tech and ups and leaves and lives on his own little private island somewhere? What happens to Selkies?Do you have any advice for people who've really done this, right? They have a pet project, they put it on GitHub, it starts to gain some traction, but ultimately, it's still sort of their project. Do you have any advice for how people can take that project and actually build a reliable, growing, thriving community around it?Dan: Honestly, I'm still trying to figure that out [laugh] myself. It's not easy. Having the right people on your team helps a lot. Like, having a developer advocate, developer relations to showcase what it's capable of in order to create interest around the project, I think is a big component of that. The license that you choose is also pretty important to that.You know, there's some software licenses that kind of force the open-sourcing of any derivative of what you build, and so that can kind of keep it open, as well, as you know, move it forward a little bit. So, I think that's a component. And then, you know, just, especially with conferences being not a thing in the last couple of years, it's been really hard to get the word out and generate buzz about some of these newer open-source technologies. One of the things I kind of like really hope comes out of a two-year heads-down time for developers is that we're going to see some, like, crazy, amazing tech on the other side. So, I'm really looking forward to the conferences later this year as they're opening up more to see what people have been building. Yeah, very interested in that.Jason: I think the conversation around open-source licenses is one that's particularly interesting, just because there's a lot involved there. And there's been some controversy over the past couple of years as very popular open-source projects have decided to change licenses, thinking of things like Elastic and MongoDB and some other things.Dan: Yeah. Totally.Jason: You chose, for Selkies, it looks like it's Apache v2.Dan: Yep. That was mostly from a Google legal point of view. When I was open-sourcing it, everything had to be—you know, had to have the right license, and Apache was the one that we published things under. You know, open-source projects change their license frequently. You saw that, like what you said, with Elastic and Mongo.And that's a delicate thing, you know, because you got to make sure you preserve the community. You can definitely alienate a lot of your community if you do it wrong. So, you got to be careful, but you also, you know, as companies build this tech and they're proud of it and they want to turn it into a product, you want to—it's a very delicate process, trying to productize open-source. It can be really helpful because it can give confidence to your customers, meaning that, like, “Hey, you're building this thing; if it goes away, it's okay. There's this open-source piece of it.”So, is instills a little bit of confidence there, but it also gets a little tricky, you know? Like, what features are we adding the add value that people will still pay for versus what they can get for free? Because free is great, but you know, it's a community, and I think there are things that private companies can add. My philosophy is basically around packaging, right? If you can package up an open-source product to make it more easier to consume, easier to deploy, easier to observe and manage, then you know, that's a lot of value that the rest of the free community may not necessarily need.If they're just kind of kicking the tires, or if they have very experienced Kubernetes team on-site, they can run this thing by themselves, go for it, you know? But for those, the majority that may not have that, you know, companies can come in and repackage things to make it easier to run open-source. I think there's a lot of value there.Jason: So, speaking of companies repackaging things, you mentioned that itopia had really sort of acquired you in order to really build on top of Selkies. What are the folks at itopia doing and how are they leveraging the software?Dan: That's a good question. So, itopia's mission is to radically improve work-from-anywhere. And we do that by building software to orchestrate and automate access to remote computing. And that orchestration and automation is a key component to this, like, SaaS-like model for cloud computing.And so, Selkies is a core piece of that technology. It's designed for orchestrating per-user workloads, like, remote environments that you would need to stand up. And so, you know, we're adding on things that make it more consumable for an enterprise, things like VPN peering and single-sign-on, a lot of these things that enterprises need from day one in order to check all the boxes with their security teams. And at the heart of that is really just increasing the amount of the productivity you have through onboarding.Basically, you know, setting up a developer environment can take days or weeks to get all the dependencies set up. And the point of itopia—Spaces is the product I'm working on—is to reduce that amount of time as much as possible. And, you know, this can increase risk. If you have a product that needs to get shipped and you're trying to grow or scale your company and team and they can't do that, you can slip deadlines and introduce problems, and having a environment that's not consistent, introduces reliability problems, right, because now you have developers that, “Hey, works on my machine.” But you know, they may have—they don't have the same machine, same environment as everyone else, and now when it comes to reproducing bugs or even fixing them, that you can introduce more problems to the software supply chain.Julie: I mean, that sounds like a great problem to solve and I'm glad you're working on it. With your background being varied, starting as an intern to now where you personally are being acquired by organizations. What's something that you've really learned or taken from that? Because one thing that you said was that you failed your first Google interview badly? And—Dan: Yes. [laugh].Julie: I find that interesting because that sounds like you know, you've taken that learning from failure, you've embraced the fact that you failed it. Actually, I just kind of want to go back. Tell us, do you know what you did?Dan: It was definitely a failure. I don't know how spectacular it was, but, like, [laugh] google interviews are hard. I mean—and that's just how it is, and it's been—it's notorious for that. And I didn't have enough of the software, core software experience at the time to pass the interview. These are, like, five interviews for a software engineer.And I made it through, like, four of them. The last one was, like, just really, really, really hard and I could not figure it out. You know, because this is, like, back in the day—and I think they still do this, like, where you're, like, coding on a whiteboard, right? Like, okay, right, this C code on a whiteboard, and it has to work. You know, the dude is, like, right, there compiling it, right? Like, “Okay, [unintelligible 00:23:29], boy.” [laugh].So, not only is a high stress, but it has to be right as well. [laugh]. And so, like, it was just a very difficult experience. And what I learned from that was basically, “Okay, I need to, one, get more experience in this style and this domain of programming, as well, as you know, get more comfortable speaking and being in front of people I don't know.” [laugh].So yeah, there's definitely components there of personal growth as well as technical growth. From a technical point of view, like, my philosophy as being an engineer in general, and software developer, is have a really big toolbox and use the tools that are appropriate for the job. This is, like, one of my core philosophies. Like, people ask, you know, ‘what language do you use?' And I'm like, “Whatever language you needed to solve the problem.”Like, if you're writing software, in a—with libraries that are all written in C, then don't try to do that in, like, Java or something, in some other language that doesn't have those language bindings. Don't reinvent the language bindings. You follow the problem and you follow the tech. What language, what tool will best solve this problem? And I'm always working backwards from the problem and then bringing in the right tools to solve it.And that's something that has paid off in dividends because it's very—problem-solving is fun and it's something I always had a passion for, but when you have a toolbox that is full of interesting gadgets and things you can use, you get excited every time you get to use that tool. Like, just like power tools here, I have a—I don't know, but it's like, “Yeah, I get to use the miter saw for this thing. Awesome. I don't have one? Okay, I'm going to go buy one.” [laugh].Julie: That's actually—that's a really good point, one of the talks that I gave was, “You Can't Buy DevOps.” And it was really all about letting developers be part of the process in choosing the tools that they're going to use. Because sometimes I think organizations put too many constraints around that and force you to use these tools that might not be the best for what you're trying to accomplish. So, I like that you bring up having the ability to be excited about your toolbox, or your miter saw. For me, it would be my dremel. Right? But what tool is going to—Dan: [crosstalk 00:25:39] cool.Julie: Yeah, I mean, they really are—what tool is going to be best for the job that you are trying to accomplish? And I think that that's, that's a big thing. So, when you look to bring people onto your team, what kind of questions do you ask them? What are you looking for?Dan: Well, we're just now starting to really grow the company and try and scale it up. And so we're, you know, we're starting to get into more and more interview stuff, I try to tell myself, I don't want to put someone through the Google experience again. And part of that is just because it wasn't pleasant, but also, like, I don't know if it was really that useful [laugh] at the end of the day. And so, you know, there's a lot about culture fit that is really important. People have to be able to communicate and feel comfortable with your team and the pace that your team is working at. And so, that's really important.But you know, technically, you know, I like to see a lot of, you know—you got to be able to show me that you can solve problems. And that can be from, you know, just work that you've done an open-source, you know, having a good resume of projects you've worked on is really important because then we can just talk about tech and story about how you solve the problem. I don't have to—I don't need you to go to the whiteboard and code me something because you have, like, 30 repos on GitHub or something, right? And so, the questions are much more around problem-solving: you know, how would you solve this problem? What technology choices would you use, and why?Sometimes I'll get the fundamentals, like, do you understand how this database works at its core or not? You know, or why is it… why is that good or bad? And so, looking for people who can really think within the toolbox they have—it doesn't have to be a big one, but do they know how to use the tools that they've acquired so far, and really, just really, really critically think through with your problems? So, to me, that's a better skill to have than just, you know, being able to write code on the whiteboard.Julie: Thanks for that, Dan. And earlier, before we started the official recording here, you were talking a little bit about time drift. Do you want to fill everybody in on what you were talking about because I don't think it was Doctor Strange and the Multiverse of Madness?Dan: No. [laugh]. I think there were some—we were talking about um…clocks?Julie: Clocks skew.Dan: Daylight savings time?Julie: Yeah.Dan: Clock skew, clock drift. There was a time at JPL when we were inserting a leap second to the time. This actually happened all throughout the world, where periodically that the clocks will drift far enough because the orbits and the rotation of the planet are not, like, perfectly aligned to 365 days in a year and 24 hours in a day. And so, every so decades, you have to insert these leap seconds in order to catch up and make time more precise. Well, space travel, when you're planning, you have to—you're planning to the position of the stars and the planets and the orbital bodies, and those measurements are done at such a large scale that you have—your precision goes, like, way out, you know, many, many decimal places in order to properly plan to the bodies up big.And with the Mars Rover, one of these leap seconds happened to come in, like, right, before we launched. And it was like, oh my gosh, this is going to be to—change all of our ephemeris files—the data that you use to track positions—and we had to do it, like, synchronize it all, like, right, when the leap second was going in. And we tested this extensively because if you get it wrong with your spacecraft is traveling, like, 15,000 miles an hour towards Mars, and a one-second pointing error from Earth means, like, you missed the whole planet, you won't even get there. [laugh]. We're not talking about, like, missing the landing site of, like, a few kilometers. No, it's like thousands of kilometers in pointing error.So yeah, things are astronomical [laugh] in units. Actually, that's why they're called AU, astronomical units, when you're measuring the distance from the Sun. So yeah, it was a pretty fun time. A little bit nerve-wracking just because the number of systems that had to be updated and changed at the same time. It's kind of like doing a rolling update on a piece of software that just had to go out all at the same time. Yeah.Jason: I think that's really interesting, particularly because, you know, for most of us, I think, as we build things whether that's locally or in the cloud or wherever our servers are at, we're so used to things like NTP, right, where things just automatically sync and I don't have to really think about it and I don't really have to worry about the accuracy because NTP stays pretty tight. Usually, generally.Dan: Mm-hm.Jason: Yeah. So, I'm imagining, obviously, like, on a spacecraft flying 15,000 miles a second or whatever, no NTP out there.Dan: [laugh]. Yeah, no NTP and no GPS. Like, all the things you take for granted, on Mars are just not there. And Mars even has a different time system altogether. Like the days on Mars are about 40 minutes longer because the planet spins slower.And my first 90 sols—or days on Mars—of the mission, the entire planning team on earth that I was a part of, we lived on Mars time. So, we had to synchronize our Earth's schedule with what the rover was doing so that when the rover was asleep, we were planning the next day's activities. And when it woke up, it was ready to go and do work during the day. [laugh]. So, we did this Mars time thing for 90 days. That was mostly inherited from the Mars Exploration rovers, Spirit and Opportunity because they were only designed to live for, like, 90 days.So, the whole team shifted. And we—and now it's kind of done in spirit of that mission. [laugh]. Our rover, we knew it was going to last a bit longer, but just in case, let's shift everyone to Mars time and see what happened. And it was not good. We had to [laugh] we had to end that after 90 days. People—your brain just gets completely fried after that. But it was bizarre.And there's no time. You have invent your own time system for Mars. Like, there's no, it was called LMST, or Local Mars Standard Time, local mean standard time. But it was all, like, relative to, you know, the equator and where you were on the planet. And so, Mars had his own Mars time that counted at a different rate per second.And so, it was funny, we had these clocks in the Mission Control Room that—there was this giant TV screen that had, like, four different time clocks running. It had, like, Pasadena time, UTC time, Mars time, and, like, whatever time it was at the Space Network. And I was like, “Oh, my gosh.” And so, we were always doing these, like, time conversions in our heads. It was mental. [laugh]. So, can't we just all be on UTC time? [laugh].Jason: So, I'm curious, with that time shift of being on Mars time and 40 minutes longer, that inherently means that by the end of that 90 days, like, suddenly, your 8 a.m. Mars local time is, like, shifted, and is now, like, hours off, right? You're waking—Dan: Yeah.Jason: Up in the middle of the night?Dan: Totally, yeah.Jason: Wow.Dan: Yeah, within, like, two weeks, your schedule will be, like, upside down. It's like, every day, you're coming in 40 minutes later. And yeah, it was… it was brutal. [laugh]. Humans are not supposed to do that.If you're actually living on Mars, you're probably okay, but like, [laugh] trying to synchronize those schedules. I thought you were going from East Coast to West Coast time, working remote was hard. And, like, [laugh] that's really remote.Julie: Dan, that's just astronomical.Dan: [laugh].Julie: I'm so sorry. I had to do it. But with that—[laugh].Jason: [laugh].Dan: [laugh]. [unintelligible 00:33:15].Julie: With that, Dan, I really just want to thank you for your time on Break Things on Purpose with us today. And as promised, if I can find the links to Dan's talks, if they're available before this episode posts, we will put those in the show notes. Otherwise, we'll put the link to the YouTube channel in the show notes to check for updates. And with that, I just want to thank you, Dan, and wish you a wonderful day.Jason: Before we go, Dan, do you have anything that you'd like to plug? Any projects that people should check out, where they can find you on the internet, stuff like that?Dan: Yeah, thank you guys very much for having me. It was a great conversation. Really enjoyed it. Please check out our new product, itopia Spaces, remote developer environments delivered, powered by Selkies. We launched it last fall and we're really trying to ramp that up.And then check out the open-source Selkies project, selkies.io will get you there. And yeah, we're looking for contributors. Beyond that, you can also find me on Twitter, I'm @danisla, or on LinkedIn.Jason: Awesome. Well, thanks again for being a part of the show. It's been fantastic.Dan: You're very welcome. Thanks for having me.Jason: For links to all the information mentioned, visit our website at gremlin.com/podcast. If you liked this episode, subscribe to the Break Things on Purpose podcast on Spotify, Apple Podcasts, or your favorite podcast platform. Our theme song is called, “Battle of Pogs” by Komiku, and it's available on loyaltyfreakmusic.com.

Sean: We have a question from Nino. How did the agricultural supply chain disruption caused by the pandemic affect your agribusiness? Jason: And that's a good question. We started Zagana as traditional e-commerce, we take your order and we deliver it in 3 to 5 days. So when we take the order, we send it to the farm saying "here are the orders" then we'll send it to our storages then that's when we distribute it. So that's what we have been doing for the B2B side - that is how we do it before the pandemic because we were delivering to restaurants. And then when the pandemic happened, because of the lockdown, we changed a lot of our supply chain, so it really disrupted it. Jason: Our farmers were not able to supply the products we asked from them. And there was also a rise in demand because everyone went online. March 15, we opened up our website. That was our first time opening up our website, and then thousands of our orders came in. But because we're doing that traditional e-commerce, there was a big number of orders that we sent to the farm, and that made our farmers baffled as some of the products are not yet ready for harvest. So a lot of items are out of stock and need to be replaced, our customers really got mad saying "why are there no products coming from the farm?" and then the delivery trucks that we use to carry out the deliveries to our customers before, they often get delayed because they need to go through a lot of checkpoints. Jason: Number one, checkpoints. Number two is the waiting time for the customers to go down either from their condo or inside of their subdivision. So one customer will take around 30 to 40 minutes and think about it that one truck usually carries 30 loads of orders. So by the time, it gets to the 30th customer, some of the goods are already spoiled. So eventually we had to pivot our model. Within a month we were able to pivot it to doing that 'instant commerce.' So what is instant commerce? It's bringing the goods to our storage, keeping them there, and then we start taking orders. So the moment they ordered online, either Zagana.com, Grab, Foodpanda, Lazada, Shopee, Pick-a-roo, or Metro mart. They will be able to view the products that are on stock and out of stock. So if it's out of stock they won't be able to order. So that made it a painless transaction. So if they order and we have it on-stock, we pick it up, pack it and deliver it within 30 minutes to your house. So revolutionizes how we are shopping now digitally. Sean: Amazing. 00:02:55 Sean: Now we have another question. What sets it apart? So we know that there are other solutions trying to also go into the same industry, and provide the same service. What sets Zagana apart and how did you formulate your unique selling proposition? Jason: For us, I will say it's the 30 minutes delivery. I don't see any farm-to-table grocery shops that have been doing that and we have the selections, we have selections from high land to low land crops. We have frozen goods like seafood all the way from Capiz, which is the seafood capital of the Philippines. We were able to work closely with the fishermen - what they offer is freshly harvested seafood. Whether it's scallop, Lapu-Lapu, or milkfish, they blast freeze it so that it will maintain its freshness and we ship it to our storage and from there we deliver it to you. Once you open your package you can really smell its freshness from the sea. We are really proud of the technology that we're putting into it. We're studying how we would be able to prolong and maintain the product, and we're working closely with the Department of Agriculture and UPLB in terms of making sure that we maintain this. - - - Youtube: https://www.youtube.com/leadershipstack Join our community and ask questions here: from.sean.si/discord Facebook: https://www.facebook.com/leadershipstack

In this episode, we cover: Introduction (00:00) “Embracing Change Fearlessly” (01:45) Fearless change enabling good work (04:00) The culture change that needs to happen (06:10) How to talk to your leaders (10:45) “The Adolescent Version” of engineering (14:40) How Natalie prioritizes time, speed, and efficiency (18:42) Natalie's keynote (26:48) Links Referenced: Gremlin: https://www.gremlin.com/ gremlin.com/podcast: https://gremlin.com/podcast loyaltyfreakmusic.com: https://loyaltyfreakmusic.com TranscriptNatalie: I like this—I call it the adolescent version of engineering. It's where, you know, we're through the baby part, we need to start to grow up a little bit, we need to go from getting stuff done in some way or another, to something that's repeatable and scalable. And so, it's like, that adolescent years, that's my fun. That's what I enjoy doing. I call it creating something out of chaos.Basically, taming the chaos is what it really looks like because it's very chaotic initially, and that's true of every, like, small organization; they always start like that. And as they start to grow, you know, you've got ten different engineers who have ten different opinions on how something should be done, and so they do it ten different ways. And that's fine when you're only ten, but then when you need to go from 10 to 20 to 30 to 100, it no longer works.Julie: Welcome to Break Things on Purpose, a podcast about reliability, culture change, and learning from failure. In this episode, we talk with Natalie Conklin, head of engineering at Gremlin, about the importance of embracing change, and how we can all work through our fears and work together to build more reliable systems. Natalie, I'm so excited to have you here with us today. And today is actually a really big day because it is the fifth year of DevOpsDays Boise, which you are doing the closing keynote for. So, really excited to have you both on the podcast and at the conference today. And your talk is titled “Embrace Change Fearlessly.” So, do you want to kick off by telling our listeners a little bit about you and what you're going to be talking about?Natalie: Sure. Thanks for having me. I am excited about both, sort of, [laugh] which is exactly what the talk is about. [laugh]. The talk is really about being able to embrace change fearlessly, and that it's rarely ever fearlessly truly, but mostly around being able to do what makes you afraid anyway.I'm not a big public speaker, so that's something I've had to work hard at trying to be able to be more comfortable doing. And so, this is an exciting time for me. But background-wise, I am the head of engineering currently for Gremlin and had been leading engineering teams for growth companies for just over a decade. And a lot of what I end up doing centers around this: It's helping those engineering teams be willing to move forward in risky—because in growth companies, a lot of times you're building things that are brand new, this is not something that, you know, has been out there and done, so they typically have to do something new for the first time. And so, being able to take calculated risks is tough. It's hard stuff. And so, getting into the right mindset to be able to push through that, that's a lot of what I ended up doing.Julie: I love that. And that's actually a really good point that you're bringing up, you know, growth companies and being in the right mindset. So, one of the things you and I talked about when I was starting here at Gremlin and getting to know you a little bit about your background, which is really cool. You lived in India for a few years, correct?Natalie: I did. I lived there for two years. I was working for a company, we were doing big data analytics for telcos, building big, large platform that we would then do some custom development work off the top of for these various telco companies. And the team over there had experienced some turnover, and so there was a lot of quality issues and things of that nature starting to show up for the first time. This had been a very rock-solid team, honestly, and so the company asked if I would be willing to go to India to figure out what was going on. And so, that was what I did. It was a great opportunity; loved doing it.Julie: So now, as you work with teams to embrace change fearlessly, and we talk about you mentioned the ROI and doing things in new ways and building new things, do you have an example of maybe when you built something new or your team built something new, and it changed the way we work?Natalie: Well yes, an easy answer would just be to fall back on the India example for a second, right? So, a lot of what I did when I went there was they were a very waterfall shop, converted them over to Agile practices and DevOps. They had really none of that practice existing. So, when you ask the company—or the, I'll just say the team to go through that sort of transition, you're pretty much asking them to change everything about the way they work. And we focused a lot more —there was a lot of manual processes that they had been doing previously and we were automating all of those had to do the automations, but then also, you know, make sure that work fit into this new automated way of doing things.They also had, just, also the trepidation over am I going to still be needed, right? Those are all those things that come into your mind when you're basically changing from a manual process to an automated process, “Am I still going to be needed? Is my work going to still be important? What am I going to do in this new world, in this new environment?” There's a lot of that that pops up into people's heads.So, a lot of making the change successful, there's certainly the technical aspects of getting it automated and all those things, but to really make a change successful on that kind of scale, it requires getting people to think about it differently and to be okay, and to realize that they can learn new stuff and they'll come out of this better than how they went in. And a lot of that takes a lot of, just, communication and talking, being very personal with people, making sure that they personally understand how to do this, but then just also, things like training and coaching and making sure that there are people there to counter the negative energy that comes along with change. There's always negative energy that comes along with it, people are nervous, they're scared, and you have to be able to counter that in some way.Julie: You know, there was a talk I gave a while ago, and I'm trying to remember the name of it, but one of the things that I talked about was the Pareto Principle, which is, what, 20% of people are going to be amazing in an organization, 60% are going to be, you know, middle of the road, then you have that bottom 20% that are going to kind of fight that change. And you shouldn't really necessarily focus on that top 20%, but you should put a lot of the focus on bringing that bottom 20% along with you. And we talk a lot about just the cultural change that needs to happen when we talk about Chaos Engineering, for example. I mean, there's a huge cultural change that organizations need to switch that mindset into embracing failure. Which we talk a lot about, but it's hard for folks to embrace change fearlessly, embrace failure fearlessly.When you've been going through these experiences in the past—and you mentioned that you really need to think about the people—what's one of the common fears? You said, you know, people worry about their jobs and worry about being left behind. Work us through how do you help folks with that?Natalie: Yeah, I think that's actually one of the most interesting aspects of this. When you start looking at—when I [start talking about 00:07:18] about, you know, people don't change, when it's something that's personal like getting married or having kids or going off to college, you know, these are all huge life changes, and we celebrate those, we have parties, we're super happy, we think they're fantastic, right? And I mean, if I go back to India for a second, these are the same people that are struggling on, you know, the fact that I'm going to change from a manual testing to an automated testing, will actually go through an arranged marriage where they're marrying someone that they don't know super well, but they're very happy about it, right? So, that's one of the things that I like to point out and have a discussion with people about is that you're not afraid of change; you're afraid of change in your work life, right? And we have to be very specific about that because we start talking about humans are afraid of change, I actually don't agree. I think we're just afraid of changing what we do at work.And usually, that's because that's somehow tied to our needs pyramid, right? Like, that's how we get our needs met from food and shelter and all of these other kinds of things. And so, when we start to threaten that, it gets really, you know, sketchy for a minute, right? So, that's when we have to, like, take a minute and realize what we're doing and realize that we're being overly protective of a part of our world that, you know, we somehow feel like it's going to then have us begging on the street, is the example I give in my talk, right? That's not going to happen. Like, you know, that's just an irrational fear.And it's highly unlikely that that's your right answer. So, what I encourage people to do is to actually find a logical, kind of, sounding board, person, a mentor, a friend—and again, if you don't have this person in your life, then you know, find that person, but start talking to them about, like, what's most likely to happen in this scenario? Or, better yet, what can I get out of it? I think if you spent less time on that and spent, you know, more time on, like, what can I actually get out of this, how could this benefit me, and sort of flip that in your brain.Because what our brains are incredibly good at doing is going down that worst possible path. But the real truth is, we're just as capable of imagining the good. It's just a matter of focus. So, why don't we just focus on that instead? We can focus on what's the positive part of this, what could happen, and we're actually much more likely—there's a whole lot of studies around manifestation—and we can manifest that in our life if we want to, right? So, we just need to focus on the positive side of it.So, I—literally it's honestly a bunch of personal conversations, and getting people to just calm down and realize that the likelihood of their worst-case scenario is not really real. And then start to think through, okay, what can you actually learn from this? You know, is there something that you would like to get out of this? Would you like to try a new role? Would you like to try to lead an initiative? Would you like to be part of this in some way, right?So, those conversations—and again, it has to be personal. That's the thing that I think, you know, when you start doing widespread, full organizational changes, which I was doing over there and I had 120 engineers, it's hard to do it personally because you literally have to have one-on-one conversations with everybody and understand what they are going to get out of it. But that is what's required. I think, to really get people to a comfort zone, you've got to make sure that they understand how they fit in, and their why; why they're doing it.Julie: And that is all amazing. Now, as the leader, as the head of engineering and an organization, how do you recommend individual contributors talk to their leaders? Or how do they bring up concerns in a way that's productive in an organization? Because I know for me, sometimes—and you're right, I am excellent at going down that every possible negative outcome path; I've planned it out pretty well, to my peril, but that means that when I bring up concerns with leadership, I tend to do so in a heightened emotional state. So, what's your advice for folks?Natalie: Well, and it's just that. I think it's exactly where you're headed with that is that take the emotions out of it—or attempt to—and try to present your concerns logically. Because there's going to be situations where what you're bringing up is something they need to consider, and if you can present it in a logical way, chances are they will, and they'll take that into consideration. So, I would—like, even if they are going to still move forward with the plans that you've somehow don't agree with, like, let's assume that some portion of this change, you don't feel is correct, which is actually one of the most legitimate reasons to worry about this, then what you should do is say, “Okay, look, I have this concern, so here's the Plan B. But just in case, this doesn't work. But I think it might not, so here's a Plan B.”Like, that's a way of presenting that in a way that's not challenging to the situation. So, I'll give you an example. In the India conversations, one of the things was that I actually did create a Plan B around was the fact that the person was bringing up—I was attempting to have Agile teams where they needed to have very strong ownership, they also needed to be able to self-manage. We talked about self-managed teams in Agile. And India is a very hierarchical culture, and so the thing that they brought up with me is that this isn't going to work here; it culturally isn't a good fit.And frankly, I knew that I was going to—I had this issue it within the company, but was it so widespread within India that I couldn't possibly change it? I hadn't lived there my whole life, I couldn't say, right? So, I needed to actually answer that question. And I thought it was a legitimate question, right? And I thought—but it was presented in, you know, a very factual, logical way, and kind of without the emotions, and so it's like, “Okay, let me think through that.”And so, we did this as a—you know, we created an experimental team where we tried this out to see if it would work. And it actually did, ultimately, succeed with that team. And I love this team because —I mean, to be fair, I did handpick who went on this team. Like, I did, you know, try to pick people who I thought might be the most likely to succeed. I'm not crazy; I did want it to work, and so you know, I did sort of seed it a bit.But at the same time, when they came out of that—and they tend to be a little bit younger than I think some of the, you know—because I think their minds were a little bit more open as part of that, but they came out of that, and after about nine sprints, you started to see the junior engineers challenging the more senior engineers, which in India is not like something that you see all that often. They were also able to —the junior engineers were having opinions, they were contributing to the technical discussions. Like, it was actually a pretty radical shift. But they also kind of walked around with this, like, certain swagger that I cannot describe. But it was, like, super fun to watch.So, you know, you've got to see that this was actually going to work, and it could work. And then it became a really good example, for the rest. So, I think the main thing is to help mitigate risk. If you have a real concern over a change that's coming your way, and it's something you don't feel like the company should do, just understand that they may do it instead and that's not personal, but at the same time, you know, you can help by offering a Plan B or some risk mitigation to double-check that it is going to work or to help it work.Julie: Absolutely. It's kind of that whole testing hypothesis, right? We're going to see if this works; we're going to evaluate it. One of the things that you brought up that I love and it was something that when I was at PagerDuty, we used to talk about a lot with the postmortem process, which was to involve junior engineers because they tend to look at things differently with that fresh set of eyes.Natalie: Right.Julie: And they kind of get us a little bit—the people who've been doing it for a very long period of time—a little bit out of your comfort zone because all of a sudden, maybe you're having to explain something. Jason and I have talked about this a few more times probably than necessary, but just, “Well, we've always done it this way because…” and then having to explain that because. You know, one of the things that I find interesting just from your background is—you know, we've talked about this, where you scaled that engineering team from 0 to 100, to deliver on custom software engineering contracts, and you've done quite a few things over your career. I mean, even working at Oracle—which we were actually just talking about an Oracle outage this morning—but, driving technical programs. And that seems to be a lot of your background. I mean, even at Facet, that you introduced engineering best practices to standardize code reviews and improve test coverage. Do you want to talk a little bit about that?Natalie: Yeah, I think—I like this—I call it the adolescent version of engineering. It's where, you know, we're through the baby part, we need to start to grow up a little bit, we need to go from getting stuff done in some way or another, to something that's repeatable and scalable. And so, it's like, that adolescent years, that's my fun. That's what I enjoy doing. I call it creating something out of chaos.Basically, taming the chaos is what it really looks like because it's very chaotic initially, and that's true of every, like, small organization; they always start like that. And as they start to grow, you know, you've got ten different engineers who have ten different opinions on how something should be done, and so they do it ten different ways. And that's fine when you're only ten, but then when you need to go from 10 to 20 to 30 to 100, it no longer works. And you do have to create some standards and still leave enough leeway for people to be able to have their tool of choice based on, you know, what makes sense, right?So, there needs to be some pragmatism in there, you can't just, like, also go the [unintelligible 00:16:54] where it's just one thing. But at the same time, there is some standards and there is some consistency that needs to be created so that, like, when you're onboarding a new engineer, there's not 20 things to learn; you can reduce that down to something that's manageable and you can get somebody onboard and productive within a reasonable amount of time. Otherwise, that's difficult, even that becomes difficult. So, every part of it that needs to have some level of standards around it—I think the fun in it, too, is finding that balance between introducing enough process that you have some standardization, you have some consistency, but not so much that you slow it down to the point that it's no longer moving. Because you can; you can strangle a small organization with too much process.So, it's finding that middle ground. And yeah, that's what I've pretty much done, like, my whole career in some form or another; it's what I enjoy. And if it gets to the point where things become too standard, too stable, to done, then I'm probably… I'm going to need to move on to something different and new. You know, that's going to be where I go do this again, with somebody else.Julie: Hashtag #startuplife, right?Natalie: [laugh].Julie: [laugh]. That's interesting that you bring up, you know, going from ten people to more, right, where you can just buy any tool you want and reimburse it, and there might not even be a central repo of all the tools that the organization has, to whittling that down into processes that you own, that you control, versus processes that control you. And then bringing those ten people that were there at the beginning that could kind of do whatever they want because the whole goal is to bring this product to market, to refining that organization and helping build out features in service of the customer. So, when you're looking at the new things that you want to do or prioritizing your time or the engineering team's time, what are some of the things that you take into consideration?Natalie: It's kind of actually very similar to performance when you look at the performance of a system, right? The engineering organization is no different. You need to find your bottlenecks and then you work from there. And the bottlenecks are different depending on which team that you're looking at, right? So, I like to start to kind of get a feel for what's working, what's not working, and where things are slow, [unintelligible 00:19:15] oftentimes what I'm trying to do is to get some speed, to get some speed and consistency tend to be really big things without losing quality. You know, all of those kinds of—those are the always the buckets, right?And so, when you start looking at speed, it really starts to look very much like that performance bottleneck exercise where you just start hitting them one at a time until you, you know, you get through the easy ones and then you start tweaking from there. But for instance, I'll tell you when I first started with Gremlin, we had a very large team and because of that, stand-ups were very huge, there was too much conversation, they took too long, people —actually the odd thing is that you'll find people have less ownership when the team is too large because they don't feel like they're as part of something that they're making a huge —as much of an impact on; they don't feel their impact on a team that's too large, so when you're organized in such a way that the teams are very large, you tend to lose some of the qualities of Agile that you're trying to achieve when you're doing these little small Agile teams, or at least that's the thought. So, one of the things I did was split the team. And one of the first things that I did—and that automatically started to create a different dynamic within the teams, and we're starting to see the results of that. And so, I feel like those are the kinds of things that you do.Like, that was an easy one; we have to do this, like, that first. Now, like, what do we do next? It depends. It depends, like, where, like, in some cases—I'll take India, for example—there was a lot of tech debt. So, I had some tech debt that I had to contend with and deal with that was—the way it was built, it was built with this very huge monolithic-style service, and I needed to help them start breaking that into smaller services, mainly because—and they were such a large team, and it was still a monolithic sort of situation, the problem was actually more so than the performance because they had tuned the heck out of that, so that wasn't it.Like, the data was very large, so they had already dealt with performance. But the conflict within the engineering teams was a lot because there was so much coordination. And so, by being able to split this up into services that make sense, then the teams can start to own the services and be able to deliver on that with some speed without having to coordinate so much. And every moment of coordination costs you time, right? So, that's the type of things that you start to look at.And it could be a technical solution, like in this case, it was breaking the technology, from an architectural standpoint, down into something that make the teams operate differently, or it can be splitting the teams itself without changing the architecture. It can be any number of things. But really start to have to look at what's causing this to go slow.Julie: Now, I love that because when everybody owns everything, nobody owns anything, right? And you talked about breaking the teams down into service teams that makes sense. And so, it sounds like it was incredibly intentional; owning your services all the way through into production is really helpful with that speed and that quality. And you mentioned that briefly earlier, which is—what is that? The iron triangle, or whatever they call it, but speed, cost, quality. There's three things; you can only have two. Which two do you pick?Natalie: Right. [laugh]. Exactly.Julie: And I've seen that titled as a fallacy saying that you can really have all three, but I don't really know. What do you think? Speed, cost, quality, can you have all three?Natalie: Well, so you can maybe have speed, cost, and quality, but if you throw scope in there, [laugh] and you throw that into your [unintelligible 00:22:41], right? Like, because [unintelligible 00:22:42] where you have to start throwing that in. Like, if you look at—so, you know, the triangle that we tend to look at is the time that you're going to deliver it in, the scope, and the price. Those are the three that I think you can only hold two of. You can go—so by speed when you say speed, cost, and quality, if you go back to your you know, your original one, depends on what how you define speed on whether or not you get quality out of that, right? [laugh].And so, when you say—but when you start putting deadlines on things, then yeah, you can get quality so long as I can control the scope, right? Because then I can scope it down enough that I can deliver something within that timeline that is of high quality, right? So, those are the trade-offs that you have to make? And no I don't —I still feel like in that particular three-legged stool, you know, there's only two of those you get, that somebody else outside of your organization can handle. You do have to —otherwise, you know, you can't possibly deliver everything in the world within a really short timeframe and expect the quality to be high.Julie: Yeah, wouldn't that be nice if you could, right? But that's why we talk about learning from our failures. That's why we talked about Chaos Engineering and understanding our systems. Because in all reality, we do have timeframes that we need to get things out, and we have to make our systems as reliable as possible. But then where do we find the gaps that we may have missed because of speed, because of that timeliness?Natalie: Well, and when you start looking at things like, you know, quality, there's certainly things that you can do, but if you go back to Chaos Engineering—we talk about that for just a second, and we look at the changes that people are afraid of. What happens when you go in and you tell a place, “To improve your quality I'm going to actually start shutting down your host.” They're like, “I'm sorry, what?” [laugh].Julie: [laugh].Natalie: That's a very difficult conversation, right? So, I feel like it's one of those things where once you see that and why you would do it and then, like, you make the adjustments to that, and then it becomes a part of your—doing this sort of change is actually, you know, something that you just do on a continuous basis; it's no longer something that you're afraid of, right? And I think that's true of just [unintelligible 00:24:48] in general. Like, you know, once you start getting into the habit of it, whatever that habit might be—and automation, by the way, is one of those things—and whether it be automating regular tests, whether it be automating Chaos Engineering tests, like any of this automation, that's actually a key to speed with engineering. And the reason for that is because those are so closely linked.I go back and I talk about automation and confident mindset. This is really the two things that give you speed in engineering organization. And the reason is because if you can automate it enough, you can—you know, obviously there's just some speed that comes from automation, you know, that you're not doing things manually, that's great. But the thing that you miss in that, or that you don't necessarily think of, is the fact that there, like, an automated safety net under you, like, through testing, through, like, you know, the systems-level testing, Chaos Engineering, you know, the engineers now feel more free, they're more confident, they're able to make changes at a much more rapid pace. It feels less risky because they're able to make this change and then they know that the tests are going to catch them, right?So, if they've screwed something up, something else is going to stop it before it heads to production. So, they're just more—they're able to just move forward at a faster pace than they would otherwise, right? So, that automation, the speed that you get out of it goes far beyond just you taking the manual process down to an automated one; it's creating the safety net that gives them the confidence to just move without thinking. And that's huge. Like, that's a big deal.It's also—back to your thoughts on junior engineers—it's also why I think it's really important to make sure there's people in the engineering team who [unintelligible 00:26:26] three years, like, three years of experience. It's like you know enough that you can make really good progress and you can be useful, but you don't know so much that you're afraid. Like, there—laugh] because that confident mindset I'm back to, it really matters. Like, it makes such a big difference in the teams that will move quickly and teams that will not.Julie: I love everything that you just said. And I just saw a tweet from Kelsey Hightower that he tweeted just a couple of days ago; I saw it just before we recorded this. So, he said, “…as an industry we've been pushing… Automate. Automate. Automate. And we haven't been saying… Understand. Understand. Understand. Because if you understand what you're doing, you can automate it if you want to.”And I think you just touched on that. And I think you touched on a lot of the having confidence, that what you're doing—that there's safety and even if there are failures, that they're going to be caught. And I think that all ties together beautifully. Now, with that, because I do realize that we are running out of time, I just want to say, so for you, you are giving the closing keynote today at DevOpsDays Boise. And we've talked a lot about overcoming fear during this podcast, and I know that this was something that made you a little bit uncomfortable. Can you tell me why you chose to do this? Why did you choose to overcome this fear?Natalie: Because of my position and the fact that I'm female, I get offers. And I just made a deal with myself about, you know, a few months ago that said, you know, I wouldn't turn these down. And primarily it's because I feel like it's important that at least some women are out there and are serving as examples for others. Like, I'm not saying that I'm going to have, like, the best things to say all the time, and I think that's okay. I don't think every man that comes on a podcast has the best things to say either, right?So, I feel like it's just one of those situations where we need examples for ourselves, and I think it's important that, you know, we see ourselves in the—in what's—in what's, I guess, the speakers and the participants, right? And so, I want to make sure that I do my part in that, I guess.Julie: Well, thank you. And you heard it here first, folks. If you need Natalie to speak at your conference, she made a deal with herself [laugh] that she would not say no. We're really excited to have you both on the podcast and speaking at DevOpsDays Boise. So, thank you, Natalie, and thank you for joining us on Break Things on Purpose. And good luck on your talk today.Natalie: Thank you. Appreciate it. Enjoyed it. [laugh].Julie: Have a wonderful day.Natalie: You too.Jason: For links to all the information mentioned, visit our website at gremlin.com/podcast. If you liked this episode, subscribe to the Break Things on Purpose podcast on Spotify, Apple Podcasts, or your favorite podcast platform. Our theme song is called “Battle of Pogs” by Komiku, and it's available on loyaltyfreakmusic.com.

In this episode, we cover:00:00:00 - Introduction00:00:57 - Rootly, an incident management platform 00:02:20 - Why build Rootly00:06:00 - Unique aspects of Rootly00:09:50 - How people should use Rootly Links Referenced:rootly.com/demo: https://rootly.com/demo TranscriptJJ: How do you now get this massive organization to change the way that they work? Even if they were following, like, a checklist and Google Docs, that still marks as a fairly significant cultural change, and so we need to be very mindful of it.Jason: Welcome to another episode of Build Things on Purpose, part of the Break Things on Purpose podcast. In our build episodes, we chat with the engineers and developers who create tools that help us build modern applications, or help us fix them when they break. In this episode, JJ Tang, co-founder of Rootly, joins us to chat about incident response, the tool he's built, and the lessons he's learned from incidents.So, in this episode, we've got with us JJ Tang, who's the co-founder of a company and a tool called Rootly, welcome to the show.JJ: Thank you, Jason, super excited to be here. Big fan of what you guys are doing over at Gremlin and all things Chaos Engineering. Quick intro on my side. I'm JJ, as you mentioned. We are building Rootly, which is an incident management platform built on top of Slack.So, we help a bunch of different companies automate what we believe to be some of the most manual and tedious work when it comes to incidents, like creating virtual war rooms, Zoom Bridges, tracking your action items on Jira, generating your postmortem timeline, adding the right responders, and generally just helping build that consistency. So, we work with a bunch of different fast-growing tech companies like Canva, Grammarly, Bolt, Faire, Productboard, and also some of the more traditional ones like Ford and Shell. So, super excited to be here. Hopefully, I have some somewhat engaging insight, I hope. [laugh].Jason: Yeah, I think you will because in our discussions previously, we've always had fantastic conversations. So, you've kind of covered a lot of the first question that I normally ask, and that's what did you build? And so as you explained, Rootly is an incident management tool; works with Slack. But that naturally leads into the other question that I asked our Build Things guests, and that's why did you build this? Was it something from your experience as an engineer that you're just like, “I need a tool to solve this?” What's the story behind Rootly?JJ: Yeah, definitely. Sorry to jump the gun on the first question. I was a little bit too excited, I think. But yeah, so my co-founder, and I—his name is Quinton—we both used to work at Instacart, the grocery delivery startup. He was there super, super early days; he was actually one of the first SREs there and kind of built out that team.And I was more on the product side of things, so I helped us build out our enterprise and last-mile delivery products. If you're curious what does [laugh] grocery have to do with reliability, actually, not that much, but the challenges we were dealing with were at very great scale. So, it all started back when the pandemic first started getting kicked off. Instacart was growing rapidly at the time, we were scaling really well, we were heading the numbers where we want it to be, but with suddenly the lockdowns occurring, everyone overnight who didn't care about grocery delivery and thought, “Well, why don't I just drive to Walmart,” [laugh] suddenly wanted to order things on Instacart. So, the company grew 5, 600%, nearly overnight.And with that, our systems just could not handle the load. And it'd be the most obscure incidents you wouldn't think would break, but under such immense stress and demand, we just couldn't keep the site up all the time. And what that really exposed on our end was, we don't have a really good incident management process. What we were doing was, we kind of just had every engineer in a single incident channel on Slack. And if you got paged, you just kind of ping in there. “I just got woken up. Did anyone else? Does this look legit?”And there was no formal way, so there was no consistency in terms of how the incidents were created. And then, of course, from that top-of-funnel into the postmortem, there wasn't too much discipline there. So, we really thought about, you know, after the dust kind of settled, there must be a better way to do this. And like most organizations that we work with, you start thinking about how can I build this myself?I think there's probably a little bit of a gap right now in this space. People generally understand monitoring tools really well, like New Relic, Datadog, alerting tools super well, PagerDuty, Opsgenie, they do a really good job at it. But everything afterwards, the actual orchestration and learning from the incidents tends to be a little bit sparse. So, we started embarking on our own. And for my co-founder's side of things, he was more at the heart of the incident than I was. I think I was the one complaining about and breathing down his neck a little bit about why things [laugh] sometimes weren't working.And—yeah, and, you know, as we started thinking about internal solutions, we took a step back and thought, “Well, you know, if Instacart is facing this problem then I think a lot of companies must be as well.” And luckily, our hypothesis has proven to be true, and yeah, the rest is just history now.Jason: That's really fascinating, particularly because, I mean, it is such a widespread issue, right? And I think I've experienced that as well, where you've got a general on-call or incidents channel, and literally everybody in the organization's in there, not just engineers, but—like yourself—product people and customer success or support folks are all in there. And the idea is this, sort of—it's a giant, giant crowd of folks who are just, like, waiting and wondering. And so having a tool to help manage that is extremely useful. As you started building out this tool, I'm starting to think there are starting to become a lot more incident management tools or incident response management tools, so talk to me about what are the unique points about Rootly?Because I suspect that a lot of it is influenced from, “These are the pain points that I had during my incidents,” and so you pulled them over? And so I'm curious, what are those that you brought to the tool that really help it shine during an incident?JJ: Yeah, definitely. I think the space that we're in right now is certainly heating up as you go to the different conferences and the content that's put out there. Which is great because that means everyone is educating the broader audience of what's going on and just makes my job just a little bit easier. There's a couple, you know, original hypothesis that we had for the product that just ended up not being as important. And that has really defined how we think about Rootly and how we differentiate a lot of what we do.How we did incidents at Instacart wasn't all that unique, you know? We used the same tools everyone else did. We had Opsgenie, we used Slack, Datadog, Jira, we wrote our postmortems on Confluence, stuff like that, and our initial reaction was, “Well, people are using the same tools, they must be following a very similar process.” And we also looked and worked a lot with people that are deep into the space, you know, Google, Stripe, the Airbnbs of the world, people that have a very formal process. And so we actually embarked on this journey building a relatively opinionated tool; “This is how we think the best incidents can be run.” And that actually isn't the best fit for everyone.I think if you had no incident management process whatsoever, that's great. You know, we give you super powerful defaults out of the box, like we do today, and you kind of can just hit the ground running super fast. But what we found is despite everyone using basically the same kind of tools, the way they use it is super different. You might only want to create a Zoom Bridge for, you know, high severity incidents, whereas someone else wants to create it for every single incident, for example. So, what we did was really focus on how do we balance between building something that's opinionated versus flexible, where should customers be able to turn the knobs and the dials.And a big part of it is we built what we call our workflows, and that allows customers to create a process that it's very similar to theirs. And a part of that we didn't anticipate at the very beginning was, although the tool is super simple to use, I think or average install time is probably 13 minutes, all the integrations and everything on a quick call with our customers, the really heavy lifting comes with, how do you now get this massive organization to change the way that they work? Even if they were following, like, a checklist in Google Docs, that still marks as a fairly significant cultural change, and so we need to be very mindful of it. So, we can't be just ripping tools out of their existing stack, we can't be wildly changing every process; everything has to happen progressively, almost, in a way. And that is a lot more digestible than saying you're going to replace everything.So, I think that's probably one of the key differences is we tend to lean more on the side of playing with your existing stack versus changing everything up.Jason: That's a really good insight, particularly because coming from Chaos Engineering, and that is almost entirely changing the way that people work, right, is Chaos Engineering is a new practice, so I definitely empathize with you, or sympathize with you on that struggle of, like, how do you change what people are doing and really get them to embrace it? That said, being opinionated is also a really good thing because you have a chance to lead people, and so that leads me to our final question that we always ask folks—and this is where being opinionated is good—but if folks were to use Rootly, or just even wanted to improve their incident response processes in general, what are some of those opinions that you had about how people should be doing that, that they should consider embracing?JJ: Yeah, that's an awesome question. So, a couple things, a little bit related to your second question that we initially thought but just proved to not be as important for us, everything that we build at the beginning—and still build—is relatively laser-focused on helping you get to that resolution as fast as possible. But from an organizational perspective, what we found is, people don't think about incident management success as how quickly they can resolve an incident. A lot of it's actually just having that security and framework and consistency around the incident. So ironically, as a tool in incident management, the most important things are actually around your people and the process and the culture that you can develop around the tool.No matter how good of something that we build, you know—let's say you're an organization, you just bring in Rootly, you have a very blameful way of handling postmortems, no one generally understands how severities in organization work, you're super laser-focused on, you know, tracking MTTR, which can not always be the best metric, but you still want to interpret it as such, it's very difficult to make the tool successful. So, that's the biggest advice that we give to our customers is when we see those type of red flags from, like, a process and culture standpoint, we'll try to guide them the best that we can. And we'll also do it from a product perspective. What you get out of the box today, we have companies as small as, you know, 20 for example, just kind of being able to hit the ground running; they'll use workflow templates that are pre-built based on some best practices that we've seen to just kind of layer in that framework. So, I think that would be a really big one that we've noticed is it's not all about us; it's not all about the product and the benefits that we can provide; it's about how we can actually enable our customers to get to that stage.Jason: I love that answer. Well, JJ, thanks for being a guest on the show and sharing a bit more about your journey and the journey of Rootly. If folks are interested in trying out the product and getting better at incident response, where can they find more info about you and about Rootly?JJ: Yeah. You can just visit rootly.com/demo. We do offer a 14-day trial if you want to sign up for free.If you want to talk to one of us or partnerships team, you're welcome to book a personalized session. I recommend that because then you get to see my super cute dog that isn't with me right now and wouldn't matter because this is audio only, but I love showing her off. That's my favorite part of my job.Jason: So, if you want to go see JJ's dog, or learn more about Rootly and incident management, go check it out. Thanks again.JJ: Yeah, thanks for having me.Jason: For links to all the information mentioned, visit our website at gremlin.com/podcast. If you liked this episode, subscribe to the Break Things on Purpose podcast on Spotify, Apple Podcasts, or your favorite podcast platform. Our theme song is called “Battle of Pogs” by Komiku, and it's available on loyaltyfreakmusic.com.

In this episode, we cover: Introduction (00:00) Elizabeth, AppLand, and AppMap (1:00) Why build AppMap (03:34) Being open-source (06:40) Building community  (08:50) Some tips on using AppMap (11:15) Links Referenced: VS Code Marketplace: https://marketplace.visualstudio.com/items?itemName=appland.appmap JetBrains Marketplace: https://plugins.jetbrains.com/plugin/16701-appmap AppLand: https://appland.com TranscriptElizabeth: “Whoa.” [laugh]. That's like getting a map of all of the Planet Earth with street directions for every single city, across all of the continents. You don't need that; you just want to know how to get to the nearest 7/11, right? Like, so just start small. [laugh]. Don't try and map your entire universe, galaxy, you know, out of the gate. [laugh].Jason: Welcome to another episode of Build Things on Purpose, part of the Break Things on Purpose podcast. In our build episodes, we chat with the engineers and developers who create tools that help us build and operate modern applications. In this episode, Elizabeth Lawler joins us to chat about the challenges of building modern, complex software, and the tool that she's built to help developers better understand where they are and where they're going.Jason: Today on the show, we have Elizabeth Lawler who's the founder of a company called AppLand, they make a product called AppMap. Welcome to the show, Elizabeth.Elizabeth: Thank you so much for having me, Jason.Jason: Awesome. So, tell us a little bit more about AppLand and this product that you've built. What did you build?Elizabeth: Sure. So, AppMap is a product that we're building in the open. It's a developer tool, so it's free and open-source. And we call it Google Maps for code. You know, I think that there has been a movement in more assistive technologies being developed—or augmenting technologies being developed for developers, and with some of the new tools, we were looking to create a more visual and interactive experience for developers to understand the runtime of their code better when they code.So, it's interesting how a lot of the runtime of an application when you're writing it or you're actually crafting it is sort of in your imagination because it hasn't yet been. [laugh]. And so, you know, we wanted to make that information apparent and push that kind of observability left so that people could see how things were going to work while they're writing them.Jason: I love that idea of seeing how things are working while you're writing it because you're so right. You know, when I write code, I have a vision in mind, and so, like, you mentally kind of scaffold out here are the pieces that I need and how they'll fit together. And then as you write it, you naturally encounter issues, or things don't work quite as you expect, and you tweak those. And sometimes that idea or the concept in your head gets a little fuzzy. So, having a tool that actually shows you in real-time seems like an extremely valuable tool.Elizabeth: Thank you. Yes. And I think you've nailed how it's not always the issue of dependency, it's really the issue of dependent behavior. And that dependent behavior of other services or code you're interacting with is the hardest thing to imagine while you're writing because you're also focusing on feature and functionality. So, it's really a fun space to work in, and crafting out that data, thinking about what you would need to present, and then trying to create an engaging experience around that has been a really fun journey that the team has been on since 2020. We announced the project in 2021 in March—I think almost about this time last year—and we have over 13,000 users of AppMap now.Jason: That's incredible. So, you mentioned two things that I want to dive into. One is that it's open-source, and then the second—and maybe we'll start there—is why did you build this? Is this something that just was organic; you needed a tool for yourself, or… what was the birth of AppMap?Elizabeth: Oh, I think that's such a great question because I think it was—this is the third startup that I've been in, third project of this kind, building developer tooling. My previous company was a cybersecurity company; before that, I helped build applications in the healthcare sector. And before that, I worked in government and healthcare. And—also, again, building platforms and IT systems and applications as part of my work—and creating a common understanding of how software operates—works—understanding and communicating that effectively, and lowering that kind of cognitive load to get everybody on the same page is such a hard problem. I mean, when we didn't all work from home, we had whiteboards [laugh] and we would get in the room and go through sprint review and describe how something was working and seeing if there was anything we could do to improve quality, performance, reliability, scalability, functionality before something shipped, and we did it as a group, in-person. And it's very difficult to do that.And even that method is not particularly effective because you're dealing with whiteboards and people's mental models and so we wanted to, first of all, create something objective that would show you really how things worked, and secondly, we wanted to lower the burden to have those conversations with yourself. Or, you know, kind of rubber ducky debugging when something's not working, and also with the group. So, we created AppMaps as both interactive visualizations you could use to look at runtime, debug something, understand something better, but also something that could travel and help to make communication a lot easier. And that was the impetus, you know, just wanting to improve our own group understanding.Jason: I love that notion of not just having the developer understand more, but that idea of yeah, we work in teams and we often have misalignment simply because people on different sides of the application look at things differently. And so this idea of, can we build a tool that not only helps an individual understand things, but gets everybody on the same page is fantastic.Elizabeth: And also work in different layers of the application. For example, many observability tools are very highly focused on network, right? And sometimes the people who have the view of the problem, aren't able to articulate it clearly or effectively or expeditiously enough to capture the attention of someone who needs to fix the problem. And so, you know, I think also having—we've blended a combination of pieces of information into AppMap, not only code, but also web services, data, I/O, and other elements and so that we can start to talk more effectively as groups.Jason: That's awesome. So, I think that collaboration leads into that second thing that I brought up that I think is really interesting is that this is an open-source project as well. And so—Elizabeth: It is.Jason: Tell me more about that. What's the process? Because that's always, I think, a challenge is this notion of we love open-source, but we're also—we work for companies, we like to get paid. I like to get paid. [laugh]. So, how does that work out and what's that look like as you've gone on this journey?Elizabeth: Yeah. You know, I think we think quietly working are certainly looking for other fellow travelers who are interested in this space. We started by creating an open data framework—which AppMap is actually both the name of a code editor extension you can install and use to see the runtime of your code to understand issues and find a fix them faster, but it also is a data standard. And with that data standard, we're really looking to work with other people. Because, you know, I think this type of information should be widely accessible for people and I think it should be available to understand.I think, you know, awareness about your software environment is just kind of like a basic developer right. And so, [laugh] you know, the reason why we made the tools free, and the reason why we've made the data structure open-source is to be able to encourage people to get the kind of information that they need to do their job better. And by making our agents open-source, by making our clients open-source, it simply allows people to be able to find and adopt this kind of tooling to improve their own job performance. And so, you know, that was really kind of how we started and I think, ultimately, you know, there are opportunities to provide commercial products, and there will be some coming down the road, but at the moment, right now we're really interested in working with the community and, you know, understanding their needs better.Jason: That's awesome. Number one, I love the embrace of, you know, when you're in the startup land, there's the advice, have never tried to monetize too early, right? Build something that's useful that people enjoy and really value, and then it'll naturally come. The other question that I had is, I'm assuming you eat your own dog food, slash drink your own champagne. So, I'm really curious, like, one of the problems that I've had in open-source is the onboarding of new community members, right? Software is complex, and so people often have troubles, and they're like, how do I fix this? They file an issue on GitHub or whatever system you're using, and there's sometimes a notion with open-source of like, that's a good thing that you called out. You can fix that because it's open-source, but people are like, “I don't know how.”Elizabeth: Yeah.Jason: Does AppMap actually help in enabling AppMap open-source contributors? Like, have you seen that?Elizabeth: So, we've had issues filed. I would say that most of the fixes still come from us. If people wanted to run AppMap on AppMap to identify the bug, [laugh] that would be great, but it doesn't really work that way. So, you know, for us at this time, most of it is community filed issues and that we are working to resolve. But I do think—and I will say—that we have actually used AppMap on open-source projects that we use, and we've found [laugh] flaws and bugs using AppMap with those projects, and have filed issues with them. [laugh].Jason: That's awesome. I love that. I mean, that's what it means to be an open-source, right, and to use open-source is that notion of, like—Elizabeth: Right.Jason: Contribute wherever you can.Elizabeth: Yeah. And if that's the way, you know, we can contribute, you know—and I think similarly, I mean, our relationship to open-source is very strong. So, for example, you know, we came from the Ruby community and there's lots of different kinds of open-source projects that are commonly used for things like security and authentication and we've done a lot of work in our own project to tag and label those commonly-used libraries so that they can be—when you pop open an AppMap everything is all beautiful and tagged and, you know, very nicely and neatly organized for you so you can find everything you're looking for. Similarly, we're working with open-source communities in Python and Java and now JavaScript to do the same thing, which is, you know, to make sure that important information, important commonly used libraries and tools are called out clearly.Jason: So, as you're adding more languages, you're going to get more users. So, that brings me to our final question. And that's, as you get all these new users, they probably need some guidance. So, if you were to give some users tips, right? Someone goes out there, like, “I want to use AppMap,” what's some advice that you'd give them related to reliability? How can they get the best experience and build the best code using AppMap?Elizabeth: Yes. So, this has actually been a key piece of feedback, I think, from the community for us, which is, we released this tool out to the world, and we said, “We're going to bring here; we come with gifts of observability in your code editor.” And people have used it for all kinds of different projects: They've used it for refactoring projects, for debugging, for onboarding to code, for all of these different use cases, but one of the things that can be overwhelming is the amount of information that you get. And I think this is true of most kinds of observability tools; you kind of start with this wall of data, and you're like, “Where am I going to start?”And so my recommendation is that AppMap is best used when you have a targeted question in mind, not just kind of like, you know, “I'd like to understand how this new piece of the codebase works. I've shifted from Team A to Team B, and I need to onboard to it.” “I'd like to figure out why I've got a slow—you know, I've been told that we've got a slowdown. Is it my query? Is it my web service? What is it? I'd like to pinpoint, find, and fix the issue fast.”One of the things that we're doing now is starting to leverage the data in a more analytic way to begin to help people focus their attention. And that's a new product that we're going to be bringing out later this spring, and I'm very, very excited about it. But I think that's the key, which is to start small, run a few test cases that are related to the area of code that you're interested in if that's an onboarding case, or look for areas of the code you can record or run test cases around that is related to the bug you have to fix. Because if you just run your whole test suite, you will generate a giant amount of data. Sometimes people generate, like, 10,000 AppMaps on the first pass through. And they're like, “Whoa.” [laugh]. That's like getting a map of all of the Planet Earth with street directions for every single city, across all of the continents. You don't need that; you just want to know how to get to the nearest 7/11, right? Like, so just start small. [laugh]. Don't try and map your entire universe, galaxy, you know, out of the gate. [laugh].Jason: That's fantastic advice, and it sounds very similar to what we advise at Gremlin for Chaos Engineering of starting small, starting very specific, really honing in on sort of a hypothesis, “What do I think will happen?” Or, “How do I think I understand things?” And really going from there?Elizabeth: Yeah. It does, it focuses the mind to have a specific question as opposed to asking the universe what does it all mean?Jason: Yeah. Well, thanks for being a guest on the show today. Before we go, where can people find AppMap if they're interested in the tool, and they want to give it a try?Elizabeth: So, we are located in the VS Code Marketplace if you use the VS Code editor, and we're also located in JetBrains Marketplace if you use any of the JetBrains tools.Jason: Awesome. So yeah, for our VS Code and JetBrains users, go check that out. And if you're interested in more about AppMap or AppLand, where can folks find more info about the company and maybe future announcements on the analysis tooling?Elizabeth: That would be appland.com A-P-P-L-A-N-D dot C-O-M. And our dev docs are there, new tooling is announced there, and our community resources are there, so if anyone would like to participate in either helping us build out our data model, feedback on our language-specific plans or any of the tooling, we welcome contributors.Jason: Awesome. Thanks again for sharing all of that info about AppMap and AppLand and how folks can continue to build more reliable software.Elizabeth: Thank you for having me, Jason.Jason: For links to all the information mentioned, visit our website at gremlin.com/podcast. If you liked this episode, subscribe to the Break Things on Purpose podcast on Spotify, Apple Podcasts, or your favorite podcast platform. Our theme song is called “Battle of Pogs” by Komiku, and it's available on loyaltyfreakmusic.com.

In this episode, we cover: Introduction (00:00) How Chris got into the world of chaos and teaching middle school science (02:11) The Cengage seasonal model and preparing for the (5:56) How Cengage schedules the chaos and the “day of darkness” (11:10) Scaling and migration and “the inches we need” (15:28) Communicating with different teams and the customers (18:18) Chris's biggest lesson from practicing chaos engineering (24:30) Chris and working at Cengage/Outro (27:40) Links Referenced: Cengage: https://www.cengagegroup.com/ Chris Martello on LinkedIn: https://www.linkedin.com/in/christophermartello/ TranscriptJulie: Wait, I got it. You probably don't know this one, Chris. It's not from you. How does the Dalai Lama order a hot dog?Chris: He orders one with everything.Julie: [laugh]. So far, I have not been able to stump Chris on—[laugh].Chris: [laugh]. Then the follow-up to that one for a QA is how many engineers does it take to change a light bulb? The answer is, none; that's a hardware problem.Julie: Welcome to Break Things on Purpose, a podcast about reliability, quality, and ways to focus on the user experience. In this episode, we talk with Chris Martello, manager of application performance at Cengage, about the importance of Chaos Engineering in service of quality.Julie: Welcome to Break Things on Purpose. We are joined today by Chris Martello from Cengage. Chris, do you want to tell us a little bit about yourself?Chris: Hey, thanks for having me today, Julie, Jason. It's nice to be here and chat with you folks about Chaos Engineering, Chaos Testing, Gremlin. As Julie mentioned I'm a performance manager at Cengage Learning Group, and we do a fair amount of performance testing, both individual platforms, and coordinated load testing. I've been a software manager at Cengage for about five years, total of nine altogether there at Cengage, and worn quite a few of the testing hats, as you can imagine, from automation engineer, performance engineer, and now QA manager. So, with that, yeah, my team is about—we have ten people that coordinate and test our [unintelligible 00:01:52] platforms. I'm on the higher-ed side. We have Gale Research Library, as well as soft skills with our WebAssign and ed2go offerings. So, I'm just one of a few, but my claim to fame—or at least one of my passions—is definitely chaos testing and breaking things on purpose.Julie: I love that, Chris. And before we hear why that's your passion, when you and I chatted last week, you mentioned how you got into the world of QA, and I think you started with a little bit of different type of chaos. You want to tell us what you did before?Chris: Sure, even before a 20-year career, now, in software testing, I managed chaos every day. If you know anything about teaching middle school, seventh and eighth-grade science, those folks have lots of energy and combine that with their curiosity for life and, you know, their propensity to expend energy and play basketball and run track and do things, I had a good time for a number of years corralling that energy and focusing that energy into certain directions. And you know back, kind of, with the jokes, it was a way to engage with kids in the classroom was humor. And so there was a lot of science jokes and things like that. But generally speaking, that evolved into I had a passion for computers, being self-taught with programming skills, project management, and things like that. It just evolved into a different career that has been very rewarding.And that's what brings me to Cengage and why I come to work every day with those folks is because instead of now teaching seventh and eighth-grade science to young, impressionable minds, nowadays I teach adults how to test websites and how to test platforms and services. And the coaching is still the same; the mentoring is still the same. The aptitude of my students is a lot different, you know? We have adults, they're people, they require things. And you know, the subject matter is also different. But the skills in the coaching and teaching is still the same.Jason: If you were, like, anything like my seventh-grade science teacher, then another common thing that you would have with Chaos Engineering and teaching science is blowing a lot of things up.Chris: Indeed. Playing with phosphorus and raw metal sodium was always a fun time in the chemistry class. [laugh].Julie: Well, one of the things that I love, there are so many parallels between being a science teacher and Chaos Engineering. I mean, we talk about this all the time with following the scientific process, right? You're creating a hypothesis; you're testing that. And so have you seen those parallels now with what you're doing with Chaos Engineering over there at Cengage?Chris: Oh, absolutely. It is definitely the basis for almost any testing we do. You have to have your controlled variables, your environment, your settings, your test scripts, and things that you're working on, setting up that experiment, the design of course, and then your uncontrolled variables, the manipulated ones that you're looking for to give you information to tell you something new about the system that you didn't know, after you conducted your experiment. So, working with teams, almost half of the learning occurs in just the design phase in terms of, “Hey, I think this system is supposed to do X, it's designed in a certain way.” And if we run a test to demonstrate that, either it's going to work or it's not. Or it's going to give us some new information that we didn't know about it before we ran our experiment.Julie: But you also have a very, like, cyclical reliabilities schedule that's important to you, right? You have your very important peak traffic windows. And what is that? Is that around the summertime? What does that look like for you?Chris: That's right, Julie. So, our business model, or at least our seasonal model, runs off of typical college semesters. So, you can imagine that August and September are really big traffic months for us, as well as January and part of February. It does take a little extra planning in order to mimic that traffic. Traffic and transactions at the beginning of the semester are a lot different than they are at the middle and even at the end of the semester.So, we see our secondary higher education platforms as courseware. We have our instructors doing course building. They're taking a textbook, a digitized textbook, they're building a course on it, they're adding their activities to it, and they're setting it up. At the same time that's going along, the students are registering, they are signing up to use the course, they're signing up to their course key for Cengage products, and they're logging into the course. The middle section looks a lot like taking activities and tests and quizzes, reading the textbook, flipping pages, and maybe even making some notes off to the side.And then at the end of the semester, when the time is up, quite literally on the course—you know, my course semester starts from this day to this day, in 15th of December. Computers being as precise as they are, when 15th of December at 11:59 p.m. rolls off the clock, that triggers a whole bunch of cron jobs that say, “Hey, it's done. Start calculating grades.”And it has to go through thousands of courses and say, “Which courses expired today? How many grades are there submitted? How many grades are unsubmitted and now I have to calculate the zeros?” And there's a lot of math that goes in with that analytics. And some of those jobs, when those midnight triggers kick off those jobs, it will take eight to ten hours in order to process that semester's courses that expire on that day.Julie: Well, and then if you experience an outage, I can only assume that it would be a high-stress situation for both teachers and students, and so we've talked about why you focus so heavily on reliability, I'd love to hear maybe if you can share with us how you prepare for those peak traffic events.Chris: So yeah, it's challenging to design a full load test that encompasses an entire semester's worth of traffic and even the peaks that are there. So, what we do is, we utilize our analytics that give us information on where our peak traffic days lie. And it's typically the second or third Monday in September, and it's at one or two o'clock in the afternoon. And those are when it's just what we've seen over the past couple of years is those days are our typical traffic peaks. And so we take the type of transactions that occur during those days, and we calibrate our load tests to use those as a peak, a one-time, our performance capacity.And then that becomes our x-factor in testing. Our 1x factor is what do we see in a semester at those peaks? And we go gather the rest of them during the course of the semester, and kind of tally those up in a load test. So, if our platforms can sustain a three to six-hour load test using peak estimate values that come from our production analysis, then we think we're pretty stable.And then we will turn the dial up to two times that number. And that number gives us an assessment of our headroom. How much more headroom past our peak usage periods do we have in order to service our customers reliably? And then some days, when you're rolling the dice, for extra bonus points, we go for 3x. And the 3x is not a realistic number.I have this conversation with engineering managers and directors all the time. It's like, “Well, you overblow that load test and it demonstrated five times the load on our systems. That's not realistic.” I says, “Well, today it's not realistic. But next week, it might be depending on what's happening.”You know, there are things that sometimes are not predictable with our semesters and our traffic but generally speaking it is. So, let's say some other system goes down. Single-sign-on. Happens to the best of us. If you integrate with a partner and your partner is uncontrolled in your environment, you're at their mercy.So, when that goes down, people stop entering your application. When the floodgates open, that traffic might peak for a while in terms of, hey, it's back up again; everybody can log in. It's the equivalent of, like, emptying a stadium and then letting everybody in through one set of doors. You can't do it. So, those types of scenarios become experimental design conversations with engineering managers to say, “At what level of performance do you think your platform needs to sustain?”And as long as our platforms can sustain within two to three, you know, we're pretty stable in terms of what we have now. But if we end up testing at three times the expected load and things break catastrophically, that might be an indication to an architect or an engineering director, that, hey, if our capacity outlives us in a year, it might be time to start planning for that re-architecture. Start planning for that capacity because it's not just adding on additional servers; planning for that capacity might include a re-architecture of some kind.Julie: You know, Chris, I just want to say to anybody from Coinbase that's out there that's listening, I think they can find you on [LinkedIn](https://www.linkedin.com/in/christophermartello/) to talk about load testing and preparing for peak traffic events.Chris: Yeah, I think the Superbowl saw one. They had a little QR code di—Julie: Yeah.Chris: —displayed on the screen for about 15 seconds or so, and boy, I sure hope they planned for that load because if you're only giving people 15 seconds and everybody's trying to get their phone up there, man I bet those servers got real hot real fast. [laugh].Julie: Yeah, they did. And there was a blip. There was a blip.Chris: Yeah. [laugh].Julie: But you're on LinkedIn, so that's great, and they can find you there to talk to you. You know, I recently had the opportunity to speak to some of the Cengage folks and it was really amazing. And it was amazing to hear what you were doing and how you have scheduled your Chaos Engineering experiments to be something that's repeatable. Do you want to talk about that a little bit for folks?Chris: Sure. I mean, you titled our podcast today, “A Day of Darkness,” and that's kind of where it all started. So, if I could just back up to where we started there with how did chaos become a regular event? How did chaos become a regular part of our engineering teams' DNA, something that they do regularly every month and it's just no sweat to pull off?Well, that Day of Darkness was 18 hours of our educational platforms being down. Now, arguably, the students and instructors had paid for their subscriptions already, so we weren't losing money. But in the education space and in our course creations, our currency is in grades and activities and submissions. So, we were losing currency that day and losing reputation. And so we did a postmortem that involved engineering managers, quality assurance, performance folks, and we looked at all the different downtimes that we've had, and what are the root causes.And after conferring with our colleagues in the different areas—we've never really been brought together in a setting like that—we designed a testing plan that was going to validate a good amount of load on a regular basis. And the secondary reason for coordinating testing like that was that we were migrating from data center to cloud. So, this is, you know, about five, six years ago. So, in order to validate that all that plumbing and connections and integrations worked, you know, I proposed I says, “Hey, let's load test it all the same time. Let's see what happens. Let's make sure that we can run water through the pipes all day long and that things work.”And we plan this for a week; we planned five days. But I traveled to Boston, gathered my engineers kind of in a war room situation, and we worked on it for a week. And in that week, we came up with a list of 90 issues—nine-zero—that we needed to fix and correct and address for our cloud-based offerings before it could go live. And you know, a number of them were low priority, easy to fix, low-hanging fruit, things like that. But there were nine of them that if we hadn't found, we were sure to go down.And so those nine things got addressed, we went live, and our system survived, you know, and things went up. After that, it became a regular thing before the semesters to make sure, “Hey, Chris, we need to coordinate that again. Can you do it?” Sure enough, let's coordinate some of the same old teams, grab my run sheet. And we learned that we needed to give a day of preparation because sometimes there were folks that their scripts were old, their environment wasn't a current version, and sometimes the integrations weren't working for various reasons of other platform releases and functionality implementation.So, we had a day of preparation and then we would run. We'd check in the morning and say, “Everybody ready to go? Any problems? Any surprises that we don't know about, yet?” So, we'd all confer in the morning and give it a thumbs up.We started our tests, we do a three-hour ramp, and we learned that the three-hour ramp was pretty optimal because sometimes elastic load balancers can't, like, spin up fast enough in order to pick up the load, so there were some that we had to pre-allocate and there were others that we had to give enough time. So, three hours became that magic window, and then three hours of steady-state at our peak generation. And now, after five years, we are doing that every month.Jason: That's amazing. One of the things you mentioned in there was about this migration, and I think that might tie back to something you said earlier about scaling and how when you're thinking of scaling, especially as I'm thinking about your migration to the cloud, you said, “Scaling isn't just adding servers. Sometimes that requires re-architecting an application or the way things work.” I'm curious, are those two connected? Or some of those nine critical fixes a part of that discovery?Chris: I think those nine fixes were part of the discovery. It was, you can't just add servers for a particular platform. It was, how big is the network pipe? Where is the DNS server? Is it on this side or that side? Database connections were a big thing: How many are there? Is there enough?So, there was some scaling things that hadn't been considered at that level. You know, nowadays, fixing performance problems can be as easy as more memory and more CPU. It can be. Some days it's not. Some days, it can be more servers; some days, it can be bigger servers.Other times, it's—just, like, quality is everybody's job, performance fixing is not always a silver bullet. There are things like page optimization by the designers. There's code optimization by your front-end engineers. And your back-end engineers, there are database optimizations that can be made: Indexing, reindexing on a regular basis—whatever that schedule is—for optimizing your database queries. If your front-end goes to an API for five things on the first page, does it make five extra calls, or does it make one call, and all five things come across at the same time?So, those are considerations that load performance testing, can tell you where to begin looking. But as quality assurance and that performance lead engineer, I might find five things, but the fixes weren't just more testing and a little bit of extra functionality. It might have involved DevOps to tweak the server connections, it might have involved network to slim down the hops from four different load balancers to two, or something like that. I mean, it was always just something else that you never considered that you utilized your full team and all of their expertise and skills in order to come up with those inches.And that's one of my favorite quotes from Every Given Sunday. It's an older football movie starring Al Pacino. He gives this really awesome speech in a halftime type of setting, and the punch line for this whole thing is, “The inches we need are everywhere around us.” And I tell people that story in the terms of performance is because performance, at the software level, is a game of inches. And those inches are in all of our systems and it's up to us as engineers to find them and add them up.Julie: I absolutely love everything about that. And that would have made a great title for this episode. “The Inches we Need are Everywhere Around Us.” We've already settled on, “A Day of Darkness with Chris Martello,” though. On that note, Chris, some of the things that you mentioned involve a lot of communication with different teams. How did you navigate some of those struggles? Or even at the beginning of this, was it easy to get everybody on board with this mindset of a new way of doing things? Did you have some challenges?Chris: There were challenges for sure. It's kind of hard to picture, I guess, Cengage's platform architecture and stuff. It's not just one thing. It's kind of like Amazon. Amazon is probably the example is that a lot of their services and things work in little, little areas.So, in planning this, I looked at an architecture diagram, and there's all these things around it, and we have this landscape. And I just looked down here in the corner. I said, “What's this?” They said, “Well, that's single-sign-on.” I says, “Well, everything that touches that needs to be load tested.”And they're like, “Why? We can't do that. We don't have a performance environment for that.” I said, “You can't afford not to.” And the day of darkness was kind of that, you know, example that kind of gave us the [sigh] momentum to get over that obstacle that said, “Yeah, we really do need a dedicated performance environment in order to prove this out.”So, then whittling down that giant list of applications and teams into the ones that were meaningful to our single-sign-on. And when we whittled that down, we now have 16 different teams that regularly participate in chaos. Those are kind of the ones that all play together on the same playing field at the same time and when we find that one system has more throughput than another system or an unexpected transaction load, sometimes that system can carry that or project that load onto another system inadvertently. And if there's timeouts at one that are set higher than another, then those events start queuing up on the second set of servers. It's something that we continually balance on.And we use these bits of information for each test and start, you know, logging and tracking these issues, and deciding whether it's important, how long is it going to take to fix, and is it necessary. And, you know, you're balancing risk and reward with everything you're doing, of course, in the business world, but sometimes the, you know—“Chris, bring us more quality. You can do better this month. Can you give us 20 more units of quality?” It's like, “I can't really package that up and hand it to you. That's not a deliverable.”And in the same way that reputation that we lose when our systems go down isn't as quantifiable, either. Sure, you can watch the tweets come across the interwebs, and see how upset our students are at those kinds of things, but our customer support and our service really takes that to heart, and they listen to those tweets and they fix them, and they coordinate and reach out, you know, directly to these folks. And I think that's why our organization supports this type of performance testing, as well as our coordinated chaos: The service experience that goes out to our customers has to be second to none. And that's second to none is the table stakes is your platform must be on, must be stable, and must be performing. That's just to enter the space, kids. You've got to be there. [laugh].You can't have your platform going down at 9 p.m. on a Sunday night when all these college students are doing their homework because they freak out. And they react to it. It's important. That's the currency. That is the human experience that says this platform, this product is very important to these students' lives and their well-being in their academic career. And so we take that very seriously.Jason: I love that you mentioned that your customer support works with the engineering team. Because makes me think of how many calls have you been on where something went wrong, you contacted customer support, and you end up reaching this thing of, they don't talk to engineering, and they're just like, “I don't know, it's broken. Try again some other time.” Or whatever that is, and you end up lost. And so this idea of we often think of DevOps is developers and operations engineers working together and everybody on the engineering side, but I love that idea of extending that.And so I'm curious, in that vein, does your Chaos Engineering, does your performance testing also interact with some of what customer support is actually doing?Chris: In a support kind of way, absolutely. Our customer call support is very well educated on our products and they have a lot of different tools at their disposal in order to correct problems. And you know, many of those problems are access and permissions and all that kind of stuff that's usual, but what we've seen is even though that our customer base is increasing and our call volume increases accordingly, the percentage decreases over time because our customer support people have gotten so good at answering those questions. And to that extent, when we do log issues that are not as easily fixed with a tweak or knob toggle at the customer support side, those get grouped up into a group of tickets that we call escalation tickets, and those go directly to engineering.And when we see groups of them that look and smell kind of the same or have similar symptoms, so we start looking at how to design that into chaos, and is it a real performance issue? Especially when it's related to slowness or errors that continuously come at a particular point in that workflow. So, I hope I answered that question there for you, Jason.Jason: Yeah, that's perfect.Julie: Now, I'd like to kind of bring it back a little bit to some of the learnings we've had over this time of practicing Chaos Engineering and focusing on that quality testing. Is there something big that stands out in your mind that you learned from an experiment? Some big, unknown-unknown that you don't know that you ever could have caught without practicing?Chris: Julie, that's a really good question, and there isn't, you know, big bang or any epiphanies here. When I talk about what is the purpose of chaos and what do we get out of it, there's the human factor of chaos in terms of what does this do for us. It gets us prepared, it gets us a fire drill without the sense of urgency of production, and it gets people focused on solving a problem together. So, by practicing in a performance, in a chaos sort of way, when performance does affect the production, those communication channels are already greased. When there's a problem with some system, I know exactly who the engineer is to go to and ask him a question.And that has also enabled us to reduce our meantime to resolution. That meantime to resolution factor is predicated on our teams knowing what to do, and how to resolve those. And because we've practiced it, now that goes down. So, I think the synergy of being able to work together and triangulate our teams on existing issues in a faster sort of way, definitely helps our team dynamic in terms of solving those problems faster.Julie: I like that a lot because there is so much more than just the technical systems. And that's something that we like to talk about, too. It is your people's systems. And you're not trying to surprise anybody, you've got these scheduled on a calendar, they run regularly, so it's important to note that when you're looking at making your people's systems more resilient, you're not trying to catch Chris off guard to see if he answered the page—Chris: That's right.Julie: —what we're working on is making sure that we're building that muscle memory with practice, right, and iron out the kinks in those communication channels.Chris: Absolutely. It's definitely been a journey of learning both for, you know, myself and my team, as well as the engineers that work on these things. You know, again, everybody chips in and gets to learn that routine and be comfortable with fighting fires. Another way I've looked at it with Chaos Engineering, and our testing adventures is that when we find something that it looks a little off—it's a burp, or a sneeze, or some hiccup over here in this system—that can turn into a full-blown fever or cold in production. And we've had a couple of examples where we didn't pay attention to that stuff fast enough, and it did occur in production.And kudos to our engineering team who went and picked it up because we had the information. We had the tracking that says we did find this. We have a solution or recommended fix in place, and it's already in process. That speaks volumes to our sense of urgency on the engineering teams.Julie: Chris, thank you for that. And before we end our time with you today, is there anything you'd like to let our listeners know about Cengage or anything you'd like to plug?Chris: Well, Cengage Learning has been a great place for me to work and I know that a lot of people enjoy working there. And anytime I ask my teams, like, “What's the best part of working there?” It's like, “The people. We work with are supportive and helpful.” You know, we have a product that we'd like to help change people's lives with, in terms of furthering their education and their career choices, so if you're interested, we have over 200 open positions at the current moment within our engineering and staffing choices.And if you're somebody interested in helping out folks and making a difference in people's educational and career paths, this is a place for you. Thanks for the offer, Julie. Really appreciate that.Julie: Thank you, Chris.Jason: Thanks, Chris. It's been fantastic to have you on the show.Chris: It's been a pleasure to be here and great to talk to you. I enjoy talking about my passions with testing as well as many of my other ones. [laugh].Jason: For links to all the information mentioned, visit our website at gremlin.com/podcast. If you liked this episode, subscribe to the Break Things on Purpose podcast on Spotify, Apple Podcasts, or your favorite podcast platform. Our theme song is called “Battle of Pogs” by Komiku, and it's available on loyaltyfreakmusic.com.

In this episode, we cover: 00:00:00 - Introduction  00:02:00 - Carissa's first job in tech and first bootcamp  00:04:30 - Early Lessons: Carissa breaks production—on a Friday! 00:08:40 - Carissa's work at ClickBank and listening to newer hires  00:10:55 - The metrics that Carissa measures and her attitude about constantly learning 00:16:45 - Carissa's Chaos Engineering experiences  00:18:25 - Some advice for bringing new folks into the fold 00:23:08 - Carissa and ClickBank/Outro Links: ClickBank: https://www.clickbank.com/ LinkedIn: https://www.linkedin.com/in/carissa-morrow/ TranscriptCarissa: It's all learning. I mean, technology is never going to stop changing and it's never going to stop being… a lot to learn, [laugh] so we might as well learn it and try to keep up with the [laugh] times and make our lives easier.Julie: Welcome to Break Things on Purpose, a podcast about reliability, asking questions, and learning from failure. In this episode, we talked with Carissa Morrow about what it's like to be new in tech, and how to learn from mistakes and build your skills.Julie: Carissa, I'm really excited to talk to you. I know we chatted in the past a little bit about some horror stories of breaking production. I think that it's going to be a lot of fun for our listeners. Why don't you tell us a little bit about yourself?Carissa: Yeah, so I actually have only been in this industry about three years. So, I come with kind of a newbie's perspective. I was a certified ophthalmic tech before this. So, completely different field. Hit my ceiling, and my husband said, “You want to try coding?” I said, “Not really.” [laugh]. But I did. And I loved it.So, long story short, I ended up just signing up for a local boot camp, three-month full stack. And then I got really lucky; when I graduated there and walked into my previous employer's place. They said, “Do you know what DevOps is?” I said, “I have no idea.” And they still hired me.And it was really great, really, really great experience. I learned so much in a couple years with them. So, and now I'm here at ClickBank and I'm three years in and trying not to break things every day, especially on a Friday.Julie: [laugh]. Why? That's the best day to break things, Carissa—Carissa: [laugh]. No, it's really not.Julie: —preferably at 4:45. Well, that's really amazing. So, that's quite the jump. And as you mentioned, you started with a boot camp and then ended up at an employer—and so, what was your role? What were you doing in your first role?Carissa: So, I started on a really small team; there was just three of us including myself. So, I learned pretty much everything from the ground up, knowing nothing coming into DevOps. So, I had, you know, coding background from the boot camp, but I had to learn Python from scratch. And then from there, just kind of learning everything cloud. I had no idea about AWS or Google or anything in the cloud realm.So, it was very much a rough—very, very rough first year, I had to put my helmet on because it was a very bumpy ride. But I made it and I've come out a heck of a lot stronger because of it.Julie: Well, that's awesome. How about do you have people that you were working with that are mentoring you?Carissa: Yep. So, I actually have been very lucky and have a couple of mentors, from not only my previous employer, but also clients that I worked with that have asked to be my mentor and have stuck it out with me, and helped not just in the DevOps realm or the cloud realm, but for me as a person in that growing area. So, it's been pretty great.Julie: Well, that's awesome. And I guess I should give the disclosure that Carissa and I both worked together, for me a couple of jobs ago. And I know that, Carissa, I've reached out to you for folks who are interested in the boot camp that you went through. And I know it's not an advertisement for the boot camp, but I also know that you mentored a friend of mine. Did you want to share where you went?Carissa: Yeah, definitely. So, I went to Boise CodeWorks, which is a local coding school here in Boise. And they did just move locations, so I'm not quite sure where they're at now, but they're definitely in Boise.Julie: And if I remember correctly, that was a three-month very intensive, full-time boot camp where you really didn't have time for anything else. Is that right?Carissa: Yes, it is absolutely 1000% a full-time job for three months. And you will get gray hairs. If you don't, you're doing something wrong. [laugh]. Yep.Julie: So, what would you say is one of the most important things you learned out of that?Carissa: I would say just learning how to be resilient. It was very easy to want to quit because it was so difficult. And not knowing what it was going to look like when I got out of it, but part of me just wanted to throw my hands up half the time. But pushing through that made it just that much sweeter when I was done.Julie: Well now, when we were talking before, you mentioned that you broke production once. Do you want to tell me about that—Carissa: Maybe a few times. [laugh].Julie: —[crosstalk 00:04:34] a few times? [laugh]. You want to share what happened and maybe what you learned from it.Carissa: Yeah, yep. So, I was working for a company that we had clients, so it was a lot of client work. And they were an AWS shop, and I was going in to kind of clean up some of their subnets and some of their VPN issues—of course, this is also on a Friday. Yeah. It has to be on a Friday.Julie: Of course.Carissa: So, I will never forget, I was sitting outside thinking, “This is going to be a piece of cake.” I went in, I just deleted a subnet, thinking, “That's fine. Nothing's going to happen.” Five minutes later Slack's blowing up, production's down and, you know, websites not working. Bad. Like, worst-case scenario.So, back then we had, like, a team of, I think I would say ten, and every single person jumped on because you could tell I was panicking. And they all jumped in and we went step-by-step, tried to figure it out, figured out how we could fix it. But it took a good four hours of traumatizing stress [laugh] before we got it fixed. And then I learned my lesson, you know? Double-triple check before you delete anything and try to just make Fridays read-only if you can. [laugh].Julie: Well, and I think that's one of the things right? You always have to have that lesson-learning experience, and it's going to happen. And showing empathy for friends during that, I think, is the really important piece. And I love the fact that you just talked about how the whole team jumped on because they saw that you were stressed out. Were you in person or remote at the time?Carissa: I was remote at the time.Julie: Okay.Carissa: Yeah. And we were traveling in our RV, so nothing like being out in the woods, panicking by yourself, and [laugh] roaming around.Julie: So, did you run a postmortem on it?Carissa: So, back then—actually, we ended up doing that, yes, but that was when I had never really experienced a postmortem before, and that's one thing that, you know, when we talk about this kind of stuff—and everyone has a horror story or two, but that's something that I've had to learn to get better at is RCAs and postmortems because they're so important. I think they're incredibly important. Because these things are going to happen again; they're going to happen to the best of us. So, definitely, everything is a learning experience. And if it's not, you're missing out. So, I try to make everything a learning experience, for sure.Julie: Absolutely. And that's one of the things we talk about is now take that, and how do you learn from this? And how do you put the gates in place so that you can't just delete a subnet? I mean, to be fair, you did it, but were there other things that could have prevented this from happening, some additional checks and balances?Carissa: Mm-hm.Julie: And as you mentioned, that's not the only time that you've broken production. But let me ask you was that—did the alerting mechanisms work? Did all of the other—did the monitoring and observability? Like, did everything work correctly, or did you find some holes in that as well?Carissa: So, that's a great question. So, this specific client did not use any monitoring tools whatsoever. So—Julie: Huh.Carissa: Yeah, so that was one of those unique situations where they just tried to get on their own website and it didn't work. And then, you know, it was testing and everything was failing. But it was all manual testing. And I actually—believe it or not—I've seen that more often than I ever thought I would in the last three years. And so with what you guys do, and kind of what I'm seeing with a bunch of different clients, it's not just do they have monitoring, it's how do they use that? And when it's, kind of, bits and pieces here and there and they're not using it to their full potential, that's when a lot of things slip through the cracks. So, I've definitely seen a lot of that.Julie: Absolutely. And it's interesting because I really think that, especially these advanced organizations, that they're just going to have all the ducks in the row, all the right monitoring setup, and it turns out that they don't always have everything set up or set up correctly. And that's one of the things that we talk about, too, is validating with Chaos Engineering, and looking at how can we make sure it's not just that our systems are resilient, but that our tools pick things up, that our people and processes work? And I think that's really important. Now… you're working at ClickBank today?Carissa: Mm-hm.Julie: You want to tell us a little bit about that and about what you do over there?Carissa: Yep. So, I came on a few months ago as a cloud engineer for their team. And they are—I have actually learned a lot of monitoring tools through what they have already set up. And as they're growing and continue to grow, I'm learning a lot about what they have in place and maybe how we can improve it. So, not just understanding the metrics has been a learning curve, but understanding what we're tracking, why, and what's an emergency—what's critical, what's not—all of those things is definitely a huge, huge learning curve.But regardless of if it's ClickBank or other companies that I know people that work out or I've worked at, everyone knows there's a humbling aspect when you're using all these tools. We all want to pretend like we know everything all the time, and so being humble enough to ask the questions of, “Why do we use this? Are we using it to its full potential? And what am I looking at?” That's how I've learned the most, even in the last couple of months here is just asking those very humbling questions.Julie: Well, I have to say, you know, you mentioned that you are really still new; it's three years out of school for you doing this, and I think that there actually is quite a lot to be said about listening to newer people because you're going to ask questions that other folks haven't thought of, like, the whys. “Why are we doing things this way?” Or, “Why are we tracking that?” And sometimes—I think you've probably seen this as organizations—we just get into these habits—Carissa: Mm-hm.Julie: —and we do things because somebody who worked here, like, five years ago, set it up that way; we've just always done it this way.Carissa: Mm-hm.Julie: And it's a great idea to look into some of our practices and make sure that they're still serving us. One thing that you mentioned that I love, though, is you said metrics. And metrics are really important when practicing Chaos Engineering because it's good to know where you are now so that you can see improvement. Can you talk about some of the metrics that you measure or that might be important to ClickBank?Carissa: Yeah. So, a lot of the things that we measure have to do with orders. So, the big thing with ClickBank with how the model, the infrastructure of this company is set, orders are incredibly important, so between the vendors and the buyers in ClickBank. So, we are always monitoring in great detail how our orders are coming in, going out, all the payment information, you know, make sure everything's always secure and running smoothly. So, those are where most of our metrics that we watch where those live.The one thing that I think is—I've noticed is really important is whether you're monitoring one thing or ten, monitor to the best of your ability so that you're not just buying stuff and using 50% of it. And I think we get really excited when we go and we're like, “Yes, this is a great third-party tool or third-party—we're going to use it.” And then 10% of it, you know, you use and the rest of it, it's like, “That's really cool. Maybe we'll do that later, maybe we'll implement that part of it later.” And that's something that it's just, it's like, I know it's painful, [laugh] but do it now; get it implemented now and start using it, and then go from there.But I feel like why do we bother if we're only going to use 10% to 50% of these amazing things that really make our lives easier, and obviously, more secure and more resilient.Julie: I think you're onto something there. That is really good advice. I remember speaking at a conference in New Zealand and one of the speakers there talked about how their organization will buy any new tool that comes out, any and every new tool that comes out. But just buying that—and as you mentioned, just using a tiny, small portion of that tool can really be kind of ridiculous. You're spending a lot of money on these tools, but then these features were built for a reason, and oftentimes—and I saw this, too, at my past company—folks would purchase our tool, but not realize that our tool did so many other things.And so then there are multiple tools that are doing the same things within an organization when in reality, if you look at all the features and truly understand a tool—I would say some folks have a hard time with saying well, it just takes too much time to learn all of that. What's your advice for them?Carissa: Yeah. I think I've caught myself saying that to [laugh] at some point in time. You know, the context-switching, already having our full-time jobs and then bringing on tools, other tools that we need to learn. And it is overwhelming, but my advice is, why make more pain for yourself? [laugh]. Why not make your life easier, just like automation, right?When you're automating things, it's going to be a lot of work up front, but the end goal is make everything more secure, make it easier on yourself, take out the single point of failure or the single-person disaster because they did one wrong thing. Monitoring does the same thing. You know, if you put the investment up ahead of time, if you do it right upfront, it's going to pay off later.The other thing I've seen, and I've been guilty of as well is just looking at it and saying, “Well, it looks like it's working,” but I don't really know what I'm looking at. And so going back to that, you know, if you don't know why things are failing, or what to look out for to catch things from failing, then why even bother having that stuff in front of you? So, it's a lot of learning. It's all learning. I mean, technology is never going to stop changing and it's never going to stop being… a lot to learn, [laugh] so we might as well learn it and try to keep up with the [laugh] times and make our lives easier.There was actually a—I wrote this quote down because I ran across this last week, and I loved it because we were talking about failures. It said, “Not responding to failures is one characteristic of the organizational death spiral.” And I loved that because I sat there and thought, “Yeah, if you do have a failure, and you think, ‘Well, I have my monitoring tools in place. It looks like it worked itself out. I don't really know what happened.' And that continues to happen, and everyone on the team has that same mentality, then eventually, things are going to keep breaking, and it's going to get worse and worse over time.” And they're not going to realize that they had a death spiral. [laugh]. So, I just love that quote, I thought that was pretty great.Julie: I love that as well, who was that from?Carissa: Oh, I'll have to pull it up, but it was online somewhere. I was kind of going through—because really bothering me when we were talking about some of our monitoring, and I was asking some kind of deep questions about, why? What's the critical threshold? What's the warning? Why are we looking at this? And so I started looking at deeper dives into resiliency, and so that popped up, and I thought that was pretty spot on.Julie: I love it. We will find the author of that. We'll post it in the show notes. I think that is an amazing quote. I think I'm going to steal it from you at some point because that's—it's very true.And learning from those failures and understanding that we can prevent failures from occurring, right? So—Carissa: Absolutely.Julie: —if you have a failure and you've remediated it, and you still want to test to make sure that you're not going to drift back into that failure, right? Our systems are constantly changing. So, that's one of the things we talk about with Chaos Engineering, as well, and building that reliability in. Now, have you experienced or practiced Chaos Engineering at all with any of your customers that you've worked on, or at ClickBank?Carissa: There was one, [sigh] one client that we had that I would say yes, but the testing itself needed to be more robust, it needed to be more accurate. It was kind of like an attempt to build testing around—you know, for Chaos Engineering, but looking back now, I wish we would have had more guidance and direction on how to build really strategic testing, not just, “Oh, look, it passed.” It might have been a false pass, [laugh] but it was just kind of absolute basic testing. So, I think there's a growth with that. Because I've talked to a lot of engineers over the years that we say testing is important, right, but then do we actually do it, especially when we're automating and we're using all these third-party tools.A lot of times, I'm going to go with what we don't. We say it's really important, we see the importance of it, but we don't actually implement it. And sometimes it's because we need help to be able to build accurate testing and things that we know really are going to be sustainable testing. So, it's more of probably an intimidation thing that I've seen over the years. And it's kind of going back to, we don't like to ask for help a lot of times in this industry, and so that plays a role there. Sometimes we just need help to be able to build these things out so we're not walking on eggshells waiting for the next thing to break.Julie: Now, I love it because you've drilled down kind of into that a few times about asking for help. And you've worked with some folks that I know you've done a great job. So far, I'm really impressed just seeing your growth over the last three years because I do remember your first day—Carissa: Oh—Julie: [laugh].Carissa: [laugh]. Oh, God.Julie: —and seeing you and in these little corner cubes. That was—[laugh]—Carissa: I was sweating bullets that day.Julie: —quite a long time ago. What advice would you give to senior folks who are helping newer folks or more junior folks? What would you want them to know about working with newer people?Carissa: Yeah, that's a good question. So, in my last job, I actually ended up becoming a lead before I left. And so [sigh] the one thing I learned from my mentor at my previous company that really just brought me up from knowing nothing. One thing I learned from him was, when he looked at me on the first day, he said, “Do not be afraid to ask for help. Period. Just don't. Because if you don't, something bad's going to happen and you're not going to learn and you're not going to grow.”And he also was one that said, “Put your helmet on. It's going to be a bumpy ride.” [laugh]. And I loved that. He even got me a little, uh—oh, it kind of like—it was a little bobblehead, and it had a helmet. [laugh]. And I thought that was so spot-on.I think we forget when we get really good at something or we've been doing something for a while, as human beings, we forget what it's like to be new, and to be scared, and to not know what our left and right hand is doing. So, I would say keep that in the forefront of your mind as you're mentoring people, as you're helping ramp them up, is they're going to be afraid to ask questions or remind them it's okay, and also just taking a step back and remembering when you were really new at something. Because it's hard to do. We all want to become experts and we don't want to remember how horrible that felt when we did not know what was in front of us. So, that would be my couple pieces of advice.Julie: Well, and then kind of circling back to that first time that you broke production, right, and everybody rallied around to help you—which is amazing; I love that—after it was over, what was the culture like? Were they supportive? What happened?Carissa: Yeah, that's a really good question because I've heard people's horror stories where it was not a good response afterwards, and they felt even more horrible after it was fixed. And my experience was a complete opposite. The support was just 1000% there. And we even hung out—we started a Zoom call and after we'd fixed it, there were people that hopped back on the call and said, “Let me tell you about my production story.” And we just started swapping horror stories.And it was 1000% support, but also it was a nice human reminder that we break things and it's okay. And so that was—it was a pretty great experience, I hope the best—we're all going to break things, but I hope that everyone gets that experience because the other experience, no fun. You know, we already feel terrible enough after we break it. [laugh].Julie: I think that's important. And I love that because that goes back to the embrace failure statement, right? Embrace it, learn from it. If you can take that and learn. And what did you learn? So, you mentioned you learn double, triple, quadruple check.Carissa: Mm-hm.Julie: So, have you made that same mistake again?Carissa: I have not. Knock on wood. I have not. [laugh].Julie: [crosstalk 00:21:58]Carissa: [crosstalk 00:21:59]. [laugh].Julie: It could happen—Carissa: Yep.Julie: —as we all are learning so much, sometimes you make the same mistake twice, right?Carissa: Yeah, absolutely. I would say there's two things. So, I learned that, and then I also learned that not just double and triple check before you do something, but going back to the don't be afraid to ask questions, sometimes you have to ask clarifying questions of your client or your customer before you pull the trigger. So, you might say I've done this a million times, but sometimes the ask is a little vague. And so, if you don't ask detailed questions, then yes, you might have done what needed to be done, but not in the way that they hoped for, not in the way that they wanted, your end game results were now not what was hoped for.So, definitely ask layered questions if you need to. To anyone: To your coworkers, to your manager, to your whoever you're using your monitoring tools through. Just ask away because it's better to do it upfront than to just try to get the work done and then, you know, then more fun happens.Julie: More fun indeed. [laugh].Carissa: [laugh]. Yes.Julie: Now, why don't you tell our listeners who aren't familiar with ClickBank, do you want to promote them a little bit, talk a little bit about what you're doing over there?Carissa: Yeah. So, ClickBank is awesome, which is why I'm there. [laugh]. No, they're a great company. I'm on a fairly, I wouldn't say large team, but it's a good-sized team.They're just really good people. I think that's been one of the things that's incredibly important to me, and I knew when I was making a switch that everyone talks about, they have a great working environment, they have great work-life balance. And for me, it's like you can talk the talk, but I want you to walk the walk, as a company. And I want—you know, if you say you're going to have a family environment, I want to see that. And I have seen that at ClickBank.It's been an awesome couple of months. There's a lot of support on the teams. There's a lot of great management there, and I'm kind of excited to see where this goes. But coming with a fresh perspective of working at ClickBank, it's a really great company. I'm happy.Julie: Well, I love that. And from what I'm aware of, y'all have some positions that are open, so we'll post a link to ClickBank in the as well. And, Carissa, I just want to thank you for taking the time to be a little vulnerable and talk about your terrifying breaking production experience, but also about why it's so important to be open to folks asking questions and to show empathy towards those that are learning.Carissa: Mm-hm. Yeah, absolutely. I think that is the number one thing that's going to make us all successful. It's going to make mentors more successful, and they're going to learn as they're doing it and it's going to make—it's going to build confidence in people that are coming into this industry or that are new in this industry to say, “Not only can I do this, I'm going to be really great. And I'm going to eventually mentor somebody someday.”Julie: I love that. And thank you. And thank you for spending time with us today. And, folks, you can find Carissa on LinkedIn. Pretty impressed that you're not on Twitter, so not a huge social media person, so it's just LinkedIn for Carissa. And with that—Jason: For links to all the information mentioned, visit our website at gremlin.com/podcast. If you liked this episode, subscribe to the Break Things on Purpose podcast on Spotify, Apple Podcasts, or your favorite podcast platform. Our theme song is called, “Battle of Pogs” by Komiku, and it's available on loyaltyfreakmusic.com.

In this episode, we cover: 00:00:00 - Intro 00:01:45 - AWS Severless Hero and Gunnar's history using AWS 00:04:42 - Severless as reliability 00:08:10 - How they are testing the connectivity in serverless 00:12:47 - Gunnar shares a suprising result of Chaos Engineering 00:16:00 - Strategy for improving and advice on tracing  00:20:10 - What Gunnar is excited about at AWS 00:28:50 - What Gunnar has going on/Outro Links: Twitter: https://twitter.com/GunnarGrosch LinkedIn: https://www.linkedin.com/in/gunnargrosch/ TranscriptGunnar: When I started out, I perhaps didn't expect to find that many unexpected things that actually showed more resilience or more reliability than we actually thought.Jason: Welcome to the Break Things on Purpose podcast, a show about Chaos Engineering and building more reliable systems. In this episode, we chat with Gunnar Grosch, a Senior Developer Advocate at AWS about Chaos Engineering with serverless, and the new reliability-related projects at AWS that he's most excited about.Jason: Gunnar, why don't you say hello and introduce yourself.Gunnar: Hi, everyone. Thanks, Jason, for having me. As you mentioned that I'm Gunnar Grosch. I am a Developer Advocate at AWS, and I'm based in Sweden, in the Nordics. And I'm what's called a Regional Developer Advocate, which means that I mainly cover the Nordics and try to engage with the developer community there to, I guess, inspire them on how to build with cloud and with AWS in different ways. And well, as you know, and some of the viewers might know, I've been involved in the Chaos Engineering and resilience community for quite some years as well. So, topics of real interest to me.Jason: Yeah, I think that's where we actually met was around Chaos Engineering, but at the time, I think I knew you as just an AWS Serverless Hero, that's something that you'd gotten into. I'm curious if you could tell us more about that. How did you begin that journey?Gunnar: Well, I guess I started out as an AWS user, built things on AWS. As a builder, developer, I've been through a bunch of different roles throughout my 20-plus something year career by now. But started out as an AWS user. I worked for a company, we were a consulting firm helping others build on AWS, and other platforms as well. And I started getting involved in the AWS community in different ways, by arranging and speaking at different meetups across the Nordics and Europe, also speaking at different conferences, and so on.And through that, I was able to combine that with my interest for resiliency or reliability, as someone who's built systems for myself and for our customers. That has always been a big interest for me. Serverless, it came as I think a part of that because I saw the benefits of using serverless to perhaps remove that undifferentiated heavy lifting that we often talk about with running your own servers, with operating things in your own data centers, and so on. Serverless is really the opposite to that. But then I wanted to combine it with resilience engineering and Chaos Engineering, especially.So, started working with techniques, how to use Chaos Engineering with serverless. That gained some traction, it wasn't a very common topic to talk about back then. Adrian Hornsby, as some people might know, also from AWS, he was previously a Developer Advocate at AWS, now in a different role within the organization. He also talked a bit about Chaos Engineering for serverless. So, teamed up a bit with him, and continue those techniques, started creating different tools and some open-source libraries for how to actually do that. And I guess that's how, maybe, the AWS serverless team got their eyes opened for me as well. So somehow, I managed to become what's known as an AWS Hero in the serverless space.Jason: I'm interested in that experience of thinking about serverless and reliability. I feel like when serverless was first announced, it was that idea of you're not running any infrastructure, you're just deploying code, and that code gets called, and it gets run. Talk to me about how does that change the perception or the approach to reliability within that, right? Because I think a lot of us when we first heard of serverless it's like, “Great, there's Nothing. So theoretically, if all you're doing is calling my code and my code runs, as long as I'm being reliable on my end and, you know, doing testing on my code, then it should be fine, right?” But I think there's some other bits in there or some other angles to reliability that you might want to tune us into.Gunnar: Yeah, for sure. And AWS Lambda really started it all as the compute service for serverless. And, as you said, it's about having your piece of code running that on-demand; you don't have to worry about any underlying infrastructure, it scales as you need it, and so on; the value proposition of serverless, truly. The serverless landscape has really evolved since then. So, now there is a bunch of different services in basically all different categories that are serverless.So, the thing that I started doing was to think about how—I wasn't that concerned about not having my Lambda functions running; they did their job constantly. But then when you start building a system, it becomes a lot more complex. You need to have many different parts. And we know that the distributed systems we build today, they are very complex because they contain so many different moving parts. And that's still the case for serverless.So, even though you perhaps don't have to think about the underlying infrastructure, what servers you're using, how that's running, you still have all of these moving pieces that you've interconnected in different ways. So, that's where the use case for Chaos Engineering came into play, even for serverless. So, testing how these different parts work together to then make sure that it actually works as you intended to. So, it's a bit harder to create those experiments since you don't have control of that underlying infrastructure. So instead, you have to do it in a few different ways, since you can't install any agents to run on the platform, for instance, you can't control the servers—shut down servers, the perhaps most basic of Chaos Engineering experiment.So instead, we're doing it using different libraries, we're doing it by changing configuration of services, and so on. So, it's still apply the same principles, the principles of Chaos Engineering, we just have to be—well, we have to think about it in different way in how we actually create those experiments. So, for me, it's a lot about testing how the different services work together. Since the serverless architectures that you build, they usually contain a bunch of different services that you stitch together to actually create the output that you're looking for.Jason: Yeah. So, I'm curious, what does that actually look like then in testing, how these are stitched together, as you say? Because I know with traditional Chaos Engineering, you would run a blackhole attack or some sort of network attack to disrupt that connectivity between services. Obviously, with Lambdas, they work a little bit differently in the way that they're called and they're more event-driven. So, what does that look like to test the connectivity in serverless?Gunnar: So, what we started out with, both me and Adrian Hornsby was create these libraries that we could run inside the AWS Lambda functions. So, I created one that was for Node.js, something that you can easily install in your Node.js code. Adrian has created one for Python Lambda functions.So, then they in turn contain a few different experiments. So, for instance, you could add latency to your AWS Lambda functions to then control what happens if you add 50 milliseconds per invocation on your Lambda function. So, for each call to a downstream service, say you're using DynamoDB as a data store, so you add latency to each call to DynamoDB to see how this data affect your application. Another example could be to have a blackhole or a denial list, so you're denying calls to specific services. Or it could be downstream services, other AWS services, or it could be third-party, for instance; you're using a third-party for authentication. What if you're not able to reach that specific API or whatever it is?We've created different experiments for—a typical use case for AWS Lambda functions has been to create APIs where you're using an API Gateway service, an AWS Lambda function is called, and then returning something back to that API. And usually, it should return a 200 response, but you could then alter that response to test how does your application behave? How does the front-end application, for instance, behave when it's not getting that 200 response that it's expecting, instead of getting a 502, a 404, or whatever error code you want to test with. So, that was the way, I think, we started out doing these types of experiments. And just by those simple building blocks, you can create a bunch of different experiments that you can then use to test how the application behaves under those adverse conditions.Then if you want to move to create experiments for other services, well, then serverless, as we talked about earlier, since you don't have control over the underlying infrastructure, it is a bit harder. Instead, you have to think about different ways to do with by, for instance, changing configuration, things like that. You could, for instance, restrict concurrent operations on certain services, or you could do experiments to block access, for instance, using different access control lists, and so on. So, different ways, all depending on how that specific service works.Jason: It definitely sounds like you're taking some of those same concepts, and although serverless is fundamentally different in a lot of ways, really just taking that, translating it, and applying those to the serverless.Gunnar: Yeah, exactly. I think that's very important here to think about, that it is still using Chaos Engineering in the exact same way. We're using the traditional principles, we're walking through the same steps. And many times as I know everyone doing Chaos Engineering talks about this, we're learning so much just by doing those initial steps. When we're looking at the steady-state of the application, when we're starting to design the experiments, we learn so much about the application.I think just getting through those initial steps is very important for people building with serverless, as well. So, think about, how does my application behave if something goes wrong? Because many times with serverless—and for good reasons—you don't expect anything to fail. Because it's scales as it should, services are reliant, and they are responding. But it is that old, “What if?” What if something goes wrong? So, just starting out doing it in the same way as you normally would do with Chaos Engineering, there is no difference, really.Jason: And know, when we do these experiments, there's a lot that we end up learning, and a lot that can be very surprising, right? When we assume that our systems are one way, and we run the test, and we follow that regular Chaos Engineering process of creating that hypothesis, testing it, and then getting that unexpected result—Gunnar: Right.Jason: —and having to learn from that. So, I'm interested, if you could share maybe one of the surprising results that you've learned as you've done Chaos Engineering, as you've continued to hone this practice and use it. What's a result that was unexpected for you, that you've learned something about?Gunnar: I think those are very common. And I think we see them all the time in different ways. And when I started out, I perhaps didn't expect to find that many unexpected things that actually showed more resilience or more reliability than we actually thought. And I think that's quite common, that we run an experiment, and we often find that the system is more resilient to failure than we actually thought initially, for instance, that specific services are able to withstand more turbulent conditions than we initially thought.So, we create our hypothesis, we expect the system to behave in a certain way. But it doesn't, instead—it doesn't break, but instead, it's more robust. Certain services can handle more stress than we actually thought, initially. And I think those cases, they, well, they are super common. I see that quite a lot. Not only talking about serverless Chaos Engineering experiments; all the Chaos Engineering experiments we run. I think we see that quite a lot.Jason: That's an excellent point. I really love that because it's, as you mentioned, something that we do see a lot of. In my own experience working with some of our customers, oftentimes, especially around networking, networking can be one of the more complex parts of our systems. And I've dealt with customers who have come back to me and said, “I ran a blackhole attack, or latency attack, or some sort of network disruption and it didn't work.” And so you dig into it, well, why didn't it work? And it's actually well, it did; there was a disruption, but your system was designed well enough that you just never noticed it. And so it didn't show up in your metrics dashboards or anything because system just worked around it just fine.Gunnar: Yeah, and I think that speaks to the complexity of the systems we're often dealing with today. I think it's Casey Rosenthal who talked about this quite early on with Chaos Engineering, that it's hard for any person to create that mental model of how a system works today. And I think that's really true. And those are good examples of exactly that. So, we create this model of how we think the system should behave, but [unintelligible 00:15:46], sometimes it behaves very unexpected… but in the positive way.Jason: So, you mentioned about mental models and how things work. And so since we've been talking about serverless, that brought to mind one of those things for me with serverless is, as people make functions and things because they're so easy to make and because they're so small, you end up having so many of them that work together. What's your strategy for starting to improve or build that mental model, or document what's going on because you have so many more pieces now with things like serverless?Gunnar: There are different approaches to this, and I think this ties in with observability and the way we observe systems today because as these systems—often they aren't static, they continue to evolve all the time, so we add new functionality, and especially using serverless and building it with AWS Lambda functions, for instance, as soon as we start creating new features to our systems, we add more and more AWS Lambda functions or different serverless ways of doing new functionality into our system. So, having that proper observability, I think that's one of the keys of creating that model of how the system actually works, to be able to actually see tracing, see how the system or how a request flows through the system. Besides that, having proper documentation is something that I think most organizations struggle with; that's been the case throughout all of my career, being able to keep up with the pace of innovation that's inside that organization. So, keeping up with the pace of innovation in the system, continuing to evolve your documentation for the system, that's important. But I think it's hard to do it in the way that we build systems today.So, it's not about only keeping that mental model, but keeping documentation and how the system actually looks, the architecture of the system, it's hard today. I think that's just a fact. And ways to deal with that, I think it comes down to how the engineering organization is structured, as well. We have Amazon and AWS, we—well, I guess we're quite famous for our two-pizza teams, the smaller teams that they build and run their systems, their services. And it's very much up to each team to have that exact overview how their part on the bigger picture works. And that's our solution for doing that,j but as we know, it differs from organization to organization.Jason: Absolutely. I think that idea of systems being so dynamic that they're constantly changing, documentation does fall out of step. But when you mentioned tracing, that's always been one of those really key parts, for me at least coming from a background of doing monitoring and observability. But the idea of having tracing that just automatically going to expose things because it's following that request path. As you dive into this, any advice for listeners about how to approach that, how to approach tracing whether that's AWS X-Ray or any other tools?Gunnar: For me, it's always been important to actually do it. And I think what I sometimes see is that's something that's added on later on in the process when people are building. I tend to say that you should start doing it early on because I often think it helps a lot in the development phase as well. So, it shouldn't be an add-on later on, after the fact. So, starting to use tracing no matter if it's as you said, X-Ray or any third-party's service, using it early on, that helps, and it helps a lot while building the system. And we know that there are a bunch of different solutions out there that are really helpful, and many AWS partners that are willing to help with that as well.Jason: So, we've talked a bunch about serverless, but I think your role at AWS encompasses a whole lot of things beyond just serverless. What's exciting you now about things in the AWS ecosystem, like, what are you talking about that just gets you jazzed up?Gunnar: One thing that I am talking a lot about right now that is very exciting is fortunately, we're in line with what we've just talked about, with resilience and with reliability. And many of you might have seen the release from AWS recently called AWS Resilience Hub. So, with AWS Resilience Hub, you're able to make use of all of these best practices that we've gathered throughout the years in our AWS Well-Architected Framework that then guides you on the route to building resilient and reliable systems. But we've created a service that will then, in an, let's say, more opinionated but also easier way, will then help you on how to improve your system with resilience in mind. So, that's one super exciting thing. It's early days for Resilience Hub , but we're seeing customers already starting to use it, and already making use of the service to improve on their architecture, use those best practices to then build more resilient and reliable systems.Jason: So, AWS Resilience Hub is new to me. I haven't actually haven't really gotten into it much. As far as I understand it, it really takes the Well-Architected Framework and combines the products or the services from Amazon into that, and as a guide. Is this something for people that have developed a service for them to add on, or is this for people that are about to create a new service, and really helping them start with a framework?Gunnar: I would say that it's a great fit if you've already built something on AWS because you are then able to describe your application using AWS Resilience Hub. So, if you build it using Infrastructure as Code, or if you have tagging in place, and so on, you can then define your application using that, or describe your application using that. So, you point towards your CloudFormation templates, for instance, and then you're able to see, these are the parts of my application. Then you'll set up policies for your application. And the policies, they include the RTO and the RPO targets for your application, for your infrastructure, and so on.And then you do the assessment of your application. And this then uses the AWS Well-Architected Framework to assess your application based on the policies you c reated. And it will then see if your application RTO and RPO targets are in line with what you set up in your policies. You will also then get an output with recommendations what you can do to improve the resilience of your application based, once again, on the Well-Architected Framework and all of the best practices that we've created throughout the years. So, that means that you, for instance, will get it, you'll build an application that right now is in one single availability zone, well, then Resilience Hub will give you recommendations on how you can improve resilience by spreading your application across multiple availability zones. That could be one example.It could also be an example of recommending you to choose another data store to have a better RTO or RPO, based on how your application works. Then you'll implement these changes, hopefully. And at the end, you'll be able to validate that these new changes then help you reach your targets that you've defined. It also integrates with AWS Fault Injection Simulator, so you're able to actually then run experiments to validate that through the help of this.Jason: That's amazing. So, does it also run those as part of the evaluation, do failure injection to automatically validate and then provide those recommendations? Or, those provided sort of after it does the evaluation, for you to continue to ensure that you're maintaining your objectives?Gunnar: It's the latter. So, you will then get a few experiments recommended based on your application, and you can then easily run those experiments at your convenience. So, it doesn't run them automatically. As of now, at least.Jason: That is really cool because I know a lot of people when they're starting out, it is that idea of you get a tool—no matter what tool that is—for Chaos Engineering, and it's always that question of, “What do I do?” Right? Like, “What's the experiment that I should run?” And so this idea of, let's evaluate your system, determine what your goals are and the things that you can do to meet those, and then also providing that feedback of here's what you can do to test to ensure it, I think that's amazing.Gunnar: Yeah, I think this is super cool one. And as a builder, myself who's used the Well-Architected Framework as a base when building application, I know how hard it can be to actually use that. It's a lot of pages of information to read, to learn how to build using best practices, and having a tool that then helps you to actually validate that, and I think it's great. And then as you mentioned, having recommendations on what experiments to run, it makes it easier to start that Chaos Engineering journey. And that's something that I have found so interesting through these last, I don't know, two, three years, seeing how tools like Gremlin, like, now AWS FIS, and with the different open-source tools out there, as well, all of them have helped push that getting-started limit closer to the users. It is so much easier to start with Chaos Engineering these days, which I think it's super helpful for everyone wanting to get started today.Jason: Absolutely. I had someone recently asked me after running a workshop of, “Well, should I use a Chaos Engineering tool or just do my own thing? Like do it manually?” And, you know, the response was like, “Yeah, you could do it manually. That's an easy, fast way to get started, but given how much effort has been put into all of these tools, there's just so much available that makes it so much easier.” And you don't have to think as much about the safety and the edge cases of what if I manually do this thing? What are all the ways that can go wrong? Since there are these tools now that just makes it so much easier?Gunnar: Exactly. And you mentioned safety, and I think that's a very important part of it. Having that, we've always talked about that automated stop button when doing Chaos Engineering experiments and having the control over that in the system where you're running your experiments, I think that's one of the key features of all of these Chaos Engineering tools today, to have a way to actually abort the experiments if things start to go wrong.Jason: So, we're getting close to the end of our time here. Gunnar, I wanted to ask if you've got anything that you wanted to plug or promote before we wrap up.Gunnar: What I'd like to promote is the different workshops that we have available that you can use to start getting used to AWS Fault Injection Simulator. I would really like people to get that hands-on experience with AWS Fault Injection Simulators, so get your hands dirty, and actually, run some Chaos Engineering experiments. Even though you are far away from actually doing it in your organization, getting that experience, I think that's super helpful as the first step. Then you can start thinking about how could I implement this in my organization? So, have a look at the different workshops that we at AWS have available for running Chaos Engineering.Jason: Yeah, that's a great thing to promote because it is that thing of when people ask, “Where do I start?” I think we often assume not just that, “Let me try this,” but, “How am I going to roll this out in my organization? How am I going to make the business case for this? Who needs to be involved in it?” And then suddenly it becomes a much larger problem that maybe we don't want to tackle. Awesome.Gunnar: Yeah, that's right.Jason: So, if people want to find you around the internet, where can they follow you and find out more about what you're up to?Gunnar: I am available everywhere, I think. I'm on Twitter at @GunnarGrosch. Hard to spell, but you can probably find it in the description. I'm available on LinkedIn, so do connect there. I have a TikTok account, so maybe I'll start posting there as well sometimes.Jason: Fantastic. Well, thanks again for being on the show.Gunnar: Thank you for having me.Jason: For links to all the information mentioned, visit our website at gremlin.com/podcast. If you liked this episode, subscribe to the Break Things on Purpose podcast on Spotify, Apple Podcasts, or your favorite podcast platform. Our theme song is called, “Battle of Pogs” by Komiku, and it's available on loyaltyfreakmusic.com.

In this episode, we cover: 00:00:00 - Intro 00:02:23 - Iwata is the best, rest in peace 00:06:45 - Sam sneaks some SNES emulators/Engineer prep 00:08:20 - AWS, incidents, and China 00:16:40 - Understanding the big picture and moving from project to product 00:19:18 - Sam's time at Snacphat 00:26:40 - Sam's work at Gremlin, and culture changes 00:34:15 - Pokémon Go and Outro TranscriptSam: It's like anything else: You can have good people and bad people. But I wouldn't advocate for no people.Julie: [laugh].Sam: You kind of need humans involved.Julie: Welcome to the Break Things on Purpose podcast, a show about people, culture, and reliability. In this episode, we talk with Sam Rossoff, principal software engineer at Gremlin, about legendary programmers, data center disasters at AWS, going from 15 to 3000 engineers at Snapchat, and of course, Pokémon.Julie: Welcome to Break Things on Purpose. Today, Jason Yee and I are joined by Sam Rossoff, principal software engineer at Gremlin, and max level 100. Pokémon trainer. So Sam, why don't you tell us real quick who you are.Sam: So, I'm Sam Rossoff. I'm an engineer here at Gremlin. I've been in engineering here for two years. It's a good time. I certainly enjoyed it. And before that, I was at Snapchat for six years, and prior to that at Amazon for four years. And actually, before I was at Amazon, I was at Nokia Research Center in Palo Alto, and prior to that, I was at Activision. This was before they merged with Blizzard, all the way back in 2002. I worked in QA.Julie: And do you have any of those Nokia phones that are holding up your desk, or computer, or anything?Sam: I think I've been N95 around here somewhere. It's, like, a phone circa 2009. Probably. I remember, it was like a really nice, expensive phone at the time and they just gave it to us. And I was like, “ oh, this is really nice.”And then the iPhone came out. And I was like [laugh], “I don't know why I have this.” Also, I need to find a new job. That was my primary—I remember I was sitting in a meeting—this was lunch. It wasn't a meeting.I was sitting at lunch with some other engineers at Nokia Research, and they were telling me the story about this app—because the App Store was brand new in those days—it was called iRich, and it was $10,000. It didn't do anything. It was, like, a glowing—it was, like, NFTs, before NFTs—and it was just, like, a glowing thing on your phone. And you just, like, bought it to show you could waste $10,000 an app. And that was the moment where I was like, “I need to get out of this company. I need a new job.” It's depressing at the time, I guess.Julie: So. Sam, you're the best.Sam: No. False. Let me tell you story. There's a guy, his name is Iwata, right? He's a software developer. He works at a company called HAL Laboratories. You may recall, he built a game called Kirby. Very famous game; very popular.HAL Laboratories gets acquired by Nintendo. And Nintendo is like, “Hey, can you”—but Iwata, by the way, is the president of HAL Laboratories. Which is like, you know, ten people, so not—and they're like, “Hey, can you, like, send someone over? We're having trouble with this game we're making.” Right, the game question, at the time they called it Pokémon 2, now we call it Gold and Silver, and Iwata just goes over himself because he's a programmer in addition to be president of HAL Laboratories.And so he goes over there and he's like, “How can I help?” And they're like, “We're over time. We're over budget. We can't fit all the data on the cart. We're just, like, cutting features left and right.” He's like, “Don't worry. I got this.”And he comes up with this crazy compression algorithm, so they have so much space left, they put a second game inside of the game. They add back in features that weren't there originally. And they released on time. And they called this guy the legendary programmer. As a kid, he was my hero.Also famous for building Super Smash Brothers, becoming the president of all of Nintendo later on in his life. And he died a couple years ago, of cancer, if I recall correctly. But he did this motion when he was president of Nintendo. So, you ever see somebody in Nintendo go like this, that's a reference to Iwata, the legendary programmer.Jason: And since this is a podcast, Sam is two hands up, or just search YouTube for—Sam: Iwata.Jason: That's the lesson. [laugh].Sam: [laugh]. His big console design after he became President of Nintendo was the Nintendo Wii, as you may recall, with the nunchucks and everything. Yeah. That's Iwata. Crazy.Julie: We were actually just playing the Nintendo Wii the other day. It is still a high-quality game.Sam: Yeah.Jason: The original Wii? Not like the… whatever?Julie: Yeah. Like, the original Wii.Jason: Since you brought up the Wii, the Wii was the first console I ever owned because I grew up with parents that made it important to do schoolwork, and their entire argument was, if you get a Nintendo, you'll stop doing your homework and school stuff, and your grades will suffer, and just play it all the time. And so they refuse to let me get a Nintendo. Until at one point I, like, hounded them enough-I was probably, like, eight or nine years old, and I'm like, “Can I borrow a friend's Nintendo?” And they were like, sure you can borrow it for the weekend. So, of course, I borrowed it and I played it the whole weekend because, like, limited time. And then they used that as the proof of like, “See? All you did this weekend was play Nintendo. This is why we won't get you one.” [laugh].Sam: So, I had the exact same problem growing up. My parents are also very strict. And firm believers in corporal punishment. And so no video games was very clear. And especially, you know, after Columbine, which was when I was in high school.That was like a hard line they held. But I had friends. I would go to their houses, I would play at their houses. And so I didn't have any of those consoles growing up, but I did eventually get, like, my dad's old hand-me-down computer for, like, schoolwork and stuff, and I remember—first of all, figuring out how to program, but also figuring out how to run SNES emulators on [laugh] on those machines. And, like, a lot of my experience playing video games was waking up at 2 a.m. in the morning, getting on emulators, playing that until about, you know, five, then turning it off and pretending to go back to bed.Julie: So see, you were just preparing to be an engineer who would get woken up at 2 a.m. with a page. I feel like you were just training yourself for incidents.Sam: What I did learn—which has been very useful—is I learned how to fall asleep very quickly. I can fall asleep anywhere, anytime, on, like, a moment's notice. And that's a fantastic skill to have, let me tell you. Especially when [crosstalk 00:07:53]—Julie: That's a magic skill.Sam: Yeah.Julie: That is a magic skill. I'm so jealous of people that can just fall asleep when they want to. For me, it's probably some Benadryl, maybe add in some melatonin. So, I'm very jealous of you. Now I—Jason: There's probably a reason that I'm drinking all this cheap scotch right now.Sam: [laugh].Julie: We should point out that it's one o'clock in the morning for Jason because he's in Estonia right now. So, thank you, A, for doing this for us, and we did promise that you would get to talk about Pokémon. So—Sam: [laugh].Julie: [laugh].Sam: I don't know if you noticed, immediately, that's what I went to. I got a story about Pokémon.Julie: So, have you heard any of our episodes?Sam: I have. I have listened to some. They're mostly Jason, sort of, interviewing various people about their experience. I feel like they come, like, way more well-prepared than I am because they have, like, stuff they want to talk about, usually.Julie: They also generally have more than an hour or two's notice. So.Sam: Well, that's fair. Yeah. That probably [laugh] that probably helps. Whereas, like, I, like, refreshed one story about Iwata, and that's, like, my level of preparation here. So… don't expect too much.Julie: I have no expectations. Jason already had what you should talk about lined up anyway. Something about AWS incidents in China.Sam: Oh, my God. The first question is, which one?Jason: [laugh].Sam: So, I don't know how much you're familiar with the business situation in China, but American businesses are not allowed to operate in China. What happens is you create a Chinese subsidiary that's two-thirds owned by Chinese nationals in some sort of way, you work through other companies directly, and you form, like, these partnerships. And I know you know, very famously, Blizzard did this many years ago, and then, like, when they pulled out China, that company, all the people worked at are like, “Well, we're just going to take your assets and make our own version of World of Warcraft and just, like, run that instead.” But Amazon did, and it was always this long game of telephone, where people from Amazon usually, like, VP, C-level people were asking for various things. And there were people whose responsibility it was to, like, go and make those things happen.And maybe they did or, like, maybe they just said they did, right? And, like, it was never clear how much of it was lost in translation, or they're just, like, dealing with unreasonable requirements, and they're just, like, trying to get something done. But one story is one of my favorites because I was on this call. Amazon required all of their data centers to be multiple zones, right? So, now they talk about availability zones in a region. Internally at Amazon, that's not how we referred to things; it'd be like, there's the data center in Virginia, and there's, like, the first one, the second one, the third one, right? They're just, like, numbered; we knew what they were.And you had to have three of them, and then all services had to be redundant such they could handle a single data center failure. In the earlier days of Amazon, they would actually go turn off data centers to, like, make you prove this as the case. It's was, like, a very early version of chaos engineering. Because it's just, like, unreliable. And unfortunately, AWS kind of put the kibosh on that because it turns out people purchasing VMs on AWS don't like it when you turn off their VMs without warning. Which, like, I'm sympathetic, uh… I don't know.As a side note, if you are data center redundant, that means you're running excess capacity. So, if I'm about to lose a data center, I need to be able to maintain traffic without a real loss in error rates, that means I've got to be running, like, 50% excess capacity if I've only got three data centers, or 33% if you're four data centers. And so capacity of course was always the hard problem when you're dealing with data centers. So, when we were running the Chinese website— z.cn or amazon.cn—there was a data center in China, as you might imagine, as required by the complex business regulations and whatnot.And it had, you know, three availability zones, for lack of a better term. Or we thought it had three availability zones, which of course, this is what happened. One day, I got paged into this call, and they were dealing with a website outage, and we were trying to get people on the ground in China on the call, which as I recall, actually is a real hard problem to get. It was the middle of the night there; there was a very bad rainstorm; people were not near internet connectivity. If you're unfamiliar with the Chinese landscape—well, it's more complex today, but in those days, there were just basically two ISPs in China, and, like, Amazon only paired with one of them.And so if you were on the other one, it was very difficult to get back into Amazon systems. And so they'd have places they could go to so they could connect them when they—and so it was pair to. And so it was a very difficult situation. It took us a while to get people on the phone, but basically, we lost two data centers at the same time, which was very surprising. And later we find out what happened is one of the data centers had flooded, which is bad, bunch of electrical machines flooding for a rainstorm that's got whatever else going on.It turns out the other data center was physically inside of the first [laugh] data center. Which is not the sort of isolation you want between two regions. It's not really clear where in the conversation, you know, things got lost, such that this is what got implemented. But we had three data centers and in theory, and in practice, we had two data centers, since one was inside the other. And when the first one flooded, the, like, floor gave away, and the servers crashed down on top of the other one. [laugh].And so they were literally inside of each other after that point. They took down the Chinese website for Amazon. It was an experience. It was also one of those calls where there's not a lot I could do to help, which is always frustrating for a lot of reasons.Julie: So, how did you handle that call? Out of curiosity, I mean, what do you say?Sam: Well, I'll be honest with you, it took us a long time to get that information, to get save the world. Most of the call actually was trying to get ahold of people try to get information, get translators—because almost everybody on the line did not speak either Cantonese or Mandarin, which is what the engineers were working with—and so by the time we got an understanding—I was in Seattle at the time—Seattle got an understanding of what was happening in—I think it was Beijing. I don't recall off the top of my head—the people on the ground had done a lot of work to isolate and get things up and running, and the remainder of the work was reallocating capacity in the remaining data center so that we wouldn't be running data center redundant, but at the very least, we would be able to serve something. It was, as I recall, it was a very long outage we had to take. Although in those days, the Amazon cn website was not really a profit center.The business was—the Amazon business—was willing to sell things at steep discounts in China to establish themselves in that market, and so, there was always sort of a question of whether or not the outage was saving the company money. Which is, like, sort of a—Julie: [laugh].Sam: —it's like a weird place to be in as an engineer, right? Because you're, like, “You're supposed to be adding business value.” I'm like, “I feel like doing nothing might be adding business out here.” It's not true, obviously because the business value was to be in the Chinese market and to build an Amazon presence for some eventual world. Which I don't know if they ever—they got to. I don't work at Amazon, and haven't in almost a decade now.But it was definitely—it's the kind of thing that wears our morale, right? If you know the business is doing something that is sort of questionable in these ways. And look, in the sales, you know, when you're selling physical goods, industry loss leaders are a perfectly normal part of the industry. And you understand. Like, you sell certain items or loss to get people in the door, totally.But as engineering lacked a real strong view of the cohesive situation on the ground, the business inputs, that's hard on engineering, right, where they're sort of not clear what the right thing is, right? And anytime you take the engineers very far away from the product, they're going to make a bunch of decisions that are fundamentally in a vacuum. And if you don't have a good feel for what the business incentives are, or how the product is interacting with customers, then you're making decisions in a vacuum because there's some technical implementation you have to commit in some way, you're going to make a lot of the wrong decisions. And that was definitely a tough situation for us in those days. I hear it's significantly better today. I can't speak to it personally because I don't work there, but I do hear they have a much better situation today.Julie: Well, I'll tell you, just on the data center thing, I did just complete my Amazon Certified Cloud Practitioner. And during the Amazon training, they drilled it into you that the availability zones were tens of miles apart—the data centers were tens of miles apart—and now I understand why because they're just making sure that we know that there's no data centers inside data centers. [laugh].Sam: It was a real concern.Julie: [laugh]. But kind of going back though, to the business outcomes, quite a while ago, I used to give a talk called, “You Can't Buy DevOps,” and a lot of the things in that talk were based off of some of the reading that I did, in the book, Accelerate by Dr. Nicole Forsgren, Gene Kim, and Jez Humble. And one of the things they talked about is high-performing teams understanding the business goals. And kind of going back to that, making those decisions in a vacuum—and then I think, also, when you're making those decisions in a vacuum, do you have the focus on the customer? Do you understand the direction of the organization, and why are you making these decisions?Jason: I mean, I think that's also—just to dovetail on to that, that's sort of been the larger—if we look at the larger trend in technology, I think that's been the goal, right? We've moved from project management to product management, and that's been a change. And in our field, in SRE and things, we've moved from just thinking of metrics, and there were all these monitoring frameworks like USE (Utilization, Saturation, Errors) and RED (Rate, Errors, Duration) and monitoring for errors, and we've moved to this idea of SLOs, right? And SLOs are often supposed to be based on what's my customer experience? And so I think, overall, aside from Accelerate and DevOps, DevOps I feel like, has just been one part of this longer journey of getting engineers to understand where they fit within the grander scheme of things.Sam: Yeah. I would say, in general, anytime you have some sort of metric, which you're working towards, in some sort of reasonable way, it's easy to over-optimize for the metric. And if you think of the metric instead as sort of like the needle on a compass, it's like vaguely pointing north, right, but keep in mind, the reason we're heading north is because X, Y, Z, right? It's a lot easier for, like, individuals making the decisions that they have to on a day-to-day basis to make the right ones, right? And if you just optimize for the metric—I'm not saying metrics aren't helpful; they're extremely important. I would rather be lost with a compass than without one, but I also would like to know where I'm going and not just be wandering to northwards with the compass, right?Julie: Absolutely. And then—Jason: I mean, you don't want to get measured on lines of code that you commit.Sam: Listen. I will commit 70 lines of code. Get ready.Julie: Well, and metrics can be gamed, right? If people don't understand why those metrics are important—the overall vision; you've just got to understand the vision. Speaking of vision, you also worked at Snap.Sam: I did. I did. That was a really fun place to work. I joined Snapchat; there were 30 people at the company and 14 engineers. Very small company. And a lot of users, you know, 20-plus million users by that point, but very small company.And all the engineers, we used to sit in one room together, and so when you wanted to deploy the production back end, you, like, raised your hand. You're like, “Hey, I'm going to ship out the code. Does anyone have changes that are going out, or is everyone else already doing it?” And one of my coworkers actually wrote something into our deploy script so the speakers on your computer would, like, say, “Deploying production” just so, like, people could hear when it went out the door. Because, like, when you're all in one room, that's, like, a totally credible deployment strategy.We did build automation around that on CircleCI, which in those days was—I think this was 2014—much less big than it is today. And the company did eventually scale to at least 3000 engineers by the time I left, maybe more. It was hard for me to keep track because the company just grown in all these different dimensions. But it was really interesting to live through that.Julie: So, tell me about that. You went from, what, you said, 30 engineers to 3000 in the time that you were there.Sam: Fifteen engineers, I was the fifteenth.Julie: Fifteen. Fifteen engineers. What were some of the pain points that you experienced? And actually maybe even some advice for folks going through big company growth spurts?Sam: Yeah, that hypergrowth? I think it's easier for me to think about the areas that Snap did things wrong, but those were, like, explicit decisions we made, right? It might not be the case that you have these problems at your company. Like, one of the problems Snap had for a long time, we did not hire frontline managers or TPMs, and what that did is it create a lot of situations where you have director-levels with, like, 50-plus direct reports who struggled to make sure that—I don't know, there's no way you're going to manage 50-plus direct reports as engineers, right? Like, and it took the company a while to rectify that because we had such a strong hiring pipeline for engineers and not a strong hiring pipeline for managers.I know there's, like, a lot of people saying companies like, “Oh, man, these middle managers and TPM's all they do is, like, create work for, like, real people.” No. They—I get to see the world without them. Absolutely they had enormous value. [laugh]. They are worth their weight in gold; there's a reason they're there.And it's not to say you can't have bad ones who add negative value, but that's also true for engineers, right? I've worked with engineers, too, who also have added negative value, and I had to spend a lot of my time cleaning up their code, right? It's like anything else: You can have good people and bad people. But I wouldn't advocate for no people.Julie: [laugh].Sam: You kind of need humans involved. The thing that was nice about Snap is Snap was a very product-led company, and so we always had an idea of what the product is that we were trying to build. And that was, like, really helpful. I don't know that we had, like, a grand vision for, like, how to make the internet better like Google does, but we definitely had an idea of what we're building and the direction we're moving it in. And it was very much read by Evan Spiegel, who I got to know personally, who spent a lot of time coming down talking to us about the design of the product and working through the details.Or at least, you know, early on, that was the case. Later on, you know, he was busy with other stuff. I guess he's, like, a CEO or something, now.Julie: [laugh].Sam: But yeah, that was very nice. The flip side meant that we under-invested in areas around things like QA and build tools and these other sorts of pieces. And, like, DevOps stuff, absolutely. Snapchat was on an early version of Google Cloud Platform. Actually an early version of something called App Engine.Now, App Engine still exist as a product. It is not the product today that it was back in 2014. I lived through them revving that product, and multiple deprecations and the product I used in 2014 was a disaster and huge pain, and the product they have today is actually semi-reasonable and something I've would use again. And so props to Google Cloud for actually making something nice out of what they had. And I got to know some of their engineers quite well over the—[laugh] my tenure, as Snap was the biggest customer by far.But we offboarded, like, a lot of the DevOps works onto Google-and paid them handsomely for it—and what we found is you kind of get whatever Google feels like level of support, which is not in your control. And when you have 15 engineers, that's totally reasonable, right? Like, if I need to run, like, a million servers and I have 15 engineers, it's great to pay Google SREs to, like, keep track of my million servers. When you have, you know, 1000 engineers though, and Google wants half a billion dollars a year, and you're like, “I can't even get you guys to get my, like, Java version revved, right? I'm still stuck on Java 7, and this Java 8 migration has been going on for two years, right?”Like, it's not a great situation to be in. And Snap, to their credit, eventually did recognize this and invested heavily in a multi-cloud solution, built around Kubernetes—maybe not a surprise to anyone here—and they're still migrating to that, to the best of my knowledge. I don't know. I haven't worked in that company for two years now. But we didn't have those things, and so we had to sort of rebuild at a very, sort of, large scale.And there was a lot of stuff we infrastructure we set up in the early days in, like, 2014, when, like, ah, that's good enough, this, like, janky python script because that's what we had time for, right? Like, I had an intern write a janky Python script that handled a merge queue so that we could get changes in, and that worked really great when there was like, a dozen engineers just, like, throwing changes at it. When there was, like, 500 engineers, that thing resulted in three-day build times, right? And I remember, uh, what was this… this was 2016… it was the winter of either 2016 to 2017 or 2017 to 2018 where, like, they're like, “Sam, we need to, like, rebuild the system because, like, 72 hours is not an acceptable time to merge code that's already been approved.” And we got down to 14 minutes.So, we were able to do it, right, but you need to be willing to invest the time. And when you're strapped for resources, it's very easy to overlook things like dev tools and DevOps because they're things that you only notice when they're not working, right? But the flip side is, they're also the areas where you can invest and get ten times the output of your investment, right? Because if I put five people on this, like, build system problem, right, all of a sudden, I've got, like, 100x build performance across my, like, 500 engineers. That's an enormous value proposition for your money.And in general, I think, you know, if you're a company that's going through a lot of growth, you have to make sure you are investing there, even if it looks like you don't need it just yet. Because first of all, you do, you're just not seeing it, but second of all, you're going to need it, right? Like, that's what the growth means: You are going to need it. And at Snap I think the policy was 10% of engineering resources were on security—which is maybe reasonable or not; I don't know. I didn't work on security—but it might also be the case that you want maybe 5 to 10% of the engineering resources working on your internal tooling.Because that is something that, first of all, great value for your money, but second of all, it's one of those things where all of a sudden, you're going to find yourself staring at a $500 million bill from Google Cloud or AWS, and be like, “How did we do this to ourselves?” Right? Like, that's really expensive for the amount of money we're making. I don't know what the actual bill number is, but you know, it's something crazy like that. And then you have to be like, “Okay, how do we get everything off of Google Cloud and onto AWS because it's cheaper.” And that was a—[laugh] that was one heck of a migration, I'll tell you.Julie: So, you've walked us through AWS and through Snap, and so far, we've learned important things such as no data centers within data centers—Sam: [laugh].Julie: —people are important, and you should focus on your tooling, your internal tooling. So, as you mentioned before, you know, now you're at Gremlin. What are you excited about?Sam: Yeah. I think there's, like, a lot of value that Gremlin provides to our customers. I don't know, one of the things I liked working at Snapchat is, like, I don't particularly like Facebook. I have not liked Facebook since, like, 2007, or something. And there's, like, a real, like, almost, like, parasitic aspect to it.In my work at Snap, I felt a lot better. It's easy to say something pithy, like, “Oh, you're just sending disappearing photos.” Like, yeah, but, like, it's a way people stay connected that's not terrible the way that Facebook is, right? I felt better about my contribution.And so similarly, like, I think Gremlin was another area where, like, I feel a lot be—like, I'm actually helping my customers. I'm not just, like, helping them down a poor path. There's some, like, maybe ongoing conversation around if you worked in Amazon, like, what happens in FCs and stuff? I didn't work in that part of the company, but like, I think if I had to go back and work there, that's also something that might, you know, weigh on me to some degree. And so one of the—I think one of the nice things about working at Gremlin is, like, I feel good about my work if that makes sense.And I didn't expect it. I mean, that's not why I picked the job, but I do like that. That is something that makes me feel good. I don't know how much I can talk about upcoming product stuff. Obviously, I'm very excited about upcoming product stuff that we're building because, like, that's where I spend all my time. I'm, like, “Oh, there's, like, this thing and this thing, and that's going to let people do this. And then you can do this other thing.”I will tell you, like, I do—like, when I conceptualize product changes, I spend a lot of time thinking, how is this going to impact individual engineers? How is this going to impact their management chain, and their, like, senior leadership director, VP, C-suite level? And, like, how do we empower engineers to, like, show that senior leadership that work is getting done? Because I do think it's hard—this is true across DevOps and it's not unique to Chaos Engineering—I do think it's hard sometimes to show that you're making progress in, like, the outages you avoided, right? And, like, that is where I spend, like, a lot of my thought time, like, how do I like help doing that?And, like, if you're someone who's, like, a champion, you're, you're like, “Come on, everyone, we should be doing Chaos Engineering.” Like, how do I get people invested? You care, you're at this company, you've convinced them to purchase Gremlin, like, how do I get other engineers excited about Chaos Engineering? I think, like, giving you tools to help with that is something that, I would hope, I mean, I don't know what's actually implemented just yet, but I'd hope is somewhere on our roadmap. Because that's the thing like, that I personally think a lot about.I'll tell you another story. This was also when I was at Amazon. I had this buddy, we'll call him Zach because that's his name, and he was really big on testing. And he had all this stuff about, like, testing pyramid, if you're familiar with, like, programming unit testing, integration testing, it's all that stuff. And he worked as a team—a sister team to mine—and a lot engineers did not care heavily about testing. [laugh].And he used to try to, like, get people to, like, do things and talk about it and stuff. They just, like, didn't care, even slightly. And I also kind of didn't care, so I wasn't any better, but something I did one day on my team is I was like, “You know, somebody else at Amazon”—because Amazon invested very heavily in developer tools—had built some way that was very easy to publish metrics into our primary metrics thing about code coverage. And so I just tossed in all the products for my team, and that published a bunch of metrics. And then I made a bunch of graphs on a wiki somewhere that pulled live data, and we could see code coverage.And then I, like, showed it in, like, a team meeting one week, and everyone was like, “Oh, that's kind of interesting.” And then people were like, “Oh, I'm surprised that's so low.” And they found, like, some low-hanging fruit and they started moving it up. And then, like, the next year bi-weekly with our skip-level, like, they showed the progress, he's like, “Oh, this it's really good.” You made, like, a lot of progress in the code coverage.And then, like, all of a sudden, like, when they're inviting new changes, they start adding testing, or, like, all sudden, like, code coverage, just seemed ratchet up. Or some [unintelligible 00:30:51] would be like, “Hey, I have this thing so that our builds would fail now if code coverage went down.” Right? Like, all of a sudden, it became, sort of like, part of the culture to do this, to add coverage. I remember—and they, like, sort of pollinated to the sister teams.I remember Zach coming by my desk one day. He's like, “I'm so angry. I've been trying for six months to get people to care. And you do some dumb graphs and our wiki.” And I'm like, “I mean, I don't know. I was just, like, an idea I had.” Right? Like, it wasn't, like, a conscious, like, “I'm going to change the culture moment,” it was very much, like, “I don't know, just thought this was interesting.”And I don't know if you know who [John Rauser](https://www.youtube.com/watch?v=UL2WDcNu_3A) is, but he's got this great talk at Velocity back in 2010, maybe 2011, where he talks about culture change and he talks about how humans do change culture readily—and, you know, Velocity is very much about availability and latency—and what we need to do in the world of DevOps and reliability in general is actually we have to change the culture of the companies we're at. Because you're never going to succeed, just, like, here emoting adding chaos engineering into your environment. I mean because one day, you're going to leave that company, or you're going to give up and there'll be some inertia that'll carry things forward, but eventually, people will stop doing it and the pendulum will swing back the other way, and the systems will become unreliable again. But if you can build a culture, if you can make people care—of course, it's the hardest thing to do in engineering, like, make other engineers care about something—but if you can do it, then it will become sort of self-perpetuating, right, and it becomes, like, a sort of like a stand-alone complex. And then it doesn't matter if it's just you anymore.And as an engineer, I'm always looking for ways to, like, remove myself as a critical dependency, right? Like, if I could work myself out of a job, thank you, because, like, [laugh] yeah, I can go work on something else now, right? Like, I can be done, right? Because, like, as we all know, you're never done with software, right? There's always a next version; there's always, like, another piece; you're always, like, migrating to a new version, right? It never really ends, but if you can build something that's more than just yourself—I feel like this is, like, a line from Batman or something. “Mr. Wayne, if you can become a legend”—right? Like, you'd be something more yourself? Yeah, absolutely. I mean, it's not a great delivery like Liam Neeson. But yeah.Jason: I like what you said, though. You talked about, like, culture change, but I think a big thing of what you did is exposing what you're measuring or starting to measure this thing, right? Because there's always a statement of, “You can't improve until you measure it,” right? And so I think simply because we're engineers, exposing that metric and understanding where we're at is a huge motivator, and can be—and obviously, in your case—enough to change that culture is just, like, knowing about this and seeing that metric. And part of the whole DevOps philosophy is the idea that people want to do the best job that they can, and so exposing that data of, “Look, we're not doing very well on this,” is often enough. Just knowing that you're not doing well, is often enough to motivate you to do better.Sam: Yeah, one of the things we used to say at Amazon is, “If you can't measure it, it didn't happen.” And like, it was very true, right? I mean, that was a large organization that moves slowly, but, like, it was very true that if you couldn't show a bunch of graphs or reports somewhere, oftentimes people would just pretend like it never happened.Julie: So, I do you want to bring it back just a little bit, in the last couple of minutes that we have, to Pokémon. So, you play Pokémon Go?Sam: I do. I do play Pokémon Go.Julie: And then how do people find you on Pokémon Go?Sam: My trainer—Jason: Also, I'm going to say, Sam, you need to open my gifts. I'm in Estonia.Sam: [laugh]. It's true. I don't open gifts. Here's the problem. I have no space because I have, like, all these items from all the, like, quests and stuff they've done recently.They're like, “Oh, you got to, like, make enough space, or you could pay us $2 and we'll give you more space.” I'm like, “I'm not paying $2,” right? Like—Jason: [laugh].Sam: And so, I just, like, I have to go in every now and then and, like, just, like, delete a bunch of, like, Poké Balls or something. Like maybe I don't need 500 Poké Balls. That's fair.Jason: I mean, I'm sitting on 628 Ultra Balls right now. [laugh].Sam: Yeah. Well, maybe you don't need—Jason: It's community day on Sunday.Sam: I know, I know. I'm excited for it. I have a trainer code. If you need my trainer to find me on Pokémon Go, it's 1172-0487-4013. And you can add me, and I'll add you back because, like, I don't care; I love playing Pokémon, and I'd play every day. [laugh].Julie: And I feel it would be really rude to leave Jason out of this since he plays Pokémon a lot. Jason, do you want to share your…Jason: I'm not sharing my trainer code because at this point, I'm nearing the limit, and I have all of these Best Friends that I'm actually Lucky Friends with, and I have no idea how to contact them to actually make Lucky trades. And I know that some of them are, like, halfway around the world, so if you are in the Canary Islands and you are a friend of mine on Pokémon Go, please reach out to me on Twitter. I'm @gitbisect on Twitter. Message me so that we can actually, like, figure out who you are. Because at some point, I will go to the Canary Islands because they are beautiful.Sam: Also, you can get those, like, sweet Estonia gifts, what will give you those eggs from Estonia, and then when you trade them you get huge mileage on the trades. I don't know if this is a thing you [unintelligible 00:36:13], Jason, but, like, my wife and I both compete for who can get the most mileage on the trip. And of course, we traded each other but that's, like, a zero-sum game, right? And so the total mileage on trades is a big thing in my house.Jason: Well, the next time we get together, I've got stuff from New Zealand, so we can definitely get some mileage there.Sam: Excellent.Julie: Well, this is excellent. I feel like we have learned so much on this episode of Break Things on Purpose, from obviously the most important information out there—Pokémon—but back to some of the history of Nintendo and Amazon and Snap and all of it. And so Sam, I just want to thank you for being on with us today. And folks again, if you want to be Sam's friend on Pokémon Go—I'm sorry, I don't really know how it works. I don't even know if that's the right term—Sam: It's fine.Julie: You've got his code. [laugh]. And thanks again for being on our podcast.Jason: For links to all the information mentioned, visit our website at gremlin.com/podcast. If you liked this episode, subscribe to the Break Things on Purpose podcast on Spotify, Apple Podcasts, or your favorite podcast platform. Our theme song is called, “Battle of Pogs” by Komiku, and it's available on loyaltyfreakmusic.com.

In this episode, we cover: 00:00:00 - Introduction 00:30:00 - Fastly Outage 00:04:05 - Salesforce Outage 00:07:25 - Hypothesizing  00:10:00 - Julie Joins the Team! 00:14:05 - Looking Forward/Outro TranscriptJason: There's a bunch of cruft that they'll cut from the beginning, and plenty of stupid things to cold-open with, so.Julie: I mean, I probably should have not said that I look forward to more incidents.[audio break 00:00:12]Jason: Hey, Julie. So, it's been quite a year, and we're going to do a year-end review episode here. As with everything, this feels like a year of a lot of incidents and outages. So, I'm curious, what is your favorite outage of the year?Julie: Well, Jason, it has been fun. There's been so many outages, it's really hard to pick a favorite. I will say that one that sticks out as my favorite, I guess, you could say was the Fastly outage, basically because of a lot of the headlines that we saw such as, “Fastly slows down and stops the internet.” You know, “What is Fastly and why did it cause an outage?” And then I think that people started realizing that there's a lot more that goes into operating the internet. So, I think from just a consumer side, that was kind of a fun one. I'm sure that the increases in Google searches for Fastly were quite large in the next couple of days following that.Jason: That's an interesting thing, right? Because I think for a lot of us in the industry, like, you know what Fastly is, I know what Fastly is; I've been friends with folks over there for quite a while and they've got a great service, but for everybody else out there in the general public, suddenly, this company, they never heard of that, you know, handles, like, 25% of the world's internet traffic, like, is suddenly on the front page news and they didn't realize how much of the internet runs through this service. And I feel it that way with a lot of the incidents that we're seeing lately, right? We're recording this in December, and a week ago, Amazon had a rather large outage, affecting us-east-1, which it seems like it's always us-east-1. But that took down a bunch of stuff and similar, they are people, like you know, my dad, who's just like, “I buy things from Amazon. How did this crash, like, the internet?”Julie: I will tell you that my mom generally calls me—and I hate to throw her under the bus—anytime there is an outage. So, Hulu had some issues earlier this year and I got texts from my mom actually asking me if I could call any of my friends over at Hulu and, like, help her get her Hulu working. She does this similarly for Facebook. So, when that Facebook outage happened, I always—almost—know about an outage first because of my mother. She is my alerting mechanism.Jason: I didn't realize Hulu had an outage, and now it makes me think we've had J. Paul Reed and some other folks from Netflix on the show. We definitely need to have an engineer from Hulu come on the show. So, if you're out there listening and you work for Hulu, and you'd like to be on the show and dish all the dirt on Hulu—actually don't do that, but we'd love to talk with you about reliability and what you're doing over there at Hulu. So, reach out to us at podcast@gremlin.com.Julie: I'm sure my mother would appreciate their email address and phone number just in case—Jason: [laugh].Julie: —for the future. [laugh].Jason: If you do reach out to us, we will connect you with Julie's mother to help solve her streaming issues. You had mentioned one thing though. You said the phrase about throwing your mother under the bus, and that reminds me of one of my favorite outages from this year, which I don't know if you remember, it's all about throwing people under the bus, or one person in particular, and that's the Salesforce outage. Do you remember that?Julie: Oh. Yes, I do. So, I was not here at the time of the Salesforce outage, but I do remember the impact that that had on multiple organizations. And then—Jason: Yes—Julie: —the retro.Jason: —the Salesforce outage was one where ,similarly ,Salesforce affects so much, and it is a major name. And so people like my dad or your mom probably knew like, “Oh, Salesforce. That's a big thing.” The retro on it, I think, was what really stood out. I think, you know, most people understand, like, “Oh, you're having DNS issues.” Like, obviously it's always DNS, right? That's the meme: It's always DNS that causes your issues.In this case it was, but their retro on this they publicly published was basically, “We had an engineer that went to update DNS, and this engineer decided to push things out using an EBF process, an Emergency Brake Fix process.” So, they sort of circumvented a lot of the slow rollout processes because they just wanted to get this change made and get it done without all the hassle. And turns out that they misconfigured it and it took everything down. And so the entire incident retro was basically throwing this one engineer under the bus. Not good.Julie: No, it wasn't. And I think that it's interesting because especially when I was over at PagerDuty, right, we talked a lot about blamelessness. That was very not blameless. It doesn't teach you to embrace failure, it doesn't show that we really just want to take that and learn better ways of doing things, or how we can make our systems more resilient. But going back to the Fastly outage, I mean, the NPR headline was, “Tuesday's Internet Outage was Caused by One Customer Changing a Setting, Fastly says.” So again, we could have better ways of communicating.Jason: Definitely don't throw your engineers on their bus, but even moreso, don't throw your customers under the bus. I think for both of these, we have to realize, like, for the engineer at Salesforce, like, the blameless lesson learned here is, what safeguards are you going to put in place? Or what safeguards were there? Like, obviously, this engineer thought, like, “The regular process is a hassle; we don't need to do that. What's the quickest, most expedient way to resolve the issue or get this job done?” And so they took that.And similarly with the customer at Fastly, they're just like, “How can I get my systems working the way I want them to? Let's roll out this configuration.” It's really up to all of us, and particularly within our companies, to think about how are people using our products. How are they working on our systems? And, what are the guardrails that we need to put in place? Because people are going to try to make the best decisions that they can, and that obviously means getting the job done as quickly as possible and then moving on to the next thing.Julie: Well, and I think you're really onto something there, too, because I think it's also about figuring out those unique ways that our customers can break our products, things that we didn't think through. And I mean, that goes back to what we do here at Gremlin, right? Then that goes back to Chaos Engineering. Let's think through a hypothesis. Let's see, you know, what if ABC Company, somebody there does something. How can we test for that?And I think that shouldn't get lost in the whole aspect of now we've got this postmortem. But how do we recreate that? How do we make sure that these things don't happen again? And then how do we get creative with trying to figure out, well, how can we break our stuff?Jason: I definitely love that. And that's something that we've done internally at Gremlin this year is, we've really started to build up a better practice around running Chaos Engineering internally on our own systems. We've done that for a long time, but a lot of times it was just specific teams, and so earlier this year, the advocacy team was partnering up with the various engineering teams and running Chaos Engineering experiments. And it was interesting to learn and think through some of those ideas of as we're doing this work, we're going to be trying to do things expediently with the least amount of hassle, but what if we decide to do something that's outside of the documented process, but for which there is no technical guardrails? So, some of the things that we ended up doing were testing dependencies, right, things that again, are outside of the normal process.Like, we use LaunchDarkly for feature flagging. What happens if we decide to circumvent that, just push things straight to production? What happens if we decide to just block LaunchDarkly all together? And we found some actual critical issues and we're able to resolve those without impacting our customers.Julie: That's the key element: Practice, play, think through the what ifs. And I love the what ifs part. You know, going back to my past, I have to tell you that the IT team used to always give me all of the new tech because if something was going to break for some reason—they used to call me the “AllSpark”  to be honest with everybody out there—for some reason, if something was going to break, with me it would break in the most unique possible way, so before anything got rolled out to the entire company, I was the one that got to test it.Jason: That's amazing. So, what you're saying is on my next project, I need to give that to you first?Julie: Oh, a hundred percent. Really, it was remarkable how things would break. I mean, I had keyboards that would randomly type letters. I definitely took down some internal things, but I'm just saying that you should leverage those people within your organization, as well. The thing was, it was never a, “Julie is awful; things break because of Julie.” It was, “You know what? Leverage Julie to learn about what we're using.” And it was kind of fun. I mean, granted, this was years ago, and that name has stuck, and sometimes they still definitely make fun of me for it, but really, they just used me to break things in unique ways. Because I did.Jason: That's actually a really good segue to some of the stuff that we've been doing because you joined Gremlin, now, a few months back—more than a few months—but late summer, and a lot of what we were doing early on was just, we had these processes that, internally for myself and other folks who'd been around for a while, it was just we knew what to do because we'd done it so much. And it was that nice thing of we're going to do this thing, but let's just have Julie do it. Also, we're not going to tell you anything; we're just going to point you at the docs. It became really evident as you went through that of, like, “Hey, this doc is missing this thing. It doesn't make sense.”And you really helped us improve some of those documentation points, or some of the flows that we had, you would execute, and it's like, “Why are we doing it this way?” And a lot of times, it was like, “Oh, that's a legacy thing. We do it because—oh, right, that thing we did it because of doesn't exist anymore. Like, we're doing it completely backwards because of some sort of legacy thing that doesn't exist. Let's update that.” And you were able to help us do that, which was fantastic.Julie: Oh, yeah. And it was really great on my end, too because I always felt like I could ask the questions. And that is a cultural trait that is really important in an organization, to make sure that folks can ask questions and feel comfortable doing so. I've definitely seen it the other way, and when folks don't know the right way to do something or they're afraid to ask those questions, that's also where you see the issues with the systems because they're like, “Okay, I'm just going to do this.” And even going back to my days of being a recruiter—which is when I started in tech, but don't worry, everybody, I was super cool; I was not a bad recruiter—that was something that I always looked for in the interview process. When I'd ask somebody how to do something, would they say, “I don't know, I would ask,” or, “I would do this,” or would they just fumble their way through it, I think that it's important that organizations really adopt that culture of again, failure, blamelessness, It's okay to ask questions.Jason: Absolutely. I think sort of the flip side of that, or the corollary of that is something that Alex Hidalgo brought up. So, one of our very first episodes of 2021 on this podcast, we had Alex Hidalgo who's now at Nobl9, and he brought up a thing from his time at Google called Hyrum's Law. And Hyrum's Law is this guy Hyrum who worked at Google basically said, “If you've got an API, that API will be used in every way possible. If you don't actually technically prevent it, somebody is going to use your API in a way it wasn't designed for. And that because it allows that, it becomes totally, like, a plausible or a valid use case for this.”And so as we think about this, and thinking about blamelessness, use the end-runaround to deploy this DNS change, like, that's a valid process now because you didn't put anything in place to validate against it, and to guarantee that people weren't using it in ways that were not intended.Julie: I think that that makes a lot of sense. Because I know I've definitely used things in ways that were not intended, which people can go back and look at my quest for Diet Cherry 7 Up during the pandemic, when I used tools in ways they weren't intended, but I would like to say that Diet Cherry 7 Up is back, from those tools. Thank you PagerDuty and some APIs that were open to me to be able to leverage in interesting ways.Jason: If you needed an alert for Diet Cherry 7 Up, PagerDuty, I guess it's a good enough tool for that.Julie: Well, the fact is, is I [laugh] was able to get very creative. I mean, what are terms of service, Jason?Jason: I don't know. Does anybody actually read those?Julie: Yeah. I would call them ‘light guardrails.'Jason: [laugh]. So Julie, we're getting towards the end of the year. I'm curious, what are you looking forward to in 2022?Julie: Well, aside from, ideally, the end to the pandemic, I would say that one of the things that I'm looking forward to in 2022, from joining Gremlin, I had a really great opportunity to work on certifications here, and I'm really excited because in 2022 we'll be launching some more certifications and I'm excited for what we're going to do with that and getting creative around that. But I'm also really interested to just see how everybody evolves or learns from this year and the outages that we had. I always love fun outages, so I'm kind of curious what's going to happen over the holiday season to see if we see anything new or interesting. But Jason, what about you? What are you looking forward to?Jason: You know I, similarly, am looking forward to the end of the pandemic. I don't know if there's really going to be an end, but I think we're starting to see a return to some normalcy. And so, we've already participated in some great events, went to KubeCon a couple months ago, went to Amazon re:Invent a few weeks ago, and both of those were fantastic just to see people getting out there, and learning, and building things again. So, I'm super excited for this next year. I think we're going to start seeing a lot more events back in person, and a lot of people really eager to get together to learn and build things together. So, that's what I'm excited about. Hopefully, less incidents, but as systems get more complex, I'm not sure that that's going to happen. So, at least if we don't have less incidents, more learning from incidents is really what I'm hoping for.Julie: I like how I'm looking forward to more incidents and you're looking forward to less. To be fair, from my perspective, every incident that we have is an opportunity to talk about something new and to teach folks things, and just sometimes it's fun going down the rabbit holes to find out, well, what was the cause of this? And what was the outcome? So, when I say more incidents, I don't mean that I don't want to be able to watch the Queen's Gambit on Netflix, okay, J. Paul? Just throwing that out there.Jason: Well, thanks, Julie, for being on. And for all of our listeners, whether you're seeing more incidents or less incidents, Julie and I both hope that you're learning from the incidents that you have, that you're working to become more reliable and building more reliable systems, and hopefully testing them out with some chaos engineering. If you'd like to hear more from the Break Things on Purpose podcast, we've got a bunch of episodes that we've published this year, so if you haven't heard some of them, go back into our catalog. You can see all of the episodes at gremlin.com/podcast. And we look forward to seeing you in our next podcast.Jason: For links to all the information mentioned, visit our website at gremlin.com/podcast. If you liked this episode, subscribe to the Break Things on Purpose podcast on Spotify, Apple Podcasts, or your favorite podcast platform. Our theme song is called “Battle of Pogs” by Komiku, and it's available on loyaltyfreakmusic.com.

In this episode, we cover: 00:00:00 - Introduction  00:04:30 - Early Dark Days in Chaos Engineering and Reliability 00:08:27 - Anecdotes from the “Long Dark Time” 00:16:00 - The Big Changes Over the Years 00:20:50 - Mandi's Work at PagerDuty 00:27:40 - Mandi's Tips for Better DevOps 00:34:15 - Outro Links:PagerDuty: https://www.pagerduty.com TranscriptJason: — hilarious or stupid?Mandi: [laugh]. I heard that; I listened to the J. Paul Reed episode and I was like, “Oh, there's, like, a little, like, cold intro.” And I'm like, “Oh, okay.”Jason: Welcome to Break Things on Purpose, a podcast about reliability and learning from failure. In this episode, we take a trip down memory lane with Mandi Walls to discuss how much technology, reliability practices, and chaos engineering has evolved over her extensive career in technology.Jason: Everybody, welcome to the show, Julie Gunderson, who recently joined Gremlin on the developer advocacy team. How's it going, Julie?Julie: Great, Jason. Really excited to be here.Jason: So, Mandi is actually a guest of yours. I mean, we both have been friends with Mandi for quite a while but you had the wonderful opportunity of working with Mandi.Julie: I did, and I was really excited to have her on our podcast now as we ran a podcast together at PagerDuty when we worked there. Mandi has such a wealth of knowledge that I thought we should have her share it with the world.Mandi: Oh, no. Okay.Julie: [laugh].Jason: “Oh, no?” Well, in that case, Mandi, why don't you—Mandi: [crosstalk 00:01:28]. I don't know.Jason: Well, in that case with that, “Oh no,” let's have Mandi introduce herself. [laugh].Mandi: Yeah hi. So, thanks for having me. I am Mandi Walls. I am currently a DevOps advocate at PagerDuty, Julie's last place of employment before she left us to join Jason at Gremlin.Julie: And Mandi, we worked on quite a few things over a PagerDuty. We actually worked on things together, joint projects between Gremlin, when it was just Jason and us where we would run joint workshops to talk about chaos engineering and actually how you can practice your incident response. And I'm sure we'll get to that a little bit later in the episode, but will you kick us off with your background so everybody knows why we're so excited to talk to you today?Mandi: Oh, goodness. Well, so I feel like I've been around forever. [laugh]. Prior to joining PagerDuty. I spent eight-and-a-half years at Chef Software, doing all kinds of things there, so if I ever trained you on Chef, I hope it was good.Prior to joining Chef, I was assistant administrator for AOL.com and a bunch of other platform and sites at AOL for a long time. So, things like Moviefone, and the AOL Sports Channel, and dotcom, and all kinds of things. Most of them ran on one big platform because the monolith was a thing. So yeah, my background is largely in operations, and just systems administration on that side.Jason: I'm laughing in the background because you mentioned Moviefone, and whenever I think of Moviefone, I think of the Seinfeld episode where Kramer decides to make a Moviefone competitor, and it's literally just his own phone number, and people call up and he pretends to be that, like, robotic voice and has people, like, hit numbers for which movie they want to see and hear the times that it's playing. Gives a new meaning to the term on-call.Mandi: Indeed. Yes, absolutely.Julie: And I'm laughing just because I recently watched Hackers and, you know, they needed that AOL.com disc.Mandi: That's one of my favorite movies. Like, it's so ridiculous, but also has so many gems of just complete nonsense in it. Absolutely love Hackers. “Hack the planet.”Julie: “Hack the planet.” So, with hacking the planet, Mandi, and your time working at AOL with the monolith, let's talk a little bit because you're in the incident business right now over at PagerDuty, but let's talk about the before times, the before we practiced Chaos Engineering and before we really started thinking about reliability. What was it like?Mandi: Yeah, so I'll call this the Dark Ages, right? So before the Enlightenment. And, like, for folks listening at home, [laugh] the timeline here is probably—so between two-thousand-and-fi—four, five, and 2011. So, right before the beginning of cloud, right before the beginning of, like, Infrastructure as Code, and DevOps and all those things that's kind of started at, like, the end of my tenure at AOL. So, before that, right—so in that time period, right, like, the web was, it wasn't like it was just getting started, but, like, the Web 2.0 moniker was just kind of getting a grip, where you were going from the sort of generic sites like Yahoo and Yellow Pages and those kinds of things and AOL.com, which was kind of a collection of different community bits and news and things like that, into more personalized experiences, right?So, we had a lot of hook up with the accounts on the AOL side, and you could personalize all of your stuff, and read your email and do all those things, but the sophistication of the systems that we were running was such that like, I mean, good luck, right? It was migration from commercial Unixes into Linux during that era, right? So, looking at when I first joined AOL, there were a bunch of Solaris boxes, and some SGIs, and some other weird stuff in the data center. You're like, good luck on all that. And we migrated most of those platforms onto Linux at that time; 64 bit. Hurray.At least I caught that. And there was an increase in the use of open-source software for big commercial ventures, right, and so less of a reliance on commercial software and caught solutions for things, although we did have some very interesting commercial web servers that—God help them, they were there, but were not a joy, exactly, to work on because the goals were different, right? That time period was a huge acceleration. It was like a Cambrian explosion of software pieces, and tools, and improvements, and metrics, and monitoring, and all that stuff, as well as improvements on the platform side. Because you're talking about that time period is also being the migration from bare metal and, like, ordering machines by the rack, which really only a handful of players need to do that now, and that was what everybody was doing then.And in through the earliest bits of virtualization and really thinking about only deploying the structures that you needed to meet the needs of your application, rather than saying, “Oh, well, I can only order gear, I can only do my capacity planning once a year when we do the budget, so like, I got to order as much as they'll let me order and then it's going to sit in the data center spinning until I need it because I have no ability to have any kind of elastic capacity.” So, it was a completely, [laugh] completely different paradigm from what things are now. We have so much more flexibility, and the ability to, you know, expand and contract when we need to, and to shape our infrastructures to meet the needs of the application in such a more sophisticated and almost graceful way that we really didn't have then. So, it was like, “Okay, so I'm running these big websites; I've got thousands of machines.” Like, not containers, not services.Like, there's tens of thousands of services, but there's a thousand machines in one location, and we've got other things spread out. There's like, six different pods of things in different places and all this other crazy business going on. At the same time, we were also running our own CDN, and like, I totally recommend you never, ever do that for any reason. Like, just—yeah. It was a whole experience and I still sometimes have, like, anxiety dreams about, like, the configuration for some of our software that we ran at that point. And all of that stuff is—it was a long… dark time.Julie: So, now speaking of anxiety dreams, during that long, dark time that you mentioned, there had to have been some major incidents, something that stands out that that you just never want to relive. And, Mandi, I would like to ask you to relive that for us today.Mandi: [laugh]. Okay, well, okay, so there's two that I always tell people about because they were so horrific in the moment, and they're still just, like, horrible to think about. But, like, the first one was Thanksgiving morning, sometime early in the morning, like, maybe 2 a.m. something like that, I was on call.I was at my mom's, so at the time, my mom had terrible internet access. And again, this time period don't have a lot of—there was no LTE or any kind of mobile data, right? So, I'm, like, on my mom's, like, terrible modem. And something happened to the database behind news.aol.com—which was kind of a big deal at the time—and unfortunately, we were in the process of, like, migrating off of one kind of database onto another kind of database.News was on the target side but, like, the actual platform that we were planning to move to for everything else, but the [laugh] database on-call, the poor guy was only trained up in the old platform, so he had no idea what was going on. And yeah, we were on that call—myself, my backup, the database guy, the NOC analyst, and a handful of other people that we could get hold of—because we could not get into touch with the team lead for the new database platform to actually fix things. And that was hours. Like, I missed Thanksgiving dinner. So, my family eats Thanksgiving at midday rather than in the evening. So, that was a good ten hour call. So, that was horrifying.The other one wasn't quite as bad as that, but like, the interesting thing about the platform we were running at the time was it was AOL server, don't even look it up. Like, it was just crazytown. And it was—some of the interesting things about it was you could actually get into the server platform and dig around in what the threads were doing. Each of the servers had, like, a control port on it and I could log into the control port and see what all the requests were doing on each thread that was live. And we had done a big push of a new release of dotcom onto that platform, and everything fell over.And of course, we've got, like, sites in half a dozen different places. We've got, you know, distributed DNS that's, like, trying to throw traffic between different locations as they fall over. So, I'm watching, like, all of these graphs oscillate as, like, traffic pours out of the [Secaucus 00:11:10] or whatever we were doing, and into Mountain View or something and, like, then all the machines in the Secaucus recover. So, then they start pinging and traffic goes back, and, like, they just fall over, over and over again. So, what happened there was we didn't have enough threads configured in the server for the new time duration for the requests, so we had to, like, just boosted up all of the threads we could handle and then restart all of the applications. But that meant pushing out new config to all the thousands of servers that were in the pool at the time and then restarting all of them. So, that was exciting. That was the outage that I learned that the CTO knew how to call my desk. So, highly don't recommend that. But yeah, it was an experience. So.Julie: So, that's really interesting because there's been so many investments now in reliability. And when we talk about the Before Times when we had to cap our text messages because they cost us ten cents a piece, or when we were using those AOL discs, the thought was there; we wanted to make that user experience better. And you brought up a couple of things, you know, you were moving to those more personalized experiences, you were migrating those platforms, and you actually talked about your metrics and monitoring. And I'd like to dig in a little on that and see, how did that help you during those incidents? And after those incidents, what did you do to ensure that these types of incidents didn't occur again in the future?Mandi: Yeah, so one of the interesting things about, you know, especially that time period was that the commercially available solutions, even some of the open-source solutions were pretty immature at that time. So, AOL had an internally built solution that was fascinating. And it's unfortunate that they were never able to open-source it because it would have been something interesting to sort of look at. Scale of it was just absolutely immense. But the things that we could look at the time to sort of give us, you know, an indication of something, like, an AOL.com, it's kind of a general purpose website; a lot of different people are going to go there for different reasons.It's the easiest place for them to find their email, it's the easiest place for them to go to the news, and they just kind of use it as their homepage, so as soon as traffic starts dropping off, you can start to see that, you know, maybe there's something going on and you can pull up sort of secondary indicators for things like CPU utilization, or memory exhaustion, or things like that. Some of the other interesting things that would come up there is, like, for folks who are sort of intimately tied to these platforms for long periods of time, to get to know them as, like, their own living environment, something like—so all of AOL's channels at the time were on a single platform.—like, hail to the monolith; they all live there—because it was all linked into one publishing site, so it made sense at the time, but like, oh, my goodness, like, scaling for the combination of entertainment plus news plus sports plus all the stuff that's there, there's 75 channels at one time, so, like, the scaling of that is… ridiculous.But you could get a view for, like, what people were actually doing, and other things that were going on in the world. So like, one summer, there were a bunch of floods in the Midwest and you could just see the traffic bottom out because, like, people couldn't get to the internet. So, like, looking at that region, there's, like, a 40% drop in the traffic or whatever for a few days as people were not able to be online. Things like big snowstorms where all the kids had to stay home and, like, you get a big jump in the traffic and you get to see all these things and, like, you get to get a feel for more of a holistic attachment or holistic relationship with a platform that you're running. It was like it—they are very much a living creature of their own sort of thing.Like, I always think of them as, like, a Kraken or whatever. Like, something that's a little bit menacing, you don't really think see all of it, and there's a lot of things going on in the background, but you can get a feel for the personality and the shape of the behaviors, and knowing that, okay, well, now we have a lot of really good metrics to say, “All right, that one 500 error, it's kind of sporadic, we know that it's there, it's not a huge deal.” Like, we did not have the sophistication of tooling to really be able to say that quantitatively, like, and actually know that but, like, you get a feel for it. It's kind of weird. Like, it's almost like you're just kind of plugged into it yourself.It's like the scene in The Matrix where the operator guy is like, “I don't even see the text anymore.” Right? Like, he's looking directly into the matrix. And you can, kind of like—you spend a lot of time with [laugh] those applications, you get to know how they operate, and what they feel like, and what they're doing. And I don't recommend it to anyone, but it was absolutely fascinating at the time.Julie: Well, it sounds like it. I mean, anytime you can relate anything to The Matrix, it is going to be quite an experience. With that said, though, and the fact that we don't operate in these monolithic environments anymore, how have you seen that change?Mandi: Oh, it's so much easier to deal with. Like I said, like, your monolithic application, especially if there are lots of different and diverse functionalities in it, like, it's impossible to deal with scaling them. And figuring out, like, okay, well, this part of the application is memory-bound, and here's how we have to scale for that; and this part of the application is CPU-bound; and this part of the application is I/O bound. And, like, peeling all of those pieces apart so that you can optimize for all of the things that the application is doing in different ways when you need to make everything so much smoother and so much more efficient, across, like, your entire ecosystem over time, right?Plus, looking at trying to navigate the—like an update, right? Like, oh, you want to do an update to your next version of your operating system on a monolith? Good luck. You want to update the next version of your runtime? Plug and pray, right? Like, you just got to hope that everybody is on board.So, once you start to deconstruct that monolith into pieces that you can manage independently, then you've got a lot more responsibility on the application teams, that they can see more directly what their impacts are, get a better handle on things like updates, and software components, and all the things that they need independent of every other component that might have lived with them in the monolith. Noisy neighbors, right? Like, if you have a noisy neighbor in your apartment building, it makes everybody miserable. Let's say if you have, like, one lagging team in your monolith, like, nobody gets the update until they get beaten into submission.Julie: That is something that you and I used to talk about a lot, too, and I'm sure that you still do—I know I do—was just the service ownership piece. Now, you know who owns this. Now, you know who's responsible for the reliability.Mandi: Absolutely.Julie: You know, I'm thinking back again to these before times, when you're talking about all of the bare metal. Back then, I'm sure you probably didn't pull a Jesse Robbins where you went in and just started unplugging cords to see what happened, but was there a way that AOL practiced Chaos Engineering with maybe not calling it that?Mandi: It's kind of interesting. Like, watching the evolution of Chaos Engineering from the early days when Netflix started talking about it and, like, the way that it has emerged as being a more deliberate practice, like, I cannot say that we ever did any of that. And some of the early internet culture, right, is really built off of telecom, right? It was modem-based; people dialed into your POP, and like, that was the reliability they were expecting was very similar to what they expect out of a telephone, right? Like, the reason we have, like, five nines as a thing is because you want to pick up dial tone, and—pick up your phone and get dial tone on your  line 99.999% of the time.Like, it has nothing to do with the internet. It's like 1970s circuits with networking. For part of that reason, like, a lot of the way things were built at that time—and I can't speak for Yahoo, although I suspect they had a very similar setup—that we had a huge integration environment. It's completely insane to think now that you would build an integration environment that was very similar in scope and scale to your production environment; simply does not happen. But for a lot of the services that we had at that time, we absolutely had an integration environment that was extraordinarily similar.You simply don't do that anymore. Like, it's just not part of—it's not cost effective. And it was only cost effective at that time because there wasn't anything else going on. Like, you had, like, the top ten sites on the internet, and AOL was, like, number three at the time. So like, that was just kind of the way things are done.So, that was kind of interesting and, like, figuring out that you needed to do some kind of proactive planning for what would happen just wasn't really part of the culture at the time. Like, we did have a NOC and we had some amazing engineers on the NOC that would help us out and do some of the things that we automate now: putting a call together, or when paging other folks into an incident, or helping us with that kind of response. I don't ever remember drilling on it, right, like we do. Like, practicing that, pulling a game day, having, like, an actual plan for your reliability along those lines.Julie: Well, and now I think that yeah, the different times are that the competitive landscape is real now—Mandi: Yeah, absolutely.Julie: And it was hard to switch from AOL to something else. It was hard to switch from Facebook to MySpace—or MySpace to Facebook, I should say.Mandi: Yeah.Julie: I know that really ages me quite a bit.Mandi: [laugh].Julie: But when we look at that and when we look at why reliability is so important now, I think it's because we've drilled it into our users; the users have this expectation and they aren't aware of what's happening on the back end. They just kn—Mandi: Have no idea. Yeah.Julie: —just know that they can't deposit money in their bank, for example, or play that title at Netflix. And you and I have talked about this when you're on Netflix, and you see that, “We can't play this title right now. Retry.” And you retry and it pops back up, we know what's going on in the background.Mandi: I always assume it's me, or, like, something on my internet because, like, Netflix, they [don't ever 00:21:48] go down. But, you know, yeah, sometimes it's [crosstalk 00:21:50]—Julie: I just always assume it's J. Paul doing some chaos engineering experiments over there. But let's flash forward a little bit. I know we could spend a lot of time talking about your time at Chef, however, you've been over at PagerDuty for a while now, and you are in the incident response game. You're in that lowering that Mean Time to Identification and Resolution. And that brings that reliability piece back together. Do you want to talk a little bit about that?Mandi: One of the things that is interesting to me is, like, watching some of these slower-moving industries as they start to really get on board with cloud, the stairstep of sophistication of the things that they can do in cloud that they didn't have the resources to do when they were using their on-premises data center. And from an operation standpoint, like, being able to say, “All right, well, I'm going from, you know, maybe not bare metal, but I've got, like, some kind of virtualization, maybe some kind of containerization, but like, I also own the spinning disks, or whatever is going on there—and the network and all those things—and I'm putting that into a much more flexible environment that has modern networking, and you know, all these other elastic capabilities, and my scaling and all these things are already built in and already there for me.” And your ability to then widen the scope of your reliability planning across, “Here's what my failure domains used to look like. Here's what I used to have to plan for with thinking about my switching networks, or my firewalls, or whatever else was going on and, like, moving that into the cloud and thinking about all right, well, here's now, this entire buffet of services that I have available that I can now think about when I'm architecting my applications for the cloud.” And that, just, expanded reliability available to you is, I think, absolutely amazing.Julie: A hundred percent. And then I think just being able to understand how to respond to incidents; making sure that your alerting is working, for example, that's something that we did in that joint workshop, right? We would teach people how to validate their alerting and monitoring, both with PagerDuty and Gremlin through the practice of incident response and of chaos engineering. And I know that one of the practices at PagerDuty is Failure Fridays, and having those regular game days that are scheduled are so important to ensuring the reliability of the product. I mean, PagerDuty has no maintenance windows, correct?Mandi: No that—I don't think so, right?Julie: Yeah. I don't think there's any planned maintenance windows, and how do we make sure for organizations that rely on PagerDuty—Mandi: Mm-hm.Julie: —that they are one hundred percent reliable?Mandi: Right. So, you know, we've got different kinds of backup plans and different kinds of rerouting for things when there's some hiccup in the platform. And for things like that, we have out of band communications with our teams and things like that. And planning for that, having that game day to just be able to say—well, it gives you context. Being able to say, “All right, well, here's this back-end that's kind of wobbly. Like, this is the thing we're going to target with our experiments today.”And maybe it's part of the account application, or maybe it's part of authorization, or whatever it is; the team that worked on that, you know, they have that sort of niche view, it's a little microcosm, here's a little thing that they've got and it's their little widget. And what that looks like then to the customer, and that viewpoint, it's going to come in from somewhere else. So, you're running a Failure Friday; you're running a game day, or whatever it is, but including your customer service folks, and your front-end engineers, and everyone else so that, you know, “Well, hey, you know, here's what this looks like; here's the customers' report for it.” And giving you that telemetry that is based on customer experience and your actual—what the business looks like when something goes wrong deep in the back end, right, those deep sea, like, angler fish in the back, and figuring out what all that looks like is an incredible opportunity. Like, just being able to know that what's going to happen there, what the interface is going to look like, what things don't load, when things take a long time, what your timeouts look like, did you really even think about that, but they're cascading because it's actually two layers back, or whatever you're working on, like that kind of insight, like, is so valuable for your application engineers as they're improving all the pieces of architecture, whether it's the most front-end user-facing things, or in the deep back-end that everybody relies on.Julie: Well, absolutely. And I love that idea of bringing in the different folks like the customer service teams, the product managers. I think that's important on a couple of levels because not only are you bringing them into this experience so they're understanding the organization and how folks operate as a whole, but you're building that culture, that failure is acceptable and that we learn from our failures and we make our systems more resilient, which is the entire goal.Mandi: The goal.Julie: And you're sharing the learning. When we operate in silos—which even now as much as we talk about how terrible it is to be in siloed teams and how we want to remove silos, it happens. Silos just happen. And when we can break down those barriers, any way that we can to bring the whole organization in, I think it just makes for a stronger organization, a stronger culture, and then ultimately a stronger product where our customers are living.Mandi: Yeah.Julie: Now, I really do want to ask you a couple of things for some fun here. But if you were to give one tip, what is your number one tip for better DevOps?Mandi: Your DevOps is always going to be—like, I'm totally on board with John Wallace's [CAMS 00:27:57] to, like, move to CALMS sort of model, right? So, you've got your culture, your automation, your learning, your metrics, and your sharing. For better DevOps, I think one of the things that's super important—and, you know, you and I have hashed this out in different things that we've done—we hear about it in other places, is definitely having empathy for the other folks in your organization, for the work that they're doing, and the time constraints that they're under, and the pressures that they're feeling. Part of that then sort of rolls back up to the S part of that particular model, the sharing. Like, knowing what's going on, not—when we first started out years ago doing sort of DevOps consulting through Chef, like, one of the things we would occasionally run into is, like, you'd ask people where their dashboards were, like, how are they finding out, you know, what's going on, and, like, the dashboards were all hidden and, like, nobody had access to them; they were password protected, or they were divided up by teams, like, all this bonkers nonsense.And I'm like, “You need to give everybody a full view, so that they've all got a 360 view when they're making decisions.” Like you mentioned your product managers as part of, like, being part of your practice; that's absolutely what you want. They have to see as much data as your applications engineers need to see. Having that level of sharing for the data, for the work processes, for the backlog, you know, the user inputs, what the support team is seeing, like, you're getting all of this input, all this information, from everywhere in your ecosystem and you cannot be selfish with it; you cannot hide it from other people.Maybe it doesn't look as nice as you want it to, maybe you're getting some negative feedback from your users, but pass that around, and you ask for advice; you ask for other inputs. How are we going to solve this problem? And not hide it and feel ashamed or embarrassed. We're learning. All this stuff is brand new, right?Like, yeah, I feel old talking about AOL stuff, but, like, at the same time, like, it wasn't that long ago, and we've learned an amazing amount of things in that time period, and just being able to share and have empathy for the folks on your team, and for your users, and the other folks in your ecosystem is super important.Julie: I agree with that. And I love that you hammer down on the empathy piece because again, when we're working in ones and zeros all day long, sometimes we forget about that. And you even mentioned at the beginning how at AOL, you had such intimate knowledge of these applications, they were so deep to you, sometimes with that I wonder if we forget a little bit about the customer experience because it's something that's so close to us; it's a feature maybe that we just believe in wholeheartedly, but then we don't see our customers using it, or the experience for them is a little bit rockier. And having empathy for what the customer may go through as well because sometimes we just like to think, “Well, we know how it works. You should be able to”—Mandi: Yes.Julie: Yes. And, “They're definitely not going to find very unique and interesting ways to break my thing.” [laugh].Mandi: [laugh]. No, never.Julie: Never.Mandi: Never.Julie: And then you touched on sharing and I think that's one thing we haven't touched on yet, but I do want to touch on a little bit. Because with incident—with incident response, with chaos engineering, with the learning and the sharing, you know, an important piece of that is the postmortem.Mandi: Absolutely.Julie: And do you want to talk a little bit about the PagerDuty view, your view on the postmortems?Mandi: As an application piece, like, as a feature, our postmortem stuff is under review. But as a practice, as a thing that you do, like, a postmortem is an—it should be an active word; like, it's a verb, right? You hol—and if you want to call it a post-incident review, or whatever, or post-incident retrospective, if you're more comfortable with those words, like that's great, and that's—as long as you don't put a hyphen in postmortem, I don't care. So, like—Julie: I agree with you. No hyphen—Mandi: [laugh].Julie: —please. [laugh].Mandi: Please, no hyphen. Whatever you want to call that, like, it's an active thing. And you and I have talked a number of times about blamelessness and, like, making sure that what you do with that opportunity, this is—it's a gift, it's a learning opportunity after something happened. And honestly, you probably need to be running them, good or bad, for large things, but if you have a failure that impacted your users and you have this opportunity to sit down and say, all right, here's where things didn't go as we wanted them to, here's what happened, here's where the weaknesses are in our socio-technical systems, whether it was a breakdown in communication, or breakdown in documentation, or, like, we we found a bug or, you know, [unintelligible 00:32:53] defect of some kind, like, whatever it is, taking that opportunity to get that view from as many people as possible is super important.And they're hard, right? And, like, we—John Allspaw, on our podcast, right, last year talked a bit about this. And, like, there's a tendency to sort of write the postmortem and put it on a shelf like it's, like, in a museum or whatever. They are hopefully, like, they're learning documents that are things that maybe you have your new engineers sort of review to say, “Here's a thing that happened to us. What do you think about this?” Like, maybe having, like, a postmortem book club or something internally so that the teams that weren't maybe directly involved have a chance to really think about what they can learn from another application's learning, right, what opportunities are there for whatever has transpired? So, one of the things that I will say about that is like they aren't meant to be write-only, right? [laugh]. They're—Julie: Yeah.Mandi: They're meant to be an actual living experience and a practice that you learn from.Julie: Absolutely. And then once you've implemented those fixes, if you've determined the ROI is great enough, validate it.Mandi: Yes.Julie: Validate and validate and validate. And folks, you heard it here first on Break Things on Purpose, but the postmortem book club by Mandi Walls.Mandi: Yes. I think we should totally do it.Julie: I think that's a great idea. Well, Mandi, thank you. Thank you for taking the time to talk with us. Real quick before we go, did you want to talk a little bit about PagerDuty and what they do?Mandi: Yes, so Page—everyone knows PagerDuty; you have seen PagerDuty. If you haven't seen PagerDuty recently, it's worth another look. It's not just paging anymore. And we're working on a lot of things to help people deal with unplanned work, sort of all the time, right, or thinking about automation. We have some new features that integrate more with our friends at Rundeck—PagerDuty acquired Rundeck last year—we're bringing out some new integrations there for Rundeck actions and some things that are going to be super interesting for people.I think by the time this comes out, they'll have been in the wild for a few weeks, so you can check those out. As well as, like, getting better insight into your production platforms, like, with a service graph and other insights there. So, if you haven't looked at PagerDuty in a while or you think about it as being just a place to be annoyed with alerts and pages, definitely worth revisiting to see if some of the other features are useful to you.Julie: Well, thank you. And thanks, Mandi, and looking forward to talking to you again in the future. And I hope you have a wonderful day.Mandi: Thank you, Julie. Thank you very much for having me.Jason: For links to all the information mentioned, visit our website at gremlin.com/podcast. If you liked this episode, subscribe to the Break Things on Purpose podcast on Spotify, Apple Podcasts, or your favorite podcast platform. Our theme song is called “Battle of Pogs” by Komiku, and it's available on loyaltyfreakmusic.com.

In this episode, we cover:00:00:00 - Introduction 00:05:00 - Itiel's Background in Engineering00:08:25 - Improving Kubernetes Troubleshooting00:11:45 -  Improving Team Collaboration 00:14:00 - OutroLinks: Komodor: https://komodor.com/ Twitter: https://twitter.com/Komodor_com TranscriptJason: Welcome back to another episode of Build Things On Purpose, a part of the Break Things On Purpose podcast where we talk with people who have built really cool software or systems. Today with us, we have Itiel Shwartz who is the CTO of a company called Komodor. Welcome to the show.Itiel: Thanks, happy to be here.Jason: If I go to Komodor's website it really talks about debugging Kubernetes, and as many of our listeners know Kubernetes and complex systems are a difficult thing. Talk to me a little bit more—tell me what Komodor is. What does it do for us?Itiel: Sure. So, I don't think I need to tell our listeners—your listeners that Kubernetes looks cool, it's very easy to get started, but once you're into it and you have a big company with complex, like, micros—it doesn't have to be big, even, like, medium-size complex system company where you're starting to hit a couple of walls or, like, issues when trying to troubleshoot Kubernetes.And that usually is due to the nature of Kubernetes which makes making complex systems very easy. Meaning you can deploy in multiple microservices, multiple dependencies, and everything looks like a very simple YAML file. But in the end of the day, when you have an issue, when one of the pods is starting to restart and you try to figure out, like, why the hell is my application is not running as it should have, you need to use a lot of different tools, methodologies, knowledge that most people don't really have in order to solve the issue. So, Komodor focus on making the troubleshooting in Kubernetes an easy and maybe—may I dare say even fun experience by harnessing our knowledge in Kubernetes and align our users to get that digest view of the world.And so usually when you speak about troubleshooting, the first thing that come to mind is issues are caused due to changes. And the change might be deploying Kubernetes, it can be a [configurment 00:02:50] that changed, a secret that changed, or even some feature flag, or, like, LaunchDarkly feature that was just turned on and off. So, what Komodor does is we track and we collect all of the changes that happen across your entire system, and we put, like, for each one of your services a [unintelligible 00:03:06] that includes how did the service change over time and how did it behave? I mean, was it healthy? Was it unhealthy? Why wasn't it healthy?So, by collecting the data from all across your system, plus we are sit on top of Kubernetes so we know the state of each one of the pods running in your application, we give our users the ability to understand how did the system behave, and once they have an issue we allow them to understand what changes might have caused this. So, instead of bringing down dozens of different tools, trying to build your own mental picture of how the world looks like, you just go into Komodor and see everything in one place.I would say that even more than that, once you have an issue, we try to give our best efforts on helping to understand why did it happen. We know Kubernetes, we saw a lot of issues in Kubernetes. We don't try complex AI solution or something like that, but using our very deep knowledge of Kubernetes, we give our users, FYI, your pods that are unhealthy, but the node that they are running on just got restarted or is having this pressure.So, maybe they could look at the node. Like, don't drill down into the pods logs, but instead, go look at the nodes. You just upgraded your Kubernetes version or things like that. So, basically we give you everything you need in order to troubleshoot an issue in Kubernetes, and we give it to you in a very nice and informative way. So, our user just spend less time troubleshooting and more time developing features.Jason: That sounds really extremely useful, at least from my experience, in operating things on Kubernetes. I'm guessing that this all stemmed from your own experience. You're not typically a business guy, you're an engineer. And so it sounds like you were maybe scratching your own itch. Tell us a little bit more about your history and experience with this?Itiel: I started computer science, I started working for eBay and I was there in the infrastructure team. From there I joined two Israeli startup and—I learned that the thing that I really liked or do quite well is to troubleshoot issues. I was in a very, very, like, production-downtime-sensitive systems. A system when the system is down, it just cost the business a lot of money.So, in these kinds of systems, you try to respond really fast through the incidents, and you spend a lot of time monitoring the system so once an issue occur you can fix it as soon as possible. So, I developed a lot of internal tools. For the companies I worked for that did something very similar, allow you once you have an issue to understand the root cause, or at least to get a better understanding of how the world looks like in those companies.And we started Komodor because I also try to give advice to people. I really like Kubernetes. I liked it, like, a couple of years ago before it was that cool, and people just consult with me. And I saw the lack of knowledge and the lack of skills that most people that are running Kubernetes have, and I saw, like—I'd have to say it's like giving, like, a baby a gun.So, giving an operation person that doesn't really understand Kubernetes tell him, “Yeah, you can deploy everything and everything is a very simple YAML. You want a load balancer, it's easy. You want, like, a persistent storage, it's easy. Just install like—Helm install Postgres or something like that.” I installed quite a lot of, like, Helm-like recipes, GA, highly available. But things are not really highly available most of the time.So, it's definitely scratching my own itch. And my partner, Ben, is also a technical guy. He was in Google where they have a lot of Kubernetes experience. So, together both of us felt the pain. We saw that as more and more companies moved to Kubernetes, the pain became just stronger. And as the shift-left movement is also like taking off and we see more and more dev people that are not necessarily that technical that are expected to solve issues, then again we saw an issue.So, what we see is companies moving to Kubernetes and they don't have the skills or knowledge to troubleshoot Kubernetes. And then they tell their developers, “You are now responsible for the production. You are deploying? You should troubleshoot,” and the developers really don't know what to do. And we came to those companies and basically it makes everything a lot easier.You have any issue in Kubernetes? No issue, like, no issue. And no problem go to Komodor and understand what is the probable root cause. See what's the status? Like, when did it change? When was it last restarted? When was it unhealthy before today? Maybe, like, an hour ago, maybe a month ago. So, Komodor just gives you all of this information in a very informative way.Jason: I like the idea of pulling everything into one place, but I think that obviously begs the question: if we're pulling in this information we need to have good information to begin with. I'm interested in your thoughts of if someone were to use Komodor or just want to improve their visibility into troubleshooting Kubernetes, what are some tips or advice that you'd have for them in maybe how to set up their monitoring, or how to tag their changes, things like that? What does that look like?Itiel: I will say the first thing is using more metadata and tagging capabilities across the board. It can be on top of the monitors, the system, the services, like, you name it, you should do it. Once an alert is triggered, you don't necessarily have to go to the perfect playbook because it doesn't really exist. You should understand what's the relevant impact, what system it impacted, and who is the owner, and who should you wake up, like, now or who should look at it?So, spending the time tagging some of the alerts and resources in Kubernetes is super valuable. It's not that hard, but by doing so you just reduced the mental capacity needed in order to troubleshoot an issue. More than that, here in Komodor we read of this metadata label stacks, and we harness it for our own benefits. So, it is best practice to do so and Komodor also utilize this data.And for example, for an alert, say like, the relevant team name that is responsible, and for each service in Kubernetes write the team that owns this service. And this way you can basically understand what teams are responsible for what services or issues. So, this is the number one tip or trick. And the second one is just spend time on exposing these data. You can use Komodor I think, like, it's the best solution, but even if not, try to have those notification every time something change.Write those, like, web hooks to which one of your resources and let the team know that things change. If not, like, what we see in companies is something break, no one really know what changed, and in the end of the day they are forced to go into Slack and doing, like, here—someone changed something that might cause production break. And if so, please fix it. It's not a good place to be. If you see yourself asking questions over Slack, you have an issue with the system monitoring and observability.Jason: That's a great point because I feel like a lot of times we do that. And so you look back into your CI/CD logs, like, what pushes are made, what deploys are made. You're trying to parse out, like, which one was it? Especially in a high-velocity organization of multiple changes and which one actually did that breaking.Itiel: We see it across the board. There are so many changes, so many dependencies. Because microservice A talks with microservice B that speak with microservice C using SQS or something like that. And then things break and no one know what is really happening. Especially the developers, they have no idea what is happening. But most of the time also the DevOps themselves.Jason: I think that's a great point of, sort of, that shared confusion. As we've talked about DevOps and that breaking down of the walls between developers and operations, there was always this, “Well, you should work together,” and there is this notion now of we're working together but nobody knows what's going on.As we talk about this world of sharing, what are some of your advice as somebody who's helped both developers and operations? Aside from getting that shared visibility for troubleshooting, do you have any tips for collaborating better to understand as a team how things are functioning?Itiel: I have a couple of thoughts on this area. The first thing is you must have the alignment. Both the DevOps, or operation and the developers need to understand they are in this together. And this, like, base point in other organization you see they struggle. Like, the developers are like, yeah, I don't really need—like, it's the ops problem if production is down, and the ops are, like, angry at the devs and say they don't understand anything so they shouldn't be responsible for issues in production.So, first of all, let's create the alignment. The organization needs to understand that both the dev and the ops team need to take shared responsibility over the system and over the troubleshooting process. Once this very key pillar is out of the way, I will say that adding more and more tools and making sure that those tools can be shared between the ops and the dev team.Because a lot of the times we see tools that are designed for the DevOps, and a developer don't really understand what is happening here, what are those numbers, and basically how to use them. So, I think making sure the tools fit both personas is a very crucial thing. And the last thing is learning from past incidents. You are going to have other incidents, other issues. The question is, do you understand how we improve the next time this incident or a similar incident will happen? What processes and what tools are missing in the link between the DevOps and the system to optimize it. Because it's not after you snap your finger and everything works as expected.It is an iterative process and you must have, like, the state of mind of, okay, things are going to get better, or they are going to get better, and so on. So, I think this is the third, like, three most important things. One make sure you have that alignment, two, create tools that can be shared across different teams, and three, learn from past incidents and understand this is like a marathon. It's not a sprint.Jason: Those are excellent tips. So, for our listeners, if you would like a tool that can be shared between devs and DevOps or ops teams, and you're interested in Komodor—Itiel, tell us where folks can find more info about Komodor and learn more about how to troubleshoot Kubernetes.Itiel: So, you can find us on Twitter, but basically on komodor.com. Yeah, you can sign up for a free trial. The installation is, like, 10 seconds or something like that. It's basically Helm install, and it really works. We just finished, like, a very big round, so we are growing really fast and we have more and more customers. So, we'll be happy to hear your use case and to see how we can accommodate your needs.Jason: Awesome. Well, thanks for being on the show. It's been a pleasure to have you.Itiel: Thank you. Thank you. It was super fun being here.Jason: For links to all the information mentioned, visit our website at gremlin.com/podcast. If you liked this episode, subscribe to the Break Things on Purpose podcast on Spotify, Apple Podcasts, or your favorite podcast platform. Our theme song is called “Battle of Pogs” by Komiku, and it's available on loyaltyfreakmusic.com.

In this episode, we cover: 00:00:00 - Introduction 00:02:45 - Adopting the Cloud 00:08:15 - POC Process  00:12:40 - Infrastructure Team Building 00:17:45 - “Disaster Roleplay”/Communicating to the Non-Technical Side  00:20:20 - Leadership 00:22:45 - Tomas' Horror Story/Dashboard Organziation 00:29:20 - Outro Links: Productboard: https://www.productboard.com Scaling Teams: https://www.amazon.com/Scaling-Teams-Strategies-Successful-Organizations/dp/149195227X Seeking SRE: https://www.amazon.com/Seeking-SRE-Conversations-Running-Production/dp/1491978864/ TranscriptJason: Welcome to Break Things on Purpose, a podcast about failure and reliability. In this episode, we chat with Tomas Fedor, Head of Infrastructure at Productboard. He shares his approach to testing and implementing new technologies, and his experiences in leading and growing technical teams.Today, we've got with us Tomas Fedor, who's joining us all the way from the Czech Republic. Tomas, why don't you say hello and introduce yourself?Tomas: Hello, everyone. Nice to meet you all, and my name is Tomas, or call me Tom. And I've been working for a Productboard for past two-and-a-half year as infrastructure leader. And all the time, my experience was in the areas of DevOps, and recently, three and four years is about management within infrastructure teams. What I'm passionate about, my main technologies-wise in cloud, mostly Amazon Web Services, Kubernetes, Infrastructure as Code such as Terraform, and recently, I also jumped towards security compliances, such as SOC 2 Type 2.Jason: Interesting. So, a lot of passions there, things that we actually love chatting about on the podcast. We've had other guests from HashiCorp, so we've talked plenty about Terraform. And we've talked about Kubernetes with some folks who are involved with the CNCF. I'm curious, with your experience, how did you first dive into these cloud-native technologies and adopting the cloud? Is that something you went straight for, or is that something you transitioned into?Tomas: I actually slow transition to cloud technologies because my first career started at university when I was like, say, half developer and half Unix administrator. And I had experience with building very small data center. So, those times were amazing to understand all the hardware aspects of how it's going to be built. And then later on, I got opportunity to join a very famous startup at Czech Republic [unintelligible 00:02:34] called Kiwi.com [unintelligible 00:02:35]. And that time, I first experienced cloud technologies such as Amazon Web Services.Jason: So, as you adopted Amazon, coming from that background of a university and having physical servers that you had to deal with, what was your biggest surprise in adopting the cloud? Maybe something that you didn't expect?Tomas: So, that's great question, and what comes to my mind first, is switching to completely different [unintelligible 00:03:05] because during my university studies and career there, I mostly focused on networking [unintelligible 00:03:13], but later on, you start actually thinking about not how to build a service, but what service you need to use for your use case. And you don't have, like, one service or one use case, but you have plenty of services that can suit your needs and you need to choose wisely. So, that was very interesting, and it needed—and it take me some time to actually adopt towards new thinking, new mindset, et cetera.Jason: That's an excellent point. And I feel like it's only gotten worse with the, “How do you choose?” If I were to ask you to set up a web service and it needs some sort of data store, at this point you've got, what, a half dozen or more options on Amazon? [laugh].Tomas: Exactly.Jason: So, with so many services on providers like Amazon, how do you go about choosing?Tomas: After a while, we came up with a thing like RFCs. That's like ‘Request For Comments,' where we tried to sum up all the goals, and all the principles, and all the problems and challenges we try to tackle. And with that, we also tried to validate all the alternatives. And once you went through all these information, you tried to sum up all the possible solutions. You typically had either one or two options, and those options were validated with all your team members or the whole engineering organization, and you made the decision then you try to run POC, and you either are confirmed, yeah this is the technology, or this is service you need and we are going to implement it, or you revised your proposal.Jason: I really like that process of starting with the RFC and defining your requirements and really getting those set so that as you're evaluating, you have these really stable ideas of what you need and so you don't get swayed by all of the hype around a certain technology. I'm curious, who is usually involved in the RFC process? Is it a select group in the engineering org? Is it broader? How do you get the perspectives that you need?Tomas: I feel we have very great established process at Productboard about RFCs. It's transparent to the whole organization, that's what I love the most. The first week, there is one or two reporters that are mainly focused on writing and summing up the whole proposal to write down goals, and also non-goals because that is going to define your focus and also define focus of reader. And then you're going just to describe alternatives, possible options, or maybe to sum up, “Hey, okay, I'm still unsure about this specific decision, but I feel this is the right direction.” Maybe I have someone else in the organization who is already familiar with the technology or with my use case, and that person can help me.So, once—or we call it a draft state, and once you feel confident, you are going to change the status of RFC to open. The time is open to feedback to everyone, and they typically geared, like, two weeks or three weeks, so everyone can give a feedback. And you have also option to present it on engineering all-hands. So, many engineers, or everyone else joining the engineering all-hands is aware of this RFC so you can receive a lot of feedback. What else is important to mention there that you can iterate over RFCs.So, you mark it as resolved after through two or three weeks, but then you come up with a new proposal, or you would like to update it slightly with important change. So, you can reopen it and update version there. So, that also gives you a space to update your RFC, improve the proposal, or completely to change the context so it's still up-to-date with what you want to resolve.Jason: I like that idea of presenting at engineering all-hands because, at least in my experience, being at a startup, you're often super busy so you may know that the RFC is available, but you may not have time to actually read through it, spend the time to comment, so having that presentation where it's nicely summarized for you is always nice. Moving from that to the POC, when you've selected a few and you want to try them out, tell me more about that POC process. What does that look like?Tomas: So typically, in my infrastructure team, it's slightly different, I believe, as you have either product teams focus on POCs, or you have more platform teams focusing on those. So, in case of the infrastructure team, we would like to understand what code is actually going to be about because typically the infrastructure team has plenty of services to be responsible for, to be maintained, and we try to first choose, like, one specific use case and small use case that's going to suit the need.For instance, I can share about implementation of HashiCorp Vault, like our adoption. We leveraged firstly only key-value engine for storing secrets. And what was important to understand here, whether we want to spend hours of building the whole cluster, or we can leverage their cloud service and try to integrate it with one of our services. And we need to understand what service we are going to adopt with Vault.So, we picked cloud solution. It was very simple, the experience that were seamless for us, we understood what we needed to validate. So, is developer able to connect to Vault? Is application able to connect to Vault? What roles does it offer? Was the difference for cloud versus on-premise solution?And at the end, it's often the cost. So, in that case, POC, we spin up just cloud service integrated with our system, choose the easiest possible adaptable service, run POC, validate it with developers, and provide all the feedback, all the data, to the rest of the engineering. So, that was for us, some small POC with large service at the end.Jason: Along with validating that it does what you want it to do, do you ever include reliability testing in that POC?Tomas: It is, but it is in, like, let's say, it's in a later stage. For example, I can again mention HashiCorp Vault. Once we made a decision to try to spin up first on-premise cluster, we started just thinking, like, how many master nodes do we need to have? How many availability zones do we need to have? So, you are going to follow quorum?And we are thinking, “Okay, so what's actually the reliability of Amazon Web Services regions and their availability zones? What's the reliability of multi-cross-region? And what actually the expectations that is going to happen? And how often they happen? Or when in the past, it happened?”So, all those aspects were considered, and we ran out that decision. Okay, we are still happy with one region because AWS is pretty stable, and I believe it's going to be. And we are now successfully running with three availability zones, but before we jumped to the conclusion of having three availability zones, we run several tests. So, we make sure that in case one availability zone being down, we are still fully able to run HashiCorp Vault cluster without any issues.Jason: That's such an important test, especially with something like HashiCorp Vault because not being able to log into things because you don't have credentials or keys is definitely problematic.Tomas: Fully agree.Jason: You've adopted that during the POC process, or the extended POC process; do you continue that on with your regular infrastructure work continuing to test for reliability, or maybe any chaos engineering?Tomas: I actually measure something about what we are working on, like, what we have so far improved in terms of post-mortem process that's interesting. So, we started two-and-a-half year ago, and just two of us as infrastructure engineers. At the time, there was only one incident response on-call team, our first iteration within the infrastructure team was with migration from Heroku, where we ran all our services, to Amazon Web Services. And that time, we needed to also start thinking about, okay, the infrastructure team needs to be on call as well. So, that required to update in the process because until then, it works great; you have one team, people know each other, people know the whole stack. Suddenly, you are going to add new people, you're going to add new people a separate team, and that's going to change the way how on-call should be treated, and how the process should look like.You may ask why. You have understanding within the one team, you understand the expectations, but then you have suddenly different skill set of people, and they are going to be responsible for different part of the technical organization, so you need to align the expectation between two teams. And that was great because guys at Productboard are amazing, and they are always helpful. So, we sat down, we made first proposal of how new team is going to work like, what are going to be responsibilities. We took inspirations from the already existing on-call process, and we just updated it slightly.And we started to run with first test scenarios of being on call so we understand the process fully. Later on, it evolved to more complex process, but it's still very simple. What is more complex: we have more teams that's first thing being on call; we have better separation of all the alerts, so you're not going to route every alert to one team, but you are able to route it to every team that's responsible for its service; the team have also prepared a set of runbooks so anyone else can easily follow runbook and fix the incident pretty easily, and then we also added section about post-mortems, so what are our expectations of writing down post-mortem once incident is resolved.Jason: That's a great process of documenting, really—right—documenting the process so that everybody, whether they're on a different team and they're coming over or new hires, particularly, people that know nothing about your established practices can take that runbook and follow along, and achieve the same results that any other engineer would.Tomas: Yeah, I agree. And what was great to see that once my team grew—we are currently five and we started two—we saw excitement of the team members to update the process so everybody else we're going to join the on-call is going to be excited, is going to take it as an opportunity to learn more. So, we added disaster roleplay, and that section talks about you are new person joining on-call team, and we would like to make sure you are going to understand all the processes, all the necessary steps, and you are going to be aligned with all the expectations. But before you will actually going to have your first alerts of on-call, we would like to try to run roleplay. Imagine what a HashiCorp Vault cluster is going down; you should be the one resolving it. So, what are the first steps, et cetera?And that time you're going to realize whatever is being needs to be done, it's not only from a technical perspective, such as check our go to monitoring, check runbook, et cetera, but also communication-wise because you need to communicate not only with your shadowing buddy, but you also need to communicate internally, or to the customers. And that's going to change the perspective of how an incident should be handled.Jason: That disaster roleplay sounds really amazing. Can you chat a little bit more about the details of how that works? Particularly you mentioned engaging the non-technical side—right—of communication with various people. Does the disaster roleplay require coordinating with all those people, or is it just a mock, you would pretend to do, but you don't actually reach out to those people during this roleplay?Tomas: So, we would like to also combine the both aspects. We would like to make sure that person understands all the communication channels that are set within our organization, and what they are used for, and then we would like to make sure that that person understand how to involve other engineers within the organization. For instance, what was there the biggest difference is that you have plenty of options how to configure assigning or creating an alert. And so for those, you may have a different notification settings. And what happened is that some of the people have settings only for newly created alert, but when you made a change of assigned person of already existing alert, someone else, it might happen that that person didn't notice it because the notification setting was wrong. So, we encountered even these kind of issues and we were able to fix it, thanks to disaster roleplay. So, that was amazing to be found out.Jason: That's one of the favorite things that I like to do when we're using chaos engineering to do a similar thing to the disaster roleplay, is to really check those incident response processes, and validating those alerts is huge. There's so many times that I've found that we thought that someone would be alerted for some random thing, and turns out that nobody knew anything was going on. I love that you included that into your disaster roleplay process.Tomas: Yeah, it was also great experience for all the engineers involved. Unfortunately, we run it only within our team, but I hope we are going to have a chance to involve all other engineering on-call teams, so the onboarding experience to the engineering on-call teams is going to rise and is going to be amazing.Jason: So, one of the things that I'm really interested in is, you've gone from being a DevOps engineer, an SRE individual contributor role, and now you're leaving a small team. I think a lot of folks, as they look at their career, and I think more people are starting to become interested in this is, what does that progression look like? This is sort of a change of subject, but I'm interested in hearing your thoughts on what are the skills that you picked up and have used to become an effective technical leader within Productboard? What's some of that advice that our listeners, as individual contributors, can start to gain in order to advance where they're going with their own careers?Tomas: Firstly, it's important to understand what makes you passionate in your career, whether it's working with people, understanding their needs and their future, or you would like to be more on track as individual contributor and you would like to enlarge your scope of responsibilities towards leading more technical complex initiatives, that are going to take a long time to be implemented. In case all the infrastructure, or in case of the platform leaders, I would say the position of manager or technical leader also requires certain technical knowledge so you can be still in close touch with your team or with your most senior engineers, so you can set the goals and set the strategic clearly. But still, it's important to be, let's say, people person and be able to listen because in that case, people are going to be more open to you, and you can start helping them, and you can start making their dreams true and achievable.Jason: Making their dreams true. That's a great take on this idea because I feel like so many times, having done infrastructure work, that you start to get a mindset of maybe that people just are making demands of you, all the time. And it's sometimes hard to keep that perspective of working together as a team and really trying to excel to give them a platform that they can leverage to really get things done. We were talking about disaster roleplaying, and that naturally leads to a question that we like to ask of all of our guests and that's, do you have any horror stories from your career about an incident, some horror story or outage that you experienced and what you've learned from it?Tomas: I have one, and it actually happened at the beginning of my career of DevOps engineer. What is interesting here that it was one of the toughest incidents I experienced. It happened after midnight. So, the time I was still new to a company, and we have received an alert informing about too many 502, 504 errors written from API. At the time API process thousands of requests per second, and the incident had a huge impact on the services we were offering.And as I was shadowing my on-call buddy, I tried to check our main alerting channel, see what's happening, what's going on there, how can I help, and I started with checking monitoring system, reviewing all the reports from the engineers of being on-call, and I initiated the investigation on my own. I realized that something is wrong or something is not right, and I realized I was just confused and I want sleep, so it took me a while to get back on track. So, I made the side note, like, how can I start my brain to be working as during the day? And then I got back to the incident resolution process.So, it was really hard for me to start because I didn't know what [unintelligible 00:24:27] you knew about the channel, you knew about your engineers working on the resolution, but there were plenty of different communication funnels. Like, some of the engineers were deep-focused on their own investigation, and some of them were on call. And we needed to provide regular updates to the customers and internally as well. I had that inner feeling of let's share something, but I realized I just can't drop a random message because the message with all the information should have certain format and should have certain information. But I didn't know what kind of information should be there.So, I tried to ping someone, so, “Hey, can you share something?” And in the meantime, actually, more other people send me direct message. And I saw there are a lot of different tracks of people who tried to solve the incident, who tries to provide the status, but we were not aligned. So, this all showed me how important is to have proper communication funnel set. And we got the lucky to actually end up in one channel, we got lucky to resolve incident pretty quickly.And what else I learned that I would recommend to make sure you know where to work. I know it's pretty obvious sentence, but once your company has plenty of dashboards and you need to find one specific metric, sometime it looks like mission impossible.Jason: That's definitely a good lesson learned and feeds back to that disaster roleplays, practicing how you do those communications, understanding where things need to be communicated. You mentioned that it can be difficult to find a metric within a particular dashboard when you have so many. Do you have any advice for people on how to structure their dashboards, or name their dashboards, or organize them in a certain way to make that easier to find the metric or the information that you're looking for?Tomas: I will have a different approach, and that do have basic dashboard that provides you SLOs of all the services you have in the company. So, we understand firstly what service actually impacts the overall stability or reliability. So, that's my first advice. And then you should be able to either click on the specific service, and that should redirect you to it's dashboard, or you're going to have starred one of your favorite dashboards you have. So, I believe the most important is really have one main dashboard where you have all the services and their stability resourced, then you have option to look.Jason: Yeah, when you have one main dashboard, you're using that as basically the starting point, and from there, you can branch out and dive deeper, I guess, into each of the services.Tomas: Exactly, exactly true.Jason: I like that approach. And I think that a lot of modern dashboarding or monitoring systems now, the nice thing is that they have that ability, right, to go from one particular dashboard or graphic and have links out to the other information, or just click on the graph and it will show you the underlying host dashboard or node dashboard for that metric, which is really, really handy.Tomas: And I love the connection with other monitoring services, such as application monitoring. That gives you so much insight and when it's even connected with your work management tool is amazing so you can have all the important information in one place.Jason: Absolutely. So, oftentimes we talk about—what is it—the three pillars of observability, which I know some of our listeners may hate that, but the idea of having metrics and performance monitoring/APM and logs, and just how they all connect to each other can really help you solve a lot, or uncover a lot of information when you're in the middle of an incident. So Tomas, thanks for being on the show. I wanted to wrap up with one more question, and that's do you have any shoutouts, any plugs, anything that you want to share that our listeners should go take a look at?Tomas: Yeah, sure. So, as we are talking about management, I would like to promote one book that helped make my career, and that's Scaling Teams. It's written by Alexander Grosse and David Loftesness.And another one book is from Google, they have, like, three series, one of those is Seeking SRE, and I believe other parts are also useful to be read in case you would like to understand whether your organization needs SRE team and how to implement it within organization, and also, technically.Jason: Those are two great resources, and we'll have those linked in the show notes on the website. So, for anybody listening, you can find more information about those two books there. Tomas, thanks for joining us today. It's been a pleasure to have you.Tomas: Thanks. Bye.Jason: For links to all the information mentioned, visit our website at gremlin.com/podcast. If you liked this episode, subscribe to the Break Things on Purpose podcast on Spotify, Apple Podcasts, or your favorite podcast platform. Our theme song is called “Battle of Pogs” by Komiku, and it's available on loyaltyfreakmusic.com.

In this episode, we cover: 00:00:00 - Introduction 00:03:20 - VMWare Tanzu 00:07:50 - Gustavo's Career in Security  00:12:00 - Early Days in Chaos Engineering 00:16:30 - Catzilla  00:19:45 - Expanding on SRE 00:26:40 - Learning from Customer Trends 00:29:30 - Chaos Engineering at VMWare 00:36:00 - Outro Links: Tanzu VMware: https://tanzu.vmware.com GitHub for SREDocs: https://github.com/google/sredocs E-book on how to start your incident lifecycle program: https://tanzu.vmware.com/content/ebooks/establishing-an-sre-based-incident-lifecycle-program Twitter: https://twitter.com/stratus TranscriptJason: Welcome to Break Things on Purpose, a podcast about chaos engineering and building reliable systems. In this episode, Gustavo Franco, a senior engineering manager at VMware joins us to talk about building reliability as a product feature, and the journey of chaos engineering from its place in the early days of Google's disaster recovery practices to the modern SRE movement. Thanks, everyone, for joining us for another episode. Today with us we have Gustavo Franco, who's a senior engineering manager at VMware. Gustavo, why don't you say hi, and tell us about yourself.Gustavo: Thank you very much for having me. Gustavo Franco; as you were just mentioning, I'm a senior engineering manager now at VMware. So, recently co-founded the VMware Tanzu Reliability Engineering Organization with Megan Bigelow. It's been only a year, actually. And we've been doing quite a bit more than SRE; we can talk about like—we're kind of branching out beyond SRE, as well.Jason: Yeah, that sounds interesting. For folks who don't know, I feel like I've seen VMware Tanzu around everywhere. It just suddenly went from nothing into this huge thing of, like, every single Kubernetes-related event, I feel like there's someone from VMware Tanzu on it. So, maybe as some background, give us some information; what is VMware Tanzu?Gustavo: Kubernetes is sort of the engine, and we have a Kubernetes distribution called Tanzu Kubernetes Grid. So, one of my teams actually works on Tanzu Kubernetes Grid. So, what is VMware Tanzu? What this really is, is what we call a modern application platform, really an end-to-end solution. So, customers expect to buy not just Kubernetes, but everything around, everything that comes with giving the developers a platform to write code, to write applications, to write workloads.So, it's basically the developer at a retail company or a finance company, they don't want to run Kubernetes clusters; they would like the ability to, maybe, but they don't necessarily think in terms of Kubernetes clusters. They want to think about workloads, applications. So, VMWare Tanzu is end-to-end solution that the engine in there is Kubernetes.Jason: That definitely describes at least my perspective on Kubernetes is, I love running Kubernetes clusters, but at the end of the day, I don't want to have to evaluate every single CNCF project and all of the other tools that are required in order to actually maintain and operate a Kubernetes cluster.Gustavo: I was just going to say, and we acquired Pivotal a couple of years ago, so that brought a ton of open-source projects, such as the Spring Framework. So, for Java developers, I think it's really cool, too, just being able to worry about development and the Java layer and a little bit of reliability, chaos engineering perspective. So, kind of really gives me full tooling, the ability common libraries. It's so important for reliable engineering and chaos engineering as well, to give people this common surface that we can actually use to inject faults, potentially, or even just define standards.Jason: Excellent point of having that common framework in order to do these reliability practices. So, you've explained what VMware Tanzu is. Tell me a bit more about how that fits in with VMware Tanzu?Gustavo: Yeah, so one thing that happened the past few years, the SRE organization grew beyond SRE. We're doing quite a bit of horizontal work, so SRE being one of them. So, just an example, I got to charter a compliance engineering team and one team that we call ‘Customer Zero.' I would call them partially the representatives of growth, and then quote-unquote, “Customer problems, customer pain”, and things that we have to resolve across multiple teams. So, SRE is one function that clearly you can think of.You cannot just think of SRE on a product basis, but you think of SRE across multiple products because we're building a platform with multiple pieces. So, it's kind of like putting the building blocks together for this platform. So then, of course, we're going to have to have a team of specialists, but we need an organization of generalists, so that's where SRE and this broader organization comes in.Jason: Interesting. So, it's not just we're running a platform, we need our own SREs, but it sounds like it's more of a group that starts to think more about the product itself and maybe even works with customers to help their reliability needs?Gustavo: Yeah, a hundred percent. We do have SRE teams that invest the majority of their time running SaaS, so running Software as a Service. So, one of them is the Tanzu Mission Control. It's purely SaaS, and what teams see Tanzu Mission Control does is allow the customers to run Kubernetes anywhere. So, if people have Kubernetes on-prem or they have Kubernetes on multiple public clouds, they can use TMC to be that common management surface, both API and web UI, across Kubernetes, really anywhere they have Kubernetes. So, that's SaaS.But for TKG SRE, that's a different problem. We don't have currently a TKG SaaS offering, so customers are running TKG on-prem or on public cloud themselves. So, what does the TKG SRE team do? So, that's one team that actually [unintelligible 00:05:15] to me, and they are working directly improving the reliability of the product. So, we build reliability as a feature of the product.So, we build a reliability scanner, which is a [unintelligible 00:05:28] plugin. It's open-source. I can give you more examples, but that's the gist of it, of the idea that you would hire security engineers to improve the security of a product that you sell to customers to run themselves. Why wouldn't you hire SREs to do the same to improve the reliability of the product that customers are running themselves? So, kind of, SRE beyond SaaS, basically.Jason: I love that idea because I feel like a lot of times in organizations that I talk with, SRE really has just been a renamed ops team. And so it's purely internal; it's purely thinking about we get software shipped to us from developers and it's our responsibility to just make that run reliably. And this sounds like it is that complete embrace of the DevOps model of breaking down silos and starting to move reliability, thinking of it from a developer perspective, a product perspective.Gustavo: Yeah. A lot of my work is spent on making analogies with security, basically. One example, several of the SREs in my org, yeah, they do spend time doing PRs with product developers, but also they do spend a fair amount of time doing what we call in a separate project right now—we're just about to launch something new—a reliability risk assessment. And then you can see the parallels there. Where like security engineers would probably be doing a security risk assessment or to look into, like, what could go wrong from a security standpoint?So, I do have a couple engineers working on reliability risk assessment, which is, what could go wrong from a reliability standpoint? What are the… known pitfalls of the architecture, the system design that we have? How does the architectural work looks like of the service? And yeah, what are the outages that we know already that we could have? So, if you have a dependency on, say, file on a CDN, yeah, what if the CDN fails?It's obvious and I know most of the audience will be like, “Oh, this is obvious,” but, like, are you writing this down on a spreadsheet and trying to stack-rank those risks? And after you stack-rank them, are you then mitigating, going top-down, look for—there was an SREcon talk by [Matt Brown 00:07:32], a former colleague of mine at Google, it's basically, know your enemy tech talk in SREcon. He talks about this like how SRE needs to have a more conscious approach to reliability risk assessment. So, really embraced that, and we embraced that at VMware. The SRE work that I do comes from a little bit of my beginnings or my initial background of working security.Jason: I didn't actually realize that you worked security, but I was looking at your LinkedIn profile and you've got a long career doing some really amazing work. So, you said you were in security. I'm curious, tell us more about how your career has progressed. How did you get to where you are today?Gustavo: Very first job, I was 16. There was this group of sysadmins on the first internet service provider in Brazil. One of them knew me from BBS, Bulletin Board Systems, and they, you know, were getting hacked, left and right. So, this guy referred me, and he referred me saying, “Look, it's this kid. He's 16, but he knows his way around this security stuff.”So, I show up, they interview me. I remember one of the interview questions; it's pretty funny. They asked me, “Oh, what would you do if we asked you to go and actually physically grab the routing table from AT&T?” It's just, like, a silly question and they told them, “Uh, that's impossible.” So, I kind of told him the gist of what I knew about routing, and it was impossible to physically get a routing table.For some reason, they loved that. That was the only candidate that could be telling them, “No. I'm not going to do it because it makes no sense.” So, they hired me. And the student security was basically teaching the older sysadmins about SSH because they were all on telnet, nothing was encrypted.There was no IDS—this was a long time ago, right, so the explosion of cybersecurity security firms did not exist then, so it was new. To be, like, a security company was a new thing. So, that was the beginning. I did dabble in open-source development for a while. I had a couple other jobs on ISPs.Google found me because of my dev and open-source work in '06, '07. I interviewed, joined Google, and then at Google, all of it is IC, basically, individual contributor. And at Google, I start doing SRE-type of work, but for the corporate systems. And there was this failed attempt to migrate from one Linux distribution to another—all the corporate systems—and I tech-led the effort making that successful. I don't think I should take the credit; it was really just a fact of, like you know, trying the second time and kind of, learned—the organization learned the lessons that I had to learn from the first time. So, we did a second time and it worked.And then yeah, I kept going. I did more SRE work in corp, I did some stuff in production, like all the products. So, I did a ton of stuff. I did—let's see—technical infrastructure, disaster recovery testing, I started a chaos-engineering-focused team. I worked on Google Cloud before we had a name for it. [laugh].So, I was the first SRE on Google Compute Engine and Google Cloud Storage. I managed Google Plus SRE team, and G Suite for a while. And finally, after doing all this runs on different teams, and developing new SRE teams and organizations, and different styles, different programs in SRE. Dave Rensin, which created the CRE team at Google, recruited me with Matt Brown, which was then the tech lead, to join the CRE team, which was the team at Google focused on teaching Google Cloud customers on how to adopt SRE practices. So, because I had this very broad experience within Google, they thought, yeah, it will be cool if you can share that experience with customers.And then I acquired even more experience working with random customers trying to adopt SRE practices. So, I think I've seen a little bit of it all. VMware wanted me to start, basically, a CRE team following the same model that we had at Google, which culminated all this in TKG SRE that I'm saying, like, we work to improve the reliability of the product and not just teaching the customer how to adopt SRE practices. And my pitch to the team was, you know, we can and should teach the customers, but we should also make sure that they have reasonable defaults, that they are providing a reasonable config. That's the gist of my experience, at a high level.Jason: That's an amazing breadth of experience. And there's so many aspects that I feel like I want to dive into [laugh] that I'm not quite sure exactly where to start. But I think I'll start with the first one, and that's that you mentioned that you were on that initial team at Google that started doing chaos engineering. And so I'm wondering if you could share maybe one of your experiences from that. What sort of chaos engineering did you do? What did you learn? What were the experiments like?Gustavo: So, a little bit of the backstory. This is probably because Kripa mentioned this several times before—and Kripa Krishnan, she actually initiated disaster recovery testing, way, way before there was such a thing as chaos engineering—that was 2006, 2007. That was around the time I was joining Google. So, Kripa was the first one to lead disaster recovery testing. It was very manual; it was basically a room full of project managers with postIts, and asking teams to, like, “Hey, can you test your stuff? Can you test your processes? What if something goes wrong? What if there's an earthquake in the Bay Area type of scenario?” So, that was the predecessor.Many, many years later, I work with her from my SRE teams testing, for my SRE teams participating in disaster recovery testing, but I was never a part of the team responsible for it. And then seven years later, I was. So, she recruited me with the following pitch, she was like, “Well, the program is big. We have disaster recovery tests, we have a lot of people testing, but we are struggling to convince people to test year-round. So, people tend to test once a year, and they don't test again. Which is bad. And also,” she was like, “I wish we had a software; there's something missing.”We had the spreadsheets, we track people, we track their tasks. So, it was still very manual. The team didn't have a tool for people to test. It was more like, “Tell me what you're going to test, and I will help you with scheduling, I'll help you to not conflict with the business and really disrupt something major, disrupt production, disrupt the customers, potentially.” A command center, like a center of operations.That's what they did. I was like, “I know exactly what we need.” But then I surveyed what was out there in open-source way, and of course, like, Netflix, gets a lot of—deserves a lot of credit for it; there was nothing that could be applied to the way we're running infrastructure internally. And I also felt that if we built this centrally and we build a catalog of tasks ourselves, and that's it, people are not going to use it. We have a bunch of developers, software engineers.They've got to feel like—they want to, they want to feel—and rightfully so—that they wanted control and they are in control, and they want to customize the system. So, in two weeks, I hack a prototype where it was almost like a workflow engine for chaos engineering tests, and I wrote two or three tests, but there was an API for people to bring their own test to the system, so they could register a new test and basically send me a patch to add their own tests. And, yeah, to my surprise, like, a year later—and the absolute number of comparison is not really fair, but we had an order of magnitude more testing being done through the software than manual tests. So, on a per-unit basis, the quality of the ultimate tasks was lower, but the cool thing was that people were testing a lot more often. And it was also very surprising to see the teams that were testing.Because there were teams that refused to do the manual disaster recovery testing exercise, they were using the software now to test, and that was part of the regular integration test infrastructure. So, they're not quite starting with okay, we're going to test in production, but they were testing staging, they were testing a developer environment. And in staging, they had real data; they were finding regressions. I can mention the most popular testing, too, because I spoke about this publicly before, which was this fuzz testing. So, a lot of things are RPC or RPC services, RPC, servers.Fuzz testing is really useful in the sense that, you know, if you send a random data in RPC call, will the server crash? Will the server handling this gracefully? So, we fought a lot of people—not us—a lot of people use or shared service bringing their own test, and fuzz testing was very popular to run continuously. And they would find a ton of crashes. We had a lot of success with that program.This team that I ran that was dedicated to building this shared service as a chaos engineering tool—which ironically named Catzilla—and I'm not a cat person, so there's a story there, too—was also doing more than just Catzilla, which we can also talk about because there's a little bit more of the incident management space that's out there.Jason: Yeah. Happy to dive into that. Tell me more about Catzilla?Gustavo: Yeah. So, Catzilla was sort of the first project from scratch from the team that ended up being responsible to share a coherent vision around the incident prevention. And then we would put Catzilla there, right, so the chaos engineering shared service and prevention, detection, analysis and response. Because once I started working on this, I realized, well, you know what? People are still being paged, they have good training, we had a good incident management process, so we have good training for people to coordinate incidents, but if you don't have SREs working directly with you—and most teams didn't—you also have a struggle to communicate with executives.It was a struggle to figure out what to do with prevention, and then Catzilla sort of resolved that a little bit. So, if you think of a team, like an SRE team in charge of not running a SaaS necessarily, but a team that works in function of a company to help the company to think holistically about incident prevention, detection, analysis, and response. So, we end up building more software for those. So, part of the software was well, instead of having people writing postmortems—a pet peeve of mine is people write postmortems and them they would give to the new employees to read them. So, people never really learned the postmortems, and there was like not a lot of information recovery from those retrospectives.Some teams were very good at following up on extra items and having discussions. But that's kind of how you see the community now, people talking about how we should approach retrospectives. It happened but it wasn't consistent. So then, well, one thing that we could do consistently is extract all the information that people spend so much time writing on the retrospectives. So, my pitch was, instead of having these unstructured texts, can we have it both unstructured and structured?So, then we launch postmortem template that was also machine-readable so we could extract information and then generate reports for to business leaders to say, “Okay, here's what we see on a recurring basis, what people are talking about in the retrospectives, what they're telling each other as they go about writing the retrospectives.” So, we found some interesting issues that were resolved that were not obvious on a per retrospective basis. So, that was all the way down to the analysis of the incidents. On the management part, we built tooling. It's basically—you can think of it as a SaaS, but just for the internal employees to use that is similar to externally what would be an incident dashboard, you know, like a status page of sorts.Of course, a lot more information internally for people participating in incidents than they have externally. For me is thinking of the SRE—and I manage many SRE teams that were responsible for running production services, such as Compute Engine, Google Plus, Hangouts, but also, you know, I just think of SRE as the folks managing production system going on call. But thinking of them a reliability specialists. And there's so many—when you think of SREs as reliability specialists that can do more than respond to pages, then you can slot SREs and SRE teams in many other areas of a organization.Jason: That's an excellent point. Just that idea of an SRE as being more than just the operation's on-call unit. I want to jump back to what you mentioned about taking and analyzing those retrospectives and analyzing your incidents. That's something that we did when I was at Datadog. Alexis Lê-Quôc, who's the CTO, has a fantastic talk about that at Monitorama that I'll link to in the [show notes 00:19:49].It was very clear from taking the time to look at all of your incidents, to catalog them, to really try to derive what's the data out of those and get that information to help you improve. We never did it in an automated way, but it sounds like with an automated tool, you were able to gather so much more information.Gustavo: Yeah, exactly. And to be clear, we did this manually before, and so we understood the cost of. And our bar, company-wide, for people writing retrospectives was pretty low, so I can't give you a hard numbers, but we had a surprising amount of retrospectives, let's say on a monthly basis because a lot of things are not necessarily things that many customers would experience. So, near misses or things that impact very few customers—potentially very few customers within a country could end up in a retrospective, so we had this throughput. So, it wasn't just, like, say, the highest severity outages.Like where oh, it happens—the stuff that you see on the press that happens once, maybe, a year, twice a year. So, we had quite a bit of data to discuss. So, then when we did it manually, we're like, “Okay, yeah, there's definitely something here because there's a ton of information; we're learning so much about what happens,” but then at the same time, we were like, “Oh, it's painful to copy and paste the useful stuff from a document to a spreadsheet and then crunch the spreadsheet.” And kudos—I really need to mention her name, too, Sue [Lueder 00:21:17] and also [Yelena Ortel 00:21:19]. Both of them were amazing project program managers who've done the brunt of this work back in the days when we were doing it manually.We had a rotation with SREs participating, too, but our project managers were awesome. And also Jason: As you started to analyze some of those incidents, every infrastructure is different, every setup is different, so I'm sure that maybe the trends that you saw are perhaps unique to those Google teams. I'm curious if you could share the, say, top three themes that might be interesting and applicable to our listeners, and things that they should look into or invest in?Gustavo: Yeah, one thing that I tell people about adopting the—in the books, the SRE books, is the—and people joke about it, so I'll explain the numbers a little better. 70, 75% of the incidents are triggered by config changes. And people are like, “Oh, of course. If you don't change anything, there are no incidents, blah, blah, blah.” Well, that's not true, that number really speaks to a change in the service that is impacted by the incident.So, that is not a change in the underlying dependency. Because people were very quickly to blame their dependencies, right? So meaning, if you think of a microservice mesh, the service app is going to say, “Oh, sure. I was throwing errors, my service was throwing errors, but it was something with G or H underneath, in a layer below.” 75% of cases—and this is public information goes into books, right—of retrospectives was written, the service that was throwing the errors, it was something that changed in that service, not above or below; 75% of the time, a config change.And it was interesting when we would go and look into some teams where there was a huge deviation from that. So, for some teams, it was like, I don't know, 85% binary deploys. So, they're not really changing config that much, or the configuration issues are not trigger—or the configuration changes or not triggering incidents. For those teams, actually, a common phenomenon was that because they couldn't. So, they did—the binary deploys were spiking as contributing factors and main triggers for incidents because they couldn't do config changes that well, roll them out in production, so they're like, yeah, of course, like, [laugh] my minor deploys will break more on my own service.But that showed to a lot of people that a lot of things were quote-unquote, “Under their control.” And it also was used to justify a project and a technique that I think it's undervalued by SREs in the wild, or folks running production in the wild which is canary evaluation systems. So, all these numbers and a lot of this analysis was just fine for, like, to give extra funding for the scene that was basically systematically across the entire company, if you tried to deploy a binary to production, if you tried to deploy a config change to production, will evaluate a canary if the binary is in a crash loop, if the binary is throwing many errors, is something is changing in a clearly unpredictable way, it will pause, it will abort the deploy. Which back to—much easier said than done. It sounds obvious, right, “Oh, we should do canaries,” but, “Oh, can you automate your canaries in such a way that they're looking to monitoring time series and that it'll stop a release and roll back a release so a human operator can jump in and be like, ‘oh, okay. Was it a false positive or not?'”Jason: I think that moving to canary deployments, I've long been a proponent of that, and I think we're starting to see a lot more of that with tools such as—things like LaunchDarkly and other tools that have made it a whole lot easier for your average organization that maybe doesn't have quite the infrastructure build-out. As you started to work on all of this within Google, you then went to the CRE team and started to help Google Cloud customers. Did any of these tools start to apply to them as well, analyzing their incidents and finding particular trends for those customers?Gustavo: More than one customer, when I describe, say our incident lifecycle management program, and the chaos engineering program, especially this lifecycle stuff, in the beginning, was, “Oh, okay. How do I do that?” And I open-sourced a very crufty prototype which some customers pick up on it and they implement internally in their companies. And it's still on GitHub, so /google/sredocs.There's an ugly parser, an example, like, of template for the machine-readable stuff, and how to basically get your retrospectives, dump the data onto Google BigQuery to be able to query more structurally. So yes, customers would ask us about, “Yeah. I heard about chaos engineering. How do you do chaos engineering? How can we start?”So, like, I remember a retail one where we had a long conversation about it, and some folks in tech want to know, “Yeah, instant response; how do I go about it?” Or, “What do I do with my retrospectives?” Like, people started to realize that, “Yeah, I write all this stuff and then we work on the action items, but then I have all these insights written down and no one goes back to read it. How can I get actionable insights, actionable information out of it?”Jason: Without naming any names because I know that's probably not allowed, are there any trends from customers that you'd be willing to share? Things that maybe—insights that you learned from how they were doing things and the incidents they were seeing that was different from what you saw at Google?Gustavo: Gaming is very unique because a lot of gaming companies, when we would go into incident management, [unintelligible 00:26:59] they were like, “If I launch a game, it's ride or die.” There may be a game that in the first 24, or 48 hours if the customers don't show up, they will never show up. So, that was a little surprising and unusual. Another trend is, in finance, you would expect a little behind or to be too strict on process, et cetera, which they still are very sophisticated customers, I would say. The new teams of folks are really interested in learning how to modernize the finance infrastructure.Let's see… well, tech, we basically talk the same language, with the gaming being a little different. In retail, the uniqueness of having a ton of things at the edge was a little bit of a challenge. So, having these hubs, where they have, say, a public cloud or on-prem data center, and these of having things running at the stores, so then having this conversation with them about different tiers and how to manage different incidents. Because if a flagship store is offline, it is a big deal. And from a, again, SaaS mindset, if you're think of, like, SRE, and you always manage through a public cloud, you're like, “Oh, I just call with my cloud provider; they'll figure it out.”But then for retail company with things at the edge, at a store, they cannot just sit around and wait for the public cloud to restore their service. So again, a lot of more nuanced conversations there that you have to have of like, yeah, okay, yeah. Here, say a VMware or a Google. Yeah, we don't deal with this problem internally, so yeah, how would I address this? The answers are very long, and they always depend.They need to consider, oh, do you have an operational team that you can drive around? [laugh]. Do you have people, do you have staffing that can go to the stores? How long it will take? So, the SLO conversation there is tricky.a secret weapon of SRE that has definitely other value is the project managers, program managers that work with SREs. And I need to shout out to—if you're a project manager, program manager working with SREs, shout out to you.Do you want to have people on call 24/7? Do you have people near that store that can go physically and do anything about it? And more often than not, they rely on third-party vendors, so then it's not staffed in-house and they're not super technical, so then remote management conversations come into play. And then you talk about, “Oh, what's your network infrastructure for that remote management?” Right? [laugh].Jason: Things get really interesting when you start to essentially outsource to other companies and have them provide the technology, and you try to get that interface. So, you mentioned doing chaos engineering within Google, and now you've moved to VMware with the Tanzu team. Tell me a bit more about how do you do chaos engineering at VMware, and what does that look like?Gustavo: I've seen varying degrees of adoption. So, right now, within my team, what we are doing is we're actually going as we speak right now, doing a big reliabilities assessment for a launch. Unfortunately, we cannot talk about it yet. We're probably going to announce this on October at VMworld. As a side effect of this big launch, we started by doing a reliability risk assessment.And the way we do this is we interview the developers—so this hasn't launched yet, so we're still designing this thing together. [unintelligible 00:30:05] the developers of the architecture that they basically sketch out, like, what is it that you're going to? What are the user journeys, the user stories? Who is responsible for what? And let's put an architecture diagram, a sketch together.And then we tried to poke or holes on, “Okay. What could go wrong here?” We write this stuff down. More often than not, from this list—and I can already see, like, that's where that output, that result fits into any sort of chaos engineering plan. So, that's where, like—so I can get—one thing that I can tell you for that risk assessment because I participated in the beginning was, there is a level of risk involving a CDN, so then one thing that we're likely going to test before we get to general availability is yeah, let's simulate that the CDN is cut off from the clients.But even before we do the test, we're already asking, but we don't trust. Like, trust and verify, actually; we do trust but trust and verify. So, we do trust the client is actually another team. So, we do trust the client team that they cache, but we are asking them, “Okay. Can you confirm that you cache? And if you do cache, can you give us access to flush the cache?”We trust them, we trust the answers; we're going to verify. And how do we verify? It's through a chaos engineering test which is, let's cut the client off from the CDN and then see what happens. Which could be, for us, as simple as let's move the file away; we should expect them to not tell us anything because the client will fail to read but it's going to pick from cache, it's not reading from us anyways. So, there is, like, that level of we tell people, “Hey, we're going to test a few things.”We'll not necessarily tell them what. So, we are also not just testing the system, but testing how people react, and if anything happens. If nothing happens, it's fine. They're not going to react to it. So, that's the level of chaos engineering that our team has been performing.Of course, as we always talk about improving reliability for the product, we talked about, “Oh, how is it that chaos engineering as a tool for our customers will play out in the platform?” That conversation now is a little bit with product. So, product has to decide how and when they want to integrate, and then, of course, we're going to be part of that conversation once they're like, “Okay, we're ready to talk about it.” Other teams of VMWare, not necessarily Tanzu, then they do all sorts of chaos engineering testing. So, some of them using tools, open-source or not, and a lot of them do tabletop, basically, theoretical testing as well.Jason: That's an excellent point about getting started. You don't have a product out yet, and I'm sure everybody's anticipating hearing what it is and seeing the release at VMworld, but testing before you have a product; I feel like so many organizations, it's an afterthought, it's the, “I've built the product. It's in production. Now, we need to keep it reliable.” And I think by shifting that forward to thinking about, we've just started diagramming the architecture, let's think about where this can break. And how we can build those tests so that we can begin to do that chaos engineering testing, begin to do that reliability testing during the development of the product so that it ships reliably, rather than shipping and then figuring out how to keep it reliable.Gustavo: Yeah. The way I talked to—and I actually had a conversation with one of our VPs about this—is that you have technical support that is—for the most part, not all the teams from support—but at least one of the tiers of support, you want it to be reactive by design. You can staff quite a few people to react to issues and they can be very good about learning the basics because the customers—if you're acquiring more customers, they are going to be—you're going to have a huge set of customers early in the journey with your product. And you can never make the documentation perfect and the product onboarding perfect; they're going to run into issues. So, that very shallow set of issues, you can have a level of arterial support that is reactive by design.You don't want that tier of support to really go deep into issues forever because they can get caught up into a problem for weeks or months. You kind of going to have—and that's when you add another tier and that's when we get to more of, like, support specialists, and then they split into silos. And eventually, you do get an IC SRE being tier three or tier four, where SRE is a good in-between support organizations and product developers, in the sense that product developers also tend to specialize in certain aspects of a product. SRE wants to be generalists for reliability of a product. And nothing better than to uncover reliability for product is understanding the customer pain, the customer issues.And actually, one thing, one of the projects I can tell you about that we're doing right now is we're improving the reliability of our installation. And we're going for, like, can we accelerate the speed of installs and reduce the issues by better automation, better error handling, and also good—that's where I say day zero. So, day zero is, can we make this install faster, better, and more reliable? And after the installs in day one, can we get better default? Because I say the ergonomics for SRE should be pretty good because we're TKG SREs, so there's [unintelligible 00:35:24] and SRE should feel at home after installing TKG.Otherwise, you can just go install vanilla Kubernetes. And if vanilla Kubernetes does feel at home because it's open-source, it's what most people use and what most people know, but it's missing—because it's just Kubernetes—missing a lot of things around the ecosystem that TKG can install by default, but then when you add a lot of other things, I need to make sure that it feels at home for SREs and operators at large.Jason: It's been fantastic chatting with you. I feel like we can go [laugh] on and on.Gustavo: [laugh].Jason: I've gone longer than I had intended. Before we go, Gustavo, I wanted to ask you if you had anything that you wanted to share, anything you wanted to plug, where can people find you on the internet?Gustavo: Yeah, so I wrote an ebook on how to start your incident lifecycle program. It's not completely out yet, but I'll post on my Twitter account, so twitter.com/stratus. So @stratus, S-T-R-A-T-U-S. We'll put the link on the [notes 00:36:21], too. And so yeah, you can follow me there. I will publish the book once it's out. Kind of explains all about the how to establish an incident lifecycle. And if you want to talk about SRE stuff, or VMware Tanzu or TKG, you can also message me on Twitter.Jason: Thanks for all the information.Gustavo: Thank you, again. Thank you so much for having me. This was really fun. I really appreciate it.Jason: For links to all the information mentioned, visit our website at gremlin.com/podcast. If you liked this episode, subscribe to the Break Things on Purpose podcast on Spotify, Apple Podcasts, or your favorite podcast platform. Our theme song is called “Battle of Pogs” by Komiku, and it's available on loyaltyfreakmusic.com.

In this episode, we cover: 00:00:00 - Introduction  00:03:30 - An Engineering Anecdote  00:08:10 - Lessons Learned from Putting Out Fires 00:11:00 - Building “Guardrails” 00:18:10 - Pushing the Chaos Envelope  00:23:35 - OpenGitOps Project 00:30:37 - Where to Find Leo/Costa Rica CNCF Links: Weaveworks: https://www.weave.works GitOps Working Group: https://github.com/gitops-working-group/gitops-working-group OpenGitOps Project: https://opengitops.dev Github.com/open-gitops: https://github.com/open-gitops Twitter: https://twitter.com/murillodigital LinkedIn: https://www.linkedin.com/in/leonardomurillo/ Costa Rica CNCF: https://community.cncf.io/costa-rica/ Cloudnative.tv: http://cloudnative.tv Gremlin-certified chaos engineering practitioner: https://www.gremlin.com/certification TranscriptJason: Welcome to the Break Things on Purpose podcast, a show about our often self-inflicted failures and what we learn from them. In this episode, Leonardo Murillo, a principal partner solutions architect at Weaveworks. He joins us to talk about GitOps, Automating reliability, and Pura Vida.Ana: I like letting our guests kind of say, like, “Who are you? What do you do? What got you into the world of DevOps, and cloud, and all this fun stuff that we all get to do?”Leo: Well, I guess I'll do a little intro of myself. I'm Leonardo Murillo; everybody calls me Leo, which is fine because I realize that not everybody chooses to call me Leo, depending on where they're from. Like, Ticos and Latinos, they're like, “Oh, Leo,” like they already know me; I'm Leo already. But people in Europe and in other places, they're, kind of like, more formal out there. Leonardo everybody calls me Leo.I'm based off Costa Rica, and my current professional role is principal solutions architect—principal partner solutions architect at Weaveworks. How I got started in DevOps. A lot of people have gotten started in DevOps, which is not realizing that they just got started in DevOps, you know what I'm saying? Like, they did DevOps before it was a buzzword and it was, kind of like, cool. That was back—so I worked probably, like, three roles back, so I was CTO for a Colorado-based company before Weaveworks, and before that, I worked with a San Francisco-based startup called High Fidelity.And High Fidelity did virtual reality. So, it was actually founded by Philip Rosedale, the founder of Linden Lab, the builders of Second Life. And the whole idea was, let's build—with the advent of the Oculus Rift and all this cool tech—build the new metaverse concept. We're using the cloud because, I mean, when we're talking about this distributed system, like a distributed system where you're trying to, with very low latency, transmit positional audio, and a bunch of different degrees of freedom of your avatars and whatnot; that's very massive scale, lots of traffic. So, the cloud was, kind of like, fit for purpose.And so we started using the cloud, and I started using Jenkins, as a—and figure it out, like, Jenkins is a cron sort of thing; [unintelligible 00:02:48] oh, you can actually do a scheduled thing here. So, started using it almost to run just scheduled jobs. And then I realized its power, and all of a sudden, I started hearing this whole DevOps word, and I'm like, “What this? That's kind of like what we're doing, right?” Like, we're doing DevOps. And that's how it all got started, back in San Francisco.Ana: That actually segues to one of the first questions that we love asking all of our guests. We know that working in DevOps and engineering, sometimes it's a lot of firefighting, sometimes we get to teach a lot of other engineers how to have better processes. But we know that those horror stories exist. So, what is one of those horrible incidents that you've encountered in your career? What happened?Leo: This is before the cloud and this is way before DevOps was even something. I used to be a DJ in my 20s. I used to mix drum and bass and jungle with vinyl. I never did the digital move. I used DJ, and I was director for a colocation facility here in Costa Rica, one of the first few colocation facilities that existed in the [unintelligible 00:04:00].I partied a lot, like every night, [laugh] [unintelligible 00:04:05] party night and DJ night. One night, they had 24/7 support because we were collocations [unintelligible 00:04:12], so I had people doing support all the time. I was mixing in some bar someplace one night, and I don't want to go into absolute detail of my state of consciousness, but it wasn't, kind of like… accurate in its execution. So, I got a call, and they're like, “We're having some problem here with our network.” This is, like, back in Cisco PIX times for firewalls and you know, like… back then.I wasn't fully there, so I [laugh], just drove back to the office in the middle of night and had this assistant, Miguel was his name, and he looks at me and he's like, “Are you okay? Are you really capable of solving this problem at [laugh] this very point in time?” And I'm like, “Yeah. Sure, sure. I can do this.”We had a rack full of networking hardware and there was, like, a big incident; we actually—one of the primary connections that we had was completely offline. And I went in and I started working on a device, and I spent about half an hour, like, “Well, this device is fine. There's nothing wrong with the device.” I had been working for half an hour on the wrong device. They're like, “Come on. You really got to focus.”And long story short, I eventually got to the right device and I was able to fix the problem, but that was like a bad incident, which wasn't bad in the context of technicality, right? It was a relatively quick fix that I figured it out. It was just at the wrong time. [laugh]. You know what I'm saying?It wasn't the best thing to occur that particular night. So, when you're talking about firefighting, there's a huge burden in terms of the on-call person, and I think that's something that we had experienced, and that I think we should give out a lot of shout-outs and provide a lot of support for those that are on call. Because this is the exact price they pay for that responsibility. So, just as a side note that comes to mind. Here's a lot of, like, shout-outs to all the people on-call that are listening to this right now, and I'm sorry you cannot go party. [laugh].So yeah, that's telling one story of one incident way back. You want to hear another one because there's a—this is back in High Fidelity times. I was—I don't remember exactly what it was building, but it had to do with emailing users, basically, I had to do something, I can't recall actually what it was. They was supposed to email all the users that were using the platform. For whatever reason—I really can't recall why—I did not mock data on my development environment.What I did was just use—I didn't mock the data, I actually used just to a copy of the production [unintelligible 00:07:02] the users. I basically just emailed everybody, like, multiple times. And that was very embarrassing. And another embarrassing scenario was, one day, I was working on a firewall that was local to my office, and I got the terminals mixed up, and I shut down not my local office firewall, but the one that was at the colocation facility. And that was another embarrassing moment. So yeah, those are three, kind of, self-caused fires that required fighting afterwards.Ana: The mock data one definitely resonates, especially when you're starting out in engineering career where you're just like, “Hey, I need to get this working. I'm trying to connect to pull this data from a production service,” or, “I'm trying to publish a new email, I want to see how it all goes out. Yeah, why not grab a copy of what actually usually is being used by my company and, like, press buttons here? Oh, wait, no, that actually is hitting a live endpoint? I did not know that.”Which brings me to that main question; what do you end up learning when you go through these fires? After you went through this incident that you emailed all of your customers, what is something that you learn that you got to take back.Leo: I learned how you have to pay attention. It's hard to learn without having gone through this experiences because you start picking up on cues that you didn't pick up in the past. You start seeing things that you didn't pay attention to before, particularly because you didn't know. And I'm pretty sure, even if somebody would have told me, “Don't do this,” or, “Don't do that. Be careful,” you still make those mistakes.There is certain things that you only achieve through experience. And I think that's one of the most important things that I realized. And I've actually see the analogy of that with my children. There's certain things that I, no matter how well I articulate, they will not learn until they go through those experiences of themselves. But I think that's one of the things that I'd argue, you ha—you will go through this, and it's—it's not okay, but it's okay.Everybody makes mistakes. You'll also identify whether—like, how supporting your team is and how supportive your—the organization you're working with is when you see the reaction to those errors. Hopefully, it wasn't something too bad, and ideally there's going to be guiderails that prevent that really, really bad scenario, but it's okay to make mistakes. You learn to focus through those mistakes and you really should be paying attention; you should never take anything for granted. There is no safety net. Period.So, you should never assume that there is, or that you're not going to make a mistake. So, be very careful. Another thing that I learned, how I can I work in my development environment. How different patterns that I apply in my development environment, how I now I'm very careful to never have, kind of like, production [x 00:10:11] readily available within my development environment. And also to build those guiderails.I think part of what you learn is all the things that could go wrong, might go wrong, so take time to build those guiderails. I think that's important. Like anything else that comes with seniority, when you have a task to accomplish, the task itself is merely a margin, only a percentage of what you really should consider to reach that objective. And a lot of the times, that means building protection around what you're asked, or thinking beyond that scope. And then leverage the team, you know? If you have people around you that know more, which is kind of great about community and collaboration. Like, being—don't—you're not alone.Ana: I love that you mentioned guardrails and guardrails being a way that you're able to prevent some of these things. Do you think something like chaos engineering could help you find those guardrails when you don't know that you don't have a guardrail?Leo: I think it definitely. The more complex your job, the more complex your architecture, the more complex of the solution you're building—and we've gotten in an increase in complexity over time. We went from monoliths to microservices to fully distributed architectures of services. We went from synchronous to asynchronous to event-driven to—like, there's this increase in complexity that is basically there for a reason because of an increase in scale as well. And the number of possible failure conditions that could arise from this hugely diverse and complex set of variables means that we've gotten to a point that likely always was the way, but now it's reached, again, and because of targets aligned with this complexity, new levels of scale, that there is currently more unknown unknowns than we've ever had.The conditions that you can run into because of different problem states of each individual component in your distributed architecture, brings up an orders-of-magnitude increase in the possible issues that you might run into, basically a point where you really have to understand that you have no idea what could fail, and the exercise of identifying what can fail. Or what are the margins of stability of your solution because that's, kind of like, the whole point, the boundaries? There's going to be a set of conditions, there's going to be a combination of conditions that will trigger your—kind of, will tip your solution beyond that edge. And finding those edges of stability can no longer be something that just happens by accident; it has to be premeditated, it has to be planned for. This is basically chaos engineering.Hypothesizing, given a set of conditions, what is the expected outcome? And through the execution of this hypothesis of increasing or varying scope and complexity, starting to identify that perimeter of stability of their solution. So, I guess to answer your question, yes. I mean, chaos engineering allows you to ide—if you think about that perimeter of stability as the guardrails around your solution within which have to remain for your solution to be stable, for instance, there goes—[unintelligible 00:13:48] chaos engineering. I was actually talking to somebody the other day, so I'm the organizer for the Costa Rica Cloud-Native Community, the chapter for [unintelligible 00:14:00], and I have this fellow from [unintelligible 00:14:04] who, he works doing chaos engineering.And he was talking to me about this concept that I had not thought about and considered, how chaos engineering can also be, kind of like, applied at a social level. What happens if a person xyz is not available? What happens if a person other has access to a system that they shouldn't have? All these types of scenarios can be used to discover where more guiderails should be applied.Jason: You know, you start to learn where the on-call person that's completely sober, maybe, is unavailable for some reason, and Leo comes and [crosstalk 00:14:45]—Leo: Right. [laugh]. Exactly. Exactly. That's what you have to incorporate in your experiment, kind of like, the DJ variable and the party parameter.Jason: It's a good thing to underscore as well, right? Back to your idea of we can tell our children all sorts of things and they're not going to learn the lesson until they experience it. And similarly with, as you explore your systems and how they can fail, we can imagine and architecture systems to maybe be resilient or robust enough to withstand certain failures, but we don't actually learn those lessons or actually know if they're going to work until we really do that, until we really stress them and try to explore those boundaries.Leo: Wouldn't it be fantastic if we could do that with our lives? You know, like, I want to bungee jump or I want to skydive, and there's a percentage of probability that I'm going to hit the ground and die, and I can just introduce a hypothesis in my life, jump, and then just revert to my previous state if it went wrong. It would be fantastic. I would try many, many things. [laugh].But you can't. And it's kind of like the same thing with my kids. I would love to be able to say, “You know what? Execute the following process, get the experience, and then revert to before it happened.” You cannot do that in real life, but that's, kind of like, the scenario that's brought up by chaos engineering, you don't have to wait for that production incident to learn; you can actually, “Emulate” quote-unquote, those occurrences.You can emulate it, you can experience without the damage, though, if you do it well because I think that's also part of, kind of like, there's a lot to learn about chaos engineering and there's a lot of progress in terms of how the practice of chaos engineering is evolving, and I think there's likely still a percentage of the population or of the industry that still doesn't quite see chaos engineering beyond just introducing chaos, period. They know chaos engineering from calling the Chaos Monkeys kill instances at random, and fix things and, you know, not in the more scientific context that it's evolved into. But yeah, I think the ability to have a controlled experience where you can actually live through failure states, and incidents, and issues, and stuff that you really don't want to happen in real life, but you can actually simulate those, accelerates learning in a way that only experience provides. Which is the beauty of it because you're actually living through it, and I don't think anything can teach us as effectively as living through [unintelligible 00:17:43], through suffering.Ana: I do also very much love that point where it's true, chaos engineering does expedite your learning. Not only are you just building and releasing and waiting for failure to happen, you're actually injecting that failure and you get to just be like, “Oh, wait, if this failure was to occur, I know that I'm resilient to it.” But I also love pushing that envelope forward, that it really allows folks to battle-test solutions together of, “I think this architecture diagram is going to be more resilient because I'm running it on three regions, and they're all in just certain zones. But if I was to deploy to a different provider, that only gives me one region, but they say they have a higher uptime, I would love to battle, test that together and really see, I'm throwing both scenarios at you: you're losing your access to the database. What's going to happen? Go, fight.” [laugh].Leo: You know, one thing that I've been mentioning to people, this is my hypothesis as to the future of chaos engineering as a component of solutions architecture. My hypothesis is that just as nowadays, if you look at any application, any service, for that application or service to be production-ready, you have a certain percentage of unit test coverage and you have a certain percentage of end-to-end coverage of testing and whatnot, and you cannot ignore and say I'm going to give you a production-ready application or production-ready system without solid testing coverage. My hypothesis is that [unintelligible 00:19:21]. And as a side note, we are now living in a world of infrastructure as code, and manifested infrastructure, and declarative infrastructure, and all sorts of cool new ways to deploy and deliver that infrastructure and workloads on top of it. My theory is that just as unit testing coverage is a requirement for any production-ready solution or application nowadays, a certain percentage of, “Chaos coverage,” quote-unquote.In other words, what percentage of the surface of your infrastructure had been exercised by chaos experiments, is going to also become a requirement for any production-ready architecture. That's is where my mind is at. I think you'll start seeing that happen in CI/CD pipelines, you're going to start seeing labels of 90% chaos coverage on Terraform repos. That's kind of the future. That I hope because I think it's going to help tremendously with reliability, and allow people to party without concern for being called back to the office in the middle of the night. It's just going to have a positive impact overall.Ana: I definitely love where that vision is going because that's definitely very much of what I've seen in the industry and the community. And with a lot of the open-source projects that we see out there, like, I got to sit in on a project called Keptn, which gets a chance to bring in a little bit more of those SRE-driven operations and try to close that loop, and auto-remediate, and all these other nice things of DevOps and cloud, but a big portion of what we're doing with Keptn is that you also get a chance to inject chaos and validate against service-level objectives, so you get to just really bring to the front, “Oh, we're looking at this metric for business-level and service-level objectives that allow for us to know that we're actually up and running and our customers are able to use us because they are the right indicators that matter to our business.” But you get to do that within CI/CD so that you throw chaos at it, you check that SLO, that gets rolled out to production, or to your next stage and then you throw more chaos at it, and it continues being completely repetitive.Leo: That's really awesome. And I think, for example, SLOs, I think that's very valuable as well. And prioritize what you want to improve based on the output of your experiments against that error budget, for example. There's limited time, there's limited engineering capacity, there's limited everything, so this is also something that you—the output, the results, the insights that you get from executing experiments throughout your delivery lifecycle as you promote, as you progress your solution through its multiple stages, also help you identify what should be prioritized because of the impact that it may have in your area budgets. Because I mean, sometimes you just need to burn budget, you know what I'm saying?So, you can actually, clearly and quantifiably understand where to focus engineering efforts towards site reliability as you introduce changes. So yeah, I think it's—and no wonder it's such a booming concept. Everybody's talking about it. I saw Gremlin just released this new certification thing. What is it, certified chaos engineer?Jason: Gremlin-certified chaos engineering practitioner.Leo: Ah, pretty cool.Jason: Yeah.Leo: I got to get me one of those. [laugh].Jason: Yeah, you should—we'll put the link in the [show notes 00:23:19], for everybody that wants to go and take that. One of the things that you've mentioned a bunch is as we talk about automation, and automating and getting chaos engineering coverage in the same way that test coverage happens, one of the things that you're involved in—and I think why you've got so much knowledge around automation—is you've been involved in the OpenGitOps Project, right?Leo: Mm-hm. Correct.Jason: Can you tell us more about that? And what does that look like now? Because I know GitOps has become this, sort of, buzzword, and I think a lot of people are starting to look into that and maybe wondering what that is.Leo: I'm co-chair of the GitOps Working Group by the CNCF, which is the working group that effectively shepherds the OpenGitOps Project. The whole idea behind the OpenGitOps Project is to come to a consensus definition of what GitOps is. And this is along the lines of—like, we were talking about DevOps, right?Like DevOps is—everybody is doing DevOps and everybody does something different. So, there is some commonality but there is not necessarily a community-agreed-upon single perspective as to what DevOps is. So, the idea behind the OpenGitOps Project and the GitOps Working Group is to basically rally the community and rally the industry towards a common opinion as to what GitOps is, eventually work towards ways to conformance and certification—so it's like you guys are doing with chaos engineering—and in an open-source community fashion. GitOps is basically a operating model for cloud-native infrastructure and applications. So, idea is that you can use the same patterns and you can use the same model to deploy and operate the underlying infrastructure as well as the workloads that are running on top of it.It's defined by four principles that might resonate as known in common for some with some caveats. So, the first principle is that your desired state, how you want your infrastructure and your workloads to look like is declarative. No, it's—you're not—there's a fundamental difference between the declarative and imperative. Imperative is you're giving instructions to reach a certain state. The current industry is just… defining the characteristics of that state, not the process by which you reached it.The current state should be immutable and should be versioned, and this is very much aligned with the whole idea of containers, which are immutable and are versioned, and the whole idea of the Gits, that if used… [unintelligible 00:26:05] if used following best practices is also immutable and versioned. So, your declared state should be versioned and immutable.it should be continuously reconciled through agents. In other words, it eliminates the human component; you are no longer executing manual jobs and you're no longer running imperative pipelines for the deployment component of your operation. You are allowing your [letting 00:26:41] agents do that for you, continuously and programmatically.And the fourth principle is, this is the only way by which you interact with the system. In other words it completely eliminates the human component from the operating model. So, for example, when I think about GitOps as a deployment mechanism, and for example, progressive delivery within the context of GitOps, I see a lot of… what's the word I'm looking for? Like, symbiosis.Jason: Yeah. Symbiosis?Leo: Yeah. Between chaos engineering, and this model of deployment. Because I think chaos engineering is also eliminating a human component; you're no longer letting humans exercise your system to find problems, you are executing those by agents, you are doing so with a declarative model, where you're declaring the attributes of the experiment and the expected outcome of that experiment, and you're defining the criteria by which you're going to abort that experiment. So, if you incorporate that model of automated, continuous validation of your solution through premeditated chaos, in a process of continuous reconciliation of your desired state, through automated deployment agents, then you have a really, really solid, reliable mechanism for the operation of cloud-native solutions.Ana: I was like, I think a lot what we've seen, I mean, especially as I sit in more CNCF stuff, is really trying to get a lot of our systems to be able to know what to do next before we need to interfere, so we don't have to wake up. So, between chaos engineering, between GitOps, between Keptn, [unintelligible 00:28:32] how is it that you can make the load of SRE and the DevOps engineer be more about making sure that things get better versus, something just broke and I need to go fix it, or I need to go talk to an engineer to go do a best practice because now those things are built into the system as a guardrail, or there's better mental models and things that are more accurate to real conditions that can happen to a system?Leo: Actually, I sidetracked. I never ended up talking more about the OpenGitOps Project and the GitOps Working Group. So, it's a community effort by the CNCF. So, it's open for contribution by everybody. You're all in the CNCF Slack, there is an OpenGitOps Slack channel there.And if you go to github.com/open-gitops, you'll be able to find ways to contribute. We are always looking to get more involvement from the community. This is also an evolving paradigm, which I think also resonates with chaos engineering.And a lot of its evolution is being driven by the use cases that are being discovered by the end-users of these technologies and the different patterns. Community involvement is very important. Industry involvement is very important. It would be fantastic and we're an open community, and I'd love to get to know more about what you're all doing with GitOps and what it means for you and how these principles apply to the challenges that your teams are running into, and the use cases that and problems spaces that you're having to deal with.Jason: I think that's a fantastic thing for our listeners to get involved in, especially as a new project that's really looking for the insight and the contribution from new members as it gets founded. As we wrap up, Leo, do you have any other projects that you want to share? How can people find you on the internet? Anything else that you want to plug?Leo: I love to meet people on these subjects that I'm very passionate about. So yes, you can find me on Twitter. I guess, it's easier to just type it, it's @murillodigital, but you'll find that in the show notes, I imagine. As well as my LinkedIn.I have to admit, I'm more of a LinkedIn person. I don't, I hope that doesn't age me or made me uncool, but I never figured out how to really work with Twitter. I'm more of a LinkedIn person, so you can find me there. I'm an organizer in the community in Costa Rica CNCF, and I run.So, for those that are Spanish speakers, I'm very much for promoting the involvement and openness of the cloud-native ecosystem to the Hispanic and Latin community. Because I think language is a barrier and I think we're coming from countries where a lot of us have struggled to basically get our head above water from lesser resources and difficult access to technology and information. But that doesn't mean that there isn't a huge amount of talent in the region. There is. And so, I run a—there's a recent initiative by the CNCF called cloud-native TV, which is we're ten shows that are streaming on Twitch.You go to cloudnative.tv, you'll see them. I run a show called Cloud Native LatinX, which is in Spanish. I invite people to talk about cloud-native technologies that are more cloud-native communities in the region.And my objective is twofold: I want to demonstrate to all Hispanics and all Latin people that they can do it, that we're all the same, doesn't matter if you don't speak the language. There is a whole bunch of people, and I am one of them that speak the language that are there, and we're there to help you learn, and support and help you push through into this community. Basically, anybody that's listening to come out and say these are actionable steps that I can take to move my career forward. So, it's every other Tuesday on cloudnative.tv, Cloud Native LatinX, if you want to hear and see more of me talking in Spanish. It's on cloudnative.tv. And the OpenGitOps Project, join in; it's open to the community. And that's me.Ana: Yes I love that shout-out to getting more folks, especially Hispanics and Latinx, be more involved in cloud and CNCF projects itself. Representation matters and folks like me and Leo come in from countries like Costa Rica, Nicaragua, we get to speak English and Spanish, we want to create more content in Spanish and let you know that you can learn chaos engineering in English and you can learn about chaos engineering in Spanish, Ingeniería de Caos. So, come on and join us. Well, thank you Leo. Muchisimas gracias por estar en el show de hoy, y gracias por estar llamando hoy desde Costa Rica, y para todos los que están oyendo hoy que también hablen español...pura vida y que se encuentren bien. Nos vemos en el próximo episodio.Leo: Muchas gracias, Ana, and thanks everybody, y pura vida para todo el mundo y ¡hagamos caos!Jason: For links to all the information mentioned, visit our website at gremlin.com/podcast. If you liked this episode, subscribe to the Break Things on Purpose podcast on Spotify, Apple Podcasts, or your favorite podcast platform. Our theme song is called, “Battle of Pogs” by Komiku, and it's available on loyaltyfreakmusic.com.

In this episode, we cover:00:00:00 - Introduction 00:04:25 - Cadence to Temporal 00:09:15 - Breaking down the Technology 00:15:35 - Three Tips for Using Temporal 00:19:21 - OutroLinks:Temporal: https://temporal.io TranscriptJason: And just so I'm sure to pronounce your names right, it's Maxim Fateev?Maxim: Yeah, that's good enough. That's my real name but it's—[laugh].Jason: [laugh]. Okay.Maxim: It's, in American, Maxim Fateev.Jason: And Samar Abbas.Samar: That sounds good.Jason: Welcome to another episode of Build Things on Purpose, part of the Break Things on Purpose podcast. In our build episodes, we chat with the engineers and developers who create tools that help us build and operate modern applications. In this episode, Maxim Fateev and Samar Abbas join us to chat about the problems with orchestrating microservices and the software they've created to help solve those problems.Jason: Hey, everyone, welcome to Break Things on Purpose, the podcast about reliability, chaos engineering, and all things SRE and DevOps. With me today, I've got Maxim Fateev and Samar Abbas, the cofounders of a company called Temporal. Maxim, why don't you tell us a little bit more about yourself?Maxim: Hi, thanks for inviting me to the podcast. I have been around for quite a while. In 2002, I joined Amazon and Amazon was pretty small company back then, I think 800 developers; compared to its current size it was really small. I quickly moved to the platform team, among other things. And it wasn't AWS back then, it was just the software platform, actually was called [Splat 00:01:36].I worked in a team which owned the old publish-subscribe technologies of Amazon, among other things. As a part of the team, I oversaw implementation of service and architecture, Amazon [unintelligible 00:01:47] roll out services at large scale, and they built services for [unintelligible 00:01:51] Amazon so all this asynchronous communication, there was something my team was involved in. And I know this time that this is not the best way to build large-scale service-oriented architectures, or relying on asynchronous messaging, just because it's pretty hard to do without central orchestration. And as part of that, our team conceived and then later built a Simple Workflow Service. And I was tech leader for the public release of the AWS Simple Workflow Service.Later, I also worked in Google and Microsoft. Later I joined Uber. Samar will tell his part of the story but we together built Cadence, which was, kind of, the open-source version of the same—based on the same ideas of the Simple Workflow. And now we driving Temporal open-source project and the company forward.Jason: And Samar, tell us a little bit about yourself and how you met Maxim.Samar: Thanks for inviting us. Super excited to be here. In 2010, I was basically wanted to make a switch from traditional software development like it used to happen back at Microsoft, to I want to try out the cloud side of things. So, I ended up joining Simple Workflow team at AWS; that's where I met Maxim for the first time. Back then, Maxim already had built a lot of messaging systems, and then saw this pattern where messaging turned out—[unintelligible 00:03:08] believe that messaging was the wrong abstraction to build certain class of applications out there.And that is what started Simple Workflow. And then being part of that journey, I was, like, super excited. Since then, in one shape or another, I've been continuing that journey, which we started back then in the Simple Workflow team while working with Maxim. So, later in 2012, after shipping Simple Workflow, I basically ended up coming back to Azure side of things. I wrote this open-source library by the name of Durable Task Framework, which looks like later Azure Functions team ended up adopting it to build what they are calling as Azure Durable Functions.And then in 2015, Uber opened up office here in Seattle; I ended up joining their engineering team in the Seattle office, and out of coincidence, both me and Max ended up joining the company right about the same time. Among other things we worked on together, like, around 2017, we started the Cadence project together, which was you can think of a very similar idea as like Simple Workflow, but kind of applying it to the problem we were seeing back at Uber. And one thing led to another and then now we are here basically, continuing that journey in the form of Temporal.Jason: So, you started with Cadence, which was an internal tool or internal framework, and decided to strike out on your own and build Temporal. Tell me about the transition of that. What caused you to, number one, strike out on your own, and number two, what's special about Temporal?Maxim: We built the Cadence from the beginning as an open-source project. And also it never was, like, Uber management came to us and says, “Let's build this technology to run applications reliably,” or workflow technology or something like that. It was absolutely a bottoms-up creation of that. And we are super grateful to Uber that those type of projects were even possible. But we practically started on our own, we build it first version of that, and we got resources later.And [unintelligible 00:05:09] just absolutely grows bottoms-up adoption within Uber. It grew from zero to, like, over a hundred use cases within three years that this project was hosted by our team at Uber. But also, it was an open-source project from the beginning, we didn't get much traction first, kind of, year or two, but then after that, we started to see awesome companies like HashiCorp, Box, Coinbase, Checkr, adopt us. And there are a lot of others, it's just that not all of them are talking about that publicly. And when we saw this external adoption, we started to realize that thing within Uber, we couldn't really focus on external events, like, because we believe this technology is very widely applicable, we needed, kind of, have separate entity, like a company, to actually drive the technology forward for the whole world.Like most obvious thing, you cannot have a hosted version [unintelligible 00:06:00] at Uber, right? We would never create a cloud offering, and everyone wants it. So, that is, kind of like, one thing led to another, Samar said, and we ended up leaving Uber and starting our own company. And that was the main reasoning is that we wanted to actually make this technology successful for everybody in the whole world, not just within Uber. Also the, kind of, non-technical but also technical reasons, one of the benefits of doing that was that we had actually accumulated quite pretty large technical debt when running, like, Cadence, just because we were in it for four years without single backwards-incompatible change because since [unintelligible 00:06:37] production, we still were on the same cluster with the same initial users, and we never had downtime, at least, lik, without relat—infrequent outages.So, we had to do everything in backwards-compatible manner. At Temporal, we could go and rethink that a little bit, and we spent almost a year just working on the next generation of technology and doing a lot of fixes and tons of features which we couldn't do otherwise. Because that was our only chance to do backwards-incompatible change. After our first initial production release, we have this promise that they're not going to break anyone, at least—unless we start thinking about the next major change in the project, which probably is not going to come in the next few years.Samar: Yeah. One thing I would add is back then one of the value propositions that Uber was going after is provide rides as reliable as running water. And that translated into interesting system requirements for engineers. Most of the time, what ended up happening is product teams at Uber are spending a large amount of time building this resiliency and reliability into their applications, rather than going after building cool features or real features that the users of the platform cares about. And I think this is where—this is the problem that we were trying to solve with us back at Uber, where let us give you that reliability baked into the platform.Why does every engineer needs to be a distributed systems engineer to deal with all sorts of failure conditions? We want application teams to be more focused on building amazing applications, which makes a lot of sense for the Uber platform in general. And this is the value proposition that we were going after with Cadence. And it basically hit a nerve with all the developers out there, especially within Uber. One of the very funny incidents early on, when we are getting that early adoption, one of the use cases there, the way they moved onto Cadence as an underlying platform is there was actually an outage, a multi-day outage in one of the core parts of that system, and the way they mitigated the outage is they rewrote that entire system on top of Cadence in a day, and able to port over that entire running system in production and build it on top of Cadence and run it in production. And that's how they mitigated that outage. So, that was, in my opinion, that was a developer experience that we were trying to strive, with Cadence.Jason: I think, let's dive into that a little bit more because I think for some of our listeners, they may not understand what we're talking about with this technology. I think people are familiar with simple messaging, things like Kafka, like, “I have a distributed system. It's working asynchronously, so I do some work, I pass that into a queue of some sort, something pulls that out, does some more work, et cetera, and things act decoupled.” But I think what we're talking about here with workflows, explain that for our listeners a little bit more. What does it provide because I've taken a look at the documentation and some of the demos and it provides a lot of really cool features for reliability. So, explain it a little bit more, first.Maxim: [crosstalk 00:09:54] describe pretty well how systems are built now. A lot of people, kind of, call it choreography. But basic idea is that you have a bunch of callbacks which listen on queues, then update certain data sources' databases, and then put messages into the queues back. And also they need to actually—in a real system also need to have durable timers, so you either build your own timer service, so you just poll your databases for messages to be in certain state to account for time. And skill in these things are non-trivial.The worst part is that your system has a bunch of independent callbacks and you practically have very complex state machine and all these business requirements just practically broken in, like, a thousand, thousand little pieces which need to work together. This choreography in theory kind of works, but in practice is usually a mess. On top of that, you have very poor visibility into your system, and all this other requirements about retries and so on are actually pretty hard to get. And then if something goes wrong, good luck finding the problem. It goes into the orchestrat—it does orchestration.Means that you implement your business logic in one place and then you just call into these downstream services to implement the business logic. The difference is that we know how to do that for short requests. Practically, let's say you get a request, your service makes the five downstream API calls, does something with those calls, maybe makes a little bit more calls, than [unintelligible 00:11:14] data. If this transactions takes, let's say, a second, is pretty easy to do, and you don't care about reliability that much if it fails in the middle. But as soon as they come to you and say, “Okay, but any of those calls can fail for three minutes,” or, “This downstream call can take them ten hours,” you practically say, “Okay, I have this nice piece of code which was calling five services and doing a few things. Now, I need to break it into 50 callbacks, queues, and whatever in database, and so on.”It's Temporal [unintelligible 00:11:38] keep that code [unintelligible 00:11:39]. The main abstraction, which is non-obvious to people is that they practically make your process fully fault-tolerant, including stack variables, [unintelligible 00:11:49], and so on. So, if you make a call, and this call takes five hours, you're still blocked in exactly the same line of code. And in five hours, this line of code returns and then continues. If you're calling sleep for one month, you're blocked on this sleep line of code for one month, and then it just returns to the next line of code.Obviously, there is some magic there, in the sense that we need to be able to [unintelligible 00:12:11] and activate state of your workflow—and we call it workflows but this code—but in exactly the same state, but this is exactly what Temporal provides out of the box. You write code, this failure doesn't exist because if your process fails, we just reconstruct the exactly the same state in a different process, and it's not even visible to you. I sometimes call it a fault-oblivious programming because your program not even aware that fault happened because it just automatically self-healing. That is main idea. So, we'll give you a fault-tolerant code is guaranteed to finish execution.And on top of that, there are a lot of things which kind of came together there. We don't invoke these services directly, usually, we invoke them from queues. But these queues are hidden in a sense because all you say execute, for instance, some activity, call some API, and then this API is in workflow asynchronously. But for your coding code, it's not visible. It's, kind of, just normal RPC call, but behind the scenes, it's all asynchronous, has infinite retries, exponential retries, a [unintelligible 00:13:11] granular tasks, and so on.So, there are a lot of features. But the main thing which we call workflow which ties all these together, is just your business logic because all it does is just practically makes this call. And also it has state. It's stateful because you can keep state in variables and you don't need to talk to database. So, it can have you—for example, one of the use cases we saw is customer loyalty program.Imagine you want to implement the UI airline, you need to implement points, give points to people. So, you listen to external events every time your trip finished, your flight finished, and then you will get event or a need to increment that. So, in normal system, you need to get better-based cues, and so in our world, you would just increment local variables saying, “Yeah, it's a counter.” And then when this counter reaches a hundred, for example, you will call some downstream service and say, “Okay, promote that person to the next tier.” And you could write this practically your type of application in 15, 20 minutes on your desktop because all of the rest is taken care of by Temporal.It assumes that you can have millions of those objects and hundreds of millions of those objects running because you have a lot of customers—and we do that because we built it at Uber, which had hundreds of millions of customers—and run this reliably because you don't want to lose data. It's [unintelligible 00:14:29] your financial data. This is why Temporal is very good for financial transactions, and a lot of companies like Coinbase, for example, uses them for their financial transactions because we provide much better [unintelligible 00:14:39] that alternative solutions there.Jason: That's amazing. In my past as an engineer of working on systems and trying to do that, particularly when you mentioned things like retries, I'm thinking of timeouts and things where you have a long-running process and you're constantly trying to tune, what if one service takes a long time, but then you realize that up the stack, some other dependency had ended up timing out before your timeout hit, and so it's suddenly failed even though other processes are going, and it's just this nightmare scenario. And you're trying to coordinate amongst services to figure out what your timeout should be and what should happen when those things timeout and how retries should coordinate among teams. So, the idea of abstracting that away and not even having to deal with that is pretty amazing. So, I wanted to ask you, as this tool sounds so amazing, I'm sure listeners will want to give this a try if they're not already using it.If I wanted to try this out, if I wanted to implement Temporal within my application, what are, say, three things that I need to keep in mind to ensure that I have a good experience with it, that I'm actually maintaining reliability, or improving reliability, due to using this framework, give us some tips for [crosstalk 00:15:53] Temporal.Maxim: One thing which is different about Temporal, it requires you rethinking how application is structured. It's a kind of new category of software, so you cannot just take your current design and go and translate to that. For example, I've seen cases that people build system using queues. They have some downstream dependency which can be down. For example, I've seen the payment system, the guys from the payment system came to us and said, “What do we do? We have downstream dependency to bank and they say in the SLA that can be done for three days. Can we just start workflow on every message if the system is down, and keep retrying for three days?” Well, technically, answer is yes, you can absolutely create workflows, productivity retried options for retry for a month, and it is going to work out of the box, but my question to those people first time is what puts message in that Kafka queue which your are listen, you know?And they are, “Oh, it's a proxy service which actually does something and”—“But how does the service is initiated?” And they say, “Oh, it's based on another to Kafka queue.” And the whole thing, I kind of ended up understanding that they had this huge pipeline, this enormous complexity, and they had hard time maintaining that because all these multiple services, multiple data sources, and so on. And then we ended up redesigning the whole pipeline as just one workflow, and instead of just doing this by little piece, and it helped them tremendously; it actually practically completely changed the way application was designed and simplified, they removed a lot of code, and all these, practically, technology, and reliability was provided by Temporal out of the box. So, my first thing is that don't try to do piecemeal.Again, you can. Yeah, actually, they initially even did that, just to try to prove the technology works, but at the end, think about end-to-end scenario, and if you use Temporal for end-to-end scenario, you will get [10x 00:17:37] benefits there. That probably would be the most important thing is just think about design.Samar: So, I think Max, you gave a pretty awesome description of how you should be approaching building applications on top of Temporal. One of the things that I would add to that is think about how important durability is for your application. Do you have some state which needs to live beyond a single request response? If the answer to those questions is yes, then I think Temporal is an awesome technology, which helps you deal with that complexity, where a traditional way of building those applications, using databases, queues, retry mechanisms, durable timers, as Max mentioned, for retrying for three days because we have—instead of building a sub-system which deals with this retrying behavior, you can literally just, when you schedule an activity, you just put an activity options on it to, say, put your retry policy and then it will retry based on three days, based on your policy. So, I think—think holistically about your system, think about the statefulness and how important that state is, and I think Temporal is an amazing technology which really developer-friendly because the way currently industry—I think people have accepted that building these class of applications for cloud environment is inherently complex.So, now a lot of innovation which happens is how to deal with that complexity as opposed to what Temporal is trying to do is, no, let's simplify the entire experience of building such applications. I think that's the key value that we are trying to provide.Jason: I think that's an excellent point, thinking of both of those tips of thinking about how your application is designed end-to-end, and also thinking about those pieces. I think one of the problems that we have when we build distributed applications, and particularly as we break things into smaller and smaller services, is that idea of for something that's long-running, where do I store this information? And a lot of times we end up creating these interesting, sort of, side services to simply pass that information along, ensure that it's there because we don't know what's going to happen with a long-running application. So, that's really interesting. Well, I wanted to thank you for joining the podcast. This has been really fantastic if folks want to find more information about Temporal or getting involved in the open-source project or using the framework, where should they head?Maxim: At temporal.io. And we have links to our Git repos, we have links to our community forum, and we also have, at the bottom that page, there is a link to the Slack channel. So, we have pretty vibrant community, so please join it. And we are always there to help.Jason: Awesome, thanks again.Maxim: Thank you.Jason: For links to all the information mentioned, visit our website at gremlin.com/podcast. If you liked this episode, subscribe to the Break Things on Purpose podcast on Spotify, Apple Podcasts, or your favorite podcast platform. Our theme song is called, “Battle of Pogs” by Komiku, and it's available on loyaltyfreakmusic.com.

In this episode, we cover: 00:00:00 - Introduction  00:03:15 - FinOps Foundation and Multicloud  00:07:00 - Costs  00:10:40 - John's History in Reliability Engineering  00:16:30 - The Actual Cost of an Outages, Security, Etc. 00:21:30 - What John Measures  00:28:00 - What John is Up To/Latinx in Tech Links: Palo Alto Networks: https://www.paloaltonetworks.com/ FinOps Foundation: https://www.finops.org Techqueria.org: https://techqueria.org LinkedIn: https://www.linkedin.com/in/johnmartinez/ TranscriptJohn: I would say a tip for better monitoring, uh, would be to, uh turn it on. [laugh]. [unintelligible 00:00:07] sounds, right?Jason: Welcome to the Break Things on Purpose podcast, a show about chaos engineering and operating reliable systems. In this episode we chat with John Martinez, Director of Cloud R&D at Palo Alto Networks. John's had a long career in tech, and we discuss his new focus on FinOps and how it has been influenced by his past work in security and chaos engineering.  Jason: So, John, welcome to the show. Tell us a little bit about yourself. Who are you? Where do you work? What do you do?John: Yeah. So, John Martinez. I am a director over at Palo Alto Networks. I have been in the cloud security space for the better of, I would say, seven, eight years or so. And currently, am in transition in my role at Palo Alto Networks.So, I'm heading headstrong into the FinOps world. So, turning back into the ops world to a certain degree and looking at what can we do, two things: better manage our cloud spend and gain a lot more optimization out of our usage in the cloud. So, very excited about new role.Jason: That's an interesting new role. I'd imagine that at Palo Alto Networks, you've got quite a bit of infrastructure and that's probably a massive bill.John: It can be. It can be. Yeah, [laugh] absolutely. We definitely have large amount of scale, in multi-cloud, too, so that's the added bonus to it all. FinOps is kind of a new thing for me, so I'm pretty happy to, as I dig back into the operations world, very happy to discover that the FinOps Foundation exists and it kind of—there's a lot of prescribed ways of both looking at FinOps, at optimization—specifically in the cloud, obviously—and as well as there's a whole framework that I can go adopt.So, it's not like I'm inventing the wheel, although having been in the cloud for a long time, and I haven't talked about that part of it but a lot of times, it feels like—in my early days anyway—felt like I was inventing new wheels all the time. As being an engineer, the part that I am very excited about is looking at the optimization opportunities of it. Of course, the goal, from a finance perspective, is to either reduce our spend where we can, but also to take a look at where we're investing in the cloud, and if it takes more of a shift as opposed to a straight-up just cut the bill kind of thing, it's really all about making sure that we're investing in the right places and optimizing in the right places when it comes down to it.Jason: I think one of the interesting perspectives of adopting multi-cloud is that idea of FinOps: let's save money. And the idea, if I wanted to run a serverless function, I could take a look at AWS Lambda, I could take a look at Azure Functions to say, “Which one's going to be cheaper for this particular use case,” and then go with that.John: I really liked how the FinOps Foundation has laid out the approach to the lifecycle of FinOps. So, they basically go from the crawl, walk, run approach which, in a lot of our world, is kind of like that. It's very much about setting yourself up for success. Don't expect to be cutting your bill by hundreds of thousands of dollars at the beginning. It's really all about discovering not just how much we're spending, but where we're spending it.I would categorize the pitting the cloud providers against each other to be more on the run side of things, and that eventually helps, especially in the enterprise space; it helps enterprises to approach the cloud providers with more of a data-driven negotiation, I would say [laugh] to your enterprise spend.Jason: I think that's an excellent point about the idea of that is very much a run. And I don't know any companies within my sphere and folks that I know in the engineering space that are doing that because of that price competition. I think everybody gets into the idea of multi-cloud because of this idea of reliability, and—John: Mm-hm.Jason: One of my clouds may fail. Like, what if Amazon goes down? I'd still need to survive that.John: That's the promise, right? At least that's the promise that I've been operating under for the 11 years or so that I've been in the cloud now. And obviously, in the old days, there wasn't a GCP or an Azure—I think they were in their infancy—there was AWS… and then there was AWS, right? And so I think eventually though you're right, you're absolutely right. Can I increase my availability and my reliability by adopting multiple clouds?As I talk to people, as I see how we're adopting the multiple clouds, I think realistically though what it comes down to is you adopted cloud, or teams adopt a cloud specifically for, I wouldn't say some of the foundational services, but mostly about those higher-level niche services that we like. For example, if you know large-scale data warehousing, a lot of people are adopting BigQuery and GCP because of that. If you like general purpose compute and you love the Lambdas, you're adopting AWS and so on, and so forth. And that's what I see more than anything is, I really like a cloud's particular higher level service and we go and we adopt it, we love it, and then we build our infrastructure around it. From a practical perspective, that's what I see.I'm still hopeful, though, that there is a future somewhere there where we can commoditize even the cloud providers, maybe [laugh]. And really go from Cloud A to Cloud B to Cloud C, and just adopt it based on pricing I get that's cheaper, or more performant, or whatever other dimensions that are important to me. But maybe, maybe. We'll remain hopeful. [laugh].Jason: Yeah, we're still very much in that spot where everybody, despite even the basics of if I want to a virtual machine, those are still so different between all the clouds. And I mean even last week, I was working on some Terraform and the idea of building it modularly, and in my head thinking, “Well, at some point, we might want to use one of the other clouds so let's build this module,” and thinking, “Realistically, that's probably not going to happen.”John: [laugh]. Right. I would say that there's the other hidden cost about this and it's the operational costs. I don't think we spend a whole lot of time talking about operational costs, necessarily, but what is it going to cost to retrain my DevOps team to move from AWS to GCP, as an example? What are the underlying hidden costs that are there?What traps am I going to fall into because of that? It seems cool; Terraform does a great job of getting that pain into the multiple clouds from an operations perspective. Kubernetes does a great job as well to take some of that visibility into the underlying—and I hate to use it this way but ‘hardware' [laugh] virtual hardware—that's like EC2 or Google Compute, for example. And they do great jobs, but at the end of the day we're still spending a lot of time figuring out what the foundational services are. So, what are those hidden costs?Anyway, long story short, as part of my journey into FinOps, I'm looking forward into not just uncovering the basics of FinOps, where is what are we spending? Where are we spending it? What are the optimization opportunities? But also take a look at some of the more hidden types of costs. I'm very interested in that aspect of the FinOps world as well. So, I'm excited.Jason: Those hidden costs are also interesting because I think, given your background in security—John: Mm-hm.Jason: —one of the challenges in multi-cloud is, if I'm an expert in AWS and suddenly we're multi-cloud and I have to support GCP, I don't necessarily know all of those correct settings and how to necessarily harden and build my systems. I know a model and a general framework, but I might be missing something. Talk to me a bit more about that as a security person.John: Yeah.Jason: What does that look like?John: Yeah, yeah. It's very nuanced, for sure. There are definitely some efforts within the industry to help alleviate some of that nuance and some of those hidden settings that I might not think about. For example, CIS Foundations as a community, the foundations of benchmarks that CIS produces can be pretty exhaustive—and there are benchmarks for the major clouds as well—those go a long way to try and describe at least, what are the main things I should look at from a security perspective? But obviously, there are new threats coming along every day.So, if I was advising security teams, security operations team specifically, it would be definitely to keep abreast into what are the latest and go take a look at what some of the exploit kits are looking for or doing and adopting some of those hidden checks into, for example, your security operations center, what you react to, what the incident responses are going to be to some of those emerging threats. For sure it is a challenge, and it's a challenge that the industry faces and one that we go every day. And an exploit that might be available for EC2 may be different on Google Compute or maybe different on Azure Compute.Jason: There's a nice similarity or parallel there to what we often talk about, especially in this podcast, is we talk about chaos engineering and reliability and that idea of let's look at how things fail and take what we know about one system or one service, and how can we apply that to others? From your experience doing a wide breadth of cloud engineering, tell me a bit more about your experience in the reliability space and keeping—all these great companies that you've worked for, keeping their systems up and running.John: I think I have one of the—fortunate to have one of the best experiences ever. So, I'll have to dig way back to 11 years ago, or so [laugh]. My first job in the cloud was at Netflix. I was at Netflix right around the time when we were moving applications out of the data center and into AWS. Again, fortunate; large-scale, at the cusp of everything that was happening in the cloud, back in those days.I had just helped finish—I was a systems engineer; that's where I transitioned from, systems engineering—and just a little bit of a plug there, tomorrow is Sysadmin Day, so I still am an old school sysadmin at heart so I still celebrate Sysadmin Day. [laugh]. But I was doing that transition from systems engineering into cloud engineering at Netflix, just helped move a database application out from the data center into AWS. We were also adopting in those days, very rapidly, a lot of the new services and features that AWS was rolling out. For example, we don't really think about it today anymore, but back then EBS-backed instances was the thing. [laugh].Go forth and every new EC2 instance we create is going to be EBS-backed. Okay, great. March, I believe it was March 2011, one of AWS's very first, and I believe major, EBS outages occurred. [laugh]. Yeah, lots of, lots of failure all over the place.And I believe from that a lot of what—at least in Gremlin—a lot of that Chaos Monkey and a lot of that chaos engineering really was born out of a lot of our experiences back then at Netflix, and the early days of the cloud. And have a lot of the scars still on me. But it was a very valuable lesson that I take now every day, having lived through it. I'm sure you guys at Gremlin see a lot of this with your customers and with yourselves, right, is that the best you can do is test those failure scenarios and hope that you are as resilient as possible. Could we have foreseen that there was going to be a major EBS outage in us-east-1? Probably.I think academically we thought about it, and we were definitely preaching the mantra of architect for failure, but it still bit us because it was a major cascading outage in one entire region in AWS. It started with one AZ and it kept rolling, and it kept rolling. And so I don't know necessarily in that particular scenario that we could have engineered—especially with the technology of the day—we could have engineered full-on failover to another region, but it definitely taught us and me personally a lot of lessons around how to architect for failure and resiliency in the cloud, for sure.Jason: I like that point of it's something that we knew theoretically could maybe happen, but it always seems like the odds of the major catastrophes are so small that we often overlook them and we just think, “Well, it's going to be so rare that it'll never happen, so we don't think about it.” As you've moved forward in your career, moving on from Netflix, how has that shaped how you approach reliability—this idea of we didn't think EBS could ever go down and lead to this—how do you think of catastrophic failures now, and how do you go about testing for them or architecting to withstand them?John: It's definitely stayed with me. Every ops job that I've had since, it's something that I definitely take into account in any of those roles that I have. As the opportunity came up to speak with you guys, wanted to think about reliability and chaos in terms of cloud spend, and how can I marry those two worlds together? Obviously, the security aspect of things, for sure, is there. It's expecting the unexpected and having the right types of security monitoring in place.And I think that's—kind of going back to an earlier comment that I made about these unexpected or hidden costs that are there lying dormant in our cloud adoption, just like I'm thinking about the cost of security incidents, the cost of failure, what does that look like? These are answers I don't have yet but the explorer in me is looking forward to uncovering a lot of what that's going to be. If we talk in a year from now, and I have some of that prescribed, and thought of, and discovered, and I think it'll be awesome to talk about it in a year's time and where we are. It's an area that I definitely take seriously I have applied not just to operational roles, but as I got into more customer-facing roles in the last 11 years, in between advising customers, both as a sales engineer, as head of customer success, and cloud security startup that I worked for, Evident.io, and then eventually moving here to Palo Alto Networks, it's like, how do I best advise and think about—when I talk to customers—about failure scenarios, reliability, chaos engineering? I owe it all to that time that I spent at Netflix and those experiences very early on, for sure.Jason: Coming back to those hidden costs is definitely an important thing. Especially I'm sure that as you interact with folks in the FinOps world, there's always that question of, “Why do I have so much redundancy? Why am I paying for an entire AZs worth of infrastructure that I'm never using?” There's always the comment, “Well, it's like a spare tire; you pay for an extra tire in case you have a flat.” But on some hand, there is this notion of how much are we actually spending versus what does an outage really cost me?John: Right. We thought about that question very early on at another company I worked at after Netflix and before the startup. I was fortunate again to work in another large-scale environment, at Adobe actually, working on the early days of their Creative Cloud implementation. Very different approach to doing the cloud than Netflix in many ways. One of the things that we definitely made a conscious effort to do, and we thought about it in terms of an insurance policy.So, for example, S3 replication—so replicating our data from one region to another—in those days, an expensive proposition but one that we looked at, and we intentionally went in with, “Well, no, this is our customer data. How much is that customer data worth to us?” And so we definitely made the conscious decision to invest. I don't call it ‘cost' at that point; I call that an investment. To invest in the reliability of that data, having that insurance policy there in case something happened.You know, catastrophic failure in one region, especially for a service as reliable and as resilient as S3 is very minuscule, I would say, and in practice, it has been, but we have to think about it in terms of investing. We definitely made the right types of choices, for sure. It's an insurance policy. It's there because we need it to be there because that's our most precious commodity, our customers' data.Jason: Excellent point about that being the most precious commodity. We often feel that our data isn't as valuable as we think it is and that the value for our companies is derived from all of the other things, and the products, and such. But when it comes down to it, it is that data. And it makes me think we're currently in this sort of world where ransomware has become the biggest headline, especially in the security space, and as I've talked with people about reliability, they often ask, “Well, what is Gremlin do security-wise?” And we're not a security product, but it does bring that up of, if your data systems were locked and you couldn't get at your customer information, that's pretty similar to having a catastrophic outage of losing that data store and not having a backup.John: I've thought about this, of course, in the last few weeks, obviously. A very, very public, very telling types of issues with ransomware and the underlying issues of supply chain attacks. What would we do [laugh] if something like that were to happen? Obviously, rhetorically, what would we do? And lots of companies are paying the ransom because they're being held at gunpoint, you know, “We have your data.”So yeah, I mean, a lot of it, in the situation, like the example I gave before, could not just the replication of, for example, my entire S3 bucket where my customer data is thwarted a situation like that? And then you think about, kind of like, okay, let's think about this further. If we do it in the same AWS account, as an example, if the attacker obtained my IAM credentials, then it really comes down to the same thing because, “Oh, look it, there's another bucket in that other region over there. I'm going to go and encrypt all of those objects, too. Why not, right?” [laugh].And so, it also begs the question or the design principles and decisions of, well, okay, maybe do I ship it to a different account where my security context is different, my identity context is different? And so there's a lot of areas to explore there. And it's very good question and one that we definitely do need to think about, in terms of catastrophic failure because that's the way to think about it, for sure.Jason: Yeah. So, many parallels between that security and reliability, and all comes together with that FinOps, and how much are you—how much do we pay for all of this?John: Between the reliability and the security world, there's a lot of parallels because your job is about thinking what are the worst-case scenarios? It's, what could possibly go wrong? And how bad could it be? And in many cases, how bad is it? [laugh].Especially as you uncover a lot of the bad things that do happen in the real world every day: how bad is it? How do I measure this? And so absolutely there's a lot of parallels, and I think it's a very interesting point you make. And so… yeah so, Jason, how can we marry the two worlds of chaos engineering and security together? I think that's another very exciting topic, for sure.Jason: That is, absolutely. You mentioned just briefly in that last statement, how do you measure it?John: Yep.Jason: That comes up to something that we were chatting about earlier is monitoring, and what do you measure, and ensuring that you're measuring the right things. From your experience building secure systems, talk to me about what are some of the things that you like to measure, that you like to get observability on, that maybe some folks are overlooking.John: I think the overlooking part is an interesting angle, but I think it's a little bit more basic than that even. I'll go to my time in the startup—so at Evident.io—mainly because I was in customer success and my job was to talk to our customers every day—I would say that a bunch of our customers—and they varied based on maturity level, but we were working with a lot of customers that were new in the cloud world, and I would say a lot of customers were still getting tripped up by a lot of the basic types of things. For example—what do I mean by that? Some of the basic settings that were incorrect were things just, like, EC2 security groups allowing port 22 in from the world, just the simple things like that. Or publicly accessible S3 buckets.So, I would say that a lot of our customers were still missing a lot of those steps. And I would say, in many of the cases, putting my security hat on, the first thing you go to is, well, there's an external hacker trying to do something bad in your AWS accounts, but really, the majority of the cases were all just mistakes; they were honest. I'm an engineer setting up a dev account and it's easier for me, instead of figuring out what my egress IP is for my company's VPN, it's easier for me just to set port 22 to allow all from the world. A few minutes later, there you go. [laugh]. Exploit taken, right? It's just the simple stuff; we really as an industry do still get tripped up by the simple things.I don't know if this tracks with the reliability world or the chaos engineering world, but I still see that way too much. And that just tells me that even if we are in the cloud—mature company or organization—there's still going to be scenarios where that engineer at two in the morning just decides that it's just easier to open up the firewall on EC2 than it is to do, quote-unquote, “The right thing.” Then we have an issue. So, I really do think that we can't let go of not just monitoring the basics, but also getting better as an industry to alert on the basics and when there are misconfigurations on the basics, and shortening that time to alert because that really is—especially in the security world—that really is very critical to make sure that window between when that configuration setting is made to when that same engineer who made the misconfiguration get alerted to the fact that it is a misconfiguration. So. I'll go to that: it's the basics. [laugh].Jason: I like that idea of moving the alert forward, though. Because I think a lot of times you think of alerts as something bad has happened and so we're waiting for the alert to happen when there's wrongful access to a system, right? Someone breaks in, or we're waiting for that alert to happen when a system goes down. And we're expecting that it's purely a response mechanism, whereas the idea of let's alert on misconfigurations, let's alert on things that could lead to these, or that will likely lead to these wrong outcomes. If we can alert on those, then we can head it off.John: It's all the way. And in the security world, we call it shifting left, shifting security all the way to the left, all the way to the developer. Lots of organizations are making a lot of the right moves in that direction for embedding security well into the development pipeline. So, for example, I'll name two players in the Infrastructure as Code as we call it in the security space. And I'll name the first one just because they're part of Palo Alto Networks now, so Bridgecrew; so very strong, open-source solution in that space, as well as over on the HashiCorp side where Sentinel is another example of a great developer-forward shift-left type of tool that can help thwart a lot of the simple security misconfigurations, right from your CI/CD pipelines, as opposed to the reaction time over here on the right, where you're chasing security misconfigurations.So, there's a lot of opportunity to shorten that alert window. And even, in fact, I've spent a lot of time in the last couple of years—I and my team have spent a lot of time in the last couple of years thinking about what can the bots do for us, as opposed to waiting for an alert to pop up on a Slack message that says, “Hey, engineer. You've got port 22 open to the world. You should maybe think about doing something.” The right thing to do there is for something—could be something as simple as an alert making it to a Lambda function and the Lambda function closing it up for you in the middle of the night when you're not paying attention to Slack, and the bot telling you, “Hey, engineer. By the way, I closed the port up. That's why it's broken this morning for you.” [laugh]. “I broke it intentionally so that we can avoid some security problems.”So, I think there's the full gamut where we can definitely do a lot more. And that's where I believe the new world, especially in the security world, the DevSecOps world, can definitely help embed some of that security mindset with the rest of the cloud and DevOps space. It's certainly a very important function that needs to proliferate throughout our organizations, for sure.Jason: And we're seeing a lot of that in the reliability world as well, as people shift left and developers are starting to become more responsible for the operations and the running of their services and applications, and including being on call. That does bring to mind that idea, though—back to alerting on configurations and really starting to get those alerts earlier, not just saying that, “Hey, devs, you're on call so now you share a pain,” but actually trying to alleviate that pain even further to the left. Well, we're coming up close to time here. So, typically at this point, one thing that I like to do is we like to ask folks if they have anything to plug. Oftentimes that's where people can find you on social media or other things. I know that you're connected with Ana through Latinx in Tech, I would love to share more about that, too. So.John: For sure, yeah. So, my job in terms of my leadership role is definitely to promote a lot of diversity, inclusion, and equity, obviously, within the workspace. Personally, I do also feel very strongly that I should be not just preaching it, but also practicing it. So, I discovered in the last year—in fact, it's going to be about a year since I joined Techqueria—so techqueria.org—and we definitely welcome anybody and everybody.We're very inclusive, all the way from if you're a member of the Latinx community and in technology, definitely join us, and if you're an ally, we definitely welcome you with open arms, as well, to join techqueria.org. It is a very active and very vibrant community on Slack that we have. And as part of that, I and a couple of people in Techqueria are running a couple of what we call cafesitos which is the Spanish word for coffees, coffee meetings.So, it's a social time, and I'm involved in helping lead both the cybersecurity cafecito—we call it Cafecito Cibernético, which happens every other Friday. And it's security-focused, it's security-minded, we go everywhere from being very social and just talking about what's going on with people personally—so we like to celebrate personal wins, especially for those that are joining the job market or just graduating from school, et cetera, and talk about their personal wins, as well as talk about the happenings, like for example, a very popular topic of late has been supply chain attacks and ransomware attacks, so definitely very, very timely there. As well as I'm also involved—being in the cloud security space, I'm bridging, sort of, two worlds between the DevOps world and the security world; more recently, we started up the DevOps Cafecito, which is more focused on the operations side. And that's where, you know, happy to have Ana there as part of that Cafecito and helping out there. Obviously, there, it's a lot of the operations-type topics that we talk about; lots of Kubernetes talk, lots of looking at how the SRE and the DevOps jobs look in different places.And I wouldn't say I'm surprised by it, but it's very nice to see that there is also a big difference with how different organizations think about reliability and operations. And it's varied all over the place and I love it, I love the diversity of it. So anyway, so that's Techqueria, so very happy to be involved with the organization. I also recently took on the role of being the chapter co-director for the San Francisco chapter, so very happy to be involved. As we come out of the pandemic, hopefully, pretty soon here [laugh] right—as we're coming out of the pandemic, I'll say—but looking forward to that in-person connectivity and socializing again in person, so that's Techqueria.So, big plug for Techqueria. As well, I would say for those that are looking at the FinOps world, definitely check out the FinOps Foundation. Very valuable in terms of the folks that are there, the team that leads it, and the resources, if you're looking at getting into FinOps, or at least gaining more control and looking at your spend, not so much like this, but with your eyes wide open. Definitely take a look at a lot of the work that they've done for the FinOps community, and the cloud community in general, on how to take a look at your cloud cost management.Jason: Awesome. Thanks for sharing those. If folks want to follow you on social media, is that something you do?John: Absolutely. Mostly active on LinkedIn at johnmartinez on LinkedIn, so definitely hit me up on LinkedIn.Jason: Well, it's been a pleasure to have you on the show. Thanks for sharing all of your experiences and insight.John: Likewise, Jason. Glad to be here.Jason: For links to all the information mentioned, visit our website at gremlin.com/podcast. If you liked this episode, subscribe to the Break Things on Purpose podcast on Spotify, Apple Podcasts, or your favorite podcast platform. Our theme song is called, “Battle of Pogs” by Komiku, and it's available on loyaltyfreakmusic.com.

In this episode, we cover: What Kessel Run is Doing: 00:01:27 Failure Never has a Single Point: 00:05:50 Lessons Learned: 00:10:50 Working the DOD:00:13:40 Automation and Tools: 00:18:02 Links: Kessel Run: https://kesselrun.af.mil Kessel Run LinkedIn: https://www.linkedin.com/company/kesselrun/ TranscriptOmar: But I'll answer as much as I can. And we'll go from there.Jason: Yeah. Awesome. No spilling state secrets or highly classified info.Omar: Yes.Jason: Welcome to Break Things on Purpose, a podcast about chaos engineering and building reliable systems.Jason: Welcome back to Break Things on Purpose. Today with us we have guest Omar Marrero. Omar, welcome to the show.Omar: Thank you. Thank you, man. Yeah, happy to be here.Jason: Yeah. So, you've been doing a ton of interesting work, and you've got a long history. For our listeners, why don't you tell us a little bit more about yourself? Who are you? What do you do?Omar: I've been in the military, I guess, public service for a while. So, I was military before, left that and now I've joined as a government employee. I love what I do. I love serving the country and supporting the warfighters, making sure they have the tools. And throughout my career, it's been basically building tools for them, everything they need to make their stuff happen.And that's what drives me. That's my passion. If you've got the tool to do your mission, I'm in and I'll make that happen. That's kind of what I've done for the whole of my career, and chaos has always been involved there in some fashion. Yeah, it's been a pretty cool run.Jason: So, you're currently doing this at a company called Kessel Run. Tell us a little bit more about Kessel Run.Omar: So, we deliver combat capability that can sense or respond to conflict in any domain, anywhere, any time. Or deliver award-winning software that our warfighters love. So, Kessel Run's kind of… you might think of it as a software factory within the DOD. So, the whole creation of Kessel Run is to deliver quickly, fast. If you follow the news, you know DOD follows waterfall a little bit.So, the whole creation of Kessel Run was to change that model. And that's what we do. We deliver continuously non-stop. Our users give us feedback and within hours, they got it. So, that's the nature behind Kessel Run. It's like a hybrid acquisition model within the government.Jason: So, I'm curious then, I mean, you obviously aren't responsible for the company naming, but I'm sure many of our listeners being Star Wars fans are like, “Oh, that sounds familiar.” Omar: Yep, yep.Jason: If you haven't checked out Kessel Run's website, you should go do that; they have a really cool logo. I'm guessing that relates to just the story of Kessel Run being like, doing it really fast and having that velocity, and so bringing that to the DOD, is that the connection?Omar: Actually, it goes into the smuggling DevSecOps into the DOD, so the 12 parsecs. So, that's where it comes from. So, we are smuggling that DevSecOps into the DOD; we're changing that model. So, that's where it comes from.Jason: I love that idea of we're going to take this thing and smuggle it in, and that rebellious nature. I think that dovetails nicely into the work that you've been doing with chaos engineering. And I'm curious, how did you get into chaos engineering? Where did you get your start?Omar: I've been breaking things forever. So, part of that they deliver tools that our warfighters can use, that's been my jam. So, I've been doing, you can say, chaos forever. I used to walk around, unplug power cables, network cables, turn down [WAN 00:03:24]. Yeah, that was it.Because we used to build these tools and they're like, “Oh, I wonder if this happens.” “All right, let's test it out. Why not?” Pull the cable and everybody would scream and say, “What are you doing?” It was like, “We figured it out.”But yeah, I've been following chaos engineering for a while, ever since Netflix started doing it and Chaos Monkey came out and whatnot, so that's been something that's always been on my mind. It's like, “Ah, this would be cool to bring into the DOD.” And Kessel Run just made that happen. Kessel Run, the way we build tools, our distributed system was like, “Yep, this is the prime time to bring chaos into the DOD.” And Kessel Run just adopted it.I tossed the idea, I was like, “Hey, we should bring chaos into Kessel Run.” And we slowly started ramping up, and we build a team for it; team is called Bowcaster. So, we follow the breaking stuff. And that's it. So, we've matured, and we've deployed and, of course, we've learned on how to deploy chaos in our different environments. And I mean, yeah, it's been a cool run.Jason: Yeah, I'm curious. You mentioned starting off simply, and that's always what we recommend to people to do. Tell us a little bit more about that. What were some of the tests that you ran then, and then maybe how have they matured, and what have you moved into?Omar: So, our first couple of tests were very simple. Hey, we're going to test a database failover, and it was really manual at that point. We would literally go in and turn off Database A and see what happened. So, it was very basic, very manual work. We used to record them so we can show them off like, “Hey, check this out. This is what we did.”So, from there, we matured. We got a little bit more complex. We eventually got to the point where we were actually corrupting databases in production and seeing what happens. You should have seen everybody's faces when we proposed that. So, from there, we're running basically, we call it ‘Chaos Plus' in Kessel Run.So, we've taken chaos engineering, the concept of chaos engineering, right, breaking things on purpose, but we've added performance engineering on top of it, and we've added cybersecurity testing on top of it. So, we can run a degraded system, and at the same time say, “All right, so we're going to ramp up and see what a million users does to our app while it's fully degraded.” And then we would bring in our cyber team and say, “All right, our system is degraded. See if you can find a vulnerability in it.” So, we've kind of evolved.And I call it, put chaos on a little bit of steroids here. But we call it Chaos Plus; that's our thing. We've recently added fuzzing while we're doing chaos. So, now we got performance chaos, our cyber team, and we're fuzzing the systems. So, I'm just going to keep going until somebody screams at me and says, “Omar, that's too much.” But that's essentially a little bit of our ride in Kessel Run.Jason: That's amazing. I love that idea of we're going to do this test, and then we're going to see what else can happen. One of the things that I've been chatting with a bunch of folks recently about is this idea, we always talk about, especially in the resilience engineering space, that failure never has a single point. It's not a singular root cause; it's always contributing factors. And the problem is, when you're doing chaos engineering, you're usually testing one thing.And then it's like, “Oh, I did the failover on that database and that worked.” I've been suggesting that people now start to do, “Well, if this is in a degraded state, what are the contributing factors—if that's still working, what are the contributing factors that can lead to a major catastrophe?” That's one of the nice things that actually performing these failures allows you to do rather than just imagining them and trying to work up some sort of response process to your imagination.Omar: That's our thing. So, from our perspective, that's what I charge the team to do is like, “Hey, we need to make sure these things are working.” Comes back to my passion, right? Were delivering tools to the warfighters; the warfighter needs to have tools that work. And that's what Kessel Run does; that's what Kessel Run exists for.We deliver that award-winning software that our airmen love. So, following that trend, that's where chaos comes in place. So, we're building fancy tools, and we got an awesome platform that supports it and all that stuff. We're just there to make sure, “Hey yeah, this is engineered correctly. It's responsive to fault or any kind of failure.” And we just—I mean, we're literally blasting it with anything we can imagine to make sure it could support that.Jason: I'm curious if you could dive into some details about one of your recent chaos engineering experiments. Was anything unusual or unexpected? And what did you learn from it?Omar: So, I think one of the cool ones, which is the latest one, was that database corruption. There was a lot of questions on, “Hey, we have some tools in place we built. The engineering is in place to make sure that if the database goes down, nothing is impacting our system and whatnot. What would happen if the database gets corrupted?” For some odd reason. I don't know, that's probably going to happen once in a million, I don't know.But it's like, “Hey, let's figure it out.” So, my team came up with an experiment; we went and we started corrupting databases in staging. It's like, “All right yeah, that was cool.” Oh, and then we went to the leader, she was like, “Hey, we want to do this in production and call an outage and see how the teams responds.” And at the same time, we're going to throw a whole bunch of curves.We're going to disappear key people, we're going to make sure you don't have access to certain things. It was not just database corruption; we're going to throw curveballs at you like there's no tomorrow here. So, we did, and it was actually a pretty good experience. So, we figured out, hey, yeah, the database corruption just happens, whatnot and the team like our SRE team actually figured out. It took them a little bit because it was a lot of curveballs, but we learned, all right, if this does happen and we have all these issues happening at once, it's probably a non-realistic—I'd call it—fire drill, but it's something we got to prepare for just in case.We've learned from it and we actually practiced it again. So, from the initial time it took us to go through the curveballs, we did another one, threw different curveballs at them, and that was like a no-brainer. They're like, “Yep. We got this. Don't worry about it. We ran this through once, so we know.”Which is why we do these things. You want to practice and then, if there's an outage, shorten the time, make sure it's not impacting. What was really cool to see is, like, it didn't matter how many databases we corrupted and how many curveballs we threw at the system, there was never an impact to the end-user, which is the goal. We practice chaos to make sure that it's always working. So, we validated that our system can tolerate all these curveballs and all these things we were doing at it. And it's something that we've never tried before, so it was pretty cool.Jason: I love that you mentioned what you threw at people was maybe not realistic, it's not something that would happen in the real world, but I think it brings up that idea of when you're training for things, if you train harder, if you're an athlete and you train harder than you wouldn't normally in a game, and you're constantly stressing yourself when it comes to that real-world situation, it just seems easy.Omar: Yeah. And that's what the SRE team—because we do the normal, “Hey, we did the test,” and then we go, [it's like 00:10:31], “This is what we saw.” And then we actually asked for feedback from the team's. It's like, “Any way we could have done this test better?” The normal process.And they're like, “We loved this. We've learned so much that helps us either automate more scripts or streamline our process.” So, from our standpoint, we'll keep throwing curveballs. And I think they did that, aside from, hey, this is a very realistic scenario, and then we go to the—this is probably a little bit over the edge, but we still want to do it. We do both. It's good.Plus, it doesn't keep a same [unintelligible 00:11:04]. We're used to it. All of a sudden you're throwing all these curveballs at the team, they can nitpick from all these lessons learned and put better processes in place, make it faster, better engineering. The team's awesome. All the team that supports Kessel Run, our SRE team, our platform team, everybody's super smart, super amazing, and I'm just there to test their ability to respond. Which is why I like my job.Jason: You mentioned lessons learned, and I'm curious, as somebody who's been doing chaos engineering for quite a long time, actually, what are some of the top lessons that you would give, or the top advice you would give to our listeners as they start to do chaos engineering?Omar: I would say. So, you start simple, and that's key. You start simple. If you really mention chaos to somebody who's not familiar, the first thing they're going to do is they're going to Google ‘chaos engineering,' and what they're going to find out is Netflix and Chaos Monkey. That's an awesome tool, but do your research, figure out what other people are doing, and get involved in the chaos community world; there's a lot of people doing some cool stuff.Start with a small test so you can see and get the data from there, and scale up. As you learn and as you go, you scale up. And it helps—chaos scares, sometimes—or not, sometimes. For the most part—your senior leadership because you're telling them, “Hey, I'm going to come in and break stuff.” So, doing small-scale tests allows you to prove and provide, hey, this is why it's beneficial.The actual event is not chaos. We call it chaos engineering, but the actual event is very controlled. We know what we're doing, we're watching, we have somebody in place to say stop in case things are going haywire. So, you have to explain that while you're doing. And just do it; it's just like testing, you have to test your applications, and the more testing you do, the better.The closer you shift left the better, too but you have to test. You got to make sure your apps are working. So, chaos engineering is just another flavor to that. The word chaos usually scares people. So, you just got to slowly do it and show them the value of doing chaos.Hey, you're doing chaos, this is what it brings. Hey, we just proved your database can failover. That's a good thing. And if it didn't fail over, it's like, how can we make it happen? So, that's a small-scale test that provides that feedback and data you need to say, this is why we have to adopt chaos engineering.And as you going, get—do—go crazy, right? As your leadership allows you to do stuff like, yeah, let's just do it. And work with your teams. Work with the SRE teams, work with the app teams and get feedback. What do you need?What is your biggest problem? That's one thing I ask my team to do. So, every month, they go to the team and say, “All right, so what's your biggest hurdle? What right now is your—why don't you sleep?” And we go, “Okay, can we replicate ‘the why don't you sleep' so we can let you sleep?”So, that's an approach that's worked for us. And a whole bunch of our tests are based on that. It's like, okay, “What keeps you up at night?” We'll test it so you can sleep. And then next month, give me the next thing that keeps you up at night. And we go in and we test it.Jason: And like that iterative approach of let's work on, what's your biggest pain point? What keeps you up at night? And then let's solve that. And then what's the next thing? And keep working down that chain until, hopefully, nothing keeps you up at night.Omar: Yeah, that'll be good. We all sleep and it's like, “Oh, this thing's on cruise control. Let's go.” Jason: You mentioned convincing management or the upper levels of management in allowing you to do this. What's that process like at Kessel Run? And then, what's that process look like as Kessel Run convinces the broader Department of Defense to adopt this?Omar: Oh, that's a fun one. Yeah, so we when we first brought it up, we got the, “What are you trying to do?” Look—because it was like, “Hey, we want to do chaos engineering.” It was like, “Okay, yeah, we've heard a little bit about this. What does that mean?”It's like, “I'm just going to break stuff.” Which probably wasn't the smart approach at the moment, but that's what I said. And they're like, “No, wait. What do you mean?” And I'm like, “Yeah, and eventually I want to do it in production.”So, I just went all out. That was my presentation. You know, I've learned from that. It's like, okay, baby steps, Omar. But initially, it was like, “I want to do chaos and I want to get to production.” They were like, “Yeah, sounds good, but I need a plan.” I was like, “Okay. I'll come up with a plan. And we'll figure it out.”And so that's how we slowly started. And I stood up the team, Bowcaster, and from there we kind of, all right, how do we show the value of chaos engineering? How do we learn chaos and all that stuff? So, it was easy to get them to adopt it. It was the actual execution of tests that was a concern.Because there was a lot of unknowns. We didn't know what we're going to break. We don't know how it's going to react. And how do we actually do this? And we slowly just kind of did those little tests. It was like, all right, we're going to do this, we're going to do that. And that's how we got it.And now that we're moving to the rest of the DOD, that's a really cool adventure because our framework, what Bowcaster has built in Kessel Run, is what they want to move to the rest of the DOD. So, the Chaos Plus model is what's interesting. The fact that we are moving to the rest of the DOD is very cool because it's something I believe should be in the rest of the DOD. And we're happy to experiment. From the Kessel Run perspective, that's what we're here for.We'll experiment and we'll let you know what fails what doesn't fail because we're an experimental lab. And, yeah. But the senior leadership in DOD in charge with all the software development and stuff like that, they're all over it. They just want to—hey, how do we make it happen? What do you need?You'll see there's a different mind change now that chaos engineering is more familiar around the DOD and the tech space. “Hey, yeah. This thing called chaos engineering.” It's not just, yeah, Netflix does chaos engineering. It's like, yeah, everybody's doing chaos engineering.So, you see the little mind shift from, initially, when I bought it in. It was like, “Hey, I want to break stuff in production.” And everybody's like, “Whoa, hold up there. There's [no 00:17:05] baby steps here, Omar.” Now, it's like, “Hey, let's go and do it.” Is like, “Yeah, let's do it. How do we execute? But let's do it.” So, it's a very cool thing to see.Jason: I'm wondering if maybe that readiness to adopt things like this since you've spent time in the military—I haven't, but from what I understand, it sounds like the military has ideas of really, really doing testing. And in some cases, not production testing. We don't start wars just to train the military, but there is the idea of things like live-fire testing. Do existing practices within the military influence the perception of chaos engineering, and to help people actually understand it better, maybe more so than with standard civilians and corporate enterprise?Omar: Yes. Testing is very important in our systems. So, it's a different mindset, I would say. So, because in corporate world, it's all about the money, making the system work and make sure it's not going down because you lose profit. Or if you're—that's the mindset on that one.For us, we are in charge of defending the nation, so our system has to be proven and ready to rock within seconds. So, we do a lot of tests, and chaos engineering is just one extra layer to those tests. And now that we are moving to this massive DevSecOps transformation, chaos engineering is key. There's no way we can do this without having chaos engineering involved. So, that's what our senior leadership is pushing.Hey, yeah, this is another flavor of testing. It's important because we're building distributed complex systems across the cloud and whatnot, to support the DOD mission. So, chaos engineering is there. Same thing with the live-fire testing. We got to do live-fire testing to make sure that the ammunition is working, and the guns are working, and everything's working right. This is just a different flavor of live fire testing, just on software, and applications, and infrastructure, and the whole deal.Jason: You mentioned running game days and throwing curveballs, and that sounds like more of a manual game day where you've got people running the attacks and people responding. You've mentioned Kessel Run and really that velocity, and getting faster at things, and automating. Have you started automating the chaos engineering process as well?Omar: So, we have and we're following the same approach as when we started. So, the baby steps approach. So, we are going to slowly work with the SRE team to automate some of these tests. And that's ongoing. My team's working on it right now, so we're getting there.It's part of our slowly learning and kind of process. The manual, like, game days won't stop. Those will keep going because of the curveballs we want to keep throwing at the teams, but the automations is coming. The idea is to get the chaos engineering closer to the dev cycle as we can, so shift left as much as possible. And that's our next goal.So, we're working on that. And I think a lot of it comes down to where do we do it. So, we work in different environments. It's not just what we call the internet right now. We have different environments, so how do we automate across all environments?And part of it is how are we architect that so it works. So, if we make it work on one environment, how does it work on all the environments? So, that's usually where our timelines are. So, trying to make sure that our architecture supports all environments versus having to spend a lot of resources, you know, all right, we're going to engineer one environment, we've got to engineer another environment, we've got to engineer another environment. We want to make sure just to—out of the box, here we go. But that is part of our goal, and we are starting baby steps, so the database failover test is probably the one we will automate first.Jason: As you've done chaos engineering, you're doing the game days manually; what was the process like in terms of tools and adoption? I think a lot of people start off and they hear of Chaos Monkey and so they immediately jump over and, “Cool, let me grab Chaos Monkey and see if I can use that.” For any listeners that have tried that you've probably have quickly recognized that that tool, not so great for public consumption, was very much designed for Netflix. So, I'm curious if you could tell me more about your tools adoption, what have you used? What are you using now? What does that evolution look like?Omar: Yeah, so we actually—the first thing I told my team was you are going to research tools. [laugh]. I know Chaos Monkey is out there, but I'm like, there's definitely more tools that we should look at. I'm sure there's been a whole bunch of tools created, depending on our platform. And that's what they did.So, they went and they researched a whole bunch of tools. And they came back and they presented the tools they wanted to use, or kind of just integrate into our architecture. When the team started, right, so when we started that chaos team, the Bowcaster team was supposed to focus just on chaos engineering, but the more I kept thinking about it, it was like we need to focus on chaos and some other stuff. So, that's where the performance engineering and the fuzzing came in plays, and bringing the cyber team into the game. So, from a tool perspective, when you look at us, Bowcaster the team is also the tool.So, they have a tool, Bowcaster is the tool that we deploy across KR to do chaos engineering. Now, within that tool or that framework, there's the tools behind it. And there's a combination of open-source tools and other tools that we do there, but those just provide the engine for us to perform all of our tests on what we call Chaos Plus. So, Bowcaster is our tool. Yeah, it's the team and the tool is kind of weird, right?But the team and the tool, so when you go into KR and you say, “Hey, I want to chaos engineering.” It's like, “All right. Go do chaos engineering with the Bowcaster tool that the Bowcaster team built.” But the architecture behind that, there's a lot of tools. And it was that—that was the task I gave the team.It's like, “I need you to research tools. I know, Chaos Monkey is out there. I know Simian Army, I know all these tools that originally come out when you Google.” It's like, if Netflix created it, that's the first thing that comes up. But there has to be more, especially in the Kubernetes world. There's a whole bunch of tools. So, that's what they did, and we took a combination of those tools and we built Bowcaster. And that's what we got.Jason: That's an excellent point, though, about not just a chaos engineering tool. And I think a lot of times when people think of chaos engineering because it's chaos engineering it sounds like this well-defined practice of, this is it. If you have chaos engineering, you must have chaos engineers, and so it seems siloed when in actuality, it's just one of many practices that SREs and DevOps and all engineers should practice. So, this idea of, we're going to build a tool that has not just the chaos engineering, but all of these other things that you need, and providing that as a service is, I think, a fantastic idea.Omar: That's always been the charter I've given the teams. Yes, we want to do chaos engineering; chaos engineering is awesome. We all dig it, we preach it, we're huge advocates of it, but what else can we provide? I mean, we're already degrading the system, so what else can we test? [unintelligible 00:24:25] break the system and blast it with a million users and see what happens. And it's like, “All right, systems degraded; we're blasting it. Let's see if we can hack it.”And maybe while that's degraded and getting blasted, maybe we figure out there's a vulnerability or something. So, that's always been the concept. It's like putting chaos engineering a little bit on steroids, we call it. And that's what Bowcaster does. Bowcaster's job is to build these things and support it.And I'm sure we'll come up with other crazy stuff as we get feedback from team, like, “Hey, it would be cool if you can do this.” And we'll just build it into our framework and it will just be another service that Bowcaster provides aside from performance and chaos engineering.Jason: Omar, thanks for coming on the show. Fantastic information. It's inspiring to see the journey of where you've come from and where you're headed, especially with the Bowcaster team at Kessel Run. Before we go, though, I wanted to ask, do you have anything that you want to plug or promote, job openings, upcoming speaking? Where can people find you on the internet to learn more about the stuff you've been doing?Omar: So, Kessel Run, very active, so you can find us at LinkedIn: Kessel Run, or just go to our site, kesselrun.af.mil and you'll find a whole bunch of information there, careers, so if you're interested come work, we're cool people. I promise we do cool stuff.And if you come work for Bowcaster, we'll hire you and you can break stuff with us, which is why we—can't get better than that, right? Yeah, come check us out, kesselrun.af.mil. Lots of information there, careers, you can follow us and yeah.Jason: Awesome. Thanks again for coming on the show.Omar: Thanks.Jason: For links to all the information mentioned, visit our website at gremlin.com/podcast. If you liked this episode, subscribe to the Break Things on Purpose podcast on Spotify, Apple Podcasts, or your favorite podcast platform. Our theme song is called, “Battle of Pogs” by Komiku, and it's available on loyaltyfreakmusic.com.

In this episode, we cover: Intro and an Anecdote: 00:00:27 Early Days of Chaos Engineering: 00:04:13 Moving to the Cloud and Important Lessons: 00:07:22 Always Learning and Teaching: 00:11:15 Figuring Out Chaos: 00:16:30 Advice: 00:20:24 Links: Apex: https://www.apexclearing.com LinkedIn: https://www.linkedin.com/in/mdcsaenz TranscriptJason: Welcome to the Break Things on Purpose podcast, a show about chaos engineering and operating reliable systems. In this episode, Ana Medina is joined by Carmen Saenz, a senior DevOps engineer at Apex Clearing Corporation. Carmen shares her thoughts on what cloud-native engineers can learn from our on-prem past, how she learned to do DevOps work, and what reliable IT systems look like in higher education.Ana: Hey, everyone. We have a new podcast today, we have an amazing guest; we have Carmen Saenz joining us. Carmen, do you want to tell us a little bit about yourself, a quick intro?Carmen: Sure. I am Carmen Saenz. I live in Chicago, Illinois, born and raised on the south side. I am currently a senior DevOps engineer at Apex and I have been in high-frequency trading for 11 out of 12 years.Ana: DevOps engineers, those are definitely the type of work that we love diving in on, making sure that we're keeping those systems up-to-date. But that really brings me into one of the questions we love asking about. We know that in technology, we sometimes are fighting fires, making sure our engineers can deploy quickly and keep collaboration around. What is one incident that you've encountered that has marked your career? What exactly happened that led up to it, and how is it that your team went ahead and discovered the issue?Carmen: One of the incidents that happened to us was, it was around—close to the beginning of the teens [over 00:01:23] 2008, 2009, and I was working at a high-frequency trading firm in which we had an XML configuration that needed to be deployed to all the machines that are on-prem at the time—this was before cloud—that needed to connect to the exchanges where we can trade. And one of the things that we had to do is that we had to add specific configurations in order for us to keep track of our trade position. One of the things that happened was, certain machines get a certain configuration, other machines get another configuration. That configuration wasn't added for some machines, and so when it was deployed, we realized that they were able to connect to the exchange and they were starting to trade right away. Luckily, someone noticed from external system that we weren't getting the positions updates.So, then we had to bring down all these on-prem machines by sending out a bash script to hit all these specific machines to kill the connection to the exchange. Luckily, it was just the beginning of the day and it wasn't so crazy, so we were able to kill them within that minute timeframe before it went crazy. We realized that one of the big issues that we had was, one, we didn't have a configuration management system in order to check to make sure that the configurations we needed were there. The second thing that we were missing is a second pair of eyes. We need someone to actually look at the configuration, PR it, and then push it.And once it's pushed, then we should have had a third person as we were going through the deployment system to make sure that this was the new change that needed to be in place. So, we didn't have the measures in place in order for us to actually make sure that these configurations were correct. And it was chaos because you can lose money because you're down when the trading was starting in the day. And it was just a simple mistake of not knowing these machines needed a specific configuration. So, it was kind of intense, those five minutes. [laugh].Ana: [laugh]. So, amazing that y'all were able to catch it so quickly because the first thing that comes to mind, as you said, before the cloud—on-prem—and it's like, do we start needing to making ‘BC', like, ‘Before Cloud' times when we talk about incidents? Because I think we do. When we look at the world that we live in now in a more cloud-native space, you tell someone about this incident, they're going to look at us and say, “What do you mean? I have containers that manage all my config management. Everything's going to roll out.”Or, “I have observability that's going to make us be resilient to this so that we detect it earlier.” So, with something like chaos engineering, if something like this was to happen in an on-prem type of data center, is there something that chaos engineering could have done to help prepare y'all or to avoid a situation like this?Carmen: Yeah. One of the things that I believe—the chaos engineering, for what it's worth, I didn't actually know what chaos engineering was till 2012, and the specific thing that you mentioned is actually what they were testing. We had a test system, so we had all these on-prem machines and different co-locations in the country. And we would take some of our test systems—not the production because that was money-based but our test systems that were on simulated exchanges—and what would we do to test to make sure our code was up-to-date is we actually had a Chaos Monkey to break the configuration.We actually had a Chaos Monkey and it would just pick a random function to run that day. It would be either send a bad config to a machine or bring down a machine by disconnecting its connection, doing a networking change in the middle to see how we would react. [unintelligible 00:05:01] with any machine in our simulation. And then we had to see how it was going to react with the changes that was happening, we had to deduce, we had to figure out how to roll it back. And those are the things that we didn't have at the time. In 2012—this was another company I was working for in high-frequency trading—and they implemented chaos engineering in that simulation, specifically for them, we would catch these problems before we hit production. So yeah, that's definitely was needed.Ana: That's super awesome that a failure encountered four years prior to your next company, you ended up realizing, wait, if this company actually follows what they do have of let's roll out a bad deploy; how does our system actually engage with it? That's such an amazing learning experience. Is there anything more recent that you've done in chaos engineering you'd want to share about?Carmen: Actually, since I've just started at this company a couple of months ago, I haven't—thankfully—run into anything, so a lot of my stories are more like war stories from the PC days. So.Ana: Do you usually work now, mostly on-prem systems or do you find yourself in hybrid environments or cloud type of environments?Carmen: Recently, in the last three to four years I spent in cloud-only. I rarely have to encounter on-prem nowadays. But coming from an on-prem world to a cloud world, it was completely different. And I feel with the tools that we have now we have a lot of built-in checks and balances in which even with us trying to manually delete a node in our cluster, we can see our systems auto-heal because cloud engineering tries to attempt to take care of that for us, or with, you know, infrastructure as code, we're able to redeploy at will. So, with the cloud infrastructure, a lot of what would cause me anxiety and give me more white hairs is slightly less than [unintelligible 00:06:51].Ana: I love the way of putting it the less amount of white hairs is because of cloud. So, thank you all, cloud providers. As this comes to mind and we think about your background of coming in from on-prem systems, is there anything that you've encountered in this cloud world that you think that's a gotcha? Like, I've had an incident in bare metal that cloud is not really necessarily having a use case or reliability mechanism built-in, just out of the box.Carmen: It's easy to catch, but it's a gotcha at the same time. So, when you come from on-prem into Cloud, the networking is… not all the same. The words are there from networking, like ‘gateway,' and ‘firewall,' click a few buttons, as opposed to you running [Arista 00:07:38] commands versus on a router. [laugh]. And then you have your VPC, which you can say that's your little world and your internal network.The words are there, but they're different in cloud, and that's the got me part of that transition. But at the same time, you have an easier way to visualize those things. For example, if my machine can't connect to another machine, are they in the same subnet? I don't have to run Arista commands to figure that out, or look at the logs on the router; it's literally right there in front of me. So, a lot of that pain that we would have to going—you know, switching from—going to your Linux machine to then getting into the router and then running these different commands, I feel like you needed to learn more commands and more different types of languages of the things that you were using in order to interact with, as opposed to now in the cloud, I feel that those things are more blatantly in front of you to fix.And they were a little bit more abstract in on-prem and that's why you would need someone like a network engineer more as opposed to a DevOps engineer who—I feel like it's easier in that sense. So, once you know it, you're able to solve those problems that you would need a networking engineer for.Ana: I guess now when we look at the DevOps site reliability engineers have this cloud world or this hybrid world, you end up wearing a lot of hats and you end up having to master, to an extent, various levels of networking, or knowing at least the operator side of how a lot of our infrastructure is running. It does bring me to the next question where you get a chance to come from that BC world—Before Cloud—and we now see that a lot of DevOps SREs that are joining in, they come into the magic of the cloud. What do you think is one of the things that the engineers that are just getting started and are just touching the cloud are not getting a chance to dive into, that the cloud abstraction layer really misses out on this amazing fundamental of the work that we do.Carmen: I think it goes down to the nitty-gritty. With DevOps, you wear many hats. You're good at everything, but not a master of one thing. You're a little bit everything [unintelligible 00:09:49]. Before cloud, even though the term DevOps didn't exist and you were called, like, an operations developer, operations engineer, you worked closely with the people who wore that one hat and you were working with them.And now that people are coming into the cloud only with no on-prem, they get the layers abstracted on what is the three-way handshake in networking? What is indexes really used for in databases? How do you know that you're not using—doing a linear search because your indices are incorrect in your database, versus doing an algorithmic search with that specific algorithm, that specific query language is using. Those are things that are so abstracted but are still very necessary because you may have to work with an on-prem system that connects to your cloud infrastructure and you may need to use Wireshark.Who still uses that? But you do. There's systems, older mainframe systems that mainly finance uses, or there's still COBOL systems out there. So, I feel that's what's missing from being in cloud. But I hope that education and other programs like Courseras and Lindas that if people feel like they're lacking in something, they're able to go and learn those fundamentals somewhere.Ana: I know you love learning. Two things come to mind. Do you have any resources where DevOps and SREs can start learning more? And do you want to share with our listeners a little bit more about your passion and your path in learning?Carmen: Sure. So, a lot of the things that people look at are common things like Udemy. I feel that Udemy has a lot of great DevOps courses, believe it or not. I have used them to study, I've used them for refreshers. I came in with Amazon cloud experience but no Google Cloud experience, so I basically took a Udemy to get my feet wet, as you would say, to get into that world.Linda is also good. If you have your student ID email, you can get it for free. So, [laugh] as a student, now, I use that. And then there's just various resources. A good thing, also, like, finding groups like Techqueria DevOps, as well as Latinas in Tech, and TECHNOLOchicas that if you join groups where you meet other people who are starting in that space or have been in that space for a long time, they have the resources as well. But those are the resources.Ana: Do you want to share a little bit more about your path on learning about DevOps and what you're up to now?Carmen: With DevOps, since I'm passionate in learning—because in DevOps, you have to always keep learning—as I was going through my education, and even now being in industry, is that I don't know that many Latinas, especially back in the 2000s. What I noticed when I was in school is also that the majority of my teachers were men, they went to Harvard and MIT, and their great schools, majority of them, were [unintelligible 00:12:35] and I never had a Latina teacher or any of that. And I said to myself that I wanted to be that teacher that I didn't have. So, I started teaching part-time at my alma mater, at Loyola University. And I loved it.And I loved—I taught, like, data structures in [C+ 00:12:55] and Java. I've taught DevOps classes, I've taught bash scripting, I've taught open-source computing, intro to object-oriented programming. And I just loved engaging with students. I've noticed that I knew I was missing something and then I realized it was teaching, being that difference, being that change, being the face that I didn't have. And I figured, what's the best place than my alma mater to start that at?As I was doing this for my fifth year, I realized that if I love it so much, I should do something about it. So, I decided to get a Ph.D. So, 10 years later [laugh], I went back to the school, and I'm currently in my third year at DePaul University in Chicago as a Ph.D. student, working in the American Sign Language Lab Avatar Project, creating an avatar to do not just American Sign Language but other sign languages—yes, there are many different ones. And that's where I'm currently at now because I would love to teach again somewhere, full time, be it after industry or maybe at the same time I'm still in industry. Who knows what the path is. I love teaching, and I love helping, and I love engaging, and I love technology, so that's why I wanted to go back to school and become a teacher, at one point.Ana: So, amazing to see your passions and your background come together into that mission of pushing forward the industry and bringing more representation to it.Carmen: Definitely. Thanks. It's still a [ride 00:14:19]. I don't know what the outcome will be, but [unintelligible 00:14:22] so I just hope that I pass my second exam for my Ph.D. that's coming up in September. [laugh].Ana: I wish you a lot of luck, and I'm sure our listeners are also rooting for you. Are we going to be seeing Dr. Carmen Saenz that's going to be teaching DevOps, teaching [unintelligible 00:14:39], teaching chaos engineering, or would you stick to something more in ASL?Carmen: I think a little bit of both. I think that the experience that I bring as a DevOps engineer is that some of these systems don't exist. Some of the stuff that I was doing that I brought to the lab was, they already have the programmers, but they don't have—they're running on, like, a Windows machine in an internal network at school. So, how can we make this widely available? One of my posters that I did my second year was specifically in engineering, and architecting, and infrastructure that can handle creating high visualizations with, you know, a GPU graphics card, but also being scalable and then it has to be [GDP 00:15:22] compliant because we work with European schools.So, there is DevOps in my Ph.D.; it's just a little hidden. But I'm hoping that I continue to bring that to the table in my lab and the work that I have won't just be on an internal network in three years. I hope in three years, you'll be able to—you all—can connect to it to be like, “That's the work her lab did, and we all know that the reason why it's visually seen by everybody was because Carmen was the one that created the infrastructure for it to work, with a group of other engineers.” So, I'm hoping that I can bring my two loves together in that sense, in three years' time.Ana: I mean, I think you're already doing it. It's super amazing to just even hear when you get those learning stories of someone is an engineer in the industry, has 10-plus years, and then they go and they help assist them that is not tied to our technology space, that is not running on the cloud, that they're running on just one Windows Server and they're hoping to reach, what, 3000 people a month; like, how is that possibly even going to scale for them to be successful? So, I think even you just going into the lab and putting in some of those DevOps principles. And I love also what you mentioned, you know like, how do you make it be a highly scalable system when you're running with so much GPUs, and you have all this different types of compliance in it? Which I think is always interesting when we talk about certain industries, whether it's healthcare or finance, that it's like, yes, we need to have compliancy based on data that we store, but then there's certain regulations and government that might tell us we also might need to have a certain uptime or you might be breaching this type of service-level-agreement.Carmen: So, a lot of the things I'm used to is more internal in the sense of we need to keep our logs X amount of years, and we need to know who logs into what machines. And so a lot of the compliance is pretty standard across the board for a lot of internal networks. But for something as big as this project for the Ph.D. that I'm doing on, our group has to make sure that [GDR 00:17:27] compliance is very different. And what PIA, what data do we have and how can we abstract it or break it down enough where then it won't actually go back to a person? And those are the things that I feel that I personally don't really have to deal with right now at work, but I have to deal with them at school.So, there is a trade-off, something that I'm lacking at my position at work, I actually have to think about, working with different countries, to have this software and the things that they're lacking like now having a scalable uptime system that people can communicate with, that is something that I do here at work; I'm trading off in both places. And compliance is very difficult to [ticket 00:18:10]. That's chaos engineering, too, because you're going to have to hire someone, or third-party company, or yourself have to literally attack your own system and see what you're missing and make sure you're compliant. And I think that's the beauty of—also—chaos engineering, trying to figure that out and making sure [laugh] that you're good, you know?Ana: For these highly visual systems that you have in your Ph.D. program, what are some of the unknowns that you've had to encountered as you're working on them since they're not very similar to the stuff that you work on your day-to-day?Carmen: I actually had to backtrack to my on-prem experience to a point. Unfortunately, the code was written in the late '90s and it's still [unintelligible 00:18:54] like that now, some of it. And it uses an executable; it had to be built by my professor, they gave me the EXE, I had to put it on the machine. And one of the things is, in Amazon, they have your GPU systems, right, and you could say, “I want this server with this graphics card and AMD and so forth.” One of the things that a lot of people don't recall if they never worked on on-prem systems is that drivers are problematic.And as I was trying to run this executable, I kept getting this error and I was like, “This has to be a driver issue, but how do we troubleshoot a driver on a cloud system that is pre-built for you?” So, [laugh] I was trying to figure it out? I'm like, “Is it the executable and how it was built on whatever machine? Or is it the machine that's in the cloud? And if it is, how do I update the driver? How do I downgrade the driver?”And so I had to Google how to downgrade drivers in VMs in the cloud. There's specific commands that you have to run that are AWS only. You don't have the manageability that you had when it's your own on-Prem system. Like, you just know, you run a general AMD command or a general package installer for the driver. It's not the case, all the time, for cloud systems.You have to run a specific AWS command. Luckily, what I found out was my professor, I brought it up to him, and he's like, “Oh, I have this driver. You're using this driver. I need to do some magic on my end to build this executable and it should work on the driver for this VM.” And I was like, “Sure.” But I didn't know how to troubleshoot those things in the cloud. But I knew how to troubleshoot them from back in the day when it was my on-prem system so there—it's weird understanding that drivers are still an issue, you just didn't think so because they're so abstracted nowadays.Ana: It's always interesting to remember where the abstraction layers push us forward in so many ways, but that they always bring this kind of catch on the other side of it of, “Wait, no, now you actually don't get a chance to just drive over to your data center, switch out a certain type of resource. Oh, the cable is starting to look a little hot, maybe we should stretch it out.” We now assume that a lot of these things are being handled for us.Carmen: Yes.Ana: Do you have any advice on how do you maintain systems that you don't build, or how is it that you can hand over things better when you're working with systems that are maybe even BC, Before Cloud? I'm going to trademark this. [laugh].Carmen: You should trademark it because, seriously, that is such a great way to explain it. That was literally what you do when you started a new job, right? They're like, “There's this old system.” I asked if there's any documentation, they usually laugh and chuckle at you. And then [laugh] they give you some notes that tech that some person left for you to look at. “Do you have any infrastructure as code?” They also might slightly chuckle at you and just give you some version that's 15 versions behind, that if you try to [unintelligible 00:21:52], they'll tell you, you're missing 50 other things.So, you do have to work with what you've got. And you're the whole point of being a DevOps engineer is that you investigate, investigate. And you shouldn't be afraid to ask questions. And I think that's something I learned as I got older. I was always afraid to ask questions.And I always felt like people were going to judge the crap out of me because I was asking questions. But how are you going to understand the system that you didn't build, and try to get into the head of the person that did build it in order for you to make it better? And… that's okay to ask those questions. And you should get those notes, that tech, and that rando Terraform that only works for a quarter of the things that were built. And see what's missing and try to see if you can devise a plan of attack of how are you going to break this down for yourself.And then, there may not be no diagrams. So, I'm not telling you, use Creately or anything like that to diagram, but it's also good just to have a piece of paper and a pen and just start drawing some of that stuff out. And then, a lot of it also is okay, let's make a test. On Saturday, I'm going to bring this down—chaos engineering—and I'm going to see who yells about it. Who's going to care?Who's going to care if I break this. And that's how you know who are the stakeholders. Sometimes, that's what you need to do; you need to create a little chaos, to understand what your next steps are in order to get rid of all that technical debt to make your company and your product better. That's how you have to start, and then from there, you'll get more stakeholders that are going to care because you caused a little chaos, in order to bring the system up-to-date, that is not yours, that now is yours. [laugh].Ana: You actually touched upon something that I was telling someone about two weeks ago where it's that we have this mental model of what our system looks like, they gave us an architecture diagram because, you know, this was only built five years ago, but we now have the thought that all of this is perfect, and until you start unplugging things, you start doing some chaos engineering of what in this architecture diagram is actually correct? What is not? Do I really have a database in my high critical services, or do I not? And then you can kind of really start thinking about understanding your system and build it to be better.Carmen: And also, one of the things that is just because it says dev, don't assume it's dev. It might be prod. Just because it's called dev doesn't mean that it's dev. That is one of my biggest rules now that I've learned recently in the last three years. Because with cloud, it's a little bit different than on-premise: if that's the name, that's what it's going to stay and most likely it is what it is. But here, because we're iterating so quickly, we try to fail fast in order for us to learn from our mistakes and build our product, dev becomes prod. [laugh]. More so now than it did before, you know?Ana: It brings me to that portion you mentioned also earlier: always ask questions. Always poke holes at it. If someone tells us, “Oh, no, don't worry, nothing is running here on production,” take a deep dive and try to find out what are some of those services, or what are some of those dependencies that could be going on. I know from my time at Uber, it took forever for us to find out which of the 2000 microservices are needed to just take a trip on the cloud. And it was like, “Uh, we don't know. We know they're running on prod, they're running on dev, but what is needed for this service to actually happen?”Carmen: Exactly. Sometimes just getting on the machine. And if you have root—which you should—if you're a DevOps engineer, usually—look at the history and then look at the directories of who has a home directory. People don't realize the history can give you so much good nuggets about what's going on in the system. And those are the things that help you figure out, like you said at Uber, what's running on here, and who's using it, and what is the systemd daemon telling me? And like… I mean, right?Ana: And it's funny, you mentioned that of take a look at the history because that was actually one of the things that I've always done, like, reading post-mortems—Carmen: Yeah.Ana: Understanding history that's being run on systems, understanding past PRDs to try to get a better understanding. And a lot of it actually is because of that other point that you also touched upon, being afraid of ask questions. Like, similar to you, I've also been one of the only Latinas in the room, and I'm like, “I don't want to raise my hand in this class, or in this meeting. I don't want to be the person that has to ask.” But if I have ways of starting to do my own searching so I make a more informed question, that gave me confidence. So, that was one of the things that I was always doing. But now I tell people, “No. Just ask the questions. Don't spend those five hours trying to look at history because the person next to you might actually know the answer in just two minutes.”Carmen: Yeah, exactly. And I noticed that. Just asking that question was literally like, “Oh, it was because of X, Y, and Z.” “Okay, cool.” And then, now that I know that, at least when I look at the history, I have some background of why this was this way, and now I can just pull out what I really care about in the history, as opposed to saying, “Why is this happening in the [unintelligible 00:26:59] in the first place?”But it's sucks being the first person to ask that question. And especially if it's just, like, you and a bunch of dudes—which usually it was, and at the time I was usually the youngest, too. I was, like, 22. Up until now, obviously; now I'm one of the oldest, but at one point, I was the youngest. And also age was a thing, and being the only Latina in the room, and it—you know, and it's finance; it was scary. [laugh].Ana: [laugh]. That's the awesome part. We got to have folks like you, like me, organizations like TECHNOLOchicas and Techqueria that allow for us to create spaces that are going to say, “You're welcome here. Ask as many questions as you want. No question is going to be stupid.”Because we've all had to start somewhere. And maybe you do get a chance to have Carmen as your teacher and get to pick their brain on what DevOps is. For that, I think that's all the questions that I had. Do you have anything else you want to share with our listeners that you have upcoming for you? Or just any words of advice?Carmen: My advice is just keep trucking along. There's many Carmens, there's many Anas, there's many Jason's out there that are willing to help. And there's spaces now where we can ask those deep questions, like you said, like Techqueria, Latinas in Tech, TECHNOLOchicas, [unintelligible 00:28:17] Girls Can Code, Girls Who Code. There's so many places now where you can really dig in and find the community to uplift you and keep pushing you forward in your technology inquiries and your technology career path. So, stick with it, keep going.Ana: I love it. What are some ways that folks can get in touch with you?Carmen: You can go to my LinkedIn, and the LinkedIn will be the slash name, slash M-D-C-S-A-E-N-Z. Or you can find me under Carmen Saenz at Techqueria Slack, or in Latinas in Tech Slack.Ana: Awesome. Thank you so much, Carmen.Carmen: Thank you so much for having me. I had such a great time with both of you.Jason: For links to all the information mentioned, visit our website at gremlin.com/podcast. If you liked this episode, subscribe to the Break Things on Purpose podcast on Spotify, Apple Podcasts, or your favorite podcast platform. Our theme song is called, “Battle of Pogs” by Komiku, and it's available on loyaltyfreakmusic.com.

Improving Your Business Strategy With Jason Button   You might be wondering, how do you improve your business strategy? A few weeks ago, we spoke with Marty Lewis about how to prepare a business plan. If you've already got a business plan or you want to improve your business plan, this is going to be the episode for you. We've got Jason Button from JB Strategic, and he's going to be talking about improving your business plan.   Get more tips on how to improve your business strategy at dorksdelivered.com.au   What is meant by “business strategy”?   What does business strategy mean?   Jason: Let's sort of get into the nuts and bolts of it. If anyone hasn't listened in before, I really encourage anyone to jump in and listen to Marty Lewis's piece around preparing a business plan, which is used to start a business and direct operations. That's going to cover your “Who” and “What” within the business.   Jason: From the strategic side of things, a strategic plan will look at implementing and managing the strategic direction of your existing organisation or the business that you're in. Think of that as your “How” and “When”.   Know Your Why  There's this pretty cool book called Know Your Why about understanding why you're in business. It allows you to understand your “Why”, “What”, and “Who” to further build on your business strategies.   Jason: That's got to be the starting point every single time. You've got to start with the “Why”.   Jason: For anyone who owns or operates a business, it's really critical for you, as a business owner or operator, to understand what that “Why” proposition looks like to you. Looking at it from a customer's point of view, what is the “Why” for them? Starting with that is a really good grounding point for you to build out that “Who” and “What” of your business plan and then the “How” and “When” we get there from the strategic proposition.   Jason: The other things to take into consideration with your strategic plan is how to tie in things like your vision, mission, objectives and how you are going to achieve everything as well as the “Who” and “What” of the business plan.   Know Your Customers' “Why”   You brought up a good thing there: knowing your customers' “Why” as well as the business' “Why” and objectives. A lot of the times they don't line up. How do you make sure that from a business owner's perspective, your “Why” is solid but clear to your clients and you're not pulling any emotion into that?   Jason: The exercise that I'll generally go through with people in this position is you've got to turn off all the noise around you. Take the emotion away and then think of why you are in business. What do you want to ultimately achieve within that?   Jason: Once you've got that as a framework and then you start to look at it from a customer's point of view with whatever product or service offering that you've got, there has to be some sort of alignment. That's not to say that it's got to be exactly the same, but there's got to be some sort of an element that ties both of those things together for you to have a really strong value proposition, which you can live and breathe in your employees or contractors or whoever is working within your business or organisation.   Jason: From the customer's perspective, they start to understand what your value proposition is and how that ties into whatever the offering is and ticks that box for them.   Know Why People Choose You If you've already been in business for a little bit and maybe you had a napkin-type of business plan to start off with that you created when you're at the pub with a few mates, and now you've already got some clients, is it polite just to ask them, 'Why do you work with us?' or 'Why don't you work with someone else?' How do you pull in that data?    Jason: Generally, what we'd look at is getting feedback from your existing customer base and suppliers. Go and talk to your employees or contractors. You could go to friends and family just to get that sounding board and start to really shake that up.   Jason: If the feedback that you're receiving isn't lining up with your “Why”, there is something amiss at that point. There's got to be some realignment because the “Why” stitches everything together and where you want to head. If you can't get that alignment and what you're projecting out there, there's got to be some time and attention spent on that.   How Much Time to Spend on Improving Your Business Strategy  We read Know Your Why a few years ago, and then last year we revisited it and I looked deeper into why we are in business. It racked my brain, nearly turned me into turmoil. I didn't know what I was doing. I just dived straight into it and ended up coming back full circle on why we're in business. How much time is the right amount of time to be spending on this sort of stuff?   Jason: If you've genuinely got some concerns in your “Why”, I encourage anyone to take the time. When we're all stuck in the grind, Monday to Friday is not the time to do it. You need to get away from things and turn off those outside influences to really drill down on that.   Jason: Have some time on that and start to line it up starting with the feedback. There is absolutely no reason why you can't spend time on this during the different phases of your life. You've always got to get that alignment and revisit your “What” on a regular basis.   Revisit Your Why on a Regular Basis  Jason: You've also got to start to spend time looking at your “Why” on a regular basis. I work with clients and we re-evaluate our “Why” on an annual basis and then that starts to shape that strategic plan. We do it on a financial year basis, but that doesn't mean you can't do it on a calendar year basis or whatever suits you, whatever you're comfortable with. What's really important is to always bring that back to your “Why” while you can.   We have a boring book. I've got all my different business plans that I've written in this book, and every year, I review it and see where I'm at. It's also going back through and seeing what the idea and direction and vision of the business was for us in, say, 2007—when we started—and then how that changed in 2010. Who was our target client and who were our suppliers and why were we in business?   Having a look and going far out, we've changed. I just get this big hit of energy because I've actually done a lot. Sometimes you feel like you're just on the grind, you're spinning your wheels, and you think I'm not getting ahead as fast as I want to but reflecting and looking at that, you'll find that you've done a lot. Keeping the book is a very huge value-add for us.    What should be included in your business strategy? With a business strategy, should you be talking about how many staff or is that not important? Is it more important to talk about revenue or is it more important to reflect on the numbers or the emotion? How do you know that you're balancing things properly? What are the three basic business strategies that you should be having in a business plan?   The 4 Ps and Your Why  Jason: Whenever I'm looking at a strategic plan, I look at the 4 Ps: plans, people within the business (i.e. internal stakeholders), process (i.e. how we go about certain things), and the purchaser (i.e. customers). I love to just segregate the 4 Ps into four quadrants there, start with those areas and start to drill down. You might write your “Why”—the value proposition—right in the middle of that.    The People Jason: Start to look at what's the value of all your People. We start that with a survey to get some feedback and we might do a bit of an audit on the skills and experience. We look at career pathway mapping, the culture within the business, how engaged they are, what sort of capacity we've got. There are so many different things that we can look at with just our People.   The Plan Jason: When I say Plans, we also have to look at the timeframe. When do I want to achieve X, Y and Z? Taking it back to your book, I think it's really cool that you've got a reference point to go back to when you put your strategic hat on. Start to look at your plans at that snapshot in time. Then, look at any outside influence or any change to your “Why” and then update that to ensure that you're on the right track. That's sort of a lot of planning.   The Purchaser Jason: Jumping across to Purchaser, it's looking at your “Why” standpoint from your customer side of things. It's improving the experience and communicating the value proposition. You need to do some thinking on how you communicate that effectively to customers and making sure you've got the right customers.    The Process Jason: In terms of your Process, what are you doing in terms of capturing your training internally so that it can be replicated for new people coming into the business? I can come up to your “What” proposition really quickly and look at all documented actions.   Jason: I know you do a lot with business process automation so I'm keen to get your thoughts around that and what process you go through when you talk to a client.   I started the business off as a cowboy with no processes and everything was in my head. That story is pretty familiar to a lot of people. I was running a very profitable business, but it was just myself. I brought one extra person, and then if I shall use the term, he was the Robin to my Batman. He was fantastic, but he sadly had a stroke while we're working 80-hour weeks.   I thought I'd try to do my best to do the work that he was doing as well as the work that I was doing. There are only 168 hours in a week, so I don't have very much time to do the work he was doing and the work that I was doing. I ended up having to instigate the 80-20 rule, and get rid of a whole bunch of clients that we kind of didn't want to work with anyway.   It sounds terrible, but everyone has them. I now had a profitable business that I was able to manage myself. But the push came when I brought on the first employee after him. That was when I realised I really did have some better processes in place. It took me 6 months to have that person become profitable, so we started putting in fantastic processes internally to start off with and then we noticed a lot of businesses don't really have this sort of stuff.   What is automation? For us, what we call automation is a mix between software and delegation. It's about understanding the processes yourself that might be in your head, pulling them down onto pen and paper, and then looking at how things work to then make them work better.   If you're doing any level of repetition, we have a look at how long that takes and then we take that and look at how long it would take for us to automate that process. Then we make an assessment: is the squeeze worth the juice?   A good example would be I was learning about electronics when I was 12 years old. I wanted to automate my door—I could press a button on a remote control to open or unlock the door. I worked out how the pneumatics were going to work and thought this is going to be awesome. I showed my brother, who's 13 years older than me and an electronic engineer, and went through the whole thing.    He asked how long it's going to take to make. I said, probably 100 hours. He said, 'How many times could you have stood up out of your chair and just open up the door in 100 hours?' I said, 'That's fair. Many, many, many more times than I'll ever need to do it.' But it was the process to learn how to do it.   Don't automate something that doesn't need to be automated. Don't create a process or have a faceless process if there are people involved.   Profitability and Efficiency Jason: I really like that point and the early part resonated with me. This is going to seem a bit funny. Let's pretend that we're at a restaurant. If you watch MasterChef, there was this little trend called deconstructed everything. They deconstructed desserts and other food.    Jason: Everyone knows what a pavlova looks like, but then you get a deconstructed pavlova—may still taste nice, but you just haven't had the time to do it so you've done all the elements separately. That can often happen in business when we're talking about process. Just because we are making money, it doesn't mean it's the most efficient way to do it.   Jason: I'd like everyone to look at their business and what they're offering once it hits the purchaser or the customer. Am I serving up a whole pavlova dessert in its completion or is it a bit of a deconstructed base where I've got a lot of good little elements and they're still leaving you satisfied, but it's still not quite right?   That's a great way to look at it because a lot of businesses will have all the bits, but they might not be the best chef to put it together either. It's good to be engaging people like yourself to be able to gain visibility.   An Extra Pair of Eyes Helps I am the worst critic of our own business. In our head, we always know we shouldn't have done this or we should be doing that. A lot of the time, you can't see the forest for the trees and you need to have someone come in to help deconstruct some of the different processes to find out if there is a better way to do things.    You don't know what you don't know until you know it, you know? That resonates strongly with a lot of stuff that we do with businesses.    For instance, we went in and he said they've been doing something for years. I asked if they have any problems with it locking up and they do, but everyone just jumps out of the file and then they all—10 people—jump back. I asked how often it locks up, and he said, 'Twice a week.' He has to organise 10 people to jump out and then back in twice a week. He calls them up on the phone to make sure everyone jumps out of the file. It's 30 minutes that everyone's jumping out and waiting to use it twice a week; an hour for 10 people. That's ridiculous.   Jason: From an outsider's perspective, it's really difficult in the day-to-day grind to see some of this stuff. When it is pointed out to you as obvious as that, that's your profit margin. You might still be making money, but all that inefficiency could be either replaced by tightening up your training or introducing a technical solution is profit.    Jason: It's always good to zoom out of your business and spend some time looking at what your strategies for those areas are. Look at those 4 Ps—the people, the plan, the process and your purchasers—so you can get the most out of your business and yourself. You're a happier individual with it as well.   Step Out of Your Business to Work on Your Business I think stepping out of your business is one of the most intelligent things you can do in your business. We were talking about some of the different house renovations that I'm doing, and I made it very clear that I'm not a builder, but I find that your brain never turns off.    When you're sleeping, your brain is not turned off. You're always thinking about something—a business problem, a personal thing. Whatever happens, you're always thinking about it.    I think swapping skillsets sharpens your sword. It's not just diving into your emails and becoming monotonous and repetitious with what you're doing. Just by changing out to doing something else like a run in the morning, you'll find that thoughts just come to you.    Have a Record of Your Ideas Almost everyone experiences taking a shower and then the answer they've been looking for comes to them. Or you're asleep and then you write it down. There's actually a funny Jerry Seinfeld episode where he wrote down a funny joke when he was asleep and woke up the next morning didn't know what the hell he wrote.   I've got the Amazon Echo units. I'll think of something great and I'll say it, but I'll say it with such blubbery mess in the middle of the night. In the morning, I'd wonder what the hell I was talking about.   Jason: That's a really good point. If you have some way to capture these ideas, whether it is a notebook, a whiteboard or a Google sheet, and just go back and sort of reference it when you've got time. I'm hopeless when it comes to ideas and idea generation. It often happens when I'm driving all over the countryside in traffic. I use just an audio recorder, and I'll just talk out loud by myself in the car. I can sort of throw all sorts of crazy ideas out there and dedicate a couple of hours a week to working on the business, not working on the business operations.    Jason: I use that reference point, pull out that notepad or play that audio file. I go, 'The guy in the car that day made a really good point, and I could probably translate that into the business in some way, shape or form.' That's a little technique that I use, and it might help.   Thank you. That's fantastic because they don't always come at the best of times, but we have to be able to find these ideas usually when we're nearly running out of time.   What should NOT be in your business strategy? I want to just ask a couple more questions. People ask a lot: what is a good business strategy? I want to ask you, what are the things that you should avoid having in your business strategy? What is a bad business strategy? It helps to know what you shouldn't be spending too much time on or shouldn't be putting into it?   No Accountability Jason: A bad business strategy for me is one that is missing any sort of accountability.   Unrealistic Timeframe Jason: The other thing that springs to mind straightaway is timeframes that are unachievable. You've just got to understand and get a feel. Don't bite more than you can chew and make things too ambitious. Make it so that it is achievable and will push you out of your comfort zone. Let's just make sure that it's something tangible and achievable and we know the steps to get to the final point.    Jason: Don't get hung up on the design of your plan, the front cover and the pages, and the look of it. That's where we all sit and procrastinate. Rip the Band-Aid off and then you can come back tomorrow and start to put a bit of a sense to it. It's sometimes the best way to go about that.   I think you hit the nail on the head. Starting is always the hardest. At least, if you just put something on the paper, you can then start refining it and massaging it into something better. What I do is I look at the big audacious goal and then I do the timeline in reverse.    What is it you're looking to achieve? For instance, you want to earn $10 million in two months. What are the steps that you need to take before that gets to $10 million? If you're at $1 million now, then how are you going to get to $10 million?    Is it times 10 of your client base or are you going to change around the products that you're selling to clients? What is it you're doing to allow that process to come to fruition and then put steps on that? Is it going to be a marketing plan that needs to go towards promoting these new products and everything else? You can make it a more realistic timeframe.    Avoid saying what a lot of people starting off in business say: 'I can make triple of what the boss is paying me because he charges a triple of what he's paying me, so I can do all this and I'm going to be a millionaire and pay my house off in a year for getting all of the mechanics.'   Jason: I couldn't agree with you more on that point. Once we start to get the data down and write the figures down, such as insurances, you start to work out pretty quickly that the proposition with your boss wasn't looking too bad if everything's just about money. When you're earning just over $12 an hour, once you pay yourself, it might not be the best suit for you.   Jason: It's really important that you set your goals and then work backwards from that—what it's going to take, what the steps are, and then go back and look at those 4 Ps of how they're going to help you achieve that.   Suggested Read: Good to Great  What would you say is your favourite book or a book that covers business strategy and improving your business strategy?   Jason: My go-to is Good to Great by Jim Collins. I encourage everyone to go and have a look at any of Jim Collins' works, particularly Good to Great and Built to Last. I find them really inspiring. Every time I go back and look at it, I can go with a different context and framework, and I'll get something out of it every time.   Is that the type of book you can listen to as an audiobook?   Jason: Absolutely. It's quite long, but just try and break it down and get a really good understanding of the concept that they're promoting and then take that little bit of time, work it back within your business and then go on to the next one. A lot of great stuff there.    Jason: The reason why I love it so much is I'm a data guy. 'The numbers don't lie' is one of my favourite sayings. That book was created out of years and years of research by some of the best companies in America at the time and as far back as World War II and the trends they've got in common to make them great companies.   I actually haven't read it, so I'm going to check that on the side.    What is business freedom to you? The podcast is called Business Built Freedom. What is business freedom to you or what is freedom and why are you in business?   Jason: Freedom for me is always time. Time to be a better parent later and mentor to people. It affords me the freedom or the space to continue my learnings and absorb different concepts so that I can start to shape that in my own experiences and then share that with others to inspire them. That's why I do what I do.   Jason: I'm one of four children, and my father had a small business. He left before we got up and came home in the evening and we didn't get to spend a lot of time together. I appreciate the time, effort and energy that it takes to own and run a business. If I can come in and help a business, give them that bit of freedom back to be a better parent or a better individual while their business is sustainable and growing at the same time, happy days. Everyone wins.   If anyone wants to contact Jason from JB Strategic, jump across to his LinkedIn profile, check him out, and have a bit of a chat.    If anyone has any questions to put those up for the greater community, go to our Facebook group.    Jason: I'm more willing to share anything and speak out.    That's what it's all about. The world goes round on knowledge, and I think everyone should be sharing as much as possible and not holding their cards too close to their chests. We will grow and become better together.   If you have enjoyed this podcast, jump across to iTunes, leave us some love, give us some feedback. Stay good and stay healthy.   We specialise in improving processes to boost your businesses' profitability and efficiency. Call 07 3166 5465 to enquire!

The next time you play golf on company time make sure insights into how you spend your time on the job out in the field is not being tracked. Today’s guest is Barima Kwarteng from Timeero, a GPS app that reduces overhead costs and streamlines time tracking. Basically, Timeero started to prevent golf (at least on company time) because it’s expensive!  You’ll Learn... [01:52] Barima’s Background: Originally from Ghana in West Africa, but attended BYU. [02:35] Timeero: Some employees work in the field. How do they spend their time? [03:01] Drunk Coworker Confesses: On company time, they play golf on Fridays. [03:35] Software not from Scratch: Acquired cold base from Texas cop and rebuilt it. [05:00] Theft Prevention: Timeero tracks time, location, mileage, and much more. [06:10] Screenshot Safety: Pressure to prove work is being done to get paid. [06:58] Big Brother? Find out if employees are being accountable or taking advantage. [07:22] Bottom Line: Companies save 5-15% payroll costs by tracking employees’ time. [08:16] COVID Cash Crunch: Tighten belt to offload employees not pulling their weight. [10:00] Technology and Time Management: Paper timecards/sheets take too much time. [12:07] Timeero: How the Web application works after downloading app on smartphone. [15:06] Geofence: Reminder to clock in and clock out as soon as you arrive/leave work. [16:36] Better Culture within Business: Nobody likes to be micromanaged. Trust me. [20:49] More Money? Most are more motivated by recognition than monetary benefits. [22:05] Simplicity vs. Tech-Savvy: Get people to use new things or do things differently. [24:10] Why is Timeero different from other apps? Gives outside team more insights. [30:42] ROI: Timeero is $5 per user per month, plus $10 base fee for business account. Tweetables “If golf is helping the bottom line and helping you get revenue, great, but otherwise…” Jason Hull “People right off the bat think, ‘Oh, it has a little big brother feel to it. What we're seeing is, it's actually helping the bottom line in terms of payroll.” Barima Kwarteng “Technology saves you a lot of time. As a business owner, you can dedicate those hours or time to something more beneficial.” Barima Kwarteng “The app only tracks time and movement while you're clocked in. The moment you clock out, it stops tracking you because we really value privacy.” Barima Kwarteng “It helps people track time more efficiently and more accurately.” Barima Kwarteng Resources Timeero Brigham Young University (BYU) DoorGrow on YouTube DoorGrowClub DoorGrowLive Transcript Jason: Welcome, DoorGrow Hackers, to the DoorGrow Show. If you are a property management entrepreneur that wants to add doors, make a difference, increase revenue, help others, impact lives, and you are interested in growing your business and life, and you are open to doing things a bit differently, then you are a DoorGrow Hacker. DoorGrow Hackers love the opportunities, daily variety, unique challenges, and freedom that property management brings. Many in real estate think you’re crazy for doing it, you think they’re crazy for not because you realize that property management is the ultimate high-trust gateway to real estate deals, relationships, and residual income. At DoorGrow, we are on a mission to transform property management businesses and the owners. We want to transform the industry, eliminate the BS, build awareness, change the perception, expand the market, and help the best property management entrepreneurs win. I’m your host, property management growth expert, Jason Hull, the founder and CEO of DoorGrow. Now, let’s get into the show. My guest today, I'm hanging out with Barima Kwarteng. Did I say it right? Barima: I think you got that right more than most people. Jason: Okay, awesome. Barima has a company called Timeero. Before we get into that, I'm excited to hear a little bit about this. I was checking out your landing page for your software. It looks like it's kind of unique with the GPS stuff and some of those kinds of things. Before we get into that, why don't you introduce yourself to those listening or watching this later? Just help them understand what brought you to Timeero, and share a little bit about your entrepreneurial background here. Barima: Yeah, absolutely. I'm originally from Ghana, a country in West Africa. I'm not sure if many of you listeners know where that is. I originally came to school out here in the US at BYU. At BYU, I was able to learn a little bit more about entrepreneurship and starting businesses. It's something I've always wanted to be able to do. That opportunity presented itself with Timeero. The whole idea behind Timeero was that many businesses out there have employees out in the field. They have teams out in the field. They want to have a little bit more insight into how their employees or team members spend their time. Timeero provides you that opportunity to be able to have more insights into how your outsight spends their time. One of the events that really triggered this was I have a friend who owns a construction company. His employees, every Friday, they would bring their golf clubs and check it in their truck. On company time, they will play golf on Friday. The company didn't know about this. They only found out during a company party that they were playing golf on company time. One of the employees was drunk and finally spilled out the secret about what they were doing. He needed a solution to be able to track again and have insights into where his employees were spending their time. Working with him, that was how the whole idea was born. I initially did not build it from scratch. There was a company out in Texas that the owner of the company wanted to sell his software. I remember thinking to myself, you know what, this is what Brad wants. This is the kind of solution he needs. I actually ended up acquiring the initial cold base from this guy out in Texas. He was a cop. He just wanted to get rid of it and focus on something else. What we ended up doing was rebuilding the whole system again and making them much better. That's a little bit about how we got started on Timeero. Jason: Timeero started to prevent golf, basically. Barima: I have a problem with people playing golf. When you're doing it on company time or when you're playing when you're not supposed to be playing, it gets pretty expensive. We’re on company time. Jason: Yeah. If golf is helping the bottom line and helping you get revenue, great, but otherwise... Barima: If it's helping the bottom line, yeah. Exactly. Jason: Okay. Barima: Sometimes, it feels like it has a big brother feel to it, but you'll be surprised in many ways that people use it. It tracks time, location, mileage, and a whole lot of things. Jason: My company's virtual. One of the challenges I’ve noticed over the last decade or so that I've been running a virtual team is you sometimes get people that are not full of integrity. They come into the business, you think they're going to do a good job. If there isn't a level of accountability that needs to be in the business, people sometimes will take advantage of that. The challenge is that I've had people stealing time from me in the past. Eventually, you find out. Eventually, you figure out they're not getting stuff done and you're paying for that time. That's a challenge people deal with. It's theft in the business. It sounds like we're talking about preventing theft. One of the side effects I've noticed in having time tracking and time software for some of the team members, contractors, or people that I hire, some of these do screenshots, some of these do things like that. It's created safety for my team members or as contractors as well. One of the challenges a lot of contractors deal with is they have to prove that they're doing something and they feel that pressure. It could very easily be somebody that hired them to say I don't think you did anything, or I don't think you did enough. I don't want to pay you. There's also the issue of if there are interactions or dealings with customers, so having accurate records protects the customer, it protects the employer, the team member, and it protects entrepreneurs, the CEO, or the boss of the company. I think people forget that and they sometimes like to focus on it feels so big brother. I generally found that I only get that feedback from people that want to not have accountability. Barima: I think you hit the nail on the head. That's exactly what we're seeing as well. There are a whole lot of uses for it. Generally, people right off the bat think, oh, it has a little big brother feel to it. What we're seeing is it's actually helping the bottom line in terms of payroll as well. One of the things I haven't really mentioned is just with time theft, it's natural for us to have our times. I may come up to you and say I work from 8:00 AM-5:00 PM. The reality is I probably worked 8:13 AM until 4:42 PM. Five minutes, 10 minutes here and there start to add up all the time. It starts to affect the bottom line. We're seeing companies saving anywhere from 5%-15% payroll cost by just having a system in place to help them accurately track more time and also have more insights as to where the employees are spending their time. That's one of the few statements that we're giving businesses that you’d also have to [...]. Jason: I think one positive advantage that has come from the whole coronavirus, COVID-19 epidemic is in March, everybody sort of experienced the significant cash crunch—most businesses did—and they had to tighten their belt. What we saw is a lot of businesses laying people off, a lot of property managers were tightening the belt because they worried rent wasn't going to come in. That created this massive mass cleanse of wait, bloat, and fluff inside of companies. There was just this offloading of that. That makes people really conscious. I think people, more so now than ever, are really conscious of the fact that a 5%-15% reduction in payroll cost is significant because that's typically the largest expense in a business. That could be a significant reduction in expenses, which means you have a lot more cash flow health. If cash flow goes into the business, the business is dead. The second cash flow is gone, the business is done, unless you get loans or something to squeeze out and get a little free cash available. Cash flow is king in a business, and payroll is going to directly impact that. Barima: Right. I think you're spot-on on about that. What the coronavirus and everything going on, as a business owner, your natural instinct is to try to cut down costs. Like you mentioned, most people may be looking to lay off people. A lot of times, as business owners, we’re so focused on bringing in more business. How do I increase my revenue? It also makes sense to look at your bottom line and figure out, how do I cut down costs using technology? I think it's easily forgotten by a lot of team leaders or business owners. In figuring out how to cut down on costs using technology, you can dramatically cut down on costs and help improve your bottom line. In terms of time management, let's just focus on time management. There are a lot of companies out there who still use paper timesheets, they're having their employees to text in their time, or using some form of [...] system. What we're not even talking about is the amount of time that's involved in processing your payroll. First of all, your employees may not be turning in their timecards on time. Depending on how often you’re running payroll, you may be finding out two weeks later how many hours they may have spent on a job. They may not have the greatest handwriting and you're spending a lot of time trying to figure that out. Or you may have to open a call and try to get everyone to turn in their time card in a timely fashion. Today is Friday. The last thing you wanted to do is to spend your Friday afternoon or evening trying to chase people for their timecards. If you're using a system like Timeero, you don't have to sit around and wait for any of that. We're talking about cutting the amount of hours anywhere from 30 minutes to even 5 hours for some companies or more to minutes or even seconds. You can just pull up a report of how many hours have been worked, how much mileage, and have insights to where they've been. Technology saves you a lot of time. As a business owner, you can dedicate those hours or time to something more beneficial. It might be spending more time with your family or other areas of your business. It definitely does help save you a lot of time and money as well. Jason: Okay. We talked about some of the benefits of saving time. Let's talk a little bit about how it works. Say, a property management business owner has some boots on the ground, has an agent or a person at the field, a property manager, and they're doing showings. They're going to open up a property for somebody in getting a lease handle. They're moving around and doing stuff. But they're not sure if this person is going and just picking up the gates in school, hanging out and watching Netflix for an hour here. They want to verify this, how does this software work? Barima: The way it works is that you have your web application, which you can run on an iPad or a normal desktop machine. Your employees are going to have their smartphone. We have mobile apps that they can download on their smartphones. Once they have the app on their smartphone, they can log in using the credentials that they’ve been assigned. Whenever you're ready to start with it, let's say we'll start at 8:00 AM for me. I come in, I punch in, and it starts tracking my time, my location, and my mileage. Whenever I travel around it logs my positions. Let's say I clock in and decide to drive off to go play golf. Now Jason is going to come in and see that I'm at a golf course. That's probably where I'm not supposed to be—at a golf course. You're going to see a trail path of where every single employee has been while they're on the clock. Managers can review that and ask questions. Why were you at the golf course? Or were you supposed to be there in the first place? Jason: For the manager that's looking in the app—managing their staff that is tracking their time—they will see a map for this? Barima: They will see a map, exactly. There's a Google map. For every time entry, there's a Google map that shows your trail path of where you’ve been while you were on the clock. Again, the app only tracks time and movement while you're clocked in. The moment you clock out, it stops tracking you because we really value privacy. We're not interested in knowing what you were doing off the clock. It's only while you're in the clock that it tracks your trail path, where you've been, and any information that’s needed Jason: Great. If they have a maintenance person going around doing maintenance in some properties, they can say, hey, this tenant called me. They said you haven't shown up yet. Where are you at? Oh, yeah, I'm on my way. You check and they're at the golf course. I know that's not the case. It creates some accountability, which I think lessens the temptation for people to lie, steal, and do things. It also creates more accuracy because they know that their location is being entered in. Now, say somebody forgets to clock out. They go do an errand at the grocery store. They're like, oh, I forgot to do that. Are they able to edit their time entry to remove that portion and eliminate that pin from the map? Barima: Absolutely. You're able to go edit your time entries and change all of that. One of the things we're doing or some of the few things we've got is to make sure we can remind people to clock out as well. Not just clock in, but also to clock out. You can set a reminder. For instance, we use a technology called geofence. We can set a geofence around your work area. The moment you leave your work area, it notifies you to clock out because you probably may have forgotten to clock out, you're leaving off for lunch, or leaving off. The other thing we're coming out with is an automatic clock in and clock out. Whenever you arrive at the job site, it automatically clocks you in. When you leave, it also automatically clocks you out. These are optional settings you can turn on and off. Again, it helps people track time more efficiently and more accurately. Now, you don't have to worry about forgetting to clock in and out. It just handles that for you if you turn on those options. Jason: Yes. I'm a big fan of creating or implementing systems that can do the job of managing certain pieces rather than micromanaging your team. I love when information's pushed to me instead of me having to go and ask them, and get it, or find out, or that sort of thing is. You may want to know, as a boss, what were you [...]. What were you doing this time? If you go and ask them all the time, if they are doing the right thing, they're going to feel invaded. They're going to feel like you're not trusting them. Having a system like this, I can see the advantage in being able to check and say okay. It can alleviate those fears you have in the back of your mind. Oh, they were on the job site. They were doing this job at this property. They were in the office this time while I was on vacation when I checked. It just allows you to lower that pressure, those noises, those fears, and those doubts. And it allows you to facilitate greater trust in your team. Barima: Absolutely. Speaking of micromanaging, I'm not a micromanager at all. I don't like environments where people feel micromanaged. Sometimes using the software may come off as micromanaging. One of the things we do is we like to train our users to have a better culture within your businesses, within your companies. Foster an environment where people feel trusted. And also, using tools like this to foster more accountability. You can have a great accountability system without micromanaging people. Sometimes, people don't think those two things go together. You can have good technology in place, make sure people are held accountable, but then also avoid micromanaging people. Jason: Yeah. This is an ironic thing in business or some people think these things don't go together. But really, when you create really good accountability systems inside your business, first of all, it's going to prevent the hiders or the people that aren't real believers in you or in your business from being able to get away with theft, stealing time, or being lazy. These are the team members we sometimes have in business that just want to leave for the weekend, complain about their boss, and they hate their job. None of us want those people on our team. Ultimately, we want people that are believers, people that buy into our vision, that want to enact the change that our business' mission is focused on. Those are the people that we want working for us—people that are inspired rather than controlled. What I found is that those types of people love accountability systems because they get recognized. They want recognition. But when you're hiding, when you're trying to steal, when you're trying to do as little work as possible, and just get a paycheck, you don't want accountability or recognition. You just want to fly below the radar. For my team members, we have a weekly commitment meeting. We show up and everybody says what they got done that week based on what they committed to doing in the previous week and what they didn't. My team members love being able to say, I got these things done. Everybody can see it. Great job you got these things done. Nobody wants to be the person that has a bunch of things that are in the red that they didn't accomplish, that are nos on the to-do list and want to feel like the weakest link. It creates a performance culture in which there's positive peer pressure. I could see how some people would resent this system and some people would respond well to it. Ultimately, as entrepreneurs, we want those that respond well to accountability and they love the recognition. The big mistake we make a lot as entrepreneurs is we think that they just want more money. We think they want bonuses. That's going to incentivize their behavior. Most people, besides entrepreneurs and salespeople, are more motivated by recognition than they are by increasing monetary benefits. Barima: I totally agree with that. It's very easy to think money motivates. Not everyone is solely motivated by money. Money helps but there are so many other ways to motivate people. It comes down to figuring out what each person on your team values. Recognition is one of them. It really, really, helps to help people feel recognized. It helps them feel that a good job is very important. There are many ways to motivate people outside of money. I've seen it go both ways where money motivates people. Actually, I've also seen it not motivate people. You just never know. It's up to you, as the leader, to figure out what motivates each and every single person. Also, like you've mentioned, there are some people who just are not ready to work. They will be the ones complaining about accountability systems, and they will be trying to figure out ways to beat the system. With those people who fall in that bucket, you have to forego different ways to work with them. Jason: Now, when it comes to technology, the biggest challenge with the team is adoption, which means getting them to use it. This is usually the biggest challenge with software is getting people to use a new thing or do things in a new way. What's your experience with people being onboarded into Timeero and getting their team members to use this? Barima: It's been on a wide spectrum. Overall, one of the big reasons why people go with us is because of simplicity. We put a lot of effort into making sure that our software is very easy and very simple to use. We know that not everyone is very tech-savvy. We really invest a lot into the design. Simplicity is always at the very top of what really shapes the design. When it comes to onboarding, it's very simple. We try to make the process extremely simple so that with a few clicks or a few taps you’re in the system and you're starting to log hours without very much help. We also invest in a lot of our customer support. You can get on the phone with us and talk with us. You can get on live chat. We’re happy to do a video call and help anyone get onboarded. There are few players out in the space. In software in general, it's a well-known fact that it's not that easy to get support when you need it. Sometimes, people will put you through an email system and you may hear back from them several days after. With us, we really invested a lot in our customer support, so you can get on the phone with us in the live chat. I can tell you, Jason, once you put out your request, you can expect to hear from us within 10 minutes. That's it if you're within our core hours of the business. We really take that. It's a high priority for us in giving people the support they need. Jason: You mentioned that there's a lot of other apps in the time tracking space. Customer service and being intuitive are two areas in which you stand out. What are some other features or benefits we haven't touched on yet? Or is there anything else that leans people to using your software to do this rather than whatever they're comfortable already or that they're using currently? Barima: When it comes to time tracking, time tracking has been around for a long time. You have companies that have been doing this for over 30 years or probably even longer. It's a very saturated space. There are so many time tracking apps out there. Some of them are free and some of them are paid as well. Where we separate ourselves from the gazillion apps out there we are mostly focused on helping you give your outside team more insights, having more insights into how your outside team is spending their time. We have a lot of property management companies that use us. Also, we help you track mileage. There are few apps out there. I can only probably count two apps I know that probably do that—track your mileage and track your time. Many companies will want to track your mileage. Perhaps you reimburse employees for mileage driven as well. They may probably be paying also for a separate time tracking app with GPS tracking capabilities. What we've done is we’ve matched those two. You don't even have to pay for two separate apps. You get time tracking, your mileage tracking. You're also getting scheduling and your host of other things we want to come up with like expense tracking as well. We've been able to mash all of these different technologies. It's very appealing to a lot of property management companies, businesses, and even property managers with one or two employees will use us. It just saves them a ton of time. As a business owner, you're focused on trying to grow your business. The last thing you want to be doing is trying to chase paper timecards, figuring out mileage, and whatnot. With technology, you're able to do all of these very, very quickly, and much more easily. Your time can be focused on what's supposed to be important. There are several things we do: time tracking, mileage, GPS tracking, scheduling, and a few other things. Again, with our software, you have a lot more insight into how your outside team is spending their time. Jason: Awesome, that's great. There's a lot of property management business owners that have property managers that are spending some time in the office, some time out in the field. They'll be able to assess whether they're out, in, or not. They'll be able to see how much time they’re spending at the office. They'll be able to see that they are going to the properties at the times they're supposed to. I'm sure every property management business owner has heard, I showed up for the showing and nobody was there. Usually, the assumption is this tenant is totally out there. But it may be the case that you have team members that are not really measuring up, being fully honest, or doing the things that you need them to do. That level of accountability is going to keep them at that high level. Barima: Right, exactly. You probably end up getting a bad review from a customer that says, They never showed up. As a business owner, you can go to their system and say, Where was John at 3:00 PM on this day? It will pull up that system and tell you. Jason: Yeah. If they were there, then that could be your comment on that review online. We use GPS technology to track. That's a selling point. That's a feature. We use GPS technology to track each of our team members that are out in the field, and this team member showed up at 2:58 PM—two minutes before and was there. We have this verified. You have facts and data. This will protect you legally in certain legal disputes. You'll be able to verify the things where people were in certain places when they were supposed to be. If you were supposed to be there to deal with a constable, an eviction, or a policeman. Any of these things, there'll be a record that somebody was there at that time stated and you'll have history. If you end up in a courtroom, some sort of a challenge, or a discrepancy with an owner or a tenant, you've got verification. You've got validation from a third party showing that there is a GPS time tracking being done. Barima: Absolutely. Jason, it's very interesting with my experience just talking with different business owners. What I'm seeing is the businesses that are investing in technology are the ones starting to get ahead. Over time, they'll start to get ahead because it just gives you such a huge advantage. It's tremendous the amount of advantage that technology just gives you. Sometimes, you have some business owners that just want to stick with the old way of doing things because that's what they're used to and it works. Yes, it might work, but your competition is just getting ahead, miles ahead of you just because you're investing in the right technology. The ROI that you're getting from using technology is so much bigger than what you're paying for. Jason: Yeah. You're saying that this can save people on average between 5%-15% in payroll? Barima: Exactly. 5%-15% of payroll. That's just the payroll side of it. Now, there's the human. We're not even talking about your time spent trying to do payroll and capture people's times. We haven't also talked about the errors that happen from getting the payroll log. The last thing you want to do is overcompensating people or under-compensating them. Again, you use technology that takes care of that. You don't have to spend your time worrying about, did I end up overpaying this person or did I underpay them? We're talking about tremendous time-saving costs as well. Jason: Okay. I'm looking at your website. The pricing looks really affordable. Let's just do some quick math for property managers here. Say, you've got one team member. They’re maybe between $15-$20 an hour. They're full time. You've got one team member full time. That could be anywhere from $3000-$4000 a month that they're spending on this person. 5% of $3000 is $150. Your software is not going to cost them $150 a month. Not even close. Even if it just helps them a little bit, it sounds like it can easily pay for itself right out the gate just by reducing a little bit of extra fat that people are padding accidentally on to their timesheets. Barima: Absolutely. It pays for itself right out of the gate. Again, looking at the ROI, you were paying this amount of money and you're saving this amount of money by using our system. It pays itself off right away. In a lot of ways, you can say it's a no-brainer to use that. Jason: All right. Is there anything else we're missing? Barima: Our pricing is very simple. It's $5 per user per month. It doesn't mean it's going to stay that way. In the future, we'll eventually introduce extra pricing tiers as we add more technology as well. We do have a $10 base fee for your accounts. The $10 base fee just covers your whole company account. Let's say you have 10 employees in your company. You're going to pay 10 times your $5, so that puts you up at $50. Then, there's the $10 account base fee that you pay every month. In total, you're paying $60 a month to manage your 10 employees—whereabouts, the time entries, the mileage, the scheduling, and a whole bunch of things—only for $60 a month. Like you mentioned, you can look at how much you're saving in payroll costs by using that. Purely just for payroll cost, you can— Jason: If it only helps them save 1%, even just a single percentage in staffing cost, it would be a no-brainer. Barima: Yes. It is a no-brainer, but you'll also be surprised not everyone finds it to be a no-brainer. You get it, but not everyone might get it that way. Yeah, it's very interesting. Jason: Very cool. Barima, it's been great having you on the show. We haven't had anybody on the show yet talk about anything like reducing or tracking time with GPS out in the field. I think this is a common issue that property managers run into, or it's a blind spot they just have they've just not been paying attention to. Something like this would probably create a little bit more safety and certainty for them. It sounds like it would also lower some of the effort, pressure, noise, work involved with timesheets, payroll, and dealing with people that are out in the field or in-house vendors. People that they have maintenance, people inhouse, property managers that they have out in the field, people doing showings. I think there are a lot of benefits here for property managers. I appreciate you coming on the show and sharing this with us. How can people find out more about Timeero? Get in touch with you? Plug your stuff. Barima: Yeah. You can visit our website at timeero.com. Visit our website. Use our live chat. You can also call into our office. If we’re in, we will answer it. If we're not, we'll respond back to you. Please do visit our website and use our live chat. Get on with one of our support agents or a customer sales agent who will get in touch with you and help you solve problems. We're here to solve problems. Get on and talk to us. We'll figure out if we'll be a good fit for you or not. If we're not, we’re happy to recommend any other solutions that might be a better fit for you as well. We look forward to hearing from you. Jason: Awesome, Barima. Thanks for coming on the DoorGrow Show. Barima: Thank you. Thanks for having me on, Jason. Jason: All right. We'll wrap this up. If you are a property management entrepreneur that wants to add doors, you want to make a difference, all the things we talked about in the beginning, in the intro, be sure to reach out to DoorGrow. We've been having great success helping clients, coaching clients, helping them clean up their business, clean up their branding, clean up everything that's preventing them from getting the deals, and the leads that they really believe they should or could be getting. The reality is that SEO won't save you. Can SEO help your business? Absolutely, but there is not a lot of search volume for property management for this industry. The best deals and leads are snatched up before they start searching on Google. You need a game plan, you need a system in place that you can grow your business, and not be waiting and relying on having the top spot on Google in order to grow. We want to help you get there. We want to help the best property manager succeed. Reach out to DoorGrow. Check us out at doorgrow.com. And be sure to check out Timeero. You can check it out at timeero.com. Until next time, everybody. To our mutual growth. Bye, everyone.

I sat down with world-renowned vegan chef and author Jason Wyrick who has co-authored a NY Times Bestseller "21 Day Weight Loss Kickstart" as well as the book "Powerfoods for the Brain" with Dr. Neal Barnard, MD. Other books he has written are "Vegan Tacos" and "Vegan Mexico". He was the food editor for "Living the Farm Sanctuary Life" with Gene Baur and Gene Stone. He's a coauthor of "Clean Protein" with Kathy Freston and Bruce Friedrich. Jason has published the world's first vegan food magazine, The Vegan Culinary Experience which is now defunct and has been featured in the NY Times, the LA Times, VegNews, and Vegetarian Times. He has traveled the world teaching cooking classes and is the first vegan instructor to teach in the prestigious Le Cordon Bleu program. We talk about being vegan, health benefits, dairy, cheese, his home delivery service of amazing vegan food called The Vegan Taste and his restaurant Casa Terra. Jason gives us such a great insight of his progression of eating like most of the population to becoming a vegetarian and finally a full out vegan. It was such an honor for me, to have such a celebrated chef and author on my show. Because I've eaten his food, this conversation had so much more of a meaning due to my various attempts of being vegan myself. I hope you enjoy this conversation and the knowledge Jason shares with us all from his heart. Jason Wyrick: Vegan Food Delivery Service: The Vegan Taste Vegan Restaurant: Casa Terra Co-authored a NY Times Bestseller: "21-Day Weight Loss Kickstart" and "Powerfoods for the Brain" with Dr. Neal Barnard, MD. Other books he has written are "Vegan Tacos" and "Vegan Mexico"He was the food editor for "Living the Farm Sanctuary Life" with Gene Baur and Gene Stone. He's a coauthor of "Clean Protein" with Kathy Freston and Bruce Friedrich. Connect with Jason: YouTube: https://www.youtube.com/user/thevegantaste/videos Facebook: https://www.facebook.com/jason.wyrick.5 Instagram: https://www.instagram.com/casaterrarestaurant Twitter: https://twitter.com/VeganChefJason https://youtu.be/6jzSCBvX7PA ********** Podcast Music By: Andy Galore, Album: "Out and About", Song: "Chicken & Scotch" 2014 Andy's Links: http://andygalore.com/ https://www.facebook.com/andygalorebass ********** If you enjoy the podcast, would you please consider leaving a short review on Apple Podcasts/iTunes? It takes less than 60 seconds, and it really makes a difference in helping to convince hard-to-get guests. For show notes and past guests, please visit: https://joecostelloglobal.com/#thejoecostelloshow Subscribe, Rate & Review:I would love if you could subscribe to the podcast and leave an honest rating & review. This will encourage other people to listen and allow us to grow as a community. The bigger we get as a community, the bigger the impact we can have on the world. Sign up for Joe's email newsletter at: https://joecostelloglobal.com/#signup For transcripts of episodes, go to https://joecostelloglobal.com/#thejoecostelloshow Follow Joe: Twitter: https://twitter.com/jcostelloglobal Instagram: https://www.instagram.com/jcostelloglobal/ Facebook: https://www.facebook.com/jcostelloglobal/ YouTube: https://www.youtube.com/channel/UCUZsrJsf8-1dS6ddAa9Sr1Q?view_as=subscriber Transcript Jason Wyrick: Joe: All right, welcome, Chef Jason Wyrick, this has been a long time coming for me. I have looked forward to interviewing you the moment I tasted the food that was delivered to my house. So here we are and I'm so excited to have you on the podcast and I really appreciate the time and you actually saying yes to me, so thank you so much and welcome! Jason: Well, you're welcome, I appreciate you having me on here. Joe: Yeah man this is a, the way this came about for me was I got a flyer in the mail and it was one of those things like come to this free, healthy dinner to hear some, I don't know, some sort of talk about healthy eating and nutrition. And it happened to be from a nutritionist, a company in town, like an office in town. And I went and then I, I got pulled into it, you know. The food we had was great, but it wasn't necessarily vegan, it was just healthy. But then when I got into the program, which was not cheap by the way, but I felt I was worth it. They started to say, you know, do all this blood work and then we found out my, I knew my cholesterol is always a little high. So their program is doing vegan for 30 days on their menu. And then from there, you, you know, you the hope is you stay with it or you alter it a little bit or whatever, so that's how I got into this. And the problem for me was I literally was so busy I did not have the time to prep my food. It was taking me like half days on Saturdays, half days on Sundays. And I was like, my weekend is shot and I've prepped all this food and, and I, you know, any small amount of time I had was gone. So then I really went on the hunt for trying to find healthy vegan food that I could just literally eat and not do anything with. I had already done, I think I did Sun Basket a while back. You know, all the food prep things that you know Jason: Right. Joe: of and we talk about. So that's how you and I got connected. I, I don't even know how I ended up finding you. I say it was just purely, I was so desperate doing a Google search and I found you and I was like, SOLD! You mean I can just heat it and eat it, right? That's that's your thing, it's just heart and eat. So here we are. So I want to start from wherever you want to start. I know that this was a health thing for you in combination of other things. But knowing the stories that I've read and interviews I've seen of you, that this came about more for a health reason initially for you. And then it just blew up from there and and it became your passion, which is really cool to me, because this is what I preach on this show and on my videos, is that I want people to live or fulfilled lives doing what they love. And it's cool that you went into that direction knowing some of your past, which you can talk about om how this all started for you. So take Jason: I'm Joe: It away! Jason: Sure it was a kind of a winding journey, I think I mean, it, it seems kind of straightforward when you look at it. I was unhealthy, I went vegan, I got my health back. Hurray! But that's, that's really not how it started, I mean. It's starts when I'm a little kid because, I think I didn't eat great, but I didn't eat bad for the kind of regular American diet. Which meant, you know, my mom cooked some of the meals and occasionally ordered out and I played sports all the time, I was always active. So I was a super healthy rail thin kid. And then as I got older, towards the end of high school and in college, I kept eating the same way I had been eating the last few years and last few years had changed because my mom went to work, she got busier and so our food choices changed to, "What, which one of these seven different chicken dishes do you want tonight that I know how to make? or would you like Taco Bell or Burger King or Pizza Hut or something like that?" So when I stopped playing sports all the time and was super active, the calorie and taken and honestly, like the terrible food I was eating, started to catch up with me. And so I, I probably put on 30 pounds from when I was 16 to probably 19 and just kept going up about 10 pounds a year from there. Jason: So I was already getting overweight. And then right at the end to college, I started learning how to cook. So I went to, I went to this really great Egyptian restaurant in Fort Worth where I went to college, had the ah this amazing meal with the first amazing meal I'd ever had. And I was like, "I want to learn how to eat like this!" And I'm broke because I'm in college. So I started to learn how to cook for myself. And then right after that, it was like two months after that, I went vegetarian and that was solely for ethical reasons. No real idea of the health impact or anything like that, that it has. I didn't care at the time, I was just going to keep eating food that was super tasty and not worry about the health part. So, of course, even going vegetarian, a couple gaining weight. In fact, I was kind of a stupid vegetarian, I'll just be blunt about it. I took the meat I was eating and I replaced it with blocks of cheese. So instead of these instead of like these super fatty steak fajitas loaded with sour cream and cheese that I was eating before. Now I was eating cheese lover's pizza from Pizza Hut and the additional topping was extra cheese. Exactly! [laughter] Joe: [laughter] Jason: And that was that was my dinner. I was with someone at the time, she had her own pizza. It was it was terrible. And so I became incredibly overweight. I weighed about 330 pounds and I got type two diabetes by the time I was in my mid 20s. And I was, I was faced with having to take insulin for the rest of my life and in basically starting to deteriorate even more. Like I was already deteriorating, my eyesight sucked, sleeping 10 to 12 hours a day. Everything you can think of with Type two diabetes was going wrong with me. So I was facing having to take medication and deteriorate for the rest of my life, which was probably not going to be that long at this point or changed my diet. And so it's, it's funny because I was, I've been vegetarian for five years and I had, I had heard of vegans, but I didn't really know what they were. And I even made fun of it a little bit.[laughter] Joe: Right. Right. Jason: This was back in the late 90s. And then all of a sudden it's 2001 and I'm faced with having to make this choice, do I do I give up this food that I love, which is cheese, and live a better life or just keep going with the cheese and and it's funny because even though it it sounds like a no brainer, like eat cheese and die or give up cheese and regain your health. I mean, it sounds like an obvious choice, but there is so much there's so much pain involved in a lifestyle change, that the stress of that was really bad in itself and, and going vegan in 2001 when really no one else around me was, was vegan. It meant I had to learn how to cook, I had to learn how to fend for myself, I had to completely change all these foods that I knew how to make and eat when I was growing up. And so it was super stressful at first. And so I relaxed a little bit and decided I was going to give myself a cheat day. So I was going to be a cheating vegan once a week. So every Wednesday night I'd go out and I get all you can eat enchiladas at my favorite Mexican restaurant and they bring them out in pairs they'll bring you two enchiladas at a time. And the first time I went in there, the waiter was like, "OK, yeah, whatever, it cool! He brings out enchiladas, except I eat 14 of them. Joe: Oh, my gosh. Jason: And then they come back the next week and all of a sudden the waiter's like, "Hmmmmm" because I need another 14 enchiladas. So by the third week, the waiters like "I hate you but I have to serve you anyway." Joe: You're like the, you're like that all you can eat buffet, crab, Jason: Right. [laughter] Joe: Leg guy. [laughter] Jason: It's it's probably familial in some way because I know my, my little brother would go to a Mongolian stir fry places and he take the bowl and see how much he could pack in the bowl because it was one pass through. And so he'd, he'd have the regular bowl and it only come up like three inches and then there was like the six inch pile of stuff on top Joe: Oh, Jason: Of the. [laughter] Joe: My gosh. It's. Jason: So there must be something familial about that, that buffet all you can eat thing. I, so I, but anyway, the point is, I, I did that for a few months and even then I managed to start losing weight and my symptoms went away. So I'd be vegan for the entire week, except for this one, one rather egregious cheat meal but it was still just one meal. And then it went to once every other week when I would go to this place. And then once a month. And then I remember the last time I purposely had went to this place in order cheese that I order in the enchiladas and I, it was a weird experience because I looked at them and I realized they didn't taste good to me anymore. They didn't have that, that feeling you get when you cheese that Homer Simpson like, "dooonnuuttt" like when you eat dairy, so I didn't have that anymore. They didn't taste good and I realized I was ordering them out of habit and not because I actually wanted them. So I didn't even eat the enchiladas, I pushed them away, paid the waiter, who probably sighed relief Joe: Right. Jason: that I was getting had their there and that was the last time I ever stepped foot in that place. And at that point, I was a full on vegan, which took me about eight months. And it also coincided with me completely getting rid of diabetes. Jason: And Joe: Incredible! Jason: After the first year, I dropped about 60 pounds and then when I added in some real exercise, I dropped another 60, so I dropped about 120 pounds over two years. Joe: That's incredible. And I think Jason: Yeah. Joe: What people need to understand about you, you're a big guy. Like I know Jason: Yeah. Joe: from the interviews and stuff, 6' 3", right? Yeah, I mean, that's you know, and and I think at one point you said you, you went to school and lived in San Antonio...Fort Worth, sorry. So you're like in steak town. Jason: Yeah, I mean, Joe: Right. Jason: The nickname of Fort Worth is Cowtown. Joe: Yeah, ok, so there you go! Yeah, so that must, the be, that must be hard. It's just the stigmatism with, you know, vegan and yoga and all of those kind Jason: Ok. Joe: Of things. Right. It's tough. Jason: It depends. OK, it was weird because Texas is really interesting. I mean, I grew up here in Arizona but my dad is Texan. And so I was already pretty familiar with Texas before I actually moved there for school and stayed there afterwards. And Texas has this reputation of being big and boisterous and rednecky and it is. But it also has has this huge liberal side and has this huge health side, has this huge vegan side to it. I mean, I remember when I was in college, I went to the Texas Vegetarian Chili Cookoff. And this was in the mid 90s and it was like this huge gathering of people from all over Texas doing this Chili Cookoff. Like Texas had one of the biggest vegetarian societies in the 90s, at least when I was there participating in that stuff. And so Texas is just this really cool mix of all these different things, religion and Atheism and big hair money and rebel activists and steak eaters and vegans and no one is quiet about it. Maybe that's the one thing about Texans is, you know, everybody kind of gets by in the big city but they're, they're friendly but boisterous about that stuff, which makes it really cool. Anyway, that's my tangent on Texas. Joe: No, but that's great, because it's exactly you, you saying that is exactly how it educates people to know that it's not just big hats and boisterous voices and steak and whatever, it's, I had no idea that you would think that long ago people were vegan in the state of Texas. Jason: I mean, I think, I think Fort Worth had one of the first vegan restaurants in the country, which was Spiral Diner that opened up in 2001. Joe: Yes, I don't think anybody would ever know that. So that's, that's cool. So the tangent was great. OK, so you are, this is what year now that you go full vegan? Jason: So that was the, I started the beginning in 2001 and then I was full vegan by the end of 2001. Joe: Got it. Jason: And I think, I think I might be more like a lot of other people with this, like I've, you know, I've written books with a lot of the vegan doctors and usually their message is that's all or nothing proposition. You go from zero to 60. And from a physiological standpoint, you're going to regain your health really fast that way. But if you're miserable doing it, chances are you're going to quit out. And so I think for a lot of people transitioning, as long as they have it in their mind that it is a transition, it makes it easier for people. So that's that's what I did. It took me it took me about eight months to fully transition over. And I tried to zero to 60 approach for Joe: Right. Jason: three weeks, and it, I was miserable. Joe: Yeah, and for me, the 30 day thing I did not find hard, the part I found hard about it was the meal prep and that's literally what was difficult for me. And I even heard you in some other interviews, the good thing that we have going for us these days is that it's, it's much more accepted in the world. And when you go out to a restaurant, there are options that would have never been there 10 years ago. Jason: Yeah, there are plenty of options, Joe: Right. Jason: Which has made it an interesting landscape for vegan businesses. Because I think in the past, vegan's gravitated towards vegan businesses because that was their only choice. And now at least in the Phoenix area, vegan businesses are just one amongst a bunch of other vegan options. Joe: Right, but I think the key and the reason I was so excited to have you on is what helped me get through the, the, the next 30 days that they asked me to do because they could see that my cholesterol was dropping. So Jason: Great! Joe: They were like, will you, "Are you willing to buy into doing it another 30 days? And towards the middle or end of the first, as I think when I came across your website and then it was easier for me to say yes, because I literally just could not afford the time to prep. Jason: Right. Right. Joe: But but besides that, the biggest thing for me was the taste. And I don't know, like this could be a trademark or something that I'm saying, but I didn't know vegan food could taste so good, and you can still Jason: No it's true, Joe: if you want. If it's not taken by somebody, it's all yours. But, yeah, that's what it was for me, man. When I first dug into it and the way I worked with you was that I wanted it spicy, which you were all down for. I think even when I, I got from my doctor what I needed to do, he said, OK, well, if you're gonna get this food from The Vegan Taste, just make sure, ask them if it's low and oil, right?. And it so... Joe: It everything was a yes. Like all, you know, that was when I wrote to you, Yes, you know, it's either low or minimal oil or no oil. And I can get it the way I like it, so you made it spicy, which is the way you said you liked it in email. Jason: Right. Joe: So it was like the perfect marriage. I was like sold! Jason: Yeah, I think that's, that's the key to getting people to make a change. It's about honestly, I think it's like about the in the environment that you put people in. So I know Dan Buettner, who wrote the Blue Zones by it. And one of the things that he told me that really impacted the way I thought about food and getting food to people and the way we treat people, is that the the biggest determinant for someone making choices that let them live a long time was not their willpower, was not a doctor's prescription or anything like that, it was the environment in which they lived. And so if the choices were easy to make, to go out and exercise, statistically speaking, more people would go out and exercise...that way. And so to me, food is part of the environment that you're in. And so the easier I can make it on someone to make a better choice for themselves, the bigger chance they are they're going to have to actually make that choice. And so for me, that's putting ready to eat meals in front of someone that's going to make them happy. Joe: Yeah. Jason: The less you have to worry about it, the easier it is for you to be healthy. Joe: Yeah, it's it was so nice to find the website. It was that, I could hear that sound when the heavens open, I was like "Thank you!". It's the only thing that's gonna keep me on track. Now, you know, before, before we get too deep into this, I'm not full vegan. Since doing nutrition program, I've cut out a lot of, like I would use, I would snack before dinner. I'd be so hungry I'd come home at four o'clock, whatever, and I'd pull out the the block of cheddar cheese and some Triscuits and, you know, just take the edge off. I, I stopped doing that a lot more than I use, you know, it's, it's cut way back to almost minimal, you know, to none. I don't drink, I used to drink half and half of my coffee and now all I use is either oat milk or almond milk. So I've completely switched over to that type of stuff. So while we're on the subject of, of, you know, how this has helped you, why do you think dairy is so bad? Is it just that it's like, was it not meant to be eaten or drank? Is it just like we've created this product that should not have existed? Jason: I think so. I mean, dairy's primary uses to grow a baby. And so you're you're consuming something that's meant to grow another being and as, as adults, we're not, I don't think we're supposed to be consuming foods that are continue endlessly making us grow to that scale. Like I have a five year old daughter, I watch how much she eats and sometimes as much as I do, because she, she's always out there running around and she's, like I look at her in a week later, she's taller and I'm like, oh, my God! And so calorically dense foods are good for her, I mean, that's why human mothers breastfeed and you know, all this other stuff. But then when you stop growing and you keep eating those foods, you're consuming growth hormone and all this other stuff that I don't think we're meant to be consuming. And then, you know, there are a couple other issues that go with it, which it turns out casein, which is the protein in milk seems to be carcinogenic, even, even in that milks appropriate species after their weaning, it seems it seems like the incidence of cancer goes up in that species if they continue to consume milk even from their own species after they're supposed to stop drinking it. And then, I mean, look at us where we're drinking stuff that's meant to grow a baby cow into this big monster cow compared to humans I mean a cow is pretty heavy. Jason: So, you know, there's, there's that it's, it's loaded with fat and it's all if you have cheese, it's all condensed down into this calorically dense product with all these other, all these other ingredients into it that are probably not meant for us to just get fuel. And it's all like if you take milk, milk is this big volume, take cheese and it comes down to  this little thing, all that condensed down. It's like a black hole of food. And then you're you're eating that, so, of course, no wonder you're you're getting fat, you're having arteriosclerosis as you age and all these other problems. So that's why I think the health problem is with dairy. From, from an evolutionary standpoint, it's was a good thing because you could have this nutrient dense food even in times of famine. That's, that was one of the benefits of cheese because cheese was basically shelf stable in a long period of human history when we didn't really have very many shelf stable foods, the same way that after a fashion beer, a shelf stable, just one of the reasons that beer was traded there and there are all these ways to preserve foods during times of famine and we just don't live in that anymore. Joe: Right. So on the dairy part of this, what I guess people have a hard time thinking of how they would substitute a cheese for these recipes, and I know that in you know, you have this enchilada recipe and you, there's I mean, you have a ton of different recipes. What are just some off the top of your head, some substitutes that you do use for cheese? Like, how would someone make a pizza? What would they put on it as their cheese? Jason: You know, it depends. There are a lot of nondairy commercial cheeses out there. I think from a health standpoint, they're good insofar as you're not getting casein and all these hormones that go with it, but I can't pretend that they are health food. Joe: Right. Jason: I mean, it's base, it's like cheese is solidified fat when it's dairy and the non vegan cheeses are still a solidified fat. They just have all the other junk that goes with them. So, you know, if you if you limit that look, if you're going to have a pizza and you have it once a week and you put some vegan cheese that's made out of almonds or cashews or something like that on it, you're going to be OK. If you do that every single day, you're not going to be so OK anymore. You can still be a junk food vegan. In fact, it's easier now to be a junk food vegan than it is to be a healthy vegan, because you can run over to Carl's Jr. and get a Beyond burger, that's, you know, still loaded up with all this fat and it's still a burger where as when I went vegan almost 20 years ago, if I was craving a burger, I had to make it myself. Joe: All right. Yeah, I mean, the creativity Jason: So that's. Joe: That, that you have to come up with for these recipes must be daunting. Jason: I sometimes, but only because when I do a lot of recipes, Joe: Right. Jason: I mean most, most chefs at a restaurant might do 30 recipes throughout the year. If they're really pushing themselves. I think with the delivery service, we're doing 300. Joe: WOW! Jason: Every, every year, at each year, it's different too. Joe: Ok. So you're rotating 300 recipes a year from The Vegan Taste. Jason: And we're just making about as we cook every week. Joe: It's amazing! Jason: Yeah, it's, it's, it's daunting, but it's cool. Joe: Yeah, it's. Jason: Yeah, I mean, and like back to the cheese thing, sometimes it's replacing that, that fatty mouthful, mouthfeel that cheese gives you so you can even use something like an avocado or you can use, what are my favorites is this thing called pipián verde, which is just this ah pepitas and tomatillo puree. It's it's a classic Mexican dip and I'll just use that on enchiladas or we'll make our own cheese at the restaurant, sometimes we'll make it just out of almonds and some other ingredients and we'll make our own queso fresco like that and we make our own mozzarellas and stuff. That's a little laborious, I think, for the for the home cook, it's just getting that, that creamy texture which you can get from nuts and seeds. Joe: Right. Yeah. Because even on the recipes at Casa Terra, your restaurant, I saw that there was I think you have is it brick oven pizzas or just... Jason: Yeah, Joe: Or Jason: We have worked fire Joe: Wood Jason: With Joe: Fire. Jason: Fire pizzas Joe: Right. Sorry. Wood fire. Yeah. And so and I did see one of the recipes are one of the descriptions of the you know, the pizza said mozzarella. So I was like, OK, how does he doing that? Jason: Right. It's just a, when you get to that type of cheese, that's it's a little time consuming and it's a mix of art and chemistry. Joe: Yup. It's just it's incredible. So I know we just kind of skipped over it a little bit but we talked about your daughter and, and I and I know we talked about, we didn't quite say that she's vegan, but I know that she is from based on my research about you. And I know it's tough with kids these days with all of the gluten allergies and, and everything that's going on that or used to be a lot tougher. Now, its parents are more aware there are more options and I would think that it's almost the same thing with your daughter as it is with a child that has a gluten allergy. When they go to a house for a birthday party and let's just go back to using pizza as a example, because that's how I grew up, right? That your parents would buy a bunch of pizzas, and... What does she do in that case? Or how how do you let the parents know that she's vegan and that, you know, that isn't something she would (A.) like to eat or (B.) she shouldn't eat or (C.) it might make her sick of she eats because she's not used to eating cheese. Jason: We just we tell them and ask them not to make a big deal out of it. And then we make sure our daughter has food that totally owns everybody else's. Joe: Perfect. Jason: I Joe: That's awesome! Jason: When she was in school before COVID hit, the teachers were asking if we could bring stuff for them. Joe: That is so funny. I can imagine, no I, listen, I know what it smells and tastes like. Every kid we sit there with, their pizza from Dominos going, WWO!, what are you eating? I'll trade you, I'll trade you two slices for that, that's perfect. Well good, she's totally vegan incorrect? That's amazing. So you, what is the Vcology project? Is that how you say it? Vcology Project? Jason: Vcology. Joe: Vcology. So. Jason: It's pretty much the umbrella for all the stuff that I do. Joe: That's what I thought, I just wanted to make sure. And I, because I know that you spoke about The Vegan Taste, which is the home delivery food service, Casa Terra, which is the restaurant out in Glendale, Arizona. And then I heard you speak about other things potentially coming down down the road, so I assumed that that was the umbrella where all of these things would fall under. Jason: Yeah, I mean, we're working on commercializing our cheeses on a large scale. We've already had one big vegan restaurant chain express some interest in it, which was really cool, it came out of the blue. But that was, that was a nice surprise. And Joe: Yeah. Jason: And we just want to roll out really high quality vegan cheeses onto the, onto the food service market and then retail, if we can. Joe: That's great. Jason: But if I can. I mean, if I can get, like some of the best restaurants in Phoenix using high quality of vegan cheeses, all of a sudden it opens up really great menu options for vegans around the entire town. Joe: Right. And I Jason: And Joe: Was Jason: I Joe: Thinking Jason: Think Joe: Good Jason: Go ahead. Joe: While I was sitting Jason: I think. Joe: On the dairy part of it, and I didn't even know that this underlying thing about the cheese had a broader scope or what was happening. I just I kind of chose the one thing that I know, like you, you know, it's like, how do you have ravioli? How do you have a pizza? How do you, if you you're so used to having half and half in your coffee, how do you make the move away from dairy? And I think that's, I think that's harder almost than the meat part of this or that Jason: It's way Joe: Or the Jason: Harder. Joe: Protein part of it. Right. Jason: I didn't know why until Dr. Barnard told me a few years ago that the casein in cheese is called the casomorphin and that basically means that acts like morphine. It acts like an opiate in your system. And I was like, "That makes sense!!", because one day I just gave up meat and it was like, whatever but when I gave up cheese, I had withdrawal symptoms. I was jonesing, I mean, like the hands were shaking and I had headaches and I was irritable and everything else that I had heard from people that were trying to give up cigarettes or drugs or something like that, I was going through and I'm like, "What the hell is going on?" That was, that was one way where I knew, like, I've really gotta get off this stuff, because Joe: All right. Jason: If I'm having that reaction, this is probably pretty bad for me. But it was a few years later when he told me why. And so Joe: That's Jason: Anyway, Joe: It. Jason: I think that's why cheese is so hard. Joe: That's incredible. How did the two of you get connected for that book? Your book? I wrote it down. I'm going to have it in the show Jason: Sure. Joe: Notes. Jason: The "21-day Weight Loss kickstart". So he was coming through town to do a talk and they wanted someone to do a cooking demo and I was the only one in Phoenix, doing this kind of stuff, so I just volunteered to do it. They were gonna pay me and I was like, don't worry about it, I'll just I'll just do it. And so we became friends through that and then I started teaching the cancer project classes here in Phoenix for a few years, which later became their Food for Life program. And, and during that, I just developed tons of recipes every single week. Because I think back then they were kind of in the same boat that a lot of healthy, healthy doctors are in, we're like, they're like, you have to change your diet. Here's how you do it. But they're not really experts at the here's how you do part. Joe: Right. Jason: And so, you know, their recipes were easy to do, but they weren't necessarily great. They were just like, "Ahhh". And so during that class, I just continuously develop stuff that was usually easy to make, but also really spectacular. And then because of that, we just wrote the book together. Joe: And that's really cool. It's just amazing how things, you know, you can make these connections and they just turn into something amazing like that, so, yeah. I'm trying not to skip around, there's so many things I have to ask you, I have so many notes, it's like this is, like I said, I, I was doing the meals for when I was doing the 30 day thing, basically for lunch and dinner. And then I started to do them just for lunch because my partner, Jo Ellen, we were like we were eating separate times, separate things at dinner, it felt like it wasn't this Jason: Right. Joe: Community. Jason: You loose the social part. Joe: Yeah, and so it's this balance for me. But so I thought at least at a bare minimum, and I think this is one thing that we talk about stepping stones and doing this in stages, is that it's worth at least trying to say to yourself, OK, "I'm going to eat vegan for lunch", just take a meal of the Jason: Right. Joe: day and say, this is what I'm going to do. And literally, breakfast is super easy because for me, it's, it's like a vegan smoothie, right? There's nothing and so I don't have to worry about that. It's not sausage, an egg and bacon and all this other stuff. So then you handle the vegan lunch part and you're already better than probably seventy five percent of the world in regards to how healthy you're eating. Jason: That's Joe: And Jason: What Joe: Then. Jason: I think. Joe: Right and then you just. So and that's kind of the approach I took. I don't know yet, just being honest with you, if I can completely eliminate that occasional steak or burger or Jason: Right. Joe: And I'm sure I can at some point, like for me, like you, I, I refuse to go on medication. So I'm 58 years old and I'm like, I'm not going on cholesterol medication. I don't take anything for high blood pressure. I'm not going to do any of that stuff. So if it's a, if it's food, it's going to make the difference, then that's the difference that I'll make. Go into the gym five days a week is already easy for me. But if I have to do that and get rid of the burgers and the steaks and whatever, and that's the mood that I would make. Jason: And if you could make that, did you make it fun and pleasurable, then why not? Joe: Right. That's Jason: If Joe: It. Jason: It's this chore, you know, like most people are gonna be like, ahhh screw it. I don't want to do it, Joe: Now, Jason: But. Joe: For me, it's it's talking my girlfriend into seeing if we can do it together, so that'll be the that'll be the piece we'll see. Yeah. So tell me a little bit about, oh, I also heard an interview where you said that your daughter growing up with two chefs. So is your wife also working with you at either at The Vegan Taste or Casa Terra? Jason: She she was Joe: Ok. Jason: Doing The Vegan Taste for a while. Joe: Ok. Jason: I mean, for, for years, she was with me in the kitchen. And sometimes when I was off doing other stuff, she was running at it for months at a time. Joe: Got it. Jason: But I now we're in a situation where it's hard for us to split our time like that. And so she takes care of the household and raises our daughter while I take care of the business. We tried where we were splitting it both ways and it was like, I think it's hard to multitask. Right? It's hard to be great at a bunch of different stuff at the same time. And so we just finally decided, well, I'll have to go off and kind of slug it out and be the champion for the business, while she's the champion for keeping the rest of the family sane. Joe: Which is the admirable thing for sure. So The Vegan Taste, let's talk about that really quickly. So The Vegan Taste as home delivery, vegan meals that come in these great packages that are, like you said, are the goal is to heat and eat. And Jason: Right. Joe: They I don't know. I'll let you just talk about it because I don't want to, I know I had a certain schedule and the whole thing with the coolers, but I'd like you to describe it so that the audience will know what it's all about and then they can make their decision from there. Jason: Yeah, it's it's super easy. So the menu changes every single week. It's a fixed menu. You put your order in by Friday night. My crew comes into the kitchen on the weekends, makes everything. We plate it up over the weekend. Pack it up for delivery on Monday and then my team of drivers go out every Monday and they deliver all the meals at once for your entire week, that Monday. They leave it in a cooler loaded up with ice packs so even in the middle of July, the meals will stay chilled until you can pick them up and then you put them in your fridge. I know, some of our clients will reheat them on the stovetop. They'll take the ingredients out and reheat them on the stove, top it honestly, talking to people, most of them stuff it in the microwave and they have a lunch in two minutes. Joe: Yup and those containers are microwaveable. Jason: Yes, Joe: Is that correct? Jason: Yes. Joe: Yes. I know I've done both. I've depending on what the food was, sometimes I would heat it on the stove and sometimes I would heat it in the microwave. And I think that's all, also another thing in my brain about microwaves, they know make me a little nervous thinking that maybe something's there that eventually Jason: Right. Joe: someone's going to admit to, so if I if I have enough time, I'll go to the stove. If I don't, I just use the Jason: I Joe: Microwave. Jason: Am exactly the same way. I mean, I don't even have time to cook for myself very much anymore, so so I use our delivery service for me and most of the time I just slide the contents out of the container and right to a pan. Joe: So in regards to the meals that are available, is it, are they just lunches and dinners? Are they breakfast, lunch and dinners or... Jason: It's basically lunches and dinners right now, but will add in a breakfast option and the juicing option and some desserts pretty soon. Joe: And and like me, at one point, I was getting doubles of things so that I could have something for lunch and then something completely different for dinner. So I assume you have clients across the board that are only lunch, only dinner or a combination of enough meals for, is that how many, how many Jason: Yeah, Joe: can they get? Is it Jason: So, Joe: The. Jason: Yeah, basically we do six different dishes every week and you can get a single portion of each one or you can get a double portion of each one. And the people that want to have our meals for lunch and dinner, get the double portion. Joe: Right and that's what I was doing for a time, that's, that's right. And then in my case, I said that I wanted it spicy but so you actually keep tabs of certain things that people request on a small, I assume a small level because you can't be doing personalized, you know, things across the board for everybody. Jason: Yeah, we have spice is one of the standard options we have for people. And then we have a gluten free option, soy free option, although we use pretty limited soy already anyway. And then no oil option in the meals, again, are are pretty much pretty low oil already. So we just talked to people like, do you really, really want no oil? Or is that that's that you're trying to minimize your your oil? Are you trying to minimize your soy? Are you trying to minimize gluten? Because we don't we don't use those types of ingredients heavily in the meal service. And then if there's something that we can, leave off as a garnish for someone like if someone's like, "I hate right onions." I'll tell them, you know, if it's mixed into the dish, we can't change it but if it's a garnish, we can make a note to leave it off for you. Joe: Right. Jason: I mean, most people are good about it, but then sometimes I get someone that sends me a list of like 10 different things, I can't, sorry, I can't do that. Joe: Thank God I do that I don't want to sit here and look at you in the camera and go, oh, I was one of those people. And Jason: No, not Joe: I Jason: At Joe: Think Jason: All. Joe: The only thing that I said, I everything was great for me. The only thing I request that I think was less tofu in some of my stuff only because I'm I, it's just me getting used to it, it's it, and, and it's not, I would, I wouldn't even say it's a texture thing for me because I eat oysters, right? That's about as weird of a texture as you can Jason: That's sure. Joe: get. So I don't know why I definitely have had tofu from your food service, that was amazing. And it's almost like it's firm and some of it sometimes is even like crispy, like it's it's hasn't where I've had it other times where it just, just, it's just weird. Jason: Yeah, I mean. Joe: I don't know if there's good or bad tofu, maybe there's just the quality of it, I don't know. Jason: It's the way, it's the way it's prepared. And I think it's also what you're used to growing up with. I mean, if you're used to growing up with, say, diced up firm tofu in a miso soup, you're not going to bat an eye at it. But if you're not used to that, the texture might be weird for you. And I think, when dealing with American culture where we're not used to that stuff, too many people just take tofu and throw it in a soup or a stew and they're like, "Okay, that's good enough." But it's not I mean, it's like to me that's like throwing in a raw hunk of meat and is something and being like whatever. So, Joe: Yeah, Jason: You know, it's just it's Joe: Ok. Jason: All in the preparation. Joe: Ok, good to know because I started to get to like it. And thanks to you once again, because I was definitely I grew up with, in an Italian restaurant family and my father was a chef and so all of this stuff is new to me. Jason: Right! Joe: I was eating pizza and pasta and bread and, and you name it. So I wanted to ask you about Cassa Terra. I noticed that on the website, like a lot of places, especially during this time we're living in right now with COVID-19, that the kitchen is closed for the summer, right? That's what it says on the website. Jason: Yeah, Joe: Is that true? OK. Jason: A lot of the high end restaurants, it seems, around town actually close up for the summer. Unless there are these big corporate things that can afford to take the loss that restaurants just suffer with the summer here. Joe: Is Casa Terra where you do actually all the food prep and making them? So that that kitchen is still being used for the food delivery service? Jason: Yeah, it's our Joe: It's. Jason: R&D kitchen and our delivery service kitchen. We do catering and stuff out of there, too. Joe: When does the restaurant open or when do you expect it to open back up in the fall or ? Jason: I'm not sure yet Joe: Ok. Jason: Because honest answer is for a, for the type of food that we do, our location is not that great. And so if we can find a location that's more central or on the east side, that makes more sense for us right now than trying to just reopen in Glendale. And Phoenix is a weird city, so, we have these really accessible freeways and it's actually pretty easy to get around here but I don't know if our food culture is is there yet, because if someone else to drive more than 20 minutes here for food, it's painful. And the chances are they won't do it. Joe: You know. Jason: Or if they do it, they'll come once a year. And Joe: Yeah. Jason: So it's, it's difficult that way we're compared to like Los Angeles and New York or Chicago, people will spend an hour getting to, getting to a place to have dinner. And if it's a good meal, that's just part of the it's part of the experience. That might not be a great part of the experience, but it's something you're willing to do. So. Joe: Yeah, absolutely, Jason: So Joe: Yeah. It's Jason: We Joe: Funny. Jason: Have to be, yeah, we have to be in a more central location. Joe: Yeah, because I know we're in, and I live in Arcadia and the boundary for me is pretty much like the 51. If it's on the other side of the 51, I have a hard time going that far west but I understand that. You, one of the things that I did read was that about the Le Cordon Bleu the school and it was something about you being, was it the first graduate of vegan Jason: First Joe: Or Jason: Instructor. Joe: First instructor of vegan? Jason: Remember when it was theater, 2007 or 2008 that I was teaching at the Scottsdale Culinary Institute Joe: Yeah. Jason: And right when I, right when I started teaching there, they became part of the Le Cordon Bleu program. And so I, because I became the first official vegan instructor in that program. Joe: That's really cool! Jason: There was there was cool. Joe: Yeah. There's so many things, the other thing was I remember either hearing or reading that philosophy was your major? And I think what, what struck me about it, when I when I read it and then who you are and, and I even, there was an interview about making the argument of why to go vegan, like how when someone find something like this and this is why this has been like I've wanted to talk about, even though I haven't gone full vegan, I think that the health benefits are so important and just the, the eliminating of dairy alone. I mean, I've told people when they said, oh, yeah, you know, it sucks getting old. I'm like, well, I'm 58, I agree with you, but I don't, I'm, I don't wake up feeling achy. And, and, and I never did a lot of dairy, but even cutting out what I've already done, I think the inflammation piece of this is what other, you know, is another part that people are missing. Jason: I'd, Joe: And so, Jason: Yeah, it's. Joe: You know, so getting back to the philosophy part about how you're able to convey this in a not like beating someone over the head with a club, you've got to do this, it's, it's the only way. Your approach to it is your first of all, your demeanor of how your, you know, your a 6' 3" guy who you would never think if I met you in the street, would say you're vegan. And then the way you intelligently talk about the food and then the bonus of all of it is how it tastes. And so there's just so many amazing things about this, it's why I was so excited to finally do this. Jason: Well, cool! Thank you. Joe: So the Jason: It's. Joe: Go ahead with the phil..., with the philosophy part of this, I think it's helped a lot. Jason: That that's actually what got me to go vegetarian, but also it it taught me a few things about the way people make decisions because I socially and just because of the way I was raised, I didn't want to go vegetarian because it meant changing my lifestyle. And intellectually, I've been kind of bandying it about for a couple months before I pulled the trigger on it. And I didn't do it, it was just something I had thought about it. And then I had an epiphany because I was watching, I was playing with my cat. And I, intellectually, I knew my cat is this other being with its own thoughts and her own emotions. But then there was something where I was just playing with her and I had that emotional epiphany and that's where it went off and I was like, I understood that my cat was this separate creature that was valuable and she had her own rich emotional life and because she was sitting there problem solving and she was getting excited about bringing this little bottle cap back to me and playing fetch with me. It wasn't like this, this robotic, emotionless, thinking-less, piece of matter that, that's how Descartes used to view animals and that's how he justified doing all these horrible experiments he did on them because he, you know, even though they would, they would scream and all this other stuff, he passed it off as they didn't have a soul and they weren't really conscious and all this other BS. And so you can intellectually know that, but then you have the understanding there is that connection. And within a second I was like, wait a minute, it's not ok for me to just, like, take a hammer and smash my cat apart right now, that's really jacked up, that's something serial killers do. Why? Why can't I do that to my cat but why am I paying someone to do it to a cow? And I was like, "I have to stop!" So I stopped, went vegetarian and then spent a month arguing against vegetarianism to see if any of the arguments hold up. And none of the arguments were self-consistent. And so I was like, I'm going to stay vegetarian. And that was the the rational part of that. But what I learned was I had to have that emotional epiphany to fully make that leap in my decision making. And then when I went vegan, it was even more so because I was doing it for health reasons. But then I found out about factory farming. So it's ironic because being vegetarian for a few years, I had no idea about factory farming and then all of a sudden I'm looking at it for health reasons and learning about factory farming and I know that it's what happens in a factory farming is horrible and I don't want to partake in it. But yet I'm going out and having all you can eat enchiladas once a week. Because I emotionally had that tie to the enchiladas and, and so I think for most people, decision making is ah, pain pleasure balance. And it's, it's a very immediate and very immediate decision. And it's funny because people that can make that decision for the long term, we call them wise, because in the short term, going out and jogging or lifting weights sucks for most people. But the wise people go out and do that because, you know, it's going to pay off in the long term. And so I think going through that myself, even though I was trying to be rational about it and I knew what the right decision was and not being able to make it because I had this emotional thing is what got me into food in the first place. Because I knew if I could if I could take the pain part of that calculus away for people and just give them an environment where they could make a good decision for themselves and for the planet and for the animals, then, then I had to do it. Joe: Yeah, it's, it's really cool. I mean, I learned so much more about you just doing the research that I wanted to do up front and, and I think it's important how the philosophy part of your, what your brain has done through, you know, getting that degree in school and then then I heard about the soul sucking marketing job that, you Jason: Oh, Joe: Know. Jason: It was horrible. Joe: Right. Yeah. And it's and this is it all plays, this is why this Jason: It's. Joe: is such a cool interview for me. And I don't want to keep you any longer because I know that, you know, you work really hard and but I, I would love to do more at some point, Jason: Yeah, that'll be Joe: You Jason: Fun. Joe: Know, it's just cool that you, you are doing your passion. It really means a lot to you. You're you know, you eat, sleep and breathe what you preach, but you preach it in a way that it's not preaching. The food tastes amazing! It was just a godsend for me to find it. We find out tonight as you're setting up here and give it a talk, you play the drums. It's like, what, what more of a kinship could we possibly have? And all I do is try to preach on my podcast and on my, you know, social media and all that is just people following their dream. And it's really cool to see you do this. It's, it's, it's great. And and I'm glad you're healthy. Glad you made the choice when you did. You're here Jason: Yeah. Joe: To help keep us all healthy and feed us. Jason: Well it's funny, so it's funny you brought that up, because I feel like I'm in another transition point in what I'm doing because, ah you know, I had this amazing journey where I lost all this weight, I cured my diabetes, became a chef and went and helped out other people. And in the last couple years my, my health started to decline and I was like, what's going on because I'm eating right. But there's, there's all this other stuff. So, I mean, you know, in the last couple of years, I almost got divorced. I was working 100 hours a week. I was doing all this other, other stuff. I was, you know, we went to set up to open up this restaurant, we had some guys steal about 50K from us and steal, ah... He probably cost us about 200 grand in the long term, which was almost all my family's money and almost all of my best friend's money that she had. And then we opened up this, opened up this restaurant, which you were in the restaurant business, so, you know, like it is a lot of work. And on top of that, we're doing these other businesses. Jason: And so there are all these other stressors and I realize it actually happened right wing COVID hit. Because we were thinking about like, we were really looking forward to the summer when we could shut the restaurant down for a while and get a breather. And then COVID hit and all of a sudden, oddly, my life got better. Because I was spending time with my family and I was killing myself anymore and my health started to improve. That was it, I had this very narrow focus in my life, which I was really good at but it also carried all the stress that I think, I think you have when you get a little bit older in your career and you're kind of at the, you're operating at a higher level, it's also a more stressful level. And there's a lot more at stake about point. And so when COVID hit, I had more time for my family. And then I started going on bike rides again and hiking and I started spending time playing the drums, I hadn't touched my drum set in three years. Joe: WOW! Jason: And I started playing again, which was actually cool. I have this thing where I get my, stop something for a while when I pick it up and better at it. So now I can actually play some of the Rush songs that I couldn't get through Joe: Nice. Jason: For three years. Like, where did this come from? Joe: It's awesome! Jason: You know, so that was cool. And so, so I realized, like, I'd been talking about environment with food choices. But I've been ignoring everything else that goes into being a healthy person and taking care of your mental state, taking care of your family, making sure you have time to not be insane with all this other other stuff and so I think my crew is shifting into a point where I'm going to start talking about more about holistic health and creating good environments for your, for your well-being as an adult. It's, I'm sure it's true for for kid or whatever part you're in but since I'm in my 40s and kind of went through the midlife crisis part, that's how I solved it, was figuring out that I had to create a good environment to make good choices throughout my whole life and not just with the food, because I'd just been concentrated on the food, which is one key. Joe: You. Yeah, it's amazing how many people I know, it's it's hurt a lot of people. But I personally, it's been the best three months and so long because I was running so hard. And like I said, I've gotten to do things that I want to do. I it's just it's been a good thing. And I'm glad to hear that everything is turning back around for you, too, as well. I worried about you when it happened, to be honest, because, you know, I, I know it devastated the event world for me, I mean everything just stopped. And so I was worried just purely whether or not you know how how well you would do during that time. And it's funny, speaking of, you know, COVID-19. Was there any concerns about, you know, your clients with Joe: The food delivery and any, any things that you had to do differently in order to to be, you know, follow the CDC guidelines or anything like that? Jason: We just did extra sanitation, but we were already doing that stuff anyway. Joe: Right. Jason: We were just more hardcore about it than normal. But that was it. Because I think with the food delivery, it's contactless, so our drivers just show up and Joe: Drop the Jason: They're Joe: Cooler. Jason: At their doorstep Joe: Yeah. Jason: In and head out. Joe: Yeah. Jason: So, so in a way, it didn't really affect the delivery service at all. Joe: Got Jason: It was Joe: It. Jason: horrible for the restaurant, but that ended up being a boom for us personally. Joe: Yep, yep. Well, awesome! Man. I cannot tell you how grateful I am that you're here. Like I said, I was disappointed when I had a sort of postpone it last time, I just took on too much. It was one of those deals where I thought I could I forget how much time postproduction takes after I get off this thing to get it, Jason: Yeah. Joe: You know, ready for prime time. But I am super, super grateful that you said yes and you came on, I love your food and you're an amazing human being. The more I've done the research and get to know you now. And it sounds like your daughter is definitely waiting for you to put her to bed. So I'm glad, I could go on, I swear to God for another hour, there's so many questions about food and just things that you've done, but we'll do it another time for sure. Jason: Yeah, that'll be fun. I'd love to come back. Joe: I again, I can't thank you enough. It's an honor to have you on here. And I'd love to have you back again. Just for the audience sake and things like that, where's the best place to get in touch with you? And I'll put I'll do in the show notes, I'll list every, you know, your social media things but like in regards to, let's say, The Vegan Taste, what's the best way for people to reach out? Jason: Just go right to thevegantaste.com Joe: Okay, perfect. Jason: I mean, we have all the social media platforms, but it seems like, you know, Facebook changes what they want to show to people every few months and Instagram is the same way. You know, all these other ones. So just just go straight to thevegantaste.com Joe: Perfect. I'll put in all the other links, I'll take care of all of that. Again, thank you so much, I appreciate it, it's so, I look forward to actually meeting you live in person. Maybe we can sit around and jam one night. Jason: That would be awesome! Joe: I would love it. So. Jason: Cool. Joe: All right. Thank you so much, man. I appreciate it. Jason: Hey, thank you. Have a good night. Joe: You too!

What do most skiers and snowboarders dream about? Owning mountain property. But most of them don’t have the necessary wealth or money to purchase a second home, especially in expensive real estate markets. Today, I am talking to Rob Stephens of Avalara MyLodgeTax. As a Denver resident and lifelong skier, Rob was fortunate enough to buy a three-bedroom condo in Vail about 20 years ago. But even then, it took Rob, his wife, and wife's brother-in-law to make the dream a reality.  You’ll Learn... [02:57] Short-term Rental Market Options:  Hire local property manager/real estate agent. Post property online via VRBO, Airbnb, Expedia, etc. Rent-by-Owner: Book guests, collect money, and provide on-site services. [03:48] Complex Tasks: Apply technology and manage challenges for property owners, managers, and operators. Know what taxes to charge, collect, file, and pay to agencies. [05:50] Lack of Awareness: Property owners trying to manage their short-term rentals have never dealt with these types of transactional taxes.  [06:20] It’s not rocket science, but multiple layers of government are involved. State and local tax rates and requirements are specific to rental property location and address. [07:10] Avalara MyLodgeTax: Online hosted, Cloud service similar to TurboTax but for short-term rentals with vacation, occupancy, resort, and other taxes.  [09:27] Penalties to Avoid: Long-term, multifamily operators getting into short-term rental space need to understand rules and risks involved.  [11:40] One less thing to worry about. Partnerships with property management companies is when Avalara handles everything occupancy tax related.  [13:02] Common Questions and Concerns: Shortcuts and consolidation for creating awareness and understanding the mechanics of administrative work. Tweetables Increasing scrutiny and regulation on the short-term rental space, makes for more paperwork and forms to be filed. Lack of Awareness: Property owners managing their short-term rentals have never dealt with some types of transactional taxes.  Short-term rentals involve multiple layers of government. State and local tax rates and requirements are specific to rental property address. Resources Avalara MyLodgeTax Vail, CO VRBO Airbnb Expedia TurboTax DoorGrowClub Facebook Group DoorGrowLive DoorGrow on YouTube DoorGrow Website Score Quiz Transcript Jason: Welcome, DoorGrow Hackers, to the DoorGrow Show. If you are a property management entrepreneur that wants to add doors, make a difference, increase revenue, help others, impact lives, and you are interested in growing your business and life, and you are open to doing things a bit differently, then you are a DoorGrow Hacker. DoorGrow Hackers love the opportunities, daily variety, unique challenges, and freedom that property management brings. Many in real estate think you’re crazy for doing it, you think they’re crazy for not, because you realize that property management is the ultimate high-trust gateway to real estate deals, relationships, and residual income. At DoorGrow, we are on a mission to transform property management businesses and their owners. We want to transform the industry, eliminate the BS, build awareness, change perception, expand the market, and help the best property management entrepreneurs win. I’m your host, property management growth expert, Jason Hull, the founder and CEO of DoorGrow. Now, let’s get into the show. My guest today is Rob Stephens and his company is Avalara MyLodgeTax. Rob, before we get into talking about your business, I’d love to hear your background of how you're connected maybe to the real estate or property management industry. Maybe you could share with us just a little bit about your entrepreneurial journey just to qualify yourself to our listeners. Rob: Sure, happy to. Thanks for having me. I love the intro, good stuff. My story, I live in Denver, Colorado, lifelong skier like a lot of us, lifelong skiers here in Colorado. One of our aspirations is to have a mountain property. As a relatively young professional, this is about 20 years ago, was fortunate enough to be able to purchase a three-bedroom condo in Vail, Colorado with my wife's brother-in-law. The three of us went in together. At that point in time in my life, we wanted a second home in Vail, great skiing, be up in the mountains more, but part of that is when you make that financial investment—Vail is a very expensive real estate market—when we purchased it, we really needed income on that property to make the economics work. We certainly didn't have the wealth or money sitting around just to have a second home for our own personal use when we wanted it. We really needed to generate income. This was in the short-term rental market. Back in the day, this is the late 90s, you really hire a local property manager or real estate agent that would get you bookings. At the time, somebody told us about a little website called Vrbo, it was pretty new back then. We tossed it on what's now Vrbo. It was frankly really amazing to see the bookings and we were immediately connected to travelers really across the globe. Through this one subscription would cost us about $120 a year, we're able to keep our property pretty booked and you're talking $30,000, $40,000 a year in rent. Doing that, and I'm sure your audience will appreciate this, we were pretty clueless doing the rent-by-owner thing, there are all these other tasks to go along with it. You have to interact with guests, book guests, collect money, maybe have a rental agreement, you have to have on the site service, cleaning services. My background is I’m a CPA, accounting and finance background, worked for some of the big accounting firms. But even as a CPA at the time, we just stumbled across the requirement to collect, remit sales, and occupancy taxes. It was really through that experience, one being successfully doing this on the internet, managed my own property, and then coming across the complexity of managing these taxes even from my own little rental in Vail, we realized this is a powerful model. We think this online short-term rental, these sites have a lot of opportunities, but this tax is going to be a stumbling point. A few years down the road, we started a company to really solve that problem for people. Our model was to apply technology and manage all these complexities for our customers, which would be registering with the different agencies, knowing what taxes to charge, and then once you know what taxes to charge and collect taxes, then automating the monthly, quarterly filing and payment of those taxes to the different agencies. Just to be clear, this is like a hotel occupancy tax, the same taxes that hotels pay to apply to short-term rentals. That was our inspiration. We got started and we're able to form some partnerships originally with Vrbo and some of the other larger operators at the time, but the industry has certainly grown. Now you've got Airbnb, which I think really a ubiquitous household name, but the short-term rental industry, it's just everywhere now. We've been fortunate to be a little part of that on the back-end with our tax service, but about 4½ years ago, we sold our company to Avalara. Now we're part of the Avalara family, but our whole mandate in life, Jason, is to really support, whether that's an owner, property manager, or a large operator like Expedia or Airbnb, help all these taxes, apply technology to make all these taxes simple and keep that back-office function as simple as possible. Jason: Got it. In doing all of this, what are some of the complexities, some of the challenges that people are dealing with that have rental properties? Why is this a service that they might need? Rob: I think there's a lot of confusion in this arena. I think in the smaller segments of the market, if you think about property owners trying to manage this themselves, they've just never dealt with these types of transactional taxes before. If you mentioned tax to a property owner, they're thinking income tax. There's just a great lack of awareness and the complexity comes in. This isn't necessarily rocket science, but wherever you're renting, like for my property in Vail, I have to deal with the state of Colorado, the Colorado Department of Revenue for sales tax and then the town of Vail has their own local room tax or lodging tax, so that's very common. There are multiple layers of government involved, these are state and local taxes, and it's all specific to the rental property address. If you’re a manager and you have a portfolio, depending on how dispersed you are, each city, town, or county, and certainly, state, has their own tax rates and requirements. There are different forms and it can vary by city. That's the complexity. It's highly localized and each location has their own set of requirements. Jason: Now, your service can cover any city anywhere in the US? Rob: Yeah. We cover every location in the US. We don't do anything international yet, but that means we tell people the exact tax rate for all the properties they're managing or even if it's just one property, we register them. It's very common in this environment, you have to get a business license, short-term rental permit, or rental permits. There's increasing scrutiny and regulation on the short-term rental space so there are more paperwork and form. We manage and do all that on behalf of our customers. Then on the back-end, these taxes need to be collected on all short-term transactions. Your software platform automates the monthly filing and paying of the taxes. I sometimes compare it to TurboTax. It’s kind of an online hosted in the cloud service where you can log in, put in your data, and the technology does the heavy lifting and calculation of filing. We're similar, just a different type of taxes with these hotel taxes. Jason: Got it. What are some of the filings that have to occur that the typical homeowner would probably not be thinking about? Rob: Again, the typical homeowner (like I said) thinks of these taxes in the context of income taxes. I'll go back to my Vail property as an example. Every quarter, I have to file a Colorado sales tax return with the state. I have to file—this is where it gets confusing—a Vail special marketing district return which is also paid to the state even though it's a Vail tax. Then there's a Vail sales tax that's paid directly to the town of Vail. For my little example in Vail, I've got three different filings to two different agencies every quarter. That's pretty common. If you're in Florida, it's going to be a state sales tax filing and a county tourist tax filing. If you're in Texas, it's going to be a state hotel tax filing and then a city or county hotel tax filing based on your location. It’s multiple filings, it's always a sales tax or lodging tax, a very specific tax to providing accommodations. Jason: If somebody's listening that has a property like this and they're like I haven't really been paying too much attention, I probably don't need to, what penalties could be coming down the line that they're just not aware of? Rob: One of the things we're seeing is in the multifamily space. We're seeing a lot of operators that typically are in the residential long-term rental market, that are being pulled in at the short-term rental market because I think it's become so popular, it's become easy on these big platforms like Vrbo and Airbnb. The rent, the nightly or weekly rent can be very compelling, depending on your location of property relative to what you can do on a long-term lease. That's a huge trend we're seeing in the space is typical long-term or multifamily operators getting in the short-term space. One of the issues for those folks is that now they’re doing short-term rentals, the short-term nature of their rental is what triggers the requirement to deal with all these taxes. To your point, there's a risk there. Certainly, the very larger multifamily operators are very sensitive to it. They want to have all that covered before they engage in short-term rentals. For smaller operators, I think sometimes they'll get it going and then figure out as they're doing it, seeing if a short-term model is going to work for them. Again, there's increasing scrutiny on short-term rental operators, but the risk is that if you're audited or found not to be paying the tax, you can be assessed, not only back tax, which you would have typically just collected from your guest or renter—they're very accustomed to paying it—but then you've got penalties and interest on top of that. It's certainly something that can add up. We do see those cases and we do help people that are in that situation where they've got a back tax liability to figure that out. The penalties can be pretty significant. Tax, probably like everything in life, is better to do it proactively versus reactively. We certainly encourage our customers and people we interact with to try to get out in front of it, make sure you understand the rules, are registered, and collecting the taxes, so you don't have that liability jump up and bite you as you're trying to manage your business. Jason: For property managers that are listening to the show, they have investors as clients. Some of them may have short-term rentals. Do you typically work with property management companies and how might that work for those that listen? Rob: We work with hundreds and hundreds of property managers across the US in the short-term rental space. We actually do work, like I said, in the multifamily space. We're seeing a lot of those operators and we work with companies that often have largely been in the long-term rental space, they're getting in the short-term rental space. Our largest segment is still the rent-by-owner or the Airbnb host crowd but we certainly work with a lot of managers and we're a solution for them. Everything tax, occupancy-tax related, our model is to really handle that from soup-to-nuts, A-Z, like I said, registration rates. I think our property manager partners really value us as a way to take everything occupancy-tax related which includes licensing and registration, to hand that over to a trusted partner (which is us), and just make sure everything's done correctly, accurately, on time. It's just one less thing they have to deal with. Jason: Perfect. What are the typical questions that a property manager or a homeowner listening to this would be curious about knowing about your service? Rob: Good question. There are two different buckets. In the homeowner space, there's often generally just a lack of awareness of what they even need to do. It's more of a conversation about, “Oh yeah, if you are renting in Vail, here's what you need to do.” When owners understand that—and they often are very surprised about these requirements—they're often eager to outsource. Our service is $20 a month for that single user. Not to be cavalier, but we'd say it's a very affordable way, just to make sure you know the taxes are all getting paid on the right form, to the right agency, on the right date. For the rent-by-owner crowd or host, they're just generally not aware. For the manager, generally there is awareness. The first question a manager would ask us if they're new to the space would be “Hey, do I have to file this for each of the properties I'm managing in our portfolio? How does that work?” Generally, for managers, we can aggregate their portfolio onto one set of filings. We can really do it much more efficiently. There are some locations that require property level filings, but usually for managers, we can do things very efficiently, we can combine all their listings under one, what we call an umbrella account for the management company. Instead of filing 30, 50, or 100 returns for each property, we can file a couple for the management company that covers the 30, 50, or 100 properties they're managing in their portfolio. Managers often have a lot of questions around the mechanics of those type of administrative mechanics about how it works, how can we do it, and do they have to do it by property, by owner, or can we do things more efficiently? The good news is in the manager segment, there are shortcuts and consolidations we can do to make things easier. Jason: Right. Because usually, most of the properties are in similar geographic areas, you can bulk those together. Rob: Exactly. Most of our managers are obviously concentrated in one area. Sometimes they span across two, three, four cities, or a couple of counties, so there are maybe a handful of different jurisdictions or filings that need to be managed. We do work with some of the national operators like Evolve or TurnKey Vacation Rentals, we work with the Saunders and the Lyrics. A lot of their [...] comes out of that multifamily space. Certainly, the national operators have a much more complex set of tax requirements, so I think we can be a good partner. But you're right, most of these companies are localized within a certain area and they're dealing with a couple of tax agencies, not hundreds. Jason: Basically, you have streamlined tax compliance for property managers. You're going to be able to group some of these filings together so that they're not having to do as many, and then you're focused on things at even the local level, state level, and just making sure that they're compliant all throughout all these different tax situations. Anything else that those listening should know and how can they get in touch if they're interested in trying out your service? Rob: The one thing I had passed along, we sometimes see reticence on the multifamily or long-term operators. They certainly know what's going on in the short-term rental space and they certainly know maybe there are a few homes or some of their portfolios that would really work there, but they're often intimidated by these taxes. I'm talking about even multi-billion-dollar multifamily operators that we work with, they're very concerned about managing these taxes correctly for the liability. I think that's fair but I would encourage people if you want to get into it, experiment with it, or you think there's better revenue yield in the short-term space for some of your properties, I would encourage people to jump in. These large websites are very effective at generating your rent. Certainly, relative to the tax piece, we're applying technology to really just take that burden away from people and know that they can operate in this market, be fully taxed, and license-compliant. That's my advice. In terms of reaching us, we're a hosted service. We're on the cloud. We like to do things through technology. The best way to reach us and get information is through our website, which is mylodgetax.com. Once you're on the website, there's information about tax, our services. If you want to send us an email or give us a call, we do have phone numbers published there. We're happy to pick up the phone and walk through people's specific situation like what are the requirements for their city, how doesn't work, what would it cost to use our service. We feel those types of inquiries and happy to talk about it all day long. Jason: Perfect. All right. They can check out mylodgetax.com. I appreciate you coming on the show, Rob, and I wish you guys success. Rob: All right. Thanks so much, Jason. Jason: You bet. Bye-bye. All right, check out mylodgetax.com if you're concerned about the taxes, especially for your short-term rental properties, take a look at that. If you are a property management entrepreneur that’s wanting to grow your business, add doors, it might be time to take a look at your website. Test your website out. Go to doorgrow.com/quiz, test your website out, and see how effective it is at making money because it's not just about having a pretty website. It's about having a website that's effective at creating conversions, capturing business, and creating trust. If your website isn't, you could be potentially losing out on one, two, three, four deals every single month, maybe more. The typical deal for most property managers is probably worth about $6000 lifetime value. Say you can make $2000 a year per door and you can keep them on for maybe about three years on average, that's about $6000. If you're missing out on just a door a week or maybe about four doors a month, and if you're getting on about four doors a month, you're probably missing out on just about that many. That's about $24,000 in future ROI that you're missing out on every single month. Websites are not that expensive. Making the changes are not that expensive. That's a leak that's easily shored up. Check out the DoorGrow Score Quiz and you can then easily schedule a call with our team. Nobody builds more effective property management websites than DoorGrow. Other people are trying to manipulate and focus on search engine optimization and rankings. You don't even have to have the top spot or even show up on the first page of Google in order to have a business that's crushing it with growth. We can show you how, so reach out to the DoorGrow. That's it for today. Until next time, everybody, to our mutual growth. Bye, everyone.

Have you ever played the board game, Monopoly? Were you successful at buying properties, and charging people rent? Did you go from buying and selling the little green houses to bigger houses? Did you dream about becoming a successful real estate agent, making billions, winning the game, and retiring at an early age? You’re not alone.  Today, I am talking to Pat Hiban, a real estate agent who got better over time to have an illustrious career in the real estate sales business. Pat practiced what he preached and like most agents, bought houses and then rented them out. At 46 years old, Pat retired from selling homes for commissions to living off the income he made from the real estate that he purchased.  You’ll Learn... [02:45] Labeled as Learning-Disabled: How Pat overcame it, and didn’t let it bother him. [03:35] Go Getter: Don’t reinvent the wheel. Listen and copy others to sell houses.  [04:09] Done is better than perfect: Things don’t need to be perfect, but need to get done. Hire others to make them perfect and fix problems. [05:58] Building a Billion-Dollar Business: One sale at a time, one staff member at a time, one commission at a time. Get rich quick is a slow process and takes discipline. [07:54] What holds people back from growing their business? Themselves. There's someone else that has the same goals, but there's no difference between them. [11:00] What’s going to happen? You're going to quit affirming and focusing on your goals, or they’re going to come true.  [13:25] Unwilling to Give Up: Entrepreneurs tend to have tenacity and relentlessness.  [14:31] Are they not setting goals? Or, are they setting goals and failing? If they don't have any goals, they're never going to get anywhere.  [15:30] GoBundance: Find accountability partner for positive peer pressure to set goals, create affirmations for each goal, and make sure each goal and objective gets done. [19:42] Why people fail to succeed? They give up too soon and don’t establish proper mastermind.  Tweetables Stick with Superpower: Getting business, doing business, and making money. Done is better than perfect. To get rich quick is a slow process. Get rich slowly to succeed.  Your circumstances are a direct result of your goals and how often you review them. Resources Pat Hiban on Facebook Pat Hiban on Instagram GoBundance Tribe of Millionaires Think and Grow Rich by Napoleon Hill Robert Kiyosaki The Secret Movie Jim Rohn DoorGrowClub Facebook Group DoorGrowLive DoorGrow on YouTube DoorGrow Website Score Quiz Transcript Jason: Welcome, DoorGrow Hackers to the DoorGrow Show. If you are a property management entrepreneur that wants to add doors, interested in growing your business and life, and you're open to doing things a bit differently, then you are a DoorGrow Hacker. DoorGrow Hackers love the opportunity, daily variety, unique challenges, and freedom that property management brings. Many in real estate think you're crazy for doing it, you think they're crazy for not, because you realize that property management is the ultimate, high trust gateway to real estate deals, relationships, and the residual income. At DoorGrow, we are on a mission to transform property management businesses and their owners. We want to transform the industry, eliminate the BS, build awareness, change the perception, expand the market and help the best property management entrepreneurs win. I'm your host, property management growth expert, Jason Hull, the founder and CEO of DoorGrow. Now, let's get into the show. And today's guest, I'm hanging out with Pat Hiban. Pat, welcome to the show. Pat: Good to be here, Jason. Thanks, man. I appreciate it. I'm excited to be on DoorGrow.  Jason: Give everybody a little bit of background on you and how you got involved with real estate. Help them understand who Pat is. Pat: That's a big question as far as who Pat is. It's easier to say how I got involved in real estate. I went to college and I got a degree in sociology. I was going to be a probation officer and I couldn't get a job. What happened was I became an agent, a real estate agent, a poor one in the beginning. I sold 10 house in my first year, made $13,000. Over time, I got better, and better, and better, and I went on to an illustrious career in the real estate sales business. I did practice what I preach and like most agents, I bought houses along the way and then I rented them out.  I played monopoly a little bit, sold the little greenhouses, bought bigger hotels, shopping center, lots of apartments, things like that. Then, at 46 years old, I retired from selling real estate homes for commissions, and just live off of the income from the real estate that I purchased currently. Jason: One of the things in the bio that you've mentioned is you're labeled with a learning disability at the age of eight. Maybe you could share a little bit about that and how you overcame. Pat: Basically, I was learning-disabled. It was all a label. At that point, just like anything, I didn't let it bother me. When you're 8 years old, or even 10, or even 16, you're not conscious of any of that. You're really unconscious of it until later in life when your parents tell you about it.  I got a 2.3 GPA in college. I didn't really think of myself as being really smart. I really saw myself more as a go getter. Someone who would actually be able to do whatever somebody told me to do. My office managers would tell me, "Pat, this is what you need to do to sell a house or to get a listing," I would actually listen where 99 of 100 other agents wanted to try it their own way or reinvent the wheel. That's how I grew everything in my life. It's just by copying off other people. Jason: You've had a lot of success where a lot of other agents haven't been able to experience success or they eventually folded in and just gotten out because they just couldn't make it. What do you attribute to being different? Is it just that you would listen and learn? Or do you have a little bit more tenacity and bite than most people? Pat: One of my favorite quotes is, "Perfect is the enemy of done." I never really had to have things perfect, but I always had to have things done. I think that served me in that I would get them done. If they weren't perfect and there was a problem, I would hire other people to make them perfect for me so that I could stay in my superpower, which would be getting business, doing business, making money.  It was like what Robert Kiyosaki always says, "The B students works for the C students." That's true with me, I think. I'll be able to get it done. I'll be able to come up with the idea and implement a copy of it to somebody and implement it, then just hire other people along the way to make it perfect or better. Jason: I love that idea of, "Perfect is the enemy of done," which is funny because I say to my clients of a whole training video that talking about it and getting their websites launched. I say, "Done is better than perfect," because once it's done, it can make money. It can do its job. If you're waiting for perfect, it takes forever. Let's get into the topic on hand, which is building a billion dollar business. How do you build a billion dollar business? Pat: How do you build a billion dollar business? One bite at a time. That's like an elephant [...]. It's crazy. With me, it's just one sale at a time, one staff member at a time, one commission at a time. With regards to properties and property management, it's the same thing. One unit at a time, one door at a time. You're just building on that. That would be the answer to the question.  So many people today want to get rich quick. The truth to the matter is to get rich is a slow process. You got to know how to get rich slow. If you know how to do that, you're going to succeed. About a decade ago, there's a movie out called The Secret. The whole half of the movie was talking about what you need to do to become a millionaire is to sit there and basically just tell yourself, "I am a millionaire. I am a millionaire. I am a millionaire." There's a great quote by Jim Rohn. He says, "Affirmation without discipline is delusion." What Jim meant by affirmation without discipline is delusion is you can sit all day and be like, "I am a millionaire. I am a millionaire," but at the end of the day, if you don't earn a dollar and save a dollar, you're never going to have a million. It really should be, "I save $10 a day, I save $10 a day." Or, "I earn $20 a day and save half." Whatever it is, the point is, you need to add discipline.  Jason: For those listening, they're struggling in their business or they’re wanting to grow their business, what do they need to realize that it's maybe holding them back? Pat: Themselves. The answer is themselves. There's someone else in another state, another country, that has the same goals, and aspirations as them, that's so far ahead of them already this year. There's no difference between them. As a matter of fact, that person somewhere else may be disadvantaged compared to them in some way. Meaning, they don't have the money, or they don't have the skills, or they don't have the degree, or they aren’t the right race, or the right sex, whatever the case maybe. They may be disadvantaged in many ways, but I guarantee you that there's tons of amount there that are way ahead of you with the same goals as you and there's no difference between you two. Jason: Yeah. We could hold on to our story and excuses or we can get results. We're the one creating our own blindspots. If we're the ones that creating our own blindspots, we're the ones that's holding ourselves back, then how do we see that? Pat: What you have to understand is how psychology and how people are raised, and how most people are raised. I'll speak for America or North America. The average two-year old boy hears a negative statement from his or her parents or people older than him 16 times for every one positive statement. They might tell that little boy, "Don't touch that." "You're doing it wrong." "Wipe your face, you're messy," anything that's negative. None of that is positive.  By the time you're 18 years old, your subconscious mind is conditioned to believe that you can’t do stuff because you're doing all these things wrong. The only way to reverse that effect on your subconscious mind is to work on your subconscious mind. That's where you basically take goal-setting to whole another level where you actually set goals which everybody's listening to this probably has goals set. You reduce those goals to ridiculous.  I just talked about earlier, whatever it is you want to do, let's say you want to buy a house once a year or buy a house a month, that means you need to look at 20 everyday. You set your goal to that. Then, you create an affirmation around it for your subconscious mind that says, "I analyzed 20 deals a day." If you analyzed 20 deals a day, your mind believes that you're supposed to be analyzing 20 deals a day, and your mind believes that you're supposed to be buying one house a month, then it's going to happen. You can't help it. Either one of two things are going to happen. I guarantee it. Either you're going to quit, meaning you're going to quit affirming, you're going to quit reading your goals, you’re going to quit focusing on your goals. Or number two, it's going to come true. I believed that if you focus on that goal everyday, whatever it is, buying a house, and you focus on what you need to do to get into that goal everyday, it will happen. You will actualize it. I think that's how you overcome the subconscious mind of yours that’s not believing that you’re worthy, not believing that you ever will be a millionaire. I never really had much of a doubt that I would do well, that I will be rich. I was lucky and I was naive enough. A lot of people struggle with that. They don't have that naivety. The way to work around that is reprogramming your subconscious mind but not just in glorious goals. Not just in big goals, but in how you're going to actually act to get to that big goal. Jason: I like this idea. You're saying if you have a big goal, you have to break it down into the smallest action, the action that you're going to be taking on a daily, consistent basis. Then, you create an affirmation connected to this. That affirmation is just basically that you're completing this microcommitement, this action. Like, "I'm going to cold call this many owners to see if they're out of state. To see if I can get them on for business. I'm going to do whatever." It needs to be a daily, consistent, action. I'm going to go to this many real estate network. I'm going to commit to that. Breaking down into the smallest action, "I'm going to take this many agents out for lunch and have a conversation with them. Hopefully, we meet the referrals." They need to start setting some micro commitment and creating affirmations that they're saying regarding these to affirm that they're doing it. Then they need to live with integrity and take action towards those affirmations. Pat: Absolutely. Jason: Say, somebody's doing the affirmations. They're believing in themselves. They're taking these micro commitments. Then you said they're either going to quit or it's going to come true. There's this tenacity that I sense in you, this relentlessness, that I think a lot of entrepreneurs carry, that they're just unwilling to give up. If you're unwilling to give up, eventually, the universe just got to cave to you because you're relentless. Eventually, you're going to get it. Pat: And giving up is hard. You don't want to give up on the ultimate goal, but you’re going to have to change how you get there. Things are going to pop up on your way. You're going to have to go around them. Some people would say that would be quitting but it's not really quitting. You're just doing things in a different way all the time. Jason: Right, like course correcting. Pat: Course correcting, yeah. Jason: You've worked with quite a few different entrepreneurs and business owners. What advice would you give to those listening that you would typically give out for those that are wanting to move towards goals and they're struggling to figure stuff out on their own? What would you recommend to them? Pat: Are they not setting goals? Or are they setting goals and failing? Jason: That's a good point. What if they're not setting goals? What if they don't have any goals right now? Pat: Silly. Then they don't have any goals. They're never going to get anywhere. I have goals since day one. I can't imagine life without goals, even today. Most people don't have goals. That's why they're in a situation that they are. Your circumstances are a direct result of your goals and how many times you review your goals.  Jason: Got it. First off, they've got to set some goals, then they need to review these on a regular basis.  Pat: Daily. I would add something. Maybe have an accountability partner. One of the things we do at GoBundance, The Tribe of Millionaires is we have what we call peer partners which are people in the tribe that keep each other accountable. If they're goal is to call 20 out of state owners everyday, they text them, and say, "Did you call 20 today?" Then, we have GoBuds, which are about four to five GoBros that are in The Tribe of Millionaires that meet on a bi-weekly basis to talk about their goals, talk about where they're at, what they've done, and what they haven't done, that sort of thing, and it works. The point missing would be the accountability aspect. Not only set goals, not only create a subconscious affirmation for each goal big and each goal small, meaning the act-oriented goals, the discipline-oriented goals, but bring accountability around those discipline-oriented goals to make sure that they get done. Jason: Got it. They need to be accountable with somebody. If they're accountable to one, then the likelihood of them actually it is probably none. Pat: [...] works so well. Jason: It's probably because they have a coach, right? Pat: Yeah. They have to go in and step on a scale every week or every day. They have to write down and track what they put in their mouth. If you do that and someone's looking at it, it works. But if no one's looking at it, you're not looking at it, you're not stepping on the scale, and you're not writing down what you eat, chances are you're not going to lose weight.  Jason: Yeah. I worked out with a trainer for a solid year to get in shape. He had me fill out a spreadsheet. Every time I showed up (like once a week), he was pinching me with things to measure my body fat. There was no hiding. He was like, "I could tell you didn't eat right this week," or, "I could tell you're not getting enough sleep because you're retaining water." He’s just tell these stuff. He's done these with so many people. Same thing with working with any business coach that I've worked with. There's this level of accountability, that I'm checking in with them. I know I'm going to be talking into them and say they're going to ask me, "Did you keep your commitments? Did you do what you said you're going to do?"  I think there's that positive pressure. We're so good at applying negative pressure to ourselves. I think it's rare for us as entrepreneurs to apply pressure in a lateral or a positive way among our peers or among our people that their goal is to level us up. We firmly are really good at attracting people around us to tell us that we can't do things, that it's difficult, that maybe we should get a job. We've all heard these things as entrepreneurs. We really do need to have some sort of accountability. We need friends, we need partners, we need those that are in our corner. We need a coach, we need mentors. We need people that believe and can support us in our objectives. Pat: I agree. That's why we created GoBundance. That's why we do what we do and why there are over 220 members now, why our retention is extremely high. It's just because of that accountability piece. Your life just amplifies when you put it out there in front of other people. Jason: Got it. I love the idea of adding an accountability partner. It’s a simple buddy system.  Pat, I appreciate you coming here on the DoorGrow show. Is there any other advice you'd love to share related to how people can get out of their own way, start working towards building the legacy that they want, and building the finances that they want? Pat: I'm sure everybody here has heard of the classic book, Think and Grow Rich.  Jason: You said, Think and Grow Rich by Napoleon Hill? Pat: Yeah. What Napoleon Hill did is he just went around rich people. He asked them, "How did you get rich? What are your habits?" There was a newspaper article that said Napoleon Hill had to break it down into two things. He actually asked for one thing, "Why people fail to succeed?" He said, "I don't have one but I have two." He said, "The first thing is they give up too soon." They're about to hit the gold and they stop digging. He said, "The second one is they fail to establish a proper mastermind." He was the one that came up with the mastermind concept. He had a mastermind with Harvey Firestone, Thomas Edison, Henry Ford. These are all big name people. They would hang out, grilled marshmallows at a fire, and share secrets.  That's how we came to write our latest book which is Tribe of Billionaires. I want to give everybody on the show an opportunity to get a copy of this if I could. You can get a free copy by going to tribeofmillionaires.com and all you've got to do is pay the shipping. It's a story of a guy who loses touch with his father. For 20 years, he doesn't see his dad. Then, his dad dies and he has to settle the estate. He sees the pallbearers of his dad's coffin. They're six guys. They're all billionaires and multimillionaires.  He scratches his head because he's like, "I thought my dad was a deadbeat. How are his pallbearers billionaires?" Then, he's lucky enough that in order to get his estate, his dad wants him to spend a week with all these rich guys. What ensues are lessons that he learned. He journals about these lessons after spending or during his weeklong time with these six pallbearers. That was what Tribe of Millionaires was all about. You can get it on Amazon for $20. You're welcome to have it for $7 or free. All you're doing is paying $7 shipping. You just go to tribeofmillonaires.com. Jason: Perfect. All right, I appreciate that. Check out tribeofmillionaires.com. It sounds like a really good story that teaches some lessons regarding money, finances, and growing your business. I appreciate you sharing that, Pat. Any other words you want to share before we let you go? Pat: Nope. I'm easy to find. Luckily, my last name is not really popular. Just type in Pat Hiban. You can find me in multiple places. Follow me on social media, Instagram, Facebook, everywhere. Jason: Perfect. Pat, thanks for coming on the DoorGrow Show.  Pat: My pleasure. Jason: There you have it. Check it out and get a free copy of the book. You said you can go to tribeofmillionaires.com.  If you're a property management entrepreneur that's wanting to grow your business, if you're looking to connect with other entrepreneurs, wanting some accountability from me as a coach, and some support, I recommend you reach out to DoorGrow. We would love to help you grow your business.  Until next time, to our mutual growth. Bye, everyone. You just listened to the DoorGrow Show. We are building a community of the savviest property management entrepreneurs on the planet, in the DoorGrow Club. Join your fellow DoorGrow hackers at doorgrowclub.com. Listen, everyone is doing the same stuff. SEO, PPC, pay-per-lead, content, social, direct mail, and they still struggle to grow. At DoorGrow, we solve your biggest challenge getting deals and growing your business. Find out more at doorgrow.com. Find any show notes or links from today’s episode on our blog at doorgrow.com. To get notified of future events and news, subscribe to our newsletter at doorgrow.com/subscribe. Until next time, take what you learn and start DoorGrow hacking your business and your life.

00;00;14;09 [Jason]: Jason Watson with WCG Incorporated here in ColoradoSprings, we're a local tax and accounting firm. Joined by RachaelWeber and Joseph Bassett, both tax professionals for us. We'realso hosted by Axe and the Oak here in Colorado Springs, they'vebeen gracious enough to open up early for us, part of our Bourbonand Business00;00;31;29 series, podcasts and videos. We just got done wrapping up a videoand podcast on some of the bigger deductions that we see, cars,that's always a big one for most small00;00;42;29 business owners, meals and travel. We're going to talk this time or,this time around about home office and then all the other likegoofier ones, if you will.00;00;54;25 So, you know, tell me the rules, Rachael, on the home officededuction.00;01;00;04 [Rachael]: It's got to be used regularly and exclusively00;01;03;27 [Jason]: Okay.00;01;04;26 [Rachael]: In your home.00;01;05;13 [Jason]: Okay, regular and exclusive and have a business.00;01;09;01 [Rachael]: A Business purpose.00;01;09;14 [Jason]: That's probably true for every deduction on a plan, right?00;01;12;18 [Rachael]: Yeah.00;01;12;27 [Jason]: For a business deduction to be a legitimate businessdeduction it has to have a business purpose. So, use regularly andexclusively. So, can you break those words down for me? What's"regular" mean?00;01;23;15 [Rachael]: "Regular" means you would be checking your emails,invoicing your customers, doing administrative work. Okay. Meetingwith clients, holding your inventory. It could mean a whole host of00;01;37;27 [Jason]: Right.00;01;38;03 [Rachael]: of things. You're just doing that on a regular basis, notonce a month00;01;42;26 [Jason]: Right.00;01;43;14 [Rachael]: but on a regular basis.00;01;44;21 [Jason]: Yeah. And one of the words that the IRS will also use too is"continuous", right? This is regular and continuous, it's, it's, got alife, you know, it's got a cycle.00;01;53;28 [Jason]: So yeah, absolutely, regular is a big deal. We have folksthat have a rental, one rental, you know, they have a W-2 job, theyhave all those things and they're trying to say, I have a home officeto manage my rental. It's just never going to happen. Now, if wehave 10 rentals, 6 rentals, that's all you do is manage your00;02;11;02 rentals. We have some people that have 3 or 4 VRBOs or Airbnb,short term rentals and that is all they do.00;02;18;24 [Rachael]: Time consuming.00;02;18;29 [Jason]: Their working that stuff 100% so yeah, so that's regular.How about exclusive Joe? Joseph? What's exclusive?00;02;24;27 [Joseph]: So, let's say you have an extra room in the bedroomthat's you want to use for your home office and it also can't be yourtheater room. So you know, that's gotta be exclusively used forbusiness.00;02;33;20 [Jason]: What if you're a videographer and the theater is yourbusiness? I'm teasing you.00;02;38;03 [Joseph]: Well, you have an argument there. Or, or00;02;38;24 [Jason]: No, but you can't mix the use, yeah00;02;41;07 [Joseph]: Right, unless you run a daycare out of your00;02;42;13 [Jason]: Right.00;02;42;21 [Joseph]: House as well.00;02;43;05 [Jason]: Yeah, and daycare has its own special rules and this is notthe podcast for that because00;02;47;27 [Joseph]: Right.00;02;48;01 [Jason]: I don't know those rules by a memory. I look them up oncein awhile when I have to, but that's it. But right, those are some ofthe shared use stuff can be daycare. Other than that, it's regularand exclusive with a business purpose. So, tell me some of thebank, the benefits of having a home office00;03;05;25 deduction or home office reimbursement.00;03;08;10 [Rachael]: Reimbursement? Is that part of your mortgage interest?00;03;13;15 [Jason]: Okay.00;03;13;23 [Rachael]: Your real estate taxes,00;03;15;05 [Jason]: Okay.00;03;15;19 [Rachael]: Utilities.00;03;16;21 [Jason]: Okay.00;03;17;09 [Rachael]: Could become a small deduction.00;03;19;21 [Jason]: Okay.00;03;21;11 [Rachael]: For you, a business deduction.00;03;21;18 [Jason]: Yeah absolutely. And what are some of those expensesthat aren't otherwise available to be deducted? And mortgageinsurance, we say yes, right?00;03;27;29 [Rachael]: Mm-hmm00;03;28;14 [Jason]: Schedule A property taxes, we say yes, but how about theother ones?00;03;31;04 [Rachael]: Utilities, insurance00;03;33;19 [Jason]: HOA dues.00;03;34;21 [Rachael]: Yeah. Mm-hmm.00;03;35;24 [Joseph]: Repairs. Okay, so suddenly those become deductible andin a world where otherwise it wouldn't be.00;03;40;11 [Rachael]: Right.00;03;40;21 [Joseph]: Yeah.00;03;41;01 [Jason]: Okay, and how do we calculate that home officededuction?00;03;45;17 [Rachael]: Well we do it by square footage.00;03;48;07 [Jason]: Yeah, that is probably the most common, is squarefootage. You could do it at room by room, the IRS allow that, theyactually mentioned that in Publication, what? 587, or whatever it is.But I've never seen anybody do room by room.00;04;00;15 [Rachael]: No.00;04;00;20 [Jason]: It's always, usually, I shouldn't say always, but usually it'ssquare footage, yeah. So what's the basic calculation? It's thehome office space divided by00;04;09;24 [Joseph]: Total space of the house.00;04;10;26 [Jason]: Yeah, the total space of the house. What if you use yourgarage, then what do you do?00;04;15;08 [Joseph]: You include it.00;04;16;12 [Jason]: Include it where?00;04;17;08 [Joseph]: In both.00;04;18;01 [Jason]: In both numerator and denominator? Yeah, exactly. So ifwe're going to take the benefit of the garage and it's not otherwisein the denominator then we have to add it in.00;04;29;26 [Rachael]: Mm-hmm.00;04;30;02 [Jason]: Yeah, exactly. So, that's home office. What's this 50 milerule thing? Who wants to talk about that?00;04;38;14 [Joseph]: Right, so the 50 mile rule's, you know, kind of a safeharbor if you will, that if you know, your home office is within 50miles of your tax home then you can, you know, deduct expensesassociated with00;04;50;29 commuting from the tax home to the home office.00;04;54;17 [Jason]: Yeah, exactly. It's, they want, "they" being the IRS and thetax court, they want your home office to be, there's no written ruleon this, it's more of a contrived rule.00;05;06;23 But your home office needs to be within 50 miles of your tax home.Your tax home is where you earn your revenue. So, the greatexample, in one of the tax court cases, is a surgeon had a home inPennsylvania.00;05;23;05 He drove to New York, I believe, and it was 130 miles away. He wasattempting to deduct all those commuting expenses and becausehe was like, well, I got a home office. So then my commute is frommy bedroom to the basement. And then when I hop in the car, it'sall business miles, and of course the00;05;43;06 IRS and tax court said "No." They said it's too far from your taxhome, basically. So they dis, disallowed all those expenses asdeductible expenses and00;05;54;27 consider them commuting expenses, which is normally a personalexpense.00;05;59;03 [Rachael]: Mm-hmm.00;05;59;12 [Jason]: Non deductible. So, that's this 50 mile rule. What, youknow, talk to me about the audit rate risk for home offices and00;06;09;25 [Rachael]: [Inaudible]00;06;10;00 [Jason]: and, you've, and you've been doing taxes for a little bit oftime.00;06;14;07 [Rachael]: Just a little while.00;06;14;13 [Jason]: So tell me a little bit about the history.00;06;16;12 [Rachael]: It's kind of high. Yeah and it's, it's almost like they canwalk in and assume you're doing something wrong because they're,they're not easy rules. And you know, maybe the square footageisn't complete or they can say, Hey, what's with the day bed andyour home office?00;06;31;02 [Jason]: Right.00;06;31;13 [Rachael]: Or, and it's not just the deductions that you're getting,your utilities, your small amount of additional square footage, butit's that commuting miles00;06;42;07 [Jason]: Right.00;06;42;15 [Rachael]: That are, it's going to be pricey00;06;43;26 [Jason]: Yeah.00;06;44;04 [Rachael]: If its not done right.00;06;45;07 [Jason]: Yeah, absolutely. So, home offices, 20 years ago were notvery common, so it was a high audit rate risk.00;06;53;19 [Rachael]: Mm-hmm.00;06;54;11 [Jason]: Today telecommuters and all that stuff is a lot higher. Butnow we're back to not being seen very often because if you're aW-2 individual working out of your home office for a company out ofCalifornia,00;07;07;05 you would have to deduct that on Form 2106.00;07;10;19 [Rachael]: Mm-hmm.00;07;11;04 [Jason]: And those expenses, those deductions are no longerallowed. So, home office is almost been shrunk down to just forbusiness owners.00;07;18;02 [Rachael]: Yeah.00;07;18;29 [Jason]: So, how are we going to do that? Joseph, talk to, talk to usabout how we're going to do the home office from an S Corpperspective.00;07;28;20 [Joseph]: So, we'll use an accountable plan for the home office forthe S Corp and one of the reasons why we do that, so you know SCorp's are cash basis, you know, and00;07;38;07 [Jason]: Typically.00;07;38;23 [Joseph]: Typically, typically.00;07;39;14 [Jason]: Yes, small businesses enjoy using cash as their method of00;07;43;18 [Joseph]: Right. Accounting, it's simple. Depending on their grossreceipts.00;07;45;18 [Jason]: Yeah.00;07;45;27 [Joseph]: And we just, we have you record it, you know, for like, likeRachael said, your interest, taxes, insurance, and then you getreimbursed by the S Corp for your business use percentage of00;07;58;00 [Jason]: Okay.00;07;58;06 [Joseph]: Business expenses.00;07;59;18 [Jason]: So, just to back up for a viewers and listeners, anaccountable plan is the method used to reimburse people,employees for business use of their personal assets.00;08;13;03 [Jason]: Car, cell phone, home, are probably the biggest ones,right?00;08;16;05 [Joseph]: Mm-hmm.00;08;16;19 [Jason]: So, and we forgot to put cell phone down on our big list ofdeductions, but we can talk about that in a second. So, the benefitto that is we're getting reimbursed by our business. That expense iskind of tucked away on the S Corp tax return, using00;08;35;29 an S Corp in your00;08;36;29 [Joseph]: Mm-hmm.00;08;37;28 [Jason]: example as occupancy expense. Not that you can't defendit, not that we're doing anything wrong, but it certainly is not as highof an audit rate as filing Form 8829.00;08;49;29 [Rachael]: Mm-hmm.00;08;50;06 [Jason]: Which is clearly the Office In Home worksheet.00;08;53;12 [Joseph]: Right.00;08;53;20 [Jason]: That gets tucked on or tacked onto your Schedule C, if youwere to have a business only on your 1040. So, that just shrinksdramatically, the audit rate risk, from home office perspective.00;09;07;06 [Joseph]: And too, S Corp's already face a lower audit ratethemselves.00;09;10;14 [Jason]: Yes, 0.4% given I think 2017 data00;09;14;28 [Joseph]: Mm-hmm, 2017, yeah.00;09;15;02 [Jason]: Is the latest that we have now. So the IRS takes forever tocompile00;09;19;03 [Joseph]: Yeah.00;09;19;11 [Jason]: This stuff. I mean, I guess it makes a little bit of sensebecause audits take time00;09;23;07 [Rachael]: Mm-hmm.00;09;23;18 [Jason]: to generate and to do. But I still like to think we can live ina real time world. You know what I mean? Like we should know likeright now how many audits are happening. So, alright, let's talkabout commuting expenses. You know, you get up in the morning,you drive to WCG Inc, you know, is00;09;45;22 that an expense you can deduct?00;09;47;09 [Rachael]: No, it's not.00;09;48;01 [Jason]: Okay. Are you bummed out about that?00;09;49;20 [Rachael]: Yes, I am.00;09;50;08 [Jason]: Yeah, okay, we should write our Senators and ourCongress people. So, okay, so commute expenses? No. Even ifyou travel far, let's say you moved to Denver and you drove everyday down in the Colorado Springs, it doesn't matter, right?00;10;03;18 [Rachael]: Still personal, yeah.00;10;03;27 [Jason]: Right, so there's no like, Hey, we recognize that you'retraveling really far, we'll give you that deduction. There's nothinglike that. So, commuting expenses, parking, tolls, all that associatedwith going to00;10;16;18 your tax home if you will, are not going to be deductible. So, great,Country Club Dues, Rachel?00;10;23;14 [Rachael]: No, can't do it.00;10;24;15 [Jason]: No! Wow! Just hammered, boom.00;10;28;06 [Rachael]: Sad, yeah.00;10;29;14 [Jason]: Talk to me a little more about that. So we have someonewho has a membership somewhere, but they do entertain, shouldn'tsay that00;10;35;07 [Rachael]: Nope. Yeah.00;10;35;11 [Joseph]: Yeah, discuss business.00;10;36;08 [Jason]: They do discuss business at their country club.00;10;40;17 [Rachael]: Mm-hmm.00;10;40;20 [Jason]: How does that work?00;10;41;29 [Rachael]: Those expenses for the country club dues are going tobe personal.00;10;46;19 [Jason]: Right.00;10;46;25 [Rachael]: It's great that they're generating business00;10;48;29 [Jason]: Yes.00;10;49;07 [Rachael]: At the country club00;10;50;14 [Jason]: Okay.00;10;50;20 [Rachael]: but the dues are not deductible.00;10;52;01 [Jason]: All right, so this same member, buys a meal. The businesspurpose is clear. They00;10;59;20 [Rachael]: Yup.00;10;59;23 [Jason]: Were there to discuss business and now this individual isbuying a meal that's going to get tacked on top of his or her dues.How's that work?00;11;07;16 [Rachael]: That meal portion is going to be 50%00;11;10;14 [Jason]: Okay. Deductible as a business meal. Just, just like we'vealways done.00;11;13;06 [Rachael]: Mm-hmm.00;11;13;09 [Jason]: With meals. Okay, great. Talk to me a little abouteducation. Can you run education expenses through yourbusiness?00;11;20;21 [Joseph]: It depends.00;11;21;20 [Jason]: It depends, ah look just the classic accountant.00;11;24;28 [Rachael]: Yeah, maybe.00;11;26;00 [Jason]: Yeah.00;11;26;14 [Joseph]: If those education expenses are to improve your currentfield, then possibly. If they're to do something completely different,you know so if I was going to go to school to become a doctor now,which probably won't happen.00;11;37;13 [Jason]: Yeah.00;11;37;23 [Joseph]: But, those won't be deductible.00;11;40;03 [Jason]: Right, so the rule is it has to improve your current workskills. And you can even do, deducted a degree or even like, youknow, college courses, even if it leads to a degree, provided it'simproving your current00;11;57;20 work skills. So, you're absolutely correct, the other half of that is ifyou need it for certifications, like your continuing educations and allthat stuff. So, people who are CPAs have to go do all these, youknow, nauseating00;12;10;15 [Rachael]: [All laugh]00;12;11;02 [Jason]: Continuing Ed credits, you know, I'm sure we learned a lottoo, but you know, anyway, so, so that's education. How about yourchildren? Can you hire your children and consider them employeesand have the company00;12;27;10 pay for the education? Who wants to take that one?00;12;31;01 [Joseph]: I would say yes.00;12;32;07 [Jason]: I'd say no. [Laughs]00;12;34;09 [Joseph]: Like, the client advocacy in me would say Yes.00;12;38;13 [Jason]: Yeah.00;12;38;20 [Joseph]: Because of the, the relation though it will be disallowed.00;12;41;15 [Jason]: Right? Yeah, I was giving you a hard time. So section 127says if your child is 20 years or younger, they have attribution toyou as Mom and Dad being an owner of the company.00;12;54;21 If you own 5% or more of the company, you can't deduct thateducation.00;13;00;01 [Jason]: But if your child legitimately works, and is 21 or older, sowe're talking junior or senior00;13;08;24 [Rachael]: In college.00;13;08;29 [Jason]: If you're on a six year plan, you're a sophomore, right?Then the company can pay up to 5,250 a year, I think that's 2019limit. So, that might get index every year, like everything else. So,anyway that's education. How about client gifts? How do youhandle that?00;13;23;16 [Rachael]: Oh, they're $25 cap.00;13;27;08 [Jason]: Ahh $25?00;13;27;14 [Rachael]: I know, its really, yep. Mm-hmm.00;13;28;29 [Joseph]: Well they give you the $4 for gift wrapping, so00;13;31;16 [Jason]: And they give you $4 per pen or something.00;13;33;09 [Joseph]: Per pen, yeah.00;13;33;11 [Rachael]: That's advertising, yes.00;13;37;02 [Jason]: So, talk to me more about the $25 rule. Is that like all giftsor, or is it just for gifts to specific people?00;13;48;14 [Rachael]: It's gifts to a limited clientele. If you were handing giftsout to the general public and it was a lower cost, then that would beconsidered advertising.00;14;00;07 [Jason]: Okay.00;14;00;14 [Rachael]: And I think they give $4 for each advertising gift.00;14;04;21 [Jason]: Yeah.00;14;04;27 [Rachael]: Which I'm not quite sure what, you know, a pen or acalendar or something like that.00;14;08;26 [Jason]: Yeah, I don't know how much stuff like that costs either,yeah.00;14;12;12 [Rachael]: But your $75 wine basket is going to be a $25 businessgift.00;14;17;29 [Jason]: Yeah, and as I've seen it, read it maybe in Journal ofAccountancy, other things like that, but that's an individual limit. Soif you don't donate, or if you don't provide that gift to an individual, ifyou just do it to the business00;14;33;01 [Rachael]: Mm-hmm.00;14;33;10 [Jason]: There might be different rules00;14;34;03 [Rachael]: Yes.00;14;34;13 [Jason]: allowing you to take more deduction. So if you say, DearBob, thanks for all the business00;14;39;27 [Rachael]: versus staff at.00;14;41;03 [Jason]: Yeah, exactly.00;14;42;19 [Rachael]: Yeah.00;14;43;06 [Jason]: yeah, exactly. So, and you can see why, you know, theIRS is always worried about transfer of wealth without taxation.00;14;50;02 [Rachael]: Mm-hmm.00;14;50;12 [Jason]: Right? So if you, if you come in there with a bunch of clientgifts for one person it might look like a transfer of wealth. So, howabout professional attire? I am rocking the WCG.00;15;00;18 [Joseph]: That's true, very nice.00;15;00;29 [Jason]: On my shirt here. But tell me about professional attire.People will constantly ask you00;15;07;09 [Rachael]: Yep.00;15;07;28 [Jason]: I have to look good in my business suit, I have to have mynails and hair done, I have to rock, I have to rock this image.00;15;15;25 [Rachael]: And they're all personal.00;15;17;27 [Jason]: Yes, even though they're dead sexy, right? Even thoughthey're very good looking.00;15;21;21 [Rachael]: And necessary00;15;22;01 [Jason]: Yes.00;15;22;14 [Rachael]: Absolutely necessary. Yeah. So there's a businesspurpose behind it, but no tax deduction.00;15;26;09 [Jason]: Right. So what's the rule?00;15;28;07 [Joseph]: If it's not suitable for everyday wear00;15;30;00 [Jason]: Yes.00;15;30;11 [Joseph]: You can deduct it.00;15;30;20 [Jason]: So, if it's, yeah, so if you can, if it's suitable for everydaywear, easily convertible into everyday wear, then it's not deductible.00;15;38;08 [Rachael]: Mm-hmm.00;15;38;25 [Jason]: Right? Business suits are, you know, clearly somethingyou can convert to everyday use. We do have some, TVpersonalities.00;15;47;10 [Joseph]: Yes.00;15;47;15 [Jason]: We do have some models, you know, and we can, we canidentify some of that attire as costumes, something that theywouldn't, you know, be caught dead in. And that's true for some ofthese models, for sure.00;16;01;26 They wear stuff and they're like, I'm never wearing that in public. Itjust, it looks good on a cover of a magazine, but that's about it.00;16;08;15 [Jason]: Those are costumes, they're not suitable for everyday use.Those are something that we can deduct. TV personalities, they'llbuy, you know, a thousand jackets and they'll give them away andso those become marketing toys00;16;20;19 [Rachael]: Yeah.00;16;20;25 [Jason]: Or ploys or whatever, so absolutely. Let's talk about, perdiem and I'll just kind of talk about this real quick. Per diems a funnything. If you own 10% or more of a corporation and, and also theremight be some00;16;38;21 attribution there, where if your brother or your sister or your Mom or00;16;42;16 [Joseph]: Spouse.00;16;43;12 [Jason]: Whatever, then you are assumed to have the same,greater than 10%. If you are in that boat, you cannot take a perdiem reimbursement. So the scenario would be like this, I'm 100%owner of a corporation. I pay myself $71 a day for every day thatI'm in San Francisco, because00;17;02;15 that's the per diem rate. Let's say using 2018 numbers, I haven'tseen them, I haven't looked at per diem in a while cause we don't,we don't see 2106 expenses anymore. But, that would not beallowed. WCG Inc says, Rachel, we need you to go to, let's sayCortez, we really00;17;18;23 didn't like you very much. I'm teasing, Cortez is lovely. But, and wesay, Hey, we're going to give you $71 per per day that you're00;17;27;08 there for meals, that would be acceptable.00;17;30;11 [Rachael]: Mm-hmm.00;17;30;13 [Jason]: Now that will not be revenue to you. You maybe only spend$20, you know, whatever. You still get to take that $71 as tax freeincome.00;17;40;24 [Jason]: So, because you don't own 10% or more of WCG Inc.That'll change, you know, you'll own, own it all and00;17;49;07 [Rachael]: Eighty-five percent like you.00;17;50;09 [Jason]: Joseph, I'll be working for you one day, it'll be awesome.So, but that's per diem, per diem is a little tricky. There is the, themeals and incidentals component. There is the lodging component.The meals and incidentals component, as far as I know and read it,is00;18;06;24 available to Schedule C, Sole Prop, single member LLC types. Theminute you're a corporation or you act with a corporation through anS Corp election that gets tossed out the window.00;18;18;06 Lodging, regardless, is always going to be actual expenses. Youdon't get the high, low seasonal rates and all that stuff that you seein those per diem tables as a business owner. So, we ran throughhome office, all kinds of good stuff there. We ran through all kindsof other deductions that we get entertained with,00;18;38;10 quite literally, cause some people are pretty clever, right?00;18;41;22 [Rachael]: Mm-hmm.00;18;41;29 [Jason]: With, with their deductions. The bottom line is, people askme all the time and they ask all of us all the time, how do I save ontaxes, right? And the first thing I say is, look, your job is to buildwealth, not save taxes.00;18;56;12 We can save taxes along the way, that's great. But your job in life isto build wealth. Now, if you still want to save taxes the trick is tolook at what cash you're already comfortable with leaving yourbody.00;19;10;24 [Jason]: So go through your checkbook and try to figure out if therewas one thing that you missed or maybe this expense really didhave a business connection to it and I forgot that it did, or to digdeep. So, it's to look at the money that you're already willing tospend and try00;19;28;06 to find a business connection.00;19;29;23 [Rachael]: Mm-hmm.00;19;30;15 [Jason]: Now, I say find a business connection, like discover abusiness connection00;19;35;18 [Rachael]: Not create one.00;19;35;27 [Jason]: Not fabricate a business connection. So anyway, those,those are some of the other business deductions that we see a lotof: commuting expenses, country club dues, education, client00;19;47;20 gifts, professional attire, per diem, all that good stuff. We talkedabout home office in this segment as well. We didn't talk about cellphones. You know, cell phones, you know, folks will try to deduct100%, right?00;20;02;16 [Joseph]: Mm-hmm.00;20;02;21 [Jason]: "I use it for my business," oh, I know you use it for yourbusiness, I see that. But the minute you get a text saying, Heyhoney, you know, you're out of beer you should probably pick somemore up on the way home; and milk and eggs are low too. Nowyour cell phone's no longer 100%.00;20;17;04 [Rachael]: Mm-hmm.00;20;18;17 [Jason]: So, you know our firm-wide soft ceiling is around 80%, ifyou're a realtor, you're probably on the phone all the time. Peoplehave kicked landlines to the curb but still your phone is going tohave a high personal use and I, I believe, we believe as a firm, 20%is00;20;37;00 about the minimum there, meaning 80% is for business.00;20;40;26 [Jason]: Maybe you're a dentist, right? And you use your cell phoneoccasionally, you do have an office phone and all those otherthings, so maybe that's like 30% business use and 70% forpersonal. So, commonly we see cell phones being paid for by thebusiness and they00;20;58;19 truly are a mixed-use asset, so a mixed-use asset should be00;21;03;06 [Joseph]: Paid by you personally00;21;04;10 [Jason]: Exactly.00;21;05;00 [Joseph]: And reimbursed to you on an accountable plan.00;21;06;04 [Jason]: Yup. So, assets that you own personally should be paid forpersonally. If there's a business connection or use of that assetthen get reimbursed. No different than you working for Google andGoogle says, Hey, you know, drive down to the store, pick up some,you know, some pencils and we'll00;21;21;15 reimburse you. Well, you bring in a receipt and you're bringing inyour mileage log, and maybe you have to use your cell phone andall that stuff, and they would cut you a check for the business use ofyour personal stuff. So, anyway those are some of the common taxdeductions that we see here at WCG.00;21;36;06 My name is Jason Watson with WCG. I'm alongside Rachel Weberand Joseph Bassett. We're at the Axe and the Oak and this is a partof our Bourbon and Business series of podcasts and videos and wethank you for joining us and we'll00;21;51;00 talk to you real soon.

Are you a property manager or owner who wants to recoup financial losses when stuck with a bad tenant who stops paying rent or needs to be evicted? Lower your risk? Trust somebody else to manage your properties? Protect all parties involved?  Today, I am talking to John Higgins, co-founder and CEO of Steady Marketplace, a leading technology platform for property owners and managers. Steady’s subsidiaries offer financial products, including rent default insurance.  You’ll Learn... [02:00] Background of Big Financial Numbers: Starting with event-driven, distressed, and activist hedge fund managers with billions in assets.  [06:37] Steady’s products protect property owners/managers from bad tenant outcomes.  [07:40] Rent Default Insurance: Protection against rental income loss due to tenant’s failure to pay.  [10:15] Rent Default Insurance is widely available and adopted around the world. About 70% are renters and 30% are owners. [12:38] Collaboration Over Competition: Don’t simply copy-and-paste products and policies; leads to lack of innovation. [13:55] Automate It All: Learn from online lending space using technology to streamline processes, operations, and pricing. [15:05] Perfect Businesses are Out of Business: Entrepreneurs think they've got something perfect, only to realize they need to make it better.  [16:15] By the Book: Take regulatory issues seriously, and make sure to do it right. [17:00] Adoption is #1 challenge with any solution, software, or service.  [17:55] Competitive Advantage: Education, awareness, and understanding of product.  [20:53] FAQs: How does it work? Why does this exist? What’s the catch?  [21:55] Renter’s Insurance vs. Rent Default Insurance: What’s the difference? Tweetables Every entrepreneur should make a difference. Otherwise, they're just causing problems. When there’s a loss of rental income due to tenant default, there is no protection. Automate everything: Go slow to go fast. That's how the process works. It's constant iteration to get better, and better, and better. Resources John Higgins’ Email Steady Marketplace Steady Marketplace FAQ John Higgins on LinkedIn SureVestor Rent Rescue National Association of Residential Property Managers (NARPM) DoorGrowClub Facebook Group DoorGrowLive DoorGrow on YouTube DoorGrow Website Score Quiz Transcript Jason: Welcome DoorGrow Hackers to the DoorGrow Show. If you are a property management entrepreneur that wants to add doors, make a difference, increase revenue, help others, impact lives, and you are interested in growing your business and life, and you're open to doing things a bit differently, then you are a DoorGrow hacker. DoorGrow hackers love the opportunities, daily variety, unique challenges, and freedom that property management brings. Many in real estate think you're crazy for doing it, you think they're crazy for not, because you realize that property management is the ultimate high trust gateway to real estate deals, relationships, and residual income. At DoorGrow, we are on a mission to transform property management businesses and their owners, we want to transform the industry, eliminate the BS, build awareness, change perception, expand the market, and help the best property management entrepreneurs win. I'm your host, property management growth expert, Jason Hull, the founder and CEO of DoorGrow. Now, let's get into the show. Today, I am hanging out with John Higgins of Steady Marketplace. John, welcome to the DoorGrow Show. John: It's great to be here, Jason. Thanks for having me. Jason: John, you've got a really big bio and you're really impressive. Do you want me to read all of it? John: You can read whatever you want to read. I'm not that impressive. I'll say you're more impressive hosting this show and with your following in the space. I'm just a guy trying to make a difference. Jason: I appreciate it. That's what every good entrepreneur is trying to do is make a difference, at least I hope. Otherwise, they're just causing problems.  I'll read a little bit here. It says you are the co-founder and CEO of Steady Technologies Inc., a leading technology platform for property owners and property managers. Steady, through subsidiaries, offers financial products that benefit property owners and managers. Their first product is rent default insurance, offered in partnership with the top US insurance carrier that is a Fortune 100 company, rated A+ by AM Best, and S&P.  Prior to co-founding Steady, Mr. Higgins founded Nobadeer Advisors which provided business development and capital market expertise to technology-enabled lending platforms across the variety of consumers and business, lending verticals, and backed by top venture capital firms globally.  Prior to Nobadeer, Mr. Higgins spent 2.5 years at Prosper Marketplace, Inc. where he helped build the institutional loan program growing it from $0 to over $5 billion over his tenure and help scale Prosper's monthly origination volumes over 4000% during his time at the firm.  Mr. Higgins also previously served as a director at Topwater Capital, now owned by Leucadia, where he made investments between $5-$100 million to hedge fund managers across a variety of strategies via structured managed accounts.  Prior to Topwater, Mr. Higgins spent five years working for event-driven, distressed, and activist hedge fund managers with assets as large as $1.85 billion. There's a lot of big financial numbers here, John. A lot of big financial numbers. John: Want me to dive a bit deeper on it and summarize for you? Jason: Yeah. Let's dive into that and then tell us how you got into all of these. John: Sure. I can start from how I got into the hedge funds space which led me through here. I started and talk my way into an internship my junior college, totally unqualified, at the University of New Hampshire versus people that are top of their class from top business schools. Got a shot to join big hedge fund on my way up. I worked my tail off that summer and got a full time offer. I joined that firm full time after I graduated college. I was really lucky. I worked for the really brilliant entrepreneur there who would start this business with $500,000. Four years later, he grew it to almost $2 billion.  Then, left that company and went to Topwater where I was invested in hedge fund strategies via structured managed accounts, kind of cross the bench of the long, short, and distressed credit. That company was acquired by Leucadia which is now Jefferies Investment Bank; the two merged. Leucadia was at a big stake and Jefferies a long story anyway. As that transaction was transpiring, I was approached by the former management team across the marketplace who've I known from the hedge fund industry. They had great entrepreneurs that built and sold the company that served hedge funds called Merlin Securities. They're backed by Sequoia. Sold that business to Wells Fargo and decided they were going to take over Prosper.  They reached out and said, "We're looking for someone to help us build out this business as we take it over and turn it around." Really fortunate to work with tremendous entrepreneurs and the tremendous team there. During my time there, we went from about 50 employees up to about 600+ when I left. That was my first foray into more pure play technology.  We're a financial technology platform. We're offering unsecured personal loans online to end consumers. If you're thinking about going online, applying for a personal loan, no human interaction, [...] pricing, I can get you a loan in a matter of days as opposed to having to leave your house, go to a bank, et cetera, and fill up paper forms. After leaving Prosper, I was consulting for various lending platforms as you touched on in the intro. I got to work again with tremendous entrepreneurs across a bunch of different verticals. One of the people I've got to work with was doing some lending into the small landlord space. It's fix and flip lending and also rental lending. I started looking at the opportunities. I said, "This is really interesting. I know all of these products that helped multifamily owners protect them against bad tenant outcomes."  There's a lot of companies that pop up doing that, but no one's really going after single family. I started looking at the space and opportunity. As you and everyone else in the space realizes, it's actually bigger than the multifamily space. When you live in New York, everyone thinks rental properties are the big highrise. In fact, there's roughly more than 16 million single family rental units in the US, then another 8 million duplexes, triplex quads. All in all, you have about 20 million rental units in the US owned by individual investors that owned less than 10 units. These owners actually can't solve for this risk which is if the tenant goes bad. The smart owners are getting professional property managers or actually better at picking tenants at the established processes and procedures. They're getting bad tenants out. It can help manage those properties and have better outcomes. But still, when there’s a loss of rental income due to tenant default, there is no protection. In fact, my business partner and co-founder, Viken, had a property in New York City that he was renting. Person just skips town in the middle of the night. He was left with close to $20,000. It actually might have been north of $20,000 loss because the tenant just left the unit and didn't say anything. It took awhile to get it rerented. He had no coverage. If he had, it had no protection against that. If you had Steady or some of these other providers that are popping up, they could've indemnify themselves from that loss, and could've been made whole for a modest premium.  Long story short, there's a big need in the market to this type of product. What we're really excited about is working with all the property managers across the country to help ensure this is product underlying landlords and finding ways for everyone to win. Jason: Cool. Let's talk about the product specifically. Explain this to somebody that's never heard of this. They might even be an unseasoned property manager. Describe the problem that exists, that this solves for. John: Sure. When you look at it, if the tenant goes bad whether it's professionally managed or not—let’s suppose it’s some professionally managed properties; that's really who we're serving here in this podcast, and who we speak to—if their tenants goes bad, the owner's mad at them. They might've lose that door because guess what? They probably picked the tenant. They were entrusted by the landlord or the owner to find the tenant, to select the right tenant, and now the tenant's bad. So, the owner's mad, they might lose every relationship. The owner's also rental income. As a result, property managers also lost their property management fee income. Generally, they're charging based on the property management fee.  If you look globally, across Australia, New Zealand, and Europe, this type of insurance product, rent default insurance, is widely available and widely adopted. The reason is that, if you look in other jurisdictions, primarily Europe, it's flipped from the US. It's about 70% renter 30% owner. As we know, post financial crisis, more and more US consumers are now choosing to rent instead of own. So, the property management space is going to be larger and the rental property market is getting larger.  As this is occuring, we think that more and more people will be in need of this insurance because we have a growing market. The insurance itself indemnifies and there's different flavors. We'll speak generally about rent default insurance and what's out there as opposed to Steady, specifically. What we want to do is educate the market on the availability of these types of products.  Rent default insurance, generally speaking, indemnifies the owner against losses as a result of the bad tenant outcome. It could be eviction, tenant skips, et cetera; different programs to different coverages. What this does is it allows the owner who can't self-insure due to the diversification to recoup losses if they are unfortunately stuck with the bad tenant that stops paying rent or needs to get evicted.  Different people had different approaches to it. Us at Steady, we've taken a lot of the learnings from the online lending space using technology to streamline processes, operations, and try to deliver a great product that are at a reasonable price to the end market.  A lot of property managers are saying, "Hey, this is great. This is a huge concern that my underlying owners have. What happens if the tenant doesn't pay rent?" They see property management companies out there that have eviction protection plans or other plans. You've got the SureVestors, the Rent Rescues, and a bunch of other great companies out here, all serving for these types of risks and helping solve these pain points. The reason for that is this huge market is a huge concern. If you've got one property, say you own a home and you move for work across the country. You can't sell your home or whatever reason you have. You put it with the professional property manager. They're managing that, but you're relying on that cash flow for maintenance, upkeep, taxes, et cetera. In many cases, to pay the mortgage. If that tenant goes bad, all of a sudden, you're break even or your cash flowing property gone upside down and now you're coming out of pocket. You now have a liability that you have to come out of the pocket for every month. That's a big pain point, a big concern, and what these types of products do is solve for those types of risk, help landlords have peace of mind, and protect against bad tenant outcomes. Jason: You name dropped some of your own competitors, which is very generous of you. How does Steady standout or differ? How do you compare, standout, or differ in the space? John: We've taken a bit of a different approach on how we can structure our products and policy. A lot of other competitors, not just in space but in insurance generally, what they do is copy and paste what other products work on their markets or other products that other people have launched, and there's not a lot of innovation. As a result, we haven't seen a huge take rate for these types of product in the US.  What we found—you might feel differently—my business partner, Viken, grew up in Paris. What works in Europe doesn't necessarily work in the US. What works in Australia doesn't necessarily work for the US. What Viken and I did when we came together is we deconstructed how these programs work globally. We took a lot of the learning from online lending to build what we believe is a better program here in the US.  One differentiation is automation. Our entire process is fully automated. We just set an email prior to this event saying, "We are now in 20 states." We've got the ability to be in all 50 states. The reason we're not in all 50 states right now is because we want to automate everything. It is going slow to go fast. As we start to take it off here and ramp because the updates have been very strong, it's continuing to go stronger daily, everything will be automated. What that will result in is more efficient processes, procedures, and better pricing.  Jason: Explain what that means so everyone understands. You're saying that automation is a differentiator and that it's fully automated. What's automated? John: A property manager or a property owner can go online to the website, inquire about rent default insurance on their own, and complete the entire process in less than two minutes. There's no human interaction necessary and they could do everything themselves. Now, newer company, newer brand, we’re lucky to be aligned with the very strong brand in the insurance space, but nothing's perfect. As you know, as an entrepreneur, you think you've got something perfect and they realize you need to make it better. That's how the process works. It's constant iteration to get better, and better, and better. Jason: The perfect businesses are out of business. John: Right. We continue to constantly push new development releases and streamlining things. What we believe is that, if you can make the process as easy as buying, say for instance, travel insurance when you're buying a flight and make it that easy, that will be a great outcome for us and for this market. The way which you can do that is through API integrations, the right product structures, the right creativity, the right business development strategies, et cetera.  If you look at our product, where our technology is our technology, our product is our product, the two weren't built separately. They're built together. They work very closely together and in tandem. Because of that, it allows us to deliver a great customer experience, a frictionless process, high scalability, and keep headcount well. Right now, our biggest expenses have been legal and engineering, as you can imagine. It's a technology company, but legal because we invest heavily in making sure that we do everything right and by the book. Also, that our partners do things right by the book. As you know, the property management space has some instances where people have more of a cavalier or cowboy type approach that works until it doesn't. For us, we have ambitions to be a very large company and we operate in a highly regulated space. It's non negotiable for us to run into issues on the regulatory front or have our partners run into those issues. We take that very seriously and focus on in making sure everything is done the right way. Jason: That makes sense. The number one challenge when it comes to any solution or software or third party service is adoption. It's how easy is it for them to adopt this and use. If adoption is a challenge, then it's not going to work. It's not going to grow. People are not going to use it or it's going to be confusing or frustrating.  I'm a big Apple fan. Apple made adoption very easy. My AirPods, I just hold them out, open them up, my phone just show them on the screen, and they connect. It was magic, it's easy, I didn’t have to fill around weird Bluetooth settings or hold down buttons. What you're saying makes a lot of sense.  You've mentioned that it's easy for the consumer or for the property manager. One challenge that I see a lot of firms run into is when you're servicing an audience that's servicing that same audience. You almost can become competitors with them. How do you negotiate that? How does the property manager still have a competitive advantage against them just working with you directly? John: I guess, education, awareness, and understanding. People [...] this in massive market. People don't even know about this product. One parallel I draw frequently is pet insurance. I’ve got a pet, I’ve got a dog who's five now. I have pet insurance that I pay $70 or $80 a month. They haven’t got a good plan because the vet at the time said, "Hey, you should consider pet insurance if there's ever an issue." To me, the asset there is the pet. A little bit different than a rental property, maybe not as emotional as a rental property would be. They said, "Maybe you should look at this." It's a similar thing as what you're seeing happening in the property management space. Property managers are the fiduciary, the trusted advisor to the asset and the asset owner, which is the landlord or the small rental property owner who's contracted the property manager for their services. If they can be introduced to this product, it's for their benefit.  We don't have a big direct push. We're not looking to go after single family rental landlords directly. Our entire business model is predicated on partnerships. Based on our analysis, there's roughly eight million rental units in the US managed professionally. We've love to see that grow larger. Those are also, for us, we believe the best risk. As I touched on earlier, we believe strongly that property managers are better at picking tenants, have an established processes and procedures in getting bad tenants out, and they can get units rented more quickly. Jason: Which lowers your risk as an insurance provider. John: Correct, which results in better outcomes from the underwriting perspective.  Jason: Okay, makes sense. Your interests are aligned directly with property managers. They're your focus.  John: Yes. They are our focus. We just did a giveaway today to property management conference for people that could enter. We view property managers as our partners. Again, the reason I mentioned some of our competitors earlier because the rising tide lifts all boats. We want to see everyone do well, we want to see landlords have access to the solution so they get better outcomes, and we want to see property managers to be able to benefit from this as well.  Jason: Yeah, I love it. I believe that too. I have said before, rising tide raises all ships, but sometimes the bar is so low in property management in some areas and in some markets, that I don't think every ship's going to rise. Some have too many holes and are going to sink, but that's okay. John: That's right. That's Darwinism. Jason: Right, survival of the fittest. What are some of the most frequently asked questions or concerns that property managers are asking you or have been asking in sales conversations? So that we can make sure we address them here on this show. John: A lot of things that a lot of property managers ask is simply how it work. We have an FAQ section on our website and we can share the link on it. "How does it work?" "Why does this exist?" "How can no one else is doing is?" As I catch on, this is the third time I'll mention SureVestor, Rent Rescue, and others. The awareness is growing and that's what the biggest challenge is for all of us in this space is awareness that these types of solutions are available. This isn't like rental insurance or pet insurance. Pet insurance, I guess, is now becoming widely adopted, but people don't know about it and don't understand it. Most of the reactions we got is, "Wow, this exists? This is great. How does it work?" "Wow, that's inexpensive. This makes a lot of sense." It all depends on the property address, the rent amount, and the pricing. Jason: For anyone that's confused, let's just explain the difference between renter's insurance and rent default insurance. John: Renter's insurance covers the renter's possessions and liability to the landlord, generally speaking. It's paid for by the renter and they're doing it, so if there's a fire in the unit, they're not covered from the landlord's policy. Their possessions are gone. The landlord gets the unit rebuild, the house rebuilt, but they don’t receive anything. Now with renter's insurance, then we get some coverage for that.  From the landlord's perspective, if the renter has renter's insurance, they have a guest over, they slip and fall, and break their leg, it protects the liability to the landlord for them getting sued from that slip and fall. That's renter's insurance. Rent default insurance, it depends on the program. Different people, different features. Generally speaking, it covers loss of rent due to tenant skips, eviction, and tenant nonpayment for whatever reason. Jason: Sometimes, we have to make sure things are at an 8 year old level so that everybody gets it.  John: I generally need things at an 8 year old level to understand.  Jason: Right. Most entrepreneurs do because we're just so damn impatient at paying attention to things sometimes.  All right. We talked about how it works, why is anyone doing this. Any other frequently asked questions that people are concerned about? John: "What's the catch?" generally. Insurance companies, for better or for worse, generally don't always have the best reputation for making it easy to make claims, et cetera. That's another thing. Some people want to see the policies and see things in that nature.  Again, the big thing is people just don't understand these types of products exists. That's why we're out there educating the market and letting people know that there are these types of coverages available and you can get the coverage to these types of risks.  Jason: Let's touch on the benefits for a property management business in having this in their repertoire of services and how this can help them sell and close more deals, give them the competitive advantage, maybe. John: What do you see is property managers are now looking at this and some are saying, "I'm just going to include it in all my plans," and say, "This makes a lot of sense.” Now, we've got a differentiator. All of my property management packages include three months of rent default insurance if the tenant goes bad. They're out there marketing and saying that it includes it. Others are saying, "This is interesting. How can we offer this and earn some B revenue?" The only way it works, as I touched on earlier with compliance, is you can't get paid for the sales, solicitation, negotiation of insurance, unless you're an insurance producer. You can do other things such as marketing fees, et cetera, but you can't make conditions on the sale, solicitation, negotiation, and insurance. That's why we spend so much to make sure that anything we do, anything our partners do in partnership with us, is fully vetted and above board. We make sure everyone stays on the right side of the rules. Jason: Do they become somewhat of an insurance agent? Or you're just laying that all together? John: No. They do not become insurance agents in any way, shape, or form unless they've got an insurance agent license. Then, they could be an insurance agent, obviously.  Jason: Okay. John, it's great to see an entrepreneur doing something that's impacting the industry. I believe these products are going to have massive ripple effect in the industry. They're going to create a lot more safety and certainty in the property management space. It's going to lower the risk. It's going to lower the pain threshold for landlords to trust somebody else to manage their properties. It's going to protect all the parties involved and that means it's going to help the industry grow. If Australians, somebody said their markets are any indicator, it seems like these types of products help these markets grow significantly in a relatively short period of time, over a decade. They've grown phenomenally. I heard stats like Australia's grown through 25% in a decade. Largely, they claimed that it was connected to that. I don't know if that's accurately or true, but if that were true and the industry—single family residential—were maybe about 30% are professionally managed, that almost be our industry doubling here in the US. I don't know that there's enough companies here in the US right now to handle that level of growth. That would mean we need to double the amount of companies or we need to double the size of every company that exists. Something in between that. John: Or let's double the size of every company that exists. That'll be a good outcome for everyone. Jason: Yeah. Regardless, I want to make sure that we've got the best. Let's raise the tide. I appreciate that you're seeking to raise the tide. I think collaboration over competition is what builds market, it's what builds the category. It's always important to build the category before you try to build the individual brand. That's Marketing 101, everybody.  Property management is in the same boat. Property management has very low awareness, in general, here in the US and right now, we've got a lot of people going around something in their chest, trying to fill their individual brand. We need to build the category first. There's a lesson for the industry to take away from what you've mentioned and what's going on in what you're doing, so I appreciate that. John: NARPM’s done a good job trying to get the industry moving in the right direction. People like you and a lot of others that are trying to educate and build awareness are very helpful as well. It's great to see everyone working together in some way, shape, or form. Jason: There's no scarcity in property management. There just really isn't. There's 70% in single family residential that are self-managing right now. That does not indicate scarcity. In certain channels of marketing, there is a lot of scarcity because everybody's doing the same stuff, there is scarcity. John, I appreciate you coming in the show. How can people get in touch with Steady and learn more about this? John: They can go to the website www.steadymarketplace.com or shoot me an email john@steadymarketplace.com. Jason: Perfect. John, I appreciate you coming on the show, I appreciate what you're doing, and I wish Steady success. John: Thank you, Jason. Thanks for having me. Jason: Check them out at steadymarketplace.com. If you are, for some reason, not getting the growth that you want, you're growth is good, but you want to pour a little gasoline on that fire, if you find that you're getting a lot of your business lately from word of mouth, and from the trust that you built in the marketplace, I would love to pour gasoline on that fire. That's what DoorGrow specializes in, optimizing your warmly funnel and optimizing your business for more organic growth, which is a lot less expensive than showing up tens of thousands of dollars a year towards pay per click, SEO, and everything that everybody is competing and already doing.  Like I said, I don't believe there's scarcity in the industry, but I believe there's false scarcity that's been created by marketers, and you can avoid that. For those who can't see, I'm wearing my "SEO won't save you" shirt. A lot of people are relying on SEO to save you. Don't get me wrong, SEO is great. If you have the top spot in Google, that's great to have search engine optimization. But there are things that are better than having the top spot in Google like being the most trusted company in your market. Our whole system is focused on building trust for your brand, for your business, and helping you to go after that blue ocean where there's all that business available; that 70%. I appreciate John being on the show. Until next time, to our mutual growth. Bye, everyone.  

Keirsten's Social Media!! Twitter: @traprecruiter Facebook: @traprecruiter LinkedIn: @traprecruiter Instagram: @traprecruiter Keirsten's Resources!! One of the things that I offer through Trap Recruiter, LLC is career coaching. I will offer a 20% discount on for say, 10 of your listeners. If they sign up for career coaching sessions. Jason Hello, and welcome to the cavnessHR podcast. I'm your host Jason Cavness. Our guest today i Keirsten Greggs. Keirsten are you ready to be great today Keirsten I am. Jason Keisten is a talent acquisition consultant and career coach. In 2017 she founded Trap Recruiter LLC. A small business committed to bridging the gap between the job seeker and organizations committed to tracking hiring, developing and retaining diverse talent and fostering inclusive, equitable cultures. During her 19 year career, she has implemented creative recruiting strategies for Defense Intelligence, federal and civilian contract portfolio. That accumulated a full range of talent acquisition experience, including full lifecycle recruiting, executive hiring, talent acquisition operations, training, development and delivering. Talent acquisition, Product Management, college recruiting, internal staffing. Always focused on relationship building. she engages with a broader audience via her blog as a guest speaker and as a guest in various podcasts, facilitating workshops and training Keirsten, thank you for being here today. I really appreciate it. Keirsten Thank you for having me. Jason What are you focused on now? Keirsten Right now I do focus primarily on a contract that I have with a company to staff their global trade compliance group. So that is my quote unquote, nine to five job, that will be your regular full lifecycle recruiting. Then I still do some of my independent things on the side. I am grateful and so very thankful that this company is very amenable to my schedule, allowing me to continue to do Trap Recruiter work as long as their good work is completed. Jason A lot of recruiters like have a niche, they only do construction or tech. But it's like you have a broad and general based recruiter. Keirsten Initially, I was a tech recruiter, because I started in 1999. Because of the area that I live in, a lot of companies were popping up and the way that I got into government contracting, was that the company that was one of our customers. They actually did not have a recruiting department. So they brought three of us over to be their internal recruiters instead of paying us that heavy fee that they paid for each hire. So then they just paid us salaries, then it saves them a lot of money. Somehow I transitioned that into the more Intel space, or the Intel portfolios for the companies that I worked for. But again, that work was very broad. It could be in a executive hire, it could be a contracts person, it could be a security person, it could, it could be an IT person or an engineer. So the only difference was, in a lot of cases, I was adding a clearance to the requirements for the jobs I was hiring against. Jason Keirsten, you do both defense and civilian recruiting, correct? Keirsten Yes. Jason So what are some of the differences between the two is one easier or more challenging? Keirsten Defense is definitely more challenging. Again, because you only have so many folks that are not only used to the culture of a defense contractor. Especially, when you're talking about people that are going to be sitting on site at a government site, or at a military installation. So that is a little bit more difficult when you add the caveat that they not only have to know the programs, and the little nuances of the culture, but they also have to have a security clearance Jason Can you explain your passion for recruiting? Keirsten I love helping people, there is a wonderful feeling that you get when you help someone get a job. I know what it's like to be unemployed. I didn't at the time that I started recruiting. I thought it was going to be more like a sales type of thing. I had this idea, the same that I think we see on memes now where you know, it's like what people think I do. It's like this lavish lifestyle, we're on boats, and yachts drinking champagne, and that's not really it. We really are doing work. So I had this idea of what it was going to be like to be a technical recruiter. But the more I got into the actual strategy of it, being a relationship builder, building trust, helping people get a job. Creating the organizational culture with the people that I was hiring, making sure I was hiring the best talent that we could. That's what really, you know, drives me and gets me excited every single day, Jason What do most job candidates get wrong about working with a recruiter, Keirsten Sometimes they can be a little bit abusive. Sometimes, if recruiters don't set boundaries, or if they try to be everything to everyone, and that's not just the candidates, but the hiring managers, or the customers that they're supporting, recruiters can sometimes get burned out. They think that we don't have anything else to do but attend to them. Jason So some of them are expecting 24, seven access to you and that's not a reality, that's not realistic. Keirsten It's not realistic, and I do my best, as I've gotten older and more mature to really be the one that sets those boundaries and cuts the lines of communication after a certain time. Because people will try to have access to you 100%. Especially, now you can email me which used to be the only way where you could call me and then we have cell phones. So you can call me on my cell phone, you can call me at work, you can text me, you can email me. Now you can reach out to me on social media and tell everybody in the world that I did not respond to you in a manner that you know, as timely as you would like. Now you've embarrassed me, and you probably talked yourself out of a job because maybe I was busy. I just didn't get to you yet. Or maybe your timeline wasn't my timeline. There's a lot of ways that a recruiter has to be very, very aware of how they are managing their time and the people that have access to it. Jason I have to think if a candidate has a recruiter on their side, that's really the best thing for them. Because everyone will change jobs eventually. If you have a recruiter on your side. They call you and say, I'm probably leaving my job in six months. That has to be a value add for that person, I would think. Keirsten It definitely is. I think the biggest challenge that I have, from speaking in general terms. There are always those one offs, but it's with the people that know me personally, and my family is the ones that take the most advantage. Because I get crazy requests of things to do because I have certain accesses to stuff and I'm just like, no, that's not appropriate. Like, don't ask me to do that. I'm not doing that for you. Or, you know what, I don't have time to do that right now. Because I do actually have a job that makes me do the things that you're asking me to do. Jason Keirsten how often is it that a hiring manager gives you a job to fill. You send them some candidates based on the requirements and then the hiring manager says that they are changing the requirements. How often does that happen? Keirsten A lot. It happens a lot in organizations that are new, and I'm working in an organization as 100% new. I was actually hired to staff the organization completely. That group of people I should say because it is a large company, but this group is new. So there is not a lot of confidence in what they need. Because there's no precedent already set and the things that went wrong with the previous group there. I think they're far too focused on not repeating those mistakes. That they're not really looking at what went right and what can we replicate? How can we move things forward quickly. So it happens to me, unfortunately, a great deal where I'm revising things. I'm redoing things, most of the time, it's because of the level of the position that they described. They put out one thing, and I think they want it to get away lot of responses, or see what they get back. There's a lot of that, what will I get back, they cast a wider net, and then they start to draw it in. When we do find the right person, we end up having to do a lot of administrative work in the background to correct the level of the position. It is one of my pet peeves in recruiting because you don't level people, I feel like you level up level positions. Jason Most of the companies that start off hire the people they know. But that person probably isn't the best person to do marketing and they usually hire people that look like them. Keirsten Yes. Jason How do you convince people that maybe your best buds not the best marketing person. Maybe look for other people who don't look like you. How do you convince people to do that? Keirsten It's not easy. We talk a great deal about diversity, inclusion, and equity today, every single day. I have added. I believe in deliberate diversity, I believe in intentional inclusion. I think a lot of people are deliberate in wanting to have a diverse workforce. They're intentional in going to diversity, job fairs, or reaching out to veteran organizations or going to different conferences. Bringing in interns every summer. So they're intentional, but they don't empower and allow for equity to come into the organization. So that's why they lose a number of their diversity hires Whey can't retain diversity hires at the rate that they would like, and why the workforce remains, you know, very, very, very, for lack of a better term. Vanilla. Jason Tell us something about owning your own business that you did not expect. Keirsten I did not expect to do so much work that has nothing to do with recruiting. I have always been adamant that I did not want to be branded as a recruiting agency. But there are so many things that go into owning a business, but I just was not prepared for and I just was not ready for. Jason So from your point of view, what makes someone a great recruiter? Keirsten Number one, I think passion for recruiting, you really have to, to love it, you have to have a very thick skin. You have to be able to change your course very quickly. You're not going to be able to follow a script every single day. You're not going to be able to be a to z one to 10 methodical, methodical every single day. You are going to have to change your priorities over and over and over again. If you can become okay with that, with having multiple personality disorder and OCD pressed upon you, then you're going to be a great recruiter. Jason Keirsten how often or what percentage do people ask you to do something unethical? or illegal? How often does that happen? As a recruiter. Keirsten I don't get those because I come from an environment that was highly bureaucratic. Where we did have to follow a lot of rules. Sometimes I felt like there was a lot of self imposed governance that I myself was trying to get away from. Trying to get around to make my life a little bit easier, not that I was doing anything unethical or illegal. But I think some of the controls that we put into the process can sometimes make things more difficult than they should be. I don't get any outright blatant requests to do something illegal. If I get something that is unethical or that it's out of compliance, I'll say that is probably the word that I will use. If I get something that that's not compliant, we'll make it compliant. If at all possible, for example, we know that we want to hire Joe Smith for this role. But our policies and procedures say that we have to interview three people. So we interview three other people and we do that. But I'm also the one that's trying to make people be aware that no, I didn't give you just three people that I know you weren't going to hire. I gave people that could either build the pipeline or them may knock Joe Smith, out of the running for this position. I'm going to do my due diligence. So I don't allow any, anyone any organization to put me in a situation where I don't feel like I'm being morally sound. Now, I won't do that. Jason So from my point of view, I think it's easier to find a job nowadays. The reason because before when I was coming up, it was newspaper ads? But nowadays you can go to the website. Companies advertise jobs on Twitter, Snapchat, Instagram. There are all these places people can find jobs. So do you think it is easier or harder for people to find jobs now? Keirsten I think it's easier for people to get connected to companies. But it's not necessarily easy to fill the jobs. But if we're not filling them with the right people. I think it's, it's up to the organization's to make sure that we are doing enough to say what kind of workforce we're going to want in the future, not just for today. Jason Keirsten in your time in recruiting? What have you seen the candidates consistently do wrong? Then what do you do when you see the companies do wrong when trying to fill those jobs? Keirsten Candidates, I think they sometimes for me, they oversell just because of the industry that I'm in. I think they oversell themselves. They put too much on the organization in terms of what they're going to do for them. I think organizations are a little bit naive in the sense that they don't think that employees will ever leave them or that the employee needs them. So they're like you need I think that's the change that's happened in the workforce completely is that it's no more of what can you do for me, it's what can I do for you? Jason I mean, at will works both ways, right? Keirsten It does. Jason Can you talk a little bit how recruiters are paid? Keirsten Well, I have been a corporate recruiter most of my life, so I was paid like an employee. You get bonuses for meeting goals. There are third party recruiters that can be retained for specific roles. They would get a percentage of the higher salary paid by the organization. There are recruiters that get paid by job seekers to help them find a job who actually go out and look for jobs for them. There are recruiters who are on retainer who make a monthly or quarterly amount for supporting an organization and giving them candidates across different positions. Then there are recruiters who help people who supply contingent workers or temporary workers. They get paid a percentage of a person's bill rate, they get a bill rate, and they end that company pays the employee, Jason I'm guessing the best method depends on the recruiter and another variables. Keirsten It depends on number one the position. So there's an abundance, you will always have work filling, manual labor jobs. There's a lot of those, and it helps the organization and it helps, those third party recruiters stay employed to hire their people on a temporary basis. Especially, when you don't have an organization that's very competent in their hiring practices. It's better to do that, that temp to perm type of deal. So that you get to try and buy before you commit to the person. Those companies are the ones that are doing the work for you. So you again, they are they're keeping a pipeline. If Joe Smith doesn't work out, then they can send Amy Jones. Jason Keirsten, I understand you have something for our listeners today. Keirsten Yes. So one of the things that I offer through Trap Recruiter, LLC is career coaching. I will offer a 20% discount on for say, 10 of your listeners. If they sign up for career coaching sessions. Jason Keirsten, can you share your social media with us so people can reach out to you? Keirsten I can be found everywhere. Twitter, LinkedIn, Instagram, Facebook. Did I miss any? @traprecruiter Jason For our listeners will have the links her gift offer and her social media links at www.cavnessHRblog.com Keirsten we are coming to the end of our talk. Can you provide our listeners any last minute advice on any subject you want to cover? Keirsten For recruiters, I want to encourage all of us to please just be good people. We are the ones who are creating the culture and organizations. If we aren't doing our due diligence and bringing in the best people and doing good hiring practices. Then we are going to continue to have trash garbage companies like we have now. So let's just be better people. Let's not continue bad behaviors that we see all over an let's try and change the perception of what a recruiter is. Jason Keirsten, thank for your time today. I really appreciate it. You have done a lot of great things for everyone. Thank you very much. Keirsten Thank you. Jason To our listeners. Thank you for your time as well and remember to be great every day. Be sure to connect with us on LinkedIn, Facebook, Twitter, Snapchat, Instagram, Twitch, YouTube and TikTok at cavnessHR. Also check out our weekly live streams at the cavnessHR Facebook, Twitch, YouTube and Periscope, where we focus each week on an entire topic important for small business. Every Wednesday at 9 am PST. To join our weekly HR email newsletter list. Send us an email to jasoncavness@cavness.com Thank you and remember to be great every day. See acast.com/privacy for privacy and opt-out information.

How can eCommerce businesses reduce their time to first purchase by 10X? This week on The Inbound Success Podcast, WordPress developer and eCommerce expert Jason Resnick shares the process he uses to help eCommerce businesses dramatically reduce the time from first touch to first purchase. And while Jason works primarily with eCommerce businesses, the advice he shares is equally applicable to businesses in other verticals. From visitor segmentation, to behavioral analytics and content personalization, Jason goes into detail on the process he has used to help one client reduce time to first purchase from 40 days to 8, and for another client from 9 days to 1.  Some highlights from my conversation with Jason include: Jason is a WordPress developer and eCommerce marketing expert. According to Jason, one of the keys to reducing the time to first purchase is to capitalize on the positive emotion that a visitor feels when they discover your site for the first time and get them to explore further. One way to do this is by adding a widget to your site with related blog articles. Asking your visitors a qualifying question is a good way to learn a bit more about them and then use that information to tailor what you show them. With these types of questions, and then behavioral information like the articles and pages your visitors are looking at, you can create a lead scoring model. Based on the topics that a visitor is consuming content on, you can use that information to change the copy on your CTAs to make them more relevant to your visitors' interests. In eCommerce, the first 90 days are crucial. If you can't convince a new contact to purchase something with the first 90 days, the odds of ever selling to them drop dramatically. When Jason works with new clients, he begins by taking baseline measurements of how long it takes for a new contact to go from first touch to first purchase. By using segmentation, intent awareness, and personalized copy, Jason has been able to reduce time to first purchase for one client from 40 days to 8, and for another client from 9 days to 1.  When it comes to converting new leads into customers, Jason says it all comes down to trust, and you need to build trust into every interaction you have, from your website copy to your email marketing. Resources from this episode: Save 10% off the price of tickets to IMPACT Live with promo code "SUCCESS" Visit the Rezzz website Follow Jason on Twitter Connect with Jason on LinkedIn Listen to the podcast to learn exactly how Jason helps his client shorten their sales cycles - and how you can too. Transcript Kathleen Booth (Host): Welcome back to the Inbound Success Podcast. I'm your host, Kathleen Booth, and today my guest is Jason Resnick, who is the founder of Rezzz. Welcome Jason. Jason Resnick (Guest): Thank you for having me. I'm excited to be here. Jason and Kathleen recording this episode together . Kathleen: Tell my audience a little bit about what Rezzz is, and your background, and how you came to be doing what you're doing now. About Rezzz Jason: Sure. Rezzz is my business, it's what I've been doing for, this August will be nine full years, full time for myself. I am a solopreneur. I don't have a team behind me, but it's a web development business. I've always loved the eCommerce space, and the human behavior behind all of eCommerce. Where most developers and designers shy away from eCommerce so early on when eCommerce was ramping up in the early 2000s, I flocked to it, I was attracted to it. I built my business around helping online businesses, and I call them eCommerce, it could be anybody taking a transaction. I have nonprofit clients. I have online coaches. I have clients that sell physical products. Basically anybody taking a transaction, to help them get anonymous visitors into being customers, and then customers into repeat customers, and then repeat customers into raving fans. I do that through a number of different strategies and tactics, which most of them revolve around what's called behavioral marketing, or email automation, but also a mix of onsite personalization, that's where my skillset as a web developer come into play. Kathleen: Yeah. You know, it's funny that you say that about marketers shying away from eCommerce, because I've worked with a lot of marketers in my time. I was an agency owner for 11 years, and of course now I'm at Impact. It's true, I know a lot of marketers, and a lot of them say, "I don't touch eCommerce." It's almost like they're afraid to. I know one or two who do it, and the ones I know who do it have like gone deep, I think because there's such an opportunity, or a vacuum left by everybody else. Why do you think it is that marketers shy away from it so much? Jason: I think it really stems from there's so much tech involved with it, and it's so close to the bottom line that it's easy, I mean, and this is going to come out bad, but because there is a direct correlation that business owners see X dollars coming in per month, per day, or whatever it is from the site, when you say that you can affect that, they're going to see that result immediately. Most marketers, and obviously when we build campaigns sometimes those things take some time to build up, and sometimes you have to have those difficult conversations with clients a little bit earlier on in the eCommerce space than, let's say nonprofits, or standard brochure type websites, those things. I think that because of not just that, those difficult conversations that you might have to have, but also the tech side of things, if something goes wrong, if you're the point of contact and you don't necessarily know at a deep level what those technical bells and switches are, then you're going to be like, "Uh." With your hands raised and saying, "I'm not sure. Let's go see what we can find out from the tech team." I think at least from my experience, that's what customers tell me when they come into my ecosystem. They want somebody that, they may not know all the technical aspects of things, but they do understand that some things do take time, and they just want somebody that can take care of all of it for them. They don't want that ping pong match like, oh this is the host, and this is the developer, and this is the marketing side of things. They don't want that ping pong match, and they kind of just want that holistic point of contact person to be able to say, "Yes, there's a problem." Or, "Yes, this is what we need to do." At least from my experience I think that that's the reason why a lot of people shy away from it. Kathleen: Yeah. It's interesting because web design, development, et cetera, in general comes with a lot of high stakes, especially in this day and age when so many people find your business by your website. With eCommerce, as you rightfully pointed out, it's like way, way, way higher stakes, because your business is your website. Jason: Right. Kathleen: Your website is your business. Either way you look at it, if you break something, you're breaking the entire revenue stream of the business, not just like, oh our customers couldn't see our website today, or we didn't get another form fill. Jason: Right. Kathleen: Yeah, it's not an inconvenience, it's a major, major risk. I can totally see that. Now, and I should say, you came with very high marks, because I met you through one of my past podcast guests. This is one of my favorite ways to get new guests, is when former guests reach out and say, "Hey, I have somebody you should talk to." That actually happened in your case when Val Geisler, who I interviewed a few months ago, wrote to me unsolicited and said, "I really think you should talk to this guy." Val, for those who either didn't hear her episode or don't know, is an amazing email conversion copywriter, mostly for B2B SaaS companies. I have a tremendous amount of respect for her, and as soon as she wrote to me I said, "I'll talk to anybody that you think I should talk to." Jason: Thank you very much. Kathleen: Yeah. No, that was a good introduction. You do a lot of work, because you're in eCommerce, and what is interesting to me about you is that you're not just a web developer/designer. You work on some of the other aspects of eCommerce businesses, personalization, conversion optimization. How did you get from web design and development into these other areas? Jason: I think actually it, well my career took me in that path, but I think as a person it was the other way around. I've always been interested in human behavior. I got a minor in psychology in college. For me, and I went to college in the late '90s, and that was the advent of the internet. I mean, I remember going to the computer lab and building my first webpage. That was in like '96. Kathleen: I remember learning Basic. Jason: Right. Yes. Me too. Kathleen: I'm not going to say any more, because then that'll really date me. Jason: For me, when the advent of the internet came along, that was intriguing enough, because I was actually going for computer science at the time. At that time it was a lot of compiling code, and waiting for things to happen. Yet the web was like, I put code on a screen and I hit save, and I refresh and boom it's there. Hey, that's pretty cool. That intrigued me, but also my human nature side of things, just being perceptive of the world around me and kind of how people interact with certain things, and why they do what they do, and why they don't do what they don't do, always intrigued me. When the eCommerce world hit, pre Amazon, all the rest of it, people were afraid to put their credit cards in. Even online, let alone cellphones weren't even really a thing at that point in time. I was working for a consulting firm at that time, and we dealt with a lot of startups, and all of them wanted some sort of eCommerce in some sort of fashion. For me, it was always interesting to say, okay, if we use certain buttons in a certain way, and certain text in certain colors, we could create this, I don't want to say an artificial, but a perceptive environment of being safe. Where they can submit their credit card and not feel that they are sending it over and somebody's copying that down and running away with their identity. For me, that was the genesis of where I am today. I've always just kind of had that snowball effect, and really focus in on that specific part of my development skills. Because as you said, a lot of people were shying away from it, and I always knew that I wanted to work for myself. I had to find that niche, if you will, that sweet spot to really plant my flag in, and fit into a market that I could become known for. That was how I started all that. Just as the web evolved, now with email marketing, and how much data you can collect on somebody just by asking them a few questions, you can segment, you can promote certain things based around where a person is in their journey and their awareness. You can do all these things and marry things like your email marketing platform with your website with a little bit of code. Now there are services out there that can do this too, that your website can look completely different with two different people. It's all based around what you know about that person, where they came from, demographics, or even just what you know they clicked on in your last email. That's always been interesting to me because that's like the mom and pop of like the early 1900s, where somebody would walk into the store and you would have all your stuff ready because they knew you came in every Thursday. They knew who you were. For me, having that personalization and segmentation is what allows you, as the business owner, to know where your potential customers are, where your customers are, where your repeat customers are, and know how to cater to them in the best way possible. Kathleen: You know, it's fascinating that you just brought that up, because I literally just, as we're recording this, this morning published my latest episode, which was a conversation with Shai Schechter, who's the founder of a company called Right Message. That's exactly what he talked about, was his platform that he's built lets you ask your visitor a simple question like, what brings you here today? He actually equated it to the conversation you have when you walk into a shop. Like nobody is saying, "What industry are you in?" It's, "What brings you here today?" Jason: Right. Kathleen: Based on the answer to that, you can dynamically then update the copy on the page. He was seeing like 10x improvements in landing page and CTA conversion rates from that kind of like small amount of personalization. I definitely think there's something to it. Jason: Yeah, absolutely. I know Shai, I've known him for a couple of years. We've met at some events and things of that nature. Yeah, he's built a great platform. His platform's called Right Message, and I use Right Message as well for some assets of my business. Yeah, I mean it's that idea of, we've gotten away from that broadcast everything to everybody. Now we want to really cater to the one on one. That is what's going to increase conversions, and that's what's going to help you convert non customers to customers as quickly as possible. The more you know about them, the more that you can speak their language, the more that you're serving up the thing that they want at the right time, that's going to help you with your conversions. Reducing The Time to First Purchase Kathleen: Yeah. Now you, as you said, you do a lot of work in eCommerce, and one of the biggest areas of opportunity for optimization in eCommerce is how long it takes from first touch, if you will, with a lead or a prospect, to getting them to purchase. Time to first purchase. You've done some interesting work on shortening that time period, can you talk a little bit about that? Jason: Sure. Yeah. As you said, any time somebody sees you for the very first time, there's this innate human factor inside of us that, hey, we like this thing. This is awesome. There's this emotion, this euphoria that you get on the human side. What you want to do is, from a technical perspective, is to be able to capitalize on that euphoria, that feeling of good that somebody sees in you. What you can do nowadays is just ask them a couple of questions, or in the behavioral marketing side of things, see what they click on, what is interesting to them, what do they not click on? Those kind of things, prior to them even being in your email list. When they're in your email list obviously there's more details that you can get to, but with code snippets and things of that nature you can actually change your website around what they're reading on your blog. What you can do with your own blog, if you will, and I'm sure many of you have seen it, is that you have this "widget" that says, "You may also like ..." Or, "Here's other content that might be interesting to you." Because what you're on, the article that you're reading at this point in time, there is related articles in that same category on that same website. What they want you to do is, hey, if you're interested in this, then go check out this as well. They're trying to move you along in that journey to know that if you have a specific problem, well we have some resources and we know how to solve your problem. What you can do in the background of things is you can do "lead scoring". If somebody, let's just say on your website you have a bunch of articles around pricing or things of that nature, pricing, let's say you also have things on sales, or marketing. If somebody hits a couple of articles on your marketing side of things but they never look at pricing, then you could potentially change your website around that a little bit more. Make your calls to action to talk about marketing versus pricing. I do this on my website plenty of times. If somebody comes to me from, because I specialize in convert kit and drip, if somebody comes to me from the convert kit consultation, or convert kit experts they call them, if that webpage, then my services page gets reflected on that. I don't even mention drip, I just mention convert kit, because that's where they came from, so I'm assuming, based on their behavior, that that's what they're interested in. They're not interested in anything else that I do. You can be mindful of these sort of things, and just talking their language allows you to then get them to the next stage faster. Because if I can echo what they're saying to me, based on their actions what they're saying to me, then just us as humans we're going to say, "Hey, that's what I'm looking for. You know what I'm talking about." What I'll try to do in that respect is to be able to then grab their email address, and then market to them in that end. Talk to them about convert kit. Talk to them about potentially segmentation and those kind of things, or automated workflows if that's what they're looking for. All of this data really just gets passed over into my email marketing platform and my welcome sequence tailors to that. What that does is, like for my clients, is to be able to then baseline how long it takes for someone to first opt into your email list and then buy from you, because that window of opportunity is finite. Once you go past about 90 days, and obviously this depends on the type of product or service that you're selling, but on average 90 days, then you're not going to convert, or you're a lot less likely to convert. You want to be able to then, especially if you're selling multiple things, sell quickly. You want them to get that first purchase because that's always the hardest, and then get them to repeat buy after that. With just some small tweaks, and some small segmentation, and intent awareness, because we can dive into that a little bit more. Just based around some of those things you can then shorten that time frame greatly. I have some results where I've done for my clients, take their baseline of 40 days to the first purchase, and gone down to eight. I've had another client where it was nine days down to less than a day. Kathleen: Wow. Jason: It's just a matter of knowing and understanding the actions that somebody's taking, and then putting the right promotion, if you will. I mean, it doesn't necessarily have to be a buy, it could be an email opt in or whatever. Putting the right promotion in front of them. How To Measure Time To First Purchase Kathleen: Let's wind back a little bit. Let's say I come to you and I'm an eCommerce company, and I'm interested in focusing on this time to first purchase kind of metric. You talked about how the first thing you have to do is establish a baseline of how long is it actually already taking people to get from first touch to first purchase? Walk me through exactly what you're doing to measure that. Are there certain tools that you put in place? Tracking tools, what is it you're looking at in order to determine that? Jason: Sure. I think only one person was tracking this that came to me, which makes my life easier. Most times what I look for is really I look for obviously their customer list, and I take their email addresses. Then unfortunately there's no tool to marry this stuff. I basically take a spreadsheet, an export of that, of all their customers, and then I go to their email service provider and I see when they opted in. Then I try to figure out, based on the dates around them becoming a customer and when they first opted in, and I kind of take a baseline, if you will, "baseline", on what their metric is. Then I have a conversation with the business owner to kind of gauge what their sales team sees, if they have that data, and try to come up with the best possible estimation that they have for this. A lot of times, I mean there's obviously a percentage plus or minus, but a lot of times it's pretty accurate if you know the data that's there. Because we all know when they purchased their first thing, we all know when they came onto the email list. If it happened to be that ... I try to discount those that have zero day initially, because a lot of times people in the email marketing world, and I'm sure a lot of your audience knows this, a lot of times people will opt in with a different email than they'll actually pay with. Kathleen: Yeah. Jason: If they opted in on the same day they purchased, for the baseline I take that away. Kathleen: Yeah, there's a lot of XYZ@123.com. Jason: Right. Kathleen: Don'temailme@pleasestop.com. It's amazing how creative people get with those fake email addresses. Jason: Absolutely. Obviously there's some experience factor in there for where I try to come up with that baseline. Then what I do once I have that, then I go into their email marketing platform and I essentially create rules that store when they become an opt in, but also when they actually purchase. Which is just a custom field that really just does some math to say, okay, they subscribed on this date, they became a customer today, let's minus the two, how many days are there? Over the first month or two of doing that, I kind of gauge whether that baseline estimation that we first did is accurate enough to go off of. Then we move from there more into the optimization, asking certain questions, things of that nature to try to shorten that time. Jason's Process For Shortening Time To First Purchase Kathleen: Let's talk about that stage next. I've come to you, I say, "I need help with this." You calculate those initial baseline metrics. Then what? It sounds like you're using personalization and targeted offers in order to pull people through that customer buying journey. Is there any kind of like discovery process or research that you're using in order to determine what the right offer is, or the right way to persuade them? Jason: Yeah, absolutely. A lot of it is, in my own research anyway, is looking at their analytics first. Seeing what people are actually looking at on the website, because a lot of times it's not what the owner thinks. I want to make sure that I have the data, because for me, I'm a data geek and the numbers don't lie. If the business owner tells me one thing and the data tells me another thing, then we have a conversation to try to reconcile it in some way. That's first things first, is really looking at Google Analytics, or any other metrics that they could possibly have. A lot of people use Hotjar and some of these other tools out there that help you with the customer interaction on your website. I start there. Then I have conversations with the business owner as well as certain key members on their team, if they have those kind of people. People like marketing, sales, people that are closer to the customer, if you will. Support teams, those sort of things, to really start to get an understanding of, and it's not even technical, it's just what kind of words do you hear all the time? What pain points people are struggling with. What opt ins do you have on your site that actually can map to a product? Because a lot of people, especially in the eCommerce space, they say, "Hey, we had a discount for this. Sign up on your first purchase." Is that working for you, or is something else working for you when you run a holiday sale instead? I try to gauge what that customer is thinking. Because we can assume that we're putting the best foot forward, but if the customer is coming to you depending on the product or service obviously, they're coming to you with two things in mind. One is their intent, they're intent on solving the problem. Is the page that they're on, or your product, or service, actually going to solve their problem that they have right now? Two, what's their motivation behind solving that problem? I really want to get down to those two things. It's not scientific in the way where there's actual numbers, at least initially. I want to make assumptions on that, and put campaigns out, look at welcome sequences. Look at all of these kind of things that they're already doing that we can inject some questions, or inject some relative links to blog posts, or products, or whatever, os that we can get a better gauge on what their intent is and what their motivation is without actually asking them. Kathleen: Now you talked about nurturing sequences, and onboarding workflows, and things like that. I do find it's very easy in this day and age to overwhelm audiences with email particularly. Do you have any rules of thumb that you use as far as like, how soon do we email them and how frequently do we email them? Anything as far as even style of email, because I know there's a lot of different opinions on very designed emails versus plain text. I'd love just on the topic of email to hear your thoughts. Jason: Yeah. I mean, that's a whole nother episode. Kathleen: I know. Jason: Yeah. To answer the first part of the question about how often, frequency, those kind of things. First I have to know what they're doing already. If you came to me and said, "Look, I do a once a month promotion." If you just switch that up to a daily, then your list is going to be obliterated and they're going to be like, "I don't even know who this person is." They're going to get high on subscribe rates. If you have a pretty regular cadence, say once a week or something of that nature, it's really just throw it out, if you want to add another email. Because for me, my business when I send emails, I get paid. I will always try to mix in emails where I can. For how I like to do it, I try to do it in a human way, not just like let's just keep sending links to podcasts and blog articles, or products, or services, or stuff. I try to have the subscriber opt into those things. You can do that in a way where if you had, let's just say you had a cadence of every single week on Tuesday you send out an email to your list. You could just send out an email on Tuesday saying, "Hey look, we're going to add another email, or two emails, we're going to have it on Tuesday, Wednesday and Thursday now, and we're going to talk about this. And if you're interested in that, just click this button." They're automatically opted in. You could do things in a more human way, and it goes back to that whole mom-and-pop philosophy is, I want the subscriber to tell me. All of this stuff allows you insight into them, into the subscriber at an individual subscriber level. If they're excited to hear more from you, then you know that, hey, well they may be interested in a product or service that I have that's outside of the free level. You could do those kind of things. You can surely incentivize people with discounts and all of those other things. While that stuff does have its place and works, for the long term, creating those raving fans and repeat buyers, it's all based on trust. The trust factor comes in where you're actually genuine with them and, "Hey, I have an offer, I'm going to do this. If you're interested, all you have to do is let me know." Kathleen: You talked earlier about some examples of results in terms of shortening that time span. I would love to hear a little bit more about that. Do you have a couple of maybe specific examples of it started out at this long, went to that long, and like what led to those key changes? Jason: Yeah. I mean, specifically with some of those results, the one that's interesting is that one that was almost two weeks and I shortened it to a day, inside a day. That was really based around, it is a digital product company, but they also had a service on the back end of it too. What it was, was the funnel was very linear. It was somebody opts in and we promote this product to you, and it was a flash sale. It was like within 24 hours you can buy this for 99% off. That kind of thing. If they didn't take you up on that, then you go into this long term nurture sequence, which was basically two emails a week. Out of that it was pitching that same product over and over again, but at full price. It was, I call it a soft pitch. It's more like, hey, you've seen them in the bottom of your emails I'm sure, like in the P.S., like hey, we also if you're interested in this, we have this product. Which worked fairly well, I mean, nine days to opt in to convert to a customer is good. What they wanted to do was they had a lot of different products that served a couple of different audiences. Immediately when they opted in, unless they opted in via a specific opt in, they didn't know which audience they were. What I wanted to do was I wanted to basically put that front center. I mentioned it a little bit earlier on beforehand, is we can know before they opt in what they've already looked at, through JavaScript and cookies and local storage on their browsers, and that's all in the tech world. If we know what they looked at, then we kind of know what audience they're in. Instead of just pitching them that one thing on the back end of the opt in, let's pitch them the product that makes sense to them. That was the first step, was to really try to put that in place, which made a huge impact. I mean, that was just, that initial just, hey, let's look at the blog posts that they're looking at, and store that data. How many did they look at? How frequently did they look at? Based on that, let's position that product offering that's that tripwire product, if you will, for the next 24 hours at that discount, that's the product that makes sense for that audience. That shortened it almost to three days immediately, because people were more receptive to that offer because it made sense to them. Then there were some tweaks we made to the landing page, to the copy, based on some feedback that we got from those people that actually bought the product during that time. We made some optimizations, and that even shortened the time to first purchase. Kathleen: It's interesting to listen to you talk about this, because obviously the examples are eCommerce, but in my head I keep asking myself, is there anything here that doesn't apply to another type of sale? For example, like a complex B2B sale. I'm not hearing anything that's so specific to eCommerce. It's really just, if I'm understanding you correctly, it's really just about looking more closely at their behavior, and using that behavioral information and those patterns that are created to serve up information that's more directly relevant to their interests. Is that right? Jason: Absolutely. I mean, it just goes back to business in general. If you go to a conference, let's say you go to a conference with all colleagues of yours, they're in a similar business or industry than you are. You're going to talk to them in a different way than if you're going to a higher level conference where your customers might be. It's also a matter of awareness of the person that's viewing your online store or your website. Have they never seen you before, or are they intimately familiar with you and they know your name, they know your services? It's that buyer journey that happens with everybody, whether they're buying a pack of gum or they're buying some service that's going to cost them $10,000 a month. Obviously there's sales cycles, and that all comes into play, but it's the same business. You want to earn that trust. You want to speak their language. If you know the problem that you're proposing a solution for, then that person's going to be more receptive to hearing you. When you hear, that's where the conversion is. It's a matter of just taking them along that journey in a proper way, whether it is a complex B2B or whether it's a transaction where you just pull out your credit card and put it in. How Difficult Is It To Implement? Kathleen: This sounds straightforward on the one hand, the concept is straightforward. Then on the other hand it sounds really intimidating in terms of being able to execute it. Can you talk through how complex this is, and is this something that the typical business needs to hire a developer to do for them, or are there tools out there that make it really easy to do this? Jason: Yeah. I mean, it's as complex as you want to make it really. I like to try to keep things as simple as possible. I mean, I even, I have a thing on my white board, what would this look like if it was simple? Because we can over engineer everything. Once you start thinking one thing, it leads you to another thing, and you're going down this long rabbit hole, and you're like, "Oh my god, I don't know how I'm going to even do this thing." What I try to do is, if you come to me and you have decent enough traffic, you have decent enough sales, and we can have a conversation that's around potentially segmenting your audience better, if you don't do all that already. By segment I mean more so than customers versus non customers. If you're actually doing anything in regards to helping your customers move along the journey, meaning are you doing regular email sequences? Are you blogging? Are you doing these other things? If you are, then it's as simple as starting to think about what problems or what products are related? Let's just say you have a product that solves a problem that, let's say a developer has. As a developer I might have a problem where I need more RAM, or more compute power. If I go to a website and it just says, "Hey, buy this hard drive, or buy this RAM, or buy this monitor." Okay, but if I clicked on a blog post of theirs that talked more about compute power for my computer, and then I went to their product page and then it gave me three products that could help me there, I'm more likely to buy from there because they've already positioned a couple of things based around what I know, and I didn't sign up or anything. You can just start thinking about the product that you have and what problems that solves. That will help you start to build these things out. Keep it simple. Write it down in a notebook, or write it down in a document. You don't need a overbuilt tool to do all this stuff, at least initially. We mentioned Shai before. Right Message is a tool that you can build these. You don't need code. They give you a piece of code to put on your website, but you can build these in a visual editor. There's other tools out there as well. Initially it's really just even that widget we talked about earlier about, hey, you might like this content. On a lot of WordPress websites you can build that. There's plugins out there that would help you do that stuff. You don't necessarily need the code for that either. Keep it simple if you haven't done it yet, and see what sort of results you get. I mean, if you come to me, and usually people that do come to me, they already have this idea, they have the traction. That's why I said it earlier on, it's an established online business that I help, because they have the traction, but they want to increase more sales, they want to increase better brand relationships with their customers. They kind of have an idea that they can do this, they're just not sure what the strategies and the methods to go about doing it. What Kinds Of Results Can You Expect? Kathleen: Yeah. Are there any rule of thumbs that you use for like what kind of improvements that, on average, you think businesses can expect to experience if they go from not being contextual or using personalization to once they've done it? Jason: Yeah. It's hard, it's really based around what the price point is, to be honest with you. I feel like if it's a sub $100 product and/or service, people are more impulsive and you could probably see a quicker uptick in the percentage based around that. If it's north of $100 thing, then it's going to be a slower growth. You kind of need a little bit more time and data to see what's actually going to work and pull the triggers. On the other side of that is that those that are north of $100, you could ask existing customers certain things, which I would suggest things like, where were you when you bought this? What problem did it solve? How has it been since? By asking those questions of existing customers, you can help shorten that on the front end of it. I mean for me it's such a general rule, but I always say you could get 3% to 5% of anybody you talk to, to buy something. Obviously that's a very general rule. I always want to push that a lot higher than the 5%. What I try to do is I try to get the pages in which people are landing on for the purchase like 30% or more. Trying to get the messaging right, trying to get the distractions away from the page, because that's what a lot of eCommerce sites do. Just case in point, look at Amazon, they don't do a lot of that. Once you start going into their checkout process, the closer to your wallet that you get with Amazon, they remove everything. A lot of people don't even realize it. A lot of customers anyway, don't realize that the navigation goes away, continue to shop goes away, contact us goes away. All of these things go away as you start moving closer and closer to actually paying. Who better than Amazon to follow? Because they have the traffic, they have the data, and they publish a lot of these experiments for people to look at. I always try to, obviously depending on the price, I try to figure out what their baseline is. I want to always try to 10x the ROI that they put into me for their business. Kathleen: That makes sense, yeah. I kind of figured the answer when I asked that question might be some form of, it depends, so thank you for humoring me and answering that. Kathleen's Two Questions Kathleen: Well I'm curious to hear your answers to the two questions I usually ask my guests. When it comes to inbound marketing specifically, who do you think is doing it really well right now? It could be a company or it could be a person. Jason: Yeah, I mean, as far as inbound marketing, I'd have to say somebody that does it really well is Chris Marr. He runs the Content Marketing Academy, and he's a marketer that obviously he runs workshops for larger companies. What he does well and how he talks about what he does, it's always it's like the softest sell possible, and then you're just like, "Hey, yeah, I want to go to Chris, because he knows what he's talking about and he gets great results." His methodology and everything he talks about, it makes perfect sense. For me, I've known Chris a few years now. I've had him on my own podcast. It's just, I don't know, it's simple but yet so highly effective that it's sometimes like, hey, this is easy. Kathleen: How did I wind up buying from him? Jason: Yeah. You wonder what's going on. Yeah, if it's somebody, I would recommend checking out Chris Marr if you haven't already. Kathleen: That's a good one. I'll put that link in the show notes. Then with digital marketing changing so quickly, and especially the field that you're in, it can be very hard to stay up to date on all the new developments. How do you personally stay educated? Jason: That's a tough one. I try to, because I toe the line between tech and marketing, there's a lot of noise. What I try to do is I try to curate a lot of what I see. For me, Twitter is my home away from home, if you will. I get educated through Twitter, and who I follow there, and really put together lists on my profile that are really targeted to specific people that are knowledgeable in the space. I'll go to Twitter first to just see what people are talking about, and things of that nature. If it comes up one, two, three times more than the first time that I see it, then I'm like, okay, let me see if this is something of interest. Then what I'll do is I'll sign up to specific newsletters. Some of the newsletters that I sign up to, I may only sign up to it for a month or two and the unsubscribe, but it'll get me the information that I really need at that given point in time. I really try to reduce the amount of noise and distraction, and so I kind of use that just in time learning strategy where, okay, Facebook's changing something in their ad algorithm or whatever, now while I don't do that, my clients do, so I want to be up to date on what they're doing, at least knowledgeable to have some sort of conversation if they ask me a question. I'll go check out that for a little while. I'll talk to some people that I know in the industry, say, "Hey, what's going on over here? Is this something I should pay attention to, or is this just noise?" It's really curated, and it's more outreach for me than letting it all come to me in a flood. Otherwise, I would never get any work done. Kathleen: Yeah. I hear that. Who are your top, let's say three favorite people to follow on Twitter? Jason: Well, that's tough. For business and products, I would say Justin Jackson is probably, he's always interesting to follow because he learns out loud. He tries things. He owns a product business himself, and he's been in the product game for a long, long time, and he knows about that space. In the online world for me, business wise, as far as product goes, Justin Jackson. Chris Marr I follow. He shares a lot of interesting content, marketing links, and strategies, and that sort of thing. I follow him. Then one that I've always followed for a long, long time, probably since day one of me signing up to Twitter, is Paul Jarvis. I've tried to model my business after what he does, which is I'm small potatoes compared to what he's able to do at this point. He's always remained small, and he's built his business designed around him and his lifestyle. That's how I've built my business over the past nine years, is around the life that I want to live, and so if I start going down the rabbit hole of thinking of scaling up, and hiring, and agencies, and growing in that world, while it's attractive, it's not actually what my long term game is. Seeing Paul saying, "Hey, I'm going offline for a couple of months. I'll see you in December." Whatever it is that he does, it's like, oh yeah, that's why I do what I do. He's kind of almost like a grounding rod for me. Kathleen: That's interesting. I'll have to check him out. Any particular newsletters? You mentioned that you subscribe to a few newsletters. Are there any that have stood the test of time, that you haven't unsubscribed from, that you really love? Jason: Val's is one. Kathleen: Yeah, Val's great. Jason: Yeah. She's one. Another one that I really like is Margot, what's her last name? (Margot Aaron) She's a straight shooter. She kind of pokes, she's a marketer herself, she's a copywriter, but she pokes fun at marketing. She's one that I follow because it's like, hey, here's a headline that you're supposed to read, and here's a button that you're supposed to click, but if you don't really want to, you don't have to. It's kind of like allows me to inject my own personal brand into what I do. Because as a business owner, I know that my customers come to me, they could get what I do by going to anybody that does a similar thing, but they come to me and they become a customer of mine because there's something that I'm putting out there that they jibe with. My personality comes through in a lot of what I do, my website and all that. I wear being a New Yorker on my sleeve. I'm a pretty straight shooter too. I try to over communicate in some respects with my clients. They sort of appreciate that, and so I call my clients on certain things, I wrangle them in when they need to be wrangled in, and I challenge them. That is what most of my clients have said that that's why they stay on with me, is because I don't just do what they ask me to do. I help them along the way. Kathleen: Yeah. That's great. If you remember Margot's last name, let me know, because I'll put that link in the show notes as well. Jason: Will do. Definitely. You Know What To Do Next Kathleen: Sounds like a really good one. Well if you're listening and you like what you heard or learned something new, of course I always love it when you leave a five star review on Apple Podcasts. If you know somebody else doing kick-ass inbound marketing work, tweet me @workmommywork, because they could be my next interview. Thank you so much Jason. This was really interesting. Jason: Yeah. Thanks for having me. I appreciate it.  

Our guest, Jason Lucchesi, has done it all when it comes to real estate and today he shares his extensive expertise and tell us which strategies make the most sense. Jason has been enjoying a successful career in the real estate industry since 2002 when he started as a loan officer for the Illinois bank brokerage Bancgroup Mortgage. In 2004 Jason joined the management team at Countrywide Home Loans which ultimately lead him to become a full-time entrepreneur in 2008 when he founded his real estate investment company, Global Fortune Solutions. Skip forward to today, Jason is a real estate coach and mentor. He works directly with hedge funds and does everything, including: Pre-foreclosures, foreclosures, short sales and REOs Non-performing and performing notes Bulk packages Wholesaling residential and commercial properties Rehabs Apartment buildings Income producing properties Lease options Self-storage facilities He also has No Flipping Excuses… podcast that is. As well as a book titled, Right Flipping Now. Jason talks about the transition going from being an employee to entrepreneurship and why there is no aspect of real estate he wouldn't do, as long as it made sense. Plus, find out: Advice for his younger self The worst advice he's seen given in his area of expertise What he believes is wise that other people think is crazy Strategy ratios And, so much more! Get ready to talk real estate that makes sense with Jason Lucchesi on today's episode of the Art of Passive Income! TIP OF THE WEEK Mark: Click Here to get a free step-by-step blueprint on how to flip probate properties! Scott: Check out EmailTuna.com—a great ninja marketing tool that stores pesky emails. You can go in there and search or look at designs & offers. Do competitive research and see what's trending. Jason: For those looking for cash buyers or investors, there's a free site called Section8.com that has current available listings anywhere you want across the country. There's a pop with 65% of the time the landlord's phone #.  You can either buy property from those landlords or they can be your cash buyers and eventually turn them into your private money lenders. Isn't it time to create passive income so you can work where you want, when you want and with whomever you want?

Managing people when you are not a people manager keeps most property management businesses within 0-200 doors. They can’t scale beyond that without hiring a significant number of people. How can you shift to outsourcing to scale, save money, and improve business operations? Today, I am talking with Todd Breen of VirtuallyinCredible, which helps answer property management leasing calls every day through its experienced call center. Also, VirtuallyinCredible can hire virtual assistants (VAs) and customer service representatives (CSRs) with or without phone skills for you and your property management business. You’ll Learn... [03:15] Outsourcing: Follow rules to get it right, and select best solution to avoid failure. [04:03] Outsource companies often find and hire virtual assistants (VAs) and other team members from foreign countries causing frustration instead of support . [06:00] Small businesses need an entrepreneur, people manager, and task-oriented employees. [07:43] People Manager Traits: They care for their staff and have strong coaching, communication, and development skills. [08:50] Avoid pitfall of a property management entrepreneur who manages a team, but shouldn’t; they’re nice people, but they’re not making money or sales for the business. [12:30] Grow your management company by adding more doors and a people manager. [13:35] Entrepreneurs are sales-oriented, driven, and willing to do things others aren’t; they start controlling everyone and everything, so they need “inspired staff.” [16:25] Systems can make or break businesses; is your business systemized; and has it established and documented processes, policies, and systems? [20:14] HR should find, hire, and train employees; supervision is quality assurance. [21:38] Three Outsource Options: Hire your own VA directly; hire a VA reseller/recruiter; or select turnkey solution to hire VA for you. [27:54] Ask for and get good reviews by offering good service. [38:30] Don’t underestimate or overestimate software; technology is another tool. besides outsourcing that creates leverage, and lowers operational and staffing costs. Tweetables Property management businesses can’t scale, stay stale without hiring lots of people. Shift to outsourcing to scale, save money, and improve business operations. Outsourcing: Get it right by following rules and avoid failure by selecting a solution Lifeblood of Real Estate Business: Answer phone, get listing, rent/sell new listing Resources Todd Breen’s Email VirtuallyinCredible HireSmartVAs GatherKudos SuperTenders Property Meld EZ Repair Hotline DGS 39: Property Management Outsourcing with Todd Breen DGS 43: How Virtually Incredible Can Help a Property Management Business Grow with Todd Breen DoorGrow Website Score Quiz DoorGrowClub Facebook Group DoorGrowLive The online Windows XP simulator runs in a web browser and its operation imitates the operating system. You can use it to prank someone. Transcript Jason: Welcome, DoorGrow hackers to the DoorGrow Show. If you are a property management entrepreneur that wants to add doors, and expand your rent roll, and you are interested in growing your business and life, and you are open to doing things a bit differently, then you are a DoorGrow hacker. At DoorGrow, we are on a mission to transform property management businesses and their owners. We want to transform the industry, eliminate the BS, build awareness, expand the market, and help the best property managers win. If you enjoy this episode, do me a favor, open up iTunes, find the DoorGrowShow, subscribe, and then give us a real review. Thank you for helping us with that vision. I’m your host, property management growth hacker Jason Hull, the founder of OpenPotion, GatherKudos, ThunderLocal, and of course, DoorGrow. Now, let’s get into the show. Today’s guest, I have with me, the wise, experienced, property management guru, Todd Breen. Todd, welcome to the show. Todd: Thanks, Jason. It’s awesome to be here. Jason: Todd, you are talking to us live right now from Manila, right? Todd: That’s correct. I spend a few months a year here working with our team and been enjoying my time here, and getting a lot of work done. It’s an honor to be on your show all the away from around the world. Jason: It’s a tropical island though, right? Todd: It is. It’s actually the cooler season right now, so it’s similar to where I live in Florida. It’s just a nice weather. Jason: Maybe there’s a little bit of fun in it while you’re there too. Todd: Yeah. I played golf today while you guys were sleeping. Jason: Yes, nice. Cool. Todd, today, we are going to be talking about outsourcing roles for small, medium, and large companies—perhaps there’s some differences there—and connected to Virtually inCredible. Tell everybody, what is your thinking behind outsourcing in general. This is a topic that used to be taboo. It used to be a bad word. “Uh-oh, they’re outsourcing. You’re not using people in the US, you’re like some sort of evil entrepreneur.” I think the temperature shifted. Todd: It has. I first started outsourcing eight years ago in 2010. Back in those days, I got a lot of raised eyebrows and funny looks. Now, when people hear that we’re outsourcing to the scale that we are, a lot of people are like, “Wow! That’s awesome! How can I save some money and improve my business operations in the process?” It’s really matured nicely. The topic today is about how to get it right if you’re going to try and outsource, whether you’re a small, medium, or large business. I’m really excited to share what I’ve learned in eight years of helping small, medium, and large businesses outsource. I’ve seen what’s worked and what hasn’t. In this, I’m just going to lay it out for you and say, “Hey, if you’ve tried and failed in the past, I can probably pinpoint how that happened.” We’ll do it in a very simple grid that I’ve made for you guys. If you’re a first timer and you want to get it done right the first time, we’re going to explain the rules for you and have what it takes to do it right and to pick the right outsourcing solution. Jason: Anyone that’s listening to this show knows we’ve had a few different companies that help with like either VAs or outsourcing or finding assistants for things, and I’ve revealed multiple times, I’ve had my fair share of failures. I’ve had team members in India, Bolivia, Philippines. Right now, the bulk of my team are in the US because I’ve had so many struggles dealing with a lot of that. But we’re now just now ourselves getting back into hiring and sourcing some talent in the Philippines to add greater support for my team and my staff. Our team has helped your team with some branding work and designing new logos, so check out Virtually inCredible’s new shiny logo. I’m really liking it. We also helped out HireSmart VAs and have cleaned up their branding. It’s been awesome getting familiar with people that are making a difference in the space of property management. You’re also making a difference in the lives of people in the Philippines. Todd: I grew up in the real estate business. My dad was a very busy real estate broker, very successful, and he was working days, nights, and weekends, and he was always answering the phone. Because he couldn’t afford not to answer the phone because the lifeblood of any real estate business is answer the phone, get a listing, answer the phone, rent or sell new listing. I have a real passion now with my four kids to give them a better quality of life as third generation property management company. They just look at me and say, “Dad, how did you do it before you could outsource your leasing lines or before you could have your own private virtual assistant that was full time helping you?” And I said, ‘Yeah, it was a lot of hard work and now, it’s a lot of smart work.” Let me get into what the rules are so that the viewers will have an idea what it takes to get the right solution for you no matter what stage of business you’re in. Is it alright with you if I share my screen? Jason: Yeah, you can share your screen. Just make sure you’re talking through it because some people would just be hearing this. Todd: I’m sharing my screen, I’m going here, I’m hitting play, and tell me if you can see my slides. Jason: Yup, I see them. Todd: If you’re a small business, you’re all three of what a business needs. Because a business needs an entrepreneur that actually is the driving force behind the business, a people manager and task a score of people or employees that actually perform the tasks needed to get the job done in the business. Me, I’m an entrepreneur, always have been my whole life, and I struggled building my management company with one thing and that was managing my staff. I was pretty good at managing owners and tenants, but you don’t work everyday, all day, eight hours a day with your homeowner and your tenant. You get intermittent relationships with them. You have to actually build a relationship with your team. Whether it’s inhouse, whether it’s onshore, whether it’s offshore, if you can’t manage people, then my aha moment was, “I needed a people manager.” On this screen, I'm listing the things that a people manager needs. They’re good at caring for their staff. Me, as an entrepreneur, that’s not in my DNA. I’m not going to trouble you with my troubles and I appreciate it if won’t trouble me with yours. Entrepreneurs are not the warm fuzzy people managers. People managers are good at coaching, communicating, development, they take an interest in the development of the staff. I’ve got a graphic on the screen, there’s things that a people manager does. Number one, they know me. Number two, they focus on me and help me to focus. Number three, they care about me as a person. When all three of those things happen, they inspire me to do my best. There’s a long list of things that people managers do, but that inspiration in getting into the minds and hearts of the team and building a team is what a people manager does. Jason: Just to back that up, what I’ll say is, I see this a lot. That’s what I get to do all day long is—I’ve talked to hundreds of different property management entrepreneurs, especially when they get into that 2-400 door range—they end up in this pitfall where they now have a team and the trap because they’re trying to manage this team, and they have this to-do list that’s ever growing and endlessly long, and everything on their to-do list have been sitting there for more than two weeks. It’s probably there because they’re not the person that should be doing it. They’re in a situation which they are not natural managers. It’s just not who they are. Todd: Right. Jason: They end up in a difficult situation in which they’re not managing well, the team’s not performing well, and then they’re blaming the team. Todd: They’re really in the wrong role. An entrepreneur is very rarely a people manager. If we meet an entrepreneur who’s an awesome people manager, you’re looking at the one in a thousand entrepreneurs, or one in a hundred. It’s rare to see somebody who’s a gifted entrepreneur, and a wonderful people manager. Jason: Virtually, if they are really good people managers, they’re not so great at the entrepreneurship, the sales side of the business is struggling, they’re really nice to people, but they’re not making money. Todd: On the screen, I’ve got a picture of my wife and I right now. My wife is the consummate, ultimate people manager. She’s tremendous in managing people. We have a team of well over 150 here in the Philippines. They all think of her as the mother goose or mother hen, but they love and respect her, and they all do for her above and beyond because they’re inspired by her. It’s awesome for me to see that because I don’t have that skill set and it’s wonderful, it defines our company. Let me explain to you why that’s so important. If you are a small business, you’re the entrepreneur, and you hire your first staff member—maybe it’s a property manager, maybe it’s a secretary, or a receptionist, maybe it’s an inspector, or a work person/handyman—as soon as you start managing people and you’re not a people manager, that’s what keeps most property management businesses in the 0-200 doors. Because you can’t scale beyond 200 doors without hiring a significant number of people. Whether you hire them yourself, or you outsource them, it’s up to you, but you need people to blow past 200 doors. The companies, they get from 150-200 doors and grow up to 500 doors, they have to develop a people manager and get reliable people that work for them. There’s ways that you can get a people manager. You can hire one directly. By the way, if you were 500 or more doors, the companies that are 800-1000, 2000 doors, let me tell you something, they have a people manager on staff. Jason: Sometimes multiple. Todd: That people manager is the reason why they’ve hit that level and it’s the missing ingredient. Everyone says how do I grow my management company. I say, We’ll, there’s two ways to grow your management company. One is add more doors but at some point, you’re going to need to add a people manager.” You can actually rent a people manager or you can hire one depending on how you outsource. That’s one of the keys to today’s lessons is, “Hey, if I’m 100 doors and all I really want to do is get to 200 because the cash flow just looks so amazing then I want to go to 300. How do I best do it?” There’s some real super rules that I’m going to walk you through on how to do that. I’ve switched screens here now, for those who are listening. If you don’t have a people manager, you’ll have staff turnover, more than you want staff turnover. You’ll notice that it’s easier to get bad reviews than it is to get good reviews because you don’t have inspired staff on your team. You’re going to hit a slower growth process because normally what happens is, the entrepreneurs grows the business to the first 100, 200 doors and then, unless the entrepreneur replaces him or herself and stays working in BDM or business development and growth, then they’re going to slow down on their growth. They won’t know what to do to kickstart the growth because they’re so busy putting out fires because they’re not a people manager. Their lifestyle will suffer and they’ll say, “I’m not having any fun.” Jason: You’re referring to inspired staff. I heard this phrase several years ago that’s always stuck with me that I believe it’s true, that I love sharing with people connected to this and it’s, “Whenever we fail to inspire, we always control. Whenever we fail to inspire, we always control.” What ends up happening is, as entrepreneurs, because we’re generally really driven, risk-oriented somewhat and we’re willing to do things that other people aren’t willing to do—and sometimes we’re more sales-oriented and driven—by default if we’re not able to get people to do things, we start just controlling. The opposite is to make them inspired and they just do it. You’ll end up with these business owners that are trying to micromanage, push their team, control them, force goals on them, and they aren’t inspired to do any of it. These are the team members that are B players, that’s the only ones that will stick around in a situation like that, and they are the ones that are complaining about you, the boss, and they go home and live for the weekend. They’re just hoping to get a paycheck. That creates a company culture of hiders, they’re hiding. Todd: They’re hiding even when they’re at work. They might hide behind their voicemail. What’s the solution? We pointed out that if you are a small business and you want to go to large, at some stage you’re going to need a people manager. You can actually hire a local people manager which might cost you $50,000 or $80,000—depending on where you are—or you can hire a virtual people manager for some of the tasks at your business. By doing that, you’re now a well-rounded business. You have the entrepreneur, you have the virtual and local employees, and you have a virtual people manager. We’re going to ask you guys to rate your business. How do you rate for systems? If you’re listening to this right now, “Am I systemized, yes or no?” And if you answer is, “Yeah, I’ve got a system for everything.” If I hire somebody, I sit in my desk, and they can open a training manual, and they can open policies and procedures manual and all of the answers are there. If the answer is that you’ve got your policies, procedures, and systems setup, then you’re halfway there to being able to onboard people. A lot of people, small business owners, and entrepreneurs, they do the trial by fire. They’ll hire somebody, sit him at the desk and say, “Figure it out,” and if the person survives, then they walk through the fire, and if not, they burn up, burn out, or they leave, or they get fired. And that’s all due to a lack of systems, policies, and procedures. Now, what I’ve seen is that in the 0-150 doors, unless you have a franchise that they’re giving you every system that you could imagine, and even some that you don’t need, and then you tailor them to your local market, unless you’re one of those people who has bought or created your own systems, policies, and procedures, then from 0-150, you’re winging it. You’re figuring it out yourself and it’s all in your head. If you’re listening to this you’ll say, “Yeah, that’s me.” Then what’s going to happen whether you hire a local staff or a virtual staff and it’s all in your head? Jason: For those listening, they can break it down even further because in a business there’s not just like one system. A lot of people think, I just need a process system for processes. Every business needs some sort of sales system in their business. They need a communication system as a team in order to communicate effectively. They need a planning system in their business—which a lot of businesses don’t really have—for setting goals and milestones and allowing people to know what outcomes that are being worked towards. They need an accounting system, and they need a process system, if I didn’t already say that, for documenting, capturing all the process. Whether it’s a manual or something like Process Tree. And they need a support system for managing all the customer requests and everything else. They might be a 10 on 1 of these and a 1 on another. It really takes all of these different systems. And then you need processes for all the documented. Todd: What happens is in those 0-150, most entrepreneurs still haven’t even started with these systems. From the 150-500, they slowly come to realize that, “These systems are going to make me or break me.” By the way, we’ve outsourced for large businesses that are over 500 and they still have horrible systems, horrible policies, horrible procedures, and believe it or not, really interesting staff and that’s the nice way to put it. Jason: That’s probably the norm which is sad. Occasionally, you’ll see the conversed side where you’ll see somebody that is super systems-oriented entrepreneur. They’re basically an operator trying to function as the CEO. They will focus so much on process, and operations, and systems and they have no revenue. Todd: It went off work and they’re not growing. If you’re good at systems, give yourself a pat on the back with your policies and procedures. The next we’re going to talk about is your HR. If you;re going to grow your business and scale it and have quality and lifestyle, you have to have HR that can find good people. Then you have to get those people trained, and training does not mean you sit up the desk and figure it out, it means, before we put people to work on their tasks, nobody goes to work with less than one week of training. That’s 40 hours of specific human training. It’s a live trainer, training somebody with what’s our business, what’s out vision, mission, values, and then how do you do the actual tasks that you’ve been hired to do. Supervision is quality assurance and etc. If you have all of those things, probably, you’re closer to 500 and more doors. If you’re missing any of those things, you’re probably in that 0-400 doors. You can say, “I’m good at some, but man, I got major holes and others.” That really impacts how you should outsource. I’ve got a graph up on the screen. The graph talks about if I am rough on my systems or rough on my HR, in other words, if I didn’t rate myself high on that last five minutes of conversation, how should I outsource. The answer is, you have three choices: You can hire your own virtual assistant direct by going to the destination country, for instance, the Philippines, wherever, and you can run your ad and try and hire somebody. Not recommended for anybody of any size that’s rough on their systems or rough on their HR for obvious reasons. Jason: Yeah. You’re setting them up for failure. Todd: Yeah. I see people say, “I don’t need a reseller or a turn-key. I’ll just do it myself.” I’m like, “How are you doing with your systems and you HR with you onshore operations because that’s going to directly impact or give you an insight into how your offshore is going to be?” And then they look at me and say, “Yeah, not so good.” I’m like, “Well, running out of the country isn’t going to make it better.” The first option is to hire your own virtual staff direct. The second is hire a VA reseller which is either a recruiter or somebody who will actually stick around after they recruit for you. And then the third option is, “Hey, I just want a turnkey solution that will take care of it for me. In other words, I want staff but I’m paying you to also provide me good systems and processes and good HR because I don’t have that yet. In fact, I don’t think I’m going to have it anytime soon because I’m busy in that 0-150 doors just getting volume and growing my business.” If I can figure out a way to affordably get good systems and HR brought into me by outsourcing, that sounds like a dream job to me. This tells you that turnkey is your number one solution. If you can go to somebody that offers turnkey outsourcing and just give them some process or system or some of the workflow from your business, and then it’s called set it and forget it, and just manage the results, you’ll be thrilled. If you hire a reseller, you may or may or be thrilled based on how well systemized those systems are for the work that you give. “Have you got a system, or policy, and procedure for the specific work that you’re giving?” And/or, “How good are you at managing onshore staff and how well do you think you could manage some offshore staff?” Jason: Right. Todd: Let me go to the next screen which is a hiring guide; if you have good systems already in place and you have a people manager. In my case my wife, in your case, if you have somebody who’s a good people manager, wow, what should I do? The answer is, the world is your oyster. You can do turnkey from 0-150 and that allows you to just focus on growing your business. And then when you start hiring locally, from 150-600 doors, it means that it’s one last thing to worry about. Now, 600 and more doors, you’d be looking at a big bill from a turnkey operator and you’d be saying, “Look, I can do this myself. I should take this over inhouse with either my own DA resellers,” where I pay a recruiter to hire me an offshore staff, or by going to the destination country yourself to hire which can all be done virtually or you can actually take some trips depending on how far away you go. But good systems, good HR, and a people manager is important before you start hiring individual offshore staff that are going to report to you and work for you directly. Does that make sense? Jason: Absolutely. I think people really need to realize that they need to figure out what’s the best fit and works for them. One of the biggest mistakes I see people really early on is their default thought is, “I need to go hire somebody like me right here in the US.” That’s the first employee. They’re like, “I need to go get another me and duplicate myself,” and they’re doing 20 different things; they’re wearing 20 different hats. They go to try to hire and they’re not ready for that. They can’t even bring on somebody, they really should with technology and just start with outsourcing solutions like Virtually inCredible because you’re bringing to the table a lot of the stuff they aren’t even aware of that they might need yet. Todd: One of the things that I outsource at my management company, for those who are listening who aren’t aware, I had a property management company since 1985. I used to absolutely hate the answering my leasing calls because I knew if I didn’t answer them during the day then I had no return calls. I knew if I didn’t answer them in the evening or on the weekend then I wasn’t going to fill my vacancies and I’d have to return more messages. My phone was glued to my ear. I decided to outsource my leasing calls and so many other managers heard about and said, “Will you take my calls too?” It’s such a systemized process with us that it’s not like you could just hire somebody if you’re 120 doors. You can’t hire somebody for what you pay us because you’re going to pay more for your outsourced people, and they’ll only answer 40 hours a week, and we’re answering 80 plus hours a week. There’s a stage which it makes sense to take it over yourself, but there’s a stage where you just say, “Get rid of this and let me focus on other stuff.” When we talk about capacity in online reviews, this is the DoorGrow Show and Jason, I’m one of your customers for the review, what’s the software you have? Jason: GatherKudos. Todd: GatherKudos, thank you. I think we’ve had it for several years. We’ve had good reviews in my management company and that’s because we offer good service, and we ask for the reviews. Seasonal workload variations can actually impact your reviews. Let me explain to you why. I’m going to show you what the variations look like. This is what your work orders look like. This is from Super Tenders. Hundreds of thousands of doors went into this. In January until May, you don’t have near the work orders that you do in June, July, August, September and then it’s slower a little bit again in the fall. You’ve literally got double the work orders between April and July. April is half of what July is. How do you staff for that in your business? Jason: Yeah. The question is, “Are you going to just double staff and then fire half your staff every season?” Todd: What happens is, if you look at when you get your bad reviews from tenants who are tired of work orders taking forever to get done, you’re getting those reviews typically, not April, you get them in June, July, August, September. That’s when the majority of bad reviews come in. It’s no secret, if you’re not staffed for the increase in volume, you’re going to have a problem. At my management company, we use Property Meld to technology, really accelerates the communication process and then we use EZ Repair Hotline. Now I can outsource. I’m pretty good at it, but I hire somebody else to do my work order outsourcing because they’re pretty good at it. I haven’t chosen to do that in the Philippines yet and it’s got a US-based team. What do they call about eating your own dog food or drinking your own Kool-Aid? I outsource to other people at my management company because it makes sense for me to do it and I’m under 300 doors. I’m not at that stage where it makes sense for me to take it over. The same goes for our leasing call volumes. This is data courtesy of Virtual inCredible. This is our call volumes across the entire United States on a monthly basis, and you can see, December’s the slowest month of the year. Thank goodness, right? Going into the holidays. November, December are great, but literally there’s double the volume between December and June. June is going to crush you and people say, “Man, I can’t answer my calls or my properties take longer to rent.” It’s because you’re not really staffed to handle it right. This is called a heat map. For those of you who are listening, it’s a counter Monday through Sunday, and it shows the darker hours of the day are when we have higher call volumes and the lighter colors are lower call volumes. Throughout the course of the week, on an 80-hours of answering your calls, days, evening, and weekends, there’s a tremendous call volume variation between Monday all day. By the way, there is a reason why Mondays are Mondays. Your phone’s going to ring with your highest call volume on your leasing calls on Monday. Jason: Compared from the weekend because we aren’t generally doing it over the weekend. Todd: Or they didn’t get you on the weekend. It could that too. Wednesday is a calm day. How do you staff for that? When you’re at 120 doors. When it’s just all hands on deck at all times when you’re at that volume. It makes sense to just say, “Hey, take my calls. When I grow to a certain stage, I’ll take them back.” That’s where people are able to make a tremendous good decision for their business because if you’re under capacity, under staffed, you’re going to get bad reviews. Jason: Yeah. Todd: If you have sufficient capacity at all times throughout the season of workload variations, you can get good reviews if you ask for them. If you have too much capacity, too many staff, you’ll get good reviews but your profit will suffer. Managing that becomes a full time job when you go past 500 or 600 doors, you’ll start to do that yourself but under that, it just makes sense to outsource some of these stuff. It will save you a tremendous amount of time and money, it’ll make getting good reviews easier, it’ll make your staff happier to get rid of some of your high seasonal workload that varies a lot, and just outsource it until you’re big enough to take it back inhouse. Jason: And if you’re built out to the point where you can handle the amount of seasonal growth and seasonal increase, then what happens during those lean times, you end up with team members that are sitting at unused capacity. Nobody likes being in a position, or a job in which they feel like they’re meaningless, or there’s lack of purpose except really bad team members that you wouldn’t want. Todd: They’ll never tell you that they’re not busy. Jason: Yes. They will never say, “Oh, I have so much extra time right now. I’m probably not relevant during these few months. I probably shouldn’t be here.” It makes a lot of sense. Todd: If you’re a small company and growing, this is the last part, I don’t have slides for this, maybe I’ll stop sharing my screen. Jason: I’ll point out, connected to that, it’s not a great investment to have a team member that’s sitting in the garage half the time not being used. Imagine you live in a big city, and you’re taking the subway all the time, and you have this expensive car that you’re maintaining constantly but you’re not using. Financially, it becomes incredibly costly to have a team that is under utilized especially if they’re US-based staff, they’re really expensive, and you’re not just able to use them. You’re [inaudible 00:34:56] the money whether you’re using them or not. They’re not making any money. Todd: Can you imagine what it’s like being me? We’ve got hundreds of staff and our volume is following that curve. We figure out how to do it. There’s something I wanted to share. This is on a personal note from a guy who’s grown a management company through a few cycles. If you’re in that 150 or less, I want you to ask yourself, “Is it easy to get leads to grow my business?” If it is, great, if it’s not, check out DoorGrow, check out some great ways to get improve your leads. Second is, “Am I too busy to grow my business being an operator instead of being an owner?” An owner knows how to grow a business and knows how to keep the capacity to grow the business open because that’s what it takes. You know, 60% or 70% of calls to a business are your leasing calls. When you’re sitting there answering somebody who says, “How much is that house with the red door?” You’re considering the brain damage that’s giving you. Meanwhile, your phone’s ringing on line two and it’s an owner who has a nice listing and you didn’t get to that call. If you’re struggling to grow your business and you have leads, but you’re not growing your business, man, clear your plate, get rid of some of your work, and grow your business. You can always take it back. And then work on reducing your labor costs. If you’re in the stage in your business where, “Oh I don’t get enough leads or I don’t want to grow my business.” Well then fine tune your business, that’s great. But as long as you can grow and you want to grow, clear your plate and move forward on growth which is your highest dollar. People are valuing management companies at between $2000 and $4000 per door. Each new listing that you can sign up is going to increase your net worth, your asset value of your company by $2000 or $3000. If you can get five new listings in a month, and that’s worth $3000 a listing, we’re talking $10,000-$15,000, that’s what you’re increasing your net worth. You grow up that rate in a year and you’re $150,000 and $200,000 in your net worth. It’s all because you’re focused on growth and clearing the decks to make it possible. That’s what it takes to grow your business. Then as you bring in the new volume of work, decide carefully, “Should I insource or outsource? Do I have a people manager or don’t I? What’s the best way that’s going to keep the machine moving forward?” That’s what I tell people because if they fail that outsourcing, I usually say, “Well, who’s the people manager?” And when the entrepreneur says, “I am.” I’m like, “Oh, okay.” Jason: It didn’t work for me. You just did it wrong. Todd: Did you learn something, Jason? Relative to your world and your lives that you’ve been living with our outsourcing? Jason: Absolutely. All of these makes so much sense. Over the last decade, I’ve had plenty of failures in outsourcing things or I’m trying to do things. I think if I were to add one thing to this is one thing that has really helped me is to not underestimate or overestimate technology. Because technology, like you mentioned, like Property Meld for example, technology is another tool besides outsourcing that creates leverage, and lowers operational costs, and lower staffing cost. A lot of people will try to go cheap on software. I believe that when it comes to software, you want the best tools, the best softwares. I have spent a lot of money on software tools and I buy the best. I don’t buy the Swiss Army knife that can do everything crapily or terribly, but I buy the best in each category to make sure we have the best systems for process, or the best communications system in our business. That allows my team to get more done, and be more efficient, and more effective. Whether you’re going to outsource directly or you’re going to outsource to companies, make sure that you have the best tools available software-wise because even if software costs you hundreds of dollars a month, people are always going to be more. If you can give them the tools that they need to do the job well, you’re collapsing your cost and making them far more efficient, and it always pays off. Todd: Very true. Listen, if anyone has any questions about this, I’ll just briefly tell you how we can help you at our company. We do answer your leasing calls, we also have a tenant screening department. That scales too because you get your number of applications in the summer, is a heck a lot more than it is in the winter. We do that. Those are full termkey systems, where you’re actually putting a department into place and you don’t need a local department to do either of those tasks. In addition to that, we also have a brand new service where we’ll hire an individual VA for you with or without phone skills. You can get an admin VA or you can get a CSR, a customer service rep that can answer calls for you, and they’re all screened by our company and trained in property management. I used to be a trainer for the Property Management Academy. We put all of our people through all of that training now before they show up at your job. You get screening and training though our VA. If you like some more information, just send an email to todd@virtuallyincredible.com. Be happy to help you out however I can. Jason: Awesome. For those listening, when we put on our conference, DoorGrow Live, we gave Virtually inCredible an award because they have consistently been one of the best in class or the best in their category for what they do inside the DoorGrow Club and inside the feedback that we hear from clients. It’s not just my recommendation, it’s a recommendation of a lot of people, and you provide a really good experience for people. You are making a difference in the industry. Todd: Yeah, thanks. We’re having a ball and appreciate all you’re doing to help people grow with creative ways that aren’t obvious. I’ve been through some of your education, it’s really high quality. Thank you for what you do. Jason: Appreciate it. Todd, thanks so much for coming on the show, and appreciate you sharing all these insights. I think there’s some solid takeaways that people listening this show. I think some rising like start with technology, then start with outsourcing to a solution like Virtually inCredible, and then maybe once you start getting some things dialed in, you might want to start bringing the staff in-house eventually, but you may not. It’s going well. If it ain’t broke don’t fix it. Todd: That’s right. Jason: Todd, thanks again. Todd: Alright, guys. Take care. Jason: Alright, it’s great having Todd on the show. Again, to make sure to check out the previous episodes in which we talked specifically about leasing lines and the challenges with those with Todd. You can just search for Todd Breen and DoorGrowShow, one word. If you are listening to this on iTunes, be sure to give us your feedback in iTunes, they helps us out, also makes us aware of how you feel about the show. We love getting your feedback on these different shows, and make sure you are inside our Facebook community where you can hang out with cool people like Todd and other really savvy entrepreneurs. You can get to that by going to doorgrowclub.com and this is a community unlike any other. It’s a special community of property management business owners that believe in this vision and message that collaboration is more significant and important than competition in what this industry needs right now. If you love the idea of collaboration and helping level up the entire industry, we’re just going to help every property management business owner succeed and grow this business and industry and get more market share then that’s the place for you, that’s your home. Join us in the DoorGrow Club. Thanks everybody for tuning in. Until next time to our mutual growth. Bye, everyone. You just listened to the DoorGrow show. We are building a community of the savviest property management entrepreneurs on the planet in the DoorGrow Club. Join your fellow DoorGrow Hackers at doorgrowclub.com. Listen, everyone is doing the same stuff: SEO, PPC, pay-per-lead, content, social, direct mail, and they still struggle to grow. At DoorGrow, we solve your biggest challenge getting deals and growing your business. Find out more at doorgrow.com. Find any show notes or links from today’s episode on our blog at doorgrow.com, and to get notified of future events and news, subscribe to our newsletter at doorgrow.com/subscribe. Until next time. Take what you’ve learned and start DoorGrow hacking your business and your life.

Jason, with 4 years since his last drink, shares his story……………. Sign up now, there are only 3 spots left for the RE Retreat in Bozeman, MT (www.recoveryelevator.com) Paul reviews the GQ interview with Brad Pitt.  Pitt states that he was boozing too much and learned that either you deny your feelings and stay where you are or you feel the feelings and evolve.  He did not want to live that way anymore.  Pitt is learning to accept the things about himself that he does not like.   SHOW NOTES   [9:13] Paul Introduces Jason   Jason – I have been sober for 4 years and live in Big Sky, MT.  I am a firefighter/paramedic and enjoy outdoor activities.   [11:10] How did you meet your wife?   Jason – I had walked into a bar in the middle of a scuffle.  My “soon to be” wife was on the ground and bleeding from her head.  I felt the need to come to her rescue.  We have been married for 9 years now.    [13:45] When did you realize that you had a problem?   Jason – I did not know I had a problem because all of my family were heavy drinkers.  One night I went out with friends and drank very heavily and then drove home.  The next morning I had the worst hangover of my life.  I really thought I was having a medical emergency, I felt so bad.   [16:46] What were your drinking habits like?   Jason – For the last 10 years, I would get off of work and start drinking.  I would spend the last 2 days of my days off sobering up.  We had lots of house parties where there was plenty of drinking.  My wife and I would also take yearly sailing excursions.  They would turn into 2 weeks of binge drinking.   [19:48] How did you get sober?   Jason – I reached out to a family friend who has been sober for 42 years.  At first I did not want to go any meetings but I had wanted my wife to stop drinking so we both ended up going to a meeting.  The meeting was a total mix of people and completely changed my life.   [23:59] How do you remain sober?   Jason – I go to AA meetings.  In early sobriety, I would just show up at meetings and listen.  Currently, I stay very involved with my sober community.  I also send out daily recovery related e-mails.  It helps me stay accountable.  If anyone else would like to be added to this e-mail list, send Jason and e-mail (jgras@sailingscubeadventures.com)   [29:52] Paul and Jason discuss being a grateful alcoholic   Jason – I have learned to be grateful and humble.  The program has allowed me to change.  It has been a journey through self-restoration.         [31:51] Paul and Jason discuss Sober Scuba Sailing Tours   Jason – My wife and I thought it would be a great idea to offer sober sailing excursions.  We are organizing a trip in June.  For more information on future trips, go to www.sailingscubaadventures.com and send Jason a message.   [39:42] Rapid Fire Round What was your worst memory from drinking? that horrible hangover that made me feel like I was having a medical emergency Did you ever have an “oh-shit” moment? when my hangovers would last for days What’s your favorite resource in recovery? the Big Book, Twelve Steps and Twelve Traditions, Tony Robbins “I’m Not Your Guru” What’s the best advice you’ve ever received (on sobriety)? make your bed every morning, the miracle will happen You might be an alcoholic if…..you see a half full cocktail and think, “Now that’s alcohol abuse;” then you finish it yourself   Resources mentioned in this episode: Recovery Elevator Retreat Connect with Cafe RE- Use the promo code Elevator for your first month free Sobriety Tracker iTunes Sobriety Tracker Android Jason’s e-mail = jgras@sailingscubaadventures.com www.sailingscubaadventures.com Tony Robbins – I am Not Your Guru (available on DVD and Netflix) Sober Selfies! - Send your Sober Selfie and your Success Story to info@recoveryelevator.com   Hold on tight as we follow Paul’s journey coming off his anti-depression meds.  Good luck Paul!     “We took the elevator down, we gotta take the stairs back up, we can do this!”