Podcasts about right so i'll

James: Hi listeners, welcome to Achieve Wealth Podcast. Achieve Wealth Podcast True Value in Real Estate Investing focuses on key players in valuable estate investing specifically on Commercial Real Estate asset class. Today we have Michael Becker who has done more than 7,200 units, primarily, I believe in the Dallas area, I know Michael can help me fix that. But you know, he has done a lot of deals in the past few years that he has been investing. Hey, Michael, welcome to the show.  Michael: Thanks for having me. Appreciate it.  James: Good, good. Can you tell the listeners about things that I missed out about your credentials? Michael: Yeah. So, Michael Becker, I'm based in Dallas, Texas and I'm a banker by profession. That's kind of how I got into the business was loaning money to other people and went out on my own about six years ago now, so about six years of experience. And as we talk right now, we're just closing up our 34th and 35th acquisition. So puts us about 70 to 100 units that we've done in our career. So far we going full cycle on 16 deals. So we refinanced three out, return some Capital still own and we sold 13 of them. So as we talk, we currently own about 5,000 apartment units, the vast majority of those are up here in Dallas Fort Worth, which is where I'm based. We have 400 units in Tyler and then we have 900 units in the Austin markets. So we're Texas-based focused, predominately on Dallas Fort Worth and Austin for where we look to buy. James: Awesome. Awesome. So rarely, I get to interview someone who has come from, you know, brokerage business and also the landing site, right? But I always wonder why Brokers and lenders who lend money and trade deals never really become the buyer or the owner of the assets, right? So what was your triggering Aha moment that you said, hey, I should better just, you know, go on the other side of the table here and start buying deals rather than lend money? Michael:  Yeah to be a banker, you have to have a certain like mindset and generally pretty conservative and if you start becoming successful like I was as a banker making a lot of loans, they try to tie you in the bank by giving you stock options and have more investing period so it's kind of the longer you wait, the harder it is to leave. But for me, I was 35 when I left the bank, I'm 40 now, and we're just like this little fork in the road, I felt that if I stuck around it was going to be that much harder to go. And really what I did was this all day every day was making loans to other people like yourself that would be a buyer, distress deal, renovate and sell it for big profits and I kind of realized I was on the wrong side of all those deals. It's better to be the borrower than a lender.  And you know a lot of great clients, a lot of them are friends, my friends still to this day, and I was looking at a lot of them and I was like thinking myself like if that guy can do it, I definitely could do it. You know, not that they're not smart. But what I like about the business it's a really, really simple business at its core; it's not always easy to execute but it's pretty simple to understand. So I had a lot of connections, had a lot of experience, you know, I underwrote deal after deal after deal, I knew everyone in Dallas Fort Worth, I was in the industry. I just wasn't doing anything about it.  So I met my business partner, Shawn, back when I was at the bank and he was helping people out of California buy properties in Texas. I made a loan to them. And so, he was kind of sick of working for his boss the broker and I was sick of working for my boss at the bank and so we kind of went out on our own. And like I said, we're probably the second or third most active B classifier in Dallas Fort Worth and the current market cycle. So we've been pretty active here in Dallas Forth Worth.  James: Got it. Got it. That's interesting. I always wonder, I mean, what do the Brokers and lenders see in themselves that they want to continue doing that rather than owning an asset? Michael:  You know, when you think about it though, like as a banker, you don't have any money at risk, you got other people's money at risk, you got your clients' money, you got the bank's money and you know for you to go tie up a deal, especially today, I mean, you posted up six figures in earnest money or God forbid, you know, well north of that hard earnest money day one and get all this like Risk and then you got to go out and raise, syndicate the capital. So to take that to do what we do for a living, you got to have a certain amount of guts to go out and do that because you know, you're taking a calculated risk along the way and you don't have a paycheck. So if you don't do business you don't get paid. So that's a certain minority of people in the world I can go on and take that type of risk on and thrive and if you go out setting cases up like I do, you just have to be comfortable taking that kind of risk. And on top of that, you know, most of the stuff is on recourse, where you still sign and carve out. Some bankers get pretty, pretty nervous about signing, you know, I have 4- 500 million in debt right now so I mean that's a lot of money, you know, and to try to take that mentality, it's just a different type of mindset for sure.  James: Yeah, I guess the entrepreneurship mindset and whether you want to do it, I mean, especially if you have gone through the last crash in 2008, you can be very scared.  Michael: That's right, for sure. James:  So let's come back to how did you scale up to this large portfolio, right? Because I used to listen to your podcast when I started in this multifamily investing in 2015. When I was listening, I know you had like, first year in[05:47unintelligible] you had like 1000 units and now you have like 7,000 units, right? I mean maybe now you own like 5,000 units, but what was the system's process if you put back yourself back into that time and I know you made mistakes from then until now but you know, what are the teams or what are the processes and who would you hire first to grow to this scale? Because now it seems like clockwork for you because you guys have been... Michael: Yeah, so we started out, it was pretty lean. So when we first started out, I did the first four deals, first 800 units. I still worked at the bank and then I kind of had enough scale that I felt like I could you know, keep going. I had enough credibility in the market place; you buy one deal, you get a lot of credibility. You buy four like quickly everyone in town knows you're out there buying it because like I mentioned, I had a lot of resources like from the standpoint like all I did, all day, was underwrite apartment loans. I had a lot of connections to a lot of people. What was holding me back was that everyone thought of Michael Becker as a banker, they didn't think of me as a principal so I had to kind of change the perception in the marketplace what I was from a banker to a principal. So once I did that, that changed it pretty quick and then from there, we sort of started to scale. And so it was my partner Sean and I and we had one employee when we started. We kind of did a little bit everything and we all do a little bit everything when you're that kind of small. And so, you know, we were just kind of guys who were doing deals and then all of a sudden we woke up. I think we had seven or eight deals and we had all this work on us and there was still just three guys out there doing deals. So we had to figure out how to systematize so we started out with someone that's got an IT project management background experience actually, so she came in and kind of did operation; we were disorganized with stuff everywhere. So like our Dropbox wasn't orderly, you know, just wasn't everything wasn't save down. We didn't have any documentation of processes and procedures. So she came in the systematically, you know by meeting with me for two hours at a time., she'll talk about whatever, interview me and systematically built out all our policies and procedures and organize everything. You know, our chaos for life got real organized over a six to a 12-month period from there. Then we added an analyst to kind of help on top of it. And then we started layering in an administrative help on top of that and then you know, we start getting Asset Management help, hired a professional asset manager and then you know, we hired transaction people to kind of help run process the escrow and things like that. So those are the types of teams, you know, we have a third-party management company. I think you're vertically integrated when you do management in-house.  So we're able to manage 5,000 units with nine people; basically my partner and I and seven employees. We've got ahead and taken the approach. So I want to hire really high-quality people, pay them a little bit more money, but just be a little bit leaner. So that's kind of the approach we've taken because I really don't like managing people. So the lesser quality people will take a lot more of my resources so I rather pay someone that's a killer really high salaries and trust they can go out and do the job. But you know, admin help is the first thing I think you need. Someone to make sure you get organized. You have a process, make sure you get an investor database. Be really helpful, if you do syndication dropboxes, so we use dropbox all the time.  You'll have internal chat systems. Those are things that kind of we can do quick little messaging, you know, all sorts of stuff like I talk about, about raising money more efficiently if you want to go down that path or if you want to talk about operation, we talked about that too. But just trying to use technology and work smarter not harder. And every time we do a deal, at the end of the deal, we always have a Post-mortem meeting where we go over the good and the bad and we take away lessons that were bad and then we take those and try to improve the process for the next deal.  And when we first started out, they were a lot of bigger issues and now, fortunately, the issues are really small and minor because we got the list of stuff you don't ever want to do again list, got really long pretty quick and try not to make the same mistake willingly twice. James: Yeah, so can you name like top three things that you have realized from that not to do list, can you share it with the listeners?  Michael: I mean around raising capital in particular, you know, we first started out, we had a database and I needed to raise a million. I remember I had to raise a million four for a deal, I think it was a million five something like that. And it took me about 20 25 people somewhere in that range to get a million five in, a hundred thousand minimum. We first started out I'd get a package. I need be able to an investor. I set up a call and have an hour-long call, 45 minutes to an hour long call and I had to do that 25 times. Now, what will do is we'll email the list, we hit schedule webinar and it's at, you know, seven o'clock Central Time on Wednesday. People that can attend Live, great. If not, we'll send them a recording of the webinar. And then they can watch the webinar when they want to and then I have a five-minute call with them if I need to resolve. So I presented all the materials of the deal so maybe a lot more efficient that way. Whereas, you start scaling up doing like webinars a lot more efficient way to present your opportunity than one on one calls. Because, for example, we just finish up with 24.6 million dollar equity raised and if I had to do that one call at a time like that is so huge, you can't do that. It's going to be 200 people basically invested to get 24.6 million. So, you know, you'd have to have 300 calls to get that and that just isn't an efficient way of doing it. So, that'd be one thing.  Another thing that's been official, as I said we got an investor database. So when you invest with us, you go to our database or portal up our website you fill your stuff in electronically and you electronically sign your documents. And that's a much easier way of going about it and getting the old school, paperwork out, that's kind of how we started. And then finally what was another good way to be able to work efficiently. You know, I think we got more efficient the way we've kind of work it and keep people in line and we clearly communicate what's expected of people and we're really consistent with it. So those are things you grow into, those aren't things you necessarily have money to do out the gate because we, you know, spent a couple of thousand bucks a month on our investor database. So if you have zero units to spend $24,000 a year on a database doesn't make sense. But you know, gotowebinar is certainly something you can do and you can use a Google sheet instead of a set of a database until you ultimately get enough revenue where you can afford some of the more technology tools that are available out there.  James: Yeah, yeah. In fact, I just launched my investor database yesterday, which was a lot of my investors love it. They just say it's so nice for them to see their dashboard, in terms of investment because a lot of them have multiple investments with me and it's just nice for them to see. And all the documents are in one place and they can just log in and get the report. They just love it. Michael: And it'll help you when it comes to tax time to track all your distribution in there, I'm sure and then you don't have to go recall your distributions at the end of the year to do your K1s. James: Got it. So coming to I mean you must have a good number size of passive investors. I mean, how do you select certain passive investors for certain deals? I mean is it first come first serve or how is that? Yeah, so we have, let's see, I did 900K1s last year. I think I had about 500 unique investors when we closed the year out. We just raised, I'm not quite sure what the stats are of how many are a repeat, how many are new but I probably have 600 unique investors who've literally invest with me at this point in time. And we're going to do 12-1300K1s  next year easily. So yeah, we generally will so we definitely have like a blacklist, right? So if we take your money and you're a pain, we'll make sure we don't take your money again. That's certainly the thing I think everyone should do that for sure.  On the front end if we think you're going to be a pain we'll generally kind of blacklist you as well, life's too short. Yeah, too many people, we don't have time to have a little distraction. But basically when we have an offering, we'll just go in the database and you'll get together like the MailChimp will send out a little, hey, coming soon email or save the date email, got a future opportunity coming up and then you just email the database and just generally first come, first serve.  Sometimes we have a couple of guys that we know that we have a special situation with that. They're like, hey, I have this money. I want to place it with you. Maybe we'll give them a little bit of a head start to deal from time to time. But generally, send it out first for people to pay attention, fill the paperwork out, get it all done, wire the money in, those are the ones that get into the deal. James: Yeah. I mean, I agree with some investors being a pain. I mean, it's just so hard to win. Especially sponsors like us. I mean, there's so much of moving parts and so much hard money in and on day one, I mean, so much money stuck on escrow and this has so many things going on in closing a deal. And there will be some people we just had to deal with it, right? Michael:  Yeah, so, you know, it wasn't the vast majority, people are great and but you know, one of the things that I was talking with one of my buddies, he's syndicating his first or second deal, yesterday, and he was getting a little frustrated, it wasn't going quicker and I'm like well just because you have a deal in escrow and you have a deadline and it's important to you, doesn't mean that it's not as important to investors, but they have other stuff going on their lives. So you got to be able to make sure you meet your deadlines. So you got to consistently communicate deadlines and be proactively reaching out to people and you know, you gotta push sometimes to get these people. Because if you don't stay in front of them, they're going to get distracted and something else in life is going to come up and they'll just simply forget that, you know read about your deal. They don't mean to and it's kind of like happens.   James: Yeah. Yeah, I always communicate as well to make sure that everybody knows the timeline and when do we expect things and keep on communicating to them because everybody's working on getting things done, the passive investor, the sponsors and all that. So that's important. And so the type of deal nowadays that you're doing because usually I mean, I'm not sure whether you know, I wrote a book called Passive Investing in Commercial Real Estate where I categorize three different types of deal, which one is core, the other ones are light value add the other ones a deep value add. So the type of deal that you're doing, can you describe those characteristics? Michael: Yeah. So when we first started out, we bought a whole lot of[16:37unintelligible] that's kind of generally where we started out that's where most people start out. So the first probably ten deals may be more raw 1960s 1970s vintage stuff and then about two years into the business, we started to transition more in the B-class. So Texas, things like the 1980s vintage. And then really the last two to three years the vast majority of what we have done had been kind of more B plus, A-minus. So things kind of like late 90s all the way to about 2008; that's kind of my most favorite part of the market, as we sit right now.  We have done a couple of brand new deals. We had some exchanged money, we sold a BDO and we just bought a brand-new 17:16unintelligible]  and then we bought a few deals a little bit older than the 90s. But generally speaking, if you ask me, A-minus is my favorite space and a couple of reasons for that. Now one, if you go back when I first I bought my first apartment 2013, I bought a brand new class A Deal in Dallas for about a 5 cap, a BDO was like six and a quarter six and a half cap and a CDO was like eight, eight and a half cap. Fast forward to today an ADO is like a 475, a BDO is like a 5 and the CDO like five and a quarter by five and a half, something like that, right? So what used to be a big gap is now really, really narrow.  So we have the ability to track larger amounts of capital. So it make as much sense to me to be on a risk-adjusted return basis to buy a 1970s piece of crap building if I can buy a 2004 vintage building for a similar cap rate. So that's kind of what we're focusing on. And the stuff that was built that's 15 years old, stuff kind of on the 2000s. Still, most of those have like white appliances and cheap light fixtures and you know, no backsplash and you know cheap cabinet fronts. You still do similar value add things like flooring, appliances, fixtures, backsplash, cabinet fronts and still push the rent lift up a hundred dollars or maybe more per unit by doing the work. So that's kind of my favorite part on the market and then just kind of we've been fortunate enough to have a couple of deals go full cycle and return a bunch of capital. So we have a lot of money in our database and so I can't simply go raise two or three million dollars, that's just too small, you know, we need to be raising, you know, nine ten million time minimum; it's just too small. So we're just trying to do a little bit of a larger deal. And that's kind of what we've been focused on and say light value add, A-minus that's the vast majority of what we do with a couple like more newer stabilized kind of deals then thrown them in if we do an exchange or we just think we're getting a good basis on a deal. James: Got it. Got it. And also the other thing that I mentioned the book is the passive investors will be, they would like to invest based on their preference or based on their investment cycle. So when you look at your passive investor demographic, do you see some differentiation in terms of these are the group of people that like to invest in my deal?  Michael: Yeah, I mean, listen with 700 different people that invested with us you get a little bit of everything, right? You know, but that's one of the things that we always try to make sure we stress is you know, hey, here's what to expect. You know, we're really explicit about what the projections are, the timing and amount and the timing of the cash flow and when you do a syndication, ultimately most of those things need to sell at some point. It's hard to keep a whole bunch of unrelated people to together for perpetuity; forever is not a good hold in a syndication environment. That's cool if it's like you or you and a partner or a really small group of people, but when you have, you know, a hundred unrelated people that's hard. So we want to make sure when we're communicating with them that--and they understand like, you know what to expect and I also let them know if we're going to sell it and it doesn't fit what your objectives are, then this isn't a good thing for you to invest in.  So we try to be really explicit. So we match expectations properly because what I don't want is a year down the road, for you to be upset because you thought you were investing in, you know, one thing and there's really something different so, you know trying to be explicitly and very clear to our investors is what we're trying to do.  James: Yeah, that's good. That's the best way to just make sure that everybody knows what they're getting into right? So with the market at the current cycle right now, I mean in DFW Austin, you know, the whole taxes or places where you're investing it's very hot right now so, where do you think we are right now and how your strategy has changed in terms of acquisition? Michael: Yeah, I mean. You know, this has been a hell of a run where we're nine years into this thing or something like that. I mean, it's been one hell of a run. You know, with that said, the more we focus on a predominately Austin which is where you live in Dallas which is where I live and if you look at the population projections about three weeks ago, I've done this with staff about three weeks ago. The Census Bureau came out and kind of have stats for the growth 2018. So Dallas, Fort Worth from 2010 through 2018 over an 8 year period, there are a million more people in here in 2018 that was in 2010. So, we went from that 6 and a half million people to about 7 and a half million people and their projections in Dallas Fort Worth are to grow from about 7 and a half million people to almost 10 somewhere between the next 12 to 15 years. So to put that in perspective that's about two and a half million more people coming to Dallas, Fort Worth if the projections are right. So that's the equivalent of like the entire metropolitan area of Charlotte or Orlando and then putting it on top of Dallas, Fort Worth today. And everything I just quoted to you about Dallas, if you take the percentages, it's even higher in Austin. So Austin is growing even faster on a percentage basis. If you feel like just driving around, there are just more cars, more people all that. So I don't know a whole lot, James, but I know if the equivalent of the entire metropolitan area, Charlotte is put on top of Dallas Fort Worth[22:50unintelligible] have to go higher right? They just have to go higher. So what we want to do is, you know, make sure that we're focusing on the right locations within the metropolitan area. You know, we're trying to buy away from these Supply the best we can. We're buying like Suburban multifamily deals in better school districts. We're trying to focus on basis. So we're trying not to pay Crazy Prices. One of the strategies we've done here recently is focused on properties that you can come buy and assume someone else's mortgage and you get this avoids having a large yield maintenance or the [23:24unintelligible] prepayment penalty. So you get a pass along a lower cost to you as a buyer. So that's a way to kind of counteract that a little bit.  What you give up as a buyer; you give up five years of interest only on the front end as you're assuming a mortgage that's most likely already amortizing so kind of hurt you up from yield. But if you save a million dollars or two million dollars in basis, you know, one day, that's going to burn down if you need to sell it or refinance it free and clear. So that's one strategy we've been doing. And then here's another thing. I mean you own a bunch of stuff to San Antonio like those we were talking about before we started recording. You know, this is one of the things I would say, it's completely unfair business, you know, a lot of it who you know, what you know, what chips you can trade. And you know, I own a lot of stuff in Dallas but I walk in the San Antonio, you know, you have more clout in San Antonio than I do, just because I don't own. So the Brokers are more apt to sell you something than someone that doesn't know that market. So we're at this point in the cycle doing 35 deals or some like that at this point, we know everybody, everyone knows us that our Brokers are players in town. So we get our unfair share deals. So, you know, we're looking at a lot of stuff and we're trying to be selective with it. It's also as far as strategy goes, you know, the lone assumption route has been something that's been successful for us. And then two, we put up a lot of hard money. That is the other thing that helps.  So you can put up a lot of hard money, get aggressive with your terms, you know, act quickly, you know, we got a deal in escrow that we officially never got to tour, you know, so we had to go shop it and then we never got to tour it and so we just basically got it in escrow went hard [25:10unintelligible]  without ever having an official tour and I can do that because I've done 30 something deals. You don't do that on your first deal. So I know what's up, I know what's going on and we did our due diligence and we didn't find anything that we didn't already expect. So we knew what to expect and that's what experience and repetition gives you a psyche. I got my 10,000 hours and I kind of know what's going on. I kept having to make better decisions, quicker with that level of experience.  James: Yeah and brokers love it too because for them is like you're a very easy buyer because you already know the submarket. You're not going to give a surprise and they have done deals with you. They just love it things to go much smoother. They make money as well. So they love the repeat buyers and the local players, as well. Michael: Yeah, that's right. And then we're all friends like we go and have drinks together we go to the baseball game together. We all become friends and you know people do business with people they know like and Trust so being local in the markets that we own and operate in. I was at lunch before this podcast and ran from the[26:17unintelligible] Brokers because of their office across the street from me. Walking down the street and you ended up having lunch in these just randomly. And as I was walking out, one of my competitors who own like 12,000 units whose office is around the corner for me walked across me in the hallway, you know, and on the sidewalk, I mean so this like being proximity and doing a lot of deals that stuff helps. James: Got it. Got it. So let's say nowadays, what's the process of your firm looking at a deal? So let's say today there's a deal coming. I mean, it's not on the market, the broker tells you, who looks at it first, how does it come to your eyesight before?  Michael: Yeah. The way we are set up, a deal comes in, say I get it, you know comes across my desk. You know, I basically kind of where's it located? You know, what's the basic price? Right? So I'll just kind of go to Google Map. Make sure you kind of know the location I'm in and I know whatever location that they are sending us. Like we know like the markets because we're in the market. So, you know, usually, most of the deals are like, no, it's the wrong location or no, you're prices are extremely insane. I'm not paying that price per unit for this type of product. And so usually a lot of people kind of get kicked out, but if it passes kind of that basic high-level test, then at that point usually we'll do like a real get the financial statements in from the seller. And then what we'll do like a real back of the envelope analysis.  We'll spend 20 to 30 minutes doing a real high-level underwriting just to make sure that it kind of passes the high-level test and usually a lot of those deals die right then. So, you know, the deal was just like, you know the match it doesn't work. It's just way too expensive or we don't think there's not much upside in the rinse. Just whatever it is. We kick a lot of deals out that way. Then if it passes that deal usually at that point, we'll do a full underwriting and that will take this like four hours. You know, we have a CFA that's our analysts. Our analyst will go underwrite the deal for four hours. Since it's my partner and I, then my partner will go through and kind of review the model. And once you review the model, it passes that, then, you know usually, most of the deals kind of die right there then they don't really work. But the deals that kind of pass that screening that's when you know, we'll kind of get down and get serious about it. And I think that point that's usually when I go tour. So that point, they pass all the tests so we set up a tour maybe put [28:34unintelligible]  in early kind of depends on the situation. And so, you know, we're looking at you know, 60 70 deals to get one that actually makes something like that. That's probably somewhere in that kind of General ratio is what we look at. And we just have like little series of check marks along the way that we gotta like, you know, but doesn't pass this one little test and let's just kill a deal and move on. I found on the biggest cost to have in my life anymore, stop tuning cost. So if I spent a lot of time on one thing it's at the expense of something else. So my time is precious. So just trying to make sure I get, you know, use that the most widely and don't chase these deals for you know weeks and weeks. I never had the opportunity of actually making it in a day. So that's hard to do when you're first starting out and that's a lot easier to do when you have some experience.  So when you start out, you got to learn these lessons sometimes the hard way. You got to underwrite this deal that if you would have just at the end of it just kind of be self-reflective like, you know, what could I have seen earlier on this deal that would have stopped me from wasting a week of my life on it? You know, you need to start that. I think that's what separates a better apartment owner, ownership syndication type groups from the less successful ones.  James: Yeah, I agree. I mean, I don't look at more than five parameters in any P&L to decide whether I want to dig deeper. So what's the ratio of deals that you look at verses you looking at and passing it to your analyst for the four hours underwriting? Michael:  I mean, it's probably pretty limited. So if it's called 60 deals to get one, I mean it's probably, at least half just get killed or your pricing is way too high or it's the wrong location or the deal too small or something physically about the deal I don't like. So that's probably half of them and the ones I've been going to like get a back-of-the-envelope, we probably kill, you know, the 30 that make it through on the 60 we're probably killing, you know, so that's 20 right there. Then we'll probably underwrite, you know, ten to get the one type of thing.  James: What do you look for in a location?  Michael: You know, yeah, so we're Suburban multi Family Guy. So good Suburban location that is in the better school districts, you know near major thoroughfares preferably to have access to Lifestyle and Retail amenities like, you know, like they are near a Starbucks, near a good grocery store, you know, retail restaurant, stuff that people want to live in. First and foremost, low-crime area too, I don't want to buy in the hood. So, you know, no low-crime area. Those are the things I look for and we're targeting, you know, preferably 200 plus unit, A-minus family deals, but that's kind of my perfect deals. An A-minus deal with more than 10% or an upside, you know it's well located, low crime, better School District, near employers, near retail and restaurant. That's kind of what I look for.   James: So, can we go a bit more deeper into the back of napkin underwriting? So, let's say there's a $10 million deal you know, 50 unit, maybe a 100-unit deal, how did you underwrite that? Back of the Napkin. Michael: I mean, so what is the first major metric is a, you know, one other [inaudible31:51} ransom what's our basic market survey say . So, pull a [inaudible] and look at the market rent. So then how much upside do we have in rent? So, I say, so, if there's only 5% upside in rents then it's probably not ideal for us, you know, we typically 10 plus percent in upside of rent to make the mass work. So, if I only have 5%, I know when I layer in my sponsorship compensation it's just not going to make sense. All right, so you know, like it's just not going to have no margin for us to be able to go attract capital. So, that's the first thing and then we'll then obviously go down and like other income or other income opportunities, then obviously look at the expenses as well. Michael: So, you know, one of the deals were we just got awarded, the payroll is by 1600 ,1650 a unit and it should be 1200, you know, so we can on day one, boom, take 450 out of payroll that certainly helps quite a bit. So, we're looking for things like that, that's kind of what it is. And you know, basically for maybe if you think about it at its simplest form, James, like, I need to do a deal I need to be able to deliver somewhere between 13 to 15% IRR today that's what takes me to attract capital. So if I can't get a deal layer in my compensation layer in whatever capital you need to do, um, you know, talk to the purchase price and I don't have enough upside of rents because at the end of the day, if I can't produce a 14% or 15% IRR over a five year hold period, my investors don't want to invest. So, I can't spend time on deals on can produce those types of returns. So, we're just trying to find, stuff that has enough upsides would be able to produce that. So, whatever that is, reducing expenses, increasing income, the two most common things, or is there some sort of way we can get a different type of debt quotes that may be kind of juices, some of these returns or whatever the specific situation is to that property. That's kind of what we're trying to get to the heart because, if I can't produce a 14 or 15% return, I need to shoot the deal and move on. James: Got It, got It. So, coming to 13,14% IRR is it to investors, or is it overall returns on ... Michael: Investors right. So, if it’s like 15 investors 17 and a half, 18 to the deal and you put a sponsor comp in there? So, it's got to be, I gross 8 total 18 they get up 15 and our structure or something, something like that. James: Got It, got It. Yeah. It's interesting on the debt code side, no, sorry, before I go there, how do you know that the seller is not taking some of your upside? Because nowadays that's what sellers do, right? They price it slightly higher; they give you upside, but they price it higher, which erases your upside. So how do you determine that? Michael: That's the whole thing why we don’t buy c class anymore because of the same catch, so yeah you know, that's the thing so I mean, all these deals that have a lot of upside have a lot more interest and so they can again, bit up and the cap rates are compressing. So, the trick is you got to overpay a little bit, but you can't overpay too much. Right. James: Right. Michael: And that's kind of like what you're doing. So, at the end of the day I got to, I, it's as simple as I deliver a 15 IRR and if I can't deliver, I can pay up to a certain price and then you start doing past out price and I can produce the returns I need. And that's kind of when we back off. James: Okay. Michael: So that's kind of how I think about it, so, every, most of the deals we'll work out at a price. So, we just kind of get to where this is the Max price what we can do to push to push out a 15 IRR for investors. And so that works up to 20 million and 20 million, 100,000 it doesn't work. So, you got to kind of draw the line in the sand and have a lot of arms in the fire. You get a whole bunch of deals working all at the same time. Usually, they start popping. James: Yes, yes, yes. The basis of my question is because they could be $150 or hundred dollars a rent bump potential, but the seller has priced it so much or we could have outbid-- Michael: Yes. James: --so much that it's not worth it, right. So, to do that because you might be just getting-- Michael: Yes, there's that. And then you get a little nervous for some of the less-- the newer people in the business, with little less experience like you're going to pay a five cap for 19 C class, 1917 deal. Okay, location and suburban St. Tonio or Dallas or whatever and then you're going to perform like a five and a half or five 75 extra cap. Five years down the road for a c class deal, maybe that, maybe that's the right cap rate, maybe it's not, it needs-- as you go and improve the property, you're able to increase rents and by extension, you value you’re in a why. But at the same time, the more upside you take out of these deals because your turnover, 50% units upgrade them, shrinks your buyer pool cause everyone wants value add. So, the more value you take out on the deal, your cap rate actually goes up. So, it's like a weird little dynamic you're in that you got to like, you got to factor in. It's like a 3-D puzzle you're doing because what's great because you're increasing, you're why. Because you're raising your rent, but at the same time you're also expanding your cap rate, as we sit in the same marketplace. So, it's interesting, complex puzzle, the marketplaces are right now. James: Yes, I was talking to a broker and you say hottest deal to sell nowadays it’s like deals where everything is done right, 90% is done. Michael: Yes. James: Nobody really wants it because everybody wants value add right? Michael: That's probably the opportunity to go buy a bunch of that stuff. Cause that's what today is. And then if you can get higher leverage loan, you get a 75% loan and get a good low-interest rate and get a bunch of I Own and go buy a deal that's turnkey. Maybe that's a better way of going, to be honest with you. And just kind of get a little bit more your return from current yield versus a big pop on the backend. That's thought about strategy, to be honest with you, it's a lot more safer than going and doing a bunch of work on a property-- James: Yes. Michael: --and paying a 475 cap for 1970 deal. I'd rather pay a six and a quarter cap for six and a half cap for a deal that's already done. James: Yes, because the backend is not certain. Right. Nobody knows what's going to happen-- Michael: Right. James: --at the [inaudible37:58] cap rate, so. Michael: That's right. James: So that brings to my next-- Michael: And then you do all the work, you might expand your cap rate anyways. And then you're doing all this work to only get half the payment. So, I think if I could go back in time, I would've bought every deal on a bridge loan. I would not have spent a single dollar in renovations and just operate it, wait five years and you sell it in today's environment for like a freaking 475 cap, that would have been a better decision with the benefit of hindsight. James: Yes, correct. Correct. So how would you-- sorry, in terms of cash flow vs. IRR vs. Equity multiply, right? So, what do you see, what is the most important number that-- for you, right, I know you're passive investors need to look at? Michael: Yes. You know, I think everyone, that everyone's different too. Like, all my investors have different things that are most important to them. I think, honestly at the end of the day, a pair of this investment, that investment, IRR is really kind of the driven. I'm not the biggest IRR in our store. We, I think the cash on cash certainly matters because I can't pay my bills on IRR, but I can with a check every month. So, I, that certainly protects it. But at the end of the day, really, we're focused kind of when we're-- comparing this, it's up to you in the next one, really kind of IRR. Because you know, if I'm able to come in this deal, I assume a mortgage and refinance in the third year or something like that and have a partial return of capital that pops my IRR pretty, pretty good. And I keep take some of this capital and return to my investors quickly. Two-year period, you know, 30% of their money back through a refi or something like that. That certainly is attractive. So, we'll, I think I kind of focused on IRR when I'm making the decisions on which deal, I want to buy, which deal I don't. And we've been, we like [inaudible39:54], we've been focused many deals about loan assumptions recently trying to get a lower basis. So, the first and foremost I'm focused on basis, making sure I buy a deal that's a relative value to everything else is trading right now. And I, cause I was only two things. You can't change on a property; you can't change your purchase price and you can't change location of it. Everything else you can kind of modify can always refinance it. I can always improve the property, but I can't change what price I paid or where it's located. So, we'll locate a deal with good prices, and I think everything else will kind of generally work itself out. James: Got It. And got it. How do you make decent between buy and hold for long term vs. buy and buy and refi? How do you decide? Michael: Yes, so if it's a syndicated deal, we've done a couple deals, especially when it first started out doing dentures where it's like what equity partner in us. Those deals we tend to hold longer. We bought a bunch of workforces, we sold them, we exchange, like A-minus or a product. So, we did a bunch of that. And then when it's a syndication people for like forever is not a good whole period if you're in syndication. Because people want, return on their money as well as return of their money and kind of the intermediate term. So, we're typically performing a five-year hold period. I think you'd be going much past seven. Most people kind of like, you know, shoot, I don't want to tie my money up for 10 years or 20 years. Now I kind of want to get my, I kind of want to see a return of my money as well as the return on my money. So, it kind of depends on the thing, but that's a heck of a lot of work buying and selling these things. So, it was just a lot easier just to kind of hold and it's kind of operate, especially the way we're set up with a third-party management company that does all day today. I, managing a bunch of thousands of apartment units. It's kind of like adult daycare. James: Yes, it's adult daycare, it's a good one to see. Michael: It's property management as a business of problems. I mean, there's always a problem, like every day, always, problems everywhere. So, if you have third-party management to kind of oversee that and we're set up and I have an asset manager that layered in between me and them. As a principal, the way we're set up, it's really not that bad on the day today. So, what we've been kind of focusing on is we're just selling the older stuff and buying newer, nicer stuff. Cause there's old stuff, I mean, not only, it was great, and we made a bunch of money, but you have asphalt parking lots and casts on sewers and t one 11 siding, Hardie. You go renovate a deal and two or three years later you've got to renovate the deal because the parking lot needs to be redone and you painted over wood. So, then you've got to have more wood of what, right? You got to go paint over again. And you can't cast, our sewers are collapsed in every time you turn around and get, dig it up and replaced sexting sewer pipe. So, you have all these like nonrecurring items that recurrent all the time. So, doesn't impact in a live per se, but it impacts your actual cash and the bottom line? So, I'm so I think the actual net cash you can pay out, it's not that different on a higher cap rate, older deal versus, or maybe a little bit lower cap rate, better quality deal if you're going to be in these deals for a long period of time. So, we've been just trying to get younger in our portfolio, so stuff I owned a day, I'd be much more likely to want to hold than the stuff I owned in 2014, 2013 cause those were just tougher, older, older deals. And I think that's what I've seen been kind of like the natural progression of most people that do what I do for a living. Just over time. One of the things, one of my mentors told me once when I first got in the business was, you own apartments in dog years, and every year of ownership feels like seven. So, like over time, you know that statement is very, very true. The older the property and the smaller the property, the more true that statement is. The bigger, nicer. It's just easy, just easier. So, I don't know if I answered your question,-- James: [inaudible43:42]. Michael: --but those are the-- between owning or selling a deal. James: Absolutely. Absolutely. And-- so let's go back to a bit more personal stuff, right? So, can you name like three things that you think is your secret sauce in, scaling up to this level? Michael: Yes, so, first and foremost, I mean I'm pretty tenacious and I had a lot of ambition, so, that was, that was a lot of it, right? I was like, I was willing to do what it takes to get to where I got. So, we had a lot of experience, background, and training and that certainly, so first and foremost, I just really, really, really wanted it. And like last weekend I flew to Jacksonville, not check, yes, Jacksonville, Florida, I'm sorry. Losing track of where I was. So, I was in Jacksonville for 21 hours. I spoke in front of 300 potential investors. I flew back home. I did that Saturday morning, came back Sunday morning and three weeks earlier I was in Newark, New Jersey, went to some hotel conference room on a Saturday, came back on Sunday. So, I'm willing to sacrifice a good chunk of my weekend to go out and get in front of investors so I can then do these larger deals. So, if you're not willing to put in the work and do what it takes and you're only, you're going to get a moderate your success for sure. Second thing was, I had a great background being a banker for over a decade and I just did deal after deal after deal. So, I've got a great education on my, on the bank Stein. So, most people don't have that. Cause then they're not bankers. Right. But, go get educated. That's the other thing I would, I would say get educated, higher from a reputable mentor. There's a lot of people out there put the time in. Become a student of your craft, go listen to this podcast, or listen to our podcasts, read books, do stuff like that. That’s a great way of learning. These podcasts are great. Like we host the Dole Capitol podcasts or your podcast. You're going to sit here and talk to me. So, it looks like about at least 45 minutes here- James: Yes. Michael: --at this point. And you get to your conversation from two guys that own almost 10,000 units collectively for 45 minutes for free. And there's a lot of wisdom and nuggets, but I think hopefully you can take out of that. Um, so, my background, my education was certainly it. And then really just a lot of its just relationships. You know what I mean? A lot of this is as simple as just don't be a jerk. That's, that's a lot of it, right? So, the brokers want to do business with people they know, like, and trust. They want you to be honest with them. They want you to be, do what you say you're going to do. And if you could just do that and be in a good guy and be friendly with them, man that goes a long way. It really does. So those are, those are three things I've done pretty well in this business. James: Got it, got it. And why do you do, what you do, I mean, where are you? Michael: I understood back, couple of things, right? To have a better life to be able to, the monetary if you'd have done well, the very rewarding monetarily. I sit back, so I got a couple of things happen, reflecting back on this, cause you know, we've done a lot in a short period of time. When I was 2010, so my mother passed away in 2010. So, I was like 32, I'm 32, 31, something like that at the time. And, so she was like 57 when at the time she passed away and then she-- her and my father sacrificed to save all their life to then be able to retire one day and then go have all those great traveling adventures in the sunlight and do stuff that was great in life and she didn't get to do that. She works to sacrificed and saved and I never got to-- the fruits of it. So, I kind of, that was a thing that kind of burned into my mind that I need to be able to do something young, unable to take a risk young. So, then I can then enjoy a lot of stuff in life. So shortly after, that's when I really first started was in 2011. I bought a bunch of rent houses in 2011. I [inaudible 47:28] my mom passed away and that's kind of really when I started like taking risks and doing stuff because being a banker, you're just naturally conservative. You're not really wanting to go take risks. But I started small and kind of got some confidence and then a transition in the multifamily. So that was one thing. And then, and then when I was about 34, 35, I was sitting at the bank and I worked for a large, large national bank and then, I was really successful, and they're kept trying to promote me. And, when I was looking at the bank and I looked at my boss and my boss's boss and his boss and thinking about what they do all day, it was kind of depressing, to be honest with you. Like I didn't want to do that. And I felt like a, it is a metaphorical thing, but it felt like a little fork in the road. Like I'm 34, 35 and if I don't go out and take a chance like right now, and I wait one more year, every year is, we made a little bit harder to go out and take this risk. But if I like go out right now, I saw the market, the market was right. Capital was blowing and the deals are so good. And I knew that because I was in the industry. So, I was like, if I go out and I fail I can always come back and be a banker because I was a really good banker and I can, y'all are going to need to be a banker. But if I go out and I succeed, then I can have a great life and get to go to Hawaii for three weeks. Like I'm going to this summer, I'm just going to pick up the family in Hawaii for three weeks. I'm just going to work from Hawaii for three weeks to sort of be in a hundred degrees in Dallas. Right. So that's what you, that's what I get to do today. And I get to pay for my sister and her family to go to Hawaii because we've taken the risk and been successful and those are-- that's kind of, I guess some of my whys right there. James: Yes. It's, it's interesting on how you're tenacious. I mean, whether its real estate or anything. And you can do this in anything, right to, you just have to be-- Michael: Yes. James: --persistent in doing it and know your why and just push it. And I can change your life. Right? So. Michael: In every transaction, there's always a problem, right. James: Yes. Michael: So that's the thing too. And that's what I always fall back on. Like there's always a problem. There's always stress, there's always, whatever. And you just got to like push through who's going to put your head down. You just got to push through. Just kind of will it, so do what you needed to do, you know? And not that every time I feel frustrated and you were not getting a deal, right? Like I've gone months and months on a deal, I just do more. Like, you know, I make more calls, I go do this, I'm proactive. I'm just like more always answer. So, we don't get what you want to do. More effort, not, that's usually, usually tends to work out pretty good for me. James: Good. Good. We're coming to the end. One more question. Do you have any like a daily habit or daily ritual that you do that contributes to your success or effectiveness in life? Michael: I'm not the most, I don't really read a lot of books. I don't really meditate on do any of that. So, what-- I, I do find myself from time to time, I'll go down the rabbit hole of doing something and like burn off 30 minutes by all my life around the internet or something like that in the middle of the day. And I always try to catch myself and say, okay, like I just need to prioritize. So, I have a hundred things to do every single day and I need to ensure I know what the most impactful thing is. And I focus my time on that. Cause, sometimes you let the tyranny of the urgent get in the way of the important. So just cause I have 40 emails on red, I need to go clear. It doesn't mean that's the most important thing for me to do right then. Even though that's like dinging on my screen in front of me. Sometimes I'll try to shut that out, focus on what are, what is the most important thing. And then I know when I, I'll schedule time to come back and clear my emails out an hour later down the road when I kind of get done the most important thing. Because, if you're in a Sproul, I'll leave you with, it's kind of, there's this whole thing that I've, I've definitely learned in this business, as a syndicator, as someone that does, find that puts together an apartment operators, apartment investment opportunities or any sort of opportunity like that. The best way you make, the way you make money in this business, you've got to find deals and find money. Going to find deals and find money and everything else is sort of noise. It’s all really important. You got to operate; you've got to do all their things right. But, that doesn't really, that's not driving revenue. So, if you want to focus on revenue, you've got to find deals or find money. So, I'm not talking to brokers, I'm not talking to my investors, you know, everything else is, not driving revenue. So, at the end of the day, I always try to remember that when I'm deciding, what do I spend my time on. Do I spend my time on this or that, that's always in the back of my mind? James: Got it. Got it. Is there anything else that you want to share in this podcast that you have not shared in hundreds of other podcasts that you have been? I should have [inaudible51:57]. Michael: I, I think, we do a pretty good job. So, I would, if you want to know more about me, I think really there's a couple of ways you can, the easiest way to find me, just get my company's website, which is a company spiadvisory, just go to our website www.spiadvisory.com. It's spi like spy advisory dot com. There's a contact us form, fill that out. I always happen to have in 10 or 15 minutes. A telephone call, listeners of the podcast. You guys are interested in maybe working with us or really the best way if you want to know more about me or if you listen to this podcast or [inaudible] or. So, you can listen to a dual capital podcast. So that's on iTunes or Stitcher or YouTube or anywhere you're probably listening to me right now. You can find the old capital real estate investing podcast. So, we have probably 300 episodes in the archive or more at this point. So, we do interviews with other people kind of similar to this format. As well as we do a little short one where my partner Paul interviews me and asked me one question a week and I answered about one specific topic. So, if you want to know anything about and just all-around apartment investing in your or some form or fashion. So you want to learn more about me, that's a good way to kind of-- I talk, I have a lot of stuff recorded that's out there that, but if you like this, you may, you may like that and hopefully can provide some, a little nub. It nuggets on different little talk topics, to listen to those. James: Yes. Yes. I learned a lot from you. I mean, listening to you from different, different podcasts throughout my apartment investing journey. So, I'm thankful for that. And I think that's it. Hopefully, all the audience and listeners got the value that they want to get or getting from Michael and myself. I think that's it. Thank you. Michael: All right. Thank you.  

My guest this week is Brian Nosek, co-Founder and the Executive Director of the Center for Open Science. Brian is also a professor in the Department of Psychology at the University of Virginia doing research on the gap between values and practices, such as when behavior is influenced by factors other than one's intentions and goals. The topic of this conversation is how incentives in academia lead to problems with how we do science, how we can fix those problems, the center for open science, and how to bring about systemic change in general. Show Notes Brian’s Website Brian on Twitter (@BrianNosek) Center for Open Science The Replication Crisis Preregistration Article in Nature about preregistration results The Scientific Method If you want more, check out Brian on Econtalk Transcript Intro   [00:00:00] This podcast I talked to Brian nosek about innovating on the very beginning of the Innovation by one research. I met Brian at the Dartmouth 60th anniversary conference and loved his enthusiasm for changing the way we do science. Here's his official biography. Brian nozik is a co-founder and the executive director for the center for open science cos is a nonprofit dedicated to enabling open and reproducible research practices worldwide. Brian is also a professor in the department of psychology at the University of Virginia. He's received his PhD from Yale University in 2002 in 2015. He was on Nature's 10 list and the chronicle for higher education influence. Some quick context about Brian's work and the center for open science. There's a general consensus in academic circles that there are glaring problems in how we do research today. The way research works is generally like this researchers usually based at a university do experiments then when they have a [00:01:00] result they write it up in a paper that paper goes through the peer-review process and then a journal publishes. The number of Journal papers you've published and their popularity make or break your career. They're the primary consideration for getting a position receiving tenure getting grants and procedure in general that system evolved in the 19th century. When many fewer people did research and grants didn't even exist we get into how things have changed in the podcast. You may also have heard of what's known as the replication crisis. This is the Fairly alarming name for a recent phenomena in which people have tried and failed to replicate many well-known studies. For example, you may have heard that power posing will make you act Boulder where that self-control is a limited resource. Both of the studies that originated those ideas failed to replicate. Since replicating findings a core part of the scientific method unreplicated results becoming part of Cannon is a big deal. Brian has been heavily involved in the [00:02:00] crisis and several of the center for open science is initiatives Target replication. So with that I invite you to join my conversation with Brian idzik.   How does open science accelerate innovation and what got you excited about it?   Ben: So the  theme that  I'm really interested in is  how do we accelerate Innovations? And so just to start off with I love to ask you sort of a really broad question of  in your mind. How does having a more open science framework help us accelerate Innovations? And I guess parallel to that. Why what got you excited about it first place. Brian: Yeah, yeah, so that this is really a core of why we started the center for open science is to figure out how can we maximize the progress of science given that we see a number of different barriers to or number of different friction points to the PACE and progress of [00:03:00] Science. And so there are a few things. I think that how. Openness accelerates Innovation, and I guess you can think of it as sort of multiple stages at the opening stage openness in terms of planning pre-registering what your study is about why you're doing this study that the study exists in the first place has a mechanism of helping to improve Innovation by increasing The credibility of the outputs. Particularly in making a clear distinction between the things that we planned in advance that we're testing hypotheses of ideas that we have and we're acquiring data in order to test those ideas from the exploratory results the things that we learn once we've observed the data and we get insights but there are necessarily more uncertain and having a clear distinction between those two practices is a mechanism for. Knowing the credibility of the results [00:04:00] and then more confidently applying results. That one observes in the literature after the fact for doing next steps. And the reason that's really important I think is that we have so many incentives in the research pipeline to dress up exploratory findings that are exciting and sexy and interesting but are uncertain as if they were hypothesis-driven, right? We apply P values to them. We apply a story upfront to them we present them as. These are results that are highly credible from a confirmatory framework. Yeah, and that has been really hard for Innovation to happen. So I'll pause there because there's lots more but yeah, so listen, let's touch on that.   What has changed to make the problem worse?   Ben: There's there's a lot that right there. So you mentioned the incentives to basically make. Things that aren't really following the scientific method follow the clicker [00:05:00] following the scientific method and one of the things I'm always really interested in what has changed in the incentives because I think that there's definitely this. Notion that this problem has gotten worse over time. And so that means that that something has has changed and so in your mind like what what changed to make to sort of pull science away from that like, you know sort of ice training ideal of you have your hypothesis and then you test that hypothesis and then you create a new hypothesis to this. System that you're pushing back against. Brian: You know, it's a good question. So let me start with making the case for why we can say that nothing has changed and then what might lead to thinking something has changed in unpacking this please the potential reason to think that nothing has [00:06:00] changed is that the kinds of results that are the most rewarded results have always been the kinds of results that are more the most rewarded results, right? If I find a novel Finding rather than repeating something someone else has done. I'm like. To be rewarded more with publication without latex cetera. If I find a positive result. I'm more likely to gain recognition for that. Then a negative result. Nothing's there versus this treatment is effective, which one's more interesting. Well, we know which ones for interesting. Yeah. Yeah, and then clean and tidy story write it all fits together and it works and now I have this new explanation for this new phenomenon that everyone can can take seriously so novel positive clean and tidy story is the. They'll come in science and that's because it breaks new ground and offers a new idea and offers a new way of thinking about the world. And so that's great. We want those. We've always wanted those things. So the reason to think well, this is a challenge always is [00:07:00] because. Who doesn't want that and and who hasn't wanted that right? It turns out my whole career is a bunch of nulls where I don't do anything and not only fits together. It's just a big mess right on screen is not a way to pitch a successful career. So that challenge is there and what pre-registration or committing an advanced does is helps us have the constraints. To be honest about what parts of that are actual results of credible confrontations of pre-existing hypotheses versus stuff that is exploring and unpacking what it is we can find. Okay, so that in this in the incentive landscape, I don't think has changed. Mmm what thanks have changed. Well, there are a couple of things that we can point to as potential reasons to think that the problem has gotten worse one is that data acquisition many fields is a lot easier than it ever was [00:08:00] and so with access more data and more ways to analyze it more efficient analysis, right? We have computers that do this instead of slide rules. We can do a lot more adventuring in data. And so we have more opportunity to explore and exploit the malays and transform it into things signal. The second is that the competitive landscape is. Stronger, right there are fewer than the ratio of people that want jobs to jobs available is getting larger and larger and larger and that fact and then competitiveness for Grants and same way that competition than can. Very easily amplify these challenges people who are more willing to exploit more researcher degrees of freedom are going to be able to get the kinds of results more easily that are rewarded in the system. And so that would have amplify the presence of those in people that managed to [00:09:00] survive that competitive firm got it. So I think it's a reasonable hypothesis that people that it's gotten worse. I don't think there's definitive evidence but those would be the theoretical points. At least I would point to for that. That makes a lot of sense. So you had a just sort of jumping back. You had a couple a couple points and we had we have just touched on the first one.   Point Number Two about Accelerating Innovation   Ben: So I want to give you that chance to oh, yeah go back and to keep going through that. Brian:  Right. Yeah. So accelerating Innovation is the idea, right? So that's a point of participation is accelerating Innovation by by clarifying The credibility of claims as they are produced. Yes, we do that better than I think will be much more efficient that will have a better understanding of the evidence base as it comes out. Yeah second phase is the ability is the openness of the data and materials for the purposes of verify. Those [00:10:00] initial claims right? I do a study. I pre-registered. It's all great and I share it with you and you read it. And you say well that sounds great. But did you actually get that and what would have happened if you made different decisions here here and there right because I don't quite agree with the decisions that you made in your analysis Pipeline and I see some gaps there so you're being able to access the materials that I produced in the data that came from. Makes it so that you can one just simply verify that you can reproduce the findings that I reported. Right? I didn't just screw up the analysis script or something and that as a minimum standard is useful, but even more than that, you can test the robustness in ways that I didn't and I came to that question with some approach that you might look at it and say well I would do it differently and the ability to reassess the data for the same question is a very useful thing for. The robustness particularly in areas that are that have [00:11:00] complex analytic pipelines where there's are many choices to make so that's the second part then the third part is the ReUse. So not only should we be able to verify and test the robustness of claims as they happen, but data can be used for lots of different purposes. Sometimes there are things that are not at all anticipated by the data originator. And so we can accelerate Innovation by making it a lot easier to aggregate evidence of claims across multiple Studies by having the data being more accessible, but then also making that data more accessible and usable for. Studying things that no one no one ever anticipated trying to investigate. Yeah, and so the efficiency gain on making better use of the data that already exists rather than the Redundant just really do Revenue question didn't dance it your question you did as it is a massive efficiency. Opportunity because there is a lot of [00:12:00] data there is a lot of work that goes in why not make the most use of it began?   What is enabled by open science?   Ben: Yeah that makes a lot of sense. Do you have any like really good sort of like Keystone examples of these things in action like places where because people could replicate the. The the study they could actually go back to the pipeline or reuse the data that something was enabled. That wasn't that wouldn't have been possible. Otherwise, Brian: yeah. Well, let's see. I'll give a couple of local mean personal examples just to just to illustrate some of the points, please so we have the super fun project that we did just to illustrate this second part of the pipeline right this robustness phase of. People may make different choices and those choices may have implications for the reliability results. So what we did in this project was that we get we acquired a dataset [00:13:00] of a very rich data set of lots of players and referees and outcomes in soccer and we took that data set and then we recruit a different teams. 29 in the end different teams with lots of varied expertise and statistics and analyzing data and have them all investigate the same research. Which is our players with darker skin tone more likely to get a red card then players with lighter skin tone. And so that's you know, that's a question. We'll of Interest people have studied and then we had provided this data set. Here's a data set that you can use to analyze that and. The teams worked on their own and developed an analysis strategies for how they're going to test that hypothesis. They came up with their houses strategy. They submitted their analysis and their results to us. We remove the results and [00:14:00] then took their analysis strategies and then share them among the teams for peer review right different people looking at it. They have made different choices. They appear each other and then went back. They took those peer reviews. They didn't know what each other found but they took. Because reviews and they wanted to update their analysis they could and so they did all that and then submitted their final analyses and what we observed was that a huge variation in analysis choices and variation in the results. So as a simple Criterion for Illustrated the variation results two-thirds of the teams found a significant. Write P less than 0.05 standard for deciding whether you see something there in the data, right and Atherton teams found a null. So the and then of course they debated amongst each other which was analysis strategy was the right strategy but in the end it was very clear among the teams that there are lots of reasonable choices that could be made. And [00:15:00] those reasonable choices had implications for the results that were observed from the same data. Yeah, and it's Standard Process. We do not see the how it's not easy to observe how the analytics choices influence the results, right? We see a paper. It has an outcome we say those are what the those fats those the outcomes of the data room. Right, but what actually the case is that those are the outcomes the data revealed contingent on all those choices that the researcher made and so that I think just as an illustrative illustrative. So it helps to figure out the robustness of that particular finding given the many different reasonable choices. That one would make where if we had just seen one would have had a totally different interpretation, right either. Yeah, it's there or it's not there.   How do you encode context for experiments esp. with People?   Ben:  Yeah, and in terms of sort of that the data and. [00:16:00] Really sort of exposing the the study more something that that I've seen especially in. These is that it seems like the context really matters and people very often are like, well there's there's a lot of context going on in addition to just the procedure that's reported. Do you have any thoughts on like better ways of sort of encoding and recording that context especially for experiments that involve? Brian: Yeah. Yeah. This is a big challenge is because we presume particularly in the social and life sciences that there are many interactions between the different variables. Right but climate the temperature the time of day the circadian rhythms the personalities whatever it is that is the different elements of the subjects of the study whether they be the plants or people or otherwise, yeah. [00:17:00] And so the. There are a couple of different challenges here to unpack one is that in our papers? We State claims at the maximal level of generality. We can possibly do it and that that's just a normal pattern of human communication and reasoning right? I do my study in my lab at the University of Virginia on University of Virginia undergraduates. I don't conclude in the. University of university University of Virginia undergraduates in this particular date this particular time period this particular class. This is what people do with the recognition that that might be wrong right with recognition. There might be boundary conditions but not often with articulating where we think theoretically those boundary conditions could be so in one step of. Is actually putting what some colleagues in psychology of this great paper about about constraints on [00:18:00] generality. They suggest what we need in all discussion sections of all papers is a sexually say when won't this hold yeah, just give them what you know, where where is this not going to hold and just giving people an occasion to think about that for a second say oh. - okay. Yeah, actually we do think this is limited to people that live in Virginia for these reasons right then or no, maybe we don't really think this applies to everybody but now we have to say so you can get the call it up. So that alone I think would make a huge difference just because it would provide that occasion to sort of put the constraints ourselves as The Originators of findings a second factor, of course is just sharing as much of the materials as possible. But often that doesn't provide a lot of the context particularly for more complex experimental studies or if there are particular procedural factors right in a lot of the biomedical Sciences there. There's a lot of nuance [00:19:00] into how it is that this particular reagent needs to be dealt with how they intervention needs to be administered Etc. And so I like the. Moves towards video of procedures right? So there is a journal Journal of visualized events jove visualized experiments that that that tries to that gives people opportunities to show the actual experimental protocol as it is administered. To try to improve it a lot of people using the OSF put videos up of the experiment as they administered it. So to maximize your ability to sort of see how it is that it was done through. So those steps I think can really help to maximize the transparency of those things that are hard to put in words or aren't digitally encoded oil. Yeah, and those are real gaps   What is the ultimate version of open science?   Ben: got it. And so. In your mind what is sort of like the endgame of all this? What is it? Like what [00:20:00] would be the ideal sort of like best-case scenario of science? Like how would that be conducted? So I say you get to control the world and you get to tell everybody practicing science exactly what to do. What would that look like? Brian: Well, if it if I really had control we would just all work on Wikipedia and we would just revising one big paper with the new applicants. Ask you got it continuously and we get all of our credit by. You know logging how many words that I changed our words that survived after people have made their revisions and whether those words changed are on pages that were more important for the overall scientific record versus the less important spandrels. And so we would output one paper that is the summary of knowledge, which is what Wikipedia summarizes. All right, so maybe that's that's maybe going a little bit further than what like [00:21:00] that we can consider. The realm of conceptually possible. So if we imagine a little bit nearer term, what I would love to see is the ability to trace the history of any research project and that seems more achievable in the sense that. If a every in fact, my laboratory is getting close to this, right every study that we do is registered on the OSF. And once we finish the studies, we post the materials and the data or as we're doing it if we're managing the materials and data and then we attach a paper if we write a paper at the end preprint or the final report so that people can Discover it and all of those things are linked together. Be really cool if I had. Those data in a standardized framework of how it is that they are [00:22:00] coded so that they could be automatically and easily integrated with other similar kinds of data so that someone going onto the system would be able to say show me all the studies that ever investigated this variable associated with this variable and tell me what the aggregate result is Right real-time meta-analysis of the entire database of all data that I've ever been collected that. Enough flexibility would help to really very rapidly. I think not just spur Innovations and new things but to but help to point out where there are gaps right there a particular kinds of relationships between things particular effects of predict interventions where we know a ton and then we have this big assumption in our theoretical framework about how we get from X to y. And then as we look for variables that help us to identify whether X gets us to why we feel there just isn't stuff. The literature has not filled that Gap. So I think there are huge benefits for that [00:23:00] kind of aggregate ability. But mostly what I want to be able to do is instead of saying you have to do research in any particular way. The only requirement is you have to show us how you did your research and your particular way so that the marketplace of ideas. Can operate as efficiently as possible and that really is the key thing? It's not preventing bad ideas from getting into the system. It's not about making sure that the different kinds of best things are the ones that immediately are through with not that about Gatekeepers. It's about efficiency in how it is. We call that literature of figuring out which things are credible which things are not because it's really useful to. The ideas into the system as long as they can be. Self-corrected efficiently as well. And that's where I think we are not doing well in the current system. We're doing great on generation. [00:24:00] We're General kinds of innovative ideas. Yeah, but we're not is parsing through those ideas as efficiently as it could decide which ones are worth actually investing more resources in jumping. A couple levels in advance that   Talmud for Science   Ben:  that makes a lot of sense and actually like I've definitely come across many papers just on the internet like you go and Google Scholar and you search and you find this paper and in fact, it has been refuted by another paper and there's no way to know that yeah, and so. I does your does the open science framework address that in any way? Brian:  No, it doesn't yet. And this is a critical issue is the connectivity between findings and the updating of knowledge because the way that like I said doesn't an indirect way but it doesn't in the systematic way that actually would solve this problem. The [00:25:00] main challenge is that we treat. Papers as static entities. When what their summarizing is happening very dynamically. Right. It may be that a year later. After that paper comes out one realizes. We should have analyze that data totally different. We actually analyzed it wrong is indefensible the way that we analyzed it. Right right. There are very few mechanisms for efficiently updating that paper in a way that would actually update the knowledge and that's something where we all agree. That's analyze the wrong way, right? What are my options? I could. Retract the paper. So it's no longer in existence at all. Supposedly, although even retracted papers still get cited we guess nuts. So that's a base problem. Right or I could write a correction, which is another paper that comments on that original paper that may not itself even be discoverable with the original paper that corrects the analysis. Yeah, and that takes months and years. [00:26:00] All right. So the really what I think is. Fundamental for actually addressing this challenge is integrating Version Control with scholarly publishing. So that papers are seen as Dynamic objects not static objects. And so if you know what I would love to see so here's another Milestone of this if we if I could control everything another Milestone would be if a researcher could have a very productive career with. Only working on a single paper for his or her whole life, right? So they have a really interesting idea. And they just continue to investigate and build the evidence and challenge it and figure, you know, just continue to unpack it and they just revise that paper over time. This is what we understand. Now, this is where it is. Now. This is what we've learned over here are some other exceptions but they just keep fine-tuning it and then you get to see the versions of that paper over its [00:27:00] 50-year history as that phenomenon got unpacked that. Plus the integration with other literature would make this much more efficient for exactly the problem that you raised which is we with papers. We don't know what the current knowledge base is. We have no real good way except for these. These attempts to summarize the existing literature with yet a new paper and that doesn't then supersede those old papers. It's just another paper is very inefficient system.   Can Social Sciences 'advance' in the same way as the physical sciences?   Ben: Ya know that that totally makes sense. Actually. I just I have sort of a meta question that I've argued with several people about which is do you feel like. We can make advances in our understanding of sort of like [00:28:00] human-centered science in the same way that we can in like chemistry or physics. Like people we very clearly have like building blocks of physics and the Builds on itself. And there's I've had debates with people about whether you can do this in. In the humanities and the social sciences. What are your thoughts on that? Brian:  Yeah. It is an interesting question and the. What seems to be the biggest barrier is not anything about methodology in particular but about complexity? Yeah, right, if the problem being many different inputs can have similar impact cause similar kinds of outcomes and singular inputs can have multivariate outcomes that it influences and all of those different inputs in terms of causal elements may have interactive effects on the [00:29:00] outs, so. How can we possibly develop Rich enough theories to predict the actions effectively and then ultimately explain the actions effectively of humans in a complex environments. It doesn't seem that we will get to the beautiful equations that underlie a lot of physics and chemistry and count for a substantial amount of evidence. So the thing that I don't feel like I under have any good hand along with that is if it's a theoretical or practical limit right is it just not possible because it's so complex and there isn't this predicted. Or it's just that's really damn hard. But if we had big enough computers if you had enough data, if we were able to understand complex enough models, we would be able to predict it. Right so is as a mom cycle historians, right? They figure it out right the head. [00:30:00] Oxidizing web series righty they could account for 99.9 percent of the variance of what people do next and but of course, even there it went wrong and that was sort of the basis of the whole ceilings. But yeah, I just don't know I don't have a way to. I don't yet have a framework for thinking about how is it that I could answer that question whether it's a practical or theoretical limit. Yeah. What do you think? Ben:  What do I think I think that it's great. Yeah, so I usually actually come down on the I think it's a practical limit now how much it would take to get there might make it effectively a theoretical limit right now. But that there's there's nothing actually preventing us from like if you if you could theoretically like measure everything why not? I [00:31:00] think that is just with again. It's like the it's really a measurement problem and we do get better at measuring things. So that's the that's that's where I come down on but I.   How do you shift incentives in science?   Yep, that's just purely like I have no good argument. going going back to the incentives. It seems to me like a lot of what like I'm completely convinced that these changes would. Definitely accelerate the number of innovations that we have and so and it seems like a lot of these changes require shifting scientists incentives. And so and that's like a notoriously hard thing so we both like how are you going about shifting those incentives right now and how might they be shifted in the future. [00:32:00] Brian: Yeah, that's a great question. That's what we spend. A lot of our time worrying about in the sense of there is very little at least in my experience is very distal disagreement on the problems and the opportunities for improving the pace of Discovery and Innovation based on the solutions. It really is about the implementation. How is it that you change that those cultural incentives so that we can align. The values that we have for science with the practices that researchers do on a daily basis and that's a social problem. Yeah, there are technical supports. But ultimately it's a social problem. And so the the near term approach that we have is to recognize the systems of rewards as they are. And see how could we refine those to align with some of these improved practices? So we're not pitching. Let's all work on [00:33:00] Wikipedia because that's that is so far distant from. What they systems have reward for scientist actually surviving and thriving in science that we wouldn't be able to get actually pragmatic traction. Right? So I'll give one example of can give a few but here's the starting with one of an example that integrates with current incentives but changes them in a fundamental way and that is the publishing model of registered reports. Sophie in the standard process right? I do my research. I write up my studies and then I submit them for peer review at the highest possible prestigious Journal that I can hoping that they will not see all the flaws and if they'll accept it. I'll get all the do that process me and I understand it anyway - journal and the P plus Terminal C and eventually somewhere and get accepted. The register report model makes one change to the process and that is to move. The critical point of peer review [00:34:00] from after the results are known and I've written up the report and I'm all done with the research to after I've figured out what the question that I want to investigate is and what the methodology that I'm going to use so I don't have an observed the outcomes yet. All I've done is frame question. An articulated why it's important and a methodology that I'm going to just to test that question and that's what the peer reviewers evaluate right? And so the key part is that it fits into the existing system perfectly, right? The the currency of advancement is publication. I need to get as many Publications as I can in the most prestigious Outlets. I can to advance my career. We don't try to change that. Instead we just try to change. What is the basis for making a decision about publication and by moving the primary stage of peer reviewed before the results are known does a fundamental change in what I'm being rewarded for as the author [00:35:00] right? Yeah, but I'm being rewarded for as the author in the current system is sexy results, right get the best most interesting most Innovative results. I can write and the irony of that. Is that the results of the one thing that I'm not supposed to be able to control in your study? Right? Right. What I'm supposed to be able to control is asking interesting questions and developing good methodologies to test those questions. Of course that's oversimplifying a bit. There are in there. The presumption of emphasizing results is that my brilliant insights at the outset of the project are the reason that I was able to get those great results, right, but that depends on the credibility of that entire Pipeline and put that aside but the moving it to at the design stage means that my incentive as an author is to ask the most important questions that I can. And develop the most compelling and effective and valid methodologies that I can to test them. [00:36:00] Yeah, and so that changes to what it is presumably we are supposed to be being rewarded for in science. The other thing that it changes in the there's a couple of other elements of incentive changes that it has an impact on that are important for the whole process right for reviewers instant. It's. When I am asked to review a paper in my area of research when I when all the results are there, I have skin in the game as a reviewer. I'm an expert in that area. I may have made claims about things in that particular area. Yeah, if the paper challenges my cleanse make sure to find all kinds of problems with the methodology. I can't believe they did this is this is a ridiculous thing, right? We write my paper. That's the biggest starting point problem challenge my results all well forget out of you. But the amount of course if it's aligned with [00:37:00] my findings and excites me gratuitously, then I will find lots of reasons to like the paper. So I have these Twisted incentives to reinforce findings and behave ideologically as a reviewer in the existing system by moving peer review to the design stage. It fundamentally changes my incentives to right so say I'm in a very contentious area of research and there's only ten opponents on a particular claim when we are dealing with results You can predict the outcome right it people behave ideologically even when they're not trying to when you don't know the results. Both people have the same interests, right? If I truly believe in the phenomenon that I'm studying and the opponents of my point of view also believe in their perspective, right then both want to review that study and that design and that methodology to maximize its quality to reveal the truth, which I think I [00:38:00] have and so that alignment actually makes adversaries. To some extent allies and in review and makes the reviewer and the author more collaborative, right the feedback that I give on that paper can actually help the methodology get better. Whereas in the standard process when I say here's all the things you did wrong. All the author has this to say well geez, you're a jerk. Like I can't do anything about that. I've already done the research and so I can't fix it. Yeah. So the that shifts earlier is much more collaborative and helps with that then the other question is the incentives for the journal right? So in the. Journal editors have strong incentives of their own they want leadership. They want to have impact they don't want the one that destroyed their journal and so [00:39:00] the incentives and the in the existing model or to publish sexy results because more people were read those results. They might cite those results. They might get more attention for their Journal, right? And shifting that to on quality designs then shift their priorities to publishing the most rigorous research the most rust robust research and to be valued based on that now. Yeah, so I'll pause there there's lots of other things to say, but those I think are some critical changes to the incentive landscape that still fits. Into the existing way that research is done in communicated.   Don't people want to read sexy results?   Ben: Yeah. I have a bunch of questions just to poke at that last point a little bit wouldn't people still read the journals that are publishing the most sexy results sort of regardless of whether they were web what stage they're doing that peer review. Brian:  Yeah. This is a key concern of editors and thinking about adopting registered reports. [00:40:00] So we have about a hundred twenty-five journals that are offering this now, but we continue to pitch it to other groups and other other ones, but one of the big concerns that Hunters have is if I do this then I'm going to end up publishing a bunch of no results and no one will read my journal known will cite it and I will be the one that ruined my damn door. All right. So it is a reasonable concern because of the way the system works now, so there's a couple answers to that but the one is empirical which is is it actually the case that these are less red or less cited than regular articles that are published in those. So we have a grant from the McDonald Foundation to actually study registered reports. And the first study that we finished is a comparison of articles that were done as register reports with this in the same published in the same Journal. [00:41:00] Articles that were done the regularly to see if they are different altmetrics attention, right citation and attention and Oppa in media and news and social media and also citation impact at least early stage citation impact because the this model is new enough that it isn't it's only been working for since 2014. In terms of first Publications and what we found in that is that at least in this initial data set. There's no difference in citation rates, and if anything the register report. Articles have gotten more altmetric impact social media news media. That's great. So at least the initial data suggests that who knows if that will sustain generalize, but the argument that I would make in terms of a conceptual argument is that if Studies have been vetted. In terms of without knowing the results. These are important results to know [00:42:00] right? So that's what the actors and the reviewers have to decide is do we need to know the outcome of this study? Yeah, if the answer is yes that this is an important enough result that we need to know what happened that any result is. Yeah, right. That's the whole idea is that we're doing the study harder find out what the world says about that particular hypothesis that particular question. Yeah, so it become citable. Whereas when were only evaluating based on the results. Well, yeah things that Purity people is that that's crazy, but it happened. Okay, that's exciting. But if you have a paper where it's that's crazy and nothing happened. Then people say well that was a crazy paper. Yeah, and that paper would be less likely to get through the register report kind of model that makes a lot of sense. You could even see a world where because they're being pre-registered especially for more like the Press people can know to pay attention to it. [00:43:00] So you can actually almost like generate a little bit more height. In terms of like oh we're not going to do this thing. Isn't that exciting? Yeah, exactly. So we have a reproducibility project in cancer biology that we're wrapping up now where we do we sample a set of studies and then try to replicate findings from those papers to see where where can we reproduce findings in the where are their barriers to be able to reproduce existing? And all of these went through the journal elife has registered reports so that we got peer review from experts in advance to maximize the quality of the designs and they published instead of just registering them on OSF, which they are they also published the register reports as an article of its own and those did generate lots of Interest rule that's going to happen with this and that I think is a very effective way to sort of engage the community on. The process of actual Discovery we don't know the answer to these [00:44:00] things. Can we build in a community-based process? That isn't just about let me tell you about the great thing that I just found and more about. Let me bring you into our process. How does were actually investigating this problem right and getting more that Community engagement feedback understanding Insight all along the life cycle of the research rather than just as the end point, which I think is much more inefficient than it could be.   Open Science in Competitive Fields and Scooping   Ben: Yeah and. On the note of pre-registering. Have you seen how it plays out in like extremely competitive Fields? So one of the world's that I'm closest to is like deep learning machine learning research and I have friends who keep what they're doing. Very very secret because they're always worried about getting scooped and they're worried about someone basically like doing the thing first and I could see people being hesitant to write down to [00:45:00] publicize what they're going to do because then someone else could do it. So, how do you see that playing out if at all? Brian: Yeah scoping is a real concern in the sense that people have it and I think that is also a highly inflated concern based on the reality of what happens in practice but nevertheless because people have the concern systems have to be built to address it. Yeah, so one simple answer on the addressing the concern and then reasons to be skeptical at the. The addressing the concern with the OSF you can pre-register an embargo your pre-registrations from to four years. And what that does is it still gets all the benefits of registering committing putting that into an external repository. So you have independent verification of time and date and what you said you were going to do but then gives you as the researcher the flexibility to [00:46:00] say I need this to remain private for some period of time because of whatever reason. As I need it to be private, right? I don't want the recent participants that I am engaged in this project to discover what the design is or I don't want it competitors to discover what the design is. So that is a pragmatic solution is sort of dress. Okay, you got that concern. Let's meet that concern with technology to help to manage the current landscape. There are a couple reasons to be skeptical that the concern is actually much of a real concerning practice Tristan. And one example comes from preprints. So a lot of people when they pre princess sharing the paper you have of some area of research prior to going through peer review and being published in a journal write and in some domains like physics. It is standard practice the archive which is housed at Cornell is the standard for [00:47:00] anybody in America physics to share their research through archive prior to publication in other fields. It's very new or unknown but emerging. But the exact same concern about scooping comes up regularly where they say there's so many people in our field if I share a preprint someone else with the lab that is productive lab is going to see my paper. They're going to run the studies really fast. They're going to submit it to a journal that will publish and quickly and then I'll lose my publication because it'll come out in this other one, right and that's a commonly articulated concern. I think there are very good reasons to be skeptical of it in practice and the experience of archive is a good example. It's been operating since 1991 physicists early in its life articulated similar kinds of concerns and none of them have that concern now, why is it that they don't have that concern now? Well the Norms have shifted from the way you establish priority [00:48:00] is not. When it's published in the journal, it's when you get it onto archive. Right? Right. So a new practice becomes standard. It's when is it that the community knows about what it is you did that's the way you get that first finder Accolade and that still carries through to things like publication a second reason is that. We all have a very inflated sense of self importance that our great our kids right? There's an old saw in in venture capital of take your best idea and try to give it to your competitor and most of the time you can write. We think of our own ideas really amazing and everyone else doesn't yeah people sleeping other people. Is Right Southern the idea that there are people looking their chops on waiting for your paper your registration to show up so they can steal your [00:49:00] idea and then use it and claim it as their own is is great. It's shows High self-esteem. And that's great. I am all for high self. I don't know and then the last part is that. It is a norm violation to do that to such a strong degree to do the stealing of and not crediting someone else for their work, but it's actually very addressable in the daily practice of how science operates which is if you can show that you put that registration or that paper up on a independent service and then it was it appeared prior to the other person doing it. And then that other group did try to steal it and claim it as their own. Well, that's misconduct. And if they did if they don't credit you as the originator then that's something that is a norm violation and how science operates and I'm actually pretty confident in the process of dealing with Norm [00:50:00] violations in the scientific Community. I've had my own experience with the I think this very rarely happens, but I have had an experience with it. I've posted papers on my website before there were pretty print services in the behavioral sciences since I. Been a faculty member and I've got a Google Scholar one day and was reading. Yeah, the papers that I have these alerts set up for things that are related to my work and I paper showed up and I was like, oh that sounds related to some things. I've been working on. So I've clicked on the link to the paper and I went to the website. So I'm reading the paper. I from these authors I didn't recognize and then I realized wait that's that's my paper. I need a second and I'm an author and I didn't submit it to that journal. And it was my paper. They had taken a paper off of my website. They had changed the abstract. They run it through Google translate. It looks like it's all Gobbledy gook, but it was an abstract. But the rest of it was [00:51:00] essentially a carbon copy of our paper and they published. Well, you know, so what did I do? I like contacted the editor and we actually is on retraction watch this story about someone stealing my paper and retraction watch the laughing about it and it got retracted. And as far as we heard the person that had gone it lost their job, and I don't know if that's true. I never followed. But there are systems place is the basic point to deal with the Regis forms of this. And so I have I am sanguine about those not be real issues. But I also recognize they are real concerns. And so we have to have our Technology Solutions be able to address the concerns as they exist today. And I think the those concerns will just disappear as people gain experience.   Top down v Bottom up for driving change   Ben: Got it. I like that distinction between issues and concerns that they may not be the same thing. To I've been paying attention to   sort of the tactics that you're [00:52:00] taking to drive this adoption. And there's  some bottom up things in terms of changing the culture and getting  one Journal at a time to change just by convincing them and there's also been some some top-down approaches that you've been using and I was wondering if you could just sort of go through those and what you feel like. Is is the most effective or what combinations of things are are the most effective for really driving this change? Brian: Yeah. No, it's a good question because this is a culture change is hard especially with the decentralized system like science where there is no boss and the different incentive drivers are highly distributed. Right, right. He has a richer have a unique set of societies. Are relevant to establishing my Norms you could have funders that fund my work a unique set of journals that I publish in and my own institution. And so every researcher [00:53:00] has that unique combination of those that all play a role in shaping the incentives for his or her behavior and so fundamental change if we're talking about just at the level of incentives not even at the level of values and goals requires. Massive shift across all of those different sectors not massive in terms of the amount of things they need to shift but in the number of groups that need to make decisions tissue. Yeah, and so the we need both top-down and bottom-up efforts to try to address that and the top down ones are. That we work on at least are largely focused on the major stakeholders. So funders institutions and societies particularly ones that are publishing right so journals whether through Publishers societies, can we get them like with the top guidelines, which is this framework that that has been established to promote? What are the transparency standards? What could we [00:54:00] require of authors or grantees or employees of our organizations? Those as a common framework provide a mechanism to sort of try to convince these different stakeholders to adopt new standards new policies to that that then everybody that associated with that have to follow or incentivised to follow simultaneously those kinds of interventions don't necessarily get hearts and minds and a lot of the real work in culture change. Is getting people to internalize what it is that mean is good science is rigorous work and that requires a very bottom up community-based approach to how Norms get established Within. What are effectively very siloed very small world scientific communities that are part of the larger research community. And so with that we do a lot [00:55:00] of Outreach to groups search starting with the idealists right people who already want to do these practices are already practicing rigorous research. How can we give them resources and support to work on shifting those Norms in their small world communities and so. Out of like the preprint services that we host or other services that allow groups to form. They can organize around a technology. There's a preprint service that our Unity runs and then drive the change from the basis of that particular technology solution in a bottom-up way and the great part is that to the extent that both of these are effective they become self reinforcing. So a lot of the stakeholder leaders and editor of a journal will say that they are reluctant. They agree with all the things that we trying to pitch to them as ways to improve rigor and [00:56:00] research practices, but they don't they don't have the support of their Community yet, right. They need to have people on board with this right well in we can the bottom. It provides that that backing for that leader to make a change and likewise leaders that are more assertive are willing to sort of take some chances can help to drive attention and awareness in a way that facilitates the bottom-up communities that are fledgling to gain better standing and we're impact so we really think that the combination of the two is essential to get at. True culture change rather than bureaucratic adoption of a process that now someone told me I have to do yeah, which could be totally counterproductive to Scientific efficiency and Innovation as you described. Ben: Yeah, that seems like a really great place to to end. I know you have to get running. So I'm really grateful. [00:57:00] This is this has been amazing and thank you so much. Yeah, my pleasure.