Podcasts about brian you

Achieve Wealth Through Value Add Real Estate Investing Podcast

Play Episode Listen Later Oct 22, 2019 35:21

David Stephenson, Ph.D., is the author of Big Data Demystified, a guide for executives that explores the transformative nature of big data and data analytics. He’s also a data strategy consultant and professor at the University of Amsterdam. In a previous life, David worked in various data science roles at companies like Adidas, Coolblue, and eBay. Join David and I as we discuss what makes data science projects succeed and explore: The non-technical issues that lead to ineffective data science and analytics projects The specific type of communication that is critical to the success of data science and analytics initiatives (and how working in isolation from your stakeholder or business sponsor creates risk)) The power of showing value early, starting small/lean, and one way David applies agile to data science projects The problems that emerge when data scientists only want to do “interesting data science” How design thinking can help data scientists and analytics practitioners make their work resonate with stakeholders who are not “data people” How David now relies on design thinking heavily, and what it taught him about making “cool” prototypes nobody cared about What it’s like to work on a project without understanding who’s sponsoring it Resources and Links DSI Analytics Website Connect with David on LinkedIn David’s book: Big Data Demystified On Twitter: @Stephenson_Data Quotes from Today’s Episode “You see a lot of solutions being developed very well, which were not designed to meet the actual challenge that the industry is facing.” — David “You just have that whole wasted effort because there wasn’t enough communication at inception.” — David “I think that companies are really embracing agile, especially in the last few years. They’re really recognizing the value of it from a software perspective. But it’s really challenging from the analytics perspective—partly because the data science and analytics. They don’t fit into the scrum model very well for a variety of reasons.” — David “That for me was a real learning point—to understand the hardest thing is not necessarily the most important thing.” — David “If you’re working with marketing people, an 80% solution is fine. If you’re working with finance, they really need exact numbers. You have to understand what your target audience needs in terms of precision.” — David “I feel sometimes that when we talk about “the business” people don’t understand that the business is a collection of people—just like a government is a collection of real humans doing jobs and they have goals and needs and selfish interests. So there’s really a collection of end customers and the person that’s paying for the solution.” — Brian “I think it’s always important—whether you’re a consultant or you’re internal—to really understand who’s going to be evaluating the value creation.”— Brian “You’ve got to keep those lines of communication open and make sure they’re seeing the work you’re doing and evaluating and giving feedback on it. Throw this over the wall is a very high risk model.” — Brian Transcript Brian: David Stephenson is the author of Big Data Demystified, and he’s a consulting data strategist primarily working out of Europe. I’m excited to have my chat with David on the show here today because I think David has a really good way of connecting a highly technical background with the business and kind of act like a bridge between these two departments. Brian: All right. David, are you on the line? David: Yes, I’m here. Brian: I’m super excited to talk to you. So we have David Stephenson, the consultant and author of Big Data Demystified. we met through, somehow we met, I think it was on LinkedIn originally. And you also run Predictive Analytics World Conference, in the UK edition, correct? David: Yeah, that’s right. Brian: Awesome. Tell people a little bit… I gave you an intro there, but it’s always better from the horse’s mouth. So tell, tell us a little bit about what you’re what you’re up to. David: Yeah, sure. Um so I used to run global uh business analytics for the eBay classifieds group and then uh started off on my own a couple of years ago and have been doing consulting uh mostly around Europe in uh data strategy, data science program developments. And then recently uh published a book also on the topic uh with Financial Times Press called Big Data Demystified. I was just doing a lot of work between speaking and teaching and training and consulting with companies, really focused on uh developing how they use data and analytics and helping them reach the next steps in that. So it’s been keeping me, keeping me quite busy recently. Really enjoy it. Brian: Nice. And how did you get involved with the… I don’t know if, do you guys say PA or predictive analytics? David: Yeah, yeah. Predictive analytics world. Yep. Brian: How did you get uh involved in that? David: Yeah, I’d been speaking at quite a number of conferences and the um the PA conferences were some that I was speaking at. And I’d spoken with the organizing company, Rising Media. I’d spoken on three of their conferences and at some point they contacted me and said, “You know, we’ve enjoyed having you speak with us all this time and we’re looking for a new conference chair for the London uh predictive analytics world. Are you interested?” And uh after you know so many years of going to conferences and thinking, what would I do better? What would I do better? I thought this is finally my chance to, to do everything that I’ve wanted to do to make conferences better. Uh yeah, so I said yes. So I’ve been doing it now, I’m going on my second year. Brian: Nice. And like could, even though we’re probably not going to go way deep into PA, but what did you want to make better or what have you made better? Like what was wrong and that you wanted to change? David: I was frustrated with the combination of the choice of who would be on stage. Brian: Mmhmm. David: So there’s a lot of emphasis on just let’s get you know a well-known company or someone who just has the name recognition without really focusing on quality contents. For me as a practitioner, I really wanted to make sure, look we’re going to have two days to select speakers. Let’s get speakers up there who are really going to give valuable content. Because I was frustrated in going, I’d go to so many conferences where out of you know out of 15 talks there’d be one or two that I thought were valuable. And I thought, okay, I’m going to get a conference going where almost all the talks are really high valuable talks. Brian: Mmhmm. David: So that was for me the big opportunity. Brian: Make it the conference you want to go to kind of thing. David: Exactly, yeah, exactly. Brian: Be the change you want to see in the world as they say. David: Yeah, no it was a good opportunity. It’s been a ton of work, but um yeah I’ve enjoyed it. Brian: Excellent. Excellent. Cool. So jumping in, I have a ton of questions for you. I hope we can we can fit them all in. Uh you came recommended to me um from a previous guest and friends and I was excited to talk to you. So I also notice you know in your work you practice design thinking. So I want to jump into that in a little bit. The first thing I wanted to ask you though is, um there’s so there’s a lot of technical knowledge required to do you know data science and analytics effectively. Can you talk to us a little bit about the nontechnical knowledge that’s required to make solutions that are obvious, they’re usable, useful value is created, that kind of thing. What’s missing here? There there’s a high failure rate in this industry for projects. Talk to us about these nontechnical skills that are also required. David: Yeah, I know that’s a great question and I think that’s where we’re having a lot of trouble when we look at the industry, right? The stakeholder management is really difficult because, you know as you’re a technical expert, you’re really focused on how to do these you know very technical, whether it’s an analytic model or some type of software design or software implementation. But so much of what we do in a business environment, in an enterprise environment is understanding the stakeholders, understanding what the real challenge is, and then also communicating with them throughout the process. Especially at the beginning with this design element where you’re really understanding what’s happening and your starting to produce the right solution. And you see a lot of solutions being developed very well, which were not designed to meet the actual challenge that the industry’s facing. So yeah, that’s a that’s a huge thing. Brian: You don’t have to state any names or anything, but can you give an example of a… You said something was being designed well, which I assume means technically it was being designed properly, but it wasn’t communicating its value. Its value wasn’t inherent, uh inherently obvious to the consumer. Have you had a situation where you’ve come in and it’s like, whoa. And then like what would that before and after kind of look like? David: Yeah, no, there’s quite a bit quite a bit of that. Um like for example, you’ve got, I had one situation where the customer said, “Look, we need a certain solution to advise on how to change a product. You know we have limited resources for enhancing a product in different areas. Can you advise-” Brian: A digital product? David: No. No, actually um a physical product. Brian: Okay. David: Um and so they said, “Look, we’ve got limited resources for enhancing these.” And then you know you’d start to work on this or you know third parties would start to work on this challenge and then they’d deliver it maybe two months later and say, “Here’s our recommendation of what to deliver.” And then you’d speak with the person who would actually implement that solution and they’d say, “Oh, I didn’t mention to you that you know here’s an additional limitation on which products we actually physically can improve.” And you just have that whole wasted effort because there wasn’t enough communication at inception. Brian: Throwing it over the wall kind of model. David: Yeah, yeah. Brian: Like let us know when it’s done. David: Right, right. Brian: Yeah. David: And then they end up delivering something which wasn’t practically usable. Brian: Got it. So how, how do you approach this with… You know you hear this repeated problem, which is a lot of places want to jump into using advanced techniques you know or the business stakeholder might request machine learning or request AI. And so, and then maybe there is an opportunity to actually use that tool for a good reason. But then it’s like, “Oh wait, we don’t have the data infrastructure in place to do any of that.” So like level one is like now we have to create the pipelines and all this kind of technical stuff that needs to be built out. How do you approach, because you’re talk… I know design, so this is preloaded with the fact that you know something about design thinking and and that you probably know about moving in small increments. So how do you how do you show value soon and maybe work in a small increment where progress can be seen when there is maybe a large technical requirement that needs to be there before any of the quote useful cool stuff can be delivered. How do you do that? David: Yeah, no that’s great. And that’s super important to show value soon. Um and again, I mentioned this also like in one of the chapters in my book and there’s a couple of reasons for that. You know one is because um the more often you cycle back to your stakeholder, the more closely aligned you are with their needs. But also if they haven’t communicated clearly to you and you start to give them intermediate results, that gives them a chance to refine what they want and to clarify that and to kind of keep thinking with you, right? Uh so you really want to get back to them as quickly as possible with small results, right? And there’s different ways to do that. Typically what you do is you take a small sample of the data or a small subsection of the challenge. David: Uh for example, instead of covering you know the whole world for a company, you know all the different markets. You say, “Okay, I’ll take this one small market, um you know this one European country in this one product. You know two weeks from now I’ll give you an estimate of what your solution would look like in this one limited area.” And then you do that and then you get some iterations and then, then you say, “Okay, we’ll do a pilot stage from the proof of concept.” And go to a pilot stage, where you’ll say, “Okay, I’ll give you the full solution just for this market, but it’s the full solution.” And then you iterate into kind of finally production for the full range and such. But this iteration going from proof of concept to pilot to full deployment to automation uh is a way to kind of step through that process. Brian: Yeah. This kind of reminds me of, I think it’s the thirties Jason Freed quote, like build half a product, not a half assed product. David: Yeah. Brian: And also this concept of agile, right? Which you know I saw this really well illustrated one time and it was like, it was a picture of a car from an agile perspective and like stage one of the car was like the Flintstones car, right? David: Mmhmm. Brian: And then stage two was the automobile we picture in our head. The non-agile way of being a car with no front wheels. Like it’s David: Many axles no front wheels. Brian: Wheels… But it’s all polished. It looks like a modern automobile, but it’s not, it’s not a working, there’s no value there, right? You can’t actually drive and you can’t transport yourself somewhere with that. So that’s not, you know anyhow. So that kind of reminds me of what you’re saying there. Showing showing some value on a small scale. Um you know- David: No and I think that companies are really embracing agile, especially in the last few years, right? Brian: Mmhmm. David: They’re really recognizing the value of it from a software perspective. Brian: Mmhmm. David: But it’s really challenging from the analytics perspective Brian: Mmhmm. David: For a couple of reasons. Partly because the data science and analytics, they don’t fit into the scrum model very well for a variety of reasons. Brian: Mmhmm. David: And the other thing is, uh people who really love this field, they want to do the cool work, you know they want to do deep learning, they want to do advanced models. Brian: Right. David: And when you tell them like, “Hey, just give me you know a really super simple data driven model that you know meets 60% of the solution.” Um a lot of these guys who are just focused on machine learning, that’s not what they want to do. They want to jump straight to the cool stuff. Brian: Mmhmm. David: So I find in the projects that I do at the beginning, a lot of it is just kind of talking people down and saying, “Look, let’s start simple. Let’s not boil the ocean at once. Let’s not start with the most advanced models.” For some people that takes a bit of convincing. Brian: Is that cultural at all? Is it tied to the amount of academic background they have? Like do you see a pattern that goes with that or is it just a seniority thing? I hear that tends to be more at a more junior level people that are closer to being out of school. Do you see a pattern or trend there? David: Yeah, no that’s a good question too. There’s two things. Um sadly, part of it is simply that uh a number of people I’ve talked to would say, “I only want to do interesting work.” Brian: Mmhmm. David: You know their goal is not to first and foremost bring business value, Brian: Right. David: But they want to do work that’s interesting for them. And for some people that’s simply it, and as a manager, as a leader, you have to get people in place who are aligned with your business needs. Brian: Mmhmm. David: Because not everyone with the skills is. You know? And some of them will be very forthright about it. They’ll say, “Look, I’m not interested in that work because it’s not interesting.” And the other thing is the seniority. I think the longer people are around, the more value they recognize. Two things. One is from the basic models. You know just there’s so much work that’s, so much so many projects that’ll work fairly well with just a simple regression or Bayesian model. And the other thing is just taking the time to say, “Hey, I’m you know I’m going to do the simple iteration uh and not spend three months developing something before I deliver it.” Brian: Yeah, I can understand that. So I’m curious, do you feel the solution when that kind of problem happens uh is to train and assist those technical people in learning how to broaden their skill sets so they realize you know value was not entirely driven by the technology portion? Or is it like you really just need to bring in a different role entirely, like or meet in the middle somewhere? Like how do you address that? David: I think at a leadership level you have to find the leaders who already understand that. Um and then the people who are more junior, those are the people who need to learn it. I mean, I’ll just give you a basic example. I mean in years past I worked in quantitative finance, right? So we would um we would value these financial products like interest rate swaps and you know foreign exchange swaps and such. David: And I remember valuing some of these and there would be two parts of it. One was the foreign exchange and one was the interest rate for uh cross currency swap. And the one part was super hard to do and the other part was really easy to do. And I’d get stuck on the really hard part. You know I’d be focusing all my time on that. And at some point my manager who was an MBA and not super technical you know came to me and said, “Look David, you know this, the simple part that’s 98% of the risk, you know and the complex part you’re focusing on, that’s only 2% of the risk. So you know it doesn’t matter if there’s a huge error with that part, just get the simple part right and we’ll essentially get most of the risks um quantified.” Brian: Right. David: So that for me was a real learning point. Brian: Mmhmm. David: To say, okay, understand what’s the hardest thing is not necessarily what’s the most important thing. Brian: Right. Not to mention most places are happy with a 98% test score. David: Exactly. In this area, right, right. Brian: We’ll take 75 actually. David: No, and that’s the thing, and that’s important to know you’re uh you’re stake owner too because you know if you’re working with marketing people, you know an 80% solution is fine. Um if you’re working with finance, you know these are the guys who really need exact numbers. Um so really you have to understand what you know what your target audience needs in terms of precision. Brian: Yeah. Yeah. Tell us a little bit then about, uh so I brought this up earlier, the concept of design thinking and how it can apply to you know data science and analytics work. How are you applying that in your in your consulting work? David: Yeah, no this is something really I’m focusing more and more on now because you’ll see this… Gartner has a nice visualization where they show design thinking, uh iterating into um uh the prototype phase and then iterating into the production phase. And the idea is that um you know we all sort of know, have proof of concept is important the prototype before we go into production. So we’ve all recognized that. But what we’re really missing a lot is, is that initial stage, where you’re this design stage where you’re saying, “Look, what is my real business challenge um and what’s the solution that’s really going to address that?” And taking time to get the right people in the room to get the right processes in place so that you’re starting down the right path even before you start to build a prototype. Because just from experience, I’ve seen this where, you know I was all proud of myself a few years back because I made this awesome prototype Brian: Mmhmm. David: You know of something very quickly, very powerfully. David: And I thought, “Oh, this is great. I’ve done such a great job with this.” And then by the time we went to deploy it, a few months later, they shut it down because it wasn’t meeting the need of the company. And I thought to myself, you know if I’d really taken the time um rather than just going with what the company asked me to do, because you know the CTO had said, “Let’s build this.” And so I was like sure, let’s build it. It looks cool. If I’d instead stopped at that point and said, “Look, let’s first go through the design process,” and figure out what we really should be building before jumping straight into the prototype. We would have saved a lot of time and effort. Brian: Yeah. One thing I fully agree with this, this process. Have you ever tried using, um you know for example if you’re building a predictive model for something, is prototyping something out that doesn’t even use data like and I’m particularly thinking about solutions where there’s going to be some type of visible user interface here, but actually using mock ups of the of the design in order to tease out whether or not the intended end user might use this. How are they going to react if they see a number that they don’t expect or it’s unbelievable information so that you can figure out how might we need to present this in a way that they will actually believe it if the real data actually ends up generating these kinds of results so that you can plan for that kind of contingency ahead and you don’t end up you know with the head scratch or the unbelievable reaction. Is have you worked that approach before? David: So that I haven’t done in terms of you know testing the end user’s response. Brian: Uh huh. David: There’s definitely this method, you know the smoke testing where you put a feature out that doesn’t really work just to Brian: Right. David: To see what kind of leverage it generates and such. And we’ve done other work where instead of using a model, we used user responses to gather information Brian: Mmhmm. David: In various ways. There’s several ways we’ve done that. Um but just what you’re describing now, I don’t think I can’t think of a case where I’ve done that particular application. Brian: So when you talk about needs and you know empathizing with the customer, can you talk about like the end customer versus the business sponsor? I feel sometimes that when we talk about, “The business,” sometimes people don’t understand that the business is a collection of people. Just like a government David: Mmhmm. Brian: Is a collection of real humans doing jobs and work and they have goals and needs and selfish interests and all these kinds of things. So there’s really a collection of end customers and the person that’s you know paying for the project. Brian: Do you integrate that in your work and how do you how do you think about that when you’re when we think about empathy, you know which is early upstream in the design thinking process. David: Yeah, no that’s a good question alright. Um and as a consultant, it’s always a bit tricky. I once did a project where I misunderstood who my actual budget sponsor was. Brian: Mmhmm. David: And I found that out a little bit too late. So, you know I spent all my time kind of tailoring to the needs of one person when you know and then several weeks or months later another person informed me like I’m actually you know your budget sponsor here. Um yeah. Brian: And did they have different needs? It was like they had different needs. David: Well but also it’s… Yes, slightly different needs, but also it’s very important that you maintain close communication lines. Brian: Sure. David: Right? Um so I find for myself, typically when I’m on a project I’ll have uh I’ll have projects who have different stakeholders than necessarily the budget holder. Uh and that’s typically, you know I’ll work with a budget holder to clarify expectations. Brian: Mmhmm. David: You know where should my priorities be. Um but the stakeholder for a specific project is typically the person who I’m aiming to make sure I satisfy you know their requirements and such. Because what we normally establish is that look, if this stakeholder is happy with the results then the budget holder will also be happy. David: Uh the difficulty happens when there’s a conflict of interest there. Brian: Right. David: You know and one person is clear about wanting one thing and the other person has a different view on it. That doesn’t happen so often, but from time to time it does. Brian: I’ve been in that situation before. And you know engineering’s paying, you’re paying for your nut and you’re working with product management or you know some other department and because of those lines. But I think it’s always you know important whether you’re a consultant or you’re internal to really understand kind of who’s who’s going to be evaluating the value creation here and understanding what they’re looking for is definitely important. David: Absolutely. Brian: Or else your chance of failure is is high. David: Yeah absolutely. Brian: And it shouldn’t be silent, be be uh suspicious of silence. And when there’s not a lot of communication that’s… It’s always a risk for me. I just literally checked in with a client today because I’m not working with you know the stakeholder the main stakeholder, I’m working with her team and everything was fine. But it’s you know when you’re not hearing questions and stuff that’s, it’s you got to keep those lines of communication open and making sure they’re seeing the work you’re doing and evaluating and giving feedback on it. Uh this throw over the wall thing is a very high risk model. David: No, it’s totally true. And I do, one of the things I do is I train uh junior data scientists. I do that on a regular basis. And one of the things I tell all the classes is, “You know make sure you put in a recurring appointment with um you know the project owner, Brian: Mmhmm. David: The sponsor um and if it gets canceled, you know reschedule it,” Brian: Yeah. David: Because you can’t you can’t afford to lose contact like regular contact with a sponsor. Brian: Mmhmm, Mmhmm. We talked a little bit about this when we originally got on the phone about you know what’s going on in the industry with selling products and platforms versus you know focusing on outcomes and results. Uh the industry helps us build things. Um things don’t always turn into value. So do you have some advice on how to approach this? And I’m not saying that all products are bad and platforms. I mean without some of these this new technology, none of this would be possible. At the same time, you know in the conference halls and you know a lot of people that I’ve talked to feel like, you know basically that the large platform providers are making out like bandits and yet every year at Gartner is like, “Oh, 86% of projects will fail this year.” And it’s like, “Hmm, someone’s making out really well here. What is going on here?” Like- David: Yeah. I think part of the issue is that you know it’s this thing where you can delegate um authority but not responsibility. You know and people, it’s really tempting just to say, “Okay, I’m going to pass off this challenge to a product. You know um I don’t really understand how to address this product or to address this challenge. So I’m going to buy a product which says it’s going to meet the needs.” And then later… And you don’t even know how to evaluate it because you didn’t know how to do the problem in the first place. Brian: Mmhmm. David: So I think that’s definitely dangerous. Um and you see different tools. I don’t want to name any tools, but there’s definitely tools out there that sell sort of the AI solutions. David: Um and when you back test them, you’re like, well, what is this really providing me? You know it’s perhaps it’s a terrible model. Brian: Mmhmm. David: Um and maybe the vendor even knows it’s not a good model, but it’s their model, they’re selling it and we’re buying it. Um that being said, I mean if you understand the challenge and you know what you’re buying, then it is a way to move very quickly. Brian: Right. David: You know, it’s a way there’s certain tools which will let you automate repetitive work um and that’s you know that’s a big payoff, right? If you know what you’re getting. But there’s definitely a lot of snake oil being sold now in the market and I see that come up also. Brian: So let’s jump over to um skill gaps. You were talking a little bit about some of the training that you offer so what if you’re offering training, it generally implies that there’s something missing there. What’s missing that needs to be filled in? David: Yeah. There’s a couple of things. One is that there’s a tremendous number of people, at least over in Europe who um haven’t been really trained solidly in analytic skills, Brian: Mmhmm. David: But they sort of go through bootcamp and then crossover into this analytics slash data science space, right? Brian: Mmhmm. David: So these guys are missing a lot of fundamental skills in terms of mathematics, statistics and computer programming, right? Brian: Mmhmm. David: So that’s the basics. I don’t do a lot of training in that myself Brian: Uh huh. David: Just because there’s so many people who can do that. Uh what I focus on more is the business skills of you know once these guys are coming out of a highly technical program or bootcamp um or straight out of university, how can they place themselves within an organization and be effective? Brian: Mmhmm. David: And there’s several aspects of that. One of which is understanding that the larger enterprise around them, understanding how different people think, how the nontechnical people operate, what’s important to them. So that that sort of empathy and understanding your place in the company. Brian: Mmhmm. David: And then the other thing is communicating. You know and these guys, they tend to have a lot of challenges communicating around them, both in terms of orally but also visually. Brian: Mmhmm. David: And part of that is because they’re used to functioning within an analytic ecosphere. You know, communicating with other people like them Brian: Mmhmm. David: You know in their programs and such. Um and a part of my training is really to help them understand, look, when you’re doing emails, when you’re doing presentations, PowerPoints, or charts and graphs to people in outside departments, um what is a way to communicate very effectively? Brian: Are there any particular like repeated things that come to mind in terms of guidance that you give? You know the five bullets or something that you kind of see as a repeating theme around the communication, particularly the visual, but also the you know written? David: Yeah, no there’s just a lot of different stuff I cover uh to be honest. Brian: Mmhmm. David: I mean a lot of it just boils down to understanding the perspective of the people around you. Brian: Mmhmm. David: But that encompasses a tremendous amount of of different training materials that I have. Brian: Right, right. When you have team members that are coming from that background, how do you and the solution that you’re working on has a software component, particularly you know some type of visual component, what is their role and how do you get them up to speed or how do you work with them if they’re going to be involved in that solution that’s going to go out? Is it just take a guess and try? Like if you’re starting out from a pure data science or stats math background, how do you get to that point where you can deliver a solution that may require a software interface? What’s involved in that process? David: Yeah, so it depends on sort of how it’s being deployed. Um if you’re looking at something that’s going to be deployed as part of a production stack, Brian: Mmhmm. David: That’s a long road to travel if you don’t have the software backgrounds, right? Brian: Mmhmm. David: Because then you have to know all this stuff about you know the testing, the unit testing, um Brian: Right. David: Regression testing, um sort of everything that a developer knows in order to create robust code. Brian: Mmhmm. David: Um even then further monitoring and such. Uh and if you come out of a math program or something and you don’t have that software development experience, um that’s quite a bit of training. And that’s actually a big stumbling block Brian: Mmhmm. David: For these guys because they know the machine learning techniques and they want to deploy something. Brian: Mmhmm. David: uh and they’ll start hacking stuff together with whatever code and then they’ll go to the development team or the IT and they’ll say, “Hey, I built this cool model. Can you deploy it?” David: Um and there’s no way that’s going to get deployed because it doesn’t meet the rigorous standards that are necessary. Brian: Right. David: So on the flip side though, when you have the software developers who already know you know how to make things um robust and such, and these guys say, “Hey, you know I want to try building a regression, building a vision network or something,” um there’s almost a better chance for them to be able to build something that’s deployable Brian: Mmhmm. David: Because they at least had that foundation. Brian: Mmhmm. The um, how does this tie into… Uh so so another kind of topic that’s just been in the ether I guess around me right now is this topic of model trust. Um if you’re coming at it from that technical standpoint uh and if you think or believe that uh model trust is an issue. So this is do my stakeholders believe what I’m showing them. How does that play into your process? Um how do you how do you get to the point where you don’t wait until you have a great model David: Mmhmm. Brian: But then you find out someone won’t use it? Um David: No, that’s a great question. That’s a great question. Um in the past uh I worked with a forecasting project at a client and I had a third party working on something and they went to you know they developed some cool model and they went to the stakeholders and they said, “Look, here’s our forecasting model.” And the stakeholder looked at it and said, “Yeah, okay, that’s kind of interesting. I’m never going to use it.” Right? And then later on when I was supervising a forecasting uh project myself, what I did is I you know I told my team, I said, “Look, the first delivery we give to the stakeholders, make sure it’s something super basic and super obvious you know so that they know exactly what we did, why we did it, and they look at it and say, yeah, that makes complete sense. That’s what I would have done.” David: You know give them just a basic regression or basic Arma one one model or something, not no bells or whistles. And from there, the next iteration make it a little more complex Brian: Mmhmm. David: And then a little more complex because you have to get them on board and they have to be nodding their heads, um either asking like one question at each iteration, like, “That doesn’t make sense.” Or nodding their heads and saying, “Yeah, that makes, makes complete sense.” But if you jump into, if you go all the way to an advanced model, like you said, and try and throw that at them, if it doesn’t make sense and it doesn’t agree with what they already saw, then you’ve completely lost them. Brian: Yeah, yeah, yeah. Get in touch early. David: Yeah. Brian: Stay in touch. You know that’s kind of the kicker there. Do you think this overall trend of success or lack there of, in the industry of, you know, formerly it was big data projects, now it’s AI. Is this trending in the right direction? Like the success rate? I mean it’s been consistently low for a long time. Where’s it going? David: Yeah, I mean there’s two things to that. One is that there’s always a natural sort of failure rate even with well intentioned and well designed projects. Brian: Uh huh. David: But then the second thing is we’re having this difficulty, having a difficulty because we used to be a bottom up push for data science projects where you know, the developers would say, “Hey, I’d love to do this, I’d love to do this.” And the management said, “Stop bothering us and just do your job.” Uh recently what’s happened is it’s been a top down, right? Brian: Mmhmm. David: So from board level they’d say, “Hey, we need to start doing something with AI, you know um with machine learning, with big data.” And they’ll start throwing budgets at it. But what happens then is that you’ve got a you know couple hundred thousand or a couple of million from board level coming down to start hiring a team and buying technology. David: So, these budgets that are made up you know at an annual level, after one or two years of you know building a team of 20 people or 30 people, then the budget’s being renewed and people are going to start asking questions, “What value did we get?” Well, even at the inception, Brian: Mmhmm. David: A lot of these teams, there wasn’t a clear mandate for them. There was a top down hope that we could leverage and buzzword, right? Brian: Right. David: But two, three years down the line, we spent the millions and we’ve bought the systems and everything. We weren’t really sure from the start why. Um so, of course there’s not tangible value after two or three years and there’s still no tangible direction. And then you start to see these things burnout. You know and you’ve seen, I’ve seen some programs already where after a year or two it’s sort of like where are we going with this? What’s happening? But of course those that really understand what they’re doing and why they’re doing it, of course for these there’s a very real chance of seeing value and you’re seeing a lot of the large companies really capitalizing on these efforts when they initiated the efforts with a clear vision and a clear purpose. Brian: Following onto that then, do you do you think there’s a place for, sometimes, I call it laboratory mode, right? Which is maybe the business stakeholder understands that you know we’re also a data company. You know everybody says that now. I know we need to be doing something with this, but I don’t understand the technology. I want to know that we’re flexing this muscle, we’re rehearsing, we’re playing scrimmage games. David: Mmhmm. Brian: Even if we don’t create value. Is there something there to be to having a laboratory kind of model, which is maybe you do let some of your top talent go try to build a you know a deep learning network or something like that, back out a project from something, not exclusively in their work, but this is like the only 20% time kind of concept. Is there some value in that or do you think it’s unnecessary to be totally playground uh with no no dirt? David: No, no, absolutely right, absolutely. Brian: Mmhmm. David: But this goes back to your design thinking, right? So you want to have that initial impetus. Brian: Mmhmm. David: Like why are we doing this? You know where might it go? Brian: Okay. Cool. David: Yeah. Brian: Awesome. This has been like super good talk. Do you have any like closing advice for for listeners? You know we’ve got an analytics data science, technical product managers, some designers. Like I know that’s a wide group of you know listeners, but if you had some closing advice for them. David: Yeah, I would say just as much as possible to, you know for people who are in the field and looking to bring value in the companies, as much as possible talk to people around you, you know talk to people outside of your team. Brian: Mmhmm. David: Outside of your department. Um understand what they’re doing, why they’re doing why they’re doing it, and how you can help or how the tools that you have could potentially help. Because there’s a lot of opportunities you know within companies. It’s just, there’s not that communication between departments. So yeah, I would just really encourage people to keep that communication open. Brian: Nice, nice. And by the way, how’s the mandolin going? David: The mandolin, I’m not getting a whole lot of time with it, but I was thinking about it today. So uh more focused on the guitar these days. Brian: Oh okay. David: But um yeah, yeah, it’s a really a real source of potential that I’m not tapping into. Brian: Excellent. Well, keep practicing. David: Yeah. Brian: Tell us, obviously I’m going to put a link to your book in the show notes, but where do you publish uh social media, LinkedIn, any of that stuff? Where can people find you? David: Yeah. I used to be a little bit better at publishing blogs on LinkedIn and on my website. Brian: Okay. David: To be honest, I’m really behind on that now. I’ve been so busy with other things, uh but hopefully at some point I’ll start uh blogging some more. Brian: Nice. Excellent. And you’re, if I recall, it’s DSI analytics, right? David: That’s it. Yeah. DSI analytics. Brian: Awesome. So I’ll, yeah, I’ll put a link to that and your LinkedIn profile as well. And thank you so much for coming on the show. This has been a really good conversation. David: Yeah, no, thanks for inviting me. I’m glad to be here and look forward to uh seeing you in London soon. Brian: Awesome. Yeah, I’m looking forward to speaking there. So take care.

Ep#24 Transitioning from Owning 600 units on his own to Syndication with Brian Murray

Play Episode Listen Later Oct 15, 2019 49:12

James: Hey, audience and listeners, this is James Kandasamy from Achieve Wealth Podcast where we focus a lot on value-add, commercial real estate investing and we usually talk to commercial real estate operators who have been very active buying deals nowadays. Today, I have Brian Murray. So if you have not heard about Brian Murray, he's the author of the best-selling and award-winning book: Crushing It in Apartments and Commercial Real Estate. And he owns almost 700 units right now on his own and I think out of 700, 600 of it is apartments and 100 units are on office sites. Hey, Brian, welcome to the show. Brian: I'm really happy to be here, James. Thanks for having me. James: Really happy to have you here. And so tell me about, how did you go from 0 to 600 multifamily 0 to 700 asset classes on your own without syndication? Brian: Yeah, well, you know, I started 12 years ago and I'm located in Upstate New York. That's quite a bit different market than New York City. But my first property was an office building and it was a distressed office building and from that very first deal, I did a lot of value-adds. Frankly, I really didn't know what I was doing, I was kind of figuring stuff out as I went along but I progressively made that property perform better over a couple of years and added a ton of value. On that deal, I assumed the mortgage and on my second deal, I did an owner/finance situation. It was another property that was half full, I filled it up and refinanced out of both of those and bought three more properties and followed that path the entire way. Which is find well-located properties that were not well managed or had some other large value-add component, exercise that value add and then refinance, take cash out and buy more properties. And that's the exact path that I followed to get to where I'm at today. James: That's crazy, which is good. I mean, that's the model that, I mean, it's an absolute value-add model, which is basically the theme of this podcast. And so did you buy and then improve it and then refinance the money out or did you sell it and I didn't get that far, can you clarify that? Brian: Yeah. So I refinance the money out. I am primarily buying hold, still to this day. But especially in the first 10 years, I think I sold one or two properties, smaller properties, for the most part, during that time. I am selling some of my smaller properties right now to redeploy those funds into larger properties, but my strategy has really been buying hold. James: Awesome. Awesome. So before we go further, I want to clarify about your book, Crushing It. I mean, I remember asking this question to you when we met face-to-face. So did Gary take the 'Crushing' name from you or you took it from him? Which one is that? Brian: You know, so his book, Crushing It, came out about a year after mine but he launched a book called Crush It prior to when mine came out. But he took the Crushing It and you know, but that's fine. It doesn't matter. It's all good. James: Well, it must be a good name because both of you are like a best seller, you know, in your own domain. So awesome. So right now what's your plan? I mean you own this many units on your own and what's your plan right now? Brian: So right now, I'm really focused on diversifying. I was really excited to do my first Mastermind, which was last year, which is how you and I met and I met some great people at that Mastermind and highly recommend that to other people; surround yourself with other folks that are doing what you're doing. But when I went off to this Mastermind, it was really eye-opening for me because pretty much everybody there was doing syndication and it was a model that was really new for me and I just learned a ton about what people were doing. And my model has worked great for me up to this point, but I've reached a size, we're growing purely organically. It's becoming more challenging to maintain that pace of growth. I think also with valuations at a higher point, it's more and more challenging each year to pull that much value-add out with refis. I think another factor that's come into play is I've been very, very dedicated to putting every dollar that I've earned back into my real estate. That's been a been a big part of how I've done what I've done is to continuously reinvest back in. As a result of that, to this point, I've been living fairly frugally and you know at a certain point, you want to not have to put every dollar back in but you know, to maintain that growth rate, I've got to look at other options. I also want to diversify geographically because most of my properties are in one location. And so I'm in the middle of my first syndication right now and I've met so many good people that now, I'm developing partners and looking at new markets and it's very exciting for me. I love to learn, I love to try new things and getting into these other markets and, you know, meeting accomplished people like yourself, it's very motivating. So I'm just super excited about it. James: Yeah, it's eye-opening when you go and talk to different people who are doing the same level as you are doing much more higher level because you can see a lot of different thought processes and how people do things. So why are you moving towards syndication? I mean, you own like so many units on your own, can you go into a bit more detail on why do you think syndication is going to be beneficial for you right now in this market cycle as well or on your investment side? Brian: Well, you know syndication, it does open up a lot more opportunities in terms of size. So for example, right now, I'm looking very closely at an apartment complex that's approximately 300 units. It's in a market that's new for me that I've been doing a lot of research on and that would be a real challenge to try to pull off on my own. It really wouldn't be possible right now. So the property that I've purchased strictly on my own, without raising any outside money, I did last year, it was 126 units and you know to try to purchase something that's 300 plus units that wouldn't be possible for me right now. So it's pretty exciting and I think another thing is I really enjoy working with the idea of doing some projects with partners and getting into some of these new markets. So, there's another piece of it that's kind of exciting is, I've reached a point where I've done pretty well for myself and the idea of helping other investors who want to put their money to work to achieve their goals, I think that's going to be rewarding too. That if a project does really well that, it's all those limited partners that come in that can then improve their lives through their investment as well. And if I can be a part of that, I think I'll find that very rewarding. James: Okay, that's awesome. So scalability is important and you think of helping others as well to make money, especially I think other investors or other GPs who needs your skills, I would say? Brian: Yeah, absolutely. Yeah, and that's one of the things that's great too is I've found that it's meeting these other people that are doing it, I've got a different experience. So just like I'm learning from people like you, I'm finding that partners I can bring some different perspectives and value to the table as well. So you always want to partner with people that have strengths in areas that are different from you and that's what makes a strong team. James: Absolutely, especially in commercial real estate because the number of knobs that you can tune, there are so many knobs and especially like in multifamily because it's very management intensive compared to the triple net, other commercial properties. Multi-family is very management intensive and it gives a lot of ways to make more money or to scale down or to scale up. Even though you'd be really, really skilled at that but it just gives you a lot more opportunity. And the lease is one year term or six months term; you can quickly raise or reduce rents, it gives you a lot more fungibility, I would say. I mean, you have like SAS, we talked, in the beginning. You have like 600 units multifamily and 100 office space? Brian: Yes. James: So can we go a bit more detail into the office? What kind of office is it and how did you strategically balance within the 600 and 100 office? Is it optimistic or what did you see and why did you do it? Brian: So I started off with the office and actually, my second property was retail and so, starting on that commercial side was really interesting. I think one of the things that did for me is really emphasized my focus on customer service and customer care with tenants. And when I tried my first multifamily, I think that there were differences but they're also a lot of similarities. So the value-added approach that I was taking to office retail worked just as well with multifamily. And our focus on really taking care of our tenants as our customers really served us really well in that area also. Over time, as recently as two or three years ago, we had reached a point where up to that point we had more office and Retail and then about two years ago, I would say, we were 50/50 and now we're closer to two thirds, maybe even 70% multifamily with the rest commercial in terms of the makeup of our portfolio. So as time went by, we've really gravitated toward multifamily and that's our 100% focus right now. I think the biggest thing is that there's a number of things we like about multi-family. From our experience with commercial, you've always got a little bit more risk because you tend to have, not always, but you often will have tenants that comprise a disproportionately large percentage of your income and that can leave you really vulnerable if somebody leaves. So, on more than one occasion, we've had a commercial property where someone that takes up more than half of the space in that property, leaves unexpectedly. And then you've got with one tenant leaving, you have a property that is negative cash flow. And if you don't have a portfolio in place to support that, that can be devastating and it's really not fun even if you have a portfolio to perform it. And then when you go to backfill that space, it's more challenging in commercial properties because you oftentimes have to find the exact right tenant for that space, for that location, for the tenant mix and the property, for the configuration of the floor plan. There's a lot of things that you know, different commercial tenants are looking for. If you just adjust the rents up and down or maybe offer some concessions, a lot of times, the market doesn't immediately react to that. So turning that dial like you do in multifamily, you have less control. So if you're looking for a particular type of commercial tenant, it could be, it's not unusual for us to sit on a vacant space for one two or more years before the right tenant comes along and fits in and takes that space. With multifamily, you've got those dials that you can turn and say, Hey, you know, we're going to run a special. We're going to bump rents, we're going to drop rents and you usually will see a pretty quick reaction from the market to the changes that you make and from my perspective, that's better. You always want to have more control and the ability to adjust with your market, adjust to combat your competition and different things like that. And frankly, we've enjoyed working with the tenants. I think there's a perception out there that a lot of people would love to invest in commercial because they think they have this idea that working with white collar tenants would be much better, wouldn't have the problems but in our experience, they can be more challenging. They can be more demanding and sometimes even unreasonable with what they're looking for and you don't usually find that as much with the residential tenants in multifamily. We do primarily workforce housing and the people that we deal with there, tend to be good down to earth people and reasonable. So we appreciate that. James: And when you talk about office, this is the normal office tenants, I guess? Brian: Yeah full-spectrum, mostly professional tenants. We've got plenty of medical tenants. We have lawyers, accountants, all types, we've got not-for-profit offices, engineers and architects that would pretty much any type of white-collar professionals. James: Got it. That's very interesting. So when was the aha moment that, hey, I should do multifamily because you are focusing a lot on office, what was that triggering moment where you say, okay, I may need to look at this multi-family? Brian: Well, I don't know if there was a specific moment. I think it happened gradually over time. When we had about 50/50 multifamily and Commercial, I think one of the big things was watching the performance of the two halves of the portfolio and seeing which half was performing better and part of it had to do with the types of value-add projects we were finding and I thought we were better able to execute on the value-adds on the multifamily side. And that portion of our portfolio just kept outperforming the commercial side and I just saw in the market that we're in, more opportunity there and I felt like it was more stable income based. So, I think I think it just happened gradually over time and you kind of tend to slowly move in the direction that's performing well and where the needs are in your Marketplace. James: Got it. So all the deals that you have done on multifamily, how did you choose? I mean all these deals are in Upstate, New York, is that right? Brian: Yes. James: So you may not choose the city because that's where you live, the area. But how did you select the submarket? Okay, this deal is good in this submarket, what are the parameters that you looked at When you look at a deal in multi-family? Brian: So, we have a really close familiarity with the subtleties of the market and so it's fairly nuanced like there's not one overarching thing. One of the primary drivers of the market where we are is not that far away is a fairly large military base. And so one of the factors that we look at is, well, we definitely welcome military tenants, we have shied away from the properties that are closer to the military base and tend to have a really high percentage of military population. That's just because there's so much turnover, lenders are less excited about lending those properties because they know that long-term, there could be downsizing. A base could close, there's exposure with that. So we have gravitated within our region to the areas that are maybe we will have some military but not be all military and into the communities where people want to live, in the parts of the city that we feel are strong and good safe locations and convenient locations for the major employers in the area. James: Got it. Got it. And on average right now, what is the price per door in that market? Because I never talk to anybody from New York who's buying multifamily. I mean, Upstate, New York, New York City, but in general, can you give us some guideline on price per door? What cap rated stabilize deals are being bought right now? Brian: Yes, absolutely. So it's a really, really wide range. So that's what I would say at first. The most recent stabilized property that we purchased we paid about 60,000 a door. There are properties selling in the area, 80,000 plus per door, not that often but a lot of the properties we've got, we've purchased a couple of decent sized properties at auction. We've purchased a lot of distressed properties. The 126 units that we purchased last year, we paid in the 40s per door and that's pretty low for this area actually, but also the occupancy was below 60% when we bought it and it had a lot of deferred maintenance. So I do feel like we got a fair deal and a good deal on that because there was so much upside but there was a reason that it was priced that low. And so you can come along properties in this area that have low price point sometimes even down into the 30s per door, but usually, there's a reason why they might be in severe distress. But for stabilized properties, I think you're mostly looking at maybe 50 to 70 a door. James: Okay. You also mentioned that you're looking at other markets now? Brian: Yes. James: And why is that and what're your criteria to look for in other markets? Brian: So the number one reason is really a risk management type of approach. Where anybody who's come in and taken a close look at our business and one point even a few years back, I had some graduate students come in and they analyzed it and everybody said, hey, you're kind of crazy. You've got all your properties concentrated right here in this one city and now they're all within maybe half an hour drive of that City and there's a lot of risks involved to that. So if that City that I focused on starts to decline or say that military base that's not that far away, if they downsize then that all affects my portfolio. So I've known for a long time that it would be wise to diversify geographically and it's time to do that. Another factor is frankly, this is not a huge City. It's not a big area that I'm in and we've got limited opportunities for growth here. There's a limited number of properties that come onto the market and realistically, it's time for us to look to other places. So it's a variety of things. James: So let's say you're looking at a new city, a city A and a city B, what do you look for in that city that you think is going to be appealing to you? Brian: Well, I think there's a variety of different factors. Probably the number one thing that makes the city appealing is job growth, job creation. Being located in Upstate New York, it's not a strong area for job growth. There are pros and cons to being in a market that's undesirable. So I have less competition. I can buy things at much higher cap rates and I can get properties to cash flow better if I have less competition and higher cap rates. So, there's sometimes you can look at it and say, hey, if you're in a market that's less desirable, sometimes you're getting properties at a great deal and there's something to be said for that. But as I look to new markets, I'm trying to find something where cap rates haven't dropped too far and you can get a reasonable return but you've got that benefit of healthy growth in population and jobs. But I think because I'm looking for more geographic to looking for a market that's going to show more stability, it's on an uptrend and just like any other place, no matter what market I'm looking at, I've realized over time just how critical the specific location with any city is. So almost any City has their good parts and the bad parts and so you could take any market that you choose and break it down into all different, more and less appealing locations. And so, I wouldn't just throw and say, hey, this one city is great, even though the population is growing and you and I talked about a property not that long ago that you are familiar with the location and you very wisely were like, oh, that's not the right deal. It might be a good city, but that's not the right part of the city. James: Correct. So, I mean, you are sitting in Upstate New York, you looked at the entire nation. Can you give us the top three cities that you think that you want to delve in? James: Brian, so you are sitting in Upstate New York, and you looked at the entire nation, you know how multifamily works because you own 600 on your own. So you just briefly outline what are the things that you look for in a city. So can you name like top three cities that you think that you want to be involved in that you think has a strong growth story? Brian: Well, it's a work in progress for sure. And what I would say is sort of the candidates that I've narrowed it down to the commonality would be they tend to be the places that people are migrating to and being in Upstate New York where a lot of people are leaving the area, I want to look toward the places they're going. And so, primarily in the Southeast, pretty much our candidates or everything from starting in probably North Carolina going down to Florida and you know all the way over to maybe the little bit in Texas, but I think Georgia is an interesting market that a lot of people are pursuing. I'm partnering on a project in Kentucky right now and we're looking at North Carolina and there are some very attractive markets in Florida as well. James: Got it. Got it. Got it. Before I want to go into the deal level analysis that you do, I want to quickly ask this question because you know, it's very unique to you because you had your own deals and now you're going into syndication, right? So what do you think are the skills needed from yourself when you are having your own deals, where you can skip a distribution or whatever happened to the deal is your own problem. So now you're going into syndication, where it involves a lot more people. What do you think is a few skills that syndicators need to be successful in syndication? Brian: Sure. I mean I would say start a start with one of the big ones which is something that I don't have, which is an investor base and that's a whole job unto itself. Over the years doing what I've been doing and getting some acknowledgments for that, I had a lot of people approach me over the years and say, hey, you know, can I invest and I never took them up on that and now I'm doing that. But what I've realized is in getting to know all these folks that are out there that there's a lot of people who are interested in partnering with me who already have those investor bases and have that skill set of managing those investors and taking care of all aspects of that. So at this point, I'm primarily thinking that I bring more value in the weighing on the underwriting and the property and identifying all the value-add opportunities and making sure that people look at it as more than a spreadsheet because there's so much more. I toured a property last week and was able to uncover quite a few things. The broker that was there. I was one of the last people, they had about 40 tours and I came through and identified some significant value-add opportunities that the broker said no one else picked up on. And I think that that's something I didn't discuss but we've managed all of our own properties that whole time and so, the knowledge that you get from that just brings so much better of analysis to a deal to make sure you're vetting it properly, you're not overpaying, you're also not underpaying and that there might be value there that you're not realizing. That some of the assumptions that you're making for rent growth are real and can actually be feasible for implementation. And so, you know, those are some of the things that I bring and the experience and having the portfolio I have may give lenders a lot of comfort. And so, I'm recognizing that, hey, I could focus on my strengths and bring some things to a partnership and take those areas that I don't have and other people might and partner up. So if someone's going to do it on their own, they've got to have a pretty broad skill set and that's a challenge, to have the operational knowledge and bring that side and also have the people skills and the investor relationships, it's not easy. I have a lot of respect for people that are doing it all. James: Absolutely. So you are two operators, where you underwrite deals, you understand the operation and you're doing your own asset management. You're missing the investor base creation side of it, which I think you are either partnering or slowly building that up so which is awesome. For me, the operators are at the top of the food chain because they are the backbone of the whole deal. They know what's happening in terms of the rents, how many percents of rent increase is happening on each unit? How many units are being turned? What is the make ready period, what's the delinquency? What is the idling unit period? That's a lot of parameters in the multi-family operation which can be optimized and if you know that very well, your underwriting can be very, very solid, I would say. Brian: And I think you also bring a reality check. I think that the folks that are operating in the syndication space that don't have as much operating experience, it's easy to look at numbers and assumptions in a spreadsheet and it's challenging to actually recognize what that means in terms of the actual human beings who are there living in the apartments, what it means for the contractors and the property managers and whether what you're assuming is even practical. I look at a spreadsheet and I'm looking at it realizing, hey, you know, I looked at it once a day and I told somebody I'm like, do you understand how much drama will be involved in this? So if you haven't done that you don't know. And sometimes that translates into you might need to maybe tone back your rent growth or you might need to say, hey, maybe we implement something like this over time so that we don't have an all-out rebellion on our hands. So, you know, it's a challenge to bring all those things to the table. James: Yeah, I've seen people who come to me, you know, first few deals and say, oh, this is all bills paid, I'm just going to change it to tenant pay bills. I say, well, that's easy. We can see the value. Well, you do not know how much drama you're going to have there and you might not able to do that on a specific property, a specific location. And they say they want to do them; Utility Bill back, they want to increase the rent, they want to charge covered parking, they want to do laundry increase. So many things they want to do at the same time and I can tell you, they don't have the experience actually. But the thing is, a lot of people have been making money even without all the skills. And I always tell them everybody's a champion in a bull market. Brian: Exactly, yes. A rising tide lifts all ships, right? James: Correct. So, people may not look at that skill more in detail or give due consideration to that type of skills where the operation is important, but I think it's important if you want to sustain good rent growth across different market cycles. So coming back to underwriting. So right now you are looking at deals, how many percents of deals do you reject immediately by just looking at it? Brian: Wow, I would say well over 90%. James: Okay. So the 10% that you have or what do you look for in that 10%? What do you do? What are the steps that you take to look at that 10%? Brian: You know, I think the very, very first thing I do is I look at the T12. I want to start my analysis of a property by looking at actuals. And then I'm going to base the current situation and the actuals, going to kind of weigh that against my own experience. So, how does the target asking price or the whisper price or whatever they have, how does that compare to the actuals? And then based on my experience looking through those actuals, what do I see that jumps out at me that might create value? And if you look down through and start looking at the comps and really piecing together this puzzle about, what opportunity is really here? Is the valuation based on something that's completely unrealistic? A lot of times, you'll recognize that some brokers are way better than others at doing a realistic model and pro forma and that's much appreciated. Because you see too many where they'll say, oh, you know, the labor is going to be whatever, $300 a door, and you know, hey, that's crazy. Like it should be 1100 a door or 1000 a door in that market and you know, you'll find out that well, it's been managed by the owner and they don't track the labor. But if you see that it's based on the labor is $2000 a door and you know, hey, we could get that to 900 realistically and still do a good job of maintaining that property, then you start to see an opportunity. It's a combination of running numbers and logical analysis based on experience, is really what I would say it boils down to. James: So in a new market, how would you determine payroll and [12:09unintelligible] on property taxes because this differs by market? Brian: Sure. So all those things are going to vary by market, although many of them will fall within a range. So you're going to say, well, in that market it's going to tend to be higher or lower and I will use my best judgment but if it passes a certain level of scrutiny, that's when you want to really get an established reputable local property manager involved who could look at it and say, okay, for this market specifically, these assumptions you've made are realistic or not realistic. The same thing goes with construction costs they could vary and I can look at it and say, I think that new flooring should be this much but hey, maybe in that market, flooring is much more expensive or maybe it's a lot cheaper. So, you know it's going to be within a certain range, but you just need to figure out how you need to tweak it to get to that market. James: Got it Got it. Got it. I mean since you have your own property management in your own backyard and now I presume you looking at third partying your property management in this new market, is that correct? Brian: That's correct. James: So, what would you think is the most important factor to look at that third party property management company? Brian: Well, at this point, I would say yes, we're relying on third-party property managers. We may eventually consider expanding into new markets or operations, but not doing that right now and evaluating the property managers, it's been a very interesting process. I think you need to look at the full picture. I don't think there's any one thing you can look at. For a project that we're underwriting right now, in evaluating the various property managers, of course, we weigh referrals, you know, that's always good to hear referrals but I think one of the things that are appealing about the property manager that we ended up selecting for this project that we're pursuing is they actually specialize in this specific type of property that we're looking at. So, they have a track record and experience of nearly 10,000 units that are specifically C-Class properties that they've done value-add and executed those successfully. And a fair percentage of those are in the specific market that we're looking at and so there's a lot of things that just lined up. I think if I had to pick the one thing from my interaction with this firm because they toured the property with me as well, but I actually was very impressed with their analysis of our underwriting. They actually went through our assumptions and they toured the property on their own before I got there and gave us their own analysis and without us asking, they also toured the comps and gave us some feedback on that. I was impressed. You could tell that they went out of their way to look at the right things. They looked at the types of things that I would look at and they identified things and based on that write-up, I just said, hey, this is a firm that's experienced. They get it. They did a thorough job. They were professional, they were responsive and you know, it really checked a lot of boxes in terms of giving us an overall sense of comfort with the possibility of working with them. James: Awesome. Awesome. Let's go to a bit more on the value-add side because you have done a lot of value-adds because you buy refi and keep it more long-term. So what is the most valuable value-add multifamily from your experience? Brian: I would say that the most valuable is it's different for almost every property. If I had to pick, you know, I think that sort of the Big Bang low-hanging fruit tends to be the, I'd say, clean paint landscape, kind of like the surface stuff. If a property is dirty and not well kept and then you make it clean and you put a fresh coat of paint and you landscape it, it can change the entire image of property of fairly modest cost and that can have a huge impact. The rent adjustment is sort of obvious, I think everybody looks at that. I guess big picture if the landlord is way undercharging, of course, you know, that's an obvious big easy one, but one thing that we've ended up doing in a number of cases that is less obvious that people almost never talk about is lowering rents. And in the 126 unit that I mentioned earlier, that's under distress, that's the first thing that we did is we went in and by our assessment, they were trying to charge too much which was a major factor in why the occupancy was so low. So we immediately went in and cut all the rents and that might seem counterintuitive for a value-add person but over the last six months, we've raised the occupancy 25% and one of the big reasons is we lower the rents and so the net change in terms of the net operating income of that property it skyrocketed by lowering rents. So that also further demonstrates that it really varies, you kind of have to you know. It's sort of like if you look at five different people and say, you know, what change would you make in each person to improve their overall wellness? For some people, they might say stop smoking and some people might say, well, that one needs to eat better so you can't kind of really say well, what's the one thing overall? James: How did you decide to lower the rent? What was the data that you looked at and decide, okay, I just need to reduce the rent here? Brian: Well, you know, that's one of the fantastic things when you've got so many properties in one market. You know immediately that based on your other operations that something's off. You know when it's low, you know when it's high, you know when the fees don't match what's present in that market or the concessions don't match. It becomes very simple. If you're going into a new market, you've got to study those comps and do the best you can and hopefully, tour those comps and do your own homework. But it's one of many advantages of having a concentration of properties in one area. In addition to all the many operational efficiencies that you can have is that you have that market specific knowledge that is there's no substitute for. James: Got it. Got it. So when you decide to lower the rent, I mean it is a counter-intuitive but I think it makes sense in value-add, especially when you go with that kind of low occupancy. You need to do something to bring up the occupancy because once you bring up the occupancy, you can do a lot of other things. Brian: Exactly. James: You can't do it when the occupancy is low and you're adamant about pushing up the rent. So was your thought process, rather than I leave this unit vacant, that's the biggest loss compared to giving [19:48inaudible] $25 or $30 increase that doesn't make sense. Brian: Yes. That's right. So, you know that's been one of the strategies that I've adhered to and has worked well; you lower the rents and lease it up and then you make improvements as you go and then you raise rents from there. Nothing more expensive than vacant space. The other piece of that which is an advantage of not syndicating is that I have been able in many cases to fund many of the improvements out of cash flow. So with this particular property, we did lower the rents, but the occupancy has been brought way up. So we've just crossed a threshold where now this property is cash flowing again and all that cash flow is going to be directed right back into making improvements, probably, for the next few years at least. And so, that's a perfect example of well, if you're going to syndicate and you need to pay investors, you really can't be investing all of your cash flow back into a property. So what do you need to do? You need to raise some money up front to pay for those improvements and not count on cash flow so that you can achieve your investor returns and start to get them their money back. James: Yeah. That's the one thing different with syndicated deal versus owning your own deals. You don't have to raise so much money so you can take your cash flow and just put it back. With a syndication [21:27crosstalk/inaudible] and you may lose deals because you're competing with somebody who has a lot of money versus somebody who is syndicating. Brian: That's right. James: It's very interesting. So in terms of, I'm going to your personal side, is there a proud moment in your life or not in your life, toward your real estate career, that you think, I would remember that moment throughout my life until the end; can you describe that moment? Brian: Oh, wow, you know there's been so many moments, but not all good. James: No, no, the proudest moment where you think you really made a big impact on something. Brian: I never really expected this but some of the proudest moments that I've had has been since my book came out and I would have never guessed that that would lead to that but some of the feedback that I've gotten from readers that they've shared with me that it's changed their lives that they started into investing and have already built portfolios. And to see the direct link between the book and people, you know, really making improvements in their lives has been extremely rewarding. So I think one of the great things is that I really went into the idea of writing the book just because I wanted to share what I've learned, the mistakes I've made and to help other people, but I never really thought that it would sell very many copies or that people would have that kind of effect and the fact that it did. When I get a letter, a note from somebody, it's been extremely rewarding. So now I kind of remember that I think that's been a big impact. James: Yeah. It's interesting. I mean, I get a lot of notes from my books as well and sometimes you don't really take it seriously because for us it's just common knowledge from what we have learned. But some notes do make us think, oh, I really really made an impact on someone. I mean, it's mind-blowing in how many lives can be changed with the things that you share in a book. Brian: Right, right. Yeah. Absolutely. James: Yeah. So the next second question is why do you do what you're doing? Brian: Well. You know and it's interesting. I mean actually, in the book I share at one point, this was a few years back, I had somebody come up to me and they said you know, how much is enough? Like you are so greedy, why do you keep going? And I just realized that this person doesn't understand, they missed the whole point that it's just rewarding to take a property that's not performing, that's in distress, that's maybe even a bad thing in a community and to turn it around and make it a better place for people to live. You help the tenants and you help the community and to do that and start to get involved. Like I do meetups now and I met new people and threw those in the book to help other investors, and so, you know, I look forward to going to work every day. I enjoy it. I enjoy the challenge of finding and executing on properties that aren't achieving up to their potential and making a better place for people to live and more profitable at the same time. So I just think it's fun. Like I enjoy what I do. James: Yeah, it's like a discovery, you're trying to discover these from your paper to the real stuff. Especially when you are underwriting because you're assuming a lot of things and how does that whole assumption become a reality? You know, it's very interesting to see the output of that become [25:42inaudible] people's lives, which is just... Brian: Absolutely. James: So we really had a really good knowledge box from you, Brian. So can you tell our listeners and audience how to get hold of you? Brian: Sure, you know, your listeners can find me on Facebook. You can find me on LinkedIn, you know, you can find the book on amazon.com or on the book website is crushingit.info and my company's website is Washingtonstreetproperties.com And if anybody is interested in reaching out, I'd be glad to hear from them. James: Awesome, Brian. Thank you for coming and joining us. I think that's it. Thank you. Brian: Thanks, James, was an honor.

new york texas new york city north carolina kentucky transitioning owning retail gps mastermind marketplace apartments southeast big bang sas units commercial real estate upstate new york syndication upstate crush it crushing it brian murray c class t12 james no james kandasamy james it james you brian yeah brian you james yeah brian so james so brian oh brian well james well brian yes brian right james brian james absolutely brian exactly james okay

Everyday Heros, Miracle Babies & Tourette's w/ Brian Schulman (005)

Way To Greatness

Play Episode Listen Later Oct 13, 2019 61:17

Learn about becoming an everyday hero, being a better parent, living every day as a miracle baby AND with Tourette's syndrome, and more!! Brian Schulman of Voice Your Vibe comes on the show and gets real as he describes to us the crazy things he went through growing up as he went through wild experiences - and emerged as a person who is always spinning things positive. Brian works to inspire people and his children every day! Show Highlights 02:20 - [Ari] How do you go about inspiring one person each day? 03:08 - [Brian] I wanted to give out the opposite of the negative that I had received by those around me. Don’t get me wrong. I mean I certainly had a ton of positivity around me but sometimes, we all know how one negativity can overshadow a thousand positives. 03:25 - [Brian] So I wanted to be a giver of good and positivity, and light, and strength, and encouragement. And I know life is 10 percent what happens to me and 90 percent how I react to it. 03:42 - [Brian] I know I've failed way more in life and in business than I've succeeded, that I've learned from every step, and I've gotten up every time. I've dusted myself off and I'd get going and that’s why I've succeeded no matter what the outcome is. 04:18 - [Brian] My ”why” is to inspire one person a day and that can be in so many different ways. If I can inspire one person to chase after their dreams and no one feel and believe that they can accomplish anything (continue to say more here…) 05:00 - [Brian] You know the reality is, life has and will, and continue to throw me mad curveballs and I will continue to get up to the plate and keep swinging. 12:27 - [Brian reading his daughter's essay] This story about to be told is about an extraordinary man now 42 years young, who accepted all the obstacles life had to throw at him. I know you may be thinking that everyone has different obstacles they just overcome throughout their lifetime so how is he different from you and I? 12:45 - [Brian reading his daughter's essay] What makes this particular man different from you and I is the tactic he used when faced with these obstacles. What makes him different from you and I is that he took these obstacles and used them to help make a difference and inspire others to never let life get in the way of their greatness and achievements. What makes us different from everyone else is not the obstacles life throws at us but what we do when faced with these obstacles. 13:14 - [Brian reading his daughter's essay] Will you choose to accept defeat? Or will you choose to persevere? 14:57 - [Brian reading his daughter's essay] No one knows what life has in store for them. All we can do is live until we are forced to face an obstacle and we must make a choice to either overcome or accept defeat. 15:13 [Brian reading his daughter's essay] One person can make a difference in the lives of others. But the experiences this boy has to face came a life lesson. with determination, perseverance, and support from others, anything can happen. 15:50 [Brian reading his daughter's essay] He helps lift peoples spirits and hopes in times of hopelessness and despair, He helps people turn their ideas and dreams into realities, he helps build people from the ground up and hopes that they too will one day share their stories with others, he makes the difference in this world each and every day which inspires others to make a difference as well. 16:13 [Brian reading his daughter's essay] Who may you ask is this man? Well, he is my father. Out of all the lives he's made a difference, I believe he has made the biggest difference in mine. He inspires me to never give up on what I believe in, He always told me to do what makes me happy. But most importantly to always live every day as if it were the last because we are never guaranteed a tomorrow. 16: 52 [Brian reading his daughter's essay] Be the change you wish to see in the world making a difference in the lives of others. 19:24 [Brian] "They're all here for me, they know who I am, I don't need it." 20: 47 [Brian] Tourettes, by the way, is more common on boys versus girls and usually between 16 and 18 years. 24:18 [Brian] My legacy are my kids, my legacy is every person that I have, the ability whose hearts I have devoted or touched in some way. 24:38 [Ari] If all were able to touch us, the people around us but not our own kids and our own family, aren't we may be doing something wrong? Shouldn't we be focused on family first? I don't mean the only family but if we're gonna be inspiring people, our kids come first. Why should we let somebody else be the inspiration for our kids? 25:14 [Brian] We could talk about this all day, I always wanted to be a daddy. It was no question in my mind. I love kids, I wanted to be mine, mom and dad form. Like she was so inspiring to me. 28:20 [Ari] The typical definition of success from most people is money, great job, doctor. All these things, the people just internally, without thinking about it and that's the pinnacle of success. 28:40 [Ari] The truth is if I could get to the end of my life, look back and be able to say that I'm a good dad. That would be successful. 29:00 [Brian] That's not that, at the end of the day when we're taking our last breath, that's not what matters. What matters is the impact we make in my opinion. The impact we make on people's lives. 29:32 [Brian] They're better human beings. If their hearts are bigger, they are kinder. 30:16 [Brian] "You can't turn the clock back." 32:32 [Ari] From most parents or at least for all good parents the choice is to never accept defeat. The choice is always what do I have to do to overcome or make a passes, it doesn't matter what it is, I'll make it work. 33:13 [Brian] When we become parents, it's not about us anymore, like literally it's like snap and everything changes for the good. 34:21 [Brian] We have genuine heroes around us every single day, people that take a selfless action and make a dramatic impact on somebody's life. It happens every single day, every single moment. 34:56 [Brian] I remember thinking to myself, I don't ever want anyone to feel the way I feel, I don't want anyone to ever be treated the way I've been treated. I don't want them to feel that way or feel like they are being treated that way when they're around me. 39:32 [Brian] So I just said: "well, I'll just tell my story." 41:40 [Brian] I'm inspiring people by sharing it and it's not that they have got to have gone through what I've gone through, we all go through trials and tribulations in our life. We all get beat down, it's about getting back up and I didn't think I had a voice for a story that matter, I just looked at it as life. 44:20 [Brian] This is the most important part for anyone "It's not about you, it's about sharing your experiences that can help at least one person out there" and if you think of it that way, it changes your mindset completely. 45:13 [Ari] Sometimes when we're so down when we've got so much stuff that's happening to us that are creating such havoc in our lives or such negativity or potential negativity or whatever it is and we're just all the way down there. Legitimately the one thing that helps us more than anything else is another person reaching down with their hand to lift us up. 45:40 [Brian] Yes, that's what it is about, it's about community, it's about people coming together, it's about knowing that we all go through different things in our lives and we can do that. 47:16 [Ari] Every single time you get up, and people don't realize this and this is part of the issue, and every time you get up, you're stronger. And every single time you stand up you're that much more able to withstand whatever else life throws at you in the future and whatever else it comes. "I had that moment 2 years ago when I had that really big issue but I can do it I can get passes and I know I can!" 47:41 [Brian] Great ways to remember that? Pull out your phone when you're on your worst moment that you feel you're having in your life and hit record and talk to yourself. About Brian Schulman Check out Brian Schulman's guest biography (https://www.waytogreatness.com/guest/brian-schulman/), or read what Julian Smit wrote about Brian: "If you don't know this fine man, you're not spending enough time on LinkedIn! From Miracle Baby to Forbes Featured Entrepreneur, Brian has gone from fighting for every breath as a miracle baby, born at only 1.5 pounds, to Tourette's syndrome as a pre-teen, to one of the most prolific and popular video creators on LinkedIn! He's the founder and CEO of #VoiceYourVibe, and is a co-host of #LinkedInLocal San Diego! Despite all he's been through he's one of the most positive people you'll ever meet and is The Godfather of LinkedIn Video. He's an inspiration to many and a joy to talk to!" -Julian Smit, Host of the BIYF Marketing Podcast (https://podcasts.apple.com/ca/podcast/biyf-marketing/id1372483245) Originally recorded 10/7/2019 Special Guest: Brian Schulman.

ceo influencers forbes godfather vibe tourette heros tourettes legitimately brian schulman miracle babies brian you voice your vibe brian yes julian smit

Ep#23 Finding Great Operators in Non Multifamily asset classes with Brian Hamrick

Achieve Wealth Through Value Add Real Estate Investing Podcast

Play Episode Listen Later Oct 8, 2019 48:50

James: Hi listeners and audience, this is James Kandasamy from Achieve Wealth Through Value-add Real Estate Investing Podcast. Today, we have Brian Hamrick. Brian owns 370 units which 2/3 of it is syndicated, the remaining is owned by him. He's from Grand Rapids, Michigan. He does multifamily, self-storage and also non-performing notes and Brian is also the past president of Rental Properties Owner Association. Hey, Brian, welcome to the show. Brian: Hey, James, great to be here. Thanks for having me. James: I'm really happy to have you here. I mean, you have been podcasting for the past three years. You have a really good audience because I remember after showing up on your podcast, a lot of people did contact me. So I'm sure a lot of people love your podcast as well. Brian: That's fantastic. I'm glad to hear that. James: Yes. So can we go a bit more detailed into what is this Rental Properties Owners Association, how do they add value to syndicators or landlords or tenants? Can you describe a bit more on that? Brian: Sure, the Rental Property Owners Association, which I'm a past president of, I'm currently on the executive committee and I sit on a number of different committees, they are a landlord representation organization. So we also work a lot with Real Estate Investors and provide all kinds of training for both landlords and Real Estate Investors. Every year, we have an annual conference where we have National Speakers come in and talk about all different types of investing asset classes and whatnot. And really I got involved with it because when I moved here to Grand Rapids, 15 years ago, I was looking for a professional organization that I could become part of that would help me network with other professionals in the industry. People who own rental properties and knew how to profit from it and also just an organization that would help teach best practices so I could learn the ropes how to do it and certainly through the Rental Property Owners Association and the people I've met there, I've learned a lot. We provide a lot of training but probably what I consider most important of all is we have a legislative committee that works with lawmakers, both local and at the state level, to help push through bills that help rental property owners and also help prevent bills from becoming a reality that would hurt us; anything that has to do with like rent control or some of those hot button issues that as landlords and rental property owners would like to avoid. James: Yeah, very interesting. So like New York and I think, Oregon now is rent control states, if I'm not mistaken, so they probably have similar Association like yours in that city, I guess. Brian: I would hope so. It sounds like they're fighting a losing battle as you and I both know as rental property owners, you know, I believe you invest out of state, out of your area, is that correct? James: No. No, I'm from Austin. I invest everything in Austin and San Antonio. Brian: Okay. So would you even consider investing in a city or a state that has rent control? James: No. Of course not. Brian: Yeah. It's really detrimental to the market and I think it's going to cause a lot of problems. I used to live in Santa Monica, California where they had rent control and you can see the negative results of that. James: Oh, Santa Monica in California, did they have rent control in the past? Brian: Yeah, a lot of the Los Angeles counties, you know, it's kind of county by county, city by city, area by area, but there is rent control in Los Angeles in certain areas and you can just see how rental property owners, who own buildings in rent control areas, have no incentive to put money back into them. They're not putting the capital expenditures back into their property to keep them in good shape because there's no incentive to do so. They can't raise rents beyond a certain amount each year and you know, so why would you invest $100,000 back into your building if you're not going to get that out in value? James: Yeah. Yeah. It doesn't make sense for a business. So you may not run it as a business, you may be just run it as cash flow, I don't know, it's like a cash flow investment. I guess you don't have to spend any capital on it. Brian: I can see how if you've owned the property for a long time and you bought it at the right price at the right time, you could probably be doing well with cash flow. But in these markets where you see a lot of rent control, they're expensive markets. So I'm not really sure once rent control is instituted in these markets what's going to incentivize new investors to come in and bring fresh money into the market. James: Interesting interesting. So coming back to your portfolio, can you tell me in terms of your holdings, how much is multifamily, how much is self-storage? How many percents of each one of these and how much is non-performing notes? Brian: Sure. Sure. So multi-family is my bread and butter. I've been doing that since 2008. I moved to Grand Rapids in 2005 and 2008 the bubble burst, you know, we entered the Great Recession, it was a buyers' market. I bought my first 12 unit, I was using my own money in the beginning, started using other people's money and then started syndicating. We currently have about 370 units here in the Grand Rapids area, Grand Rapids, Michigan and that's multi-family residential. In 2018 we purchased a self-storage facility, it's about 28,000 square foot, we're currently adding another 15,000 square foot to it and that's been a fantastic investment, I really love self-storage. And then, as you mentioned, I host a podcast - The Rental Property Owner and Real Estate Investor Podcast - and one of my guests over two years ago was a gentleman by the name of Gene Chandler and he was investing in non-performing notes and I really liked his strategy so much that I ended up investing well over 300,000 dollars with them and the results have just been fantastic. James: So, you now do multifamily and now you're doing two other asset class. So can you tell me what does multifamily did not offer that these two other asset class offers? Brian: Well, I like you, I'm investing in my own backyard for when it comes to multifamily. Even though I've bought and sold over 450 units, in 2015, I stopped buying multifamily altogether because the values had gone to a point where I could no longer justify syndication. I couldn't get the returns that I needed for my investors to be able to to pay the prices that people were asking. The last two deals I found - one was off-market, one was kind of in between market - and I can go into details on that but anything that I saw after that point just, I was so spoiled by the prices I was getting between 2008-2014, that I started looking for other asset classes. And there were probably about 3 years where I just sat on the fence, waiting to see if the market would change or something else would come along. And at some point, one of the people who I met through the podcast, brought me a self-storage deal that he had found off-market. I looked at it, I like the numbers. His underwriting was very conservative, but the numbers were very compelling and we ended up buying that in 2018. And just in one year of basically bringing the rents up to market value and switching to a virtual online web-based management system, we were able to add over $700,000 in value to that property. So I like the simplicity of managing and owning self-storage more so than multifamily because in multifamily, you have tenants and plumbing issues... James: So it's very Property Management intensive, right? Brian: It definitely is and the self-storage, it's not. When you have turn-over, you're basically sweeping out a metal shed, you know, so it's a lot easier to manage and own and operate self-storage, especially when you're in a good market and I think we bought in an excellent market. It's just north of Lansing, Michigan. And then with the non-performing notes, I found a strategic partner who handled a lot of the nuts and bolts of that and I was able to invest with him somewhat passively so I enjoyed that aspect of investing there and the returns we were getting were very good. James: Interesting. Yeah, I mean, as I mentioned in my book, commercial asset classes go in cycles. I mean, I know I'm a multi-family guy and your bread and butter is multifamily but if you find the right operators in other asset classes, you can make a lot more money or equal amount of money as what you're making with multi-family. So, would you think so? Brian: Absolutely. Finding the right strategic partners in other asset classes that's one of the things I set my mind to when I realize I'm just not seeing the returns I want to see in multifamily and apartments in my area where I'm comfortable investing. Now, have you looked at other asset classes? James: I did look at a few asset class. I mean the asset class that I looked at is also like, you know, self-storage or mobile home parks but it's also in demand. I'm surprised to see here that you found something in 2018 because I thought self-storage is a hot asset class as well, I will risk going after that. Brian: Yeah, it was a lucky strike and we've been looking for similar opportunities. But yeah, we're not finding them. What we're doing instead is building ground-up construction in self-storage, finding locations where the demographics are right and the need for more square footage of self-storage space is there and then we go in and fill that need. James: Yeah, but I'm happy that you are looking at multifamily is not like the only asset class throughout the whole real estate cycle. I mean you felt like in 2015, things picked up and you really can't find the prices that you want and you have changed strategy which is how an investor should be. You always want to look at what's available out there, the deal flow because the economy is still doing very well. There's a lot of capital out there and it's just harder to find a great really-making-sense deal. I wouldn't say deals, making sense deals in multi-family, something that makes sense. It's just so hard to find out nowadays. Brian: Absolutely. As an investor, you have to stay nimble and flexible and be open to other opportunities. Now, I know a lot of people in our field, our asset class of multifamily and apartments will find strategic partners outside of their area like in Texas or Georgia or wherever and partner with strategic partners who are able to find better value and better yields in their Investments. But I've had some bad experiences early on with some single-families that I owned out of state so I've always been very hesitant since then to own rental property, residential rental property, out of state. James: So you like to have any property within your own backyard, but you like to diversify within asset classes. Some people have one asset class, but they go across the nation. Like some people like to buy multi-family across the nation, wherever make sense but you are doing it the other way around. Brian: Yeah. Since I've branched out into self-storage and non-performing notes, I'm comfortable switching up asset classes. James: Awesome. So on self-storage, are you the operator, are you the primary guy? Brian: No, my strategic partner is. He's the one who found the deal off-market, he negotiated it. I basically came in and raised the money; we syndicated that and raise the funds to be able to acquire it. James: Got it. Very interesting. And on the performing notes, you have a strategic partner, I would say, right? Brian: Yeah, I have a strategic partner on that. He's the one who knows that world. He's been doing it for well over six years now and really knows how to negotiate with the lender who we're purchasing a non-performing note from. He works with the homeowners to try to keep them in the home and figure out if that's even possible and then knows who the title company is that he should work with to get the right due diligence done and he's got the different scenarios in his head of how we can profit off of these notes. If we keep the homeowner in the home, what are the strategies there for us to maximize our profit or if we have to go through the foreclosure process. How do we go about that and maximize our returns in those cases as well. James: Interesting. Interesting. So if you get a multi-family deal today, would you still do it? Brian: If I found a deal that made sense and my underwriting shows that I could get the returns to my investors that they're accustomed to, I'd do it in a second, absolutely. James: Okay. Okay. So let's talk about the market and submarket selection. So why did you move from California to Grand Rapids, Michigan? Everybody's heading to Texas and Florida from California. Brian: I'm from Michigan, originally. James: Oh, you're from Michigan? Okay, that makes a lot of sense. Brian: Yeah, my wife is from here as well. So we met in California but decided okay, if we get married, start a family we didn't want to do it in Los Angeles, it's just too busy there. James: Makes sense. Yeah, I mean just based on data that 50% of the population move to Texas And I think there's a lot more but Texas and Florida is the favorite destination for people from California. That's why I was asking the question. And how do you select the submarket in Grand Rapids, Michigan? Like how do you select which submarket to really do the deal? Brian: Well eyes because I live here, I am looking within a half hour to an hour of where I live. Grand Rapids is very strong, has very strong demographics. It's one of the few Midwest cities that really bounce back strong from the Great Recession. A lot of diversified manufacturing industry. Furniture, Amway is here, we've got a lot of different industries and employment based here. So when I look at submarkets, I'm looking more at the neighborhoods, what's the crime rate in that neighborhood? What's the income level in that? What kind of rents can we command and by the way, I'll buy B properties and C properties or you know, C minus properties that we can push into that C plus B minus range. But I will avoid the The D areas and I've seen a lot of opportunities in the D areas. And by D, I mean where you have a lot higher crime rate, where you have a lot more evictions and tenant turnover and problems. So I'm just very careful about and I work with the property management company that has a good grasp of these areas. So when we look at a property, we can really get a sense of if we buy this, is there an upside value, can we improve it and get higher rents, get better residents in here or is it going to be bound by the neighborhood it's in, that where it is now is what just where it's going to be? James: Got it. Got it. Interesting. What about underwriting? I mean, when you look at a deal like I mean when you are buying multifamily, right? So how would you select the deal? Let's say a hundred deals been sent to you, do you know how many percents of it you would reject? Brian: Right now 100%. I'm not even looking right now, but what I'll do is I'll do a quick rule of thumb. Okay, what's the net operating income? What's the cap rate that they're asking? Is there upside potential? And of course, if it's listed by a broker, they'll always tell you the market the rents are way under market. you can raise the rent. No problem. That's sometimes true, sometimes not true. But this area is so strong that any seller right now knows that they can get top dollar and while there's a lot of Institutions and out-of-state investors and even International investors who are willing to pay top dollar, the yields that they are willing to accept are much lower than what I'm willing to pay, which is why I'm not even looking at the moment. James: Very interesting. Now I see it's happening across the country. I thought it was only happening in Texas and Florida but looks like across the country, that's what's happening. It's just so hard to find deals that used to make sense to us long time ago, right? So it's crazy out there. Brian: Yeah, and it could just be that I'm spoiled because I was buying during a period when I could buy it at eight nine ten caps. And now, when I see things at five six, six and a half caps, I don't even want to consider them. But had I bought it at those cap rates between 2015 and 2017, I would have made a lot of money. So maybe I'm just a little too stringent in my criteria right now. James: Yeah. That could be it as well. Brian: Are you buying right now? James: Well, I mean, well, I'm still buying if I find the right deal. It's just so hard to find the deal that makes sense for my criteria, and I'm sure that's the same thing as your criteria. I'm still buying if I find the right deal but I'm not underwriting a hundred deals, you know, in one month. You know, whatever deal comes to me, I usually know that within the quick look, I know whether it makes sense for me to underwrite or not. And sometimes brokers will call me if they know that a certain deal is something that I would do. That's the only deal that I look at. Brian: What's your quick back of the napkin way of determining whether or not you want to invest in something? James: If it's an email blast, I probably wouldn't look at it. Brian: Yeah. Yeah, you kind of eliminate the ones that go out to everybody. James: Yeah, it's already got everybody on his shop date and coming on an email blast. You know, you have to go on a best and final and best and best and final and then this ultimate best and final offer, which is you're shooting in the dark, right? You're basically bidding against yourself. [20:45 inaudible] I'm not really in a desperate mode to buy deals that go through that kind of process. So when I look for value-add if there's a true value-add deal, I mean, minus the crime rate area, I definitely know the area that has high crime rate, I can check it out quickly Class B and C, but need to have true value-add that we can go and add value. I don't really look at the entry cap rate, but I look for the spread of the cap rate from the time I buy to in the next two years kind of thing without any rent increases. Brian: I think part of part of my problem, one of the reasons that I've just been on the fence is because we bought a value-add property back in 2015. It was an older building, built in 1920 and it was such an exhaustive process to go in and add value to that property. I was over there like every day. James: It is very tiring to do those value-add deals. To do deep value-adds, I would say. Brian: Deep, deep value-add. And so my bandwidth for more opportunities was just completely limited because I was so exhausted by working on this one particular project. Now, luckily, we got it to a point where we added tremendous value to it and we're very proud of the work we did but you have to weigh the opportunity cost when you do those value-adds because sometimes they're so intensive that some of the lower hanging fruits, you bypassed that. James: Correct. Yeah. I see some syndicators doing deals every month and they're not doing a deep value-add or they're just doing the lighter value-add. Maybe they're just doing a yield play. [22:30inaudible] they can buy every month. They can claim 5,000 units or 3,000 years versus deep value-add to be like 100 and 200 and 300. It's a really really deep value-add. You probably make a lot more money than the guy who owns 3,000 to 4,000 units, but it's a lot of work. Brian: It's more than just asset managing. You kind of become a de facto developer. James: Developer, a huge project manager. Yes, so many things but the deep value-add gives you a sense of accomplishment. Brian: It does. I'm very proud of the work we did on this particular property and more so than any of my other properties because I didn't have to put nearly as much work into them. James: Yeah, and the deep value-add it becomes a case study, right? Because it truly shows your skills to turn around property. And people who have done deep value-add it's going to be easier for them to do the lighter [23:30inaudible] Brian: Yeah, yeah, that's an excellent point. James: So that's very interesting. So can you name like 2 or 3 secret sauces to your success? Brian: The two or three secret sauces to my success. I'm sorry if you hear that printer going in the background there. James: It's okay. No worries. Brian: Hopefully that ends soon. Secret sauces to my success; I think doing the underwriting, running my numbers. I always like to say, I like to see my numbers in bullet time. To see all the Matrix, you know, everything slows down and you can see it coming at you. I want to know what are the real expense is going to be after we've acquired the property. One particular mistake that I see a lot of investors making is they assume that the property tax is going to be the same as what the previous owner was paying and that's just not the case. So right there that's one of the main factors that I look at right away, is what is the property tax going to become once I buy this property and that eliminates 50% of the deals that I would even consider. So number one secret sauce is just really understanding the numbers. Not just where they are today, but where they will be once we acquire the property. Number two is having the right team. I am all about partnering with strategic partners who add value because they understand inside and out the asset class that you're investing in. The reason I was able to expand my multifamily portfolio was that I partnered with someone who owned his own property management company and managed the type of properties that I wanted to acquire. That without his assistance and without his team that really knew how to go in and do the due diligence and help me assess upfront, what are the capital expense costs going to be? What are the true costs going to be when we acquire this property? Without that, I would have made a lot of mistakes. The same with self-storage. I partnered with someone who even though he's young and new, somewhat new to the business, he had really studied it, talked to a lot of professionals, been mentored by people and really understood inside and out how we could add value to that self-storage facility. And everything that he put in his pro forma ended up becoming a reality. With my non-performing note partner, I mean he knows that world inside and out. So when we acquire a note, the first 12 that I bought with him, we only had one that we lost money on and that was about $1,700. James: Out of how many notes? Brian: We bought 12 notes to start with because I like to test before I bring other investors in so I bought 12 notes with my partner, I JV with him. Five of the notes our average return was over 80%. James: Wow. What timeline? Brian: A year and a half. Well, actually, each note is kind of on its own timeline. So I'll tell you that of the twelve notes that he and I purchased together, five of them are closed and paid off like we've made our profit. Our average return on investment, before we split 50/50, our average return was 81% and that included the one note that we lost $1,700 on. Some of the returns that we're getting are phenomenal. Five of the notes are re-performing, which means that we were able to keep the homeowners in their homes, which is fantastic. That's our number one goal. Our average return on those notes as we collect the monthly income is 30%. And then two of them are in some form of foreclosure. In fact, we're about to sell one. We just listed it today actually, so we should make a decent return on that. We always try to work with the homeowner and keep them in the home. Half the time we're able to do that, half the time it just doesn't work out. But you asked me the timeline so, of those five notes that we closed, our average return was 81%, the average number of days that we were in each of those notes was 163 days so that took less than half a year. James: I mean, those are good great numbers. I mean, I mentioned in my book, find the right operator in that asset class and partner with them or invest with them for passive investors. So as I said in every asset class, there's always good operators. So the numbers you're telling me in non-performing notes in self-storage are huge, right? I mean, I know multifamily you can make money if the market went up and you have a really good operator that can handle that. On average, not everybody is making what you just told me right now on self-storage. So why is multifamily more popular than other asset classes? Brian: There are more people teaching it. James: That's absolutely my point. Brian: Yeah, I mean like there are some excellent instructors out there in multifamily and you and I are both the part of a group with one of them. I mean great top-notch training material. Okay. Yeah, there's just fewer people out there. Whereas you have between 10 to 20 people out there teaching multifamily, you could count on one hand the number of people teaching self-storage and it's even less teaching the non-performing note. James: I understand. Yeah, it is it is true. There's a lot more people teaching multifamily, a lot more boot camps, a lot more 2 days weekend seminars on multifamily compared to self-storage or non-performing notes. And I think multi-family is also very simple to understand, it's a house. Not many people understand what is non-performing notes. Brian: Yeah, there's all that educational like just understanding and wrapping your head around the concept. I got into multifamily because I understood the economy of scale and I understood people have to have a place to live. So if you can get them to pay their rent and that rent pays all your expenses plus the mortgage, well, you can make a lot of money that way. And then once I understood the next level of value, which is the income valuation method, how commercial multifamily is valued based on the income method and you can increase your returns exponentially if you understand that. The relationship between cap rate and your net operating income and value that was very compelling to me. And I think that still is very compelling when it comes to investing in commercial real estate whether it be multifamily or self-storage. I think non-performing notes, there's a lot more perceived risk in that because it's not valued based on any - it's hard to understand how that's valued because there are so many different scenarios in which you can profit from non-performing notes. That you can't just say well we value it this way and if you buy this note, this is what you're going to make, it's kind of a crapshoot. But if you do it right and you partner with someone who knows how to avoid the dogs, you can actually make a lot of money doing it. James: So what is the most valuable value-add in non-performing notes? Brian: You mean an example of one of our...? James: No, not an example. I'm talking about what is the one thing that if you do the most of the time or the frequency of things that you do in non-performing notes that you get the most value out of? Brian: Well, yeah, it differs note by note. I'll give you two examples. One is a property that was pretty much a teardown property that we bought the note on in Middlebury, Indiana. We paid $5,000 for this note and I asked my partner, I mean it's $5,000, this property is a teardown. How are we going to make money on this? And he said, well, we're not buying this for this property for the house that's on it. We're buying it for the land because it's right next door to a farm and this farm is owned by this Amish family. So he sent a realtor over to the Amish family and they ended up paying $35,000 for that note. So after closing costs and paying the realtor and getting our initial $5,000 investment back, our profit was over $24,000 that represented a 245% return and we did that in less than two months. James: Yeah, but you need to identify that opportunity. I mean, it's not like you can go and buy any deals right now. Okay, very interesting. Brian: Yeah. Yeah, absolutely. Another quick example of how you can profit on notes and I don't want it to lead you to believe that your best profit is always going to be a few foreclose or take possession of the property because you can still make a lot of money if you can work with the homeowners. We bought a note on a property in northern Michigan, probably about 9 or 10 months ago now. And I believe the numbers were in the line of we paid $20,000 for this note, got the homeowners re-performing, the unpaid balance on this note is $41,000. Once we have them season for 12 months, meaning that they're paying on time for 12 months - we've been working with them with a mortgage loan originator, where they can go and get new financing, permanent financing of FHA or Fannie Mae type loan in place with much better interest rate much better payments. Well, when they go do that, they're going to pay off that unpaid balance. So our $19,000 investment, now that I'm thinking about it was $19,000, our $19,000 investment, we're going to get paid that $41,000 of the unpaid balance on their note, plus the money that they've been paying each year. So our return on that is going to be 100%, it's actually over a hundred percent. James: Across how many years? Brian: We'll be out of that in under 15 months. James: Okay, interesting. Brian: Because they're going to refinance and when they refinance, we get paid that unpaid balance. James: Got it. Got it. What about on the multifamily properties that you own before 2015? What do you think is the most valuable value-add that you really like? Brian: Well, they're all great because just anything I bought between 2008 and 2012, I've achieved an infinite return on those. James: Okay. So refied it by and you kept it? Brian: Yeah. Yeah, we've refinanced, pulled our initial investment out. We have no money in the properties and we're collecting cash flow every month. So you can't calculate a return on that. Probably one of the best examples is a 37 unit that we purchased. We bought it at a short sale in 2009, was about 600,000 is what we paid for it. We put a $200,000 into it right away to replace roofs, windows. It was a hodgepodge of heating systems. There's electric baseboard heat and hot water boiler heat and then gas forced-air furnace heat. It just depended on which unit you were looking at. So we replaced a lot of the mechanicals, made it as much of a new property as we could, as far as just the mechanicals and the roof and the windows. And we refinanced it once it had over 1.1 million dollar value, pulled all of our initial investment out plus some extra cash flow and then we just refinanced it again, put a tenure fixed loan on it through the Freddie Mac. small apartment loan. So we got great terms on it, 30-year amortization. At that point, it valued over two million dollars. So we've added a lot of value to it and the compression of cap rates didn't hurt either. James: Yeah. Yeah. Those are the awesome deals, the deep value-adds. That's where you can go and refi and make it infinite written because you pulled out all your cost basis. Brian: Yeah, yeah. Yeah, that's the goal to achieve infinite return. Whenever we can do that, that's what we do. James: Absolutely. Aren't you worried about the state of the market right now in real estate in general? Brian: You know, gosh, I was more worried about it two years ago than I am now probably. James: What has changed? Brian: Probably because two years ago, I was thinking, oh, it's going to turn any minute now and then it only got better and better. You and I both know Neil Bala and we talked to him at the last event we were at together and he made a very good case for the continuation of this market. And it basically rests on the fact that the United States, it's one of the few, if not the only places in the world where you can go to get real yield on your investment. We're seeing a lot of international money coming into the United States because in their countries, they're seeing negative yield or 0 yield. Here even if you can still get three or four percent yield on your investment, that's a lot of money. It's bringing a lot of money into this country and that's going to prop up our values for quite a long time. On top of that, I've always fought or believe that interest rates were going to rise and I've been believing that since 2000 and they keep going down. And even now, as we're speaking, they're talking about lowering the rate again by the end of the year. So that interest rate risk, I know we're playing with fire here and eventually, we're going to have to pay the piper but our government seems to keep coming up with ways to prolong this growth and the increase in prices. So am I worried? Not in the short term. No. No. The Economists I listen to are saying, oh, it's going to be a roaring 20s for us. Things are really going to hit the fan and. 2027, 2028, 29. James: Interesting. Yeah, because I think I don't know, maybe my thoughts are similar to yours somehow the Fed has figured out how to do quantitative easing and quantitative tightening. Somehow they're able to contract the economy and bring it down. So they could have found some new mechanism to keep the economy going even though our thought process always has been real estate goes in cycles. But at some point, you will hit an affordability issue, it can't [40:13unintelligible] go up all the time, right? Brian: Yes. James: The prices can go up because the interest rate is coming down because now you can get more cash flow. But at the same time, you can't keep on increasing rent because our wages are not going up so much. I mean, I'm not an economist but at some point, you will hit some roadblock, but I'm not sure where is it and how is going to come. Brian: Yeah, well, we're seeing a plateauing I think right now in just the rents that we're able to charge, the prices that people are willing to pay but it's still a very strong market. Now, don't get me wrong, I'm not going out there and just buying stuff like crazy because I am very conservative and like I said if I can't get the returns that I need to bring investors into my deals, I'm just not even looking at it. I don't anticipate that the market is going to have a huge correction, there might be a bump, I think if you're in a good market, like Grand Rapids, that bump won't be nearly as severe as some other places. I'm keeping my eye on the market but at the same time, investing conservatively in asset classes that I think will be able to withstand the next correction. James: Awesome. So let's go back to a personal side of things, right? So is there a proud moment throughout your career in real estate that you will remember for your whole life, one proud moment? Brian: One for a moment to put on my tombstone. James: Yeah, absolutely. That you really think that hard, I'm really proud I did that. Brian: Yeah. So a couple of answers. I mean any time we're able to go in and improve a property and improving neighborhoods, that always makes me proud, you know, that we're adding value to a neighborhood and community. The older building that I told you about here in Grand Rapids, it was built in 1920. When we bought that it was very tired, kind of poorly managed, it was losing money. We were able to turn that around so I'm very proud of that. I'm very proud of the fact that we also fought very hard and work very closely with the city to be able to put a restaurant in that building. So the fact that when we bought it it was 96 apartment units and about 6,000 square foot of vacant commercial space. Now we had to work with the city to get it rezoned because it had been vacant for so long, it had to be reverted to being zoned residential. So we spent over a year trying to get it rezoned so we could add commercial in there, but we filled up all 6,000 square foot including a restaurant and that took about two or three years to do. So when I think about what I'm proud of I think I'm definitely proud of that. James: Awesome. That there is hard work because you're turning the zoning from residential to mixed use. Brian: Yeah, mixed-use residential commercial, just dealing with parking, number of parking spots and green space and tree canopies. I mean, it was a massive undertaking. James: Yeah. It's very interesting that kind of work. I did one that was borderline and we merged it with an apartment and we did so many things. It was a very unique value-add that we recently refinance. Brian: What was it, a lot of work for you? James: It was a lot of work because you have to go through, you know, buying the deal - you had to buy two deals at the same time. One is the apartment and one is the land and then we have to go to the city to merge these two plots. Then you had to rezone it, then you had to - I mean replot it, rezone it And then after you do a tree survey, you have to do so many different surveys have to do to get that. It's not normal in a residential, you know, where you buy today and increase rent, reduce expense kind of deal. But it's very interesting and people got 80% of our money within 15 months, which is huge, just by doing this creatively. Brian: That's fantastic. Yeah. Yeah, you talk about its zoning and tree, you know. James: Yeah, zoning and tree and all those. Brian: So it's a whole new world and it definitely is costly and time-consuming because you have to have experts on your team. You got to bring experts like architects. James: Yeah, we brought in architects, engineers. Brian: Yeah, engineers who even understand what it is that the city is asking for because if you were trying to do that yourself, you just would be a mess. James: Yeah. I mean the good thing about what you said about what I'm proud of this kind of process and 99% of the syndicators don't have that kind of experience. Brian: Yeah. I didn't have that kind of experience but now I do. James: Most of the time, you just buy buildings and, you know, look at increasing income and reducing expenses and after that, at some point you sell but you don't do different contracts buying land and doing kind of things. So another question for you, Brian, why do you do what you do? Brian: I love it. I love what I do. I feel very entrepreneurial about it because I've been an employee up until about five or six years ago. Whatever it was I was doing, whatever job, I always embraced it and did the best I could. But what I love about being an entrepreneur, being a full-time real estate investor, now syndicator/asset manager is that it's all very self-motivated. I'm the one who decides what needs to happen, what I need to pay attention to on a day-by-day basis. I don't have a boss or anyone else telling me, 'Hey, Brian, go do this' when I'm like, 'no, I want to go do this instead.' I get to call the shots. So that's what I love about it. I get to call the shots, I get to take time off if I need to take time off and I get to kind of fill my day with activities that I want to be doing. James: Awesome. Hey Brian, you want to tell our listeners and audience how to get hold of you? Brian: Sure, James. First of all, you can go to my website, which is higinvestor.com. That's HIG is Hamrick Investment Group. You can also listen to my podcast and James you've been a guest on there so you can definitely listen to me interview James. It's the Rental Property Owner and Real Estate Investor Podcast and it's sponsored by the RPOA, which we begin this conversation talking about. And if you want to get in touch with me, you can also email me Brian@higinvestor.com. James: Awesome, Brian. Thanks for coming in and adding value to my listeners and audience and to myself as well in the kind of things from our discussion here. I think that's it. Thank you very much. Brian: All right. Thanks, James. It's been a pleasure. It's a lot of fun. James: Lot of fun, thank you.

united states new york california texas los angeles secret michigan international oregon indiana matrix midwest san antonio investments fed amish santa monica institutions furniture grand rapids great recession real estate investors operators multifamily property management lansing fha fannie mae freddie mac amway class b asset classes middlebury james no real estate investing podcast james kandasamy brian it james it brian yeah gene chandler brian you james yeah brian so james yes brian no brian there james oh brian well james so brian hamrick james well brian yes brian right rental property owner james absolutely james okay

第630期:National Sport

Play Episode Listen Later Oct 6, 2019 2:44

更多英语知识，请关注微信公众号:VOA英语每日一听Fanny: Hey, Brian, what's the most popular sports in Canada?Brian: The most popular sport is definitely ice hockey.Fanny: Ice hockey! So do you play hockey by yourself?Brian: I don't actually. When I was a kid, I wanted to play ice hockey, and I was always like begging my dad and asking him but he always said 'NO'.Fanny: Why?Brian: I think the big reason is that... Well, he told me it was too expensive.Fanny: Is it?Brian: It's not cheap. You know, it costs quite a bit to get all the gear but the big reason I think is the practice was always very early in the morning.Fanny: Oh, I see.Brian: Like five a.m. is when the practice is, and I think he was too lazy to wake up and take me to the practice.Fanny: Oh, I see.Brian: He told me it was too expensive. Deep down I think he was.... he didn't want to drive me.Fanny: So are there many people playing hockey?Brian: There are. It's a great sport. It's very popular with many children, and maybe high schools and universities all have hockey teams.Fanny: Oh, nice. That means you're a lot of rich people in Canada, then.Brian: Or maybe they spend all of their money on hockey gear. Have you ever played hockey?Fanny: No, no, not really. It's not that popular in China.Brian: What kind of sports are more common in China?Fanny: People always play soccer...Brian: Ah, soccer.Fanny: And table tennis. Table tennis is very popular.Brian: You're country is very strong at table tennis I think.Fanny: We always get all the medals in the big, you know, big eventsBrian: Why is table tennis so popular now do you think?Fanny: I think the first reason is that everybody can play it because it's very easy to get the, you know, the... to get ready for the sports. It's not expensive.Brian: No, I guess you just need the ball and the paddle.Fanny: The paddle. The ball and the paddle. Yes, and a partner.Brian: Right. Right. So have you played it then?Fanny: Yeah, I'm quite good at it.Brian: Oh, really.Fanny: Because my mother plays very well and so I always played with my mom, so I got better now.Brian: OK. So she taught you how to play table tennis?Fanny: Actually she didn't teach me but we always played together.Brian: Right.Fanny: Practice makes perfect.Brian: So they sayFanny: Yeah.

canada china deep national table brian it brian you brian so brian there brian no brian oh brian right

第630期:National Sport

Play Episode Listen Later Oct 6, 2019 2:44

canada china deep national table brian it brian you brian so brian there brian no brian oh brian right

第629期:National Icons

Play Episode Listen Later Oct 5, 2019 2:07

更多英语知识，请关注微信公众号:VOA英语每日一听Fanny: Brian, my last question seems too serious.Brian: It was. It was a very academic kind of question.Fanny: Yeah.Brian: You're testing me,Fanny.Fanny: Sorry for that. I would also like to ask you some funny questions.Brian: OK, hit me.Fanny: Have you ever seen a polar bear?Brian: Have I seen a polar bear? Unfortunately, I have not.Fanny: No, really?Brian: I think you need to go really far to the north.Fanny: North. Yeah.Brian: Like, up around, like the North Pole maybe because I think the polar bears live like only on the ice, and this is like really far from any kind of like, you know, city or civilization, and I've never been up to like such a remote kind of place.Fanny: OK, I see.Brian: Unfortunately no polar bears. I've seen other bears, but no polar bears.Fanny: OK, me either.Brian: No polar bears in China?Fanny: I don't think so.Brian: No.Fanny: I don't think it's cold enough to have polar-bear there.Brian: How about Panda? I've heard there are some Pandas in China.Fanny: Yeah, I saw Panda for several times.Brian: In the wild or in a zoo?Fanny: In the zoo.Brian: Oh, OK.Fanny: And on TV. I just joking. I just saw some pandas by myself in the zoos, but I don't think they are the, you know, how do you say, because I think in the wild we can see the Pandas. We can see their activities more.Brian: Right.Fanny: How should I put that?Brian: It's more natural maybe.Fanny: Natural, yeah. It's very natural, but in the, you know, in the zoos the pandas are always sleeping. They're... or they're just eating something.Brian: Lazy animals.Fanny: No, the cannot do some, you know, outdoor activities.Brian: Right.Fanny: Poor pandas.Brian: It's a shame.Fanny: Yeah, it is.

tv china national panda icons north pole pandas brian it brian you brian how brian no brian oh brian right

第629期:National Icons

The ALPS In Brief Podcast

Play Episode Listen Later Oct 5, 2019 2:07

tv china national panda icons north pole pandas brian it brian you brian how brian no brian oh brian right

ALPS In Brief – Episode 37: Don't Mind Your Own Business

Play Episode Listen Later Sep 11, 2019 24:28

How can we make real changes within the practice of law to lessen the impact of stress on individuals in this profession? In this episode of the ALPS In Brief Podcast, Chris Newbold checks in with Dallas attorney and advocate for wellbeing, Brian Cuban, to discuss the state of lawyer wellbeing now, the lifesaving impact one lawyer can have upon another, and our ethical responsibility to step up for one another. CHRIS NEWBOLD: Good afternoon. This is Chris Newbold, guest hosting today for the ALPS In Brief Podcast. And I'm here in our offices in Missoula, Montana with attorney and advocate for wellbeing, Brian Cuban, who's in here from the Dallas area. I just spoke at our ALPS bar leaders retreat, and we thought this would be a great opportunity for us to have ... We have a similar passion in terms of seeing our profession improve on the wellbeing side, and so I thought this would be a great opportunity for us to just kind of have a conversation about where the profession's at. Where do we need to go? And Brian, you're obviously out on the speakers' network, kind of talking about this particular issue, your personal experience, and so forth. I think I'd like to start with just you kind of putting into your own frame of reference. What is the state of the profession right now when it comes to attorney wellbeing? BRIAN CUBAN: It's a state that is a lot better than it was a few years ago. We have much more awareness. We have many more engaged professionals from the bottom up, the lawyers, the bar professionals, the local bar professionals, the state bar professionals. And we have awareness in big law. We have awareness within the boutique and the solo practitioner. There are areas that we can certainly do better, and we can certainly be more impactful, but we are definitely light years ahead of we were just three years ago. CHRIS: And what do you think has driven that improvement in such a short period of time? BRIAN: I think you have to give a lot of the credit to the ABA and the Betty Ford Hazelden Report, and that would also be Patrick Krill, who authored that report, in bringing the issue to the forefront with the staggering statistics, because I think that was a catalyst in really changing the conversation. Whatever people think of the ABA, you have different opinions, but you can't deny that that report was a seminal moment. CHRIS: And why do you think that the issue right now is capturing a lot of attention in the legal community in legal circles? BRIAN: Well, because of that report and because of the cumulative awareness, now we are looking around us and actually noticing what's going on. We may have been aware of what's going on, we may have seen what's going on. When someone dies by suicide, we are aware of it and we grieve it. But we are now much better in taking a look at that, and deciding where things could've been done differently. And three years ago, four years ago, it was more about just grieving and handing out, in the issue of suicide, handing out the 1-800 hotlines. Now we are moving beyond that, and really look at how we can make systemic changes to at least lessen the odds of those things occurring. CHRIS: You talk a lot about kind of the impact that one lawyer can have on another lawyer. Right? And the responsibility that we have to not be kind of casual observers in this. Talk about that a little bit more as it relates to how we looked at, engineer a culture shift in the profession, and how every lawyer can make a difference one by one. BRIAN: Sure. I talk a lot about not minding your own business. We have to create a culture where we are comfortable, or even if we're not comfortable. Let me step back from that because that's not comfortable. It's okay to be uncomfortable not minding your own business. That's a human emotion. But we have to get comfortable understanding that for what it is and taking that step anyways. When we see someone struggling, when we think we might be able to, or we are wondering, you just don't know. Is there a drinking problem? Is there a mental health struggle? Maybe the person's just having a bad day. To be able to not mind our own business for one moment, step outside of our struggles, step outside of our busy day, our billing, the things we have going on, and say, "How are you doing? Are you doing okay? Do you know that if you're not, you can come to me, and we can talk?" That doesn't require anything but empathy. And every lawyer, every person has that ability. CHRIS: Is that a tough conversation for an associate to have with a partner? BRIAN: Absolutely. And we have to follow protocols. Law firms need to establish protocols for when people are struggling. That is not realistic to expect an associate to confront a partner. But big law all have EAPs, so there's that. We all have lawyers assistance programs. Do you know as an associate, you can call lawyers assistance program, and you can let someone know what's going on? And they're not going to out you. I know that is a tough pill to swallow, and I know you don't believe that. But you can make that call. You do not have to identify yourself in any lawyers assistance program in this country, and you can say, "I'm in this firm, and I think this guy is struggling." And they will take it from there, so you can do that. BRIAN: Within big law, we can talk about big law and then move on to something. Go down, go down. Within big law, it's important to establish protocols that are nonjudgmental, where everyone has a path. Everyone in the firm has a nonjudgmental path, a path that they feel safe voicing their concern if they see someone they think is struggling. So I can't tell them what that path is, but there should be multiple paths based on where someone is in the chain, right down to the clerk. CHRIS: Talk about your opinions on ... There's an increasing body of work out there that says that the economics of wellbeing are conducive to a stronger bottom line. Right? And as we think about talent acquisition, talent retention, I know you work a lot in kind of big law firms. Right? BRIAN: Mm-hmm (affirmative). CHRIS: I think there's a really interesting play on the horizon for those who lead our profession from a big law perspective to be thinking about a commitment to this issue that could translate economically for the firm. Talk about that. BRIAN: Absolutely. And I think, I doubt there are any managing partners, senior partners, firm CEOs are the real big ones that are not aware of that issue. It is the messaging is consistent just in general in society about the impact of addiction and mental health issues on the workplace and the economic cost. So the challenge becomes: How do we translate that into risk management? And I think they are starting to do that. That is not what I do. I'm a storyteller, I'm not a risk manager. But I think we are starting to see an industry, and people who do that, to go to a firm and say, "This is how we translate this into risk management to increase value to you," save you money. That saves the client money because on the most basic level, and we talked about the Peter Principle of Recovery. Right? How your level of competence keeps decreasing, and you keep trying to adjust your mindset to stay within that, you tell yourself you're at a high level when you're struggling. BRIAN: That can be, in a general sense, stealing money from a client because you were not effectively representing the client. That is affecting the firm's bottom line, and that is the most basic level. When a lawyer is struggling, and not functioning at the non-struggling level, he may not even, or she may not even understand what that level is because they've been in the middle of it, lacking self-awareness for so long. That is affecting the firm's bottom line. That can affect client retention because there are lawyers out there who are not struggling. Everyone's trying to get the business. Right? So you have to maximize the ... You have to minimize the risk by putting lawyers in a position to succeed and to hit the top level of competence and move beyond that if possible. Keep raising that level. And it's hard to do that when someone's struggling with addiction, problem drinking, depression. BRIAN: And I see lawyers all the time that talk about, well, I'm struggling with depression, but I was killing it, doing this. And I can't judge that. I don't know their situation. But I can say anecdotally, and what I see in the data, that I don't see how a person can look at the big picture, step back, and say, "I was going through all that and giving a dollar for a dollar." So I think all firms are aware of that, and I think that is achieved through a risk management model. CHRIS: Again, it's going to be interesting too as big law tries to recruit talent out of the law schools, how much top talented students are actually looking for a wellness play in terms of the life, work balance that I think, generationally, I think is becoming more common. BRIAN: That's a good question. I forget what the study was. Was it Am Law? Did the Am Law survey just come out? CHRIS: Mm-hmm (affirmative). BRIAN: And I couldn't find it. I think it may have been subsumed in one of the questions. But I reached out to Patrick Krill, who does a lot of the risk management stuff, and who authored the ABA Betty Ford Study, and asked him if he knew if we are surveying firms on wellbeing, if that is part of the survey. And I don't know that he had. I'll have to look and see if he responded, or he had an answer. But I think that may be not so much as a conscious play, but as a lifestyle play. It's just part of an overall lifestyle. Looking at the overall lifestyle, can we say that someone's going to say, "What's their drinking culture? I'm not going there"? There's no way to know that. But in the overall lifestyle play, I think lifestyle and wellness will become major factors, as Millennials and Generation Z, who have different priorities on what they want their life to look at as lawyers and as human beings. CHRIS: Yeah. Talk more about, it's an interesting time in our profession given the fact that we have four separate generations all operating at the same time. Right? BRIAN: Mm-hmm (affirmative). CHRIS: But there are also studies out there, particularly those that have been done within the law schools, that say some of these behaviors and substance abuse and so forth are starting earlier, and are becoming more prevalent for those who have been in practice, particularly in private practice, for less than 10 years. As you think about that dynamic, and Millennials and so forth, that's soon going to be the largest chunk of lawyers in the profession. And as you think about the generational aspects of wellbeing, what's your take on that? BRIAN: I think Millennials definitely have a different vision of what wellness looks like than ... I'm a baby boomer. The baby boomers, I come from, my lawyers' culture was a drinking culture. And I think when we look at things like the Sober Curious Movement, and what the Sober Curious Movement is, is not looking at drinking in terms of whether someone is a problem drinker, is an alcoholic, but what it looks like as a lifestyle, and as part of a healthy lifestyle, and whether you want it to be part of the healthy lifestyle without being judged on whether you're abstinent or not abstinent, and what that means to you, whether you're an alcoholic or you're not an alcoholic. I think Millennials and Generation Z are going to look at this differently in terms of just, I want to do the things that make me feel good, and that may not involve drinking. And I don't want to be judged for that. I don't want to have to explain myself. BRIAN: And I think that is going to be a much easier transition and a much easier conversation than it is for my generation because it's beginning. It is beginning. The Sober Curious Movement is out there. We have bars within New York. There aren't any in Dallas and Austin. And you see a lot of the progressive towns, where you have bars, they just serve mocktails. And they revolve the fun around other things besides getting drunk. You go out and you're drinking fake pina coladas without alcohol. And they revolve everything around those, around the mocktails. The mocktail generation, they may be that. CHRIS: That's an interesting one. If you had to assess right now, wellbeing in the legal profession, one being it's at an all-time low, 10 being, I think lawyers are both healthy, happy, engaged, where you put that on the spectrum? BRIAN: I would put we're at a three or four, three or four. And that's great, and that's great. CHRIS: A lot of room for improvement. BRIAN: A lot of room for improvement. Four is opportunity. Right? CHRIS: Yep. BRIAN: Four is opportunity. Yes. And one of the biggest challenges I think we have, and if you look at big law, we with the ABA, and this isn't a criticism of the ABA at all. I think with the Wellness Task Force and everything, they have laid out the groundwork for all levels to participate, all stakeholders, solo, medium, boutique, the bar associations, all the way up to big law, corporate. I think they are laying out that groundwork. But I think when we get further down into the stakeholders, the solo practitioner, the small firm, we have a lot more work to do. And I think in that chunk is where we have the most improvement to do in our messaging, and the most opportunity because we have other challenges when we get down there. BRIAN: If you work at big law, you have health insurance. And I knew big law lawyers who have health insurance, and still can't find a reasonable psychiatrist or therapist. They've complained to me about it. We have this health insurance crisis on so many different levels. And big law within the spectrum, you have privilege. You have health insurance privilege because you're going to have it. And you're going to have the EAP, and you're going to have this, and you're going to have that. BRIAN: I don't know what the stats are, but I know anecdotally that a lot of the solos cannot afford health insurance. So when you can't afford health insurance, what are your options? You're going to 12 step. You are going to county. A lawyer don't want to go to county and get free treatment, that's very shameful. Right? If you even have that option as a reasonable option in your city. A lot of cities have terrible county free health services. And so we have that stigma of a solo practitioner and the medium, I don't have health insurance. I'm a lawyer, I'm not taking advantage of free. I can't. So they don't tell anyone. It's shameful. So how do we solve that? CHRIS: Obviously, in our book of business with ALPS, we specialize in small firms and solo practitioners. And 65% of the policies that we issue are to solos. And they're generally a higher malpractice risk because they don't have a support network around them. BRIAN: Absolutely. CHRIS: You can't stop into Brian's office and say, "Hey. Let's have a conversation about this particular case." You have to build networks. You have to build connections in very different ways, which makes it I think, much more challenging. BRIAN: And it does. And it's a challenge where you're struggling. It's going to be dependent on the particular situation. But you're making what would be decent money, you have a family. You can barely, after everything, then you care barely support your family. And you're more able to speak to this. You have a deductible that you can't meet anyways, even though you have health insurance. That's as almost as being uninsured. So we have all of those issues, and I don't know what the solution is to that. But that is one of the things that is a huge barrier to wellness within the profession, health insurance and the ability to pay for getting well, the ability to find people to get us well. We are becoming a cash only society in terms of wellness. BRIAN: I consider myself very lucky because I have a psychiatrist, I've been seeing for 15 years, and he treats. I have one of the few treating psychiatrists out there with his therapy. But we also have the ghost networks that you may be familiar with. And I'm getting off on tangents, where you can't, even if you have health insurance, you can't find a treatment provider because they don't take insurance. CHRIS: Where do we go? A lot of good activity now happening. You've got Pledge. You've got some state task forces going. Got a lot of discussion. Societally, we're seeing more vulnerability to talk about these issues, whether it's Hollywood stars, or sports stars, there's just more discussion, which I think is healthy. If we're a three or four right now, how do we get to a six or seven? How do we start to move the needle? Culture shifts in any society- BRIAN: It's one person at a time. It's one person at a time. If you're talking, there's no magic pill to culture shift. We talked about this. It is one person at a time. There's one bar association at a time. There's one law firm at a time. And you hope, you hope, that the Malcolm Gladwell theories kick in, and you hit a tipping point. But it is much more, again, it is much more on different levels societal. If I can't afford treatment, what's the difference what the path is if I can't get there? Why should I tell anyone if I can't afford to get there? In Texas, we have a fund where if you go to them, a lawyer can get treatment. I believe it's an endowed fund privately. And maybe someone will correct me on this when they listen to it. But we have to find different ways to ... It's more than just laying the path. People have to be able to walk on it. BRIAN: And if you can't afford to get the help, other than 12 step, and 12 step is great, Smart Recovery's great, Refuge Recovery is great, but they're all mutual aid. Mutual aid is not treatment. Mutual aid is maintaining connection, which is important. If you can't afford the treatment, and you have no way through that path, that's a huge problem that goes beyond the legal profession. When we talk about the legal profession, what we can do, I think we have to have a more societal view of that. How do we correct that? CHRIS: Yeah. There's an interconnectedness to a lot of different- BRIAN: You can't sever this. You can't sever out health insurance accessibility from all the other issues within the profession because most of the profession is solo and small. CHRIS: And even on a tangent, one of the reasons I got involved in the wellbeing movement was I feel like there is a gap in expectations for what people think practicing law will be like, and ultimately what they find that it's going to be like, whether that happens in law school, or whether that happens because of law school debt. That again, to be a good lawyer, one has to be a healthy lawyer. And more and more, people are finding themselves boxed into a spot where they're actually doing something that they're not finding professional satisfaction in, which is then causing ... It can cause other things to kind of spin off from there. BRIAN: I agree. I agree. Every lawyer is a story. Every lawyer is more than just the person under stress. Every lawyer brings their entire history of trauma, of however they grew up, family. CHRIS: Family. BRIAN: They bring it all through the door of that firm. They bring it all to the courthouse. So whatever that stress is may not just be the product of what's going on at that moment, the case, fulfilled expectations, unfulfilled expectations. It may be the product of a life story that has shaped someone that made them more susceptible to those issues. Does that make sense? CHRIS: It does. BRIAN: So we have to address the story and not just the moment that the lawyer is in. CHRIS: Yeah. Anything else that you want to kind of relay as we talk to our policy holders and other interested listeners about just kind of the current state of attorney wellbeing? BRIAN: If we want to change the paradigm of attorney wellbeing, for me personally, I think the most powerful tool is continue to encourage people to tell their stories. Keep telling the stories. Everyone identifies with aspects of other people's lives. There's going to be something to identify with. The connections, stories bring connection. Keep bringing people in to tell stories. Just encourage that. And I think through the power of storytelling, we will start to see more and more people tell their stories, and then they'll tell their stories. And I think that is how. CHRIS: That reduces stigma. That reduces vulnerability. BRIAN: That's right. I think as we reduce stigma, we will better empower lawyers to seek recovery. CHRIS: Yeah. Brian, thank you. BRIAN: Thank you. CHRIS: We appreciate your time, and we appreciate your perspectives. And obviously, you're doing wonderful work in the storytelling side of the ledger because it's important that through the experiences of you and telling your personal story that it makes a difference. BRIAN: I think law firms need to realize, and I think big firms are starting to do this, is creating a wellness program has different levels. There's storytelling. There is risk management. There is- CHRIS: Scientific studies. BRIAN: Yes. And there is the pure wellness aspect. How do we reduce stress? How do we become happier? What can we do to allow our lawyers, within the framework of our representation of clients, to feel better about themselves and what they do? Law firms are in a business. This is a business, and they are not yogis. We have to be realistic. Law firms are there to represent clients at the highest level possible. What holes do we need to fill to make that happen? Because that is what we do. We represent clients. And so we have to fill all these different gaps, the storytelling gap, the risk management gap, the wellness gap. CHRIS: Got it. Again, thank you so much. And I hope you enjoyed listening to this podcast. As you know, ALPS is committed to being a leader in the wellbeing issues of the day affecting the legal profession. We hope you enjoyed this podcast. If you have any other ideas for topics on the wellbeing, please let us know. Thank you. Brian Cuban, the younger brother of Dallas Mavericks owner and entrepreneur Mark Cuban, is a Dallas based attorney, author and addiction recovery advocate. He is graduate of Penn State University and The University of Pittsburgh School of Law. Brian has been in long term recovery from alcohol, cocaine and bulimia since April of 2007. His first book, Shattered Image: My Triumph Over Body Dysmorphic Disorder,” chronicles his first-hand experiences living with, and recovering from, twenty-seven years of eating disorders, and Body Dysmorphic Disorder (BDD). Brian's most recent, best-selling book, The Addicted Lawyer, Tales of The Bar, Booze, Blow, & Redemption is an un-flinching look back at how addiction and other mental health issues destroyed his career as a once successful lawyer and how he and others in the profession redefined their lives in recovery and found redemption. Brian has spoken at colleges, universities, conferences, non-profit and legal events across the United States and in Canada. Brian has appeared on prestigious talks shows such as the Katie Couric Show as well as numerous media outlets around the country. He also writes extensively on these subjects. His columns have appeared and he has been quoted on these topics on CNN.com, Foxnews.com, The Huffington Post, Above The Law, The New York Times, and in online and print newspapers around the world. Learn more at www.briancuban.com.

united states new york university texas canada culture hollywood talk law new york times recovery tales millennials cnn redemption ceos montana bar fox news huffington post blow dallas mavericks generation z booze mark cuban pledge mutual malcolm gladwell alps penn state university aba missoula eap mind your own business pittsburgh school peter principle amlaw smart recovery above the law brian cuban eaps chris talk refuge recovery body dysmorphic disorder bdd chris yeah chris you brian it brian you chris is brian so societally brian well addicted lawyer brian yes chris where patrick krill

017 - John Cutler on Productizing Storytelling Measuring What Matters & Analytics Product Management

Play Episode Listen Later Jul 16, 2019 48:29

John Cutler is a Product Evangelist for Amplitude, an analytic platform that helps companies better understand users behavior, helping to grow their businesses. John focuses on user experience and evidence-driven product development by mixing and matching various methodologies to help teams deliver lasting outcomes for their customers. As a former UX researcher at AppFolio, a product manager at Zendesk, Pendo.io, AdKeeper and RichFX, a startup founder, and a product team coach, John has a perspective that spans individual roles, domains, and products. In today’s episode, John and I discuss how productizing storytelling in analytics applications can be a powerful tool for moving analytics beyond vanity metrics. We also covered the importance of understanding customers’ jobs/tasks, involving cross-disciplinary teams when creating a product/service, and: John and Amplitude’s North Star strategy and the (3) measurements they care about when tracking their own customers’ success Why John loves the concept of analytics “notebooks” (also a particular feature of Amplitude’s product) vs. the standard dashboard method Understanding relationships between metrics through “weekly learning users” who share digestible content John’s opinions on involving domain experts and cross-discipline teams to enable products focused on outcomes over features Recognizing whether your product/app is about explanatory or exploratory analytics How Jazz relates to business – how you don’t know what you don’t know yet Resources and Links: Connect with John on LinkedIn Follow John on Twitter Keep up with John on Medium Amplitude Designing for Analytics Quotes from Today’s Episode “It’s like you know in your heart you should pair with domain experts and people who know the human problem out there and understand the decisions being made. I think organizationally, there’s a lot of organizational inertia that discourages that, unfortunately, and so you need to fight for it. My advice is to fight for it because you know that that’s important and you know that this is not just a pure data science problem or a pure analytics problem. There’s probably there’s a lot of surrounding information that you need to understand to be able to actually help the business.” – John “We definitely ‘dogfood’ our product and we also ‘dogfood’ the advice we give our customers.” – John “You know in your heart you should pair with domain experts and people who know the human problem out there and understand the decisions being made. […] there’s a lot of organizational inertia that discourages that, unfortunately, and so you need to fight for it. I guess my advice is, fight for it, because you know that it is important, and you know that this is not just a pure data science problem or a pure analytics problem.” – John “It’s very easy to create assets and create code and things that look like progress. They mask themselves as progress and improvement, and they may not actually return any business value or customer value explicitly. We have to consciously know what the outcomes are that we want.” – Brian “We got to get the right bodies in the room that know the right questions to ask. I can smell when the right questions aren’t being asked, and it’s so powerful” – Brian “Instead of thinking about what are all the right stats to consider, [I sometimes suggest teams] write in plain English, like in prose format, what would be the value that we could possibly show in the data.’ maybe it can’t even technically be achieved today. But expressing the analytics in words like, ‘you should change this knob to seven instead of nine because we found out X, Y, and Z happened. We also think blah, blah, blah, blah, blah, and here is how we know that, and there’s your recommendation.’ This method is highly prescriptive, but it’s an exercise in thinking about the customer’s experience.” – Brian Transcript Brian: My guest today on Experiencing Data is John Cutler who is a product evangelist at Amplitude Software. I have been really enjoying John’s commentary on Twitter and some of his articles on medium about designing better decisions of work tools. If you’re in this space and you’re trying to figure out, “How do I get into the heads of what our customers need? What types of data is actually important to track?” Especially, if you’re looking at longer term outcomes that you want to be able to measure and provide insight on, I think you’re going to enjoy my conversation with John. Without further ado, here’s my chat with John Cutler. All right, we’re back to Experiencing Data, and today we’ve got the cutlefish as your Twitter handle is known, right is it cute-l-fish or cutlefish? John: We’re going to go with cutlefish, not cute-l. Brian: That’s what I thought. John Cutler is here from Amplitude Software, which is a product analytics company, and I wanted to have John on today, not because he is cute necessarily, but because I’ve really been enjoying what you’re espousing about customer experience, and particularly, product management. Which for some of our listeners that are not working in tech companies necessarily, there’s not really a product management kind of role explicitly by title. But I think some of the, as you will probably account to, the overlap between design, user experience, and product is sometimes a gray area. I think some of the things you’re talking about are in important in the context of building analytics tools. Welcome to the show, fill in, make corrections on what I just said about what you’re doing. You’re a product evangelist at Amplitude, so what does that mean and what are you up to over there? John: Well, we’re still trying to figure out the evangelist part because I don’t necessarily sell or evangelize our product, I think our product is great and I like to say it sort of sells itself. But what I’m really focusing on is helping up level teams, now that could be like our internal teams, our customers, but largely to just prospects and teams that have never even heard of Amplitude. What we’re really looking with this role is to do workshops, provide content, I do these coaching sessions with just random teams, so it’s like one hour coaching sessions. But generally trying to fill in the blanks, I think a lot of times people think, “Well, I’m just going to purchase this analytics tool or this product analytics tool,” and suddenly it’s going to answer all our questions and everything’s going to be fine. But what they don’t quite realize is that you really have to tweak a lot of things about how you work as a product development team to really make use of the great tools that are available. There are amazing tools available. I believe Amplitude is one of them, but there is so many good software as a service products to help product teams. But really at the end of the day, it’s about the team also being aligned and things like that. I really try to take a broad view of what it will take to help people make better products with this role. Brian: Yeah. Can you give an example? I think I know where you’re going with this, but give an example of where someone had to change their expectation? You need to change the way you’re working or let’s figure out what’s important to measure instead of just expecting. I think you’re alluding to like, “Oh, buy our tool, we know what the important analytics and measurement points are that you should care about and we will unveil them.” Instead it’s like, “Well, what’s important to track? Does time on the site matter? Does engagement in the application matter? Does sharing matter? What matters, right?” Can you talk about maybe where there was a learning experience? John: Oh, absolutely. I think maybe a good way to describe this as well is a lot of the learning, a lot of the questions begin way before the team is unwrapping the problem, unraveling the problem. I’m not sure this answers your question exactly but I think we could lead into something more specific. But imagine you’re a team and someone says, “It’s the second half of 2018, what’s going to be on your roadmap?” You think about it and you know what you know and you’ve heard customers tell you things, and the CEO of the company has subtly but not so subtly hinted he’d really like to see X or she’d really like to see X. You put together this roadmap, and at that point once you’ve got people thinking that those solutions are the right solutions, and you force that level of convergence, there’s not a lot of… measurement will not save you at that point, you’ve already committed at that point to deliver those things in that particular setting. One example of a practice that might change to further or amplify the use of measurement would just be not making… committing to missions, committing to move particular metrics that the company believes are associated with mid to long-term growth of the company, and commit to those things instead of committing to build features. An example, a real world example, maybe for someone’s effort, maybe what you’re shooting at before is do they shift from same time on site was important to something else? But for a lot of these teams, it’s shifting from build feature X to something like shortening the time it takes for a team to be able to complete a workflow. That’s the big shift for that. It’s nothing-to-something that makes sense, not necessarily even something-to-something. Brian: One of the things we talk about on the show is designing for outcomes instead of designing outputs. John: Yep. Brian: Because it’s very easy to create assets and create code and things that look like progress. They mask themselves as progress and improvement, and they may not actually return any business value or customer value explicitly. We have to consciously know what the outcomes are that we want let alone measure them. Do you run into the problem when you… If you’re coaching someone and getting them into this mindset of designing around an outcome and building your sprint or your next, maybe it’s even a strategy for the next six to 12 months around outcomes? That the important things to measure are not quantifiable in the tool? Do you work yourself out of a customer sometimes because the tool can’t actually measure what’s important? Does that ever happen? John: That’s a great question because I think that I do a fun exercise with people, which is called let’s predict the success of a relationship. We start with this activity and we just we forget about what we think is possible to measure and we just start mapping our beliefs. The team will say something like, “Well, I think that they shouldn’t have arguments.” Then someone will say, “Well, yeah, but it’s not just,” and maybe they’re talking about their own life like, “Well, we argue a lot, but we resolve our arguments pretty, we become stronger once we have the arguments.” Then the team will sit there and go, “Huh, okay.” It’s not just about the number of arguments, it’s ability to resolve your arguments. Brian: Resolve. John: We keep playing this game and we map our beliefs out to predicting these things, and some of these things we have more confidence about and some of these things we don’t have a lot of confidence about. Some of these things we strike and we get this big messy network of nodes and edges on the wall and that’s what we start working with. What’s really, really interesting is that we actually, as a company, there’s almost always some percentage of these things that we can contribute to in terms of what they can instrument in using our product. It’s not like…we would much rather our customers map the universe of things and acknowledge some things that might be difficult to measure or they’re just beliefs at the moment, they haven’t figured out how to measure them. Because really what Amplitude is very powerful at is doing behavioral analytics about these long standing customer journeys through products and those types of… Anyone who’s done a 15-table join and tried to communicate it to other people in your company and then tweak it and have people collaborate with it just knows how painful that is. That’s the type of pain that we solve. But back to the particular question, all the coaching really centers around mapping all the beliefs, and we’re usually confident that there are ways to measure some percentage of those things using our product, and that’s fine by us. Brian: There’s almost like a meta-question, right? John: I like, I’m meta, yeah, I got it. I’m there with you. Brian: You’re like analytics, you’re an analytics product and you talk to your clients about what’s important for them to measure. But then at some point, you have to know what’s important to measure to know that your customers are getting the value. John: Yeah. Brian: Is it directly…are you interested in what they’re setting up to measure and then that becomes your measurement? Do you piggyback off that or do you… How do you justify that the sprint or the epic we worked on last quarter provided business value? How do you…? John: Yeah, that’s amazing. Yeah, we definitely dogfood our product and we also dogfood the advice we give people usually first. To give you an example like in 2018, we had this North Star Metric called “Weekly Querying Users”, WQUs. That seemed about right and we did some analysis and it looked like, “Well, for increasing WQUs, it’s probably going to mean this and this and it’s going to be some early indicator that our monthly recurring revenue is going to keep going up”, etc. But there were obvious problems with that and we saw that. And as 2018 went along, we started to look at it more, and for any SaaS company, there’s a point at which your expansion within existing accounts starts to be really, really important in terms of percentage of revenue that you’re in. We thought, “Well, is that metric, can you hand WQUs to any new team member and say move that or move something that you think moves that,” and then be 100% confident they’re going to make good decisions? It broke down after that. What we did is we shifted to weekly learning users. Now a weekly learning user is not just someone querying, because anyone who uses one of these tools knows you could just sit there and query all day and not get an answer. In fact, querying more might indicate that you are not getting an answer. Not like doing anything with it. A weekly learning user is actually someone who shares some piece of digestible content whether it’s notebook, whether it’s a dashboard, whether it’s a chart, and they share it. We actually have this North Star, which is weekly learning users, we believe these three inputs drive weekly learning users and those are activated accounts. They need to know what they’re doing, they’re broadcasted learnings, which is the ability for the user to attempt to broadcast some number of learnings, and then a metric that is a consumption of learning metric which is the broad consumption across the organization of that particular piece of learning. This is all sounds really heady, why would we go to all these lengths to do this, and Weekly Querying User sounded good. But to us this really encapsulates a strategy. I think that that’s an important thing that a lot of people from pure analytics backgrounds or who are used to sitting with a queue of questions and answering those questions are maybe not used to the idea of moving towards a cohesive strategy as expressed by a number of metrics and the relationships between those metrics. That’s something that we really encourage our customers to do, it’s not data snacking. It’s not like, “Oh, I got this itch today so I’m going to answer this question.” That took a lot of work to come up with that, and we’re confident about those relationships between those things. But more importantly, it helps any new team member like all you need to do is show a skilled product manager or a skilled designer or a developer even and say, “This is our current mental model as described by the relationship between these things. Where do you want to slot in? What do you have in mind?” That’s really, really powerful. I don’t know if that roundabout way of saying we take this really, really seriously. Brian: If I can sum this up, and I’ll need you to repeat part of it, but you have monthly querying users, so what I take that to be is I, the customer, using, paying for the Amplitude software, a querying user means I went in and I looked for content or I literally used a search interface to probably look up an analytic or some stat. You moved away from the number of people doing that and how often they’re doing it as a measurement of your company’s success to this three-stage kind of thing that I heard included sharing some knowledge. But can you repeat what those three grains were? John: Oh, yeah, sure. The North Star is what we call “Weekly Learning Users”, so WLUs. Those are users performing the behavior of interest, which is sharing, distributing some piece of content. Then we believe there are three inputs that explain that metric or three inputs that we really focus on. One is that the accounts are activated, which are meaning that does this account just have a minimum number of people doing that? The next one is broadcasted learnings, which is me, “is the initial attempt to broadcast the learning?” Then consumption is the actual long tail consumption of that particular learning. Let’s say it is a story like I sign up with Amplitude, no one’s really using it all because we haven’t really onboarded and we haven’t really instrumented, we haven’t done any of that stuff. Okay, well, then we get that done, so we get just that we’ve activated, we have at least a certain number of users learning, some amount. I’m in the tool, I’m in a notebook that is really interesting that I’m putting together that tells a story with data, very interesting about the mission that I’m working on. I attempt to invite people to that notebook or get them involved, that’s the broadcast. Then, finally, the consumption of learning would be the accumulated interactions with everyone with a notebook. If that sounds too complex… Brian: Got it. I don’t know, I- John: But the whole idea is for people listening and I think especially folks, designers and other folks is that their experience with analytics might be something very simple like “what percentage of people used feature?” Or something. What they’re not getting is the context, the relationships, and what I’m describing here, there’s amazing belief networks, there’s causal relationship diagrams, there’s just simple stickies and string on the wall, whatever you want to call them. But we’re describing our beliefs as it relates to the data, and I think that, that’s really important. For some background too, I’m not a data scientist, I’ve been a product manager and a UX researcher and that’s been my focus for a long time. It’s not like I’m a pro at this stuff, and even for me, though, it grounds me in what I’m working with and makes my analysis a lot easier. Brian: I imagine you may have some, not resistance, but when you’re working with quote data people or analytics people or data science people in your staff, in Amplitude, are there routine things that you wish they would hear that would sink in or problems that maybe they’re not aware of that you think they should be like, “We need to look at the problem differently.” Maybe you encapsulated that and that’s why you have this three stage thing as a reaction to the data snacking mentality, which is “What data do we provide? Great, they have it, now they can eat it.” Is that their reaction to that or are there other things that… I’m thinking of our listeners, we do have data scientists and analytics type people, and I’m wondering if you were to work with them, it’s like, “Here are the things that I want you to think about here to get our head a little bit out of the tech for a second and into the decision support mentality.” Anything, what would you espouse or advocate? John: That’s a great question. I think I can answer it a little bit with a story. I was the PM for search and relevance at Zendesk, like support software. My background is not in information retrieval or the guts of search but very, very early on working on a team with very, very talented people, data scientists, data engineers really, at the end of the day. One thing that I very much advocated for is we needed to be able to get everyone in the same room, we needed to get the people who were experts in what I would just call the actors, the support agents, or the support managers, or the the person trying to get help on their Uber app. There’s experts in that, there’s domain experts. There’s also people who are experts in the surface area, the surface, like the interface. There’s people who are really, really good at searching or finding information on mobile. There’s people who’s very good at finding information on, in our case, like the support agents view in their web browser. Then you had our people who are really smart and creating data as it related to search and they were great at data engineering, etc. The main thing that I noticed was that there’s just a silo-ing, and the people on my team were just craving, craving to be sitting next to someone who understood these other things really well. I think that for a lot of listeners it’s probably you know that, you know that from a first principles angle, you’re like, “Well, I know that there’s a bigger picture here.” I know that just in our case of searching like we knew that raising the mean reciprocal rank of a search term, we are searching it, where does the person click? Do they click on the second item, the fifth item. In theory, raising that would make a difference but when we look more broadly, it really didn’t relate to deflection of tickets and things like that. Our traditional metrics, the way we were measuring success is locally related to search. If we broadened our horizon to what makes a difference for the human beings out there who need their support tickets resolved or the support agents or things, that perspective was so helpful. What I would say to the folks on listening, it’s like you know in your heart you should pair with domain experts and people who know the human problem out there and understand the decisions being made. I think, organizationally, there’s a lot of organizational inertia that discourages that, unfortunately, and so you need to fight for it. I guess my advice is fight for it because you know that that’s important and you know that this is not just a pure data science problem or a pure analytics problem. There’s probably there’s a lot of surrounding information that you need to understand to be able to actually help the business. Brian: Sure, and you’re echoing sentiment I had a Data Center from the Broad Institute on, he was mentioning how much he’s like, “My work is so much more powerful when I have a great domain expert with me who really knows the space.” We met over music, I’m a musician as well and he was trying to explore creativity in the context of jazz. He’s a enthusiast in terms of music, he’s not a musician, but he’s an enthusiast so he understood some of it but he didn’t have the lingo. It’s just interesting when you look at someone working in that space trying to answer a question about like, “How does creativity work in jazz?” They don’t have all that domain lingo. Being on for a change, being the domain expert, it was fascinating for me to be on the other side because usually I’m the hymn advocate, even though I’m not a data scientist, as a designer and a consultant, we deal with this all the time. It’s like, we got to get the right bodies in the room that know the right questions to ask. I can smell when the right questions aren’t being asked and it’s so powerful so I totally agree with you on the need to provide that bigger context sometimes so you don’t just- John: Jazz is just a mistake played more than once, right? Brian: Yeah. Oh, there’s tons of them, there’s no wrong notes, just bad choices. John: It’s very easy for them to create the model for that. You’re just making a mistake and play it more than once. Brian: Exactly. John: Then you go back to the top. Brian: Exactly. Well, even that, like play the head again. Well, what’s a head? Oh, okay. Well, it’s just one form of the tune and they cycle through it and play chorus. Well, what’s a chorus? Okay, shit. But even having that, you can imagine that on the business client, this was like a fun side project he was working on. But you can imagine that in a business context where you don’t even know what you don’t know yet about it yet. I hear this as happening, they’re still in the, especially, in the non-tech company space, the more traditional companies that are, “Oh, we have 100 years of data and let’s go, we need to go buy some data scientists and throw them at this pile of data and then magic will come out the other end.” John: Oh, I think that that happens in tech companies, too, though. I think that that’s the number of data scientist friends who’ve been hired in is like some large effort. Then, one, they’re like, “Yeah, and data engineering was the actual problem.” Okay, we spent our first year there just going around in circles on solving that problem, and then, yeah, the number of friends I have who’ve been frustrated by that dynamic, even in tech companies, I think it’s a pretty common, more common everywhere than we would think. Brian: Tell me a little bit about, so we’ve been talking about the analytical part of all this, the quantifiable parts largely but you have a UX research background as well. We talk, on this show, we talk about empathy, we talk about the needs to go talk to people to ask good questions, to ladder up, get into all that. How does that fit in? When you’re working on an analytics tool, can you fill us in on your approach to qualitative research and more the soft, mushy stuff that UX people deal with? John: Yeah, and it’s interesting. For context, I’m not a UX researcher at Amplitude but I’ve done that in prior environments that required the chops. But in talking to teams and doing it, I think so many of the basics apply in the sense that you’re really… Not to overuse the jobs-to-be-done stuff, you’re really, really trying to understand what decision this person is hoping to make. You’re really trying and then what impact that decision has on the rest of the organization and who is involved in it. I think anyone who’s done this knows that even as a UX researcher, if I do like a co-design activity with customers related to anything analytics oriented, it’s just, “Oh, we’re going to do an Excel mock up or you know.” Anytime you get customers involved with that, it’s so easy. If either side, and I’ve been on both sides of this, it’s so easy to forget what you were trying to do. I think that has a lot to do with the exploratory aspect of data in general that we have a gut instinct that if we just saw this stuff organized like this, then it would somehow be valuable for something we have to do. I think that for, and I don’t know if it answers the question, but I think it requires the same chops but also understanding that people just have a hard time, users have a hard time talking about what they are looking at and what they’re hoping to get out of the data when they’re looking at it over and over and over. I think that really, it really you have to use all the tools in the tool shed. To give you an example, there was… I don’t know if you’ve done these things too, I’ll do these exercises where it’s like, “Okay, we’re revamping the app, it’s just going to be this mobile browser with three numbers on it.” That’s it, that we’re not going to have all these fancy charts, we’re not going to have all this stuff. And three numbers and then one piece of narrative advice, like “Consider this or do this.” I love activities like that from a UX researcher standpoint when I’m working with people because it really, really forces them to just get out of their own head to think about it. That’s like a common trick and you probably have a lot more. But, yeah, I don’t know if I answered the question but it’s a lot of the same tools. But I think also you have to really… It’s a job environment, they’re making decisions, they’re hiring these analytics to do a job. But then with this added layer that I think that people are just incredibly, they find it incredibly difficult to talk about the numbers that they’re looking for. Brian: When you say it’s difficult for them to talk about it, are you talking about their digestion of what’s on the screen or their expression of what’s important to them to actually find out? What do I actually want to learn about? Is it… John: Both really, and that’s the thing that I think just makes it doubly as hard. It means that if you show them something, and I think that we can all relate to it too, like any of us who have been shown some mock or some prototype of information on the screen, you can see your gears turning. You’re having to process it and where did this come from? Can I trust it? What is it? We see that all the time just in Amplitude, it’s people… Our understanding of how people experience some of these querying screens that we have, when you actually ask them to just talk through what they’re thinking about as they move through it, it’s just it’s so complex. Data trust, where is this stuff coming from, data over time, their challenges with certain visualization techniques, even if it’s “the right technique” like, “Well, I just need a radar chart.” Just like no you don’t really. But that’s how they’ve been anchored or whatever. It’s just complex. I don’t have a fancy answer, it’s just complex. Brian: What you just told me reminds me of you had mentioned you do this exercise, and I’m wondering if it’s the same exercise that I’ve done as well with analytics tools, especially, in the context of monitoring applications. There’s some system that’s monitoring stuff and it’s supposed to advise you on what should I do next or what happens with something like this? It’s like “instead of thinking about what are all the right stats to do”, it’s “write in plain English like a prose format what would be the value that we could possibly show”, and maybe we can’t even technically do it today. But it’s “express the analytics and words like you should change this knob to seven instead of nine because we found out X, Y, and Z happened. We also think blah, blah, blah, blah, blah, and this is how we know that, and there’s your recommendation.” It’s highly prescriptive but it’s an exercise in thinking about the customer’s experience. How close to that can we get to it, where I don’t have to infer from charts or whatever the date of this format is, how close could we get to something that prescriptive and then try to work backwards from that. We probably can’t get right to that full prose. Is it something like that where you jump to this conclusion, like value conclusion or something like that? John: Yeah, and I do a couple of these like that, one is if I have an Alexa or if I have a tube of crackers or whatever I’m like, “This is the interface now.” You can ask Alexa, that’s your interface. This is a beautiful future world where you just have your smart person, your smart assistant to do these things. Yeah, similar type of, I think, what it does is it creates just enough dissonance to snap people out of just immediately trying to unravel the visualization, which can be I think all of us do that. I think that that’s our instinct whenever we look at something like that. Brian: The default next question is how should we visualize this data that we’ve captured? That’s the itch that we may not be the one to scratch? John: Yeah, but I think that’s also what we can test with, that point, when we’ve got that need to fill, that’s when we can try multiple approaches, I think to see that. That’s my experience, there is that point at which you need to you go back to the drawing board. Although, I would say that depending on the subject, the user in that case or the person you’re working with, some people are really, really good at just the co-design aspect. I don’t know your experience with that, but it seems to have a lot to do with what the people do each day and how they think about visualization and stuff. But I’ve done co-design sessions with people who the next step was, “Well, let’s start thinking about, let’s start drawing, let’s start doing some other things to do it.” I think that depends a lot on the background of the people that you’re working with. Brian: If you were starting over today with Amplitude, is there either a… Not necessarily a feature you would change but is there something that you would approach differently? If someone says, “Hey, we have this JavaScript widget, you paste it in your, all your app or whatever, and we can track almost anything, any activity, whatever. What should we show?” Is there something you would change about maybe how you guys went, the process you went about arriving at the current product that you have? John: That’s interesting. I wish Spencer and Jeffrey were here to answer because they’re the founders of the company. But I think that it’s funny how products have their history about them, so Amplitude, for example, it was a Y Combinator. The founders didn’t go to Y Combinator, they had this fancy voice app or something that they were working on, and this was actually just their effort. They were like, “Well, we kind of had this app,” and they surveyed what was available and then just said, “We really need, there’s a thing, it’s a little different. It’s like an event based measurement thing. We really want to instrument this app and know whether people are using it or not.” That was the founding story, it wasn’t their key thing. A lot of the early customers were folks from Zynga or Facebook or other places that had moved on to other startups and then they wanted something that helped with the 90% of product questions that they had around retention and engagement and complex behavior patterns. Does this behavior predict this or is there a relationship between these things? That’s the founding story, these discerning teams that had a fair amount of autonomy and were tasked with working in these environments and that they wanted a product that they could do that with. When I’m thinking about what I would change as the newcomer to the company, now maybe five years on, was it, yeah, or six years or seven years on, I think it’s what they’re starting to do now, which is interesting. This notebooks feature to me is just so, so, so good and it gets away from a traditional dashboard. But with a notebook, it’s very similar to a data science notebook, you can weave this story and this narrative and you can make the charts live and you can communicate it and you can do those things. As a product manager, that is pure gold to me, and it’s just we’ve started to do those things. I think that the answer would be more of what they’re really digging into now, which is around this learning user concept and how do you create stories with the data to motivate your team and keep everyone aligned and things. I think if it hadn’t existed and I joined a year ago, I would have been like, “Oh, you’re missing this little element like the actual part that integrates it into day to day product development.” But they’ve just started doing that now, so they stole my answer. Brian: Nice, and just for people that don’t know, tell me if I got this right, but the notebooks for people that aren’t data scientist, it’s effectively a collection of both quant data like maybe charts or tables, stats, data collection that you guys have put into some visualized widgets or whatever it may be insights plus qualitative stuff like my commentary on it. Like “Why do we care about this? Well, design is currently tracking these metrics because we’re running a study on dah, dah, dah, and we think we can move this up” and that’s a proxy for this other thing. You can provide all this context in that storytelling mentality so that when someone new comes in, they’re like, “Why do I care about time on site or whatever the metric?” John: Exactly, and that’s the huge thing. One thing that we learned, we’re in this business of teams getting going and it’s like it’s so easy to get to the point where you’ve instrumented your products and any new person joining your company can’t make heads or tails of anything. It’s like you’ve got all these events, are these duplicate events? We’ve invested a lot of time in this taxonomy feature, which helps manage your taxo- It’s way, way, when people try to build this stuff in-house, they just forget about all that stuff. Like, “Oh, it’s just events, it’s semi-structured information, we’re going to put it here and then we’re just going to run queries on it.” But all that’s really, really important, so back to the notebooks thing, one of the biggest use cases we’ve seen in notebooks is people using them to onboard people and orient them with all the available analytics that and metrics and things that are being recorded. That’s actually a really good testament to show that need. Brian: They use it to actually show how they use Amplitude at the- John: Right, it’s pretty meta. Brian: Wow, that’s awesome. John: Yeah, we see them do that or even some of them use it for training like, “Okay, let’s start with this idea that we’ve got this whole universe of users. Well, how would we segment those? Well, here are the key ways that we segment.” Okay, that we’ve gone down one layer, and so I think that that’s kind of cool. But, yeah, for people who don’t know about these data science notebooks, it is a mix of qualitative, quantitative, you can embed charts that are live or you could embed point-in-time charts, you can make comments, and you can do various things. I think for a lot of people who don’t do this for a living, they get intimidated and it’s not, a lot of the stuff is not rocket science, but it’s just annoying to have to go to someone in your company and say, “Hey, can you spend like three or four hours just explaining our information to us.” That’s really hard to do, so these notebooks help with that particular thing. I think that type of stuff is really the future of moving away from just very, very stayed dashboards and things like that. Brian: Right. I don’t know if there’s much in terms of predictive or prescriptive intelligence in the tool, does the tool provide that as well or is it mostly rear view mirror analytics? John: It’s interesting you say that, so we have this new feature called Impact Analysis, and so in Impact Analysis you are able to go from day zero of a particular use of a particular feature and then see the impact that it has on another set of things. We give some statistics and we give some other values in there. So we’re middle of the road moving to more and more complex questions. But one thing that our team realizes that anything… To prevent people from making bad decisions or making poor statements, you need to be so, so, so careful about presenting what you’re actually showing if there’s a correlation between something or even implying that there’s causation without doing the background on it. We’re not completely rear view and we’re in this middle ground, but we’re also going to go on record and say we’re predicting what this value’s going to be in six months. Brian: Right, and the reason, and not just the hype of machine learning, blah, blah, blah, that’s not my main reason for asking was going to lead into my next question, which was do you struggle at all with the expression in the tool of the evidence that backs up any types of conclusions that you’re showing? Do your customers care? Well, how did you guys arrive at this? John: They absolutely care, and so like one of the… We spent a lot of time in the ability, in Amplitude, any data point that you see, usually, if you hover over it, there’s a message it says, “Click to inspect,” or you can create a cohort off of that or you can see the paths to that particular thing. What we really made this effort to do is exactly right, is that people… Working at two analytics companies now, Pendo and now Amplitude, data trust and people being able to unravel what that number means in a way that makes sense to them seems like one of the massive limiters. It’s just that thing that it’s best laid plans start, that’s the entropy that exists with these tools as people use them more and more. There’s just it gets messier, a bunch of hands, a bunch of people are playing around. At least with Amplitude, they try to make a really big effort to like if you want to understand why that number is there and what is behind it, we try to make that really easy. John: But we could always do better because in my mind this is the number one difference between the more data snacking approach like “it kind of looks interesting, that number,” something that you can really pin your business on, which I think is what people… That’s the dream of all this, but then once people start to ask good questions really, it really challenges the tool. Brian: John, this has been fantastic chatting with you, I really appreciate you sharing this with our listeners. Do you have any parting wisdom or anything you’d like to share with people that are maybe working more on the tech side or the data side of the thing and the vents and they’re trying to, “I want to produce more use, whether it’s reports or actually software applications. But we’re trying to provide better stuff, more engaging, more useful…” Any closing advice you might give to someone like that? John: I’m going back to what we were talking about from the UX research angle is that I think that in this area, there’s so much temptation to any one of us who’ve done this is that there’s this constant push and pull between customizability and then this promise of preemptive insights like smart system, it’s intelligent, it’s doing these things. Then so how prescriptive are you? Is what you’re presenting and actually helping someone to do their job. I think that it’s probably reflective of my learning at Amplitude is that really going to human centered design, like really thinking about if the person is able to effectively do their job and really able to answer the questions that they’re answering. I think that what happens is all of us want that, but then we hit this wall and we start to get really some conflating information from users and we start to… Then we’re like, “Well, okay, we’re just going to let them find what they want to find. I think that, that exploratory type of research should be something that’s possible in these tools. In fact, I think that leads to asking some of the best questions when users can do that. But I would really hope that people don’t abandon the idea of being really patient and seeing if before they just throw their hands up in the air and will say, “Well, we’ll just make a query builder and that’s it, that’s it.” Like really seeing if that thing can solve the problem. I don’t know if that makes sense, but I think it’s something that’s really been on my mind a lot lately. Brian: Yeah, I talk about sometimes like with clients and people in this space about knowing whether or not you’re producing an explanatory product or an exploratory product. It doesn’t mean you can’t necessarily have some of both but there’s a big difference between the value, like in your case, I’m guessing a lot of these people really want some explanations when they tell us about what we can do to make our software better. They’re not there for fun, but they might run across some things they didn’t know were possible which begins the questioning. But if you put all the effort on them, you’re just shifting the tool effort over to the customer. You’re making it much harder for them to get the value out at which point they may abandon or quit. It’s not just knowing are we explanatory or exploratory or at least there’s this feature or there’s this outcome that this goal that we’re working on, the sprint. But just being aware of that I think is part of the challenge. Like should they be able to walk away with… I should be in the six to 19 apple’s range, whatever that means, like, should I be able to walk away with that level of clarity or not? I don’t know. John: I think that it’s also something like, that’s interesting you said that, because a lot of features that we’re experimenting with, one thing that Amplitude does is anytime you… We built an undo feature, so we try to make it really easy to go really deep and then just back out really gracefully. It’s like infinite, every version of the chart as you work on one is saved. You can back out of it. There’s a lot of features like Save As or you’re built like you could go to someone else’s chart, and if you have some idea of where you want to take it, you could edit it. But you’re not editing their chart, you’re editing a copy and you can think about it. But back to that point is I think that there’s many things that you can do to encourage, that you can juggle those needs concurrently for having definitive things and then also encouraging exploration. We’ve found that with our product as we experiment more. One, I just told you about it, like the ability to telescope into a metric and then do more exploration around it. That didn’t exist before and then we were like, “Oh, well, how about when you hover over any data point and you allow them to inspect that or explore that?” I would say that there are ways to accommodate both at least from our perspective and what we’ve learned. Brian: Right, and I think there’s always some of both of that, and I don’t think most people are going to take everything on its face value. But I hear what you’re saying. One of the things I’ve been recently working on is a UX framework for this called the CED framework, just conclusion, evidence, and data. It’s not necessarily a literal expression of “Where should the screens go? What goes on every screen?” But the concept that when possible, if the tool can provide conclusions with the second tier of being the evidence by which the tool or application arrived at this conclusion. Level three might be really getting into the raw data like, “What are the queries? What was the sequel that actually ran?” Or whatever the heck it may be, there’s times when maybe that data is necessary early on a customer journey. It may just be, “We need to build trust around this stuff.” We can’t be totally black box, but we don’t actually expect people to spend a lot of time at the D-level. We really want them to work in the C level, but it’ll take time and evidence is required sometimes if you’re going… Especially, I got to go to the boss, I can’t just tell him it’s 18, we should be at 18, not 12. It’s like, “Well, how did you arrive at that?” John: We find a lot is the instrumentation rigor is like that’s one of our big problems to solve really is there are these products on the market that do just try to record everything for it. There’s a lot of entropy there and there’s a lot of issues. They’re very fragile, in some ways, so we as a company definitely believe in explicitly instrumenting these events. But at the same time, you’d be amazed how many product teams… There’s this thing called a user story, you write a user story that’s from the user’s perspective, what are you trying to do? Now you would think that like, “Okay, well, we’ll tack on to the acceptance criteria for any story that you’ll use a noun and a verb, and you’ll get these properties and you’ll get these things. Integrating instrumentation on the product level, not necessarily like, “Okay, we’re instrumenting how our servers are working or anything,” but just, “What did the user do?” That’s still relatively new. People who’ve worked in environments that just do that as second nature that, okay, they’re in another thing, but we find that companies even need to change that approach. You’ve mentioned your CED thing like what’s interesting is that extends to the UX of instrumenting. It’s pretty interesting from that, it’s you’re the user trying to draw some conclusion, you’re doing these things. But it’s almost like service design, in some sense, because you need to design the approach to even instrumenting this stuff. It makes your head hurt sometimes. Brian: Yeah, all this stuff makes my head hurt. But that’s why we have conversations, hopefully, we’re knowledge sharing and it’s like giant aspirin conversations or something, I don’t know. But I found this super useful, thanks for coming on the show. Where can people follow you? I know I found you on Twitter. I forget how but what’s your [crosstalk 00:47:23]- John: Twitter is good, I’ve installed a Stay Focused app to prevent more than 20 minutes a day on Twitter. But you will find me eventually there. I write a fair amount on Medium and it’s pretty easy to find me there. Brian: Okay. John: If you just type in “John Cutler product”, I have about 400+ posts on Medium. Some are better than others but- Brian: Awesome. John: … yeah, that’s the best way for right now. Brian: Awesome. Well, I will definitely link both of those, your Medium page and your Twitter up in the show links. Man, John, it has been really fun to chat with you here. Thanks for coming on the show. John: Cool. Yeah, thanks for having me. Yeah, awesome. Brian: Yeah, super. All right, well, cheers. John: Cheers, bye-bye.

ceo english man data storytelling uber medium analytics saas measuring excel integrating ux north star product management javascript stay focused y combinator data centers zynga zendesk amplitude ced broad institute pendo productizing likewhy appfolio john you john it john cutler product evangelist john yeah north star metric john well brian yeah brian you john oh john right brian there john they brian john brian right brian exactly john then

013 – Paul Mattal (Dir. of Network Systems, Akamai) on designing decision support tools and analytics services for the largest CDN on the web

Play Episode Listen Later May 21, 2019 44:35

Paul Mattal is the Director of Network Systems at Akamai, one of the largest content delivery networks in the U.S. Akamai is a major part of the backbone of the internet and on today’s episode, Paul is going to talk about the massive amount of telemetry that comes into Akamai and the various decision support tools his group is in charge of providing to internal customers. On top of the analytics aspect of our chat, we also discussed how Paul is approaching his team’s work being relatively new at Akamai. Additionally, we covered: How does Paul access and use internal customer knowledge to improve the quality of applications they make? When to build a custom decision support tool vs. using a BI tool like Tableau? How does Akamai measure if their analytics are creating customer value? The process Paul uses with the customer to design a new data product MVP How Paul decides which of the many analytics applications and services “get love” when resources are constrained Paul’s closing advice about taking the time to design and plan before you code Resources and Links: Akamai Twitter @pjmattal Paul Mattal on LinkedIn Paul Mattal on Facebook Quotes from Today’s Episode “I would say we have a lot of engagement with [customers] here. People jump to answering questions with data and they’re quick. They know how to do that and they have very good ideas about how to make sure that the approaches they take are backed by data and backed by evidence.” — Paul Mattal “There’s actually a very mature culture here at Akamai of helping each other. Not necessarily taking on an enormous project if you don’t have the time for it, but opening your door and helping somebody solve a problem, if you have expertise that can help them.” — Paul Mattal “I’m always curious about feedback cycles because there’s a lot of places that they start with telemetry and data, then they put technology on top of it, they build a bunch of software, and look at releases and outputs as the final part. It’s actually not. It’s the outcomes that come from the stuff we built that matter. If you don’t know what outcomes those look like, then you don’t know if you actually created anything meaningful.” — Brian O’Neill “We’ve talked a little bit about the MVP approach, which is about doing that minimal amount of work, which may or may not be working code, but you did a minimum amount of stuff to figure out whether or not it’s meeting a need that your customer has. You’re going through some type of observation process to fuel the first thing, asset or output that you create. It’s fueled by some kind of observation or research upfront so that when you go up to bat and take a swing with something real, there’s a better chance of at least a base hit.” — Brian O’Neill “Pretend to be the new guy for as long as you can. Go ask [about their needs/challenges] again and get to really understand what that person [customer] is experiencing, because I know you’re going to able to meet the need much better.” — Paul Mattal Episode Transcript Brian: Hi. We’re back with Experiencing Data here and I have Paul Mattal on the line who is currently the Director of Network Systems at Akamai. How’s going, Paul? Paul: It’s going great. Thanks, Brian. Brian: I’m glad to have you on the show and you’re working at one of these companies that I think of as kind of like oxygen in the internet. It’s everywhere but you don’t really see it because it’s all invisible and that’s actually this big thing behind the scenes. You’re swimming around the internet as all these data and Akamai’s in the middle of all of a lot of that, largely responsible for making sure it’s moving quickly and is available at the right time and in the right places. As I understand it, you’re in a new position, you’ve changed domains, previously you were working in the space of legal patent work, digital forensics, and you built some tools that your previous company makes. You can tell us a little bit about those. Now, you’re moving more into the bits and bytes of the internet and you’re responsible for creating data products like decision support tools for people that keep the Akamai network going and running smoothly and anticipating demand? Did I get all that right? Paul: That’s exactly right. At Akamai, we like to think of it as we’re the ones that make the internet work. There’s a notion that the way things work on the internet is you just simply put your content up on a server and the rest is history. But these days, there’s a lot of complexity. There are many, many users who want access to the same content at the same time. Akamai makes that content all available to everyone when they need it and how they need it. In my past job, as you mentioned, was quite a bit different, although it had some similar qualities. I was helping to develop systems and tools for lawyers and for consultants for lawyers, in some cases to analyze patents, to help them better understand their subject matter of patents, so we’ve created some applications there. Here at Akamai, I’m also creating applications and tools to be used by the members of the network’s team who are responsible for deploying and maintaining the whole Akamai network. That breaks down roughly into tools that help us manage our work, tools that helps us with analytics and planning, and also tools that help us visualize data. It is somewhat of a shift. A lot of the domain knowledge is different, but it’s interesting that so many of the problems end up being similar. Brian: Tell us a bit about who the end-customer is. How many internal customers do you have? Do they break up into personas or segments? Like you have network administrators and you have whatever people. Tell us a bit about who those people are that you’re designing these tools for or you’re helping deploy these tools for. Paul: There are a couple of groups. The infrastructure group which is responsible for really deploying all of the servers and maintaining all of the servers. That’s a set of one class of user who is mostly using our tools in a logistical fashion to coordinate and organize their work. There’s a planning team who is thinking about the capacity of our network: Do we have enough for what’s coming down the pike? Do we have the right capacity in the right places? We also have users who are thinking about the architecture of the network and thinking about how we build and optimize our hardware and our network, to continue to be cutting edge and to continue to meet the needs of our customers. So, different people looking at different tools and different data for different purposes. Brian: Cool. Just a little fun question here. This is probably because I don’t know the domain very well. When there’s a big event coming on the internet, let’s take something like the Super Bowl, or the World Cup, or the new Game of Thrones, or whatever, are there literally changes that you guys go and make to facilitate a major event? Or are those actually more like a blip in terms of internet traffic and all of that? Paul: It depends. Certainly, some of those events have been some of the largest data traffic we’ve seen move across our network. Often, there are considerations especially depending on where exactly we expect the viewers to be for those events. We may deploy additional capacity in one geographic area or another. Brian: Going back to the people that are the end of these tools—again, these are decision support tools—how do you know if your team is doing a good job? How do you measure that the end-customers are getting the right information and they believe it, that they’re willing to take action on it? Do you a regular feedback cycle or interaction with these different personas that you talked about? Paul: Yes, That’s one of the most important aspects of what we do is trying to figure out how to measure, how exactly to measure how we’re doing, especially in the analytics space, right in the productivity tool space is a little simpler. We can tell pretty much where the pain points are. People come to us and say, “This interface isn’t working for me or these five things are in five different places,” and they’re going to use them as one. Those are a little bit more straightforward kinds of feedback. With analytics, we find it goes a lot to how successful were we predicting, how much excess capacity did we end up within a place we didn’t need it, for example, and all those kinds of questions. We meet with our customers pretty regularly and we also have some metrics that we compute to give us an idea of how we’re doing. Brian: Are those quantitative then? Those are all quantitative metrics or do you have any type of qualitative conversations that go deeper than like, “I wish there was a filter for the date on this chart,” or stuff like that. Those things do matter and it’s the sum of all those little, tiny details that add up into good experiences typically, but I’m curious if you have any deeper qualitative type of interaction with these end-users. Paul: A lot of what we’re discussing these days, for example, is there’s a tremendous amount of telemetry available that comes off the platform. Numbers about what’s going on in the network that could measure and we could capture. In many cases, a lot of the conversations are about, “Hey, can we capture more of this data? Is there’s somewhere we can get sample more frequently?” or, “Can we get access to this kind of data that we don’t have right now, so that we could be able to optimize more effectively on the things that actually matter, where the actual bottlenecks are in the network,” versus more simplified models based on less data. We’re finding that’s one of the very common kinds of feedback we’re getting is for more data and differently sampled data. Brian: We talked about this a little bit when we did our pre-call on whatever about topics and you mentioned that you have different classes of users in terms of who’s capable of designing an effective tool for themselves. I think you said you’ve got a mix of tools that are custom-built which might have two-way interaction, where data’s being put back in through forms or whatever in the tool. Then you have Tableau and some kind of rear-view mirror type historical reporting interfaces which, as I understand it, those start with the user a blank slate? Is that correct? Then they put together the views that they want and the reporting that they want? Kind of curious just for you to talk about how many people are using custom tools that you built versus the ones that they designed for themselves. Are people doing a good job creating the tools they need for themselves? Do you have a sense of that feedback that they’re looking at the right data, that they know how to interpret it, they know how to visualize it? Can you talk a little bit about that? Paul: Sure. Our organization has hundreds of people in it and I would say at least probably 50%–75% of those users are highly technical, which is very helpful, actually. They often come to us with a better idea of what they need. In some cases, we can give them good interfaces to go build their own tools. The historic approach to that here has been to give them pretty decent access to the data in our databases and even the engines themselves. Many of them are comfortable writing their own queries. But we also have a very mature ecosystem of query exchange. We have this tool that allows people to write their own queries and share them with others, and then others can manipulate those queries further and customize them to their own needs. They’re very familiar with that. The piece we’re bringing in next is this idea of really making visualization also of a self-serve kind of area where, with a tool like Tableau, you can point Tableau at the same data that might be the out part of these queries but then have powerful visualizations on top of that. The other piece of this is how much of it do we do and how much of it do customers create from old cloth. It’s kind of a balance. Some people come to us and say, “Here’s what I need but I don’t know how to do it,” and then they ask us to do it. Sometimes a customer actually originate it and will say, “Here’s the report or this query that I think is interesting,” and we’ll say, “Oh yeah, that’s interesting. Why don’t we bake that into something more sophisticated?” It’s kind of a mixed bag but I would say most people come in to us, there’s usually something that we already have that they can use as a basis and then they can usually modify that further. That’s been a pretty successful model for us because it really lets people get what they want, get the very detailed, precise view that meets their needs, but benefit from all of the other work that we’ve put into to making those views and those approaches effective and mature over many years. Brian: It sounds like you don’t struggle as much with engagement with the analytics. You actually have plenty of that? Or would you say that’s not necessarily entirely true? Paul: Yes. I would say we have a lot of engagement with that here. People jump to answering questions with data and they’re quick. They know how to do that and they have very good ideas about how to make sure that the approaches they take are backed by data and backed by evidence. Very mature in that sense people. Brian: Since you have this mix of these custom tools that you guys are building and how slick, how do you decide which wheel is going to get the most oil? You’ve got these custom tools, you’ve got some Tableau stuff, you’ve got people coming in, maybe they are using Tableau, but they don’t know how to build the reporting they need. Is it based on a business driver? If we get problem X wrong, this cost a lot of money, so we’re going to put our team on this problem and sorry, Jane, you’re going to have to take that Tableau tutorial and figure it out yourself. How do you resource like that? Paul: As with any place, there’s certainly scarcity. Everybody wishes they had choice in people they had and twice them. Maybe even the computing resources and everything else that they wished they had. At a high level, a lot of it is driven by a strategic plan, by an idea for what we as an organization are trying to accomplish. That determines which things get the most people and the most priority. There’s actually a very mature culture here at Akamai of helping each other. Not necessarily taking on an enormous project if you don’t have the time for it, but opening your door and helping somebody solve a problem if you have expertise that can help them. We find that it’s a balance of those things. We work on major roadmaps, large projects or tools for strategic and efficiency. Particularly efficiency reasons that we’re wishing to achieve as an organization. We spend a lot of us of the time helping the folks who need it, to get where they need to get. Brian: That makes sense to me. Is the feedback loop in place such that there’s some point in the future which you look backwards on these projects, or products, or tools that you’ve built and say, “Did we make a dent? What were the success criteria for those? What’s that three month or six month rear-view look like?” Do you guys talk about what that is, so you know whether or not you hit your objectives? “And since project X got four times the resourcing, did we get four times the value or whatever the value was that was determined?” I’m always curious about these feedback cycles because there’s a lot of places that they start with this telemetry and data, then they put technology on top of it, they build a bunch of software, and a lot of times the releases and the platforms are looked at as the outputs and the final part of this and it’s actually not. It’s the outcomes that come from the stuff we built that matter. If you don’t know what outcomes those look like, then you don’t know if you actually created anything meaningful. So, I’m curious, that feedback cycle, does your business know? Like, “We have to see. We can’t get predictions wrong or we don’t want to have a little more than 12% server waste from the wrong prediction, whatever.” I don’t know what those metrics are. Can you talk about that feedback loop from a business and a value perspective? Paul: Sure. Some of the things we’re doing are very tied to specific business goals for certain kinds of […]. These are targets for dollars saved in terms of operating the network at a lower cost. In those areas, we are very acutely being measured pretty much on a yearly basis along those lines. We’re working towards getting better at what happens in between and the rest of the year. You can often go off-track a little bit somewhere in one month and that can cost you down the road. We’ve been focused on trying to get to more of a monthly evaluation where we can break things down, try to deliver a value on a monthly basis, then get feedback from customers, and also to see how they’re affecting the numbers in real world application of this data to actually optimize. They never to learn. Are we consistently on track? Or are we moving in the right direction? I say that it’s definitely an element of what we do. Right now, we’re doing it more like every six months or a year. At a granular level, we’d like to move that to be a much shorter term and focus on constantly delivering smaller chunks of value. Brian: That’s good to hear. My understanding from when we talked to that you be almost what I would call a product manager, even though you’re not developing commercial products but you’re overseeing the creation of these different tools. I’m curious. Do you have the equivalent of a product manager role where one person’s job is to make sure that whatever analytics and/or custom tools you guys build for the network operations team or the team that deploys the servers, they live and breathe that world and they’re totally responsible to service those staff that work on those technical problems? Is that how it’s shaped or is everyone’s touching all of the different parts of Akamai? I’m just wondering how you get into that world. What’s it like to be the server administrator and predicting where to deploy servers? How is that structured? Maybe you don’t have enough staff to break it down that way and I’m asking a leading question, but I’m curious if you could talk about that a little bit. Paul: We actually do have four teams within our group and they are divided up with focus on the different stakeholder groups within the network’s organization. There is definitely some division. There’s also some who sort of cross responsibility but there are definitely folks who know specific subject matter areas very well and who are critical in those areas to anything more than the simple bug fix in an area is going to involve somebody managing that area. Now, for our largest projects of all, we do have product managers as well as project managers involved in the creation of the larger ones. I’d say about two or three are major systems and the other several hundred tools or various pieces that we manage, care and feeding over the years. That stuff is either being taken cared of by one of these SME areas or it’s sort of rolling out to me especially if it’s something new. A large part of my role is helping to at the outset to say, “Let’s define what this tool looks like. What it’s doing? Who is going to use it? What those people need? What are the processes at play here at Akamai that this is a part of? Do we understand those processes? Have we optimized those processes?” That’s a lot of what I end up doing with the rest of my team, to define those new products so that they’ll be the most successful as we build them and get off in the right direction. Brian: That sounds awfully like design to me. Paul: It is. Brian: Is that traditionally how things have been done in this group or is this something that’s new? How’s that being received? Are you getting like, “Just give us the data and we’ll put it together,” and you’re like, “No. Help me understand what are you going to do with it at the end.” It’s just like, “Well, I’ll know when I see it.” Is it that kind of thing or are they like, “Great, let’s get it right.” What’s that process like? Paul: The history of our group is that we have probably not put enough focus on planning and design, but I think it’s an area where people realize that we need to spend more. They really are now focused on that as a goal and understanding that it’s important in many context. That’s not to say that there aren’t sometimes when people will say, “Here’s what I need and I need it tomorrow,” and you know that comes up. It’s a balancing act that is always a challenge, but I think there is an increasing sense and increasing support across the network’s organization and maybe beyond that using some sort of platform organization, other parts of engineering at Akamai. It’s really a much better result if you make a plan upfront, you understand the context into which you’re creating this new thing, and you understand how it’s going to impact processes and flow that occur once you’ve built it. Brian: Maybe you haven’t been there because I know you’re somewhat new in this position but if you’ve been there long enough to go through a full cycle with that where you’ve taken someone through like, “Let’s hold on. Let’s figure out what’s actually needed. What the real problem’s face is like,” and then you’ve gone all the way through maybe building a product or a prototype or something. Have you gone through a full cycle yet? Or are you still in the design phase on some of these? Paul: For a couple of smaller projects, we’ve definitely done that. It’s been posted where people have come and said, “Hey, could you do X?” and we’ve said, “Well, we could do X but that actually requires more code and more effort. We have this other thing over here that actually can accomplish that and then it puts you more in the driver’s seat because you can help maintain it later. How’s that?” Often, the results are very positive. If we can actually get things implemented faster, people are happier in the end, it’s less maintenance for us overall in the long-haul. So, yes on the small things. On the bigger things, those are in progress and we’re excited about those design phases that’s going on now. They’re larger and more productive than they’ve been in the past. We’re excited to see probably by the middle of this year or later in the year that there is also an output of that. Brian: Can you tell us about what some of those activities are? I think some of the people listening to this are not coming from digital-native companies. The whole product design process is maybe foreign to them. Can you tell us about like, “What are you doing during this time? Why aren’t you writing code? You have the data. Put Tableau there and build some reports.” What are you doing that’s not that during this phase? Paul: Usually, the first thing we’re doing is trying to find out who are all the people that interact with this data, or these kinds of systems, or these particular business objects, or aspects of Akamai’s network. Often at the start, we find there’s common problems. There’s people and other parts of the organizations who may already have a tool that allows them to do this. Now we also want to go and observe those users. We want to go find out are they satisfied with the tool and is the tool meeting their needs, which are actually two different questions. Really seeing whether what they’re doing is a process that’s optimal and seeing whether we can create a solution to this new problem or borrow a solution to this new problem and change it in some way that helps everybody. That’s one of the interesting aspects of design here is that there are many groups that interact with the same data in so many different ways. I think a lot of that design phase is about, “Hey, one of the tools out there, how do we integrate them so that they’re the least work for us? How do we make sure that we’re choosing a good solution and we’re actually meeting the user’s needs?” Probably the last part of that, especially in our group is, and not getting stuck on not meeting 100% of any single tool, because in some cases, you’ll get 80% of the use cases for five groups and you have to say, “Okay, that’s fine. For this other case, they’ll do it this way.” That’s a lot what goes into the design process. Really just understanding what the users are looking for, how does that match up with stuff we already have, and then how do we integrate that use case into what we maintain, in a way that is streamlined and effective for them and also streamlined and effective for us. Brian: When you talk about getting to know what they’re going to do with this information and how they want to use it, is that through them self-reporting through, like talking to you in a meeting? Is it through you observing them doing what they’re doing now without the tool? Is this largely like, “Right now I can’t do any of this. I need this tool so I can enable this new thing that I currently don’t do,” or is it more like, “I have this long, convoluted process I have to do in order to achieve X. Can you help me build the tools so I can do it in less time?” One of those there is like a recipe for something already and you’re trying to optimize it and the other one is more like, “This is a new thing I’ve never been able to do but maybe I could with your help.” Do you put it into those buckets and then if it’s the former, how do you figure it out? Is it observation or just them talking to you about how they’re going to use it? How do you figure that out? Paul: There definitely are both of those scenarios come up. We often get requests about processes that already exist. At some point, there’s some tool in there already, sometimes it’s a highly manual process. In that scenario, one of the great assets of this particular group is that we have whole standards, documentation, and work co-optimization group here within network, which is a true treader to have. Usually, when that kind of problem comes up, the first thing we do is say, “Okay, let’s work with the [worker?]group and let’s get a really good map of what this process looks like end-to-end and let’s look at what the steps are, what tools are now, where the pain points are, and then once we have drawn this out so that we understand the context, let’s actually first look and see whether there’s any way we can optimize the process, because the last thing we want to do is to spend a lot of time implementing automation steps for a process that shouldn’t be that way in the first place.” We look at that process and we say, “Okay, how do we simplify it? How can we bring automation to bear, to make the process more straightforward, take less time, take less human effort.” Then, we usually at that point, sit down and actually design the automation solution around that. That’s one kind of problem and that process of workflow [analysis…] does involve what we call business process performers in each step. These are not the people who manage those areas. These are the people who are actually doing the work. We want to know what are they actually doing, we talk to them whenever we can, and we actually go [observe.] them because we can learn at least this much and probably more by watching what they’re doing and what they’re struggling with. That’s one side of it. The less well-described problems, those are the ones where nobody knows yet. This is something brand new. There, I think we tend to sit down and try to understand what these users are trying to accomplish, what problems they had in the past that this addresses, because so often, something that’s new is really some way connected to something old. We did this before. It didn’t really work or we have a gap here, there’s something that we’re not doing as well as we should or we’re not doing at all, and how do we get that better? A lot of it is about understanding what they’re looking for and I think the big element of that that’s key is breaking it down into manageable phases so we can deliver quickly and iterate quickly. The last thing you want to do is sit down and say, “Okay, we think we understand exactly what you need. Now, we’re going to go off for a year-and-a-half and build it.” That’s always a recipe for disaster. So, what we want to do is sit down and say, “Let’s take the most important crux of what you’re trying to get at here. Let’s implement something in a few weeks or a month. Then, let’s sit down and get it in your hands, get your feet back on it, and then figure out the next piece.” This doesn’t mean we can’t have a plan for like, “Here’s really roughly what we think the phases are going to be and how they’re going to be laid out. But let’s have these checkpoints along the way and let’s iterate based on what we actually are able to learn, what we actually to benefit from.” That’s what we found is the key to those kinds of new projects is the fast iteration cycle. Brian: We’ve talked a little bit about the MVP approach, which is about doing that minimal amount of work, which may or may not be working code, but you did a minimum amount of stuff to figure out whether or not it’s meeting a need that your customer has. You’re going through some type of observation process to fuel the first thing, the first asset or output that you create. It’s fueled by some kind of observation or research upfront so that when you go up to bat and take the swing, there’s a better chance of at least to base hit and not a strikeout or something. I fully support that type of effort instead of me going off, “We all have the data. We’ll send you back a kit and then you can put it together yourself. It will take a year, you’re going to dump everything into the data warehouse, and then you fall into the Gartner 85% of ‘Big Data Projects That Failed’ category, which nobody wants to be in that whole thing. I think that that’s really great you’re doing some of that. Earlier, you said you have a lot of different products and you said two to three of them are large. I’m just curious. Large by number of users? What justifies putting a dedicated product manager on it and what’s the extra love that is received because you’re one of those two or three? Is it they have a dedicated designer and dedicated engineers? It is more research time? Tell us about your big ones. Paul: I would say that the largest projects usually have someone who’s effectively an architect for the project, who may also be part of the development team. They usually have a development team. It’s usually several people. At least in an ideal world, three or four is probably typical for larger projects. Then there’s a project manager who is managing the project and also how that reports up into our overall program of initiatives for that organization. Usually, those projects, to get substantial research, are going to be priorities for the organization at some higher level. The last [piece] probably the most important piece is that there’s a product owner, who may or may not be the architect, in some cases the architect plus feedback from the stakeholders is enough to make it work, but most of the time, it’s usually somebody who is also the project owner or the product manager who’s really responsible for shaping the design of that product. For example, one of the big tools we’re working on has to do with increased virtualization that we’re rolling out within the Akamai network. This is a big project because it’s a company-wide initiative. We have somebody working on designing the interface and working to figure out how the interface to provisioning works within the context of all the processes we have here at Akamai. Another example, one of our key analysis or databases for analytics and for planning. There, the ownership is essentially a data team who is responsible for this database, the universe of this data, and roughly how it’s visualized. That team has responsibility for that database for its schema, how we got that data, where it comes from, its cleanliness, but also for the visualization aspect of it, and then it’s now also inheriting this ‘how do we use Tableau as part of that ecosystem?’ Just to give you some idea how these projects are organized and then what the roles are. Brian: Got it. Your large projects fall both into maybe a database that’s sitting behind Tableau as the interface and then you have another one, the server provisioning one, which sounds like a custom web-based application or something? Paul: That’s right. Brian: So then, for that one, to me that’s the decision support. The provisioning action would be the decision the human takes theoretically, upon some analytics or insight, that made them decide, “I need to push the button to deploy X servers in Y region or whatever it may be.” Is that decision support part of that custom product as well? Or is this a balance between two or three different Tableau instances that are behind different databases, and then you co-authored the provisioning tool and just do the action, you make the decision in that tool, but the insight about when and how and where to make the decision is not part of the tool? Or is that actually in that tool as well, where it’s like, “Hey we predict that you should do this,” or “Here are the stats. You come and make the decision on provisioning based on what’s in this tool.” How much is that wrapped together versus a series of different URLs you’re going to bounce through and piece together yourself with eyeball analysis? Paul: There’s some separation of systems and we’re actually moving into a more integrated direction. For example, a lot of us begin with a customer demand. Either we determine or the customer gives this information that helps us determine that they need capacity in a certain area. That drives the process but that also factors in to a lot of decision-making that goes on, right about exactly what gets deployed, where, and when. There’s elements of this that are integrated in a sense that the deployments that we’re planning to make to expand a network or to choose a network in some way, are inputs into this great big optimization model where you say, “Here’s what we know we think is coming, here’s what we know we think is going to happen, here’s the moves we’re planning to make when and where we will run out of capacity. I think we’re moving towards a more integrated feedback model for that where less of the work has to be sort of connect the dots by a human being and more to saying, “Okay, all the systems have this data and if they can exchange it with each other, then we have all the data in the places we need it.” Brian: You’re talking about this feedback cycle annually, then you might look back and say, “How well did we arrange for these optimizations? We planned for these predictive resource allocation or whatever it may be,” you look back and see how accurate that was by looking at the utilization rates or something? Paul: Exactly. Is there a customer demand we failed to meet? On the flip side, were there servers sitting around underutilized? Brian: Got it. When we talk about Akamai going out and deploying servers, are you talking about deploying physical hardware in a datacenter or are you just talking about provisioning up virtual servers on the cloud somewhere? I’m just curious because you guys are a network that sits on top of the internet. Does this involve lots of humans and you’re rolling out hardware and all that or are we really talking about virtual deployments? Paul: Some of each, but one of Akamai’s hallmarks, actually, is the breadth of the network. We have some servers in pretty remote locations. These are physical servers. These are places in some cases where there isn’t a lot of good cloud providers or anything like that. Brian: Johnny’s going to the Arctic to install some Dell servers. Paul: That’s right. I’ll tell you there’s a datacenter in Antarctica and it’s possible we have a server there. Brian: Someone’s got to go rewire it once in a while. Oh, we’re out of a storage. There’s still disk drives in that cloud up there. They might be flash, but they’re still a piece of hardware. Paul: One of the things that really differentiates Akamai is that we have this extensive edge network which really is pretty unparalleled to the industry. Brian: When there’s a report back then, do they look at the travel cost for Johnny going to the Arctic on an ice clipper or whatever it’s called, and then was it worth going there to deploy these servers? Paul: Sure. Increasingly, that is the kind of analysis that we’re doing. [and] we manage the network according to some of that. When there’s servers that are sitting somewhere and just not getting used or they’re there but they were extremely expensive to put there, then maybe that’s not a place to cover in the future. But in some cases, it makes sense to keep our coverage really good even if in one area where we’re sacrificing a little bit of cost to keep the coverage up over all and that might be worth it. Brian: Right. I’m curious. Now that you’ve been here awhile and through all these, do you have any stories or anecdotes about a particular user experience, a customer/internal user that found an approach that’s useful, or you’ve got some feedback or maybe it was negative, but you learned something not to do it again, or any type of anecdotes that you can think about that were insightful for you? Paul: Yes. We had a number of tools that we use for manipulating all the business data around what’s deployed in our network. I would say that I guess the best [anchor??] I had about them is that we’ve found there are tools that are very commonly used because of their flexibility. But if you actually look at the tool itself and you look at the complexity of the tool, it’s not that complex. It’s the default way of using things and people have used it continually because it has always been the way of using it, when in fact, there’s nothing particularly special about it. We’ve seen in certain circumstances where you give somebody a new tool that just works faster, it provides very similar interface, or you found some tweak to that workflow that really can save them tons and tons of time, and you just watch their eye pop out. They realized that you just probably saved them two hours a day. It’s interesting that that can happen in pockets and corners. There are many tools that have been built already to help with that but there are still plenty of opportunity for it. Brian: That’s great. That’s one of the things I think I love about being a designer. A lot of times, the big picture rewards like, “Was this product valuable or profitable?” There’s these lagging indicators which take a while and they don’t have the same hit as those small wins which were like, “I just saved this guy two hours a day doing a task that has nothing to do with his skill set. It’s just labor. He’s not using his brain. He just has to download these logs, put them in Excel, run a lookup, and then blah-blah-blah. And now it’s just bam.” I love that and that’s part of it for me, at least the joy of doing design work and stuff. I totally relate to the way you’re saying about helping someone. It is so much about helping people and you also feel like, “Man, I’m also helping the company because I’m helping this person use his brain to do much more important things than maybe he was doing with tool time, like downloading crap and uploading into a tool, sorting it, changing this, and blah-blah-blah.” Most of that is tool time. That’s not the, “Should we put more servers in Antarctica?” It’s not the thinking time and the valuable business time. Paul: It’s one of the very fulfilling aspects of the job like this where you’re building tools for internal stakeholders. In many software industries, you build product but your users might not be accessible for you or hardly at all. “I see that they’re right down the hall.” It’s a great fulfillment I think in building something that meets a person’s need and having that feedback and knowing that [did.] and having the satisfaction of that. Brian: Yeah, that’s awesome. This has been great. I’m curious. Do you have any closing advice for other product owners or data product leaders, analytics practitioners in your space, maybe about changing domain, you’re in a new domain? Any kind of insights looking back in this six months or however long that’s been that you’ve been there? Paul: I would say above all, my advice would be take the time to plan. Nobody ever thinks they have the time to design or to plan. To some extent, you just have to say, “If we don’t do this, [you know] the thing we build is not going to be worth nearly a much as the thing we could build.” You’re much better off figuring out the right design for something before you build it. Even when you think you don’t have the time, ask your managers and then your management chain for that space you need to get that pipeline started the right way because once you actually design things, you’re going to find that the number of people you’re helping and the degree at which you’re helping them is much greater. Brian: I can totally get behind that. That closing statement, I agree. First of all, you’re putting that anchor in place to do good things down the road. You’re probably reducing you’re technical debt and you’re maximizing your ability to change, especially when you’re doing small deployments. You’re probably going to need to change stuff, so a little bit of designing and planning upfront can do a lot for both the engineering part of it but also most importantly the customer experience getting that right. So, amen to that. Paul: Maybe the last part of that, just to add, is sometimes we take for granted the job we’ve been at for a long time. We actually take for granted that we think we know what everybody needs already. Sometimes, actually, it’s a blessing when you come to something brand new, because you’re not to assume you already know what that person across the hall really needs. You say, “I’m going to ask that person because I have no idea.” I would say these problems are the same everywhere. Whether you’re in a place, in a domain you’ve been for a while, there’s still going to be some aspect to that problem and you don’t understand what that person is living with. Pretend being the new guy for as long as you can. Go ask again and get to really understand what that person is experiencing because I know you’re going to able to meet the need much better. Brian: Yeah, I think that’s great advice. You don’t have that bias from your own knowledge about the domain or your assumptions there and that’s just a good design technique in general is being able to compartmentalize. We all come to the table with bias but if you can try to put that aside. For me, a lot of times it’s like leaving with new stuff with clients. It’s explain it to me like a five year old and I tell my clients sometimes this like, “What does it mean to deploy a server? What is he literally going to do and how does he know when to push the button to go do that,” and sometimes they look at you like, “What do you mean? You don’t know what a server is?” It’s like, “Well, I know what a server is, but literally I want to see every step it takes to know to go put one there. Is the guy going to walk out there with a box and rack it up? Or is this a virtual thing? Literally tell me what that’s like, that whole process.” Even though I know something about how that works, you’re going with that clean slate because you want to be open to those things you don’t know to ask about, and that the more you can come in with removing as much of that bias is possible, you might find those nuggets and stuff that just pop out to you that the customer doesn’t know to tell you about, but that they’re just going through their process. They’ll often ping you. You have these moments where you’ll learned something you didn’t go in there to ask about and sometimes it can be a really big thing like, “Wow. That’s really what the gap here is. It’s not this. It’s this another thing.” Having that really childlike innocence about the way you inquire can help enable that. Paul: Absolutely. Brian: Where can people find out about you? LinkedIn? Twitter? Are you out there in the internet? Paul: I’m on LinkedIn, for sure. I’m on Twitter. I’m on Facebook. Brian: Where are you on LinkedIn? What’s your Twitter handle? I’ll put the links in the notes, too. Paul: I think I’m @pjmattal everywhere. Brian: @pjmattal on Twitter. Okay, great. I’ll put your information up there. Thanks for coming on the show. It’s been great to hear about what you’re doing in Akamai and good luck as you guys charge forward. Paul: All right. Thanks.

director man super bowl tools network numbers game of thrones mvp services world cup large designing analytics excel antarctica largest arctic bi pretend sme gartner urls tableau akamai paulas decision support paul it brian it brian yeah brian you brian so paul one paul there brian right paul for paul all

012 – Dr. Andrey Sharapov (Data Scientist, Lidl) on explainable AI and demystifying predictions from machine learning models for better user experienc...

Play Episode Listen Later May 7, 2019 42:33

Dr. Andrey Sharapov is a senior data scientist and machine learning engineer at Lidl. He is currently working on various projects related to machine learning and data product development including analytical planning tools that help with business issues such as stocking and purchasing. Previously, he spent 2 years at Xaxis and he led data science initiatives and developed tools for customer analytics at TeamViewer. Andrey and I met at a Predicitve Analytics World conference we were both speaking at, and I found out he is very interested in “explainable AI,” an aspect of user experience that I think is worth talking about and so that’s what today’s episode will focus on. In our chat, we covered: Lidl’s planning tool for their operational teams and what it predicts. The lessons learned from Andrey’s first attempt to build an explainable AI tool and other human factors related to designing data products What explainable AI is, and why it is critical in certain situations How explainable AI is useful for debugging other data models We discuss why explainable AI isn’t always used Andrey’s thoughts on the importance of including your end user in the data production creation process from the very beginning. Also, here’s a little post-episode thought from a design perspective: I know there are counter-vailing opinions that state that explainability of models is “over-hyped.” One popular rationalization uses examples such as how certain professions (e.g. medical practitioners) make decisions all the time that cannot be fully explained, yet people believe the decision making without necessarily expecting it to be fully explained. The reality is that while not every model or end UX necessarily needs explainability, I think there are human factors that can be satisfied by providing explainability such as building customer trust more rapidly, or helping convince customers/users why/how a new technology solution may be better than “the old way” of doing things. This is not a blanket recommendation to “always include explainability” in your service/app/UI; I think many factors come into play and as with any design choice, I think you should let your customer/user feedback help you decide whether your service needs explainability to be valuable, useful, and engaging. Resources and Links: Andrey Sharapov on LinkedIn Explainable AI- XAI Group (LinkedIn) Quotes from Today’s Episode “I hear frequently there can be a tendency in the data science community to want to do excellent data science work and not necessarily do excellent business work. I also hear how some data scientists may think, ‘explainable AI is not going to improve the model’ or ‘help me get published’ – so maybe that’s responsible for why [explainable AI] is not as widely in use.” – Brian O’Neill “When you go and talk to an operational person, who has in mind a certain number of basic rules, say three, five, or six rules [they use] when doing planning, and then when you come to him with a machine learning model, something that is let’s say, ‘black box,’ and then you tell him ‘okay, just trust my prediction,’ then in most of the cases, it just simply doesn’t work. They don’t trust it. But the moment when you come with an explanation for every single prediction your model does, you are increasing your chances of a mutual conversation between this responsible person and the model…” – Andrey Sharapov “We actually do a lot of traveling these days, going to Bulgaria, going to Poland, Hungry, every country, we try to talk to these people [our users] directly. [We] try to get the requirements directly from them and then show the results back to them…” – Andrey Sharapov “The sole purpose of the tool we built was to make their work more efficient, in a sense that they could not only produce better results in terms of accuracy, but they could also learn about the market themselves because we created a plot for elasticity curves. They could play with the price and see if they made the price too high, too low, and how much the order quantity would change.” – Andrey Sharapov Episode Transcript Brian: I’m really excited to share my chat with Dr. Andrey Sharapov today from Lidl, the large grocery store chain from Europe. Andrey is a data scientist. I met him at Predictive Analytics World while we’re speaking there. He told me that he was quite interested in removing the black from the black box concept that goes with predictive models. This is called Explainable AI or Ex-AI. I think this has a lot of relevancy to designing good decision support tools and good analytics that people can believe in and will engage with. There’s obviously been a lot of talk about this area whether it’s from a compliance perspective or an ethics perspective or just a customer, an end-user experience perspective. Being able to tell what models are doing and how they’re deriving their predictions has value on multiple different levels. Without getting super into the technical side of this, we’re going to talk about how explainability within machine learning and predictive models has relevance to the design and user experience of your software application. Here’s my chat with Andrey. Hey, everyone. Welcome back to Experiencing Data. This is Brian and I’m happy to have Dr. Andrey Sharapov from Lidl. Did I say that right? Lidl, obviously, not obviously, maybe to our American listeners, people that don’t live in Europe. But is it Lee-del or Lie-del, the grocery store chain in Europe? Andrey: It’s Lee-del. Hi, everyone! Brian: Welcome to the show, Andrey. You’re a senior data scientist at Lidl. Tell us about your background. What are you doing with grocery data? Andrey: Hi, Brian. Thanks for having me on the podcast. As you said, Lidl is one of the largest retailers in Europe. We have more than 10,000 stores. We obviously have quite a lot of data that we’re trying to put to work at the moment while building this data products. For instance, we try to create decision support tools in order to help our action planners or promotion planners to make better decisions. On the other side, we’re automating various processes like for instance order disposition which means ordering of goods automatically for all the stores. We have a lot of other use cases related to marketing and everything that has to do with business as Lidl more or less. Brian: I’m excited to talk to you about [a]particular area of interest that you have which is explainable AI. But before we get into that, I’m curious, it sounds like [you’ve touched several different aspects of the business that Lidl with some ]of the data products that you’re creating. What’s hard about getting that right? Not so much from the technical standpoint, and the engineering, and the data science piece, but in terms of someone’s doing the purchasing of the carrots and someone’s doing the planning of the promotions, tell us about that experience and how you go about getting those people the information they need such that they’re willing to use your analytics and your decision support tools to actually make decisions. Can you to talk to us a little bit about that? Andrey: Yeah, sure. As you’ve pointed out correctly, these days, it’s not really much about technology or data crunching but more about weaving together the relationship between data scientists and the business in order to get a buy-in from the actual users. Let me maybe just say a little bit about how we work on our first data product, the one that I created along with the team. We went through a lot of struggles, trying to figure out [what data scientists do or do any data engineers,],and then at some point, we’ve got product owners on the team, the people who actually talk to the business in the language that the business understands. We also got businesspeople onboard as advisors for the project. The product that we built is called a planning tool, so to say. Every week, the operational people plan a promotion for certain date in the future. They have to pick a certain number of decisions, take into account conditions of the market weather, time of the season, and a lot of other things, then come up with the number to order. The sole purpose of the tool that we built was to make their work more efficient in a sense that they could, not only produce better results in terms of accuracy, but they could also learn about the market themselves because we created a certain clause for[ instance instant]elasticity curves and they could play with the price and see, if they make the price too high or too low, how much the order and quantity would change. That’s the main idea behind the product. I guess most companies have the same problem of trying to onboard the business users. The main sort of idea or the main way of thinking, “Okay, we have AI in that.” They will search in the user, they’re like, “Okay, let’s just do it.” But most likely, it’s not that easy. We went through this experience of learning that, “Okay. Although we have the coolest algorithms in our system and from coolest people working for us as data scientists and engineers, [totally doesn’t that the final user ]??will use the product.” The way that we try to convince them was by building fancy user interface in terms of making it more beautiful, so to say, but nonetheless, they were not very convinced. As far as I know, the operational people are really hard to convince because maybe the majority of operational people try to use such tools in order to execute certain tasks very fast. They don’t have a lot of time to try to learn what’s going on but rather, they would like to do a few clicks and the job is done and they move to something else. In our case, since it was a lot of machine learning, a lot of predictions, there was this problem of trusting the system because although they could use it in this way that I just described—that they could just do some clicking and then complete the task within, I don’t know, 15 minutes maybe. But nonetheless they were hesitant to use it because there was this lack of trust to the system. They would question why the prediction is maybe a little bit higher than they expect or a little lower, and there was no way to explain it to them. You basically say, “Okay, it’s an algorithm.” This aspect was, I guess, quite crucial because in their mind they also have a certain number of rules that they follow when they do the planning. This is basically kind of an invite for the next question so to say about explainable AI that we try to show the end-users various types of explanations later in order to gain more trust from their side. I guess, as I said at the beginning, this phrase that if we have AI in the data product then they will use it, “Let’s just build it.” Well, in the end we’ve learned that it’s of course not the case. People first were very skeptical but later we’ve tried to really work hand-in-hand with them trying to polish this single feature that they had in their mind. In this way, we’ve brought in more people who’ve tried the product. Brian: You hit the nail in the head there with these technologies. Whatever is new, as we talk right now, AI is definitely high on the hype cycle, it’s not magic sauce. You don’t just stick it into the cake and then all of a sudden everything is solved. You still need to map these technologies to fit the tool into the way the customer wants to use it. In this case, the way the tool does your modeling, is it based on how, for example like a purchaser, all the factors that they were using whether it was of a calculator or some kind of a manual process in Excel, I imagine that they have some kind of a recipe that they would follow to do this prior to you doing any type of AI or machine learning to help with that decision support. Is that how you help get adoption to be higher? Is it modeled on that or were you looking at other data? I’m curious especially around like if there’s experience or maybe—I don’t know if you’d call it biased—but there might be decision points that a human would be using in the traditional or the old way that they use to do those tasks that you can’t maybe perhaps integrate into the model. Does that make them not trust it as much even if maybe you’re actually factoring in more variables that they never use to like, “Oh, well, you never had weather when you forecasted crop and prices to figure out how much to purchase or whatever. You never had that. We actually provide that, but we don’t have last month’s purchase,” so you don’t know what the price was last month. That’s a bad example but are you following what I’m saying? Can you talk a little bit to that terms of adoption? Andrey: Before we even started, people were able to plan different promotions for the last, I don’t know how many years, and the main tool that they used was Excel. People with a great number of years of experience, they put together all these datasets for themselves and they’ve developed a certain tactic for how to approach the planning. Don’t forget, Lidl is operating in about 30 countries and each country has its own secret sauce on how to do the planning. As I said at the beginning, we had one person from one of the countries; one real planner who showed us how she does it for real, what factors they consider, which logic, and from then we tried to mimic the whole thing using machine learning algorithms. Of course, machine learning can take into account a lot more different factors than human planners. But nonetheless, the results that we got were of course slightly different because with human planners—I mean, it’s impossible to get the same number for a human and machine learning algorithm. They will always kind of say that, “Okay, why is it lower than I expect? I need to understand why.” The only answer we had at the time was, “Because the model said so.” That’s certainly not enough for them. Just recently, we’ve developed quite a good relationship with Polish planners. They have similar concerns, so they tried the tool, they like to tool, but again, for them it’s somewhat hard to start trusting the system immediately. They try to plan some promotions and it was fine for them but for some of them, they got unexpected results. You hit the wall; they start asking you all sorts of contrast-effects explanations. “What if it was different then what would have happened?” Without explainable AI or any kind of explanation of machine learning, you just cannot go forward with them. This is how I would put it. Brian: I do want to get into the explainable AI piece. I’m curious though, you mentioned Lidl [is]in 30 countries. For example, the purchasing department here, are there really 30 different recipes that are valid and/or there are cultural distinctions in the way stock is purchased for stores in Germany versus Poland or is it that just these Heuristic models for doing these were kind of organically-borne in different ways in each place ? Are you guys trying to centralize that or are you creating unique models? Are you trying to map it on like, “In order to get the Polish buyer to trust us, we need to show that our model is based on the way they were doing it even if it’s a different model than we use in Germany or Italy.” Can you talk a little bit about how you make those decisions and how you keep that trust that people are still going to use these decision support tools? Andrey: As I said at the beginning, each country has a small feature when they do the planning of promotions. Lidl is currently developing various tools in order to standardize this process, but it is an ongoing thing. Like with any person, be it the promotion planner, or a bank teller, or whatever, people tend to oppose changes; they just don’t buy these many cases. It takes quite a lot of time in order to convince them that, “Okay, this is the right way to do things. We’re suggesting [you a new way ]??that is not really new, it’s just kind of more generic,” so to say. Nonetheless, they [would..] say that, “Okay. We have this one data point that must be everywhere otherwise we just don’t take it.” It’s really a matter of time, matter of interaction with the client trying to convince them that if it’s so important then we can build it into the tool, then they will go along, and they will buy it. Or on the other hand, try to convince them that that is not important then maybe they, at some point, they will get convinced. These are all the different types of things that you have to discuss with the customer one-on-one. We actually do a lot of traveling these days, going to Bulgaria, going to Poland, Hungary. Every country, we try to talk to these people directly. Try to get the requirements directly from them and then show again the results back to them and say, “Okay, we did it for you specifically so let’s work together.” Brian: I think that it’s great that you’re going out and doing that one-on-one research with your customers because that’s another way to just build support is when they feel they’re being included in the process and you’re not imposing a tool, but you’re actually modeling the tool based on them that’s another way to increase engagement. I’m curious, do you find that the unique countries, the managers, whoever ultimately makes the decision on what tools or what model is going to be used to make the final decisions, are they interested in like, “Oh, look. Italy factors in the weather. They factor in this thing that we never thought to do. We never really gave it that much weight. Maybe we should do it that way?” [Is it 30 independent recipes and then you guys have been generalizing that based on the variables that you find have the most impact on the quality of the predictions? ]??Is that shared and the countries are aware of what each other are doing or is it more like, “Yeah, yeah. That’s nice but Poland is different. We want to do it this way and we know it’s right.” Andrey: It depends. Sometimes there are certain legal things that we have to take into account that are not quite transferable across different countries, so these things, we just cannot take them into account because then the tool becomes too specific for each country. But on the other side of things that we are able to generalize we just do it simply by trying to get more data from countries or blind gate or something like that. Brian: Talk to me about how explainable AI, this technology, [is]for people that don’t know what this is. Effectively, what we’re talking about when you’re doing things like showing a prediction from a model, it’s actually showing what some of the criteria were and perhaps how they were outweighed and how they had an impact on the conclusion that was derived by the system. Is that a fair summary of what explainable AI is? Why don’t you tell us instead of me trying to do it since this is a space that you’re interested in. Andrey: Absolutely. The area that use, kind of a research area of explainable AI, it’s all about trying to understand the reasoning of a black box model. This is not a new idea. It was quite popular back in the ‘90s but then was somehow forgotten and it resurfaced back in 2016, 2017 when [?? I think it’s a name of a company??].] announced a lot of funding for this area of explainable AI. What it actually does is for instance, let’s say you have a data scientist working on a sophisticated model, whether that be a neural network or anything else, and then it produces a prediction which is just a single number or it’s a binary decision, yes or no. In many cases, these black box models are really hard to explain to a non-expert. Even data scientist, in many cases, don’t know why it predicted yes versus no. There is no clear explanation or human-readable explanation that can be delivered in this case. Unless, the whole area of research of explainable AI is trying to, first of all, come up with the whole philosophy of what an explanation really is and this is not a done deal I would say. People are still trying to understand what it really means. The second part is, “Okay, how do we generate something that a human being can understand?” Whether it be, I don’t know, some factors, “Okay, for this prediction, Factor A played the biggest role and then Factor B played somewhat a lesser role,” and so on and so on. Or even generate a sequence of rules, if [??.] rules such that, “Air temperature is higher than 30 degrees, it’s the middle of the day, then the prediction for the sales of ice cream would be high.” What I’m trying to say here is that we use this technique in order to interact with our customers. For instance, when you go and talk to an operational person, a person who works in operations, who has in mind a certain number of basic rules; three, five, six rules when doing a planning. And then when you come to him with a machine learning model, something that is say, a black box, and then you tell him, “Okay. Just trust my prediction.” In most of the cases, it just simply doesn’t work. They just don’t trust it. But the moment when you come with an explanation for every single prediction your model does, you are increasing your chances of a mutual conversation between this responsible person and the model in this case. For instance, if the model predicts sales one, two, three, four, five for the May-June and then it says, “Why is it one, two, three, four, five?” And then you say, “Okay. It’s because if regular sales in June are greater than two, three, four. It’s May or June and something else, something else.” And then this person can relate these statements to something that he has in mind when doing plenty himself. This is where this kind of the eureka happens so to say because they see that the model is reasoning in a similar way as they do it. This way, the level of trust certainly goes up and then they’re willing to try it even more. I’m aware of similar stories. For instance, Yandex has been in the business of building similar tools for their customers and they also have explanation modules. It’s not a kind of a one-shot thing that we do at Lidl but it’s gaining quite a lot of momentum, I guess. Brian: I think that’s natural that trust and engagement is likely to go up if you have this in place because as you said, people can see that the tool is modeled on the work and the tasks that they want to do, and it’s not imposing a magic answer. It’s kind of like saying, “Hey, none of you here are experienced for the last 10 years of running promotions at Lidl matters anymore. Here’s what you should do. Here’s the product. Here’s the sale price and how long you should run it for.” I think it’s just human nature, there’s a natural tendency to not want to trust that, “Well, my whole job and these activities I do are completely replaceable by a magic box.” But when people start to see how it’s actually a decision support, I think it’s natural that the trust goes up. Although having said, I’m curious, you’re using some of these, would you say this is a regular ingredient in the products that you bake-up at Lidl? The data products that you guys are working on these tools[ or this is or is this an ]occasional thing? Why or why not would it be included on everything if it’s possible? Andrey: Well, there are different cases. Certainly, explainable AI is not something that you should or must use on every situation but I’m a great believer in decision support tools and human in the loop applications. That’s not necessarily in retail but in general. Every time where people have to look at certain predictions, we try to come up with explanations or at least some sort of a strategy for, “How can we come up with these explanations?” On the other side, these techniques are very useful when you do debugging of machine learning models. Even if you are not planning to show these explanations to anybody in the business, you are still benefiting quite a lot when you’re actually developing a machine learning model by using these tools, sort of just to fill in a few words. If you can avoid all sorts of overfitting in the model or removing plenty of features that actually make a model unstable and so on. I think the main point here is that in order to build things, models, what we don’t really understand a lot of what is really going on inside when they make predictions. It would be really sad, that’s the main idea. Brian: I haven’t thought of it that way, but I could see how, even as a debug, it could be useful as you’re trying to improve the quality of the decision; the advice that the tool is generating. I don’t know if you have data or even if it’s just qualitative in nature, but having included this on any of the products that are at Lidl, do you find people trust, like once they’ve seen that the model has some explainability behind the predictions that are being made, do they tend to still pay attention to all of that going forward or is it more like, “Oh, I can see that Andrey and his team factored in last month’s purchase data plus your competitor data and that’s what I always do. Now that I know that he always does that, I don’t really need to see that every time. I’m not going to second guess as much as I used to. I’m going to trust that now going forward.” Or do you find that the explainable AI, that portion of the UI, is actually an integral part of using the tool every time? Are you following what I’m saying? Do you start to ignore that over time or do customers see that as an ongoing, useful aspect of the interface? Andrey: We don’t have the explainability to build in into the user interface at the moment. We have it as more of a PoC, we show it on the month more or less but we don’t have it as a default feature. Certainly, this should be working in a way that you described. I actually have read a few papers recently about the effect of explainability. People were tested through, I don’t remember exactly the test set-up, but the point that the researchers were making was that the accuracy of predictions within this realm of human in the loop applications does not go up. Whenever people are using machine learning model that makes a prediction yes versus no, it—altogether the performance of this blend of human and machine—does not go higher but [well-significantly ]??that they could prove it. But the trust into the system can beat those by 20% higher than without any explanation. I guess what you said is exactly the point of building in explainable AI into any tool to make it transparent and then at some point, people trusted them, they don’t really have to check these explanations every single time because they know, “Okay, we are on the same page; machine and me.” Unless, they’re trying to explore some unusual situation where they really want to test the system or learn something from that, because well, this is also a possibility for a human in the loop application when humans actually learn something new from the system. I think this another case when they would use it occasionally after that. Brian: Without getting too technical here, obviously, Experiencing Data, this podcast, is more about customer experience with data products and that type of thing, but I’m curious there maybe listeners wondering, “Hey, I have this decision support tool or this analytics deployment that’s starting to use some machine learning, it doesn’t have any type of explainability. We are seeing the same symptoms you talked about like, low engagement, people don’t trust it, we put a significant data science investment in place, is it easy to retrofit in a technology investment you may have made to include some of this such that you might be able to start to improve the trust factor or is this something that really needs to be implemented from the start and it’s much more difficult to put in after the fact? Andrey: It all depends on the system itself right now. How many models you have there? What is the complexity of the whole thing? Technically speaking, there are already a number of libraries available for doing these things. Everything is open sourced. You can just Google for words like Lime or Shep or Anchore or contact me on LinkedIn for instance. I could point you out at various sources, but the point of explainable AI is that these tools are sort of model agnostic in a sense that they just need a prediction and they need just a model, and the predictions that model produces, and that’s pretty much all. Then you can write a few lines of Python and then it’s there. You can get your explanations. The short answer is you don’t have to invest too much into getting explanations if you already have a working system. Brian: Wow. That’s kind of a little bit mind-boggling. If you hear in terms of, “Why isn’t this being used more? Is there a perception that it’s costly, or it’s more difficult than it is, or is the quality not there, or do you think the business and the leaders and the people doing this don’t think it’s necessary?” It fascinates me if it’s that simple and of the quality, if this is such an easy way to build trust, why is this not happening more? Did something happen in the night between the ‘90s and now that this kind of fell out of trend? Can you talk to that a little bit? Andrey: In my personal opinion, I think over the recent 10 years, data scientist put too much attention into getting high model performance in terms of accuracy or lower error. This whole trend of toggle competitions where you try to build super accurate model and say the opponent who gets the first price is probably hundredth of a percent more precise than you, but the main question is, “Does it actually make sense? Is it what business really wants?” In my opinion is, it’s not the case. Yeah. For scientific breakthrough, maybe yes, it is useful. But certainly, in order to gain trust, you cannot just build more and more sophisticated high-precision model. It just leads you nowhere. The other thing is that over the last four or five years, the deep learning hype took place and a lot of attention was in the deep learning area where people were all in doing neural networks and nobody really cared about explainability. It’s more of, “Okay, let’s just predict this to 99.9% of accuracy.” At some point, some executives realized, “Okay, we have a lot of these models and we have no idea what is really going on inside. As I mentioned, the DAPPER program and also GDPR regulation that came to light in May 2018, put the spotlight on the right to explainability, the right to explanations, all of these factors together propelled the explainable AI topic forward and it’s now gaining a lot more attention than before. Brian: In terms of when you get into neural nets and some of those technologies, is this technology available to products that are leveraging neural networks and some of this more complicated artificial intelligence technologies? Is it widely available to add the explainability portion? Andrey: Well, it depends on what kind of data you’re working with. If you’re working with this regular table or data, the data that is in tables, no text, no images, then it doesn’t really matter. You can take in neural net or any other model but once you go into more sophisticated realm of neural nets working with text data, then it is slightly more complicated to get it to work but it’s still possible. They could just use lime or shep. It’s very interesting what they do actually. For instance, if you try to say classify legal documents or medical documents or say fake news classifications and write yes versus no, then these explainability tools can highlight the actual words in the sentence that play the most role in terms of, “Okay, if it’s fake news, then it will underline certain words,” that a human being can get them on and say, “Okay, this content is maybe more full of feelings or calling for more action,” or something like that versus a prediction for no fake, and it’s mostly facts, facts, facts. Basically, the explainability tools, they highlight words in a sentence. In terms of images, even more complicated. There are a lot of things that are model agnostic but even more things that are not. In this case, you really have to be an expert in whatever neural net and try to get it to work. Just get a code from GitHub and try to reproduce the results of a research paper. For these more complicated cases, it’s not that easy but it’s possible. Brian: You brought up something too. I’ve heard this trend repeated too that there can be a tendency in the data science community to want to do excellent data science work and not necessarily do excellent business work—building tools and solutions that help the business. I could see how maybe some data scientists may see that’s not going to improve the model adding the explainability. “I can’t write a paper on that and build my credentials with that type of information,” so maybe that’s responsible for why it’s not as widely in use. Do you think that’s going to change? Do you think this will become more of an expectation going forward that, we won’t be talking about black boxes as much in a year from now or two years from now, do you think that’ll start to go away and expectations will change? Any thoughts on that? Andrey: The whole trend is going into the direction of explainable AI anyway for one simple reason. In the last years, AI was mostly used in the labs and probably to automate certain processes where no humans are really involved. It’s more like robotics or something like that. But these days, AI is going into various fields like healthcare, or legal domains where you deal with things that affect humans directly. For instance, how would you explain why a certain person didn’t get a low at the bank and the other one who looks very much similar got it, right. There are a lot of questions that are coming up these days because AI is touching upon some points where humans are personally involved. Because yeah, we don’t really care how some robots are moving goods at Alibaba warehouse. I mean, doing an explainability for that, yeah, maybe it’s a really sophisticated model but I don’t care. I get my goods; I order them and that’s it. But whenever things go into the direction of some social interaction or things that affect people directly, or say these high-stake decisions, then interpretability and explainability is a must. I think many people would probably choose a model that is maybe not as exact and accurate but explainable versus something that is extremely accurate but okay, sometimes it can kill you. This is kind of my logic here. Brian: That’s kind of the dividing line between the business and the human side of it. The pure data science side is that you might have a super accurate model but if you find out that they’re still buying carrots the old way, does it really matter that you have an excellent prediction on how many carrots to buy or at what price if they don’t ever take advantage of that. All of that investment is kind of thrown out the door. From a business standpoint, an acceptable model quality with a highly trusted interface and user experience might be the better business decision even if it’s not the best model quality from that perspective. I think that’s important stuff to consider in all of this as we build these tools. This is awesome. It’s actually exciting from a design perspective to hear that this is available as kind of a tool that we can implement. Obviously, context matter,and you have to look at particular domains and particular types of data and all of that but it sounds like something that—as part of our tool box—it’s something that we should be leveraging regularly when possible especially if we’re talking about human in the loop tools and decision support tools as opposed to as you said, the Alibaba robots… Andrey: Yeah. That’s [ I didn’t hear a word here]. Brian: …get my book or my shoes or whatever I ordered. Just on this topic here as we wrap up here, any broad level advice for people that are kind of looking to jump into this on making efficient data products? It doesn’t have to be just products explainable AI but just from you experience at Lidl. Maybe a mistake that you’ve realized, and you changed, and how you’re approaching your projects and building these tools. Any advice for people? Andrey: I guess the only advice I would give to anyone who wants to build a product that is used is to go to the people and ask for the actual requirements; try to involve the end-user at the earliest stage possible. I think this is the only way to succeed in the end. This is how startup fail or succeed. You have to really understand what you’re doing. At Lidl, we’ve kind of cleared the way from zero to hero over the last two years and we learned it the hard way. The more you interact, I mean it doesn’t really matter to the people because data product is a piece of software that has machine learning inside of it or algorithms in the end. Nobody really cares about how sophisticated these algorithms are. People just want to make sure that they can get the job done efficiently or have a nice experience, no bugs, and stuff like that. This is [all the same stories that was probably 10 years ago this is all the same story as 10 years ago probably]when we didn’t build any data products,[ that build]?? just regular software but again the main advice is just don’t try to build or create this ‘moon-shotty’ product within a few months and try to reiterate and try to onboard your users as early as possible. This is the main advice that I could give. Of course, use explainable AI in order to convince and gain trust. This is a must in my view. I mean, of course in the cases where someone is doing something, and no one can see him. If it’s an interactive tool, interacting with users, they have to be sure that it is doing the right thing. Brian: Great advice. People that have been on my designing products mailing list and the Experiencing Data podcast have definitely heard this advice beaten into the ground many times about getting out there and talking to people early in the process to inform what you’re doing and not working in isolation because that’s almost a sure way to produce something that people aren’t going to use because it’s full of your own bias about how things should be done, and it’s not informed by what customer wants to do. Good words, good parting advice. Where can people find out more about you? Are you on Twitter? Do you have a website, LinkedIn, anything like that? Andrey: Well, I’m posting quite a lot on LinkedIn. I have a group dedicated to explainable AI; it’s called Explainable AI-XAI. Everyone who’s interested or learning more, feel free to join or contact me through LinkedIn. I don’t tweet much on Twitter. My presence is mostly on LinkedIn at the moment. Brian: Great. I’ll definitely put those links to your profile and to the Explainable AI group on LinkedIn on the show notes. This has been really great, Andrey. It’s been great to talk to you. Thanks for coming on in Experiencing Data. Andrey: Thank you, Brian. Thanks for inviting me.

011 – Gadi Oren (VP Product, LogicMonitor) on analytics for monitoring applications and looking at declarative analytics as “opinions”

Play Episode Listen Later Apr 23, 2019 46:22

My guest today is Gadi Oren, the VP of Product for LogicMonitor. Gadi is responsible for the company’s strategic vision and product initiatives. Previously, Gadi was the CEO and Co-Founder of ITculate, where he was responsible for developing world-class technology and product that created contextual monitoring by discovering and leveraging application topology. Gadi previously served as the CTO and Co-founder of Cloudscope and he has a management degree from Sloan MIT. Today we are going to talk with Gadi about analytics in the context of monitoring applications. This was a fun chat as Gadi and I have both worked on several applications in this space, and it was great to hear how Gadi is habitually integrating customers into his product development process. You’re also going to hear Gadi’s interesting way of framing declarative analytics as casting “opinions,” which I thought was really interesting from a UX standpoint. We also discussed: How to define what is “normal” for an environment being monitored and when to be concerned about variations. Gadi’s KPI for his team regarding customer interaction and why it is important. What kind of data is needed for effective prototypes How to approach design/prototyping for new vs. existing products Mistakes that product owners make falling in love with early prototypes Interpreting common customer signals that may identify a latent problem needing to be solved in the application Resources and Links: LogicMonitor Twitter: @gadioren LinkedIn: Gadi Oren Quotes from Today’s Episode “The barrier of replacing software goes down. Bad software will go out and better software will come in. If it’s easier to use, you will actually win in the marketplace because of that. It’s not a secondary aspect.” – Gadi Oren “…ultimately, [not talking to customers] is going to take you away from understanding what’s going on and you’ll be operating on interpolating from information you know instead of listening to the customer.” – Gadi Oren “Providing the data or the evidence for the conclusion is a way not to black box everything. You’re providing the human with the relevant analysis and evidence that went into the conclusion and hope if that was modeled on their behavior, then you’re modeling the system around what they would have done. You’re basically just replacing human work with computer work.” — Brian O’Neill “What I found in my career and experience with clients is that sometimes if they can’t get it perfect, they’re worried about doing anything at all. I like this idea of [software analytics] casting an opinion.” — Brian O’Neill “LogicMonitor’s mission is to provide a monitoring solution that just works, that’s simple enough to just go in, install it quickly, and get coverage on everything you need so that you as a company can focus on what you really care about, which is your business.” — Gadi Oren Episode Transcript Brian: Alright, welcome back to Experiencing Data. I’m excited to have Gadi Oren on the line from LogicMonitor. How is it going Gadi? Gadi: It’s going great. Thank you for having me. Brian: Yeah. I’m happy to have you on the show to talk about not just monitoring, but you’ve done a lot of work on SaaS, analytics products in the monitoring space, software for IT departments in particular. Can you tell us a little bit about your background and what you’re doing at LogicMonitor these days? Gadi: Too many years in different industries, I actually spent multiple industries starting from medical imaging. Let’s say in the recent 18 years mostly, some sort of monitoring solutions. I dabbled also a little bit with marketing data analytics. That was not a successful company but I might draw some examples from there. Right now, I’ve recently joined LogicMonitor for an acquisition. I was the founder and CEO of a company called ITculate here in Boston. That company was acquired in April by LogicMonitor and I’m now the VP of Product Management. What LogicMonitor is doing is solving a fairly old problem that still remains, which is monitoring is really difficult. Many companies, as they go, they reach the point where they realize how important it is for them to monitor what’s going on in order to be successful. Then they realize that it’s such a complex domain that they need to develop expertise. It’s just all around difficult. LogicMonitor’s mission is to provide a monitoring solution that just works, that’s simple enough to just go in, install it quickly, and get coverage on everything you need, so that you as a company can focus on what you really care about, which is your business. Brian: Obviously that’s a hard problem to solve and I’m curious for people that are listening to the show. I imagine a lot what this product is doing is looking for exceptions, looking for things that are out of bounds from what some assemblance of normal is, and then providing that insight back to the customer. Is that a fair evaluation? Gadi: It’s a fair evaluation. There is obviously the question of what is normal, but in general, providing that there’s many ways to define what normal is, then the answer is yes. It’s the ability to give you visibility into what’s going on, first of all, to just see that things work in general and work okay, and then when something goes out of what you define normal, to notify you on that and help you with getting things better. Brian: From your experience in this space, since in a lot of companies that are doing analytics, it may be difficult to define the boundaries of what normal is, such that you could do something like, “Oh, we’ve detected an abnormal trend in sales.” I can’t think of something off the top of my head but I like the idea that the focus of the product is on declaring a conclusion or driving an insight that’s probably derived from analytics that are happening in the background. I would put that in the camp of declarative analytics as opposed to exploratory where it’s like, “Here’s all these data. Now you go and find out some interesting signal in it.” Most customers and users don’t want the latter, they want the tool to go do that job. Do you have any suggestions for how companies that maybe aren’t quite in a domain where it’s black and white, like a binary thing, like this core is either connected or not—if it’s not connected, that’s bad and if it is, it’s good—is there a way, a kind of approach putting guard rails on things or what normalcy is? Do you follow what I’m saying? How do you move into that declarative space? Gadi: The answer is obviously, it depends. The problem is so difficult that you are even having a hard time defining the question. It is very, very difficult. By the way, you called it declarative, I like that. I actually call it in a different way that usually creates a lively discussion. I call it opinionated. Opinionated system. The reason is there’s some evolution, especially with regards to monitoring. But I think it’s the same for other type of systems that are analytic-based. Ten, fifteen years ago, it was so difficult to just gather all the data, that being non-opinionated or nondeclarative for your definition, was pretty good, because people just needed the data and they bought the context themselves. But there’s been a lot of changes since that time. First is the availability of computing. But the other is also the need is much greater now for giving the opinion, giving the bottom line. The system needs to be opinionated and then it can be a variety of things. It’s really multiple types of algorithms that can be used here. A small set of that is what people define today as machine learning and AI. But actually the domain is much larger than just AI. It’s the ability to look at multiple signals and develop in a certain degree of confidence, a conclusion that is derived from those multiple signals. To put it in the most generic way that I can, some of them can be discreet or binary, and some of them can be continuous. How do you look at all that stuff and say, “I think that this is what’s going on”? And even more than that, here is what we think is going on, here’s what you can do about it, or here are a few options for you to act on it. That is the ideal solution that we would like to have. Obviously, we have very, very little of that right now. I think it’s a journey that will take us years to get. Brian: I love that idea of opinionated because I think it softens the expectation around the technology and it also reminds people that it’s never different than when your plumber comes to house and you’re like, “Well, the shower hot water isn’t quite as hot as the sink water.” Maybe he tighten some things, he turns on some faucets, and he gives you an opinion about what might be wrong without actually tearing apart the whole system. We would tend to trust that. He might say, “Well, what I really need to do is X.” Then you make a decision whether you want to pay for that or not. But it’s not like, “If you can’t give me a 100% decision, then I’m unsatisfied.” We accept that opinion. What I found in my career and experience with clients is that sometimes if they can’t get it perfect, they’re worried about doing anything at all. I like this idea of casting an opinion. On that thought then, especially for example, you’re putting a data model in place or something like this is which is going to learn from the information, there may be insights gleaned from some type of computer-based analysis which may be unseen or unexpected by the business. That could be positive or negative. But there also might be some context of what normal or expected is from the end-users. For example, I expect the range to be between 32 and 41 most of the time. I know sometimes it goes up and I have this feeling about X, Y and Z. They have something in their head, you go and do all these technology and it says, “Well, the normal range should be 14 and so we flag the 16 here,” and he’s like, “I don’t care about that. It’s not high enough for me to care.” How do you balance that sense that maybe an end-user has, like, “I track sales,” or, “I’m doing forecasting,” and they have all these experience in their head? Gadi: A couple of things. It depends a little bit about the domain. In some situations where the end result is what’s really important, you can use black box-y type of things like newer networks or things like that, not always but most cases, they tend to be more like black box. It’s like, “Here’s the result. I can tell you why that happen. It’s based on all these training I did before.” In some situations, the result cannot be a black box. It needs to be explained. In those situations, you really need to give people an explanation on why things happen like they are. Monitoring, in many cases, tends to be the latter which is, I want to see the signal and the signal core strength on what was the shape and I want to see how it looked yesterday. If it looks the same, maybe it’s okay. In certain situations, maybe if you have a database and it has a latency of half millisecond which is very small, and then this morning it moved to one millisecond, is that normal? It’s not normal but I don’t care about it because before it gets to five milliseconds, I don’t need to know about it. In those situations, I don’t know if that’s what you refer to by cobwebs or things like that. While the system is learning and can automatically detect what’s abnormal, there is a range to what I care about. I’m going to put a threshold and say, “Only if you cause five millisecond and this is not normal behavior, then I want to see an alert.” Normal could be defined like the signal is two sigmas away from the same day last week. Something like that. There is a different level of approaches, both in terms of how consistent the data processing is and in what type of knobs you should provide to the user in order for the user to develop the right confidence level to use that solution. Brian: I would agree with that. I think there’s a balance there. Actually when we talk about our screening call, you made a comment. It was a good quote. It was something like, “The cutting edge UI is English,” if I recall. Gadi: Or any other language, yes. Brian: Exactly, whatever your interface is. But I would agree with that sentiment that I think the customers and from user experience standpoint, deriving that conclusion first or opinion as you said, then backing out from there, and providing the data or the evidence for the conclusion is a way not to black box everything. You’re providing the human with the relevant analysis and evidence that went into the conclusion and hope if that was modeled on their behavior, then you’re modeling the system around what they would have done. You’re basically just replacing human work with computer work. What I found over time was, some of these systems, with just watching customers, is they’re very curious about the beginning and if you can build that trust, they start to understand how to trust your opinions. They tend to not hold you responsible as much if an opinion is wrong because they know what went into the math and to the analytics and also what didn’t go into it, steps that they can fill the holes in themselves. Of course, this means you have to know your customer, you need to have some kind of interaction with them. Can you tell me about some of your customer interactions? You’d mention one of your KPIs for your team. Tell us about one of your KPIs for your team. Gadi: Over the years, obviously, you develop professionally and you change the way you approach to do what you do. I’ve been doing different ways of product management. I was a CTO at some point. But my basic attitude is doing product management and building the product from that understanding of what it is we want to build. What I’ve realized over the years is that there is a couple of really important points. One is the more you talk to customers, the more you understand the problem you’re trying to work on. A couple of years into working on a certain problem, you get to a point where you’re so familiar with it that you can pretty much, without talking to customers, generate a lot of really good product for some time. The problem is that this might diverge at some point or you’re going to miss something important. I think that talking to customers all the time is what grounds you to what’s going on. I’ve made my team of about 10 people. We are monitoring how many times they have interaction with customers. I’m going to chart it monthly. I started recently as one of the KPIs and I’m going to just check if certain part of the team is talking less to customers, then why is that okay or not. And then if you have a spike, some of these stopped talking to customers, then we’re going to have a discussion on why that happened because I think ultimately, it’s going to take you away from understanding what’s going on and you’ll be operating on interpolating from information you know instead of listening to the customer. Brian: Is it safe to say your team is comprised of primarily other product managers on certain portions of the product and then see up some design user experience reporting to you? Gadi: Yes. I have, let’s say, about 10 people. We’re hiring now all the time. The company’s growing very quickly. Let’s say around 10 people. Most of them are product managers and some are design people. We also have a variety of previous experience in the team, which is really something I liked. Some of them are from the industry and they have built-in knowledge. Some were in engineering before, which I think has also an interesting experience. One or two came from being sales engineers or sales, which has a different aspect of benefit to it. I don’t think I have someone that was a customer before, and if you can have that, that’s really advantageous. I might be able to do that at some point. Brian: I think that’s great. Do you involve your engineers or your technical people, data scientists, whatever with any of these interviews that you do and your customer outreach? Gadi: As much a possible, we will look at the multiple sides and a lot of engineers I work with right now are located in China. In terms of language, we might have sometimes barriers, but absolutely when possible, I know that when I transitioned from engineering to product management, the exposure to customers was very educational for me. Whenever I am able to expose people to customers, I take that opportunity. Brian: You have an interesting position. Maybe this is super common, I don’t know, but you’ve started out in a technical capacity, you have an engineering background, you were CTO, and now you’re in product. I’m curious. As someone that’s looking at holistic product both a business and also some kind of experience you need to do, you need to facilitate in order to have a relevant business. Are there biases that you need to keep in check from your technical background where the engineer in you says, “I want to do X,” and you’re like, “No, no, no”? What are some of those things to watch out for to make sure that you’re focused on that customer experience and not how it’s implemented? Gadi: The question of biases is a wide one. It’s not just about engineering. It’s bias in general. At some point, you obviously get excited about what you’re building and you see all the possibilities. “We can do something a little here, we could solve that problem or this problem,” and then you start developing a preference. It’s very natural. When you realize that this is the case, sometimes you try to just not have a bias, but you can’t. Everybody has one. The problem is, how do you make sure that this bias does not impact when you talk to a customer? It’s very easy to have a customer tell you what you want to hear. Probably the easiest person to deceive is you if you don’t pay attention. This is one of those things. With regards to engineering bias, it’s not very different than any other type of bias. Engineers and makers just really care about working on interesting things and new technologies. Sometimes, there’s a problem and I think the more advanced engineers start to think about, “How would I generalize that problem?” It might be a runaway process where they want to build more than required and that more may or may not be pointing at the right direction. That’s another type of bias. Again, definitely something to watch for. Brian: I want to move on to some other topics only because I can totally spend an hour talking about how important it is to do customer research. I love that you’re doing that and I think the theme here is you’ve actually turned that into a KPI for your particular reports and the product management division at your company, which says that it’s important to develop that habit. I would totally champion that. Gadi: It’s not that I would like it to be. I can tell you I would love it to be. Since you’re opening this, I’ll tell you what could be ideal. But it’s a lot to ask for so I’m not implementing that right now. I do check that people interact with customers and they have written down notes. Written down notes should not only be two lines. It should be telling something. Ideally, somebody can transcribe what the meeting was, but that’s almost impossible. I try but it’s very difficult. What I would have loved to do that we don’t do right now because it’s a lot to ask for, I’d love to have people repeat in their heads and in the notes the meeting and try and extract problem statements. In the past I’ve implemented that in some situations and it was successful. But you have to do it in a continuous fashion over a long time and then you see those problem statements. How many references do you have to every problem statement? It’s really giving you a good visibility into what’s going on. Now, asking that is difficult but clear notes is a good start. Brian: Just to tack onto that, it can be very hard to listen attentively and to draft notes. When I’m facilitating research sessions with a client, is you’ll have one person facilitating and one person taking notes, and then you debrief at the end. Sometimes, it does mean it’s a two-on-one instead of a one-on-one. It doesn’t need to be perfect. You can get better at this over time. That’s one way to get a little bit of a higher quality data. You can also just use something like an audio recorder on your phone instead of handwriting the notes. When that meeting’s over, you grab a phone booth or type room and just talk into a phone. Then you can just have the audio converted to text very simply and quickly with a machine. That way, you’ve got a nice dump of what the conclusions were from the sessions. Traditionally in the usability field and the human factors field, they came out of science background, so they would write these very long reports. Typically, what happens is, guess what, nobody reads the reports. So, you have to watch out for that. We’re doing all the stuff but we’re not taking any action on the information there. I like to highlight real concept, however you go about doing that, but that’s great. Can you talk to me a little bit about engagement with these data products? This is probably a little bit truer in companies that are deploying internal analytics, like non-digital native companies, non-product companies, but they’re having trouble with engagement. Customers aren’t using the services. Do you have any broad ideas on how we can increase engagement from your perspective? How do you make the tools more useful, more usable? What do you have to say about that? Gadi: How do you make the tools more useful? I think it’s a somewhat related question to how do you make your products successful to begin with? I’m going to talk about the new concept other than incremental. If you learn something incremental, I’m assuming you have enough data to place your bets successfully. If you learn a fairly new concept, what I would usually recommend is don’t code it. Try new simulations, Exceling, and modeling, whatever it is that you can to build it without building it. Prototype it and then have a few lead users. Those are users that are excited about this domain and they really care about solving that specific issue enough to work with you effectively. You need two, three, or four of those and you just start working with them. As much as possible, use their data. In the data domain, when we’re doing analytics-related product, part of the user experience in the entire cycle. It’s not just, “Oh, the user interface is the biggest expense.” No. It’s how the data is getting into the system, how is it being acquired, how is it being processed, and how is it being used on the other side when it’s producing meaningful insights. You can test a lot of that cycle without a product or with a very light sort of a product, prototype of the product. I recommend that as much as possible. If you’re going the right way, you will know very quickly and if you’re going the wrong way, also you will know quickly and you can either course-correct or eliminate completely the project and save a lot of time and money. That’s usually something that works really well for a new concept. Now, for incremental, it’s slightly different. Usually, you can use a similar type of approach but you can code something that’s kind of a prototype into your product, show capability, and then usually, you would have a lot more customers that are willing to work with you because it’s a small increment. You can validate early. I guess that’s the bottom line here. Experiment, iterate, validate early. Brian: How is it necessary to code? I don’t mean to use the word code, we’re talking about Excel or whatever it may be. It is necessary to get even into that level of technical implementation in order to do a prototype? I love the idea of working with customer data because that removes some of the classic example I’ve experienced like financial products where I was working on a trading system portfolio management. You’d have a bunch of stock positions in a table and you’re trying to test the design of the table. You have funny prices for, “Why is Apple stock trading at $12? Oh my God, what is going on?” That has nothing to do with the study but you’ve now taken the user out of the— Gadi: You will not get a meaningful […]. Brian: I love that but do you need to necessarily get into modeling and all this kind of stuff if, for example, the goal is to see, would that downstream user take action or not, based on what they’re seeing in the tool if you’re using a paper prototype or something like that? What would you do if it says it’s predicted to between 41 and 44, what would you do next? And you happen to know that that’s a sensitive range. Do you even need to have actual Excel or math happening behind the scenes? Gadi: I can see where this question is coming from. Nine-tenths is just putting a […] and mock-up might really give people a good feeling about where you’re going with this. But I do think that in many cases, not working with real data and even customer data—customer data is not a must—is not going to give you the right answer. I’ll explain when this can happen. There is the case that you mentioned, which I wasn’t even about to mention it, but it’s too late, is the data that you see doesn’t make sense, you’re emotionally detached. You’re not getting good responses from this person. Now, assuming the data is good and if it’s yours, you’ll even connect it much better to what you see. But certain type of problems, you cannot understand, you cannot get a meaningful answer if the data is not real. I’ll give an example. Right now, we’re facing a very specific situation where LogicMonitor is actually now in the process of redoing the UI and fixing usability. I’m told that this is the fourth time we’re doing it and there is a very specific problem of how to do search. We’ve been going back and forth on how does the search results should really show up because the search results are coming back in multiple levels. There’s data with dependency. Results are coming from multiple levels of dependency and need to show up on the same screen in a way that the user can use it. We’ve got to the conclusion that the problem is hard enough to answer and we need to prototype maybe one, or even two or three types of result presentation and just show it to customers. Obviously, we want to code as minimum as possible to do that, that this is something we’re going to do. We usually do it with just wireframes but in that specific situation, we are not only needing real data but we’re actually coding something very minimal, three times, to get the right answer. Brian: I think the theme here is, whether it’s code or whatever material you’re using, there’s a theme of prototyping. I would add that in the spirit of a minimum viable product or what I would call a minimum valuable product—I like that better—is figuring out what is the minimum amount of design, which could include some technical implementation like a prototype, what’s the minimum amount that you need to put out in front of a customer to learn something? To figure it out if it’s on the right track? That’s really what it’s about. In your case, maybe it does take actually building a light prototype, maybe you don’t actually query 30 data sources, and you just have one database with a bunch of seed data in it. You control the test, but at least it simulates the experience of pulling data from many places or something like that, then you can tweak the UI as you evaluate. Gadi: I think it’s a bit of going specific. I think that, again, in many cases you can do wireframes and you’d be fine. But remember, if you’re trying to test a complete flow, you’re testing a flow which may include 10–15 steps of the user for the user interface, and if you can do that with just wireframes where every step that they did produces result to makes sense, and you don’t have to model it at the background, then that’s fine. It’s probably better. But when we are talking about 10–15 steps, sometimes, the amount of effort that goes into the wireframe is big enough to consider a very light background Excel implementation. When it becomes comparable, if doing wireframe is 80% of doing a very light implementation where the background is Excel, or even 70%, then I’ll say, “Hey, let’s do a little bit more of an effort,” and then our ability to test opens up to a lot of other possibilities that are not rigid within that wireframe. So, something to think about. Brian: I would agree if you can get a higher fidelity prototype like that, with the same amount of effort. Absolutely, you’re going to uncover probably exceptions. You’re going to uncover information and an evaluation with a customer that you didn’t probably asked about. There’s so much stuff going on and there’s so much more information to be gleaned from that. The main thing is not falling in love with it too early and not overinvesting in it, such that you’re not willing to really make any change to it going forward. I find that’s the challenge with especially with data products. I’m sure you’ve experienced, there’s a tremendous amount of investment, sometimes, just to get to the point where there’s a search box and there’s data coming back. At the point, you can fool yourself and say, “Oh, we’re doing design iterations,” but in reality, no one really wants to go back and change the plumbing at that point because it’s so hard just to get to that first thing. I think the goal is to not build too much and be aware of that bias to not want to go back and rework what maybe a difficult, “Oh, it’s just a search box.” It’s like, “Yes.” But if no one can get from A to B, then the entire value of the product is moot and it sounds like, “Oh, it’s just a search box.” There’s a lot of stuff going on with getting them from A to B in the right way in that particular case. Gadi: I totally agree. I’ve seen a lot of managers do that mistake. I bet that I did that mistake once or twice in my career. “This is awesome. It looks great. Package it and let’s ship it.” That is a big mistake because then people are telling you, “No, no, no. This is just a proof of concept.” Managers sometimes cannot understand the difference, so it’s a communication problem, it’s setting expectations, and you’re right. Sometimes, the way to avoid it is just not getting to the […] at all and I agree if you can. Brian: Traditionally, from my work and the domain that you’re in, the traditional enterprise tools is that quite frankly, they can suck. The tolerances for quality was quite low, and I think that’s been changing. It’s a slow growth that the expectation that these tools can be hard to use, they’re supposed to be really complicated, and they’re for the very technical user, that’s changing. The customer and end-user is more aware of design. I’m curious. Do you find that that expectation is going up? And do you find that new technology is making it easier to provide a better experience? Or is that being negated by the fact that, in your particular domain, you’ve got cloud and on-premise? I can see the challenge is going up. Just as well as some of the tools might get better, the challenge might get harder, too. Is it net out? No change? What are your thoughts? Gadi: No. It’s not the opposite vectors that are actually pointing to the same direction, I think. What you’re saying is, I believe, no longer the case. I don’t know. Maybe in some old banks somewhere in Europe where they’re old-fashioned. I still heard that some banks in Germany are based on paper, no computers. That’s why I made this comment. But in my mind, this is long gone. It’s multiple trends of really pointing to the same direction. First of all, people are educated by Apple that you can in fact have a product that’s pleasant to use. Some young people in their 20s and 30s, most of what they’ve seen is really a lot better that what you and I have seen, being slightly older than that. Expectation is to have good products. They’ve seen that hardware and software and combination of those things can be done well. That’s one. The second is that the technology is evolving, especially in my space, there’s been virtualization, then there’s cloud, then there’s containerization, and so many big waves that are changing everything, that you constantly have to refresh your software and the ability. The users inside the enterprises are now replacing stuff much faster. They’re replacing the infrastructure much faster, and then with that, they replace software and adopted much faster. The barrier of replacing software goes down. Bad software will go out and better software will come in. If it’s easier to use, you will actually win in the marketplace because of that. It’s not a secondary. It’s one of the things people care about. They don’t care about the user experience specifically. They care about being able to complete their tasks. They don’t care how that happen. If it was easier to achieve what they need and it left them with a good feeling, it’s a better tool. That derives better usability. Everything is pointing to the same direction because you have to refresh as a vendor. You have to create software faster to adjust to the new waves of technology that’s coming in because that’s part of being competitive. While you’re at it, you have to take care of creating really strong usability because then you will have another advantage in the marketplace. I think those trends are only enforcing the same direction. You have to have great usability. By the way, usability is not limited to user interface. It’s everything. User interface is just a part of it. Brian: I didn’t want to bias my question to you, but I would wholeheartedly agree that the tolerance levels for really difficult software or software that doesn’t really provide the value clearly or quickly, the tolerances for that have gone down quite a bit and I think you’re totally spot on that consumer products have created an expectation that it doesn’t need to be that complicated. A lot of times, there’s a service to language like, “Oh, it’s ugly,” or customers will comment sometimes on the paint and the surface interface because they don’t necessarily have the language to explain why. It actually may be a utility problem or just a value problem. I think the importance here is, as you said, usability is important, but it’s not just about that. Ultimately, it’s about whether value is created. So, if you write a decision support tool, like a declarative decision support tool in your case, it’s probably often about minimal time spent using the tool, maximum signal when I do have to use the tool, and the best case scenario is probably never needing to go into the tool to begin with. That’s actually the highest business value. You can focus all day on UI, maybe it’s a one-sentence text message is really the only interface that’s required and you might deliver a ton of value with just that. Gadi: To add on what you have said, I totally agree. I actually see in the marketplace LogicMonitor is winning deals that are based on ultimately better usability. I can give you an example. A lot of customers that we see, companies are growing and they start monitoring using a few open source tools—there are so many of them—and when you’re small, like, “This is awesome. I’m going to use this open source tool and I have the problem solved.” The company grows and at some point, the open source tool, you realize that you spent so much time maintaining it and so much time on making sure that the tool keeps on working when you add another resource to the network, I think the old expression was, ‘Tool Time versus Value Time,’ or something like that. Brian: Tool Time versus Goal Time. Gadi: Goal Time, exactly. This is much bigger than user interface. This is about the whole experience, which means that in those open source tools, you need to have a team of five people that are chasing all the changes that happen in the organization and you’re never there. You never actually up-to-date with what’s going on. From a very high-level perspective, this thing just doesn’t work. It doesn’t work because of usability. So, we come in and as I’ve stated what we’re trying to do, we’re trying to do something that just works, what’s well and quickly, and you’re able to deploy quickly, automatically discover the changes, and follows what’s going on. People are amazed by that and it’s part of the rationale, depending on the problems that they have. But in many cases, it’s part of what makes them buy our product. At the same time, part of our product had older user interface. I think that the comment that you said before, where you said, “Oh, it’s an ugly UI,” or something, I think there are situations where products that are a bit older might have pieces of user interface that are not as great, but their overall experience is so good that it carries the product forward. I think in organizations that have a choice, they might actually opt for a product that, on a first glance, might look not as great from a UI perspective but overall their experience is good. I’m not saying that this is the situation with LogicMonitor because we actually have pretty good UI as well, but I think that our UI reached a point where it needs to be improved and that’s what we’re doing now. Brian: May I ask a question on that just as we get towards the end here. If you’re able to share, what are the outcomes that you want to get from the new UI? There’s some business or customer impact you’re probably looking for, right? A business justification. Gadi: There is. It’s fairly complicated because it’s also a very expensive process. There are very qualitative things that you start hearing like, “Oh, your user interface looks old,” or people tell you things like, “Your customer X is much easier to do a certain task.” I like that better because it’s a lot more specific and they can explain why and all that. But in many cases, you just get like, “Oh, this other company has a new UI and it’s so much more pretty and cool.” That’s very hard to measure and very hard to act on. We have some of that but more specifically, I think, LogicMonitor’s also moving from the mid markets to more and more enterprise. As that happens, certain things that used to be okay are no longer okay. The amount of data that we’re dealing with on the screen, how we process and present it, when you have a couple of hundreds of items, you can think about a tree or a table. When you have hundreds of thousands, then the entire thinking process is different and you need a complete different method. Doesn’t mean all those changes like trends we’re moving upmarket in terms of size, we expect a lot more data in the user interface. People are telling us that the UI looks a little bit old. We want to refresh also the technology. We can do other things. If you’re doing things that are mostly server-based, then UI tends to be more static. It doesn’t have to be that way but it’s an engineering challenge. But if you’re moving to the more new frameworks like React, Redux or things like that, you can do a lot more dynamic. Every component can take care of itself, its data, its model, and update asynchronously. It opens up the product to do things that are a lot more responsive, like a one-page application, for example. A large part of the light business logic is actually done on the client side rather than on the server side, so it makes a much better user experience. All those multiple causes, multiple trends that lead us to the conclusion that we need to refresh. Brian: Was there a particular business outcome, though? For example, are you having some attrition and you’re looking to stop that? Or do you think this is the way to start facilitating sales, to close more easily with a better UI or anything like that? Or is it mostly qualitative? Gadi: No. Obviously, we try and quantitatively justify stuff, so we look at all the requests from the last two years and how many of them are related to UI and certain things in the UI that are very hard to do today. Yes, we do think that this will encourage sales for certain reasons that we will improve in the UI. I think that over the years, because there are so many people changing things in the product, I think that some of the consistency have dissolved along the way. In most cases, you do things the same way but in other cases, you do it a little bit differently. That is both in concept and the UI. That’s confusing for new users. Old users don’t care. They’ve got used to it but I think there’s some issues of consistency. Back to your question, we do expect that to increase sales. We expect that to increase customer satisfaction. We’re actually improving a lot of the flows that we went through and we realized that simple things are missing in the UI. Those are the gold nuggets that you find on the way. Really simple things that you could add or modify in certain places that would make flows a lot better. And I mean reducing 5–10 clicks in a certain flow. My favorite one is I look at a user, he ends up working on a product, and they have six or seven open tabs. I was like, “Why do you have so many tabs?” and he explains, “It makes total sense.” It shows that you’re missing something in the product. There’s a couple of things that are easy telltales if multiple tabs are open, or you have a sticky note on the side with text, or you have Excel on the side where people copy-paste. All those things are signs to problems with the product. We have a few of those and our product is going to come up the other side much more pleasant for the users and help them achieve things faster. Brian: Great. I wish you good fortune and good luck with that redesign that you guys are going through at LogicMonitor. On that note, where can people find LogicMonitor and where can they find you if they wanted to follow you? Gadi: You can find me on Twitter. The handle is @gadioren. I have a LinkedIn page. You can look me up and find me there. You can get to our website, it’s www.logicmonitor.com and that will get you started in you’re interested with that. Brian: Awesome. Thanks, Gadi. This has been really fun to talk to you and hear about your experience here. Thanks for coming on Experiencing Data. Gadi: Thank you very much for having me.

god ceo europe english ai china apple germany co founders normal product experiments falling in love engineers customers analytics saas results cto react applications excel expectation managers user package monitoring ux kpis redux ui traditionally kpi product management prototype oren opinionated tool time gadi declarative brian yeah brian you brian how logicmonitor exceling brian exactly brian just

010 – Carl Hoffman (CEO, Basis Technology) on text analytics, NLP, entity resolution, and why exact match search is stupid

Play Episode Listen Later Apr 9, 2019 45:04

My guest today is Carl Hoffman, the CEO of Basis Technology, and a specialist in text analytics. Carl founded Basis Technology in 1995, and in 1999, the company shipped its first products for website internationalization, enabling Lycos and Google to become the first search engines capable of cataloging the web in both Asian and European languages. In 2003, the company shipped its first Arabic analyzer and began development of a comprehensive text analytics platform. Today, Basis Technology is recognized as the leading provider of components for information retrieval, entity extraction, and entity resolution in many languages. Carl has been directly involved with the company’s activities in support of U.S. national security missions and works closely with analysts in the U.S. intelligence community. Many of you work all day in the world of analytics: numbers, charts, metrics, data visualization, etc. But, today we’re going to talk about one of the other ingredients in designing good data products: text! As an amateur polyglot myself (I speak decent Portuguese, Spanish, and am attempting to learn Polish), I really enjoyed this discussion with Carl. If you are interested in languages, text analytics, search interfaces, entity resolution, and are curious to learn what any of this has to do with offline events such as the Boston Marathon Bombing, you’re going to enjoy my chat with Carl. We covered: How text analytics software is used by Border patrol agencies and its limitations. The role of humans in the loop, even with good text analytics in play What actually happened in the case of the Boston Marathon Bombing? Carl’s article“Exact Match” Isn’t Just Stupid. It’s Deadly. The 2 lessons Carl has learned regarding working with native tongue source material. Why Carl encourages Unicode Compliance when working with text, why having a global perspective is important, and how Carl actually implements this at his company Carl’s parting words on why hybrid architectures are a core foundation to building better data products involving text analytics Resources and Links: Basis Technology Carl’s article: “Exact Match” isn’t Just Stupid. It’s Deadly. Carl Hoffman on LinkedIn Quotes from Today’s Episode “One of the practices that I’ve always liked is actually getting people that aren’t like you, that don’t think like you, in order to intentionally tease out what you don’t know. You know that you’re not going to look at the problem the same way they do…” — Brian O’Neill “Bias is incredibly important in any system that tries to respond to human behavior. We have our own innate cultural biases that we’re sometimes not even aware of. As you [Brian] point out, it’s impossible to separate human language from the underlying culture and, in some cases, geography and the lifestyle of the people who speak that language…” — Carl Hoffman “What I can tell you is that context and nuance are equally important in both spoken and written human communication…Capturing all of the context means that you can do a much better job of the analytics.” — Carl Hoffman “It’s sad when you have these gaps like what happened in this border crossing case where a name spelling is responsible for not flagging down [the right] people. I mean, we put people on the moon and we get something like a name spelling [entity resolution] wrong. It’s shocking in a way.” — Brian O’Neill “We live in a world which is constantly shades of gray and the challenge is getting as close to yes or no as we can.”– Carl Hoffman Episode Transcript Brian: Hey everyone, it’s Brian here and we have a special edition of Experiencing Data today. Today, we are going to be talking to Carl Hoffman who’s the CEO of Basis Technology. Carl is not necessarily a traditional what I would call Data Product Manager or someone working in the field of creating custom decision support tools. He is an expert in text analytics and specifically Basis Technology focuses on entity resolution and resolving entities across different languages. If your product, or service, or your software tool that you’re using is going to be dealing with inputs and outputs or search with multiple languages, I think your going to find my chat with Carl really informative. Without further ado here’s my chat Mr. Carl Hoffman. All right. Welcome back to Experiencing Data. Today, I’m happy to have Carl Hoffman on the line, the CEO of Basis Technology, based out of Cambridge, Massachusetts. How’s it going, Carl? Carl: Great. Good to talk to you, Brian. Brian: Yeah, me too. I’m excited. This episode’s a little but different. Basis Tech primarily focuses on providing text analytics more as a service as opposed to a data product. There are obviously some user experience ramifications on the downstream side of companies, software, and services that are leveraging some of your technology. Can you tell people a little bit about the technology of Basis and what you guys do? Carl: There are many companies who are in the business of extracting actionable information from large amounts of dirty, unstructured data and we are one of them. But what makes us unique is our ability to extract what we believe is one of the most difficult forms of big data, which is text in many different languages from a wide range of sources. You mentioned text analytics as a service, which is a big part of our business, but we actually provide text analytics in almost every conceivable form. As a service, as an on-prem cloud offering, as a conventional enterprise software, and also as the data fuel to power your in-house text analytics. There’s another half of our business as well which is focused specifically on one of the most important sources of data, which is what we call digital forensics or cyber forensics. That’s the challenge of getting data off of digital media that maybe either still in use or dead. Brian: Talk to me about dead. Can you go unpack that a little bit? Carl: Yes. Dead basically means powered off or disabled. The primary application there is for corporate investigators or for law enforcement who are investigating captured devices or digital media. Brian: Got it. Just to help people understand some of the use cases that someone would be leveraging some of the capabilities of your platforms, especially the stuff around entity resolution, can you talk a little bit about like my understanding, for example, one use case for your software is obviously border crossings, where your information, your name is going to be looked up to make sure that you should be crossing whatever particular border that you’re at. Can you talk to us a little bit about what’s happening there and what’s going on behind the scenes with your software? Like what is that agent doing and what’s happening behind the scenes? What kind of value are you providing to the government at that instance? Carl: Border crossings or the software used by border control authorities is a very important application of our software. From a data representational challenge, it’s actually not that difficult because for the most part, border authorities work with linear databases of known individuals or partially known individuals and queries. Queries may be the form manually typed by an officer or maybe scan of a passport. The complexity comes in when a match must be scored, where a decision must be rendered as to whether a particular query or a particular passport scan matches any of the names present on a watch list. Those watch list can be in many different formats. They can come from many different sources. Our software excels at performing that match at very high accuracy, regardless of the nature of the query and regardless of the source of the underlying watch list. Brian: I assume those watch lists may vary in the level of detail around for example, aliases, spelling, which alphabet they were being printed in. Part of the value of what your services is doing is helping to say, “At the end of the day, entity number seven on the list is one human being who may have many ways of being represented with words on a page or a screen,” so the goal obviously is to make sure that you have the full story of that one individual. Am I correct that you may get that in various formats and different levels of detail? And part of what your system is doing is actually trying to match up that person or give it what you say a non-binary response but a match score or something that’s more of a gray response that says, “This person may also be this person.” Can you compact that a little bit for us? Carl: Your remarks are exactly correct. First, what you said about gray is very important. These decisions are rarely 100% yes or no. We live in a world which is constantly shades of gray and the challenge is getting us close to yes or no as we can. But the quality of the data in watch lists can vary pretty wildly, based on the prominence and the number of sources. The US border authorities must compile information from many different sources, from UN, from Treasury Department, from National Counterterrorism Center, from various states, and so on. The amount of detail and the degree of our certainty regarding that data can vary from name to name. Brian: We talked about this when we first were chatting about this episode. Am I correct when I think about one of the overall values you’re doing is obviously we’re offloading some of the labor of doing this kind of entity resolution or analysis onto software and then picking up the last mile with human, to say, “Hey, are these recommendations correct? Maybe I’ll go in and do some manual labor.” Is that how you see it, that we do some of the initial grunt work and you present an almost finished story, and then the human comes in and needs to really provide that final decision at the endpoint? Are we doing enough of the help with the software? At what point should we say, “That’s no longer a software job to give you a better score about this person. We think that really requires a human analysis at this point.” Is there a way to evaluate or is that what you think about like, “Hey, we don’t want to go past up that point. We want to stop here because the technology is not good enough or the data coming in will never be accurate enough and we don’t want to go past that point.” I don’t know if that makes sense. Carl: It does makes sense. I can’t speak for all countries but I can say that in the US, the decision to deny an individual entry or certainly the decision to apprehend an individual is always made by a human. We designed our software to assume a human in the loop for the most critical decisions. Our software is designed to maximize the value of the information that is presented to the human so that nothing is overlooked. Really, the two biggest threats to our national security are one, having very valuable information overlooked, which is exactly what happened in the case of the Boston Marathon bombing. We had a great deal of information about Tamerlan and Dzhokhar Tsarnaev, yet that information was overlooked because the search engines failed to surface it in response to queries by a number of officials. And secondly, detaining or apprehending innocent individuals, which hurts our security as much as allowing dangerous individuals to pass. Brian: This has been in the news somewhat but talk about the “glitch” and what happened in that Boston Marathon bombing in terms of maybe some of these tools and what might have happened or not what might have happened, but what you understand was going on there such that there was a gap in this information. Carl: I am always very suspicious when anyone uses the word ‘glitch’ with regard to any type of digital equipment because if that equipment is executing its algorithm as it has been programmed to do, then you will get identical results for identical inputs. In this case, the software that was in use at the time by US Customs and Border Protection was executing a very naive name-matching algorithm, which failed to match two different variant spellings of the name Tsarnaev. If you look at the two variations for any human, it would seem almost obvious that the two variations are related and are in fact connected to the same name that’s natively written in Cyrillic. What really happened was a failure on the part of the architects of that name mentioning system to innovate by employing the latest technology in name-matching, which is what my company provides. In the aftermath of that disaster, our software was integrated into the border control workflow, first with the goal of redacting false-positives, and then later with the secondary goal of identifying false negatives. We’ve been very successful on both of those challenges. Brian: What were the two variants? Are you talking about the fact that one was spelled in Cyrillic and one was spelled in a Latin alphabet? They didn’t bring back data point A and B because they look like separate individuals? What was it, a transliteration? Carl: They were two different transliterations of the name Tsarnaev. In one instance, the final letters in the names are spelled -naev and the second instance it’s spelled -nayev. The presence or absence of that letter y was the only difference between the two. That’s a relatively simple case but there are many similar stories for more complex names. For instance, the 2009 Christmas bomber who successfully boarded a Northwest Delta flight with a bomb in his underwear, again because of a failure to match two different transliterations of his name. But in his case, his name is Umar Farouk Abdulmutallab. There was much more opportunity for divergent transliterations. Brian: On this kind of topic, you wrote an interesting article called “Exact Match” Isn’t Just Stupid. It’s Deadly. You’ve talked a little bit about this particular example with the Boston Marathon bombing. You mentioned that they’re thinking globally about building a product out. Can you talk to us a little about what it means to think globally? Carl: Sure. Thinking globally is really a mindset and an architectural philosophy in which systems are built to accommodate multiple languages and cultures. This is an issue not just with the spelling of names but with support for multiple writing systems, different ways of rendering and formatting personal names, different ways of rendering, formatting, and parsing postal addresses, telephone numbers, dates, times, and so on. The format of a questionnaire in Japanese is quite different from the format of a questionnaire in English. If you will get any complex global software product, there’s a great deal of work that must be done to accommodate the needs of a worldwide user base. Brian: Sure and you’re a big fan of Unicode-compliant software, am I correct? Carl: Yes. Building Unicode compliance is equivalent to building a solid stable foundation for an office tower. It only gets you to the ground floor, but without it, the rest of the tower starts to lean like the one that’s happening in San Francisco right now. Brian: I haven’t heard about that. Carl: There’s a whole tower that’s tipping over. You should read it. It’s a great story. Brian: Foundation’s not so solid. Carl: Big lawsuit’s going on right now. Brian: Not the place you want to have a sagging tower either. Carl: Not the place but frankly, it’s really quite comparable because I’ve seen some large systems that will go unnamed, where there’s legacy technology and people are unaware perhaps why it’s so important to move from Python version 2 to Python version 3. One of the key differences is Unicode compliance. So if I hear about a large-scale enterprise system that’s based on Python version 2, I’m immediately suspicious that it’s going to be suitable for a global audience. Brian: I think about, from an experience standpoint, inputs, when you’re providing inputs into forms and understanding what people are typing in. If it’s a query form, obviously giving people back what they wanted and not necessarily what they typed in. We all take for granted things like this spelling correction, and not just the spelling correction, but in Google when you type in something, it sometimes give you something that’s beyond a spelling thing, “Did you mean X, Y, and Z?” I would think that being in the form about what people are typing into your form fields and mining your query logs, this is something I do sometimes with clients when they’re trying to learn something. I actually just read an article today about dell.com and the top query term on dell.com is ‘Google,’ which is a very interesting thing. I would be curious to know why people are typing that in. Is it really like people are actually trying to access Google or are they trying to get some information? But the point is to understand the input side and to try to return some kind of logical output. Whether it’s text analytics that’s providing that or it’s name-matching, it’s being aware of that and it’s sad when you have these gaps like what happened in this border crossing case where a name spelling is responsible for not flagging down these people. I mean, we put people on the moon and we get something like a name spelling wrong. It’s shocking in a way. I guess for those who are working in tech, we can understand how it might happen, but it’s scary that that’s still going on today. You’ve probably seen many other. Are you able to talk about it? Obviously, you have some in the intelligence field and probably government where you can’t talk about some of your clients, but are there other examples of learning that’s happened that, even if it’s not necessarily entity resolution where you’ve put dots together with some of your platform? Carl: I’ll say the biggest lesson that I’ve learned from nearly two decades of working on government applications involving multi-lingual data is the importance of retaining as much of the information in its native form as possible. For example, there is a very large division of the CIA which is focused on collecting open source intelligence in the form of newspapers, magazines, the digital equivalent of those, radio broadcast, TV broadcasts and so one. It’s a unit which used to be known as the Foreign Broadcast Information Service, going back to Word War II time, and today it’s called the Open Source Enterprise. They have a very large collection apparatus and they produce some extremely high quality products which are summaries and translations from sources in other languages. In their workflow, previously they would collect information, say in Chinese or in Russian, and then do a translation or summary into English, but then would discard the original or the original would be hidden from their enterprise architecture for query purposes. I believe that is no longer the case, but retaining the pre-translation original, whether it’s open source, closed source, commercial, enterprise information, government-related information, is really very important. That’s one lesson. The other lesson is appreciating the limits of machine translation. We’re increasingly seeing machine translation integrated into all kinds of information systems, but there needs to be a very sober appreciation of what is and what is not achievable and scalable by employing machine translation in your architecture. Brian: Can you talk at all about the translation? We have so much power now with NLP and what’s possible with the technology today. As I understand it, when we talk about translation, we’re talking about documents and things that are in written word that are being translated from one language to another. But in terms of spoken word, and we’re communicating right now, I’m going to ask you two questions. What do you know about NLP and what do you know about NLP? The first one I had a little bit of attitude which assumes that you don’t know too much about it, and the second one, I was treating you as an expert. When this gets translated into text, it loses that context. Where are we with that ability to look at the context, the tone, the sentiment that’s behind that? I would imagine that’s partly why you’re talking about saving the original source. It might provide some context like, “What are the headlines were in the paper?” and, “Which paper wrote it?” and, “Is there a bias with that paper?” whatever, having some context of the full article that that report came from can provide additional context. Humans are probably better at doing some of that initial eyeball analysis or having some idea of historically where this article’s coming from such that they can put it in some context as opposed to just seeing the words in a native language on a computer screen. Can you talk a little bit about that or where we are with that? And am I incorrect that we’re not able to look at that sentiment? I don’t even know how that would translate necessarily unless you had a playing back of a recording of someone saying the words. You have translation on top of the sentiment. Now you’ve got two factors of difficulty right there and getting it accurate. Carl: My knowledge of voice and speech analysis is very naive. I do know there’s an area of huge investment and the technology is progressing very rapidly. I suspect that voice models are already being built that can distinguish between the two different intonations you used in asking that question and are able to match those against knowledge bases separately. What I can tell you is that context and nuance are equally important in both spoken and written human communication. My knowledge is stronger when it comes to its written form. Capturing all of the context means that you can do a much better job of the analytics. That’s why, say, when we’re analyzing a document, we’re looking not only the individual word but the sentence, the paragraph, where does the text appear? Is it in the body? Is it in a heading? Is it in a caption? Is it in a footnote? Or if we’re looking at, say, human-typed input—I think this is where your audience would care if you’re designing forms or search boxes—there’s a lot that can be determined in terms of how the input is typed. Again, especially when you’re thinking globally. We’re familiar with typing English and typing queries or completing forms with the letters A through Z and the numbers 0 through 9, but the fastest-growing new orthography today is emoticons and emoji offer a lot of very valuable information about the mindset of the author. Say that we look at Chinese or Japanese, which are basically written with thousand-year-old emoji, where an individual must type a sequence of keys in order to create each of the Kanji or Hanzu that appears. There’s a great deal of information we can capture. For instance, if I’m typing a form in Japanese, saying I’m filling out my last name, and then my last name is Tanaka. Well, I’m going to type phonetically some characters that represent Tanaka, either in Latin letters or one of the Japanese phonetic writing systems, then I’m going to pick from a menu or the system is going to automatically pick for me the Japanese characters that represent Tanaka. But any really capable input system is going to keep both whatever I typed phonetically and the Kanji that I selected because both of those have value and the association between the two is not always obvious. There are similar ways of capturing context and meaning in other writing systems. For instance, let’s say I’m typing Arabic not in Arabic script but I’m typing with Roman letters. How I translate from those Roman letters into the Arabic alphabet may vary, depending upon if I’m using Gulf Arabic, or Levantine Arabic, or Cairene Arabic, and say the IP address of the person doing the typing may factor into how I do that transformation and how I interpret those letters. There’s examples for many other writing systems other than the Latin alphabet. Brian: I meant to ask you. Do you speak any other languages or do you study any other languages? Carl: I studied Japanese for a few years in high school. That’s really what got me into using computers to facilitate language understanding. I just never had the ability to really quickly memorize all of the Japanese characters, the radical components, and the variant pronunciations. After spending countless hours combing through paper dictionaries, I got very interested in building electronic dictionaries. My interest in electronic dictionaries eventually led to search engines and to lexicons, algorithms powered by lexicons, and then ultimately to machine learning and deep learning. Brian: I’m curious. I assume you need to employ either a linguist or at least people that speak multiple languages. One concern with advanced analytics right now and especially anything with prediction, is bias. I speak a couple of different languages and I think one of the coolest things about learning another language is seeing the world through another context. Right now, I’m learning Polish and there’s the concept of case and it doesn’t just come down to learning the prefixes and suffixes that are added to words. Effectively, that’s what the output is but it’s even understanding the nuance of when you would use that and what you’re trying to convey, and then when you relay it back to your own language, we don’t even have an equivalent between this. We would never divide this verb into two different sentiments. So you start to learn what you don’t even know to think about. I guess what I’m asking here is how do you capture those things? Say, in our case where I assume you’re an American and I am to, so we have our English that we grew up with and our context for that. How do you avoid bias? Do you think about bias? How do you build these systems in terms of approaching it from a single language? Ultimately, this code is probably written in English, I assume. Not to say that the code would be written in a different language but just the approach when you’re thinking about all these systems that have to do with language, where does that come in having integrating other people that speaks other languages? Can you talk about that a little bit? Carl: Bias is incredibly important in any system that tries to respond to human behavior. We have our own innate cultural biases that we’re sometimes not even aware of. As you point out, it’s impossible to separate human language from the underlying culture and, in some cases, geography and the lifestyle of the people who speak that language. Yes, this is something that we think about. I disagree with your remark about code being written in English. The most important pieces of code today are the frameworks for implementing various machine learning and deep learning architectures. These architectures for the most part are language or domain-agnostic. The language bias tends to creep in as an artifact of the data that we collect. If I were to, say, harvest a million pages randomly on the internet, a very large percentage of those pages would be in English, out of proportion to the proportion of the population of the planet who speaks English, just because English is common language for commerce, science, and so on. The bias comes in from the data or it comes in from the mindset of the architect, who may do something as simple-minded as allocating only eight bits per character or deciding that Python version 2 is an acceptable development platform. Brian: Sure. I should say, I wasn’t so much speaking about the script, the code, as much as I was thinking more about the humans behind it, their background, and their language that they speak, or these kinds of choices that you’re talking about because they’re informed by that person’s perspective. But thank you for clarifying. Carl: I agree with that observation as well. You’re certainly right. Brian: Do you have a way? You’re experts in this area and you’re obviously heavily invested in this area. Are there things that you have to do to prevent that bias, in terms of like, “We know what we don’t know about it, or we know enough about it but we don’t know if about, so we have a checklist or we have something that we go through to make sure that we’re checking ourselves to avoid these things”? Or is it more in the data collection phase that you’re worried about more so than the code or whatever that’s actually going to be taking the data and generating the software value at the other end? Is it more on the collection side that you’re thinking about? How do you prevent it? How do you check yourself or tell a client or customer, “Here’s how we’ve tried to make sure that the quality of what we’re giving you is good. We did A, B, C, and D.” Maybe I’m making a bigger issue out of this than it is. I’m not sure. Carl: No, it is a big issue. The best way to minimize that cultural bias is by building global teams. That’s something that we’ve done from the very beginning days of our company. We have a company in which collectively the team speaks over 20 languages, originate from many different countries around the world, and we do business in native countries around the world. That’s just been an absolute necessity because we produce products that are proficient in 40 different human languages. If you’re a large enterprise, more than 500 people, and you’re targeting markets globally, then you need to build a global team. That applies to all the different parts of the organization, including the executive team. It’s rare that you will see individuals who are, say, American culture with no meaningful international experience being successful in any kind of global expansion. Brian: That’s pretty awesome that you have that many languages going in the staff that you have working at the company. That’s cool and I think it does provide a different perspective on it. We talk about it even in the design firm. Sometimes, early managers in the design will want to go hire a lot of people that look like they do. Not necessarily physically but in terms of skill set. One of the practices that I’ve always liked is actually getting people that aren’t like you, that don’t think like you, in order to intentionally tease out what you don’t know, you know that you’re not going to look at the problem the same way they are, and you don’t necessarily know what the output is, but you can learn that there’s other perspectives to have, so too many like-minded individuals doesn’t necessarily mean that it’s better. I think that’s cool. Can you talk to me a little bit about one of the fun little nuggets that stuck in my head and I think you’ve attributed to somebody else, but was the word about getting insights from medium data. Can you talk to us about that? Carl: Sure. I should first start by crediting the individual who planted that idea in my head, which is Dr. Catherine Havasi of the MIT Media Lab, who’s also a cofounder of a company called Luminoso, which is a partner of ours. They do common sense understanding. The challenge with building truly capable text analytics from large amounts of unstructured text is obtaining sufficient volume. If you are a company on the scale of Facebook or Google, you have access to truly enormous amount of text. I can’t quantify it in petabytes or exabytes, but it is a scale that is much greater than the typical global enterprise or Fortune 2000 company, who themselves may have very massive data lakes. But still, those data lakes are probably three to five orders of magnitudes smaller than what Google or Facebook may have under their control. That intermediate-sized data, which is sloppily referred to as big data, we think of it as medium data. We think about the challenge of allowing companies with medium data assets to obtain big data quality results, or business intelligence that’s comparable to something that Google or Facebook might be able to obtain. We do that by building models that are hybrid, that combine knowledge graphs or semantic graphs, derived from very large open sources with the information that they can extract from their proprietary data lakes, and using the open sources and the models that we build as amplifiers for their own data. Brian: I believe when we were talking, you have mentioned a couple of companies that are building products on top of you. Difio, I think, was one, and Tamr, and Luminoso. So is that related to what these companies are doing? Carl: Yes, it absolutely is related. Luminoso, in particular, is using this process of synthesizing results from their customers, proprietary data with their own models. The Luminoso team grew out of the team at MIT that built something called Constant Net, which is a very large net of graph in multiple languages. But actually, Difio as well is also using this approach of federating both open and closed source repositories by integrating a large number of connectors into their architecture. They have access to web content. They have access to various social media fire hoses. They have access to proprietary data feeds from financial news providers. But then, they fuse that with internal sources of information that may come from sources like SharePoint, or Dropbox, or Google Drive, or OneDrive, your local file servers, and then give you a single view into all of this data. Brian: Awesome. I don’t want to keep you too long. This has been super informational for me, learning about your space that you’re in. Can you tell us any closing thoughts, advice for product managers, analytics practitioners? We talked a little about obviously thinking globally and some of those areas. Any other closing thoughts about delivering good experiences, leveraging text analytics, other things to watch out for? Any general thoughts? Carl: Sure. I’ll close with a few thoughts. One is repeating what I’ve said before about Unicode compliance. The fact that I again have to state that is somewhat depressing yet it’s still isn’t taken as an absolute requirement, which is today, and yet continues to be overlooked. Secondly, just thinking globally, anything that you’re building, you got to think about a global audience. I’ll share with you an anecdote. My company gives a lot of business to Eventbrite, who I would expect by now would have a fully globalized platform, but it turns out their utility for sending an email to everybody who signed-up for an event doesn’t work in Japanese. I found that out the hard way when I needed to send an email to everybody that was signed up for our conference in Tokyo. That was very disturbing and I’m not afraid to say that live on a podcast. They need to fix it. You really don’t want customers finding out about that during a time of high stress and high pressure, and there’s just no excuse for that. Then my third point with regard to natural language understanding. This is a really incredibly exciting time to be involved with natural language, with human language because the technology is changing so rapidly and the space of what is achievable is expanding so rapidly. My final point of advice is that hybrid architectures have been the best and continue to be the best. There’s a real temptation to say, “Just grow all of my text into a deep neural net and magic is going to happen.” That can be true if you have sufficiently large amounts of data, but most people don’t. Therefore, you’re going to get better results by using hybrids of algorithmic simpler machine learning architectures together with deep neural nets. Brian: That last tip, can you take that down one more notch? I assume you’re talking about a level of quality on the tail-end of the technology implementation, there’s going to be some higher quality output. Can you translate what a hybrid architecture means in terms of a better product at the other end? What would be an example of that? Carl: Sure. It’s hard to do without getting too technical, but I’ll try and I’ll try to use some examples in English. I think the traditional way of approaching deep nets has very much been take a very simple, potentially deep and recursive neural network architecture and just throw data at it, especially images or audio waveforms. I throw my images in and I want to classify which ones were taken outdoors and which ones were taken indoors with no traditional signal processing or image processing added before or after. In the image domain, my understanding is that, that kind of purist approach is delivered the best results and that’s what I’ve heard. I don’t have first-hand information about that. However, when it comes to human language in its written form, there’s a great deal of traditional processing of that text that boosts the effectiveness of the deep learning. That falls into a number of layers that I won’t go into, but to just give you one example, let’s talk about what we called Orthography. The English language is relatively simple and that the orthography is generally quite simple. We’ve got the letters A through Z, an uppercase and lowercase, and that’s about it. But if you look inside, say a PDF of English text, you’ll sometimes encounter things like ligatures, like a lowercase F followed by a lowercase I, or two lowercase Fs together, will be replaced with single glyph to make it look good in that particular typeface. If I think those glyphs and I just throw them in with all the rest of my text, that actually complicates the job of the deep learning. If I take that FI ligature and convert it back to separate F followed by I, or the FF ligature and convert it back to FF, my deep learning doesn’t have to figure out what those ligatures are about. Now that seems pretty obscure in English but in other writing systems, especially Arabic, for instance, in which there’s an enormous number of ligatures, or Korean or languages that have diacritical marks, processing those diacritical marks, those ligatures, those orthographic variations using conventional means will make your deep learning run much faster and give you better results with less data. That’s just one example but there’s a whole range or other text-processing steps using algorithms that have been developed over many years, that simply makes the deep learning work better and that results in what we call a hybrid architecture. Brian: So it sounds like taking, as opposed to throw it all in a pot and stir, there’s the, “Well, maybe I’m going to cut the carrots neatly into the right size and then throw them in the soup.” Carl: Exactly. Brian: You’re kind of helping the system do a better job at its work. Carl: That’s right and it’s really about thinking about your data and understanding something about it before you throw it into the big brain. Brian: Exactly. Cool. Where can people follow you? I’ll put a link up to the Basis in the show notes but are you on Twitter or LinkedIn somewhere? Where can people find you? Carl: LinkedIn tends to be my preferred social network. I just was never really good at summarizing complex thoughts into 140 characters, so that’s the best place to connect with me. Basically, we’ll tell you all about Basis Technology and rosette.com is our text analytics platform, which is free for anybody to explore, and to the best of my knowledge, it is the most capable text analytics platform with the largest number of languages that you will find anywhere on the public internet. Brian: All right, I will definitely put those up in the show notes. This has been fantastic, I’ve learned a ton, and thanks for coming on Experiencing Data. Carl: Great talking with you, Brian. Brian: All right. Cheers. Carl: Cheers.

009 – Nancy Hensley (Chief Digital Officer, IBM Analytics) on the role of design and UX in modernizing analytics tools as old as 50 years

Play Episode Listen Later Mar 26, 2019 49:44

Nancy Hensley is the Chief Digital Officer for IBM Analytics, a multi-billion dollar IBM software business focused on helping customers transform their companies with data science and analytics. Nancy has over 20 years of experience working in the data business in many facets from development, product management, sales, and marketing. Today’s episode is probably going to appeal to those of you in product management or working on SAAS/cloud analytics tools. It is a bit different than our previous episodes in that we focused a lot on what “big blue” is doing to simplify its analytics suite as well as facilitating access to those tools. IBM has many different analytics-related products and they rely on good design to make sure that there is a consistent feel and experience across the suite, whether it’s Watson, statistics, or modeling tools. She also talked about how central user experience is to making IBM’s tools more cloud-like (try/buy online) vs. forcing customers to go through a traditional enterprise salesperson. If you’ve got a “dated” analytics product or service that is hard to use or feels very “enterprisey” (in that not-so-good way), then I think you’ll enjoy the “modernization” theme of this episode. We covered: How Nancy is taking a 50-year old product such as SPSS and making it relevant and accessible for an audience that is 60% under 25 years of age The two components Nancy’s team looks at when designing an analytics product What “Metrics Monday” is all about at IBM Analytics How IBM follows-up with customers, communicates with legacy users, and how the digital market has changed consumption models Nancy’s thoughts on growth hacking and the role of simplification Why you should always consider product-market fit first and Nancy’s ideas on MVPs The role design plays in successful onboarding customers into IBM Analytics’ tools and what Brian refers to as the “honeymoon” experience Resources and Links: Nancy Hensley on Twitter Nancy Hensley on LinkedIn Quotes: “It’s really never about whether it’s a great product. It’s about whether the client thinks it’s great when they start using it.” –Nancy “Every time we add to the tool, we’re effectively reducing the simplicity of everything else around it.”–Brian “The design part of it for us is so eye-opening, because again, we’ve built a lot of best in class enterprise products for years and as we shift into this digital go-to-market, it is all about the experience…”–Nancy “Filling in that “why” piece is really important if you’re going to start changing design because you may not really understand the reasons someone’s abandoning.”–Brian “Because a lot of our products weren’t born in the cloud originally, they weren’t born to be digitally originally, doesn’t mean they can’t be digitally consumed. We just have to really focus on the experience and one of those things is onboarding.” –Nancy “If they [users] can’t figure out how to jump in and use the product, we’re not nailing it. It doesn’t matter how great the product is, if they can’t figure out how to effectively interact with it. –Nancy Episode Transcript Brian: Today on Experiencing Data, I [talked] to Nancy Hensley, the Chief Digital Officer of IBM Analytics. Nancy brings a lot of experience and has a lot to say about how user experience and design have become very integral to IBM’s success especially as they move their applications into the cloud space. They really try to bring the price point down and make their services and applications much more low touch in order to access a new base of subscribers and users. I really enjoyed this talk with her about what the designers and people focused on the product experience have been doing at IBM to keep their company relevant and keep them pushing forward in terms of delivering really good experiences to their customers. I hope you enjoy this episode with Nancy Hensley. Hello everybody. I’m super stoked to have Nancy Hensley, the Chief Digital Officer of IBM Analytics. How’s it going, Nancy? Nancy: Good. I’m happy to be here. Happy Friday. Brian: Yeah. It’s getting cold here in Cambridge, Mass ; [ you’re] in Chicago area, if I remember correctly. Nancy: Yeah, it’s a little bit chilly here as well. Brian: Nice. So it begins. You’ve done quite a bit of stuff at IBM when we had our little pre-planning call. You talked a lot about growth that’s been happening over at IBM. I wanted to talk to you specifically about the role that design and experience has played, how you guys have changed some of your products, and how you’re talking to new customers and that type of thing. Can you tell people, first of all, just a little bit about your background, what you’re currently doing, and then we could maybe […] some of those things. Nancy: Sure, happy to. Thank you for having me again. I think I’m one of those people that doesn’t fit nicely into a box of, “Are you product? Are you marketing?” I am a little bit of both. Most of my IBM career, I have moved in between product marketing and product management. That’s why I love digital so much because it really is a nice mixture. And in particular, growth hacking because it combines all the things I love, including data as a part of what we do. What I’m doing right now as a Chief Digital Officer in the Analytics Division and Hypercloud is how do we transform our products to make them more consumable, more accessible? We have best in class products in data science, in unified governments and integration, in hyper data management products, but our products and our business is built on a traditional face-to-face model. There is even a perception that we’re not as accessible to them and that’s what we’re looking to change. Creating those lower entry points, making it easier for people who didn’t have access to us before, to start small and grow through a digital channel, through a lower entry point product, and then scale up from there. That’s really what we’re trying to do and as part of a bigger mission to really democratize data science—I kind of cringe when I say that word—I think it’s really important for more clients to be able to be more data-driven, have tools that are easy to use, and leverage data science to optimize their business. Part of the way we’re doing that is to develop a digital route to market. We’re pretty excited about it. Brian: I think a lot of our listeners probably come from internal roles of companies. They might be someone that’s purchasing vendor software as opposed to a SaaS company where they may have a closer role to marketing and all that. Can you tell me what you guys are doing there? Part of the thing with my experience is that some of the legacy companies, the older companies that are out there tend to get associated with big giant enterprise installations, really crappy user experience. It’s just so powerful, you have to put up with all these stuff. People’s tendency these days to accept that poor experience as just status quo is changing. What have you guys done? Not that you’re to blame but I’m sure that opinion exist. How do you guys adapt to that and wonder if upstart analytics companies coming out with other things, what do you guys to to address the experience? Nancy: There’s certainly a perception that IBM is that big, complicated, enterprise-focused product out there. We see the data. There’s a lot of articles, there’s a lot of feedback, there’s endless report that all validate that clients are trading off complexity or features and functions for consumability, because they got to get things done, they have less people to do it. We fully recognize that. Where we started to look for that was how we first started to make things much more accessible, not just our cloud products because that’s pretty easy if you have stuff in the cloud—it’s pretty accessible—but our on-prime products as well. So, for clients that are running analysis behind the private cloud, whether it’s a statistical product, or a predictive analytics product, or data science project, or even what they’re doing on their data catalog, all of that was not something people would go to the cog to look for it. There are some things they need especially financial and health care, and there’s large and small companies on both sides. One of the things we set out to do is how do we create that cloud-like experience for clients that are running things behind their firewall. We started a project about a year ago to look at some of our on-prime products and create that experience where literally you could, within a couple of clicks, download, try, and be using a product within 15 minutes. That was our goal. As opposed to before where you would have to contact and IBM salesperson, get them to come out and meet with you, and then set-up a trial. That’s what we started to change was that at least make it accessible. As we progressed that capability, we started changing our pricing and packaging to be appropriate, to create that entry-level point, to create a shift to subscription. You want to buy everything on subscription these days, I think. The last part of that shift for us has been to really focus on the experience because a lot of these products were not born digital. We really need to make sure that when clients were coming through that channel, that it was a great experience. That’s really where design experience came into play for us. Brian: How did you know of what’s wrong beyond broad surveys or just that general feeling that like, “Oh, it’s the big giant bloated software…” the stereotype, right? How do you guys get into the meat and potatoes of like you said, sounds like there’s a benchmark there, 15 minutes on that first onboarding experience, but can you tell us a little bit of maybe if you have a specific example about how you figured it out? What do we need to change about this software application to make it easier to get value out of the analytics of the data that’s there? Nancy: I’ve got lots of examples. We’ll opt with one that clients actually are very familiar with, which is SPSS Statistics that a lot of us used back in college. That was a product that actually turns 50 years old this year. It’s been out a while, a lot of people still using it a lot, and most of our base of users for statistics, I think if you look at the demographics of it, over 60% are under the age of 25. So, their buying preferences were very different than they were when they started out in 1968. We look at the verbatims from our NPS feedback and it was clear that clients really wanted a much more simplified and flexible experience than buying SPSS Statistics and having access to it. A lot of times, students have to get it really quickly for a project because they’ve might have waited until the last minute and they wanted a much more flexible subscription-based program. They might only use it for a few months and then come back to it. That was one of the first things that we implemented was to change the buying experience for the consumption model. We didn’t actually change the product at that point. We just changed the consumption model to see if in fact that actually will help us have some growth on that product, and it absolutely did. Since then, we’ve actually gone back and change the product as well. It’s got a whole new UI for its 50th anniversary. Joke around that it’s got a face lift for it’s 50th anniversary. Brian: Does it have a green screen mode? Nancy: It is a completely different experience, not just from a buying perspective, but also from a UI perspective as well. We have other products, too, that have been around maybe not 50 years but have been very popular products like our DB2 Warehouse on Cloud and our DB2 database that clients have been buying for years to run their enterprises. We wanted to make that again, as we created a SaaS alternative of these products that it was extremely consumable. So, we’ve been looking specifically, is it easy to figure out which version to buy? How much to buy? What it’s going to do for you? Like I said, which version? How do I calculate things? We’ve been really looking at the experience of that is, if there was no salesperson at all, how do we help clients through that buying experience? Brian: I’m curious. When you decided to helping them through the buying experience, does any of that thinking or that strategy around hand-holding someone through that experience happen in the product itself? I’m guessing you’re downloading a package at some point, you’re running an installer, and at that point, did you continue that hand-holding process to get them out of the weeds of the installation and onboarding again to the actual, “Is this tool right for what I needed to do?” Everything else being friction up until that point where you’re actually working with your data, did you guys carry that through? Can you talk about that? Nancy: You’re hitting one of my favorite topics which is onboarding. Because a lot of our products weren’t born in the cloud originally, they weren’t born to be digital originally, doesn’t mean they can’t be digitally consumed. We just have to really focus on the experience and one of those things in onboarding. Let’s say, DB2 in particular, which won the process of creating onboarding experience for DB Warehouse on Cloud. For anybody who’s used DB2, we do have an updated UI for that. They can jump in and start using it. But that’s not everyone, the people that haven’t used it before. So, we just started working with a couple of different onboarding tools to create these experiences. Our goal was to be able—at least I’m offering management side alongside our partners but design—to create these experiences in a very agile way and make them measurable—my second favorite topic, which is instrumentation—but not have a burden on development, because the fact is, in almost any organization, development wants to build features and functions. Whenever we talk about this, they were prioritized lower because they want to build new capabilities. They’re less enthusiastic about building in things like onboarding experiences. With some of the tools like [.DB2..] give us, is a way to make it codeless to us. We can create these experiences, then pass the code snippet, and then measure whether those are effective or not because we actually see those flowing through segment into our amplitude as a part of the shuttle. We’ve got some great feedback as to whether they’re working or where they’re falling down. We can create checklists of things that we want the clients to do that we know makes the product sticky, and see if they actually complete that checklist. It’s giving us so much better view because before, what we would see with a client is register for trial, they downloaded the trial, they’ve created their instance, and then boom they fall off the cliff. What happened? Now we’re getting a much better view to what’s actually going on for the products that have been instrumented as well as the view we’re getting in from the onboarding experiences. Brian: For every one of these applications that you’re trying to move into a cloud model or simplify whether it’s cloud, to me the deployment model doesn’t matter. It’s really about removing the friction points whether it’s on-premise software or not. I think we all tend to use the word ‘cloud’ to kind of feel like, “Oh, is this browser-based thing? There’s no hard clients? There’s no running scripts at the terminal and all that kind of stuff?” Do you guys have a set of benchmarks or something that you establish for every one of these products that are going to go through a redesign? Nancy: We do. We’ve got a set of criteria, it’s really broken down into two pieces. Whether it’s going to be a cloud product or an on-premise product—I actually have a mix of both—there is what we call the MVP side, which might be something that’s not born in the cloud, it’s not a new product, and we’re looking to create a lower entry point, a really good trial experience, a very optimized journey. We’re even doing things like taking some of the capabilities that we used to have from a technical perspective and making those more digitally available. Online proof of concepts, hands-on labs that you do online instead of waiting for a technical salesperson to come out to see you. Tap us that can answer your questions faster even before you talk to a sales rep. All of that is included in the what we call the MVP portion of the criteria that we look at. Pricing and packaging’s got to be right for the product, for the marketplace. Got to have that right product market fit that you’ve got a good valuable product but a low-enough entry point where somebody can start small and scale up. The second part of the criteria is where the growth magic happens, where we’re dumbing down a lot more on the experimentation, where we’re making sure that we’ve got onboarding, instrumentation we want done, and the MVP phase, we don’t always get it, but our development partners really understand the value of that now, which is great. Though more often, we’re getting into the second phase of where we’re more doing the transformation. Through that, then we’re getting a lot more feedback, where we can create the onboarding experience. We can do even more on the optimized journey. We’re doing a lot of growth hacking that’s based on terms of optimizing. Things like how clear is information on the pricing page? Is it easy for the customer to figure out what they need to buy? What the pricing is for that? Can they get their questions answered quickly? Can we create a deeper technical experience for them, even outside of the trial itself? Like I mentioned, things we’re doing with our digital technical engagement, thinking that what used to be our tech sales modeling and making it more digital. Brian: That’s cool. When you guys go through this process of testing, are you primarily looking at quantitative metrics then that are coming back from the software that you guys are building, or you’re doing any type of qualitative research to go validate like, “Hey, is the onboarding working well?” Obviously, the quantitative can tell you what. It doesn’t tell you why someone might have abandoned at this point. You guys do any research there? Nancy: We do. It happens in a couple of places. We run squads that are cross-functional across marketing, product, development, and design, each product. Then every Monday we have this thing called Metrics Monday where we get the cross-functional routines together, we share the insights around the metrics. If we had a big spike or we had a big decrease, or if we had a change in engagement, or if we did some experimentation that came out with a very interesting result, we actually share that across teams. We really focus on why did things happen. We have a dashboard. Everybody is religious in using on a daily basis that tracks all of our key metrics, whether it’s visits, engage visits, trials, trial-to-win conversions, number of orders, things like that, but we also want to dive deeper into the ebbs and flows of the business itself, why things are happening, and if the experimentation we’re doing is helping or not helping. We’ve got a lot of focus on that on a daily and a weekly basis. Brian: Do you have any way to access the trial users and do one-on-one usability study or a follow-up with them that’s not so much quantitative? Nancy: Our research team and design will do that and they’ll take a very thorough approach to both recording users using the product, getting their feedback. It’s pretty thorough and also gives us some feedback. We usually don’t do that until the product’s been in the market for a little bit longer. We’ve got some hypothesis of how we think it’s doing, and then the research team will spend a couple of weeks diving a lot deeper into it. We get some great feedback from that. Honestly, as a product person, as much as I’d to think I’m focused on a beautiful experience, my lens versus our designers’ lens is completely different and they just see things we don’t. Brian: Yeah, the friction points and filling in the why’s, it takes time to go and do that, but it can tell you things, it helps you qualify the data, and makes sense especially when you’re collecting. I’m sure at the level that you guys are collecting that, you have a lot of inbound analytics coming back on what’s happening. But it’s really filling in that “why” piece that is really important if you’re going to start changing design because you may not really understand the reasons someone’s abandoning. Maybe it’s like, “I couldn’t find the installer. I don’t know where the URL is. I ended up locking the server on my thing and I don’t know how to localhost, but I forgot the port number,” and the whole product is not getting accessed because they don’t know the port number for the server they installed or whatever the heck it is, and it’s like, “Oh, they dropped off. They couldn’t figure it out how to turn it on, like load the browser…” Nancy: Right, and even behavioral things that we don’t always think of, like putting a really cool graphic in the lead space that actually takes the attention away from the callback-ends. We’re all proud of, “Hey look at this cool graphic we built.” One of our designers uses a tool that tracks eye movements and [wait a second] “We’re losing the focus here.” But again, you don’t always see from that lens. The design part of it, for us has been so eye-opening because again, we’ve built a lot of best in class enterprise products for years. As we shift into this digital go-to market, it is all about the experience. It’s all about how good the experience is, how easy the experience is, how frictionless it is, and it’s also about how consumable and accessible the product is in the marketplace. Brian: You mentioned earlier, it sounded like engineering doesn’t want to go back and necessarily add onboarding on all of this. This gets into the company culture of who’s running the ship, so to speak. Is it engineering-driven in your area? How do you guys get aligned around those objectives? I’ve seen this before with larger enterprise clients where engineering is the most dominant force and sprints are often set up around developing a feature and all the plumbing and functionality required to get that feature done, but there’s not necessarily a collective understanding of, “Hey, if someone can’t get from step A to step G, horizontally across time, then all that stuff’s a failure. Step F which you guys went in deep on is great, but no one can get from E to F, so definitely they can’t get to G.” So, that’s you’re qualifier of success. How do you guys balance that? Who’s running the ship? Does your product management oversee the engineering? Can you talk a little bit about that structure? Nancy: We call operating management aside from product management for a reason, because we really do want the operating managers to feel like they’re the CEO of their business and run the ship. Of course, development has a big say at the table, but they have a natural tendency to want to build capabilities. It’s never going to go away. It’s been that way for ages. We just don’t want to fight that tendency. We want them to focus on building, not take six months to build an onboarding experience when they could build in really valuable functionality in that six months instead. So, we really run it as a squad, just like many other companies. Operating management does leave a lot of the strategy with our products and development, but I would say that design is also a really, really chief at the table, for sure, absolutely. Brian: Tell us a little bit about your squads and is this primarily a designer or a UX professional up in your offering manager? Are they a team and then you pull in the engineering representatives as you strategize? Nancy: My team is a digital offering management. We’re a subset of offering management better known as product management. We will run the squads and the squads will be a cross-function of our product marketing team, our performance marketing team, which is demand to and type marketing. They run the campaigns, design, developments, the core product managers because we’re the digital product managers and such, and then there’s the core product managers. They have all routes to market. We’re just focused on the digital ones. With that is the cross-functional squad that gets together on a weekly basis and they run as a team. From a digital perspective, it’s led by the Digital OM for our route to market there. Brian: That’s interesting. How do you ensure that there are some kind of IBMness to all these offerings? Your UX practice and offering managers sound like they are part of one organization, but I imagine some of these tools, you might be crossing boundaries as you go from tool X to tool Y. Maybe you need to send data over like, “Oh, I have this package of stuff and I need to deploy this model,” then we have a different tool for putting the model into production and there’s some cross user experience there. Can you talk about that? Nancy: That’s really why design’s been key because their job is to keep us onus making sure that the experience is somewhat consistent across the tools so they seem familiar to us, especially within a segment data science. Some of these are using our Watson Studio tool and then moves to our statistics for our modeler tool. There should be a very familiar experience across those. That’s why design is really the lead in the experience part of it. From pricing and packaging, we try to maintain a consistency as much as possible across all the products again. Whatever level of familiarity you have and how we price and package things should be consistent across the entire segment. So we strive for that as well. On the digital side, in terms of the experience on the actual web, we partner with a team called the Digital Business Group. They are basically the host of our digital platform and they maintain a level of consistency worldwide across all the products in terms of the digital journey itself with us. Brian: That’s cool that you guys are keeping these checkpoints, so to speak, as stuff goes out the door. You’ve got the front lenses on it looking at it from different quality perspectives, I guess you could say. Earlier, you mentioned democratizing data science and we hear this a lot. Are we talking about democratizing the results of the data science, so at some point there’s maybe decision support tool or there’s some kind of outcome coming from the data science? Is that what you’re talking about democratizing? Or are you saying for a data scientist of all levels of ability, it’s more for the toolers as opposed to the [consumers..]? Nancy: It’s about the capability. The ability to put more of these products or these products in people’s hands that bought, that they might have been out of their reach, or that they were too enterprisey, or that they are for big companies. That’s one of the key things that we want to do. When you look at some of our products, they start really, really low. Cognizant Analytics is another great example where people might have had a perception that it’s really expensive but we just introduced a new version of it, and it’s less than $100 a month. You can get these powerful tools for analysis for a lot less than you think. Statistics in $99 a month, one of our pay products are significantly less, and it allows these companies that might not have considered doing business with us, to smart small and build up. That’s one of the key things we noticed as we shifted to a subscription model. With that, we started to see double digit increases in the number of clients that were new on products. Just because opened up this new route to market, doesn’t mean that we still didn’t maintain our enterprise face-to-face relationships because, of course, we did, but this allowed us to open up relationships with clients might have not gotten to before. Brian: How are the changes affecting the legacy users that you have? I imagine you probably do have some people that are used to, “Don’t change my toolset,” like, “I’ve been using DB2 for 25 years.” How are they reacting to some of the changes? I imagine at some point maybe you have some fat clients that turn to browser-based interfaces. They undergo some redesign at that point. Do you have a friction between the legacy experience and maybe do you employ the slow change mentality? Or do you say, “Nope, we’re going to cut it here. We’re jumping to the new one and we’re not going to let the legacy drag us back”? You talk about how you guys make those changes? Nancy: We’re shifting towards the subscription model. Our clients are, too. We have clients that are demanding that this is the only way that they actually want to buy software is through a subscription model. So it’s changing for them as well. I think in many ways, it’s a welcome change across the board. I can’t think of any negativity that we’ve had in both the change for the consumption models on a subscription side, as well as the new UI changes and things that we’re doing to the product that really update them and give them a modern feel. I know a lot of the onboarding is a welcome change, even for clients that are familiar with us. It helps them because they have to do less training internally to help people use the tool because now we’re building it into the product. Brian: How do you measure that they’re accepting that? Do you wait for that inbound feedback? Do you see if there’s attrition and then go talk to them? I imagine there’s some attrition that happens when you make a large tooling change. Is there a way to validate that or why that happened? Was it a result of changing too quickly? Any comment on that? Nancy: I think it’s a couple of things. We’re constantly monitoring the flow of MRR and the contraction of revenue where the attrition that we get through some of our subscription, to see if there is any anomalies there. But also we’re always were very in-tune with NPS. A lot of our product managers live and die in the verbatims and with the integration of FLAX, they get a lot of it. They’re coming right at them constantly, that they respond to. We are very, very in-tune with NPS and the feedback we’re getting there. We’re also getting a lot of reviews now on our software using tools like G2 Crowd where we keep an eye on that. I think the feedback doesn’t just come from one place. We’ll look at things like the flow through Amplitude. Our clients, when they’re coming in and during the trial, are they getting stuck someplace? Are they falling off someplace? Are they falling off either at a specific page like the pricing page? Or are they falling off as soon as they get the trial because they don’t know what to do with it? We look at things like that. We look at NPS in particular after we’ve introduced new capabilities. Did our NPS go up? What’s the feedback? Are our clients truly embracing this? I think it’s a combination of things. There is a lot of information, a lot of data that we just need to stay in-tune with. We’ve got a couple of dashboards that I know my team wakes up with everyday and takes a look at, and the product team. The core product manager stayed very focused on NPS. Brian: Do you have a way of collecting end-user feedback directly? I would imagine maybe in your newer tools, it’s easier to tool some of that in, but is there any way to provide customer feedback or something to chat or any type of interactivity that’s directly in the tools that you’re creating these days? Nancy: Sure. We are rolling out more end-product nurture capability than we ever had before. That gives them the ability to chat directly within the product, as well as schedule a time with an expert. We’re working in making that even easier through a chat bot. So if you do get stuck and you’re chatting with that bot, you can schedule the appointment with an expert right there. I think there’s lots of ways to do that. I think sometimes I worry that there’s too much data coming at us but we [didn’t have enough..] before, so I’m not going to do that. Brian: Right. It’s not about data, right? It’s, do we have information? Nancy: Do we have information? Exactly. I would say my team spends a lot of time going through that, looking at Amplitude, analyzing the flows, looking in the patterns, in the orders, in the data, and the revenue. With the NPS feedback, it’s a combination of all of that stuff that really gives us a good view. As well as looking at the chat data, and analyzing some of the keywords that’s coming across on the chat, the Watson robots are constantly learning, which is great. We’re using machine learning to get smarter about what do people ask about, and that’s giving us also some good insight into the questions they ask, the patterns of information they’re searching for by product. Brian: In terms of the net promoter score that you talked about, tell me about the fact that how do you interpret that information when not everybody is going to provide a net promoter score? You have nulls, right? Nancy: Right. Brian: How do you factor that in? That’s the argument against NPS as the leading indicator. Sometimes, it’s not having any information. So you may not be collecting positive or potentially negative stuff because people don’t even want to take the time to respond. Do you have comments on how you guys interpret that? Nancy: I think you also have to look at the NPS is going to go up and down. If you have a client who has particularly a bad experience, it’s the week of thanksgiving, there was only X amount of surveys, and one of them had a bad experience that could make your NPS score looks like it drops like a rock. [right] you’ve got to look at it like the stock market. It’s more of the patterns over the long haul, what’s coming across within those patterns of information and feedback the clients are giving you. We react but you have to look at the data set, you have to look at the environmental things that are happening, and take that all into consideration from an NPS perspective. We’re very driven by that and that comes down from our CEO. She’s very cognizant, making sure that the product teams and the development teams are getting that feedback directly from the clients. As an organization—we’re a few years old—the way we used to do that is we would have these client advisory boards. It was a small number of clients that would give us feedback on our products, roadmap, and usability of that. The reality is just that then you end up building the product for 10% of your clients. Now it’s been eye-opening for us as we really open that up. Obviously, we’re still getting feedback from a larger community and client advisory board still, but NPS comments and feedback has really widen the aperture of the feedback we’ve gotten from a broader scope of clients. Brian: You brought up a good point. I had a client who luckily was cognizant of this and they did the same things where they fly their clients, they do two-day workshops, and they gather feedback from them. I was doing some consulting there and he said, “Brian, I’d like you to just go walk around, drop in some of the conversations and just listen, but take it with a grain of salt because I hate these freaking things. All we do is invite people that are willing to come for 2–3 days and tell us how much they love our stuff, it’s a free trip, we’re not getting to the people that don’t like our stuff…” Nancy: Or don’t use it. Brian: Or don’t use it at all. I love the concept of design partners, which is new, where you might have a stable of customers who are highly engaged, but that the good ones are the ones that are engaged who will pummel you when you’re stuff is not happening. They will come down on you and they will let you know. So it’s really about finding highly communicative and people who are willing to tell it like it is. It’s not, we’ll go out and find people that rah-rah, cheerleading crowd for you. Did that inspire the changes? Nancy: Even in the client advisory councils that we had—I ran a couple of them for products like Netezza for a while—we started to change the way we even ran those. I remember the biggest aha moment was, we had a client advisory board for Netezza one year and not too long ago. We decided to run a design thinking camp as part of the agenda, so that they would actually drive what they wanted from our requirements prospectus, going through the design thinking process through that. What came out of it was truly eye-opening. You know how a design thinking process progresses. I think even they were surprised at what they ended up prioritizing across the group of requirement. I think we’re really starting using differently about that feedback from clients. I do remember that day when we were looking at those things and that was not where we thought we would end up. Brian: Do you have a specific memory about something that was surprising to the group that really stuck? Something you guys learned in particular that stuck with you? Nancy: I think we focused a lot more at that point. At the time there was a lot of issues around security and what was one of our leading things going into the next version. What clients actually were not necessarily as verbal about was that, as they were using these appliances and they were becoming more mission-critical, they were doing more mixed workloads. Yes, security was still incredibly important, but what was emerging beyond that for them was workload management because they had this mixed workload that was emerging. So many different groups were jumping in with different types of workload. They have not anticipated on their [day route?] appliance, so it was something that I think came out of the next in the design thinking process that was important to them that they actually hadn’t been able to verbalize to us. Outside of that process, which was really, really interesting to us, we were on track with the requirements that we have but beyond that, the requirements that we just hadn’t thought of and quite honestly they hadn’t verbalized. Brian: You make a good point there. Part of the job of design is to get really good clarity on what the problems are and they’re not always going to be voiced to you in words or in direct statements. It’s your job to uncover the latent problems that are already there, crystalize them, so ideally whoever your project manager in the organization and your leadership, can understand and make them concrete because then you can go and solve them. When they’re not concrete and vague, like, “We need better security.” But what does that mean specifically? If you start there and really the problem had to do with the mixed workloads and managing all that, it’s like you can go down a completely different path. You can still write a lot of code, you can build a lot of stuff, and you can do a lot of releases, but if you don’t really know what that problem is that you’re solving, then you’re just going through activity and you’re actually building debt. You’re building more technical debt, you’re wasting money and time for everybody, and you’re not really driving the experience better for the customer. I think you made a good point about the design thinking helps uncover the reality of what’s there, when it’s not being explicitly stated, support requests are not going to get that type of information. They tend to be much more tactical. You’re not going to get a, “Hey, strategically I think the project needs to go this direction.” Nancy: Right and if you would have asked of us an open-ended question, you would have gotten and answer that could have been interpreted slightly differently. I think this was when I became the biggest fan of design is that, there was this magical person who was running this design camp for me that got information that I didn’t think I could get to. I mean, I knew nothing about the product. It was pretty amazing. Brian: That can happen when you also get that fresh lens on things even when they may not be a domain expert. You get used to seeing the friction points that people have and you can ask questions in a way to extract information that’s not biased. You’re not biased by the legacy that might be coming along with that product or even that domain space. It’s sometimes having jthat almost like first grade, “Tell it to me like I’m your grandfather,” or, “Explain that to me this way,” and then you can start to see where some of those friction points are and make them real. I always enjoyed that process of when you’re really fresh. Maybe this happens for other people but especially as a designer and consultant, coming into a product and a new domain, and just having that first-grader lens on it like, “Hey, could you unpack that for me?” “What is the workload in there like?” looking at you like, “What?” and you make them unpack that but you give that full honesty there to really get them to extract out of their head into words that you [and.] everyone can understand. That’s where some of those magical things happen like, “Oh my gosh. We had no idea that this was a problem,” because he or she thought it was so obvious like, “Of course, they know this,” and it’s like, “No. No one’s ever said that.” Nancy: Right. We’re experiencing that now. We have an embedded designer into our team that’s focused on our growth products. Again, she’s coming in with a complete fresh set of eyes and her perspective that she brings on the experience is just so completely different, not completely different but there are things that she flushes out we would have never see. It’s really helping because a lot of times, too, when you’re focused on the experience as opposed to the features and functions analysis, and you come down to looking at it from that perspective. I don’t want to go to development and tell them this because it’s like calling their baby ugly. But at the end of the day, the client needs to have a great experience. They need to see the value. When they’re even just trying the product out, they don’t get to that aha experience like, “I know how this will help me within 15 minutes.” We’re just not nailing it. If they can’t figure out how to jump in and use the product, we’re not nailing it. It doesn’t matter how great the product is, if they can’t figure out how to effectively interact with it. Brian: Effectively, none of the stuff really exist in their world. It just doesn’t exist because they can’t get to it. So, effectively it’s totally worthless. Whatever that island you have on the island, if there’s no bridge to get there, it doesn’t matter because its just totally inaccessible. Nancy: Right and it’s harder sometimes for even the product managers to see it. When I was sitting down in a demo of a product that we are going to be releasing, dude was cruising through the demo, my eyes were like glazed over, I just look and I was like, “Boy, we’re going to need some onboarding with that.” Great product, amazing capabilities, very complex and dense in its capability. It’s never really about whether it’s a great product. It’s about whether the client understands that’s great, when they start using it. Brian: Yeah and I think especially for analytics tools, highly technical tools used by engineers and other people that have better working in this kind of domain. Sometimes we gloss over stuff that seems like it would be totally easy or just not important. I have this specific example I was working on a storage application. It was a tool I think for migrating storage between an old appliance and a new appliance. At some point during that workload migration, something as simple as like, “Oh, I need a list of these host names and these IP addresses,” some other information that’s just basically setup-related stuff, and all the tool needed to do was have a CSV download of a bunch of numbers to be piped into another thing so that they could talk to each other. It’s not sexy. It’s literally a CSV. It was the only technical lift required, but it was not seen as engineering. It’s not part of the product. That has to do with some other product but you have to go type it into. It’s like yes, but that bridge is never going to happen. It takes them 10 years to go figure out where all these IP addresses are listed, domain names, and all these kind of stuff. It’s not sexy but if you look at the big picture, the full end-to-end arc, and if we’re all lying around, what is that A to G workflow, there’s six steps that have to happen there. This is not sexy, it’s not a new feature but this is the blocker from getting from B to E. They’re never get to A, which is where the product begins. Nancy: We definitely had those discussions in the early days about making it more consumable instead of giving it more features and functions, and can’t we really hack growth that way? That is a mind shift that if you are a design-led organization, you get it, and we believed in every part of our being that we are. Sometimes we still have that natural resistance that we really need to add more features and functions to make this product grow, but I think we’ve really turned the corner on that. Digital really has been the task for us to do that because we build the experience in the products as if there was no IBM sales team that’s going to surround you to help make you a success. That’s a very different way that we’ve done things for so many years, and the only way you can do that is by focusing on experience. Brian: You bring up a good point and I think that it’s worth reiterating to listeners. You can add these features but they do come at a cost. The cognitive load goes up. Every time we add to the tool, we’re effectively reducing the simplicity of everything else around it. Typically as a general rule, removing choice simplifies because you’re just removing the number of things that someone has to think about. So those features don’t really come for free. It’s almost like you have a debt as soon as you add the feature and then you hope you recoup it by, “Oh there’s high engagement. People are really using that,” so that was a win. If there’s low engagement with it, you just add it. It’s like Microsoft Word 10 years ago. You just added another menu bar and another thing that no one’s gonna use, and now it’s even worse. The pile continues to grow and it’s so hard to take stuff out of software once it’s in there, because you’re going to find, “You know what? But IBM’s our client, and they’re using it. IBM makes $3 million a year. We’re not taking that button out of the tool. End of story,” and now you have that short-term like, “We can’t take that out because Nancy’s group uses this.” Nancy: That’s right and we can’t point out exactly. I think my favorite story when it comes to that is the Instagram story that people talk about, where it was launched as a tool, a product called Bourbon. It had all of these great capabilities and it was going nowhere. So they dug into the usability side of things and said, “Well, what are people actually using?” which is what we do as well from an instrumentation perspective, and found that they were really only using a couple of things. They wanted to post a picture, they wanted to comment on the picture, they want to add some sort of emojis or in like system the picture and they are like, “Let’s [do.]. Let’s just do three or four things, do them really great, and relaunch the product,” and then of course the rest is history. I think that that’s a great illustration of more features and functions. If they’re not important, relevant, and consumable, all three of those things, are not going to give you growth. It comes down to, is it easy to use? Can I get value out of it? Do I immediately see that I can get value out of it? That’s all product market fit. That’s where we shifted our focus and digital’s helped us, too. That’s why my job is so cool. Brian: Cool. This has been super fun. Can you leave us with maybe an anecdote? Do you have a big lesson learned or something you might recommend to people that are either building internal tools, internal enterprise software or even SaaS products, something like, “Hey, if I was starting fresh today, I might do this instead of doing that.” Anything from your experience you could share? Nancy: For me, the biggest thing is just really focusing on product market fit because we build something sometimes to be competitively great, but not necessarily competitively great and competitively different, or that. So to understand that you not only have something that solves somebody’s problem but does it in a way that’s unique, and that’s so valuable that they’ll pay the price that’s appropriate for whatever they’ll pay for it. You’ve got to start thinking about that upfront because oftentimes, we’ll build something we’ll see a market opportunity for, but we may not truly understand product market fit whereas we know who the target is, we know what they’ll pay for this, we know what the value is, we know how to get to them, and I think you’ve got to start with that upfront, like you really got to understand product market fit or you’re never be able to grow the product. I’ve got a lot of religion around that and we really try very, very hard to create pricing and packaging around making sure we hit that, but the product has to have that value. It can’t be too overwhelming, it can’t be too underwhelming, it’s got to hit that great value spot. Brian: Fully agree on getting that fit upfront. You save a lot of time, you could solve a lot of technical debt instead of jumping in with the projects that you going to have to change immediately because you find out after the fact and now you’re starting it like… Nancy: See you in Instagram not a Bourbon, right? Brian: Exactly. Tell us where can people find you on the interwebs out there? Nancy: I probably spend a lot of time on Twitter. Maybe not so much lately. It’s been a little bit crazy but you can find me on Twitter @nancykoppdw […] or you can find me on LinkedIn. I am going to try and do better. I am on Medium. I haven’t done as good about blogging but that’s one of my goals for trying to get back on blogging. I’m usually out there on Medium or Twitter talking about growth hacking and digital transformation. I do podcast as well. Brian: Cool. I will put those links up on the show notes for anyone. Thanks for coming to talk with us, Nancy. It’s been fun. This has been Nancy Hensley from IBM Analytics, the Chief Digital Officer. Thanks again for coming on the show and hope we get the chance to catch up again. Nancy: Thank you.

ceo chicago online design digital tools mvp medium cloud mass boy honestly cambridge ibm ip tap pricing saas statistics explain joke operating ux bourbon ui nps modernizing chief digital officers microsoft word hensley mrr flax amplitude csv spss g2 crowd db2 brian yeah brian you brian how ibm analytics netezza brian right nancy you brian exactly watson studio digital om

006 - Julien Benatar (PM for Pandora's data service, Next Big Sound) on analytics for musicians, record labels and performing artists

Play Episode Listen Later Feb 13, 2019 45:01

We’re back with a special music-related analytics episode! Following Next Big Sound’s acquisition by Pandora, Julien Benatar moved from engineering into product management and is now responsible for the company’s analytics applications in the Creator Tools division. He and his team of engineers, data scientists and designers provide insights on how artists are performing on Pandora and how they can effectively grow their audience. This was a particularly fun interview for me since I have music playing on Pandora and occasionally use Next Big Sound’s analytics myself. Julien and I discussed: How Julien’s team accounts for designing for a huge range of customers (artists) that have wildly different popularity, song plays, and followers How the service generates benchmark values in order to make analytics more useful to artists How email notifications can be useful or counter-productive in analytics services How Julien thinks about the Data Pyramid when building out their platform Having a “North Star” and driving analytics toward customer action The types of predictive analytics Next Big Sound is doing Resources and Links: Julien Benatar on Twitter Next Big Sound website Next Big Sound blog The Data Pyramid model Quotes from Julien Benatar "I really hope we get to a point where people don’t need to be data analysts to look at data." "People don’t just want to look at numbers anymore, they want to be able to use numbers to make decisions." "One of our goals was to basically check every artist in the world and give them access to these tools and by checking millions of artists, it allows us to do some very good and very specific benchmarks" “The way it works is you can thumb up or thumb down songs. If you thumb up a song, you’re giving us a signal that this is something that you like and something you want to listen to more. That’s data that we give back to artists.” “I think the great thing today is that, compared to when Next Big Sound started in 2009, we don’t need to make a point for people to care about data. Everyone cares about data today.” Episode Transcript Brian: I’m really excited today for this episode. We have Julien Benatar on the show and he’s from a company that I’m sure a lot of people here know. You probably have had headphones on at your desk, at home, or wherever you are listening to Pandora for music. Julien , correct me if I’m wrong, you were the product manager for artist tools and insights at Next Big Sound, which is a type of data product that provides information on music listening stats to, I assume, artists’ labels as well to help them understand where their fans are and social media engagement. I love this topic. I’m also a musician, I have a profile on Next Big Sound and I feel music’s a fun way to talk about analytics and design as well because everybody can relate to the content and the domain. Welcome to the show. Did I get all that correct? Julien: Yeah, it was perfect. Brian: Cool. Tell us a little about your background. You’re from France originally? Julien: Yes, exactly. I grew up next to Paris, in Versailles more specifically, and moved to New York in 2014 to join Next Big Sound. Brian: Cool, nice. You’ve been there for about four years, something like that. You have a software engineering background and then now you’re on the product side, is that right? Julien: Exactly yes. I joined the company back when we were a startup. Software engineering was perfect, there was so much to do. To our move to Pandora, I moved to a product manager role around a year ago. Brian: Next Big Sound was independent and then they were acquired by Pandora. I assume there is good stuff about your data. Why did Pandora acquire you and how did they see you guys improving their service? Julien: We got acquired in 2015. The thing is, Next Big Sound was already really involved in the music industry. We already had clients like the three major labels and a lot of artists were using us to get access to their social data. I think it was a very natural move for Pandora as they wanted to get closer to creators and provide better analytics tools. Brian: For people that aren’t on the service, I always like to know who are the actual end users, the people logging in, not necessarily the management, but who sits down and what are some of the things that they would do? Who would log in to Next Big Sound and why? Julien: Honestly, it’s really anyone having any involvement into the music industry, so that can be an artist, obviously, try looking to try their socials and their audience on Pandora. But you can also be a booker trying to book artists in their town. We have a product that can really be used by many different user personas. But our core right now is really artists and labels, having contents on Pandora and trying to tell them the most compelling story about what they’re doing on the platform. Brian: When you think about designs, it’s hard to design and we talk about this on the mailing list sometimes but it’s really hard to design one great thing that’s perfect for everybody so usually you have to make some choices. Do you guys favor the artist, or the label, or as you call them,the bookers or whom I know as presenters,in the performing arts industry? Do you have a sweet spot, like you favor one of those in terms of experience? Julien: I think it’s something we’re moving towards, but it hasn’t always been this way. Like I told you, we used to be a startup or grow us to make a product that could work for as many people as possible. What is funny is we used to have an entity on Next Big Sound called Next Big Book where we used to provide the same type of service for the book industry. If anything, it’s been great to join Pandora because then we could really refocus on creators and it really allowed us to, I believe, create much better and more targeted analytics tools to really fulfill needs for specific people like artists and labels. Brian: I would assume individual artists are your biggest audience or is it really heavily used by the labels or who tends to... Julien: I think it’s pretty much the same honestly. I think the great thing today is that, compared to when Next Big Sound started in 2009, we don’t need to make a point for people to care about data. Everyone cares about data today. I think that everyone has reasons to look at their dashboards and especially for a platform like Pandora with millions of users every month. Our goal is really just telling them a story about what does it mean to be spinning on the platform and the opportunities it opens. Brian: You talked about opportunities, do you have any stories about a particular artist or a label that may have learned something from your data and maybe they wrote to you or you found out like in an interview how they reacted like, “Hey, we changed our tool routing,” or, “Hey, we decided to focus on this area instead of that area.” Do you know anything about how it’s been put into use in the wild? Julien: Yeah, it’s used for so many different reasons. For the people who don’t use Pandora, something I really like about the platform is it’s really about quality. As you use Pandora, you have the opportunity to thumb up or thumb down songs and as you do, you’re going to get recommended more songs like the ones you like. It’s really about making sure that you get the best songs at all times. The reality then is that for artists, their top songs on Pandora can be pretty different than their top songs on other platforms because sometimes their friends are going to be just reacting more to some part of their catalog than another one. I’ve heard many times of artists changing their playlists in looking at which songs where their fans thumbing up the most on Pandora. Brian: Could you go through that again? How would they adjust their playlist? Julien: Usually, people use Pandora as a radio service. While we already have internet today, most people are listening to the radio because they’re usually are very targeted and it just works really well. The way it works is you can thumb up or thumb down songs. If you thumb up a song, you’re giving us a signal that this is something that you like and something you want to listen to more. That’s data that we give back to artists. We tell them, “This are your most thumbed songs on Pandora. These are the songs that people engage with the most on the platform.” Looking at this data, you can actually inform them songs that they believe they should be playing more on the store. Brian: I see. A lot of it has to do with the favoriting aspect to give them idea what’s resonating with their audiences. Julien: Qualitative feedback, yes. Brian: Got it. Actually, it’s funny you mentioned the qualitative feedback. In preparation for this, I was reading an article that you guys put out back in March about a new feature called weekly performance insights, which is really cool and this actually reminds me of something that I talked about in the Designing for Analytics mailing list, which is the act of providing qualitative guides with your analytics. A lot of times they analyze for turnout quantitative data and whenever there’s an opportunity to put stuff into context or provide qualifiers, I think that’s a really good thing and you guys look like you’ve have done some really nice things here. I’ll paraphrase it and then you can jump in and maybe give us some backstory on it. One of the things that I think is really cool is there're concepts of normalcy in here so that, if I’m an artist and I look at my numbers, I have an idea. For your Twitter mentions, for example, you say, “For artists with 26,000 followers, we expect you to get around 44 mentions.” When you show me that I have 146 mentions, I can tell that I’m substantially higher than what my social group would be. I think that’s a really fantastic concept that people not in music could try to apply as well which is, are there normalcy bans where you’d want to sit? Is there some other type of group, maybe, an industry, or apparent group, or another business unit, whatever it may be to provide some context for what these out of the blue numbers mean that don’t have any context? How did you guys come up with that and can you tell us a bit about the design process of going from maybe just showing, “You’re at 826 apples,” as compared to what? How did you move from just a number into this these kind of logical groupings where you provide the comparisons? Julien: I think what’s really fascinating is, we really live in an age of data. As an artist, you need to be on social media for the most part. There still a lot of artists I listen to but just decide not to. It’s part of things but at the same time, real big success in the music industry didn’t change. It’s still being on the Billboard chart, getting a Grammy and all these things. But as we see this, we have millions of artists looking at their data every day and just are not able to understand, like is it good or is it not good. Everyone starts at zero. We have a strong belief that data can only be useful when put in context. Looking at the number on its own can give you a sense of how things are doing but that can also be dismissive. An example is, a very common way to look at data is to look at a number and look at the percent changing comparison to the previous week. You’ve got a bunch of tables and you look at, am I growing or am I not growing. The reality is it’s actually impossible to always have a positive percent change. There’s no artist in the world that always does better week by week. Even Beyonce, I can assure you that the week she released Lemonade, she had more engagement on Twitter than the week after. With that in mind, we really try to give a way for artists to understand how are they doing for who they are and where they are currently in their career. Next Big Sound started in 2009. One of our goals was to basically check every artist in the world and give them access to these tools and by checking millions of artists, it allows us to do some very good and very specific benchmarks. For an artist, like the example you said, for instance an artist with a thousand Twitter mentions in a week, is it good or bad in comparison to their audience size? This feature comes because that’s just the question we’re asked. Artists want to know is it any good? What does this number actually mean for me? That’s why we really wanted to, in some ways, get out of being a content aggregator platform and really be a data analytics platform. How can we actually give information that can help artist make better decisions? Brian: I remember the first time I got what I would call an anomaly detection email from your service and it was about some spike in YouTube views or something like that. I thought it’s fantastic in two reasons. First of all, you identify an anomalous change and I think in this case it’s a positive anomalous change. That tells me that I should log in the tool. Secondly, you proactively delivered that to me. On the Designing for Analytics mailing list, we talk about is that user experience does not necessarily live inside your web browser interface or your hard client or whatever you’re using to show your analytics. Email and notifications are a big part of that. Can you tell me about how you guys also arrived at when you pushed these things out and maybe talk about this little anomaly detection service that you have? Julien: It all started when we got acquired by Pandora. We decided to just invite a bunch of users and just talk to them, understand how to use our product and what did they think about it. We had artists, managers, and label people come over and we just talk to them and basically they all said, “We love it.” But then, by looking at their actual usage, they don’t use it that much. I guess one of their questions was when should I be looking at my data? Everyone is very busy. As you’re an artist, you need to perform, you need to write music, you need to engage with your fans and same goes with everyone. When should I look at data? The reality is by being a data company, we do get all the data, we have all the numbers. We have ways to know when things are supposed to be known, when artists should be acting on something. We just turn this into this email notifications. Anytime we notice that an artist is doing better than expected, we just let them know right away. Brian: That’s great. Do you do it on the opposite end too? If there’s an unexpected drop or maybe like, “Oh, you put a new track out and your socials dropped,” or something like that, do you look at the negative side too or do you tend to only promote the positive changes? Julien: As far as pushes, we decided to only do push for positive. But as you mentioned weekly performance, weekly performance can give you some negative insights, like, “You’re not doing as well as artists with the same size of audience as yours.” The reason we didn’t do it for our notification is, anomalies are really hard to completely control. A reason, for instance, is Twitter removing bots. Basically, every single artist would have had an email telling them, “You lost Twitter followers this week.” It was a lot of work to really tune our anomaly factor to actually only send emails when something legitimate happens. That’s the reason we only decided so far to do it for positive but we actually have been thinking about doing the same for negative but that’s another type of work. Brian: Yeah, you’re right. You have to mature these things over time. You don’t want to be a noise generator. Julien: Exactly. Brian: Too many, then people start to ignore you. I’ve seen that with other data products I’ve worked on which just have really dumb alerting mechanisms that are very binary or they’re set at a hard threshold and just shootout noise and people just tune it out. Julien: I’m glad you mentioned this because this feature was in beta for a year for that specific reason. Brian: Got it. Julien: We had to learn the hard way. We had like a hundred beta users. We’ve got way too many emails because anytime there were an anomaly anywhere, they would just get an email. For the most part, it was things that were supposed to help them. If a notification becomes noise, then that’s absolutely against its purpose. Brian: I don’t know if everybody knows how the music business works, at least from the popular music side, but just to summarize. You have individual artists that are actually performers. They may or may not have an artist manager which takes care of their business affairs, represents them like negotiations with people that book shows. Then you have labels which are sort of like an artist manager except they’re really focused on the recording assets that the artist makes and they actually tend to own the recordings outright at the beginning and then over time, the artist may recoup through sales they make it the ownership act and the sound recordings they make. Of those kinds of three major groups, is there a one that’s particularly hungry or you’re the squeaky wheel that is most interested in what you’re doing? Julien: I really think that into these three groups, we have a subset of users that are really into the data and into the actionability of it. I don’t think it’s one specific group of user. It could be all around the industry like we have the data-savvy, they really want to know. We have some users that actually would rather get more notifications even if they need to on their end to figure what is right from what is wrong. But since we have such a wide user base of different type of people, we decided to go on the conservative side and make sure to only share things that we thoroughly validated through all of our filters. Brian: I assume that your group reports into some division of Pandora, I’m not sure of that. Are you reporting into a technology, like an IT, or a business unit, or marketing? Where do you guys fit in the Pandora world? Julien: We’re part of the creator’s tools. I don’t really have a perfect answer to this. Brian: Okay. I guess my main question being, because when we talk about designing services, we talk about both user experience, which is the end user thing and about business success or organization success. I’m curious, how does Pandora measure that Next Big Sound as delivering value? I can understand, I’m sure our artist can understand how the artists value it through understanding how is my music moving my audiences, et cetera. Is there a way that Pandora looks at it? Are they interested in just time spent? The analytics on the analytics, so to speak, is what I’m asking about. How do you guys look at it like, “Hey, this is really doing a good job,” or whatever? Do you know how that’s looked at? Julien: To be honest, I think you said it right. Our goal is to help artists make their decisions through data and having artists use the platform is currently the way Pandora sees us doing a good job. Actually, it hasn’t changed that much since our acquisition. One of our main KPI for the past and couple of years is something I would call insights consumes. Just making sure that our users, artists, anyone using Next Big Sound are consuming data. That can be them logging into the website or that can be them opening one of our notifications. But so far that was our main KPI. We’re trying to work on some more targeted KPI, potentially like actions taken, that would be the North Star, but we're still working on how to do that right. Brian: Do you guys facilitate actions, so to speak, directly in the tool or are there things people can do with those actions really take place outside of the context of Next Big Sound? Julien: There are actions that artists can take to the other creator’s tools provided by Pandora. For instance, artists have the ability to send audio messages to anyone listening to them. If they go on tour into the US, they can have targeted messages in every single song they’re going to play. If anyone listens to them there, they can just click and buy a ticket. We’re working to make sure that artists are aware of these tools because they are free and they’re generally helping them grow at their careers. But regarding external actions, so far we don’t have any one-click way to tweet at the right time to the right people or with the right content or anything like this. Brian: Sure and that’s understood. Not every analytics product is going to have a direct actionable insight that comes right out of it. You guys may be feeling a longer term picture about trending and maybe for a certain artist to get an idea if they’re releasing music fairly frequently, what stuff is working and resonating, and what stuff is not. I can understand that. There may not be a button to click as a result immediately. Julien: That’s the goal though. Everything we do right now is going towards this objective. Maybe I can tell you a little about the way we think about data and that can give more sense to it. In order to work on any new feature, we follow this concept called the data pyramid. It’s something that you can Google. There’s a Wikipedia page for it. Let me explain to you how it works. The data pyramid, it’s a pyramid formed of four layers. It could be upon each other and each representing an exquisitely useful application of data. At the bottom of the pyramid we have the data layer. Any sort of data that we may have. For our case, Android data, Twitter, Facebook just getting the numbers, getting the raw data. On top of it, we have the information layer. The information layer is going to be ways you have to visualize this data. I guess it’s like the very broad sense of analytics. We’re going to give you tables, graphs, pie charts, you name it. We’re giving you ways to craft stories about this data but it’s on you to figure it out. Then on top of it we have what we call the knowledge layer. That’s where things start to get interesting. The knowledge layer is the contextual part of it. It’s like, “What do this number actually mean?” It has industry expertise. For instance, the way we’re going to work about it for musicians and their true data may be different than any other industry. The knowledge layer goes like a weekly performance. It’s a perfect answer to it. It’s what does it mean for me as a musician with a hundred fans to get two mentions this week. Same for notifications. It’s telling you that you should be looking at your data right now because something is happening. That’s how we get to the North Star and the last part of the data pyramid which is intelligence. The goal of intelligence is actionability. Now that I get to understand what does this number mean to the specific context, what should I be doing? Following your question, everything we’re trying to do here is to get to a point where we can just send an email to an artist and tell them, “Hey, you should be doing this right now because, with all the data that we have, we believe that this is going to have the highest impact for you.” Brian: It‘s really fascinating that you just outlined this data pyramid. I actually haven’t heard of this before. It made me think of one of the kind of, it’s not a joke but in the music community, I’m also a composer and when we write stuff, the kind of running joke is like nothing is new. Your ideas for this new song or this new melody I’m composing, it probably came before you. You heard it there before. I wrote a post on my list that was pretty much exactly the same thing except the knowledge layer. I was calling that insight. Data have been this raw format and information being the first human-readable format that’s like say going from raw data to a chart, a histogram. Now I have a line on a chart and then the insight layer being, I have a line on the chart and another line comparing it to like you said, average, or my social group, or a parent group, or some taxonomy, or an index. Then the action or the prescription for what to do or the prediction those that kind of lead you in about action which would be that fourth state. You’re like, “Oh, is this really a new concept?” It’s like, “Nope. Someone else already thought of that.” I totally want to go read about this data pyramid. Julien: That’s amazing. Brian: I’ll find that link to the data pyramid and I’ll put that in the show notes for sure. I thought that was really funny. Julien: It’s funny that you called it insight because that’s the way we call a lot of our features are working out. The way we define insight is bite-size, noteworthy, sharable content. How can we get into the noise of all of the data that only gives you exactly what you should be looking at. That’s how we got into notification and weekly performances. This is the one thing you should be looking at. Brian: I understand what you’re getting at there. The insights are, like you said, bite-size chunks of interesting stats that someone can put some kind of context around. That’s great and it’s good. One of the things I liked, too, that you talked about was you said, “Oh we got like a hundred users, like a beta group and that kind of inspired some of this.” Your product response to how do we help people know when to come and look at our service. I think this is really good because one of the problems that I see with clients and people on the list, I think is low engagement. This is especially true for internal analytics companies. Low engagement can be a symptom of a difficult product, it doesn’t provide the right information at the right time, it may not have a lot of utility, or it’s a resistance to change. People have done something the old way and they don't want to do it the new way. One of the recipes you can follow if you’re trying to do a redesign or increase engagement is to involve the people that are going to use the service in the design process, both the stakeholders as well as the end customers. This is especially true again for the internal analytics people. Your customers or other employees and your colleagues. By engaging them in the design process, they’re much more likely to want to change whatever they’re doing now. I loved how you guys did some research. Now I want to ask, do you frequently do either usability testing or interviews? Is that an ongoing thing at your company or is it really just in front of a big feature release or something like that? How do you guys do this research? Can you tell me about that? Julien: Of course. It’s consent. We haven’t released any major feature without doing some heavy user testing. I’m very lucky to be working with two designers, Justin and Anabelle who are very user-focused. Honestly, if you come to our office, at least every week we’re going to have some user interview and just talking to them, showing them prototypes, and just see how do they play with it. Brian: So you’re doing a lot of testing it sounds like. That’s fantastic. Julien: At the same time it’s always to find the right balance because you could be overtesting things too. We really are focusing on user testing for new things and make sure that the future that we are working on actually answers their user story that we intended. Brian: I don’t know how involved you get participating in these, but do you have any interesting stories or anecdotes that you got from one of those that you could share? Julien: Let me think. I do participate into a lot of them but I’m not sure I have an example right now. Brian: Are most of the people you interview, are they current users of Next Big Sound or do you tend to focus on maybe artists that haven’t experienced the service yet or you mix it up? Julien: We mix it up. We mostly engage with users that we already have but then we can decide to go with users that haven’t used the platform for a while, or more active users if you want to understand how we’re useful into their day to day. What I would say is that, surprisingly, it’s very easy to get users to chat about their experience with the product. I didn’t assume that we would get so many responses when we tried to have people come over or just hop on the zoom to check a new feature. Brian: I’m glad you actually mentioned that because I think in some places, recruiting is perceived to be difficult and it probably isn’t. Maybe you haven’t done it before but as I tell a lot of my clients, a lot of people love to have someone listen to them talk, tell them all about their life and what’s wrong with it, and how it could be better with their tools. They love having someone listen to them and especially if they know that their feedback is going to influence a tool or a service that they’re using. They tend to be pretty engaged with it. I find it’s really rare that I do an interview with a client’s customer and they don’t want to be included in the future round like, “Hey, when we redesign the service, can we come back to you and show you what we’ve done?” “Oh, I love to do that!” Everybody wants to get engaged with it. There are places where recruiting can be difficult when it’s hard to access the users, some of the enterprise software space that can be an issue sometimes. But generally, if you can get access to them, they tend to be pretty willing to participate. I’m glad you mentioned that. Julien: I think the great part about testing with current users on the platform is to actually show them prototypes with real data, not just show them an abstract idea that we want to work on. As soon as they can see what we’re working on apply to their own career as musicians, for instance, that can lead to fascinating discussions. Brian: You made a really good point on the real data thing. I remember as far back as 10 years ago or whenever, I use to work at Fidelity Investments, we would see this issue when we’re working on the retail site for investors. When you show a portfolio that, for example, has Apple stock trading at $22 in it, you’re not really there to test what is the price of Apple stock but you might be testing something entirely different and the customer cannot bear what is going on? They’re so stuck on this thing. It’s all fake seed data in the prototype. The story here being if you’re a listener, when you test it’s important to have at least realistic data. You don’t want to have noise in the test or whatever your studying or else you can end up on this tangent. Try to make the numbers looks somewhat realistic if you’re using quantitative data. In some cases, people can be taught to roleplay. Pretend you’re Drake or pretend you’re some big artist and then they can get their head around why they have billions of streams instead of thousands which they’re used to. Julien: Absolutely. That also helps us just build better products because the reality is we have a lot of artists with maybe 10 plays in a month. As we build visualizations like something that we built a line of looking at Drake’s data, it’s not going to work as intended for a smaller artist sometimes. Having real data involved as soon as possible into the design process has been such a game changer for us. We really have a multidisciplinary team involved into the research and design of everything we do. I’m working with a data scientist, data engineer, a web engineer, and designer on a daily basis. Obviously, we all have our things to do. But as we get into creating something new, we just make sure to have someone helping us get the real data, interview the right user, and just create prototypes as soon as possible. Working with prototypes is essential into building useful data analytics tools. Brian: Yes, you do learn a lot more with a working prototype. It’s not to say you can’t test with lower fidelity goods, especially early on but for a service like yours when the range of possible use both the personas and also you’ve got the Drakes of the world, big major label artist and then down to really small independents, it’s really important to have an idea how your charts are going to scale, and what’s going to happen with data. Even just small stuff like how many decimal points should you be showing on a mobile device, some of the numbers might cram up. Julien: Exactly. Brian: All this stuff that you never think, if you only look at one version of everything, you can end up with a mess. I’m glad that you brought that up. Julien: I couldn’t say better. The decimal is actually something that we’ve had to discover through real data. Brian: To all of you in the technical people out there, I will say this. If I’ve seen one trend with engineers, is they love precision and there’s a lot of times when there’s very unnecessary precision being added to numbers. Such as charts and histograms. Histograms are usually about the trend, they’re not about identifying what was the precise value on this date at this time. It’s about the change over time. Showing what’s my portfolio worth down to three digits of micro-cents or something like that is just unnecessary detail. You can probably just round up to the dollar or even hundreds of dollars or even thousands of dollars in some cases. It actually is worse. The reason it’s worse is that adds unnecessary noise to the interface, you’re providing all these inks that someone has to mentally process, and it’s actually not really meaningful ink because the change is what’s important. Think about precision when you’re printing values. Julien: This concept of noise is so essential today for any data analytics tools. There is so much data today. There is data for everything. I think it’s our responsibility as a data analytics company to make sure what are we actually trying to help our user with this data set is not just about adding new metrics. Adding new metrics usually is just going to add noise and not be helpful in comparison to fairing what do they need to make the right decision. Brian: Right. Complexity obviously goes up. The single verb, ‘add,’ as soon as you do that, you’re generally adding complexity. One of the design tools that is not used a lot, and this is something I try to help clients with is, what can we take away? If we're not going to cut it out entirely, can we move this feature, maybe this comparison to a different level of detail? Maybe it’s hidden behind a button click, or it’s not the default. But removing some stuff is a way to obviously simplify as well, especially if you do need to add new things. Your only weapon is not the pencil, you’ve got the eraser as well in the battle so to speak. Julien: I couldn’t agree more. On Next Big Sound we have this concept of artist stages. It’s a way for us to put artist into buckets and by looking at their social instrument data. It goes from undiscovered to epic. We do that by looking at all of the data we have and looking at it in context. I don’t have the numbers right now because they update on a daily basis but every artist starts undiscovered. For instance, as they get 1000 Facebook likes, maybe they’re going to get to a promising stage. We have all of these thresholds moving everyday looking at trends among social services. But what is interesting is that for instance, for a booker, a booker doesn’t need to look at the exact number of Twitter followers for an artist. He needs to know that he’s booking for a midsized venue in the city he’s in and he’s probably going to be looking for promising to established artists and not looking for the mainstream to epic artists. It’s always about figuring a way to use the numbers to tell the story. Brian: I’m totally selfishly asking for myself here, but I was immediately curious. I live in Cambridge which is in the Boston area, and I am curious who are the big artists in our area and what is the concentration? I’m in a niche. I’m more in the performing arts market, in the jazz, in world music, and classical music but I’m just curious. Is there a way to look at it by the city and know what your artist community looks like? You guys do anything like that? Julien: We don’t currently. But I think YouTube has actually a C-level chart available. It’s not part of something we do because I think the users it would benefit are not the users we specifically try to work on new features. It’s more something for bookers than artists ,specifically ,but it’s exactly the type of thing that we need to think about when we prioritize new features. Brian: I’m curious just because the topic’s fairly hot. Everybody is trying to do machine learning projects these days. I don’t like the term AI because it tends to be a little bit overloaded but are you guys using machine learning to accomplish any particular problems or add any new value to your service right now? Is that on your horizon? Julien: How do you think about machine learning? Brian: A lot of times I associate it with predictive analytics or understanding where you might be running instead of just using statistics. I don’t know what kind of data you might have for your learning that you can feed in but maybe there’s aspects about artists that can predict. Especially, I would think like in the pop music world where there tends to be more commercialization of the music, I would say, where it’s like we need a two-minute dance track at this tempo specifically because DJs are going to play it. It’s a very commercial thing. It’s very different than what I’m used to. So I’m curious if there’s a way to predict out how an artist may do or what kinds of tracks are performing well. Like these tempo songs, we predict over the next six months that tech house music at 160 beats for a minute is going to do really well based on the trending. I don’t know. I’m throwing stuff out there. The goal, obviously, is not to try to use like, “Oh Home Depot has this new hammer, let’s run out and get it. We don’t even know what it’s for but everyone else is buying it.” That’s how I joke about machine learning. It’s like you need to have a problem that necessitates that particular tool. I don’t ask such that, “Oh there should be some.” I’m more curious as to whether or not it’s a tool that you guys are leveraging at this time. Julien: The Next Big Sound team doesn’t worked on features following the musical aspects of things. We really are focused on the user data. Brian: Engagement and social. Julien: Engagement data mostly, yes. But at the same time, I’m sure teams have worked on this because of the way that genome works. We have a lot of data about the way songs are made. Regarding machine learning, on the Next Big Song team, we actually have something that is called the prediction chart. You said predictions. We have this chart that is available every week. Basically, it really goes back to having data for a long time. The fact that we’ve had data since 2009, we’ve been able to see artists actually get from starting to charting on the Billboard 200. By having all of these data, we’ve been able to see some trends, some things that usually happen for artists at specific times in their career up until they get into the Billboard 200. We actually do have some algorithms that allow us to apply this learning to all of the artists on Next Big Sound right now and have a list every week of artist that we believe are most likely to appear on the Billboard 200 chart next year. Brian: I see. Got it. Do you track your accuracy rate on that internally and change it over time? Do you adjust the model? Julien: Yeah, we do. Brian: Cool That’s really neat. Tell me, this chat has been super fun. I’ve selfishly got a little indulgent because being a musician, it’s fun to talk about these two worlds that I’m really passionate about so I could go on forever with you about this. But I’m curious. Do you have any advice for other product managers or analytics practitioners about how to design good data products and services? How to make either your own organization happy or your customers happy? Do you have any advice to them? Julien: Yeah, of course. I guess it’s all about asking questions, honestly. What is very good with working at Next Big Sound is that it all started in 2009. Maybe actually I can go back and tell you the story about how it started and why it’s so different today. It started in 2009. It was actually a project, a university project by the three co-founders. Basically, they were wondering about one thing. How many plays does a major artist get on the biggest music platform in the world? At that time, it was MySpace. The artist they picked was Akon. Basically, they just built a crawler, went to bed, woke up, and discovered that an artist like Akon was getting 500,000 plays on MySpace in one night in 2009. The challenge in 2009 was to get the data. That’s why for the most part in Next Big Sound as it started was, I really think a data aggregation tool. Our goal was to get as many sources as possible and just make them easily accessible into the same place. We really are much into the information layer here. We’re giving you all the numbers and you can compare Tumblr to Vimeo, to YouTube, to Twitter, to Facebook, to Vine, to you name it into a table or a graph that you want to. The reality is, today things change. We don't need to fight to get data anymore. We don’t need to hike our way into getting the numbers. Now, data is accessible to everyone in a very easy way. It’s kind of a contract. You, by being an artist, you know you’re going to get access to your Spotify, YouTube, Pandora, Apple Music or any other platform data very easily just by signing up and authenticating as an artist. That’s where our goal changes. Thankfully, we don’t need to convince people to care about data, we know they do already. But now the challenge is different. Now, the challenge is to make them understand what does their data mean and how can they turn it into getting even more data, getting into having even more engagement, and having even more plays. I think that’s something that is very interesting because it really resonates into the question we’ve been asked in the past few years like, “What does my data mean and when should I be looking at my data?” If anything, these two things correlated pretty well. People don’t just want to look at numbers anymore, they want to be able to use numbers to make decisions. That’s the core of what we’re trying to achieve today. We couldn’t be there if we didn’t have users that ask us the right questions. Brian: Cool that’s really insightful. Just to maybe tie it off at the end and maybe you can’t share this but what’s your home run? What is your holy grail look like? Is there a place you guys know you want to get? Maybe it’s the lack of data or you don’t have access to the data in order to provide that service. Do you guys have kind of a picture of where it is you want to take the service? Julien: What is very noble about our goal at Next Big Sound specifically is we’re here to help artists. The North Star would be to make sure that any artist at any time in their career is doing everything they can do to play more shows, to reach to more people, and to make sure their music is heard. Brian: Nice. I guess it’s like you’re already there, just maybe the level of quality and improving that experience over time, that’s your goal. It’s not so much that there’s so much unobtainable thing at this moment. Is that kind of how you see it? Julien: I think the more we don’t feel just a data analytics tool, the more we’re getting to that goal. I really hope we get to a point where people don’t need to be data analysts to look at data. We’re always going to provide a very customizable tool for the data-savvy because they know what they need more than we can ever do it for them. We want to make sure that for everyone else, we can just make it very easy and as simple as a click for them to do something that’s going to impact them positively. Brian: Cool, man. This has been really exciting to have you on the show. Julien, can you tell the listeners where can they find you on the interwebs? Are you on Twitter or LinkedIn? How do they find you? Julien: For sure. @julienbenatar on Twitter, nextbigsound.com is free for everyone. Actually, we made our data public recently, so if you ever want to learn more about what we do, please check it out. We try to post on our blog about what we learn through data science, through design, and share more about why we build what we build. I recommend to just check blog and do some commitment to learn more about what we do. Brian: I definitely recommend people check out the site. The fun thing is again, as you said, it’s public. If there’s a band you like or whatever, you can type in any group that you like to listen to and you can get access to those insights. Just kind of get a flavor of what the service does. I’ll put those links in the show notes as well as the data pyramid. Julien, cool. Thanks for coming on. Is there anything else do you like to add before we wrap it up? Julien: No, thank you so much. I love reading your newsletters and I’m very happy to be here. Brian: Cool. Thank you so much. Let’s do it again. Julien: Cool. Brian: Cool. Thank you. We hope you enjoyed this episode of Experiencing Data with Brian O’Neill. If you did enjoy it, please consider sharing it with #experiencingdata. To get future podcast updates or to subscribe to Brian’s mailing list where he shares his insights on designing valuable enterprise data products and applications, visit designingforanalytics.com/podcast. Never forget to look up the online HTML CheatSheet when you forget how to write an image, a table or an iframe or any other tag in HTML!

006 – Julien Benatar (PM for Pandora’s data service, Next Big Sound) on analytics for musicians, record labels and performing artists

Play Episode Listen Later Feb 12, 2019 45:01

We’re back with a special music-related analytics episode! Following Next Big Sound’s acquisition by Pandora, Julien Benatar moved from engineering into product management and is now responsible for the company’s analytics applications in the Creator Tools division. He and his team of engineers, data scientists and designers provide insights on how artists are performing on Pandora and how they can effectively grow their audience. This was a particularly fun interview for me since I have music playing on Pandora and occasionally use Next Big Sound’s analytics myself. Julien and I discussed: How Julien’s team accounts for designing for a huge range of customers (artists) that have wildly different popularity, song plays, and followers How the service generates benchmark values in order to make analytics more useful to artists How email notifications can be useful or counter-productive in analytics services How Julien thinks about the Data Pyramid when building out their platform Having a “North Star” and driving analytics toward customer action The types of predictive analytics Next Big Sound is doing Resources and Links: Julien Benatar on Twitter Next Big Sound website Next Big Sound blog The Data Pyramid model Quotes from Julien Benatar “I really hope we get to a point where people don’t need to be data analysts to look at data.” “People don’t just want to look at numbers anymore, they want to be able to use numbers to make decisions.” “One of our goals was to basically check every artist in the world and give them access to these tools and by checking millions of artists, it allows us to do some very good and very specific benchmarks” “The way it works is you can thumb up or thumb down songs. If you thumb up a song, you’re giving us a signal that this is something that you like and something you want to listen to more. That’s data that we give back to artists.” “I think the great thing today is that, compared to when Next Big Sound started in 2009, we don’t need to make a point for people to care about data. Everyone cares about data today.” Episode Transcript Brian: I’m really excited today for this episode. We have Julien Benatar on the show and he’s from a company that I’m sure a lot of people here know. You probably have had headphones on at your desk, at home, or wherever you are listening to Pandora for music. Julien , correct me if I’m wrong, you were the product manager for artist tools and insights at Next Big Sound, which is a type of data product that provides information on music listening stats to, I assume, artists’ labels as well to help them understand where their fans are and social media engagement. I love this topic. I’m also a musician, I have a profile on Next Big Sound and I feel music’s a fun way to talk about analytics and design as well because everybody can relate to the content and the domain. Welcome to the show. Did I get all that correct? Julien: Yeah, it was perfect. Brian: Cool. Tell us a little about your background. You’re from France originally? Julien: Yes, exactly. I grew up next to Paris, in Versailles more specifically, and moved to New York in 2014 to join Next Big Sound. Brian: Cool, nice. You’ve been there for about four years, something like that. You have a software engineering background and then now you’re on the product side, is that right? Julien: Exactly yes. I joined the company back when we were a startup. Software engineering was perfect, there was so much to do. To our move to Pandora, I moved to a product manager role around a year ago. Brian: Next Big Sound was independent and then they were acquired by Pandora. I assume there is good stuff about your data. Why did Pandora acquire you and how did they see you guys improving their service? Julien: We got acquired in 2015. The thing is, Next Big Sound was already really involved in the music industry. We already had clients like the three major labels and a lot of artists were using us to get access to their social data. I think it was a very natural move for Pandora as they wanted to get closer to creators and provide better analytics tools. Brian: For people that aren’t on the service, I always like to know who are the actual end users, the people logging in, not necessarily the management, but who sits down and what are some of the things that they would do? Who would log in to Next Big Sound and why? Julien: Honestly, it’s really anyone having any involvement into the music industry, so that can be an artist, obviously, try looking to try their socials and their audience on Pandora. But you can also be a booker trying to book artists in their town. We have a product that can really be used by many different user personas. But our core right now is really artists and labels, having contents on Pandora and trying to tell them the most compelling story about what they’re doing on the platform. Brian: When you think about designs, it’s hard to design and we talk about this on the mailing list sometimes but it’s really hard to design one great thing that’s perfect for everybody so usually you have to make some choices. Do you guys favor the artist, or the label, or as you call them,the bookers or whom I know as presenters,in the performing arts industry? Do you have a sweet spot, like you favor one of those in terms of experience? Julien: I think it’s something we’re moving towards, but it hasn’t always been this way. Like I told you, we used to be a startup or grow us to make a product that could work for as many people as possible. What is funny is we used to have an entity on Next Big Sound called Next Big Book where we used to provide the same type of service for the book industry. If anything, it’s been great to join Pandora because then we could really refocus on creators and it really allowed us to, I believe, create much better and more targeted analytics tools to really fulfill needs for specific people like artists and labels. Brian: I would assume individual artists are your biggest audience or is it really heavily used by the labels or who tends to… Julien: I think it’s pretty much the same honestly. I think the great thing today is that, compared to when Next Big Sound started in 2009, we don’t need to make a point for people to care about data. Everyone cares about data today. I think that everyone has reasons to look at their dashboards and especially for a platform like Pandora with millions of users every month. Our goal is really just telling them a story about what does it mean to be spinning on the platform and the opportunities it opens. Brian: You talked about opportunities, do you have any stories about a particular artist or a label that may have learned something from your data and maybe they wrote to you or you found out like in an interview how they reacted like, “Hey, we changed our tool routing,” or, “Hey, we decided to focus on this area instead of that area.” Do you know anything about how it’s been put into use in the wild? Julien: Yeah, it’s used for so many different reasons. For the people who don’t use Pandora, something I really like about the platform is it’s really about quality. As you use Pandora, you have the opportunity to thumb up or thumb down songs and as you do, you’re going to get recommended more songs like the ones you like. It’s really about making sure that you get the best songs at all times. The reality then is that for artists, their top songs on Pandora can be pretty different than their top songs on other platforms because sometimes their friends are going to be just reacting more to some part of their catalog than another one. I’ve heard many times of artists changing their playlists in looking at which songs where their fans thumbing up the most on Pandora. Brian: Could you go through that again? How would they adjust their playlist? Julien: Usually, people use Pandora as a radio service. While we already have internet today, most people are listening to the radio because they’re usually are very targeted and it just works really well. The way it works is you can thumb up or thumb down songs. If you thumb up a song, you’re giving us a signal that this is something that you like and something you want to listen to more. That’s data that we give back to artists. We tell them, “This are your most thumbed songs on Pandora. These are the songs that people engage with the most on the platform.” Looking at this data, you can actually inform them songs that they believe they should be playing more on the store. Brian: I see. A lot of it has to do with the favoriting aspect to give them idea what’s resonating with their audiences. Julien: Qualitative feedback, yes. Brian: Got it. Actually, it’s funny you mentioned the qualitative feedback. In preparation for this, I was reading an article that you guys put out back in March about a new feature called weekly performance insights, which is really cool and this actually reminds me of something that I talked about in the Designing for Analytics mailing list, which is the act of providing qualitative guides with your analytics. A lot of times they analyze for turnout quantitative data and whenever there’s an opportunity to put stuff into context or provide qualifiers, I think that’s a really good thing and you guys look like you’ve have done some really nice things here. I’ll paraphrase it and then you can jump in and maybe give us some backstory on it. One of the things that I think is really cool is there’re concepts of normalcy in here so that, if I’m an artist and I look at my numbers, I have an idea. For your Twitter mentions, for example, you say, “For artists with 26,000 followers, we expect you to get around 44 mentions.” When you show me that I have 146 mentions, I can tell that I’m substantially higher than what my social group would be. I think that’s a really fantastic concept that people not in music could try to apply as well which is, are there normalcy bans where you’d want to sit? Is there some other type of group, maybe, an industry, or apparent group, or another business unit, whatever it may be to provide some context for what these out of the blue numbers mean that don’t have any context? How did you guys come up with that and can you tell us a bit about the design process of going from maybe just showing, “You’re at 826 apples,” as compared to what? How did you move from just a number into this these kind of logical groupings where you provide the comparisons? Julien: I think what’s really fascinating is, we really live in an age of data. As an artist, you need to be on social media for the most part. There still a lot of artists I listen to but just decide not to. It’s part of things but at the same time, real big success in the music industry didn’t change. It’s still being on the Billboard chart, getting a Grammy and all these things. But as we see this, we have millions of artists looking at their data every day and just are not able to understand, like is it good or is it not good. Everyone starts at zero. We have a strong belief that data can only be useful when put in context. Looking at the number on its own can give you a sense of how things are doing but that can also be dismissive. An example is, a very common way to look at data is to look at a number and look at the percent changing comparison to the previous week. You’ve got a bunch of tables and you look at, am I growing or am I not growing. The reality is it’s actually impossible to always have a positive percent change. There’s no artist in the world that always does better week by week. Even Beyonce, I can assure you that the week she released Lemonade, she had more engagement on Twitter than the week after. With that in mind, we really try to give a way for artists to understand how are they doing for who they are and where they are currently in their career. Next Big Sound started in 2009. One of our goals was to basically check every artist in the world and give them access to these tools and by checking millions of artists, it allows us to do some very good and very specific benchmarks. For an artist, like the example you said, for instance an artist with a thousand Twitter mentions in a week, is it good or bad in comparison to their audience size? This feature comes because that’s just the question we’re asked. Artists want to know is it any good? What does this number actually mean for me? That’s why we really wanted to, in some ways, get out of being a content aggregator platform and really be a data analytics platform. How can we actually give information that can help artist make better decisions? Brian: I remember the first time I got what I would call an anomaly detection email from your service and it was about some spike in YouTube views or something like that. I thought it’s fantastic in two reasons. First of all, you identify an anomalous change and I think in this case it’s a positive anomalous change. That tells me that I should log in the tool. Secondly, you proactively delivered that to me. On the Designing for Analytics mailing list, we talk about is that user experience does not necessarily live inside your web browser interface or your hard client or whatever you’re using to show your analytics. Email and notifications are a big part of that. Can you tell me about how you guys also arrived at when you pushed these things out and maybe talk about this little anomaly detection service that you have? Julien: It all started when we got acquired by Pandora. We decided to just invite a bunch of users and just talk to them, understand how to use our product and what did they think about it. We had artists, managers, and label people come over and we just talk to them and basically they all said, “We love it.” But then, by looking at their actual usage, they don’t use it that much. I guess one of their questions was when should I be looking at my data? Everyone is very busy. As you’re an artist, you need to perform, you need to write music, you need to engage with your fans and same goes with everyone. When should I look at data? The reality is by being a data company, we do get all the data, we have all the numbers. We have ways to know when things are supposed to be known, when artists should be acting on something. We just turn this into this email notifications. Anytime we notice that an artist is doing better than expected, we just let them know right away. Brian: That’s great. Do you do it on the opposite end too? If there’s an unexpected drop or maybe like, “Oh, you put a new track out and your socials dropped,” or something like that, do you look at the negative side too or do you tend to only promote the positive changes? Julien: As far as pushes, we decided to only do push for positive. But as you mentioned weekly performance, weekly performance can give you some negative insights, like, “You’re not doing as well as artists with the same size of audience as yours.” The reason we didn’t do it for our notification is, anomalies are really hard to completely control. A reason, for instance, is Twitter removing bots. Basically, every single artist would have had an email telling them, “You lost Twitter followers this week.” It was a lot of work to really tune our anomaly factor to actually only send emails when something legitimate happens. That’s the reason we only decided so far to do it for positive but we actually have been thinking about doing the same for negative but that’s another type of work. Brian: Yeah, you’re right. You have to mature these things over time. You don’t want to be a noise generator. Julien: Exactly. Brian: Too many, then people start to ignore you. I’ve seen that with other data products I’ve worked on which just have really dumb alerting mechanisms that are very binary or they’re set at a hard threshold and just shootout noise and people just tune it out. Julien: I’m glad you mentioned this because this feature was in beta for a year for that specific reason. Brian: Got it. Julien: We had to learn the hard way. We had like a hundred beta users. We’ve got way too many emails because anytime there were an anomaly anywhere, they would just get an email. For the most part, it was things that were supposed to help them. If a notification becomes noise, then that’s absolutely against its purpose. Brian: I don’t know if everybody knows how the music business works, at least from the popular music side, but just to summarize. You have individual artists that are actually performers. They may or may not have an artist manager which takes care of their business affairs, represents them like negotiations with people that book shows. Then you have labels which are sort of like an artist manager except they’re really focused on the recording assets that the artist makes and they actually tend to own the recordings outright at the beginning and then over time, the artist may recoup through sales they make it the ownership act and the sound recordings they make. Of those kinds of three major groups, is there a one that’s particularly hungry or you’re the squeaky wheel that is most interested in what you’re doing? Julien: I really think that into these three groups, we have a subset of users that are really into the data and into the actionability of it. I don’t think it’s one specific group of user. It could be all around the industry like we have the data-savvy, they really want to know. We have some users that actually would rather get more notifications even if they need to on their end to figure what is right from what is wrong. But since we have such a wide user base of different type of people, we decided to go on the conservative side and make sure to only share things that we thoroughly validated through all of our filters. Brian: I assume that your group reports into some division of Pandora, I’m not sure of that. Are you reporting into a technology, like an IT, or a business unit, or marketing? Where do you guys fit in the Pandora world? Julien: We’re part of the creator’s tools. I don’t really have a perfect answer to this. Brian: Okay. I guess my main question being, because when we talk about designing services, we talk about both user experience, which is the end user thing and about business success or organization success. I’m curious, how does Pandora measure that Next Big Sound as delivering value? I can understand, I’m sure our artist can understand how the artists value it through understanding how is my music moving my audiences, et cetera. Is there a way that Pandora looks at it? Are they interested in just time spent? The analytics on the analytics, so to speak, is what I’m asking about. How do you guys look at it like, “Hey, this is really doing a good job,” or whatever? Do you know how that’s looked at? Julien: To be honest, I think you said it right. Our goal is to help artists make their decisions through data and having artists use the platform is currently the way Pandora sees us doing a good job. Actually, it hasn’t changed that much since our acquisition. One of our main KPI for the past and couple of years is something I would call insights consumes. Just making sure that our users, artists, anyone using Next Big Sound are consuming data. That can be them logging into the website or that can be them opening one of our notifications. But so far that was our main KPI. We’re trying to work on some more targeted KPI, potentially like actions taken, that would be the North Star, but we’re still working on how to do that right. Brian: Do you guys facilitate actions, so to speak, directly in the tool or are there things people can do with those actions really take place outside of the context of Next Big Sound? Julien: There are actions that artists can take to the other creator’s tools provided by Pandora. For instance, artists have the ability to send audio messages to anyone listening to them. If they go on tour into the US, they can have targeted messages in every single song they’re going to play. If anyone listens to them there, they can just click and buy a ticket. We’re working to make sure that artists are aware of these tools because they are free and they’re generally helping them grow at their careers. But regarding external actions, so far we don’t have any one-click way to tweet at the right time to the right people or with the right content or anything like this. Brian: Sure and that’s understood. Not every analytics product is going to have a direct actionable insight that comes right out of it. You guys may be feeling a longer term picture about trending and maybe for a certain artist to get an idea if they’re releasing music fairly frequently, what stuff is working and resonating, and what stuff is not. I can understand that. There may not be a button to click as a result immediately. Julien: That’s the goal though. Everything we do right now is going towards this objective. Maybe I can tell you a little about the way we think about data and that can give more sense to it. In order to work on any new feature, we follow this concept called the data pyramid. It’s something that you can Google. There’s a Wikipedia page for it. Let me explain to you how it works. The data pyramid, it’s a pyramid formed of four layers. It could be upon each other and each representing an exquisitely useful application of data. At the bottom of the pyramid we have the data layer. Any sort of data that we may have. For our case, Android data, Twitter, Facebook just getting the numbers, getting the raw data. On top of it, we have the information layer. The information layer is going to be ways you have to visualize this data. I guess it’s like the very broad sense of analytics. We’re going to give you tables, graphs, pie charts, you name it. We’re giving you ways to craft stories about this data but it’s on you to figure it out. Then on top of it we have what we call the knowledge layer. That’s where things start to get interesting. The knowledge layer is the contextual part of it. It’s like, “What do this number actually mean?” It has industry expertise. For instance, the way we’re going to work about it for musicians and their true data may be different than any other industry. The knowledge layer goes like a weekly performance. It’s a perfect answer to it. It’s what does it mean for me as a musician with a hundred fans to get two mentions this week. Same for notifications. It’s telling you that you should be looking at your data right now because something is happening. That’s how we get to the North Star and the last part of the data pyramid which is intelligence. The goal of intelligence is actionability. Now that I get to understand what does this number mean to the specific context, what should I be doing? Following your question, everything we’re trying to do here is to get to a point where we can just send an email to an artist and tell them, “Hey, you should be doing this right now because, with all the data that we have, we believe that this is going to have the highest impact for you.” Brian: It‘s really fascinating that you just outlined this data pyramid. I actually haven’t heard of this before. It made me think of one of the kind of, it’s not a joke but in the music community, I’m also a composer and when we write stuff, the kind of running joke is like nothing is new. Your ideas for this new song or this new melody I’m composing, it probably came before you. You heard it there before. I wrote a post on my list that was pretty much exactly the same thing except the knowledge layer. I was calling that insight. Data have been this raw format and information being the first human-readable format that’s like say going from raw data to a chart, a histogram. Now I have a line on a chart and then the insight layer being, I have a line on the chart and another line comparing it to like you said, average, or my social group, or a parent group, or some taxonomy, or an index. Then the action or the prescription for what to do or the prediction those that kind of lead you in about action which would be that fourth state. You’re like, “Oh, is this really a new concept?” It’s like, “Nope. Someone else already thought of that.” I totally want to go read about this data pyramid. Julien: That’s amazing. Brian: I’ll find that link to the data pyramid and I’ll put that in the show notes for sure. I thought that was really funny. Julien: It’s funny that you called it insight because that’s the way we call a lot of our features are working out. The way we define insight is bite-size, noteworthy, sharable content. How can we get into the noise of all of the data that only gives you exactly what you should be looking at. That’s how we got into notification and weekly performances. This is the one thing you should be looking at. Brian: I understand what you’re getting at there. The insights are, like you said, bite-size chunks of interesting stats that someone can put some kind of context around. That’s great and it’s good. One of the things I liked, too, that you talked about was you said, “Oh we got like a hundred users, like a beta group and that kind of inspired some of this.” Your product response to how do we help people know when to come and look at our service. I think this is really good because one of the problems that I see with clients and people on the list, I think is low engagement. This is especially true for internal analytics companies. Low engagement can be a symptom of a difficult product, it doesn’t provide the right information at the right time, it may not have a lot of utility, or it’s a resistance to change. People have done something the old way and they don’t want to do it the new way. One of the recipes you can follow if you’re trying to do a redesign or increase engagement is to involve the people that are going to use the service in the design process, both the stakeholders as well as the end customers. This is especially true again for the internal analytics people. Your customers or other employees and your colleagues. By engaging them in the design process, they’re much more likely to want to change whatever they’re doing now. I loved how you guys did some research. Now I want to ask, do you frequently do either usability testing or interviews? Is that an ongoing thing at your company or is it really just in front of a big feature release or something like that? How do you guys do this research? Can you tell me about that? Julien: Of course. It’s consent. We haven’t released any major feature without doing some heavy user testing. I’m very lucky to be working with two designers, Justin and Anabelle who are very user-focused. Honestly, if you come to our office, at least every week we’re going to have some user interview and just talking to them, showing them prototypes, and just see how do they play with it. Brian: So you’re doing a lot of testing it sounds like. That’s fantastic. Julien: At the same time it’s always to find the right balance because you could be overtesting things too. We really are focusing on user testing for new things and make sure that the future that we are working on actually answers their user story that we intended. Brian: I don’t know how involved you get participating in these, but do you have any interesting stories or anecdotes that you got from one of those that you could share? Julien: Let me think. I do participate into a lot of them but I’m not sure I have an example right now. Brian: Are most of the people you interview, are they current users of Next Big Sound or do you tend to focus on maybe artists that haven’t experienced the service yet or you mix it up? Julien: We mix it up. We mostly engage with users that we already have but then we can decide to go with users that haven’t used the platform for a while, or more active users if you want to understand how we’re useful into their day to day. What I would say is that, surprisingly, it’s very easy to get users to chat about their experience with the product. I didn’t assume that we would get so many responses when we tried to have people come over or just hop on the zoom to check a new feature. Brian: I’m glad you actually mentioned that because I think in some places, recruiting is perceived to be difficult and it probably isn’t. Maybe you haven’t done it before but as I tell a lot of my clients, a lot of people love to have someone listen to them talk, tell them all about their life and what’s wrong with it, and how it could be better with their tools. They love having someone listen to them and especially if they know that their feedback is going to influence a tool or a service that they’re using. They tend to be pretty engaged with it. I find it’s really rare that I do an interview with a client’s customer and they don’t want to be included in the future round like, “Hey, when we redesign the service, can we come back to you and show you what we’ve done?” “Oh, I love to do that!” Everybody wants to get engaged with it. There are places where recruiting can be difficult when it’s hard to access the users, some of the enterprise software space that can be an issue sometimes. But generally, if you can get access to them, they tend to be pretty willing to participate. I’m glad you mentioned that. Julien: I think the great part about testing with current users on the platform is to actually show them prototypes with real data, not just show them an abstract idea that we want to work on. As soon as they can see what we’re working on apply to their own career as musicians, for instance, that can lead to fascinating discussions. Brian: You made a really good point on the real data thing. I remember as far back as 10 years ago or whenever, I use to work at Fidelity Investments, we would see this issue when we’re working on the retail site for investors. When you show a portfolio that, for example, has Apple stock trading at $22 in it, you’re not really there to test what is the price of Apple stock but you might be testing something entirely different and the customer cannot bear what is going on? They’re so stuck on this thing. It’s all fake seed data in the prototype. The story here being if you’re a listener, when you test it’s important to have at least realistic data. You don’t want to have noise in the test or whatever your studying or else you can end up on this tangent. Try to make the numbers looks somewhat realistic if you’re using quantitative data. In some cases, people can be taught to roleplay. Pretend you’re Drake or pretend you’re some big artist and then they can get their head around why they have billions of streams instead of thousands which they’re used to. Julien: Absolutely. That also helps us just build better products because the reality is we have a lot of artists with maybe 10 plays in a month. As we build visualizations like something that we built a line of looking at Drake’s data, it’s not going to work as intended for a smaller artist sometimes. Having real data involved as soon as possible into the design process has been such a game changer for us. We really have a multidisciplinary team involved into the research and design of everything we do. I’m working with a data scientist, data engineer, a web engineer, and designer on a daily basis. Obviously, we all have our things to do. But as we get into creating something new, we just make sure to have someone helping us get the real data, interview the right user, and just create prototypes as soon as possible. Working with prototypes is essential into building useful data analytics tools. Brian: Yes, you do learn a lot more with a working prototype. It’s not to say you can’t test with lower fidelity goods, especially early on but for a service like yours when the range of possible use both the personas and also you’ve got the Drakes of the world, big major label artist and then down to really small independents, it’s really important to have an idea how your charts are going to scale, and what’s going to happen with data. Even just small stuff like how many decimal points should you be showing on a mobile device, some of the numbers might cram up. Julien: Exactly. Brian: All this stuff that you never think, if you only look at one version of everything, you can end up with a mess. I’m glad that you brought that up. Julien: I couldn’t say better. The decimal is actually something that we’ve had to discover through real data. Brian: To all of you in the technical people out there, I will say this. If I’ve seen one trend with engineers, is they love precision and there’s a lot of times when there’s very unnecessary precision being added to numbers. Such as charts and histograms. Histograms are usually about the trend, they’re not about identifying what was the precise value on this date at this time. It’s about the change over time. Showing what’s my portfolio worth down to three digits of micro-cents or something like that is just unnecessary detail. You can probably just round up to the dollar or even hundreds of dollars or even thousands of dollars in some cases. It actually is worse. The reason it’s worse is that adds unnecessary noise to the interface, you’re providing all these inks that someone has to mentally process, and it’s actually not really meaningful ink because the change is what’s important. Think about precision when you’re printing values. Julien: This concept of noise is so essential today for any data analytics tools. There is so much data today. There is data for everything. I think it’s our responsibility as a data analytics company to make sure what are we actually trying to help our user with this data set is not just about adding new metrics. Adding new metrics usually is just going to add noise and not be helpful in comparison to fairing what do they need to make the right decision. Brian: Right. Complexity obviously goes up. The single verb, ‘add,’ as soon as you do that, you’re generally adding complexity. One of the design tools that is not used a lot, and this is something I try to help clients with is, what can we take away? If we’re not going to cut it out entirely, can we move this feature, maybe this comparison to a different level of detail? Maybe it’s hidden behind a button click, or it’s not the default. But removing some stuff is a way to obviously simplify as well, especially if you do need to add new things. Your only weapon is not the pencil, you’ve got the eraser as well in the battle so to speak. Julien: I couldn’t agree more. On Next Big Sound we have this concept of artist stages. It’s a way for us to put artist into buckets and by looking at their social instrument data. It goes from undiscovered to epic. We do that by looking at all of the data we have and looking at it in context. I don’t have the numbers right now because they update on a daily basis but every artist starts undiscovered. For instance, as they get 1000 Facebook likes, maybe they’re going to get to a promising stage. We have all of these thresholds moving everyday looking at trends among social services. But what is interesting is that for instance, for a booker, a booker doesn’t need to look at the exact number of Twitter followers for an artist. He needs to know that he’s booking for a midsized venue in the city he’s in and he’s probably going to be looking for promising to established artists and not looking for the mainstream to epic artists. It’s always about figuring a way to use the numbers to tell the story. Brian: I’m totally selfishly asking for myself here, but I was immediately curious. I live in Cambridge which is in the Boston area, and I am curious who are the big artists in our area and what is the concentration? I’m in a niche. I’m more in the performing arts market, in the jazz, in world music, and classical music but I’m just curious. Is there a way to look at it by the city and know what your artist community looks like? You guys do anything like that? Julien: We don’t currently. But I think YouTube has actually a C-level chart available. It’s not part of something we do because I think the users it would benefit are not the users we specifically try to work on new features. It’s more something for bookers than artists ,specifically ,but it’s exactly the type of thing that we need to think about when we prioritize new features. Brian: I’m curious just because the topic’s fairly hot. Everybody is trying to do machine learning projects these days. I don’t like the term AI because it tends to be a little bit overloaded but are you guys using machine learning to accomplish any particular problems or add any new value to your service right now? Is that on your horizon? Julien: How do you think about machine learning? Brian: A lot of times I associate it with predictive analytics or understanding where you might be running instead of just using statistics. I don’t know what kind of data you might have for your learning that you can feed in but maybe there’s aspects about artists that can predict. Especially, I would think like in the pop music world where there tends to be more commercialization of the music, I would say, where it’s like we need a two-minute dance track at this tempo specifically because DJs are going to play it. It’s a very commercial thing. It’s very different than what I’m used to. So I’m curious if there’s a way to predict out how an artist may do or what kinds of tracks are performing well. Like these tempo songs, we predict over the next six months that tech house music at 160 beats for a minute is going to do really well based on the trending. I don’t know. I’m throwing stuff out there. The goal, obviously, is not to try to use like, “Oh Home Depot has this new hammer, let’s run out and get it. We don’t even know what it’s for but everyone else is buying it.” That’s how I joke about machine learning. It’s like you need to have a problem that necessitates that particular tool. I don’t ask such that, “Oh there should be some.” I’m more curious as to whether or not it’s a tool that you guys are leveraging at this time. Julien: The Next Big Sound team doesn’t worked on features following the musical aspects of things. We really are focused on the user data. Brian: Engagement and social. Julien: Engagement data mostly, yes. But at the same time, I’m sure teams have worked on this because of the way that genome works. We have a lot of data about the way songs are made. Regarding machine learning, on the Next Big Song team, we actually have something that is called the prediction chart. You said predictions. We have this chart that is available every week. Basically, it really goes back to having data for a long time. The fact that we’ve had data since 2009, we’ve been able to see artists actually get from starting to charting on the Billboard 200. By having all of these data, we’ve been able to see some trends, some things that usually happen for artists at specific times in their career up until they get into the Billboard 200. We actually do have some algorithms that allow us to apply this learning to all of the artists on Next Big Sound right now and have a list every week of artist that we believe are most likely to appear on the Billboard 200 chart next year. Brian: I see. Got it. Do you track your accuracy rate on that internally and change it over time? Do you adjust the model? Julien: Yeah, we do. Brian: Cool That’s really neat. Tell me, this chat has been super fun. I’ve selfishly got a little indulgent because being a musician, it’s fun to talk about these two worlds that I’m really passionate about so I could go on forever with you about this. But I’m curious. Do you have any advice for other product managers or analytics practitioners about how to design good data products and services? How to make either your own organization happy or your customers happy? Do you have any advice to them? Julien: Yeah, of course. I guess it’s all about asking questions, honestly. What is very good with working at Next Big Sound is that it all started in 2009. Maybe actually I can go back and tell you the story about how it started and why it’s so different today. It started in 2009. It was actually a project, a university project by the three co-founders. Basically, they were wondering about one thing. How many plays does a major artist get on the biggest music platform in the world? At that time, it was MySpace. The artist they picked was Akon. Basically, they just built a crawler, went to bed, woke up, and discovered that an artist like Akon was getting 500,000 plays on MySpace in one night in 2009. The challenge in 2009 was to get the data. That’s why for the most part in Next Big Sound as it started was, I really think a data aggregation tool. Our goal was to get as many sources as possible and just make them easily accessible into the same place. We really are much into the information layer here. We’re giving you all the numbers and you can compare Tumblr to Vimeo, to YouTube, to Twitter, to Facebook, to Vine, to you name it into a table or a graph that you want to. The reality is, today things change. We don’t need to fight to get data anymore. We don’t need to hike our way into getting the numbers. Now, data is accessible to everyone in a very easy way. It’s kind of a contract. You, by being an artist, you know you’re going to get access to your Spotify, YouTube, Pandora, Apple Music or any other platform data very easily just by signing up and authenticating as an artist. That’s where our goal changes. Thankfully, we don’t need to convince people to care about data, we know they do already. But now the challenge is different. Now, the challenge is to make them understand what does their data mean and how can they turn it into getting even more data, getting into having even more engagement, and having even more plays. I think that’s something that is very interesting because it really resonates into the question we’ve been asked in the past few years like, “What does my data mean and when should I be looking at my data?” If anything, these two things correlated pretty well. People don’t just want to look at numbers anymore, they want to be able to use numbers to make decisions. That’s the core of what we’re trying to achieve today. We couldn’t be there if we didn’t have users that ask us the right questions. Brian: Cool that’s really insightful. Just to maybe tie it off at the end and maybe you can’t share this but what’s your home run? What is your holy grail look like? Is there a place you guys know you want to get? Maybe it’s the lack of data or you don’t have access to the data in order to provide that service. Do you guys have kind of a picture of where it is you want to take the service? Julien: What is very noble about our goal at Next Big Sound specifically is we’re here to help artists. The North Star would be to make sure that any artist at any time in their career is doing everything they can do to play more shows, to reach to more people, and to make sure their music is heard. Brian: Nice. I guess it’s like you’re already there, just maybe the level of quality and improving that experience over time, that’s your goal. It’s not so much that there’s so much unobtainable thing at this moment. Is that kind of how you see it? Julien: I think the more we don’t feel just a data analytics tool, the more we’re getting to that goal. I really hope we get to a point where people don’t need to be data analysts to look at data. We’re always going to provide a very customizable tool for the data-savvy because they know what they need more than we can ever do it for them. We want to make sure that for everyone else, we can just make it very easy and as simple as a click for them to do something that’s going to impact them positively. Brian: Cool, man. This has been really exciting to have you on the show. Julien, can you tell the listeners where can they find you on the interwebs? Are you on Twitter or LinkedIn? How do they find you? Julien: For sure. @julienbenatar on Twitter, nextbigsound.com is free for everyone. Actually, we made our data public recently, so if you ever want to learn more about what we do, please check it out. We try to post on our blog about what we learn through data science, through design, and share more about why we build what we build. I recommend to just check blog and do some commitment to learn more about what we do. Brian: I definitely recommend people check out the site. The fun thing is again, as you said, it’s public. If there’s a band you like or whatever, you can type in any group that you like to listen to and you can get access to those insights. Just kind of get a flavor of what the service does. I’ll put those links in the show notes as well as the data pyramid. Julien, cool. Thanks for coming on. Is there anything else do you like to add before we wrap it up? Julien: No, thank you so much. I love reading your newsletters and I’m very happy to be here. Brian: Cool. Thank you so much. Let’s do it again. Julien: Cool. Brian: Cool. Thank you. We hope you enjoyed this episode of Experiencing Data with Brian O’Neill. If you did enjoy it, please consider sharing it with #experiencingdata. To get future podcast updates or to subscribe to Brian’s mailing list where he shares his insights on designing valuable enterprise data products and applications, visit designingforanalytics.com/podcast.

005 - Jason Krantz (Dir. of Biz Analytics/Insights, Weil-McClain) on centering analytics around internal customers

Play Episode Listen Later Jan 29, 2019

Jason Krantz is the Director of Business Analytics & Insights for the 135-year old company, Weil McLain and Marley Engineered Products. While the company is responsible for helping keeping homes and businesses warm, Jason is responsible for the creation and growth of analytical capabilities at Weil McLain, and was recognized in 2017 as a “Top 40 Under 40” in the HVAC industry. I'm not surprised given his posts on LinkedIn; Jason seems very focused on satisfying his internal customers and ensuring that there is practical business value anchoring their analytics initiatives. We talked about: How Jason’s team keeps their data accessible and relevant to the issue they need to solve for their customer. How Jason strives to keep the information simple and clean for the customer. How does Jason help drive analytics in a company culture with a lot of legacy (from its people to its parts) The importance of focusing on context How Jason drives his team to be business partners, and not report generators Resources and Links: Jason Krantz on LinkedIn Quotes from Jason Krantz: "You realize that small quick wins are very effective because, at its core, it’s really important to get executive buy-in." "I’m a huge fan of simplicity. As analytics pros, we could very easily make very complex, very intricate models, and just, 'Oh, look at how smart we are.' It doesn’t help our customers. …we only use about two or three different visual types and we use mostly the exact same visual set-up. I can train a sales rep for probably five minutes on all of our reporting because if you understand one, you’re going to understand everything. That gets to the theme again of just simplicity. Don’t over complicate, keep it simple, keep it clean.” "…To get buy-in, you really got to have your business case, even to your internal customers, really dialed in. If you just bring them a bunch of crap, that’s how you’re going to lose credibility. They’re going to be like, “I don’t have the time to waste with you,” even though we’re trying to be helpful.” "What my team and I do is we really help companies weaponize their data assets." Episode Transcript Brian: Jason, are you there? Jason: I’m here my friend. Brian: Sweet. How’s it going? Jason: It’s going very well today. How’s your Friday going? Brian: I’m doing awesome. We’re going to talk a little bit about analytics. Is it Wile McLain or Weil McLain? Jason: I say Weil McLain. If I’ve been saying it wrong, I’ve been saying it wrong for a while. Brian: As I recall from my musical training, I think in German, the second syllable is the one that says its name. I guess it would be Wile McLain, like if it was W-I-E-L it would be ‘Weil.’ But I don’t know. Its anglicized as they come over the pond. Jason: I’m going to go with you on that when you sound like an expert. Brian: Nice. Well, you sound like an expert in analytics at Weil McLain. Tell us about what you’re doing over there. We met on LinkedIn, I’ve been enjoying your postings on the social feed about your approach. You seem really passionate about what you’re doing and I’m like, “I don’t know who this guy is, but that was really interesting.” I just have. Tell us about the company, what they do. I know they’re in heating, right? Jason: Yes, absolutely. The company I work for, and I work in the HVAC space, we’re a 135-year-old boiler manufacturer. Whether you realize it or not, you probably have one of our products in your house or building or very close to where you live. What my team and I do is we really help companies weaponize their data assets. As you know, a lot of companies are very skilled at acquiring data since the Big Data Movement. But the reality is that a lot of these companies don’t know what to do with all this data. That’s where we really come in. What I always tell my team and our business partners that we work with internally and externally is that our focus is on solving business problems. In order to do that, you have to identify what is the business problem that you’re trying to solve or strategic agenda that you’re trying to address. In order to do that, you really have to be anchored in the biz. Again, that’s just my perspective, but if you’re in the business day in, day out, you develop this very keen stand of what the business would need to accomplish its objectives. Just like right now, we are based in the marketing group and it’s a great spot to be. I’m a firm believer that every analytics team should be based in the business for a reason that I just talked about. But what that does being business-first is that gives us a great lens to look at data from. Sometimes analytics people would be IT-centric and they can do a lot of academic work against the data set or different data sets. But the business might look at the output and be like, “Yeah, that doesn’t help us.” We always, always, always start with, “What is the business problem we’re trying to solve or strategy we’re looking to address?” It also helps us when it comes to curating data also. That’s one of our primary response [00:03:21] this too, is to look for different data sets both internal and external that can help us identify strategic opportunities. It sounds really unsexy, I’m not going to lie. I think some of my LinkedIn post just say that data is boring. It really is. It’s mind-numbing, too, about 85% of my customers. But that’s the important part is understanding what do our customers need and that’s really the lens that we look at this through. We are a service provider, our customers are internal and external, we have customers just like any other business. We have to take this really boring, but really potent product in data and make it accessible to them. That’s really where we use design to really try to make that magic happen. Brian: I love that you said, “Trying to understand what the problem is.” This is something we talk about on the mailing list quite a bit. In fact, falling in love with the problem is a good basis for doing good work instead of kind of jumping to solutions or feeling […]. As I tell my clients sometimes like, “Our job is not to go and visualize the data. It’s not […] available for someone to put into another tool or whatever the heck it is. The job is to find an insight that already is used. Probably they’re already in your job and you’re there to make […] if you’re doing internal analytics. Help them do a better job at what they’re doing, offer more value. You need to figure out how to work that into their life.” For example, for you guys then, your customer, I assume is it primarily sales people that you’re working with? Who are your customers and your […]? Jason: Yes. Great question. One of our biggest customers is sales. Sales has been one of my biggest customers for the past 10 years of my career. I’m very intimately involved with the sales team, sales operation, sales optimization, insight gathering, pricing, things like that, but also marketing. We do a lot in terms of competitive intelligence gathering, market research. We also do a lot of operations in finance obviously related to the prices, that sort of thing. We really touch all areas of the business, but without question, our biggest customers are going to be sales and marketing. Brian: If you were to bring a new initiative like, “Hey, we have access to…” I don’t know what it might be but for you maybe your point, [might be a line 00:05:49] of data that could actually give them more leverage. We know what the negotiation brings, better than […], we know we kind of have an idea now from what the industry is doing for their sales such that we can now tell the CRM like, “This is your […] or something.” When do you get that sales person involved? Do you deliver a solution and get feedback? Do you bring [...] early and say, “Hey, we think we can tell you more about how to do better pricing on the spot with this thing.” Do you bring them in or when do they fit into your process? Jason: Great question. A lot of times because we spend so much time actually in the trenches, that’s one of things I think is unique about the way that I design my teams to do analytics. It’s not like hand off product and we’re like, “Godspeed. Good luck.” Once we deliver a solution, we’re actually in the trenches with the business trying to implement what we’re talking about because it just works better. The team work is just more effective and they know that they’ve got back up, they know they’ve got air support. Really, a lot of times when we come up with something new, a lot of times we will frame it from the lens like, “Hey, we know that we’ve got opportunity A or issue B, or whatever it is. This has been an issue or an opportunity for months or years or whatever.” We think that we’ve identified something that could help us in solving that issue or realizing the potential of that opportunity and then it becomes, “Okay, let’s sit down and talk about, do you agree that this might actually help us in this process?” Because the one thing that I’ve learned is, in order to get buy-in, you really, really got to have your business case, even to your internal customers, really dialed in. If you just bring them a bunch of crap, that’s how you’re going to lose credibility. They’re going to be like, “I don’t have the time to waste with you,” even though we’re trying to be helpful. What we found out is if you really dial in what are we trying to address with this, just as you would with any business case, and you bring that to them, I have found that they tend to be much more receptive. It’s not to say there’s not going to be resistance—resistance comes with any change—but we found that typically framing it from that lens and saying, “We’re trying to solve a problem that you have, we think that this data will help,” that’s a great starting point. Brian: Do you have an example of a before/after with that? I don’t want you to get into proprietary stuff you can’t talk about but is there like a, “Before they did it this way,” and then we brought them in and said, “Hey, we think we can get […].” and how you went [00:08:19]. Jason: Yeah. What I can talk about is just the manner in which we distribute sales information, specifically insights. I think that, for your listeners, this is going to ring true to a lot of sales forces. I know for all them that I’ve been in or worked with, this case was true 100% of the time. But one of the things that, again, keeping the customer-centric focus, that if you look at your sales reps, a lot of time is you’re going to be what I call casual data consumers. By that, I mean that these are guys and gals that aren’t really into data day in and day out like guys like you and I or some of the listeners maybe. What we have to do is, as I always encourage my team to take empathetic lens and look at, “Okay, if we give them what our first […] is going to look like, how are they going to interpret this?” A lot of times, to be honest, it’s not very good. Now that’s where we have to look at internally and kind of rationalize and say, “Okay, let’s find this. One of us will find [00:09:14].” But one example of that is traditionally, sales reps and sales teams will get the information in a flat Excel table. Just lots of rows and columns and just gibberish everywhere. That’s a very financial-centric view of sales data. But the reality is—I don’t know about the rest of mankind but I know for myself—I can’t remember much more than 10 numbers. The mental computational cost of extracting insights is just gargantuan. What happens is, I just don’t even bother to do it. I’m just like, “Yeah, whatever.” An equivalent of that is, you know if you get a big block of text in email? Even though if you took that same block of text and broke it up into two paragraphs or two sentence segments which is very easy to read when you put the effort in, but for me, if I get a big block of text, I’m not even going to read that. It’s kind of one of the same things that we see on the sales side. What we do is just say, “You know what? There’s a lot of really good information here and we need to make it digestible for our customers.” That’s where we found traditionally, visualization can be an incredibly effective tool to communicate insights to this casual data audience, to this casual data consumer. Brian: Do you have to work through the visuals with them? Do they tend to get it the first time? Is it a process of you share, “Here’s a report or here’s some new view on X.” How do you know if the visualization is actually allowing them to pull the insight out of what other [00:10:46] broad data? How do you know they’re actually “getting it”? Jason: That’s a great question. I’m a huge fan of simplicity. As analytics pros, we could very easily make very complex, very intricate models, and just, “Oh, look at how smart we are.” It doesn’t help our customers. It doesn’t help anything. Really what we do—this is going to get to the theme of simplicity—is we only use about two or three different visual types and we use mostly the exact same visual set-up. Just to kind of frame it, what I’m a big fan of is a simple bar chart. There’s more details attached to it but to the right of the bar chart, we’ll typically put a tabular data set. What we do is, as you think in US at least, we start in that left-hand side of the page or we […]. What we do is we look at the visual real estate. We say, “Our customers are going to start in the left-hand side. We want them to look at the bar chart because it allows them to very rapidly assimilate it at a high-level what’s going on.” It’s great at communicating at top-level churn very quickly but the trade-off is, is this horrible imprecision. You have no precision at all. What we like to do is then we address that issue by putting a simple table, very clean, very simple table over to the right. What that does is that then provides the precision that the customers are seeing in most financial-centric tables. What we found that does is that we have to train our sales team on one set-up and then that set-up is used virtually universally on all of our solutions. As an example, I can train a sales rep for probably five minutes on all of our reporting because if you understand one, you’re going to understand everything. That gets to the theme again of just simplicity. Don’t over complicate, keep it simple, keep it clean. Brian: I think those are good. A lot of times, when I work with engineering clients, they fall in love with consistency. I guess one point to maybe just the contrary of this is that, I think consistency is generally a good rule with design. We want to minimize unnecessary change but at the same time, I would recommend to listeners is to always look at context first, and context should always come in. Let’s say Jason comes up with report number 12 and they have 11 now or whatever, and it doesn’t feel right for number 11. That’s a place where a designer would probably push for, “Well, no. The 12th one actually needs to be different because it’s not […] 11th and even though it’s not consistent, in this context, we don’t need it to win. This version will deliver the usability and the utility that we’re looking for better than the other 11 will.” In general, I think it’s smart to not get creative unnecessarily with meaningless ink on the screen like, “Let’s try it this way. Let’s change the color palette. I’m tired of this.” Those are not good reasons for […], you’re just introducing noise and it’s unnecessary. But I like that you guys are thinking about simplicity and trying to reuse templates and not looking at it as a creative tableau. Ironically, people think it’s a creative “design” tool, but at the same time with all those weapons, you have a lot of different weapons you can use in that toolkit and part of that is knowing how to use this. It’s the same thing with Photoshop, a million buttons and all this stuff. The Photoshop doesn’t make you a designer. It’s being aware of your customer’s pain and the problems they need and knowing when to use all those filters and all those different things that it can do. I like that you guys are looking into that simplicity and reusing templates when it’s meaningful to do so. Jason: You bring up some great points and I 100% agree. My team that’s listening there, they’ll laugh because I beat it in their heads, “Context. Context, context, context.” Both in design as you’ve talked but especially with numbers in general. Like, “If I give you a number, a billion, that doesn’t mean anything, you got to have context.” I’d say the same is true for design just as you articulated. Great point. Brian: Where does the impetus for “everybody is a data company, everybody wants to do analytics”? But then there’s operationalizing that, there’s getting buy-in, leadership behind it. Where does that come from in your org? Where is the interest in taking a 130-year-old company and getting it to care about this? Where does that come from, your influence and all of that? Jason: That was driven by our current president because he saw it as part of a digital transformation. Obviously, this was an essential component of that. Obviously, we do a lot with analytics, but we’re also involved in a lot of other digital components that lead to that overall digital maturation. Analytics is a very, very big part of what we do but it’s not all that we do. We serve as kind of that quarterback for a lot of the digital initiatives to help basically, guide them through the process. Because even though some of the nuances of each of this project, each one will have its own nuances, they all come back to data. Data is the currency. We found out pretty quickly that if you want to stay relevant in this day and age, you need to be digitally evolved but more importantly, as you look at it, do you [compare the 00:16:02] advantage that you can derive from analytics? I would argue that gap is slowly closing known certain industries like manufacturing, but we probably have a little bit more runway [00:16:10] it. But for a lot of industries, analytics is becoming table stakes. It’s one of those where you can certainly expect incremental value and competitive advantage, but the question becomes how much longer. That was kind of the impetus of saying, “Hey, we got to get this going sooner rather than later.” Brian: Do you have people in sales that are resistant to using the reporting or taking advantage of your information or is it pretty ingrained in the company culture that it’s like, “This is a tool. Why would you not want to use it?” Or did you guys have a […] getting adoption? Jason: Yeah. I would say anytime you’re going through a transformation of this magnitude, it’s hard and I would say especially for other manufacturers. I found in general, manufacturing in general, tends to be one of the laggards industry-wise in analytical maturity. Unquestionably, it’s tough for no other reason than change is tough. You’re taking legacy plants, legacy steer pieces, legacy process, and some people has been around the company for decades potentially, and we’re asking them to change almost on a dime on their time scale how they do business. It’s not that it’s right or wrong but what we try to point out is that, as I always say, we have to acknowledge the past. We’ve been where we’ve been, we’ve been successful at where we been. But there’s been more change in the past two or three years than maybe you’ve seen in the past 15-20 years. In order to stay relevant, you really have to be ready to evolve, not only evolve but evolve quickly. But I have to openly acknowledge that that’s hard. It’s a hard proposition for a lot of people. Again, it comes down to change management and managing not only expectations but supporting that change. Change doesn’t happen by itself, we have to support that. That’s really what we try to coach through. The way that we try to do that is by developing a product with our customers. I’m sure as you can […], if you force something upon somebody, it doesn’t get received too well. But if you develop it in conjunction with them and do tie it around their needs, it tends to get better adaptation. Brian: You used the word product in there and I’m interested, do you see the outputs of your efforts? Primarily, it’s BI reporting as I understand it. Do you look at that as the product that you offer to sales? Is that kind of how you see it? Jason: Yeah. We offer a product in the form of the insight packages but it’s also the service. Service that goes with it where again, we serve as essentially internal consultant to help them along. If you take just the product-centric approach, you just deliver an insight package and you’re like, “Good luck. It’s [00:19:35]. Have at it.” What we do is we deliver the product and then we partner with them and say, “Okay, here’s what we see. Now, remember you’re talking about this going on in the channel last year and our note show that there’s been a lot of competitive activity in this area. Here’s some of the question that we have. You’re the expert, so what do you think?” What we found is that working together like that, we tend to get pretty good results versus just leaving these guys on an island to kind of figure it out themselves because they virtually always know the answer but sometimes it’s up to us using these products and then offering the service is to ask question that maybe aren’t getting asked. A lot of times, we find out that they know the answer it’s just that you kind of have to ask the question. Brian: Is that often like, “I was using XYZ report. Could you break this down by county instead of just by whatever because I feel there’s more people living in the East side of town and the average is here or […] the whole county. I really just need this one county because that’s where everyone lives. Is that really underserved? Blah, blah, blah,” that kind of stuff and then you guys will go off and work with them for more of that detail then maybe you release that back into the product as a feature if it seems like a one-off or something. Is that how it works? Jason: It’s actually a very fluid process. An example of what you just described is exactly what happens if hey come to us with questions. But we also do it where we flip it around because a lot of products that we create are more aggregate discussion tools. We don’t design a lot within our primary visualization package. To really get into the weeds and everything just becomes overwhelming. We have other tools like your traditional [00:21:22] pivot table to kind of dig into that stuff. But the exact example that you just gave, they will ask us those questions, but we will also flip the script and say, “Hey, we saw that the mechanical chain in the Northeast is up 50%,” I’m just making up a number, “and at a higher level, you can see that but when we segment it out, here’s what we see. Not only when we break this down to this level, we see that’s specifically being driven by A, B and C.” That gets to where I push heavier at my team to do root cause analysis. That’s really where we provide value is by digging into it and asking questions like that. Again, operating from the lens of trying to solve a problem or answer a question or root cause something in conjunction with the business. A lot of times, we will ask those questions and at the same time, they will ask us, which is great. It’s amazing because you get the better solution faster. Brian: I think that’s great. I’ve worked on several different tools that have varying sophisticated means of doing root cause analysis and I think it’s a really powerful way to bring some why to a what that has happened in the past. Most of the time, why is really where the money is at. The value comes in being able to understand why. A lot of times, we don’t have all the data. You can’t know for sure but a lot of times I tend to say, “Our guess, if they’re just going to make a WAG—a wild ass guess—then our guess, as long as we qualify what ingredients went into the pie, our guess may be better than any WAG.” They’re going to make one already. If they’re going to make a decision here and go off gut, there is maybe a chance they’re right and their experience will say something. But maybe our elementary root cause analysis, which we can improve over time, will actually be better and we can get out of the total guessing game and start with something that’s kind of a macro ballpark thing. Then overtime, you can improve that analysis as new data becomes available or maybe learn about how two variables are related in the business and you can bring that knowledge into the system. I totally hear what you’re saying. It’s a nice mix of internal product plus services and also, it sounds like it gets you guys do good discovery work as well. You guys are not just responding to questions but you’re maybe asking them questions together as a group. You kind of work through what opportunities maybe latent that no one’s talking about by asking questions using data to do that. Jason: Yeah. In the lens that we’ve been talking through, this is really sales-centric, but this applies to any group that we interact with. We have the same level of proactive discussion with any group that we interact with. In some of these, in our market research side, it’s 100% proactive. We’re going out there scouring for information and trying to see the other things that we see. That one it’s completely proactive and now we bring insights to the business and say, “What do you guys think?” The sales one is the most fun because, let’s be honest, there’s no business if you’re not selling anything and nothing happens until a sale is made. Brian: Right. I get that. You talked about other clients, do you work at all with the actual hardware, is there any IoT type of analytics going on with the boilers and machinery that you guys create? Jason: We’re early in that process. We actually are getting ready to go down that task very soon. On the hardware side, we tend to not have as much involvement. That’s really more on the engineering group. I think for any manufacturer product or engineering groups probably going to be the most involved in that. But obviously, we get involved into the discussions of answering the fundamental question. What are we actually going to do with this data when we collect it? Because as you can imagine, IoT can spit out a lot of data real quick. They can become incredibly burdensome very quickly if you don’t have a plan on how to manage it. But then, if you’re going to go through the effort of managing that, you got to be able to say, “What are we going to do with this?” Brian: Yeah. I guess the first thing that would come to mind for me would be predictive maintenance, like, “Is it going to break down soon?” I worked on a cooling company that does cooling and really as the guy told me is like, “We’re not selling refrigeration. We’re selling consistent temperature to our clients. It’s not really about coolers and all of that, so we need to deliver consistent temperature. If we don’t do that, they lose products, they can lose whatever is being stored in cold storage.” That is significant business. I’m sure for you guys, it’s heat, you want to sell heat so how do you get in front if there’s a maintenance plan or whatever, how do you stay on top of that kind of stuff? Jason: Absolutely. Brian: [00:26:13] IoT. One of my clients used this word one time, which I now use all the time which is like, “We don’t want a metrics toilet.” An example of you can get to a metrics toilet really quickly with every stat under the gun and how many ounces of water per minute through this pipe, that’s great because that’ll help me do, as a sales guy or as a technician, how am I going to use that information just because there’s a sensor on that pipe. It’s working something around like, “Oh, there’s a sensor. Put the data in the grid.” Jason: I’m going to have to borrow that. I’ll give credit whenever I use ‘metrics toilet,’ that’s a pretty good one. I may actually [00:26:56]. Brian: Nice. Tell me, where does it go from here? You had mentioned like, “Oh, the competitive edge, maybe it’s closing.” Or maybe you guys feel your competitors are all kind of maybe they’re doing the same thing that you guys are doing and we are all aware of where the data can be used to drive the business. Are there other places where you see design or technology like predictive analytics or machine learning and some of these other new technologies that are out there to help drive predictions and things like that? Are you guys leveraging any of that or have plans to look to the future? What does that look like? I know you probably can’t talk about everything but maybe broadly. Jason: Absolutely. I would say that that’s content that’s definitely, if it’s not already being done then it’s on our radar. We’ve got a pretty talented team here that goes a lot of your traditional data science turf. As you can probably surmise in this conversation, is in addition to having all skills, we’re probably the most heavily focused on the business side. As we say, we explore opportunities for a lot of this. We always look at it, again, like machine learning. Great, but we got to make sure it’s very powerful stuff. We got to make sure that whatever we’re embarking upon, because we have finite work capacity, if you pursue something, machine learning, it means we’re not doing something else. It’s not to say that it’s not important, but we really have to be able to answer to that question. Again, come back to, “This is our anchor. What are we going to do with it?” I love this stuff. I love the stats. I love machine learning, AI, all that stuff. If you’re not careful, you can really quickly get into an academic exercise that we think is really cool. “Oh wow, look at this. We’ve got this awesome algorithm here. It does all this magical stuff,” and then the business looks at it and goes, “Yeah, so what? I don’t care. How does that generate revenue? How does that improve our margins? How does that reduce our cost? How does that enable to build the sales pipeline?” If we can’t answer those base questions and we don’t get alignment, that’s probably the most important thing is executive buy-in on exactly what we’re going to be working on, why it’s important. No, we don’t pursue it but those things are most definitely, as with any analytics teams today, I think that that content is definitely being done and/or on your radar. Brian: You make a really good point. Sometimes I almost hesitate to ask the question. But I think it’s an exciting space in terms of predictive capability and removing viable analysis and what we call time tool time in the design world, there is there. But at the same time, you make a really good point which is again, these are tools that need to be leveraged to service an opportunity or a problem. The goal is not to go do the machine learning, the goal is to solve a business problem by which machine learning maybe applied a better […] do it, reduce cost or reduce effort, speed, something like that. I completely respect that. I’m glad to hear that you guy are looking that as not a leading step. I know there’s conflicting signals out there. I’ve been talking to people in the International Institute for Analytics about this and at the same time you hear a lot of stuff which is, “If AI is not part of your strategy, you’re going to be missing out,” and boards just want to hear that people are doing AI. At the same time, you’ve got academic exercises going on, you’ve got people trying to take on massive like, “We’re going to shoot for the moon,” and it’s like, “You don’t even have an airplane and you’re trying to go to the moon with this thing. Show us a small win if you’re going to do an investment in AI.” It’s okay to go try it out and say, “Let’s do a small thing but let’s try to solve a business problem or have some definable output that we’re looking to do here such that we’re not just writing code and doing experiments.” I hear there’s a problem with people putting this on their resume. It’s like people just want to have machine learning. Everyone’s a data scientist now that used to stay in analytics. [00:30:47] It’s scary in the sense of just wasting opportunity and wasting money because at some point, your smarter competitors are going to be saying, “This is a new hammer. Let’s find some nails that we can use for it. But we think […] right nails and it needs to be the right application before we whack at it. It’s not just […].” Jason: I really like your point because again, if my peers were listening to this they will laugh because they say, “We are professionals of this trade and the tools that we might want to use might not be the right tool to use for a specific job.” I couldn’t agree more with that sentiment. It’s one of those core philosophies that I have and share with my team. Also to it is with the AI. I think that you truly made a very astute observation here and comment in that, I think a lot of companies do feel compelled to have to make significant investment in AI like today. It’s not to say that there’s not merit. There clearly is plenty of merit and plenty of potential there, but kind of your point, I really believe that it’s much more beneficial when you really minimize the risk of project and budget flow and minimize overall project risk. You take that small bite and try a little bit, then try a little bit more. When you get to win, socialize the win, and your executives feel comfortable because I’ve done it on the analytics side. I went for a big bang approach and after nine months they were like, “Hey, man. Where’s the output?” All you need is to get bit by that once and then you realize that small quick wins are very effective because at its core, it’s really important to get executive buy-in. A lot of executives are not willing to wait nine months or a year for something when they’re expecting to see at three months. I totally agree with your sentiment. Brian: When you talked about the wins, I totally understand if you’re close to it and maybe hard to remember those, but is there a particular story or time where something in the product and the insights that you guys put out to your customers that it was like a real win, like a sales guy said something to you or maybe an executive said something to you about how this moved the needle, like this was a memorable moment for us. Like, “I changed a customer’s mind with this,” or, “We closed the sale that we never would have been looking over here if we didn’t do it.” Do you have any anecdotes like that that you can share? Jason: One that we had recently, again, just for confidentiality purposes I can’t get too deep. Brian: Sure. Jason: We did have one recently where we just basically revamped our insights packages that we distribute to our internal team. We really, really gathered feedback. We had version one, we gathered ton of feedback, kind of refined, iterated, got the feedback without making it a major release. Got feedback, refined it, refined it, and then what we did was, with a small group, we got that beta in their hand, they look at it and they’re like, “This is great. This is exactly what we need.” Because what we were doing, what we found was—I’m sure you’ve experienced this—everybody wants their own part of things. Everybody wants certain view of a report or they want certain insights or whatever it is, and it’s great. But if you have limited resources, really high-powered resources like an analytics team or data science team, you’re going to look at the opportunity cost of trying to do one of these one-offs, we were getting a ton of report flow. Again, what I tell my team, I don’t mean to be derogatory to the DI guys in this comment, but my team’s side, I always tell them, “We don’t create value if we’re just creating reports. We create value when we’re actually partnering with business to extract insights, identify opportunities amidst all that stuff that goes well with it.” What we realized though is that, what started out as a nice, clean, three- or four-page insights package and blow it up to like 20 and [34:21] doesn’t that meet our original criteria? Essentially, what we do is once we have the rationalization enough to say, “Okay, we’ve got all these stuffs right across 20 pages. We can actually distill it down to four pages.” It will give you the exact same information, but it might not look the exact way that you wanted it to look. The question becomes, are you willing to deal with less stuff and maybe have it look a little different, but you’ll get it in a much more concise package that you’re actually able to use and process? What we found out is that a lot of people were doing these packages and getting the reports that they want but they weren’t actually using them to drive decision-making because they can’t see the paragraph or the block of text story before. They look at it and they’re like, “I don’t know what the hell to do with this.” We would dial that in and it just been a screaming success. It’s really nice to have it where something like that you see the evolution of it. This is just one of those things that we had, and this was kind of a side package or wasn’t a primary, but it’s become a primary now because it’s so effective. Brian: Would you say that when you talk about reducing this, is the report like a PDF or do they access it through a browser the insights package? Jason: Yeah, we have the options to do both. We distribute it initially via PDF, sometimes along with our comments if there’s really, really big stuff in there. We’ll say, “Hey, we see this. Here’s a driver. Here’s a supplemental package.” A lot of times it’s PDF first and then if they want to go on the web, start interacting with it, they can do that. Those are nice, but the reality is a lot of them don’t do that which is understandable. Brian: You took it from 20 pages down to 4, is that what you’re saying? Jason: Yeah. Same information. Brian: This is a really good point. I’ve frequently had clients come in and they’re with data products and their concern is information overload. We’ve heard this a lot of times and the irony is that, the issue is usually not information overload. It’s usually a design problem that the information is not presented properly because sometimes, it can increase the density and increase the utility and usability, not the other way around. In fact, removing data can actually make it worse. A basic example of that is when you’re trying to compare A and B. If A and B are not on the same, what we call a viewport like in a browser world, it would be within the browser window there. When you require someone to toggle between two screens, they have to change context and visually, your eye can process the information a lot better when it’s within proximity. Sometimes, increasing the density actually will give you a better design. It takes more care in how you do it, but it’s not always about information overload, “Oh, it’s too crazy.” They may not get it on the first time but your sales people, if they’re looking at this stuff weekly or monthly, at some point they’re going to be pretty comfortable with this. I always tell my clients, “You need to look at the switch frequency as well because if it’s going to be used a lot, you can actually get more detailed and you can really push the, what you might see as complexity or the information density, can go up because they’re going to get familiar with the formatting. Typically, the density is actually going to probably improve the utility as long as care is given to the choices. But having that eyeball comparison without having to change pages and all of that, typically you’re going to give a better story as a broad rule. I like hearing that you guys went down in page count, up in density and in turn a better user experience at the end so that’s great. I think we’re about done here. I don’t have too many questions for you, but this is super great. One of the reasons I contacted Jason is because I remember seeing this quote, “Jason is like a category five hurricane in the data analytics world.” I’m like, “Who the hell is this guy? No one talks like that.” I started reading your stuff and I enjoyed watching your LinkedIn social posts and things like that. Where can people find out more about you? You’re obviously on LinkedIn, I can put LinkedIn in the show notes and stuff, but are you on Twitter, any social media places they can follow you? Jason: No, actually, I’m not on Twitter. But the best place unquestionably is going to be LinkedIn. I’m pretty involved there. I do like to engage. If you want to direct message me with questions, just talk, meetup, connect, whatever it is, I welcome that. I love the platform, it’s a great family. I just really started using it maybe nine months ago, really getting into it. It’s been great meeting guys like yourself. It’s actually phenomenal. Brian: Cool. I’ll put a link to Jason’s LinkedIn profile on there and you guys can find him. I recommend, especially if you’re in an internal analytics type of role at your company, to follow Jason and then check out what he has to say on there. This has been great. Thanks for coming on the show. I look forward to meeting you at some point in person. Jason: Dude, thank you for having me on here. I really appreciate it. We hope you enjoyed this episode of Experiencing Data with Brian O’Neill. If you did enjoy it, please consider sharing it with #experiencingdata. To get future podcast updates or to subscribe to Brian’s mailing list where he shares his insights on designing valuable enterprise data products and applications, visit designingforanalytics.com/podcast. Never forget to look up the online HTML CheatSheet when you forget how to write an image, a table or an iframe or any other tag in HTML! [bws_google_captcha] Subscribe for Podcast Updates Get updates on new episodes of Experiencing Data plus my occasional insights on design and UX for custom enterprise data products and apps. Email Address [text-blocks id="eu-consent-checkbox-textblock" plain="1"]

005 – Jason Krantz (Dir. of Biz Analytics/Insights, Weil-McClain) on centering analytics around internal customers

Play Episode Listen Later Jan 29, 2019 39:57

Jason Krantz is the Director of Business Analytics & Insights for the 135-year old company, Weil McLain and Marley Engineered Products. While the company is responsible for helping keeping homes and businesses warm, Jason is responsible for the creation and growth of analytical capabilities at Weil McLain, and was recognized in 2017 as a “Top 40 Under 40” in the HVAC industry. I’m not surprised given his posts on LinkedIn; Jason seems very focused on satisfying his internal customers and ensuring that there is practical business value anchoring their analytics initiatives. We talked about: How Jason’s team keeps their data accessible and relevant to the issue they need to solve for their customer. How Jason strives to keep the information simple and clean for the customer. How does Jason help drive analytics in a company culture with a lot of legacy (from its people to its parts) The importance of focusing on context How Jason drives his team to be business partners, and not report generators Resources and Links: Jason Krantz on LinkedIn Quotes from Jason Krantz: “You realize that small quick wins are very effective because, at its core, it’s really important to get executive buy-in.” “I’m a huge fan of simplicity. As analytics pros, we could very easily make very complex, very intricate models, and just, ‘Oh, look at how smart we are.’ It doesn’t help our customers. …we only use about two or three different visual types and we use mostly the exact same visual set-up. I can train a sales rep for probably five minutes on all of our reporting because if you understand one, you’re going to understand everything. That gets to the theme again of just simplicity. Don’t over complicate, keep it simple, keep it clean.” “…To get buy-in, you really got to have your business case, even to your internal customers, really dialed in. If you just bring them a bunch of crap, that’s how you’re going to lose credibility. They’re going to be like, “I don’t have the time to waste with you,” even though we’re trying to be helpful.” “What my team and I do is we really help companies weaponize their data assets.” Episode Transcript Brian: Jason, are you there? Jason: I’m here my friend. Brian: Sweet. How’s it going? Jason: It’s going very well today. How’s your Friday going? Brian: I’m doing awesome. We’re going to talk a little bit about analytics. Is it Wile McLain or Weil McLain? Jason: I say Weil McLain. If I’ve been saying it wrong, I’ve been saying it wrong for a while. Brian: As I recall from my musical training, I think in German, the second syllable is the one that says its name. I guess it would be Wile McLain, like if it was W-I-E-L it would be ‘Weil.’ But I don’t know. Its anglicized as they come over the pond. Jason: I’m going to go with you on that when you sound like an expert. Brian: Nice. Well, you sound like an expert in analytics at Weil McLain. Tell us about what you’re doing over there. We met on LinkedIn, I’ve been enjoying your postings on the social feed about your approach. You seem really passionate about what you’re doing and I’m like, “I don’t know who this guy is, but that was really interesting.” I just have. Tell us about the company, what they do. I know they’re in heating, right? Jason: Yes, absolutely. The company I work for, and I work in the HVAC space, we’re a 135-year-old boiler manufacturer. Whether you realize it or not, you probably have one of our products in your house or building or very close to where you live. What my team and I do is we really help companies weaponize their data assets. As you know, a lot of companies are very skilled at acquiring data since the Big Data Movement. But the reality is that a lot of these companies don’t know what to do with all this data. That’s where we really come in. What I always tell my team and our business partners that we work with internally and externally is that our focus is on solving business problems. In order to do that, you have to identify what is the business problem that you’re trying to solve or strategic agenda that you’re trying to address. In order to do that, you really have to be anchored in the biz. Again, that’s just my perspective, but if you’re in the business day in, day out, you develop this very keen stand of what the business would need to accomplish its objectives. Just like right now, we are based in the marketing group and it’s a great spot to be. I’m a firm believer that every analytics team should be based in the business for a reason that I just talked about. But what that does being business-first is that gives us a great lens to look at data from. Sometimes analytics people would be IT-centric and they can do a lot of academic work against the data set or different data sets. But the business might look at the output and be like, “Yeah, that doesn’t help us.” We always, always, always start with, “What is the business problem we’re trying to solve or strategy we’re looking to address?” It also helps us when it comes to curating data also. That’s one of our primary response [00:03:21] this too, is to look for different data sets both internal and external that can help us identify strategic opportunities. It sounds really unsexy, I’m not going to lie. I think some of my LinkedIn post just say that data is boring. It really is. It’s mind-numbing, too, about 85% of my customers. But that’s the important part is understanding what do our customers need and that’s really the lens that we look at this through. We are a service provider, our customers are internal and external, we have customers just like any other business. We have to take this really boring, but really potent product in data and make it accessible to them. That’s really where we use design to really try to make that magic happen. Brian: I love that you said, “Trying to understand what the problem is.” This is something we talk about on the mailing list quite a bit. In fact, falling in love with the problem is a good basis for doing good work instead of kind of jumping to solutions or feeling […]. As I tell my clients sometimes like, “Our job is not to go and visualize the data. It’s not […] available for someone to put into another tool or whatever the heck it is. The job is to find an insight that already is used. Probably they’re already in your job and you’re there to make […] if you’re doing internal analytics. Help them do a better job at what they’re doing, offer more value. You need to figure out how to work that into their life.” For example, for you guys then, your customer, I assume is it primarily sales people that you’re working with? Who are your customers and your […]? Jason: Yes. Great question. One of our biggest customers is sales. Sales has been one of my biggest customers for the past 10 years of my career. I’m very intimately involved with the sales team, sales operation, sales optimization, insight gathering, pricing, things like that, but also marketing. We do a lot in terms of competitive intelligence gathering, market research. We also do a lot of operations in finance obviously related to the prices, that sort of thing. We really touch all areas of the business, but without question, our biggest customers are going to be sales and marketing. Brian: If you were to bring a new initiative like, “Hey, we have access to…” I don’t know what it might be but for you maybe your point, [might be a line 00:05:49] of data that could actually give them more leverage. We know what the negotiation brings, better than […], we know we kind of have an idea now from what the industry is doing for their sales such that we can now tell the CRM like, “This is your […] or something.” When do you get that sales person involved? Do you deliver a solution and get feedback? Do you bring […] early and say, “Hey, we think we can tell you more about how to do better pricing on the spot with this thing.” Do you bring them in or when do they fit into your process? Jason: Great question. A lot of times because we spend so much time actually in the trenches, that’s one of things I think is unique about the way that I design my teams to do analytics. It’s not like hand off product and we’re like, “Godspeed. Good luck.” Once we deliver a solution, we’re actually in the trenches with the business trying to implement what we’re talking about because it just works better. The team work is just more effective and they know that they’ve got back up, they know they’ve got air support. Really, a lot of times when we come up with something new, a lot of times we will frame it from the lens like, “Hey, we know that we’ve got opportunity A or issue B, or whatever it is. This has been an issue or an opportunity for months or years or whatever.” We think that we’ve identified something that could help us in solving that issue or realizing the potential of that opportunity and then it becomes, “Okay, let’s sit down and talk about, do you agree that this might actually help us in this process?” Because the one thing that I’ve learned is, in order to get buy-in, you really, really got to have your business case, even to your internal customers, really dialed in. If you just bring them a bunch of crap, that’s how you’re going to lose credibility. They’re going to be like, “I don’t have the time to waste with you,” even though we’re trying to be helpful. What we found out is if you really dial in what are we trying to address with this, just as you would with any business case, and you bring that to them, I have found that they tend to be much more receptive. It’s not to say there’s not going to be resistance—resistance comes with any change—but we found that typically framing it from that lens and saying, “We’re trying to solve a problem that you have, we think that this data will help,” that’s a great starting point. Brian: Do you have an example of a before/after with that? I don’t want you to get into proprietary stuff you can’t talk about but is there like a, “Before they did it this way,” and then we brought them in and said, “Hey, we think we can get […].” and how you went [00:08:19]. Jason: Yeah. What I can talk about is just the manner in which we distribute sales information, specifically insights. I think that, for your listeners, this is going to ring true to a lot of sales forces. I know for all them that I’ve been in or worked with, this case was true 100% of the time. But one of the things that, again, keeping the customer-centric focus, that if you look at your sales reps, a lot of time is you’re going to be what I call casual data consumers. By that, I mean that these are guys and gals that aren’t really into data day in and day out like guys like you and I or some of the listeners maybe. What we have to do is, as I always encourage my team to take empathetic lens and look at, “Okay, if we give them what our first […] is going to look like, how are they going to interpret this?” A lot of times, to be honest, it’s not very good. Now that’s where we have to look at internally and kind of rationalize and say, “Okay, let’s find this. One of us will find [00:09:14].” But one example of that is traditionally, sales reps and sales teams will get the information in a flat Excel table. Just lots of rows and columns and just gibberish everywhere. That’s a very financial-centric view of sales data. But the reality is—I don’t know about the rest of mankind but I know for myself—I can’t remember much more than 10 numbers. The mental computational cost of extracting insights is just gargantuan. What happens is, I just don’t even bother to do it. I’m just like, “Yeah, whatever.” An equivalent of that is, you know if you get a big block of text in email? Even though if you took that same block of text and broke it up into two paragraphs or two sentence segments which is very easy to read when you put the effort in, but for me, if I get a big block of text, I’m not even going to read that. It’s kind of one of the same things that we see on the sales side. What we do is just say, “You know what? There’s a lot of really good information here and we need to make it digestible for our customers.” That’s where we found traditionally, visualization can be an incredibly effective tool to communicate insights to this casual data audience, to this casual data consumer. Brian: Do you have to work through the visuals with them? Do they tend to get it the first time? Is it a process of you share, “Here’s a report or here’s some new view on X.” How do you know if the visualization is actually allowing them to pull the insight out of what other [00:10:46] broad data? How do you know they’re actually “getting it”? Jason: That’s a great question. I’m a huge fan of simplicity. As analytics pros, we could very easily make very complex, very intricate models, and just, “Oh, look at how smart we are.” It doesn’t help our customers. It doesn’t help anything. Really what we do—this is going to get to the theme of simplicity—is we only use about two or three different visual types and we use mostly the exact same visual set-up. Just to kind of frame it, what I’m a big fan of is a simple bar chart. There’s more details attached to it but to the right of the bar chart, we’ll typically put a tabular data set. What we do is, as you think in US at least, we start in that left-hand side of the page or we […]. What we do is we look at the visual real estate. We say, “Our customers are going to start in the left-hand side. We want them to look at the bar chart because it allows them to very rapidly assimilate it at a high-level what’s going on.” It’s great at communicating at top-level churn very quickly but the trade-off is, is this horrible imprecision. You have no precision at all. What we like to do is then we address that issue by putting a simple table, very clean, very simple table over to the right. What that does is that then provides the precision that the customers are seeing in most financial-centric tables. What we found that does is that we have to train our sales team on one set-up and then that set-up is used virtually universally on all of our solutions. As an example, I can train a sales rep for probably five minutes on all of our reporting because if you understand one, you’re going to understand everything. That gets to the theme again of just simplicity. Don’t over complicate, keep it simple, keep it clean. Brian: I think those are good. A lot of times, when I work with engineering clients, they fall in love with consistency. I guess one point to maybe just the contrary of this is that, I think consistency is generally a good rule with design. We want to minimize unnecessary change but at the same time, I would recommend to listeners is to always look at context first, and context should always come in. Let’s say Jason comes up with report number 12 and they have 11 now or whatever, and it doesn’t feel right for number 11. That’s a place where a designer would probably push for, “Well, no. The 12th one actually needs to be different because it’s not […] 11th and even though it’s not consistent, in this context, we don’t need it to win. This version will deliver the usability and the utility that we’re looking for better than the other 11 will.” In general, I think it’s smart to not get creative unnecessarily with meaningless ink on the screen like, “Let’s try it this way. Let’s change the color palette. I’m tired of this.” Those are not good reasons for […], you’re just introducing noise and it’s unnecessary. But I like that you guys are thinking about simplicity and trying to reuse templates and not looking at it as a creative tableau. Ironically, people think it’s a creative “design” tool, but at the same time with all those weapons, you have a lot of different weapons you can use in that toolkit and part of that is knowing how to use this. It’s the same thing with Photoshop, a million buttons and all this stuff. The Photoshop doesn’t make you a designer. It’s being aware of your customer’s pain and the problems they need and knowing when to use all those filters and all those different things that it can do. I like that you guys are looking into that simplicity and reusing templates when it’s meaningful to do so. Jason: You bring up some great points and I 100% agree. My team that’s listening there, they’ll laugh because I beat it in their heads, “Context. Context, context, context.” Both in design as you’ve talked but especially with numbers in general. Like, “If I give you a number, a billion, that doesn’t mean anything, you got to have context.” I’d say the same is true for design just as you articulated. Great point. Brian: Where does the impetus for “everybody is a data company, everybody wants to do analytics”? But then there’s operationalizing that, there’s getting buy-in, leadership behind it. Where does that come from in your org? Where is the interest in taking a 130-year-old company and getting it to care about this? Where does that come from, your influence and all of that? Jason: That was driven by our current president because he saw it as part of a digital transformation. Obviously, this was an essential component of that. Obviously, we do a lot with analytics, but we’re also involved in a lot of other digital components that lead to that overall digital maturation. Analytics is a very, very big part of what we do but it’s not all that we do. We serve as kind of that quarterback for a lot of the digital initiatives to help basically, guide them through the process. Because even though some of the nuances of each of this project, each one will have its own nuances, they all come back to data. Data is the currency. We found out pretty quickly that if you want to stay relevant in this day and age, you need to be digitally evolved but more importantly, as you look at it, do you [compare the 00:16:02] advantage that you can derive from analytics? I would argue that gap is slowly closing known certain industries like manufacturing, but we probably have a little bit more runway [00:16:10] it. But for a lot of industries, analytics is becoming table stakes. It’s one of those where you can certainly expect incremental value and competitive advantage, but the question becomes how much longer. That was kind of the impetus of saying, “Hey, we got to get this going sooner rather than later.” Brian: Do you have people in sales that are resistant to using the reporting or taking advantage of your information or is it pretty ingrained in the company culture that it’s like, “This is a tool. Why would you not want to use it?” Or did you guys have a […] getting adoption? Jason: Yeah. I would say anytime you’re going through a transformation of this magnitude, it’s hard and I would say especially for other manufacturers. I found in general, manufacturing in general, tends to be one of the laggards industry-wise in analytical maturity. Unquestionably, it’s tough for no other reason than change is tough. You’re taking legacy plants, legacy steer pieces, legacy process, and some people has been around the company for decades potentially, and we’re asking them to change almost on a dime on their time scale how they do business. It’s not that it’s right or wrong but what we try to point out is that, as I always say, we have to acknowledge the past. We’ve been where we’ve been, we’ve been successful at where we been. But there’s been more change in the past two or three years than maybe you’ve seen in the past 15-20 years. In order to stay relevant, you really have to be ready to evolve, not only evolve but evolve quickly. But I have to openly acknowledge that that’s hard. It’s a hard proposition for a lot of people. Again, it comes down to change management and managing not only expectations but supporting that change. Change doesn’t happen by itself, we have to support that. That’s really what we try to coach through. The way that we try to do that is by developing a product with our customers. I’m sure as you can […], if you force something upon somebody, it doesn’t get received too well. But if you develop it in conjunction with them and do tie it around their needs, it tends to get better adaptation. Brian: You used the word product in there and I’m interested, do you see the outputs of your efforts? Primarily, it’s BI reporting as I understand it. Do you look at that as the product that you offer to sales? Is that kind of how you see it? Jason: Yeah. We offer a product in the form of the insight packages but it’s also the service. Service that goes with it where again, we serve as essentially internal consultant to help them along. If you take just the product-centric approach, you just deliver an insight package and you’re like, “Good luck. It’s [00:19:35]. Have at it.” What we do is we deliver the product and then we partner with them and say, “Okay, here’s what we see. Now, remember you’re talking about this going on in the channel last year and our note show that there’s been a lot of competitive activity in this area. Here’s some of the question that we have. You’re the expert, so what do you think?” What we found is that working together like that, we tend to get pretty good results versus just leaving these guys on an island to kind of figure it out themselves because they virtually always know the answer but sometimes it’s up to us using these products and then offering the service is to ask question that maybe aren’t getting asked. A lot of times, we find out that they know the answer it’s just that you kind of have to ask the question. Brian: Is that often like, “I was using XYZ report. Could you break this down by county instead of just by whatever because I feel there’s more people living in the East side of town and the average is here or […] the whole county. I really just need this one county because that’s where everyone lives. Is that really underserved? Blah, blah, blah,” that kind of stuff and then you guys will go off and work with them for more of that detail then maybe you release that back into the product as a feature if it seems like a one-off or something. Is that how it works? Jason: It’s actually a very fluid process. An example of what you just described is exactly what happens if hey come to us with questions. But we also do it where we flip it around because a lot of products that we create are more aggregate discussion tools. We don’t design a lot within our primary visualization package. To really get into the weeds and everything just becomes overwhelming. We have other tools like your traditional [00:21:22] pivot table to kind of dig into that stuff. But the exact example that you just gave, they will ask us those questions, but we will also flip the script and say, “Hey, we saw that the mechanical chain in the Northeast is up 50%,” I’m just making up a number, “and at a higher level, you can see that but when we segment it out, here’s what we see. Not only when we break this down to this level, we see that’s specifically being driven by A, B and C.” That gets to where I push heavier at my team to do root cause analysis. That’s really where we provide value is by digging into it and asking questions like that. Again, operating from the lens of trying to solve a problem or answer a question or root cause something in conjunction with the business. A lot of times, we will ask those questions and at the same time, they will ask us, which is great. It’s amazing because you get the better solution faster. Brian: I think that’s great. I’ve worked on several different tools that have varying sophisticated means of doing root cause analysis and I think it’s a really powerful way to bring some why to a what that has happened in the past. Most of the time, why is really where the money is at. The value comes in being able to understand why. A lot of times, we don’t have all the data. You can’t know for sure but a lot of times I tend to say, “Our guess, if they’re just going to make a WAG—a wild ass guess—then our guess, as long as we qualify what ingredients went into the pie, our guess may be better than any WAG.” They’re going to make one already. If they’re going to make a decision here and go off gut, there is maybe a chance they’re right and their experience will say something. But maybe our elementary root cause analysis, which we can improve over time, will actually be better and we can get out of the total guessing game and start with something that’s kind of a macro ballpark thing. Then overtime, you can improve that analysis as new data becomes available or maybe learn about how two variables are related in the business and you can bring that knowledge into the system. I totally hear what you’re saying. It’s a nice mix of internal product plus services and also, it sounds like it gets you guys do good discovery work as well. You guys are not just responding to questions but you’re maybe asking them questions together as a group. You kind of work through what opportunities maybe latent that no one’s talking about by asking questions using data to do that. Jason: Yeah. In the lens that we’ve been talking through, this is really sales-centric, but this applies to any group that we interact with. We have the same level of proactive discussion with any group that we interact with. In some of these, in our market research side, it’s 100% proactive. We’re going out there scouring for information and trying to see the other things that we see. That one it’s completely proactive and now we bring insights to the business and say, “What do you guys think?” The sales one is the most fun because, let’s be honest, there’s no business if you’re not selling anything and nothing happens until a sale is made. Brian: Right. I get that. You talked about other clients, do you work at all with the actual hardware, is there any IoT type of analytics going on with the boilers and machinery that you guys create? Jason: We’re early in that process. We actually are getting ready to go down that task very soon. On the hardware side, we tend to not have as much involvement. That’s really more on the engineering group. I think for any manufacturer product or engineering groups probably going to be the most involved in that. But obviously, we get involved into the discussions of answering the fundamental question. What are we actually going to do with this data when we collect it? Because as you can imagine, IoT can spit out a lot of data real quick. They can become incredibly burdensome very quickly if you don’t have a plan on how to manage it. But then, if you’re going to go through the effort of managing that, you got to be able to say, “What are we going to do with this?” Brian: Yeah. I guess the first thing that would come to mind for me would be predictive maintenance, like, “Is it going to break down soon?” I worked on a cooling company that does cooling and really as the guy told me is like, “We’re not selling refrigeration. We’re selling consistent temperature to our clients. It’s not really about coolers and all of that, so we need to deliver consistent temperature. If we don’t do that, they lose products, they can lose whatever is being stored in cold storage.” That is significant business. I’m sure for you guys, it’s heat, you want to sell heat so how do you get in front if there’s a maintenance plan or whatever, how do you stay on top of that kind of stuff? Jason: Absolutely. Brian: [00:26:13] IoT. One of my clients used this word one time, which I now use all the time which is like, “We don’t want a metrics toilet.” An example of you can get to a metrics toilet really quickly with every stat under the gun and how many ounces of water per minute through this pipe, that’s great because that’ll help me do, as a sales guy or as a technician, how am I going to use that information just because there’s a sensor on that pipe. It’s working something around like, “Oh, there’s a sensor. Put the data in the grid.” Jason: I’m going to have to borrow that. I’ll give credit whenever I use ‘metrics toilet,’ that’s a pretty good one. I may actually [00:26:56]. Brian: Nice. Tell me, where does it go from here? You had mentioned like, “Oh, the competitive edge, maybe it’s closing.” Or maybe you guys feel your competitors are all kind of maybe they’re doing the same thing that you guys are doing and we are all aware of where the data can be used to drive the business. Are there other places where you see design or technology like predictive analytics or machine learning and some of these other new technologies that are out there to help drive predictions and things like that? Are you guys leveraging any of that or have plans to look to the future? What does that look like? I know you probably can’t talk about everything but maybe broadly. Jason: Absolutely. I would say that that’s content that’s definitely, if it’s not already being done then it’s on our radar. We’ve got a pretty talented team here that goes a lot of your traditional data science turf. As you can probably surmise in this conversation, is in addition to having all skills, we’re probably the most heavily focused on the business side. As we say, we explore opportunities for a lot of this. We always look at it, again, like machine learning. Great, but we got to make sure it’s very powerful stuff. We got to make sure that whatever we’re embarking upon, because we have finite work capacity, if you pursue something, machine learning, it means we’re not doing something else. It’s not to say that it’s not important, but we really have to be able to answer to that question. Again, come back to, “This is our anchor. What are we going to do with it?” I love this stuff. I love the stats. I love machine learning, AI, all that stuff. If you’re not careful, you can really quickly get into an academic exercise that we think is really cool. “Oh wow, look at this. We’ve got this awesome algorithm here. It does all this magical stuff,” and then the business looks at it and goes, “Yeah, so what? I don’t care. How does that generate revenue? How does that improve our margins? How does that reduce our cost? How does that enable to build the sales pipeline?” If we can’t answer those base questions and we don’t get alignment, that’s probably the most important thing is executive buy-in on exactly what we’re going to be working on, why it’s important. No, we don’t pursue it but those things are most definitely, as with any analytics teams today, I think that that content is definitely being done and/or on your radar. Brian: You make a really good point. Sometimes I almost hesitate to ask the question. But I think it’s an exciting space in terms of predictive capability and removing viable analysis and what we call time tool time in the design world, there is there. But at the same time, you make a really good point which is again, these are tools that need to be leveraged to service an opportunity or a problem. The goal is not to go do the machine learning, the goal is to solve a business problem by which machine learning maybe applied a better […] do it, reduce cost or reduce effort, speed, something like that. I completely respect that. I’m glad to hear that you guy are looking that as not a leading step. I know there’s conflicting signals out there. I’ve been talking to people in the International Institute for Analytics about this and at the same time you hear a lot of stuff which is, “If AI is not part of your strategy, you’re going to be missing out,” and boards just want to hear that people are doing AI. At the same time, you’ve got academic exercises going on, you’ve got people trying to take on massive like, “We’re going to shoot for the moon,” and it’s like, “You don’t even have an airplane and you’re trying to go to the moon with this thing. Show us a small win if you’re going to do an investment in AI.” It’s okay to go try it out and say, “Let’s do a small thing but let’s try to solve a business problem or have some definable output that we’re looking to do here such that we’re not just writing code and doing experiments.” I hear there’s a problem with people putting this on their resume. It’s like people just want to have machine learning. Everyone’s a data scientist now that used to stay in analytics. [00:30:47] It’s scary in the sense of just wasting opportunity and wasting money because at some point, your smarter competitors are going to be saying, “This is a new hammer. Let’s find some nails that we can use for it. But we think […] right nails and it needs to be the right application before we whack at it. It’s not just […].” Jason: I really like your point because again, if my peers were listening to this they will laugh because they say, “We are professionals of this trade and the tools that we might want to use might not be the right tool to use for a specific job.” I couldn’t agree more with that sentiment. It’s one of those core philosophies that I have and share with my team. Also to it is with the AI. I think that you truly made a very astute observation here and comment in that, I think a lot of companies do feel compelled to have to make significant investment in AI like today. It’s not to say that there’s not merit. There clearly is plenty of merit and plenty of potential there, but kind of your point, I really believe that it’s much more beneficial when you really minimize the risk of project and budget flow and minimize overall project risk. You take that small bite and try a little bit, then try a little bit more. When you get to win, socialize the win, and your executives feel comfortable because I’ve done it on the analytics side. I went for a big bang approach and after nine months they were like, “Hey, man. Where’s the output?” All you need is to get bit by that once and then you realize that small quick wins are very effective because at its core, it’s really important to get executive buy-in. A lot of executives are not willing to wait nine months or a year for something when they’re expecting to see at three months. I totally agree with your sentiment. Brian: When you talked about the wins, I totally understand if you’re close to it and maybe hard to remember those, but is there a particular story or time where something in the product and the insights that you guys put out to your customers that it was like a real win, like a sales guy said something to you or maybe an executive said something to you about how this moved the needle, like this was a memorable moment for us. Like, “I changed a customer’s mind with this,” or, “We closed the sale that we never would have been looking over here if we didn’t do it.” Do you have any anecdotes like that that you can share? Jason: One that we had recently, again, just for confidentiality purposes I can’t get too deep. Brian: Sure. Jason: We did have one recently where we just basically revamped our insights packages that we distribute to our internal team. We really, really gathered feedback. We had version one, we gathered ton of feedback, kind of refined, iterated, got the feedback without making it a major release. Got feedback, refined it, refined it, and then what we did was, with a small group, we got that beta in their hand, they look at it and they’re like, “This is great. This is exactly what we need.” Because what we were doing, what we found was—I’m sure you’ve experienced this—everybody wants their own part of things. Everybody wants certain view of a report or they want certain insights or whatever it is, and it’s great. But if you have limited resources, really high-powered resources like an analytics team or data science team, you’re going to look at the opportunity cost of trying to do one of these one-offs, we were getting a ton of report flow. Again, what I tell my team, I don’t mean to be derogatory to the DI guys in this comment, but my team’s side, I always tell them, “We don’t create value if we’re just creating reports. We create value when we’re actually partnering with business to extract insights, identify opportunities amidst all that stuff that goes well with it.” What we realized though is that, what started out as a nice, clean, three- or four-page insights package and blow it up to like 20 and [34:21] doesn’t that meet our original criteria? Essentially, what we do is once we have the rationalization enough to say, “Okay, we’ve got all these stuffs right across 20 pages. We can actually distill it down to four pages.” It will give you the exact same information, but it might not look the exact way that you wanted it to look. The question becomes, are you willing to deal with less stuff and maybe have it look a little different, but you’ll get it in a much more concise package that you’re actually able to use and process? What we found out is that a lot of people were doing these packages and getting the reports that they want but they weren’t actually using them to drive decision-making because they can’t see the paragraph or the block of text story before. They look at it and they’re like, “I don’t know what the hell to do with this.” We would dial that in and it just been a screaming success. It’s really nice to have it where something like that you see the evolution of it. This is just one of those things that we had, and this was kind of a side package or wasn’t a primary, but it’s become a primary now because it’s so effective. Brian: Would you say that when you talk about reducing this, is the report like a PDF or do they access it through a browser the insights package? Jason: Yeah, we have the options to do both. We distribute it initially via PDF, sometimes along with our comments if there’s really, really big stuff in there. We’ll say, “Hey, we see this. Here’s a driver. Here’s a supplemental package.” A lot of times it’s PDF first and then if they want to go on the web, start interacting with it, they can do that. Those are nice, but the reality is a lot of them don’t do that which is understandable. Brian: You took it from 20 pages down to 4, is that what you’re saying? Jason: Yeah. Same information. Brian: This is a really good point. I’ve frequently had clients come in and they’re with data products and their concern is information overload. We’ve heard this a lot of times and the irony is that, the issue is usually not information overload. It’s usually a design problem that the information is not presented properly because sometimes, it can increase the density and increase the utility and usability, not the other way around. In fact, removing data can actually make it worse. A basic example of that is when you’re trying to compare A and B. If A and B are not on the same, what we call a viewport like in a browser world, it would be within the browser window there. When you require someone to toggle between two screens, they have to change context and visually, your eye can process the information a lot better when it’s within proximity. Sometimes, increasing the density actually will give you a better design. It takes more care in how you do it, but it’s not always about information overload, “Oh, it’s too crazy.” They may not get it on the first time but your sales people, if they’re looking at this stuff weekly or monthly, at some point they’re going to be pretty comfortable with this. I always tell my clients, “You need to look at the switch frequency as well because if it’s going to be used a lot, you can actually get more detailed and you can really push the, what you might see as complexity or the information density, can go up because they’re going to get familiar with the formatting. Typically, the density is actually going to probably improve the utility as long as care is given to the choices. But having that eyeball comparison without having to change pages and all of that, typically you’re going to give a better story as a broad rule. I like hearing that you guys went down in page count, up in density and in turn a better user experience at the end so that’s great. I think we’re about done here. I don’t have too many questions for you, but this is super great. One of the reasons I contacted Jason is because I remember seeing this quote, “Jason is like a category five hurricane in the data analytics world.” I’m like, “Who the hell is this guy? No one talks like that.” I started reading your stuff and I enjoyed watching your LinkedIn social posts and things like that. Where can people find out more about you? You’re obviously on LinkedIn, I can put LinkedIn in the show notes and stuff, but are you on Twitter, any social media places they can follow you? Jason: No, actually, I’m not on Twitter. But the best place unquestionably is going to be LinkedIn. I’m pretty involved there. I do like to engage. If you want to direct message me with questions, just talk, meetup, connect, whatever it is, I welcome that. I love the platform, it’s a great family. I just really started using it maybe nine months ago, really getting into it. It’s been great meeting guys like yourself. It’s actually phenomenal. Brian: Cool. I’ll put a link to Jason’s LinkedIn profile on there and you guys can find him. I recommend, especially if you’re in an internal analytics type of role at your company, to follow Jason and then check out what he has to say on there. This has been great. Thanks for coming on the show. I look forward to meeting you at some point in person. Jason: Dude, thank you for having me on here. I really appreciate it.

Changing How We Do Science with Brian Nosek [Idea Machines #3]

Idea Machines

Play Episode Listen Later Dec 7, 2018 58:17

My guest this week is Brian Nosek, co-Founder and the Executive Director of the Center for Open Science. Brian is also a professor in the Department of Psychology at the University of Virginia doing research on the gap between values and practices, such as when behavior is influenced by factors other than one's intentions and goals. The topic of this conversation is how incentives in academia lead to problems with how we do science, how we can fix those problems, the center for open science, and how to bring about systemic change in general. Show Notes Brian’s Website Brian on Twitter (@BrianNosek) Center for Open Science The Replication Crisis Preregistration Article in Nature about preregistration results The Scientific Method If you want more, check out Brian on Econtalk Transcript Intro [00:00:00] This podcast I talked to Brian nosek about innovating on the very beginning of the Innovation by one research. I met Brian at the Dartmouth 60th anniversary conference and loved his enthusiasm for changing the way we do science. Here's his official biography. Brian nozik is a co-founder and the executive director for the center for open science cos is a nonprofit dedicated to enabling open and reproducible research practices worldwide. Brian is also a professor in the department of psychology at the University of Virginia. He's received his PhD from Yale University in 2002 in 2015. He was on Nature's 10 list and the chronicle for higher education influence. Some quick context about Brian's work and the center for open science. There's a general consensus in academic circles that there are glaring problems in how we do research today. The way research works is generally like this researchers usually based at a university do experiments then when they have a [00:01:00] result they write it up in a paper that paper goes through the peer-review process and then a journal publishes. The number of Journal papers you've published and their popularity make or break your career. They're the primary consideration for getting a position receiving tenure getting grants and procedure in general that system evolved in the 19th century. When many fewer people did research and grants didn't even exist we get into how things have changed in the podcast. You may also have heard of what's known as the replication crisis. This is the Fairly alarming name for a recent phenomena in which people have tried and failed to replicate many well-known studies. For example, you may have heard that power posing will make you act Boulder where that self-control is a limited resource. Both of the studies that originated those ideas failed to replicate. Since replicating findings a core part of the scientific method unreplicated results becoming part of Cannon is a big deal. Brian has been heavily involved in the [00:02:00] crisis and several of the center for open science is initiatives Target replication. So with that I invite you to join my conversation with Brian idzik. How does open science accelerate innovation and what got you excited about it? Ben: So the theme that I'm really interested in is how do we accelerate Innovations? And so just to start off with I love to ask you sort of a really broad question of in your mind. How does having a more open science framework help us accelerate Innovations? And I guess parallel to that. Why what got you excited about it first place. Brian: Yeah, yeah, so that this is really a core of why we started the center for open science is to figure out how can we maximize the progress of science given that we see a number of different barriers to or number of different friction points to the PACE and progress of [00:03:00] Science. And so there are a few things. I think that how. Openness accelerates Innovation, and I guess you can think of it as sort of multiple stages at the opening stage openness in terms of planning pre-registering what your study is about why you're doing this study that the study exists in the first place has a mechanism of helping to improve Innovation by increasing The credibility of the outputs. Particularly in making a clear distinction between the things that we planned in advance that we're testing hypotheses of ideas that we have and we're acquiring data in order to test those ideas from the exploratory results the things that we learn once we've observed the data and we get insights but there are necessarily more uncertain and having a clear distinction between those two practices is a mechanism for. Knowing the credibility of the results [00:04:00] and then more confidently applying results. That one observes in the literature after the fact for doing next steps. And the reason that's really important I think is that we have so many incentives in the research pipeline to dress up exploratory findings that are exciting and sexy and interesting but are uncertain as if they were hypothesis-driven, right? We apply P values to them. We apply a story upfront to them we present them as. These are results that are highly credible from a confirmatory framework. Yeah, and that has been really hard for Innovation to happen. So I'll pause there because there's lots more but yeah, so listen, let's touch on that. What has changed to make the problem worse? Ben: There's there's a lot that right there. So you mentioned the incentives to basically make. Things that aren't really following the scientific method follow the clicker [00:05:00] following the scientific method and one of the things I'm always really interested in what has changed in the incentives because I think that there's definitely this. Notion that this problem has gotten worse over time. And so that means that that something has has changed and so in your mind like what what changed to make to sort of pull science away from that like, you know sort of ice training ideal of you have your hypothesis and then you test that hypothesis and then you create a new hypothesis to this. System that you're pushing back against. Brian: You know, it's a good question. So let me start with making the case for why we can say that nothing has changed and then what might lead to thinking something has changed in unpacking this please the potential reason to think that nothing has [00:06:00] changed is that the kinds of results that are the most rewarded results have always been the kinds of results that are more the most rewarded results, right? If I find a novel Finding rather than repeating something someone else has done. I'm like. To be rewarded more with publication without latex cetera. If I find a positive result. I'm more likely to gain recognition for that. Then a negative result. Nothing's there versus this treatment is effective, which one's more interesting. Well, we know which ones for interesting. Yeah. Yeah, and then clean and tidy story write it all fits together and it works and now I have this new explanation for this new phenomenon that everyone can can take seriously so novel positive clean and tidy story is the. They'll come in science and that's because it breaks new ground and offers a new idea and offers a new way of thinking about the world. And so that's great. We want those. We've always wanted those things. So the reason to think well, this is a challenge always is [00:07:00] because. Who doesn't want that and and who hasn't wanted that right? It turns out my whole career is a bunch of nulls where I don't do anything and not only fits together. It's just a big mess right on screen is not a way to pitch a successful career. So that challenge is there and what pre-registration or committing an advanced does is helps us have the constraints. To be honest about what parts of that are actual results of credible confrontations of pre-existing hypotheses versus stuff that is exploring and unpacking what it is we can find. Okay, so that in this in the incentive landscape, I don't think has changed. Mmm what thanks have changed. Well, there are a couple of things that we can point to as potential reasons to think that the problem has gotten worse one is that data acquisition many fields is a lot easier than it ever was [00:08:00] and so with access more data and more ways to analyze it more efficient analysis, right? We have computers that do this instead of slide rules. We can do a lot more adventuring in data. And so we have more opportunity to explore and exploit the malays and transform it into things signal. The second is that the competitive landscape is. Stronger, right there are fewer than the ratio of people that want jobs to jobs available is getting larger and larger and larger and that fact and then competitiveness for Grants and same way that competition than can. Very easily amplify these challenges people who are more willing to exploit more researcher degrees of freedom are going to be able to get the kinds of results more easily that are rewarded in the system. And so that would have amplify the presence of those in people that managed to [00:09:00] survive that competitive firm got it. So I think it's a reasonable hypothesis that people that it's gotten worse. I don't think there's definitive evidence but those would be the theoretical points. At least I would point to for that. That makes a lot of sense. So you had a just sort of jumping back. You had a couple a couple points and we had we have just touched on the first one. Point Number Two about Accelerating Innovation Ben: So I want to give you that chance to oh, yeah go back and to keep going through that. Brian: Right. Yeah. So accelerating Innovation is the idea, right? So that's a point of participation is accelerating Innovation by by clarifying The credibility of claims as they are produced. Yes, we do that better than I think will be much more efficient that will have a better understanding of the evidence base as it comes out. Yeah second phase is the ability is the openness of the data and materials for the purposes of verify. Those [00:10:00] initial claims right? I do a study. I pre-registered. It's all great and I share it with you and you read it. And you say well that sounds great. But did you actually get that and what would have happened if you made different decisions here here and there right because I don't quite agree with the decisions that you made in your analysis Pipeline and I see some gaps there so you're being able to access the materials that I produced in the data that came from. Makes it so that you can one just simply verify that you can reproduce the findings that I reported. Right? I didn't just screw up the analysis script or something and that as a minimum standard is useful, but even more than that, you can test the robustness in ways that I didn't and I came to that question with some approach that you might look at it and say well I would do it differently and the ability to reassess the data for the same question is a very useful thing for. The robustness particularly in areas that are that have [00:11:00] complex analytic pipelines where there's are many choices to make so that's the second part then the third part is the ReUse. So not only should we be able to verify and test the robustness of claims as they happen, but data can be used for lots of different purposes. Sometimes there are things that are not at all anticipated by the data originator. And so we can accelerate Innovation by making it a lot easier to aggregate evidence of claims across multiple Studies by having the data being more accessible, but then also making that data more accessible and usable for. Studying things that no one no one ever anticipated trying to investigate. Yeah, and so the efficiency gain on making better use of the data that already exists rather than the Redundant just really do Revenue question didn't dance it your question you did as it is a massive efficiency. Opportunity because there is a lot of [00:12:00] data there is a lot of work that goes in why not make the most use of it began? What is enabled by open science? Ben: Yeah that makes a lot of sense. Do you have any like really good sort of like Keystone examples of these things in action like places where because people could replicate the. The the study they could actually go back to the pipeline or reuse the data that something was enabled. That wasn't that wouldn't have been possible. Otherwise, Brian: yeah. Well, let's see. I'll give a couple of local mean personal examples just to just to illustrate some of the points, please so we have the super fun project that we did just to illustrate this second part of the pipeline right this robustness phase of. People may make different choices and those choices may have implications for the reliability results. So what we did in this project was that we get we acquired a dataset [00:13:00] of a very rich data set of lots of players and referees and outcomes in soccer and we took that data set and then we recruit a different teams. 29 in the end different teams with lots of varied expertise and statistics and analyzing data and have them all investigate the same research. Which is our players with darker skin tone more likely to get a red card then players with lighter skin tone. And so that's you know, that's a question. We'll of Interest people have studied and then we had provided this data set. Here's a data set that you can use to analyze that and. The teams worked on their own and developed an analysis strategies for how they're going to test that hypothesis. They came up with their houses strategy. They submitted their analysis and their results to us. We remove the results and [00:14:00] then took their analysis strategies and then share them among the teams for peer review right different people looking at it. They have made different choices. They appear each other and then went back. They took those peer reviews. They didn't know what each other found but they took. Because reviews and they wanted to update their analysis they could and so they did all that and then submitted their final analyses and what we observed was that a huge variation in analysis choices and variation in the results. So as a simple Criterion for Illustrated the variation results two-thirds of the teams found a significant. Write P less than 0.05 standard for deciding whether you see something there in the data, right and Atherton teams found a null. So the and then of course they debated amongst each other which was analysis strategy was the right strategy but in the end it was very clear among the teams that there are lots of reasonable choices that could be made. And [00:15:00] those reasonable choices had implications for the results that were observed from the same data. Yeah, and it's Standard Process. We do not see the how it's not easy to observe how the analytics choices influence the results, right? We see a paper. It has an outcome we say those are what the those fats those the outcomes of the data room. Right, but what actually the case is that those are the outcomes the data revealed contingent on all those choices that the researcher made and so that I think just as an illustrative illustrative. So it helps to figure out the robustness of that particular finding given the many different reasonable choices. That one would make where if we had just seen one would have had a totally different interpretation, right either. Yeah, it's there or it's not there. How do you encode context for experiments esp. with People? Ben: Yeah, and in terms of sort of that the data and. [00:16:00] Really sort of exposing the the study more something that that I've seen especially in. These is that it seems like the context really matters and people very often are like, well there's there's a lot of context going on in addition to just the procedure that's reported. Do you have any thoughts on like better ways of sort of encoding and recording that context especially for experiments that involve? Brian: Yeah. Yeah. This is a big challenge is because we presume particularly in the social and life sciences that there are many interactions between the different variables. Right but climate the temperature the time of day the circadian rhythms the personalities whatever it is that is the different elements of the subjects of the study whether they be the plants or people or otherwise, yeah. [00:17:00] And so the. There are a couple of different challenges here to unpack one is that in our papers? We State claims at the maximal level of generality. We can possibly do it and that that's just a normal pattern of human communication and reasoning right? I do my study in my lab at the University of Virginia on University of Virginia undergraduates. I don't conclude in the. University of university University of Virginia undergraduates in this particular date this particular time period this particular class. This is what people do with the recognition that that might be wrong right with recognition. There might be boundary conditions but not often with articulating where we think theoretically those boundary conditions could be so in one step of. Is actually putting what some colleagues in psychology of this great paper about about constraints on [00:18:00] generality. They suggest what we need in all discussion sections of all papers is a sexually say when won't this hold yeah, just give them what you know, where where is this not going to hold and just giving people an occasion to think about that for a second say oh. - okay. Yeah, actually we do think this is limited to people that live in Virginia for these reasons right then or no, maybe we don't really think this applies to everybody but now we have to say so you can get the call it up. So that alone I think would make a huge difference just because it would provide that occasion to sort of put the constraints ourselves as The Originators of findings a second factor, of course is just sharing as much of the materials as possible. But often that doesn't provide a lot of the context particularly for more complex experimental studies or if there are particular procedural factors right in a lot of the biomedical Sciences there. There's a lot of nuance [00:19:00] into how it is that this particular reagent needs to be dealt with how they intervention needs to be administered Etc. And so I like the. Moves towards video of procedures right? So there is a journal Journal of visualized events jove visualized experiments that that that tries to that gives people opportunities to show the actual experimental protocol as it is administered. To try to improve it a lot of people using the OSF put videos up of the experiment as they administered it. So to maximize your ability to sort of see how it is that it was done through. So those steps I think can really help to maximize the transparency of those things that are hard to put in words or aren't digitally encoded oil. Yeah, and those are real gaps What is the ultimate version of open science? Ben: got it. And so. In your mind what is sort of like the endgame of all this? What is it? Like what [00:20:00] would be the ideal sort of like best-case scenario of science? Like how would that be conducted? So I say you get to control the world and you get to tell everybody practicing science exactly what to do. What would that look like? Brian: Well, if it if I really had control we would just all work on Wikipedia and we would just revising one big paper with the new applicants. Ask you got it continuously and we get all of our credit by. You know logging how many words that I changed our words that survived after people have made their revisions and whether those words changed are on pages that were more important for the overall scientific record versus the less important spandrels. And so we would output one paper that is the summary of knowledge, which is what Wikipedia summarizes. All right, so maybe that's that's maybe going a little bit further than what like [00:21:00] that we can consider. The realm of conceptually possible. So if we imagine a little bit nearer term, what I would love to see is the ability to trace the history of any research project and that seems more achievable in the sense that. If a every in fact, my laboratory is getting close to this, right every study that we do is registered on the OSF. And once we finish the studies, we post the materials and the data or as we're doing it if we're managing the materials and data and then we attach a paper if we write a paper at the end preprint or the final report so that people can Discover it and all of those things are linked together. Be really cool if I had. Those data in a standardized framework of how it is that they are [00:22:00] coded so that they could be automatically and easily integrated with other similar kinds of data so that someone going onto the system would be able to say show me all the studies that ever investigated this variable associated with this variable and tell me what the aggregate result is Right real-time meta-analysis of the entire database of all data that I've ever been collected that. Enough flexibility would help to really very rapidly. I think not just spur Innovations and new things but to but help to point out where there are gaps right there a particular kinds of relationships between things particular effects of predict interventions where we know a ton and then we have this big assumption in our theoretical framework about how we get from X to y. And then as we look for variables that help us to identify whether X gets us to why we feel there just isn't stuff. The literature has not filled that Gap. So I think there are huge benefits for that [00:23:00] kind of aggregate ability. But mostly what I want to be able to do is instead of saying you have to do research in any particular way. The only requirement is you have to show us how you did your research and your particular way so that the marketplace of ideas. Can operate as efficiently as possible and that really is the key thing? It's not preventing bad ideas from getting into the system. It's not about making sure that the different kinds of best things are the ones that immediately are through with not that about Gatekeepers. It's about efficiency in how it is. We call that literature of figuring out which things are credible which things are not because it's really useful to. The ideas into the system as long as they can be. Self-corrected efficiently as well. And that's where I think we are not doing well in the current system. We're doing great on generation. [00:24:00] We're General kinds of innovative ideas. Yeah, but we're not is parsing through those ideas as efficiently as it could decide which ones are worth actually investing more resources in jumping. A couple levels in advance that Talmud for Science Ben: that makes a lot of sense and actually like I've definitely come across many papers just on the internet like you go and Google Scholar and you search and you find this paper and in fact, it has been refuted by another paper and there's no way to know that yeah, and so. I does your does the open science framework address that in any way? Brian: No, it doesn't yet. And this is a critical issue is the connectivity between findings and the updating of knowledge because the way that like I said doesn't an indirect way but it doesn't in the systematic way that actually would solve this problem. The [00:25:00] main challenge is that we treat. Papers as static entities. When what their summarizing is happening very dynamically. Right. It may be that a year later. After that paper comes out one realizes. We should have analyze that data totally different. We actually analyzed it wrong is indefensible the way that we analyzed it. Right right. There are very few mechanisms for efficiently updating that paper in a way that would actually update the knowledge and that's something where we all agree. That's analyze the wrong way, right? What are my options? I could. Retract the paper. So it's no longer in existence at all. Supposedly, although even retracted papers still get cited we guess nuts. So that's a base problem. Right or I could write a correction, which is another paper that comments on that original paper that may not itself even be discoverable with the original paper that corrects the analysis. Yeah, and that takes months and years. [00:26:00] All right. So the really what I think is. Fundamental for actually addressing this challenge is integrating Version Control with scholarly publishing. So that papers are seen as Dynamic objects not static objects. And so if you know what I would love to see so here's another Milestone of this if we if I could control everything another Milestone would be if a researcher could have a very productive career with. Only working on a single paper for his or her whole life, right? So they have a really interesting idea. And they just continue to investigate and build the evidence and challenge it and figure, you know, just continue to unpack it and they just revise that paper over time. This is what we understand. Now, this is where it is. Now. This is what we've learned over here are some other exceptions but they just keep fine-tuning it and then you get to see the versions of that paper over its [00:27:00] 50-year history as that phenomenon got unpacked that. Plus the integration with other literature would make this much more efficient for exactly the problem that you raised which is we with papers. We don't know what the current knowledge base is. We have no real good way except for these. These attempts to summarize the existing literature with yet a new paper and that doesn't then supersede those old papers. It's just another paper is very inefficient system. Can Social Sciences 'advance' in the same way as the physical sciences? Ben: Ya know that that totally makes sense. Actually. I just I have sort of a meta question that I've argued with several people about which is do you feel like. We can make advances in our understanding of sort of like [00:28:00] human-centered science in the same way that we can in like chemistry or physics. Like people we very clearly have like building blocks of physics and the Builds on itself. And there's I've had debates with people about whether you can do this in. In the humanities and the social sciences. What are your thoughts on that? Brian: Yeah. It is an interesting question and the. What seems to be the biggest barrier is not anything about methodology in particular but about complexity? Yeah, right, if the problem being many different inputs can have similar impact cause similar kinds of outcomes and singular inputs can have multivariate outcomes that it influences and all of those different inputs in terms of causal elements may have interactive effects on the [00:29:00] outs, so. How can we possibly develop Rich enough theories to predict the actions effectively and then ultimately explain the actions effectively of humans in a complex environments. It doesn't seem that we will get to the beautiful equations that underlie a lot of physics and chemistry and count for a substantial amount of evidence. So the thing that I don't feel like I under have any good hand along with that is if it's a theoretical or practical limit right is it just not possible because it's so complex and there isn't this predicted. Or it's just that's really damn hard. But if we had big enough computers if you had enough data, if we were able to understand complex enough models, we would be able to predict it. Right so is as a mom cycle historians, right? They figure it out right the head. [00:30:00] Oxidizing web series righty they could account for 99.9 percent of the variance of what people do next and but of course, even there it went wrong and that was sort of the basis of the whole ceilings. But yeah, I just don't know I don't have a way to. I don't yet have a framework for thinking about how is it that I could answer that question whether it's a practical or theoretical limit. Yeah. What do you think? Ben: What do I think I think that it's great. Yeah, so I usually actually come down on the I think it's a practical limit now how much it would take to get there might make it effectively a theoretical limit right now. But that there's there's nothing actually preventing us from like if you if you could theoretically like measure everything why not? I [00:31:00] think that is just with again. It's like the it's really a measurement problem and we do get better at measuring things. So that's the that's that's where I come down on but I. How do you shift incentives in science? Yep, that's just purely like I have no good argument. going going back to the incentives. It seems to me like a lot of what like I'm completely convinced that these changes would. Definitely accelerate the number of innovations that we have and so and it seems like a lot of these changes require shifting scientists incentives. And so and that's like a notoriously hard thing so we both like how are you going about shifting those incentives right now and how might they be shifted in the future. [00:32:00] Brian: Yeah, that's a great question. That's what we spend. A lot of our time worrying about in the sense of there is very little at least in my experience is very distal disagreement on the problems and the opportunities for improving the pace of Discovery and Innovation based on the solutions. It really is about the implementation. How is it that you change that those cultural incentives so that we can align. The values that we have for science with the practices that researchers do on a daily basis and that's a social problem. Yeah, there are technical supports. But ultimately it's a social problem. And so the the near term approach that we have is to recognize the systems of rewards as they are. And see how could we refine those to align with some of these improved practices? So we're not pitching. Let's all work on [00:33:00] Wikipedia because that's that is so far distant from. What they systems have reward for scientist actually surviving and thriving in science that we wouldn't be able to get actually pragmatic traction. Right? So I'll give one example of can give a few but here's the starting with one of an example that integrates with current incentives but changes them in a fundamental way and that is the publishing model of registered reports. Sophie in the standard process right? I do my research. I write up my studies and then I submit them for peer review at the highest possible prestigious Journal that I can hoping that they will not see all the flaws and if they'll accept it. I'll get all the do that process me and I understand it anyway - journal and the P plus Terminal C and eventually somewhere and get accepted. The register report model makes one change to the process and that is to move. The critical point of peer review [00:34:00] from after the results are known and I've written up the report and I'm all done with the research to after I've figured out what the question that I want to investigate is and what the methodology that I'm going to use so I don't have an observed the outcomes yet. All I've done is frame question. An articulated why it's important and a methodology that I'm going to just to test that question and that's what the peer reviewers evaluate right? And so the key part is that it fits into the existing system perfectly, right? The the currency of advancement is publication. I need to get as many Publications as I can in the most prestigious Outlets. I can to advance my career. We don't try to change that. Instead we just try to change. What is the basis for making a decision about publication and by moving the primary stage of peer reviewed before the results are known does a fundamental change in what I'm being rewarded for as the author [00:35:00] right? Yeah, but I'm being rewarded for as the author in the current system is sexy results, right get the best most interesting most Innovative results. I can write and the irony of that. Is that the results of the one thing that I'm not supposed to be able to control in your study? Right? Right. What I'm supposed to be able to control is asking interesting questions and developing good methodologies to test those questions. Of course that's oversimplifying a bit. There are in there. The presumption of emphasizing results is that my brilliant insights at the outset of the project are the reason that I was able to get those great results, right, but that depends on the credibility of that entire Pipeline and put that aside but the moving it to at the design stage means that my incentive as an author is to ask the most important questions that I can. And develop the most compelling and effective and valid methodologies that I can to test them. [00:36:00] Yeah, and so that changes to what it is presumably we are supposed to be being rewarded for in science. The other thing that it changes in the there's a couple of other elements of incentive changes that it has an impact on that are important for the whole process right for reviewers instant. It's. When I am asked to review a paper in my area of research when I when all the results are there, I have skin in the game as a reviewer. I'm an expert in that area. I may have made claims about things in that particular area. Yeah, if the paper challenges my cleanse make sure to find all kinds of problems with the methodology. I can't believe they did this is this is a ridiculous thing, right? We write my paper. That's the biggest starting point problem challenge my results all well forget out of you. But the amount of course if it's aligned with [00:37:00] my findings and excites me gratuitously, then I will find lots of reasons to like the paper. So I have these Twisted incentives to reinforce findings and behave ideologically as a reviewer in the existing system by moving peer review to the design stage. It fundamentally changes my incentives to right so say I'm in a very contentious area of research and there's only ten opponents on a particular claim when we are dealing with results You can predict the outcome right it people behave ideologically even when they're not trying to when you don't know the results. Both people have the same interests, right? If I truly believe in the phenomenon that I'm studying and the opponents of my point of view also believe in their perspective, right then both want to review that study and that design and that methodology to maximize its quality to reveal the truth, which I think I [00:38:00] have and so that alignment actually makes adversaries. To some extent allies and in review and makes the reviewer and the author more collaborative, right the feedback that I give on that paper can actually help the methodology get better. Whereas in the standard process when I say here's all the things you did wrong. All the author has this to say well geez, you're a jerk. Like I can't do anything about that. I've already done the research and so I can't fix it. Yeah. So the that shifts earlier is much more collaborative and helps with that then the other question is the incentives for the journal right? So in the. Journal editors have strong incentives of their own they want leadership. They want to have impact they don't want the one that destroyed their journal and so [00:39:00] the incentives and the in the existing model or to publish sexy results because more people were read those results. They might cite those results. They might get more attention for their Journal, right? And shifting that to on quality designs then shift their priorities to publishing the most rigorous research the most rust robust research and to be valued based on that now. Yeah, so I'll pause there there's lots of other things to say, but those I think are some critical changes to the incentive landscape that still fits. Into the existing way that research is done in communicated. Don't people want to read sexy results? Ben: Yeah. I have a bunch of questions just to poke at that last point a little bit wouldn't people still read the journals that are publishing the most sexy results sort of regardless of whether they were web what stage they're doing that peer review. Brian: Yeah. This is a key concern of editors and thinking about adopting registered reports. [00:40:00] So we have about a hundred twenty-five journals that are offering this now, but we continue to pitch it to other groups and other other ones, but one of the big concerns that Hunters have is if I do this then I'm going to end up publishing a bunch of no results and no one will read my journal known will cite it and I will be the one that ruined my damn door. All right. So it is a reasonable concern because of the way the system works now, so there's a couple answers to that but the one is empirical which is is it actually the case that these are less red or less cited than regular articles that are published in those. So we have a grant from the McDonald Foundation to actually study registered reports. And the first study that we finished is a comparison of articles that were done as register reports with this in the same published in the same Journal. [00:41:00] Articles that were done the regularly to see if they are different altmetrics attention, right citation and attention and Oppa in media and news and social media and also citation impact at least early stage citation impact because the this model is new enough that it isn't it's only been working for since 2014. In terms of first Publications and what we found in that is that at least in this initial data set. There's no difference in citation rates, and if anything the register report. Articles have gotten more altmetric impact social media news media. That's great. So at least the initial data suggests that who knows if that will sustain generalize, but the argument that I would make in terms of a conceptual argument is that if Studies have been vetted. In terms of without knowing the results. These are important results to know [00:42:00] right? So that's what the actors and the reviewers have to decide is do we need to know the outcome of this study? Yeah, if the answer is yes that this is an important enough result that we need to know what happened that any result is. Yeah, right. That's the whole idea is that we're doing the study harder find out what the world says about that particular hypothesis that particular question. Yeah, so it become citable. Whereas when were only evaluating based on the results. Well, yeah things that Purity people is that that's crazy, but it happened. Okay, that's exciting. But if you have a paper where it's that's crazy and nothing happened. Then people say well that was a crazy paper. Yeah, and that paper would be less likely to get through the register report kind of model that makes a lot of sense. You could even see a world where because they're being pre-registered especially for more like the Press people can know to pay attention to it. [00:43:00] So you can actually almost like generate a little bit more height. In terms of like oh we're not going to do this thing. Isn't that exciting? Yeah, exactly. So we have a reproducibility project in cancer biology that we're wrapping up now where we do we sample a set of studies and then try to replicate findings from those papers to see where where can we reproduce findings in the where are their barriers to be able to reproduce existing? And all of these went through the journal elife has registered reports so that we got peer review from experts in advance to maximize the quality of the designs and they published instead of just registering them on OSF, which they are they also published the register reports as an article of its own and those did generate lots of Interest rule that's going to happen with this and that I think is a very effective way to sort of engage the community on. The process of actual Discovery we don't know the answer to these [00:44:00] things. Can we build in a community-based process? That isn't just about let me tell you about the great thing that I just found and more about. Let me bring you into our process. How does were actually investigating this problem right and getting more that Community engagement feedback understanding Insight all along the life cycle of the research rather than just as the end point, which I think is much more inefficient than it could be. Open Science in Competitive Fields and Scooping Ben: Yeah and. On the note of pre-registering. Have you seen how it plays out in like extremely competitive Fields? So one of the world's that I'm closest to is like deep learning machine learning research and I have friends who keep what they're doing. Very very secret because they're always worried about getting scooped and they're worried about someone basically like doing the thing first and I could see people being hesitant to write down to [00:45:00] publicize what they're going to do because then someone else could do it. So, how do you see that playing out if at all? Brian: Yeah scoping is a real concern in the sense that people have it and I think that is also a highly inflated concern based on the reality of what happens in practice but nevertheless because people have the concern systems have to be built to address it. Yeah, so one simple answer on the addressing the concern and then reasons to be skeptical at the. The addressing the concern with the OSF you can pre-register an embargo your pre-registrations from to four years. And what that does is it still gets all the benefits of registering committing putting that into an external repository. So you have independent verification of time and date and what you said you were going to do but then gives you as the researcher the flexibility to [00:46:00] say I need this to remain private for some period of time because of whatever reason. As I need it to be private, right? I don't want the recent participants that I am engaged in this project to discover what the design is or I don't want it competitors to discover what the design is. So that is a pragmatic solution is sort of dress. Okay, you got that concern. Let's meet that concern with technology to help to manage the current landscape. There are a couple reasons to be skeptical that the concern is actually much of a real concerning practice Tristan. And one example comes from preprints. So a lot of people when they pre princess sharing the paper you have of some area of research prior to going through peer review and being published in a journal write and in some domains like physics. It is standard practice the archive which is housed at Cornell is the standard for [00:47:00] anybody in America physics to share their research through archive prior to publication in other fields. It's very new or unknown but emerging. But the exact same concern about scooping comes up regularly where they say there's so many people in our field if I share a preprint someone else with the lab that is productive lab is going to see my paper. They're going to run the studies really fast. They're going to submit it to a journal that will publish and quickly and then I'll lose my publication because it'll come out in this other one, right and that's a commonly articulated concern. I think there are very good reasons to be skeptical of it in practice and the experience of archive is a good example. It's been operating since 1991 physicists early in its life articulated similar kinds of concerns and none of them have that concern now, why is it that they don't have that concern now? Well the Norms have shifted from the way you establish priority [00:48:00] is not. When it's published in the journal, it's when you get it onto archive. Right? Right. So a new practice becomes standard. It's when is it that the community knows about what it is you did that's the way you get that first finder Accolade and that still carries through to things like publication a second reason is that. We all have a very inflated sense of self importance that our great our kids right? There's an old saw in in venture capital of take your best idea and try to give it to your competitor and most of the time you can write. We think of our own ideas really amazing and everyone else doesn't yeah people sleeping other people. Is Right Southern the idea that there are people looking their chops on waiting for your paper your registration to show up so they can steal your [00:49:00] idea and then use it and claim it as their own is is great. It's shows High self-esteem. And that's great. I am all for high self. I don't know and then the last part is that. It is a norm violation to do that to such a strong degree to do the stealing of and not crediting someone else for their work, but it's actually very addressable in the daily practice of how science operates which is if you can show that you put that registration or that paper up on a independent service and then it was it appeared prior to the other person doing it. And then that other group did try to steal it and claim it as their own. Well, that's misconduct. And if they did if they don't credit you as the originator then that's something that is a norm violation and how science operates and I'm actually pretty confident in the process of dealing with Norm [00:50:00] violations in the scientific Community. I've had my own experience with the I think this very rarely happens, but I have had an experience with it. I've posted papers on my website before there were pretty print services in the behavioral sciences since I. Been a faculty member and I've got a Google Scholar one day and was reading. Yeah, the papers that I have these alerts set up for things that are related to my work and I paper showed up and I was like, oh that sounds related to some things. I've been working on. So I've clicked on the link to the paper and I went to the website. So I'm reading the paper. I from these authors I didn't recognize and then I realized wait that's that's my paper. I need a second and I'm an author and I didn't submit it to that journal. And it was my paper. They had taken a paper off of my website. They had changed the abstract. They run it through Google translate. It looks like it's all Gobbledy gook, but it was an abstract. But the rest of it was [00:51:00] essentially a carbon copy of our paper and they published. Well, you know, so what did I do? I like contacted the editor and we actually is on retraction watch this story about someone stealing my paper and retraction watch the laughing about it and it got retracted. And as far as we heard the person that had gone it lost their job, and I don't know if that's true. I never followed. But there are systems place is the basic point to deal with the Regis forms of this. And so I have I am sanguine about those not be real issues. But I also recognize they are real concerns. And so we have to have our Technology Solutions be able to address the concerns as they exist today. And I think the those concerns will just disappear as people gain experience. Top down v Bottom up for driving change Ben: Got it. I like that distinction between issues and concerns that they may not be the same thing. To I've been paying attention to sort of the tactics that you're [00:52:00] taking to drive this adoption. And there's some bottom up things in terms of changing the culture and getting one Journal at a time to change just by convincing them and there's also been some some top-down approaches that you've been using and I was wondering if you could just sort of go through those and what you feel like. Is is the most effective or what combinations of things are are the most effective for really driving this change? Brian: Yeah. No, it's a good question because this is a culture change is hard especially with the decentralized system like science where there is no boss and the different incentive drivers are highly distributed. Right, right. He has a richer have a unique set of societies. Are relevant to establishing my Norms you could have funders that fund my work a unique set of journals that I publish in and my own institution. And so every researcher [00:53:00] has that unique combination of those that all play a role in shaping the incentives for his or her behavior and so fundamental change if we're talking about just at the level of incentives not even at the level of values and goals requires. Massive shift across all of those different sectors not massive in terms of the amount of things they need to shift but in the number of groups that need to make decisions tissue. Yeah, and so the we need both top-down and bottom-up efforts to try to address that and the top down ones are. That we work on at least are largely focused on the major stakeholders. So funders institutions and societies particularly ones that are publishing right so journals whether through Publishers societies, can we get them like with the top guidelines, which is this framework that that has been established to promote? What are the transparency standards? What could we [00:54:00] require of authors or grantees or employees of our organizations? Those as a common framework provide a mechanism to sort of try to convince these different stakeholders to adopt new standards new policies to that that then everybody that associated with that have to follow or incentivised to follow simultaneously those kinds of interventions don't necessarily get hearts and minds and a lot of the real work in culture change. Is getting people to internalize what it is that mean is good science is rigorous work and that requires a very bottom up community-based approach to how Norms get established Within. What are effectively very siloed very small world scientific communities that are part of the larger research community. And so with that we do a lot [00:55:00] of Outreach to groups search starting with the idealists right people who already want to do these practices are already practicing rigorous research. How can we give them resources and support to work on shifting those Norms in their small world communities and so. Out of like the preprint services that we host or other services that allow groups to form. They can organize around a technology. There's a preprint service that our Unity runs and then drive the change from the basis of that particular technology solution in a bottom-up way and the great part is that to the extent that both of these are effective they become self reinforcing. So a lot of the stakeholder leaders and editor of a journal will say that they are reluctant. They agree with all the things that we trying to pitch to them as ways to improve rigor and [00:56:00] research practices, but they don't they don't have the support of their Community yet, right. They need to have people on board with this right well in we can the bottom. It provides that that backing for that leader to make a change and likewise leaders that are more assertive are willing to sort of take some chances can help to drive attention and awareness in a way that facilitates the bottom-up communities that are fledgling to gain better standing and we're impact so we really think that the combination of the two is essential to get at. True culture change rather than bureaucratic adoption of a process that now someone told me I have to do yeah, which could be totally counterproductive to Scientific efficiency and Innovation as you described. Ben: Yeah, that seems like a really great place to to end. I know you have to get running. So I'm really grateful. [00:57:00] This is this has been amazing and thank you so much. Yeah, my pleasure.

america university founders community google science discover opportunities phd nature executive director rich innovation system psychology unity journal press target idea massive discovery studies bottom wikipedia stronger revenue pace publishers dynamic innovative studying sciences boulder colorado outreach purity builds machines gap scientific pipeline yale university norm fundamental cornell milestone hunters twisted makes papers grants notion dartmouth openness publications norms keystone illustrated reuse moves talmud mmm gatekeepers regis fairly criterion outlets google scholar atherton redundant originators technology solutions open science right right accolade scooping osf oppa version control retract standard process brian nosek brian yeah brian you brian well ben there ben so gobbledy website brian

001 – Kathy Koontz (International Institute for Analytics) on the growing need for UX and design in the analytics practice

All Angular Podcasts by Devchat.tv

Play Episode Listen Later Nov 21, 2018 38:36

Kathy Koontz is the Executive Director of the Analytics Leadership Consortium at the International Institute for Analytics and my guest for today’s episode. The International Institute of Analytics is a research and advisory firm that discusses the latest trends and the best practices within the analytics field. We touch on how these strategies are used to build accurate and useful custom data products for businesses. Kathy breaks down the steps of making analytics more accessible, especially since data products and analytics applications are more frequently being utilized by front-line workers and not PhDs and analytics experts. She uses her experience with a large property and casualty insurance company to illustrate her point about shifting your company’s approach to analytics to make it more accessible. Small adjustments to a data application make the process effective and comprehensible. Kathy brings some great insights to today’s show about incorporating analytic techniques and user feedback to get the most value from your analytics and the data products you build for the information. Conversation highlights: What is The International Institution of Analytics? What is the analytics leadership consortium? The “squishy” parts of analytics and how to compensate for them. The real value of analytics and how to use it on all levels of a company. How beta testers give perspective on data. The 3 steps to finding the ideal beta tester. Learning from the feedback and implementing it. How to keep ROI in mind during your project. Kathy’s parting advice for the audience. Resources and Links: The International Institute of Analytics Kathy Koontz on LinkedIn Surf Camps Thank you for joining us for today’s episode of Experiencing Data. Keep coming back for more episodes with great conversations about the world of analytics and data. Quotes from today’s episode: “Oftentimes data scientists see the world through data and algorithms and predictions and they get enamored with the complexity of the model and the strength of its predictions and not so much with how easy it is for somebody to use it.” — Kathy Koontz “You are not fully deployed until after you have received this [user] feedback and implemented the needed changes in the application.” — Kathy Koontz “Analytics especially being deployed pervasively is maybe not a project but more of a transformation program.” — Kathy Koontz “Go out and watch your user group that you want to utilize this data or this analytics to improve the performance.” — Kathy Koontz “Obviously, it’s always cheaper to adjust things in pixels, and pencils than it is to adjust it in working code.” — Kathy Koo Transcript Brian: I’m excited to have Kathy Koontz of the line. She is the Executive Director of the Analytics Leadership Consortium at the International Institute for Analytics. That’s quite a mouthful. Did I get that totally right? Kathy: Yeah. You did. I tell people, “Yes, it is a real job.” Yeah, you got it right, you nailed it. Brian: Tell our listeners, what does that mean? What do you do? Kathy: The International Institute for Analytics is an organization that was founded by Tom Davenport, one of the early readers and using analytics for business performance. We are research and advisory firm that helps companies realize value from analytics. Brian: How do you work with them in your leadership capacity there? What’s your specific role there? Kathy: I lead the product line that’s called the Analytics Leadership Consortium. What that is, is a group of analytics executives from different companies who are in non-competing industries that meet of a regular basis to better understand trends and best practices in analytics to vet ideas with one another and ensure that they’re doing the best that they can to deliver analytics value for their organization. A really great opportunity for leaders in this field of analytics that’s changing a lot and has a lot of emerging practices, have regular time to get together in a confidential setting to understand what’s working and what’s not, and how they can improve analytic values for their company. Brian: You mentioned trends, I obviously try to stay on top of what’s going on in that industry and that’s actually how I came across you, originally, I think was you guys had put on a webinar on five trends and analytics going on right now. At the end of that, you had mentioned that one of the things that’s starting to change now is the importance of design and user experience as we move beyond designing reports which is one of the most difficult deliverables, so to speak, of analytics—as we move into the user experience it’s becoming more important. That’s what I was like, “Oh, this could be really interesting to hear what you have to say about what’s changing. Why is UX now relevant? How is capital D design relevant to the world of analytics?” that’s what I was curious to learn about. Can you talk about that a little bit? Kathy: I think as analytics mature in organizations, the need for design is what’s going to drive adoption and utilization of those analytics. In companies that are just starting their analytics journey, it’s a little bit easier to realize analytic value by doing a couple of really big data science projects that don’t really require a lot of design thinking. It’s a powerpoint to an executive group that has some higher level of organizational level thinking that the executive lean in, understand the analytics, understand the value of making their decisions in line with the analytics and then move on and do their regular roles. But as the organizations try to make analytics more pervasive, particularly into front-line associates or individual contributors who are really using analytics to make a lot of small decisions within the execution of their work or as they try to integrate analytics into processes that are monitored, that design thinking taking an approach as user experience, can really break down a lot of barriers that organizations encounter when trying to have people who are not necessarily used to using analytics in their decision-making process, use them so that they can make a better decision for the organization. Brian: Got it. Is the trend that people are recognizing the problem because they went through some pain of maybe they delivered some big multimillion dollar platform, and like you said the front line associates didn’t use it. As I always use the example, it’s like, “They’re out driving a truck. They don’t have a laptop. They’re not going to go download a report and change the columns or whatever.” It’s like the wrong mechanism, you didn’t fit it into their job; you try to get them to change their job to accommodate your tool. Is the awareness because they went through a failure or is it like, “We already know this isn’t going to work if we don’t get the UX right,” just because companies are a little bit more design aware these days? What drove that to change? Why is it now? Kathy: I think there’s two things. First of all, they’ve failed, they invested millions of dollars in some sort of decision-support sort of application that may have millions of dollars and data integration work that we’ve done to build it and purchased a lot of really advanced software that can help users slice and dice the data and dive in and understand it. But they really took more of a data-centric approach rather than user-centric approach. They really didn’t take that extra step to say, “How does this person go through their normal day in their decision making. Where do they need this information in that decision making, and how should it be best presented so that they don’t have to do any cognitive task switching, that it just fits into how they go about making the decisions.” I think those failures are one big driver. But then I also think as organizations move up the analytics maturity curve and move from BI reports to predictive analytics to prescriptive analytics. Those prescriptive analytics are going to be much more pervasive across the organization. I think it’s this analytics maturity that’s also driving this need to put more design thinking into this creation of analytic products. Brian: Do you have any specific examples of a company that may have like, “Version one was X, and we didn’t get what we wanted. Version Y or X.2—we then went back and tried to fit this in better with that employee, and we saw some kind of change of result.” Can you cite any examples of how design allowed the data to actually be insightful and create a meaningful change? Kathy: One was a project that I was involved in at a large property and casualty insurance company. We were trying to alert our franchise insurance agents if they had somebody in their book of business who was likely to not renew. It was based on really good data science and model scores with different, clear […] that showed significant likelihood to a trait between them. It was originally deployed in a separate application with some general groupings of how likely they were to leave. The utilization of it was more of a curiosity at that point. There were some adopters, but not as broad as it would be hoped to really be able to substantially move the needle on this. As the design was redone and integrated into their overall CRM system—was legit on their CRM system that came up when they logged in in the morning, they didn’t have to go look to see, it was there for them. As we moved from groupings and scores, into one star to five stars, five star is they’re likely to go–you really need to work those. So just very simple little changes that drove a significant change in the utilization of that capability. Brian: Do you know what the blockers were such that it took a redesign? If someone was listening to this and they’re like, “I’m that person right now. We don’t want to go through the learning experience that your company went through.” What do you do to prevent that? Like, “Here are some roadblocks to watch out for.” Kathy: The thing is, these products are usually developed by folks within a data science group. We have the data, and we know this is a business problem, and we’ve done the analytics. That is expected but not sufficient. Then the next step is to really use just basic research techniques from consumer companies. Go out and watch your user group that you want to utilize this data or this analytics to improve the performance. See how they do their job, what tools do they use, where would this information be most relevant, how can it be presented in the context of their general activities, where it’s not a separate thing, where it’s integrated into the stuff that shows up of their performance evaluation. That’s the way to really avoid some of the deployments that may have really great data and science behind them, but don’t get user adoption that’s needed. Just take that extra step and really understanding the user and where this information is going to be most relevant for them. Brian: Do you think the appetite from executives and people that are at the top of the reporting chain for these things support the time and the effort to go out and do that type of research or to try to fit it in and not so much focus on “When are we releasing code? Show me some progress.” They want to see a glee. “Just show me some proof that these millions of dollars is doing something,” versus this kind of squishy. It can be squishy—at least in my experience with certain executives, especially qualitative stuff. “How many people did you talk to?” It’s like, “Oh, we talked to eight so far.” It’s like, “We have 10,000 employees.” It’s hard for certain ones to understand the value of qualitative research in these things. Do you have any experience or thoughts about that? Kathy: It can be squishy, but I think really analytics especially being deployed pervasively is maybe not a project but more of a transformation program and you have to take the transformation program perspective to it which includes such squishy things that change management and business process redesigned. Somebody using those analytics within their decision-making process is really where the organization gets value from. That code that gets released and deployed, it is an interim step. But it is not the final step to that organization getting value, maybe you’re just a data scientist of the company, trying to deploy a great app that could help a group of marketing folks better invest marketing, or supply chain folks better manage costs to suppliers. It doesn’t have to be this big concerted professional effort. One way to do that in a very agile, low-cost way is to find a couple of folks that seem to really jazzed about getting this and use them as some beta testers. Maybe you start out with just the information on a spreadsheet and say, “Hey, does this make sense just from this information that you would use?” And then have them walk you through, “Where would you use this information? How would it be best presented to you?” And then think about working within the constraints that you have of maybe you can’t change the screen for this digital marketing […] that somebody uses, but how can you make that information as accessible as possible, where it really is more of a push of information at the right time, as opposed to pull of information that a human being has to remember to go get when they’re in the process of executing their “day job”. Brian: You hit on some great stuff there. I talk about this to my list frequently which is understanding tasks and workflows and people’s day-to-day jobs that the goal is to fit your solution into their existing behavior as much as possible. It’s really hard to change behavior. “Oh, I got to go remember not to load this other screen and pull out those number of page two and paste this into the other screen and then hit enter and then it does some analytics.” These are the kind of stuff why people don’t bother to do it. It’s like, “Well, my guess is good enough. I’ve been doing this for 20 years. I’m this good. Whatever. No one’s going to know if I did that or not. They don’t know where that number came from.” They don’t want to do that. First of all, it’s to understand their pain, and then another good technique there is if you’re going to improve a process and we want people to use this tool in order to realize this new value, first of all, understand that benchmark of what their existing workflow, their playbook is and then ask them to do the same playbook using the new platform. This is a great way to uncover the things that you don’t know to ask about necessarily because if you first get that model of how they do it now, it’ll help elucidate those gaps that we can’t see. I just read an article, there is an example like, “Let’s imagine designing a new hotel room.” and you might take for granted that that hotel room has a bathroom. […] is talking about this. You’d be really surprised if you walked in and there was no bathroom because no one stated that was a requirement, and because everyone just took it for granted. It’s those kinds of things that you won’t see as a data scientist or product manager, perhaps because you don’t know what these front-line workers are necessarily doing all day. But I love that you talked about getting into the minds of what people are doing and fitting your solution into them. One problem I see is that in some places, it’s really hard to get access to customers. I think sometimes, this is more on enterprise companies that are selling a product that has a data-specific part of it, it can be hard to access them. You would think companies doing internal analytics that this would be easier. Do you think that comes from leadership? Do think that comes from the ground up? Any comments on how do you get access to the right people? What if they’re not being like, “Hey, I’m a call center. I get paid hourly by the number of calls I do, and you want me to go work on your new software?” How do you incentivize that so that they become a design partner which ultimately, is really where you want to go is to have a team of partners from subject matter experts, and product managers, the data science people, whoever it is that is going to affect the solution that comes out. What needs to change in order to incentivize that participation? It takes time to get these things right. Kathy: Yeah, for sure. We see that a lot just because as you noted, the breath of the organization or the different incentives and priorities that other groups might have. I hear companies all the time, senior leaders, we are going to be an analytics competitor, we’re going to do analytics. They think if they hire 200 Ph.D., data scientists that they are now an analytics company. I do think you’ve got to have somebody at a leadership role at least over the data science group say, “Look, this company gets value from this when we are making better decisions because of this analytics.” And those better decisions are going to come from people accessing the analytics. When they’re in their decision-making process, and really working more leader-to-leader. But that can’t be where it stops. You’ve got to create a peer relationship that all levels, number one. And then number two, if you get the leadership of the other group engaged early on—as this is a problem that they feel they have ownership in and that they are a co-creator with you, and starts at the leader level, and then works its way down—then, I think you’re going to have greater access and more ability to do that. Finally, in the absence of that, if you don’t get it right, then I would at least, as you’re deploying the new app, say, “Our next step is to watch the staff members use it and identify how to better integrate it.” So that at least you’re showing some leadership in your thinking that, “I know this is going to be a problem because I didn’t have access to them. I’m already teeing it up.” We’re going to have to come back and see how people are using it and how we can utilize some user experience and design thinking in the subsequent phase after […] is reached. Brian: Sure, sure. I think some of my clients in the past, they have to go through a failure first in order to decide that they don’t want to do it. We don’t want to do the build-first, design-second, process on the next one. There’s a certain level of convincing that can happen, and then at some point, you got to move forward because typically, IT or the business, they have been tasked with, “Deploy this new model and this software into the survey,” and they’re going to do that no matter what. They’re not going to stop and wait for design if that organization doesn’t have a matured design practice. So you might have to go through that. I would say, obviously, it’s always cheaper to adjust things in pixels, and pencils than it is to adjust it in working code. The more you can get in front of these decisions, and then form the engineering and the data science prior to deploying a large application, it’s a lot cheaper, and it’s a lot easier, and you don’t have all the change costs associated with that, both time, money, labor. No one likes to do it twice, most people want to work on new problems, they don’t want to redo the same ones. Engineers usually don’t like doing this, so getting in front of it is good. Kathy: I think what you’re touching on there is, maybe if you haven’t done your user design work, then you think about this release as a beta release. You plan this release, and you plan this app, and you manage your code knowing it’s going to change at some point, and knowing it’s going to change at some point, and knowing it’s going to change soon. The second thing is, key point is to decouple the analytics insight from the application. There is an analytic insight, whether it’s a number, or a recommendation, or some score, or something, is it is just loosely coupled to the delivery mechanism than it is easier from a code engineering perspective to have to it delivered in a different way. Brian: Right. No, that’s great. A part of that is that the attitude and culture of change and not falling in love with our first versions of stuff. To me, that has to be ingrained in both the engineering–all of the teams that are touching the product that the service that’s being put out there. A lot of companies say they’re doing agile. A lot of them are skipping one of the most important parts which is getting some customer representation involved, and actually, iterating and not just doing incremental design where you keep adding more, “Add another feature, add another data point,” that’s incremental design, that’s not iterating and changing as you get feedback. That’s something to watch out for as well. At some point, you need to get some stuff out there, and it costs to have come down obviously, to deploy software. It’s overall cheaper to get something out there quickly and to start getting feedback of it, but it’s very easy to no get the feedback and to let the working code feel like success. Until someone at the top is like, “Where is all the bang we’re supposed to get for this?” Kathy: I think the approach there is, get it deployed, so that’s great; it’s out there, and it’s working. But again, build into your deployment plan, use your feedback, and change from there. If you are not fully deployed, until after you have received this feedback and implemented the needed changes within the application, then I think that’s another way to prevent that falling in love with your first design. You know that this is just deployed so that people can use it, and that getting user feedback and making those needed changes is an expected part of the deployment process and is not a failure that the application was not sufficient. Brian: I think that should be part of any software development process to have a loop of test design, refine, deploy in the circle. For the most part, for larger applications, you’re never really done with it. It’s a process of getting better and figuring out the ROI like, “Have we hit the market? Is it worth spending more time and money on this?” but yeah, those are great insights. I’m curious in terms of the roles, I feel like someone that’s at the top of the responsibility chain here needs to have a healthy dose of skepticism about their own stuff, especially when it comes to prescriptive and predictive analytics or any service where it’s custom software they’re deploying into the organization have a healthy dose of skepticism about how great it really is. Maybe you deployed on time, bug-free, you can see the stats and all of these. But is that responsibility primarily falling into the data science realm because companies are investing in that area right now and so they become the de facto, what we would call a product manager more in the SaaS world? Did they become that and is that the right place for that responsibility to be? Kathy: I see that happen really, not necessarily through design and intention. But just because if a data scientist wants to take this great science that he’s discovered and make it accessible, in a lot of organization they have to do it, and so it’s put upon them when they’re probably not well prepared for that. I do think that that’s the problem. And then as a lot of different tool and capabilities make it easier for data scientist to deploy an application, if there’s not a really good user design construct within whatever application they’re deploying, in their data science application, then they need to be the one to take the extra step. As I said earlier, often times, the data scientist sees the world through data and algorithms and prediction. They get enamored with the complexity of the model and the strength of its prediction and not so much with how easy it is for somebody to use it. I think that’s probably a reorientation with training that should happen within data science groups around design, best practices, and user experience design. They’re not going to evolve into great user experience designers, but they are going to be able to recognize something that’s really bad and perhaps ask for help and guide to make it better. Brian: Do you think that’ll stick then that that role will continue to live there? This person that needs to understand the business value that needs to be obtained, the user experience side of it and the technology side is trifecta there. Do you think that’ll stay in that data science world? Kathy: I don’t think so. I do think there will be a separation between the data application design and the creation of the data science that informs that […] in that application. I think as we talked about as some of these companies get more companies get more mature and need to have more pervasive deployment of data science insights, they are going to realize that they need to take a different approach. I often said that between data and analytics, I see the industry mature along the lines of software development. That design focus was not a big part of software development early on. I think it’s just going to have to happen within data science. Brian: I look at it this way, product owners can take many different titles. I have had all kinds of different clients, but ultimately, the box stops with everybody that’s working on it. But it helps to have someone that’s at that intersection of, “What do we need to do? What’s the overall picture of this?” and they understand the tech, the business, and the user experience side. Whatever the title of that person is, that role to me is really critical so that technology doesn’t run with everything. You have to have all three of those ingredients to deploy successfully, at least in my experience. It seems like that’s pretty critical to have that. Kathy: Yeah, for sure. A number of organizations that ask us about how do we show the value of our data science investments, how do we demonstrate our data science ROI. I think if more data science groups really looked at how their data science products were being consumed and started quantifying those, and then using some of the business metrics that are involved with that business process, whether it’s optimizing a spend or reducing average handle time at the call center, that would give them a great task to be able to validate their ROI to the organization, but I don’t see a lot of data science groups doing that. Brian: I’m curious, who ask that question? Is that the data scientist themselves or is it the business stakeholder who’s hiring the data scientist? What role asks that? Kathy: Usually, it’s a data science leader who has a large organization in large enterprise organizations. If you have a large organization with a lot of expensive resources, there is that continuing need to show the value that this organization brings to the overall company, especially when the output of that organization is not well understood or has not then a traditional part of that company. A lot of senior executives, C-Suite executives at large organizations, didn’t have data science when they were coming up through the ranks. This is something new that they don’t really understand, they know how much money they spend on it. A lot of data science organizations will need to demonstrate, “Here’s the hardline benefit that this company has gotten for investing in data science capabilities.” Brian: This blends nicely into my next question which is about obviously, AI, and machine learning are hot topics right now in technology. I hear this from people I talk to frequently which is, “Oh, the board knows we’re supposed to be doing some AI.” They’re asking me, “How many sensors do we have installed? Do we have digital transformation?” they ask these really high-end questions, and they want to go spend some money on it because they’re so afraid they’re going to miss the boat on that. From a design standpoint, we would say, “That smell, it reeks of possibly putting a cart before the horse.” The tool comes out, “We got to go buy this hammer because everyone else is buying this hammer. We have no idea what you hit with it, but we got to have it. We got to go spend some money on it.” How do you ensure that you don’t waste money? You want to invest in this. You don’t want to miss the boat. Maybe there’s potential for a project to deploy machine learning. It’s just pretty much what a lot of these companies are doing in terms of AI right now. How do you make sure that the desired investment from the business is actually going to have some ROI? They have heard this tool is hot. Kathy: It’s what I call the hype cycle mandate. Whatever is the new thing, it was a big data, it was AI and machine learning, with data sciences right, we have to have them, we have to tell the board we’re doing this. I think that is where executives earn their money, is being able to manage the message to the senior leaders who may not understand what’s needed and how you use it so that they can say, “Yes, we’re using it.” But the leadership than the executive leadership or the ones who has to go out and figure out where are the business problems and what is actually needed within our operating environment and our company to really deliver value from this capability. I will say, oftentimes, I see executives making the right call in that way. I have seen cases where folks have gone out and bought a lot of a software, and hardware and stuff that they have no idea of how they’re going to use, and that’s the shame. Brian: Do you think the right step there is to take on a small project, find a small win, show a small value and you can at least satisfy the, “Are we doing something?” “Yes, we’re doing some machine learning or whatever.” Do you think that’s the way it starts? Kathy: I am big just in any data […] investment, a big fan of used case-based development. Come up with a used case that requires this type of capability to execute it at the scale and precision that’s needed and then do some pilots to prove out the value, show the value, and then that then builds up a business case for the larger deployment that way. Yeah, I totally agree with that, Brian. Start with a used case, start small, understand, have an eye towards scaling as you start small, but get a small win, show that you’ve done it, the CEO can, in all honesty, say, “Yeah, we’re doing machine learning, and we’re going about it in a way that is physically sound but will also put us in a position to be able to compete with this capability in a quick amount of time.” Brian: That’s great. I think that I don’t know if I’ve said it, but this concept of falling in love with the problem and if you and your team, the people I work for you, or whoever it may be can fall in love with the problem and then weaponize your machine learning against that. That’s always a great thing, it’s to get everyone jazzed about the problem so that you know, especially if you can line it up with that technology not that the goal is to do that, but that’s when big wins happen to me, at least in my experience. This have been awesome. Do you have any advice, overall what will we talk about in terms of data product managers, data science managers, data science leaders, analytics leaders in terms of design, experience, what they should be looking for, going forward and just, in general, bringing more value to the customers. Is there a theme or something on your mind right now that needs to get mitigated to them? Kathy: My theme is a data science is about big complex data and a lot of technology and really advanced math, but they’re still human beings who have to use it. Don’t forget the humans. Focus in of how great this data set is that you created or how advanced this analytics technique that you’re using, but remember, the humans are your last line to realizing value from all of the stuff you’ve done before. Make sure you keep them in mind as you go through all of those other stuff as well. Brian: I think that’s great advice. And it sucks, those pesky humans. Kathy: I know, I know! They just get in the way of all those greatness! Brian: This is awesome. I have one last question for you and that’s have you ever surfed on a river? Kathy: I have not. Brian: I’m just curious. That came up in the webinar. I just saw this article on The Times about river surfing and I’m like, “I got to ask Kathy about this.” Kathy: Yeah. I’ve seen a lot of videos around that. I think it’s like this bore tide where the tidal action creates like this perfect wave that you can surf like forever. There’s some really good videos of folks doing that down in Brazil with some of the rivers that are draining out into the Amazon and other rivers that are draining out. It looks really awesome, but there’s this surf ranch in California that has this man-made wave. Kelly Slater, I think worked to create the engineering for this technology. That’s exactly what it looks like for those river-bore waves. But I’ve seen actually somebody in Munich surfing one of those […]. It’s on my bucket list, Brian. Brian: Sweet. I’m going to put a link to the surf camp, but where can we put some links to you? Where can people find you on the interwebs? Kathy: You can find me on LinkedIn, I am, Kathy Koontz. You can also find me at the International Institute for Analytics, it’s iianalytics.com, that’s our website. My LinkedIn name is customerjourneykoontz because that’s always been my passion is using data and analytics for customer journey. That’s where you can find me on LinkedIn. Brian: Cool. I will put those links in the show notes. This was super awesome. Thanks for coming and talking to me today. I’m sure people are going to enjoy listening to this. Thank you. Kathy: Thanks for having me, Brian. Have a good day. Brian: Alright, see you.

ceo amazon california learning conversations ai design phd practice executive director focus brazil fall in love falling in love roi engineers analytics saas crm quotes munich ux bi c suite deploy international institute kelly slater koontz tom davenport brian you brian how brian right kathy you kathy it

AiA 216: Building a Complete Web Application from Scratch Alone with Amir Tugendhaft

Adventures in Angular

Play Episode Listen Later Nov 20, 2018 42:56

Panel: Aaron Frost Brian Love Special Guest: Amir Tugendhaft In this episode, Aaron and Brian talk with Amir Tugendhaft who is a web developer who is located in Israel. He finds much gratification developing and building things from scratch. Check out today’s episode where Aaron, Brian, and Amir talk about just that. Other topics include UI Design, Flexbox, UX design, PrimeNG, and ag-Grid. Show Topics: 0:00 – Advertisement: AngularBootCamp.Com 0:52 – Host: Welcome! Today’s panel is myself, Brian, and our guest is Amir Tugendhaft! 1:13 – Guest: I am a developer and experience with Angular and React. 1:56 – Host: You spend your days/nights there? 2:03 – Panel: He is committed. 2:08 – Host: I am going to back up a second, and Brian could you please introduce yourself, please? 2:26 – Brian: I am the CETO at an Angular consulting firm (Denver, CO). We have the pleasure with working with Aaron from time-to-time. My Twitter handle is @brian_love – check it out! 2:52 – Host: What is CETO stand for? 2:59 – Brian answers the question. Brian: I oversee the crew among other things. 3:31 – Host: What do you want to talk about today, Amir? You are the guest of honor today! 3:40 – Guest. 4:00 – Host: That is a lot of information – that might be more than 1 episode. We have to stay focused! 4:14 – Host: I read one of your recent blogs about Cross Filled Violators. I met you through your blog before we did the Host: Give us your own ideas about starting your own app. 4:50 – Guest answers the question. 6:17 – Host: I am biased. But here is a fact. I used to work on a large team (60 people) and everyone committing to the same page app. We were using Angular.js 1.5, which I think they are still using that. I know that it worked but it wasn’t the easiest or fastest one to maintain, but it worked. 7:05 – Brian. 7:10 – Host: What are you trying to do? React doesn’t fulfill that need. I think you are being hyperballic and using extreme cases as the norm. Let’s be honest: we do cool stuff with jQuery plugins when we didn’t have a framework. When they say that the framework is stopping them then I say: I agree to disagree. 8:00 – Host: What do you think, Amir? 8:04 – Guest: I don’t have preferences. I try to build applications through the technologies and create components and simple applications. 8:30 – Brian. 8:33 – Guest: You create the component, and then... 9:21 – Brian: You don’t have to have a template file and another file – right? 9:35 – Guest. 9:48 – Host: I do in-line styles and in-line templates. One thing I learned from React is that I like my HTML, style and code. I like it being the same file as my component. I like that about that: I like single file components. This promotes getting frustrated if it gets too big. Yeah if it’s more than 500 lines than you have to simplify. That’s one of the things that l like. 10:47 – Brian: Modules versus... 10:55 – Guest. 11:07 – Host: I think in React and Vue you have the word module but in JavaScript you have a file that exports... 11:26 – Host: I have my opinion here and talking with Joe. He made a good point: at a certain level the frontend frameworks are the same. You could be doing different things but they basically do the same thing. 13:57 – Guest: Basically what that means is that the technology used it will do the same thing. Your patterns and practices are huge. 14:17 – Brian: If you are talking about the 3 popular frameworks out there – they are basically doing the same thing. I like Angular a little big more, though. Like you said, Aaron, people tend to pick the same one. I like the opinionated things about Angular. You get properties, components or called props or inputs you are getting a lot of the same features. It comes down to your personal preference. 15:31 – Host: What else Amir? 15:35 – Guest: Let’s talk about the UI. 16:05 – Brian. 16:08 – Guest asks a question. 16:25 – Brian: How have you tackled this problem? 16:34 – Guest: I kind of ran with it. If there wasn’t something that I liked I started from scratch, because it really didn’t feel right. 16:51 – Brian: I am an enemy of starting over type of thing. You have a lot of engineers who START projects, and they can say that they start this piece, but the experts and choice team members have what it takes to ship a feature. I mean fully ship it, not just 80%, but also the final 20%. I think it takes a lot of pose decision making to say I want to rewrite it but not right now. I still need to ship this code. I have always been a bigger fan as not rewriting as much as possible; however, if you started with good patterns then that’s true, but if you are starting off with bad patterns then maybe yes. I like that opinion b/c you have to start right. Brian: How do you do your CSS? 19:05 – Guest. 19:52 – Advertisement: Get A Coder Job! 20:30 – Brian: How do you make those decisions, Amir? 20:39 – Guest: I see something that I like and ask myself how do I apply this to my design and I start scaling things. 21:50 – Host: Are you using a tool like Sketch for your initial UI design? 22:05 – Guest. 22:54 – Host: I worked on a project where the client had a designer (UX). 24:00 – Host and Guest go back-and-forth. 24:51 – Host: I am sure it’s all about the quality from your designer, too. Hopefully it works well for you and it’s quality. 25:18 – Host: There is a lot to building an app from scratch. I am not a good designer. I am not a designer – I mean straight-up. I got nothing. I appreciate team members that can do that. 26:06 – Guest: Do you write...? 26:35 – Host: Only on the most recent project. The designer didn’t own the HTML CSS but he initially wrote it and then gave it to me and now I own it, and it’s in components. If he wants updates then I have to go and make changes b/c he doesn’t know Angular. If it’s a sketch or a PNG you have to make it look like that. That’s what most of my career has been. Host: HTML and CSS got me 762x easier once Flexbox came around! I know there is a decimal there! 28:23 – Host talks about Flexbox some more. 28:42 – Guest asks a question. 28:50 – Host: I suppose if I really had heavy needs for a table then I would try CSS grid could solve some problems. I might just use a styled table. 29:12 – Brian: ag-Grid or something else. 29:21 – Host: On this recent project...I’ve used in-house design and other things. If I ever needed a table it was there. I don’t rebuild components b/c that can get expensive for me. 30:50 – Brian: Accessibility. 31:00 – Host: Your upgrade just got 10x harder b/c you own the component loop. I really don’t build tables or drop-downs. Only way is if I really need to build it for a specific request. 31:30 – Brian. 31:58 – Host: Let me give you an example. You can think I am crazy, but a designer gave me a drop-down but he told me to use PrimeNG. I had the chose of building my own drop-down or the designer has to accept whatever they gave him. I made the UI make what he wanted and I made the drop-down zero capacity and then... Host: When you click on what you see you are clicking on the... Host: Does that make sense? 33:35 – Guest. 33:50 – Host. 34:25 – Brian: That is interesting; remember when... 34:58 – Host: We will send this episode to Jeremy – come on Jeremy! Any last ideas? Let’s move onto picks! 35:20 – Advertisement – Fresh Books! 30-day free trial! END – Advertisement – Cache Fly! Links: Vue jQuery Angular React C# What is a UX Design? UI Design Flexbox Sketch ag-Grid PrimeNG Brian Love’s Twitter Aaron Frost’s Medium Amir’s Medium Amir’s Twitter Amir’s GitHub Amir’s LinkedIn Amir’s Facebook Sponsors: Angular Boot Camp Fresh Books Get a Coder Job Course Cache Fly Picks: Aaron Movie: “A Star Is Born” Concept - Model Driven Forms Amir Puppeteer Arrow Function Converter Brian TV Series: “The 100” Angular Schematics

tv movies israel medium panel react scratch user ux grid ui github tv series sketch javascript html star is born originalsubdomain css google chrome png advertisement vue ux design angular puppeteers freshbooks jquery web applications ui design cachefly html css flexbox ceto brian you brian how aaron frost hoston host let brian love host you host what coder job course advertisement get a coder job host there angular boot camp angular schematics host does primeng

AiA 216: Building a Complete Web Application from Scratch Alone with Amir Tugendhaft

Play Episode Listen Later Nov 20, 2018 42:56

AiA 216: Building a Complete Web Application from Scratch Alone with Amir Tugendhaft

Devchat.tv Master Feed

Play Episode Listen Later Nov 20, 2018 42:56

Ep. 3: Dirty Rat (Brian House)

Phantom Power: Sounds about Sound

Play Episode Listen Later Apr 9, 2018 35:06

This time we talk with a fascinating sound artist and composer Mack met at a recent meeting of the Society for Literature, Science, and the Arts. As his website puts it, "Brian House is an artist who explores the interdependent rhythms of the body, technology, and the environment. His background in both computer science and noise music informs his research-based practice. Recent interests include AI, telegraphy, and urban rats." If that description looks a little daunting on the screen, the work itself sounds really cool to cris and Mack. We'll listen to three pieces of Brian's: a composition that imprints motion-tracking data on collectible vinyl, a field recording from the Okavango Delta in Botswana, and an encounter with the wildlife that put the "burrows" in New York's five boroughs.Links to works discussed: Quotidian Record (2012), Urban Intonation (2017).Mack notes that it was incredible to edit this episode using Daniel Fishkin's daxophone arrangement of John Cage's "Ryoanji" (1983).The other music on today's episode is by Brian House and Graeme Gibson. Transcript [♪ ethereal music playing ♪][CRIS CHEEK]This… is… Phantom Power.[FEMALE COMPUTERIZED VOICE]Episode 3.[CRIS]Dirty Rat.[unidentified sounds raising and lowering in pitch, banging noises][CRIS]So, what are we listening to here, Mack?[MACK HAGOOD]What do you think we’re listening to here, Cris?[noises continue, Mack laughing][CRIS]I don’t know, what is that? Is that an owl, put through a filtering device or something?[MACK, still laughing]You think it sounds like an owl put through a filtering device? Let’s listen to some more.[CRIS]Oh, wow. So synthetic.[MACK]It sounds like an old theatre organ having a bad day.[CRIS]Oh, yeah, no, I’m hearing that now. A pipe organ.[MACK]Yeah.[CRIS]Or something that hasn’t got a lot of wheeze left in it.[MACK]Something sad is happening in the silent film.[CRIS]Something very sad is happening.[MACK]Harold Lloyd fell off the clock.[both laughing][CRIS]And so he did.[MACK]Alright, so… it’s… it’s rats.[CRIS]That’s a rat?![MACK]That’s a rat.[clanging noises begin, rat noises stop][MACK]So today we’re gonna meet the guy behind the rat recordings that you just heard a moment ago: Brian House. He’s a composer and sound artist I met last November at the Conference for the Society of Literature, Science, and the Arts, which is this really crazy conference for interdisciplinary scholarship and creative experimentation. I met Brian, and when I heard about what he was working on, I just knew we had to have him on the show. His work uses sound to express relationships between bodies, human and nonhuman bodies, social relationships, geographic relationships, temporal relationships, and sonic relationships. So we’ll be hearing three different pieces of his: a musical composition that traces human, urban, and transatlantic movement, a field recording from the wetlands of Botswana, and an installation that will take us into the underground boroughs of New York City. This is work that helps us make sense of relationships we normally can’t sense at all.[BRIAN HOUSE]Well, my name is Brian House, and I’m an artist based right now up here in Providence, though I frequently do work down in New York. Yeah, I’m up here at Brown University at the moment, working on my PhD in music.[♪ upbeat technological music ♪][CRIS]So, Mack – how does Brian get interested in rats when he’s working on music?[MACK]Well, I think in order to get into that, we need to understand more of his previous work and some of the themes that are going on in it.[BRIAN]You know, I’ve been particularly interested in the ideas of Henri Lefebvre, right, who, in his last writings, outlined this poetic methodology called “Rhythm Analysis.”[MACK]Yeah, yeah, he was the French Marxist sociologist, spent a good amount of time thinking about life in the city, and –[CRIS]And the design of the urban environment, and –[BRIAN]And that’s been the basis for a lot of my rece...

new york new york city ai science phd society arts conference literature providence brown university botswana john cage okavango delta phantom power henri lefebvre brian you dirty rat graeme gibson brian house

Did Emma Deny Joseph Practiced Polygamy?

Gospel Tangents Podcast

Play Episode Listen Later Jul 6, 2017 14:53

[paypal-donation] The RLDS Church was founded on the claim that Joseph Smith never practiced polygamy. Even on her death bed, Emma Smith made the startling claim to some that Joseph never practiced polygamy. Conservative bloggers such as Rock Waterman (see his post here) and even Denver Snuffer have made the claim that Joseph Smith was a monogamist. However, current RLDS Church prophet/president Stephen Veazey has made the claim that historical records show that Joseph Smith did practice polygamy. What does Dr. Brian Hales believe about these allegations about monogamy and Joseph Smith? I have an article that I have been pitching to different people to publish on the denials. To be quite honest with you, there are at least two other denials that clearly were not senility, were clearly not specially phrased questions where Emma just comes right out and they're fairly well documented, said “Joseph did not practice polygamy.” How do I deal with that? I don't know. I do not have a good explanation. I think Emma was an amazing woman. I look forward to meeting her someday. You know in our theology that's a possibility. She had the worst row to hoe of all the polygamist wives in my view. She did remarkably well. She stumbled, but I believe there's plenty of forgiveness on these things for her, but I honestly don't know how she was able to say these things that she said. Hopefully these will be published. I've put them together, I just haven't got them published yet. It's pretty clear that she just denied it flat out. I don't know how she made that work for her, but they're there. I also asked Brian how he deals with the idea that Joseph practiced only monogamy. Brian: You know it's interesting, Rick, that you would bring that up because by the time this interview is aired, The Interpreter {a Mormon history journal} will have published a very long article that I have written where I don't just give evidence that Joseph did it, but I actually attack the arguments by, I mention Rock Waterman, I mention the Prices, and then Denver Snuffer who is in my view just the latest false prophet. Remember I started looking at Mormon fundamentalism and I documented and wrote articles on men or came as great leaders who came for this reason or that reason and then they just fall by the wayside. There's a number of these that I talk about in my book. I think Denver is just the latest in all of this type of a pattern. For those who have questions, the easiest way probably to detect Denver is he said Joseph didn't practice polygamy, and then just look at the evidence. I outline it all here. It is true that most of the evidence that Joseph practiced plural marriage is from late sources, but not all of it. The Nauvoo Expositor states right in there that Joseph Smith had a revelation. It was read to the high council. We've got testimonials from William Law and Jane Law saying that Joseph was teaching polygamy. These are contemporaneous. There are entries in William Clayton's journal. John C. Bennett clearly had heard some rumors. I don't know that he was ever in confidence of Joseph with it. But to say that it's all old is not true. There are some very important contemporaneous sources. What do you think of the arguments Brian makes against monogamy? (As a reminder, this is part 10 of our conversation with Brian. Check out parts 1-9 to learn the facts behind polygamy.) Look below for a video and transcript. https://youtu.be/BxFWMzde2TY https://gospeltangents.com/shop/transcripts/emma-denied-joseph-practiced-polygamy/

conservatives prices mormon polygamy interpreter joseph smith practiced emma smith william law brian you brian hales denver snuffer john c bennett rock waterman rlds church jane law

Polygamy Rumors–Declaration on Marriage

Gospel Tangents Podcast

Play Episode Listen Later Jun 15, 2017 10:23

In our third conversation with Dr. Brian Hales we'll talk about the earliest rumors of polygamy. In 1835, the Doctrine & Covenants published a Declaration on Marriage. We've got a couple questions about that. First of all, did Joseph Smith actually author that revelation or was it someone else like Oliver Cowdery? Brian: You know in my volume one, I go into this in some detail, and what we find is yes Joseph was gone, and yes they were quick to get this passed. In fact they called for this conference on Sunday to be done on Monday and there were almost no leaders there. The Quorum of the Twelve there were almost no one present, the High Council really weren't represented. The theory is that they were trying to get it pushed through before Joseph got back because Oliver had these reasons and stuff. I don't think so and I'm grateful for Michael Marquardt for helping me. We sat one afternoon down in his basement and we went through all the documents, and what we find is at that at point in time they needed to go forward with printing the Doctrine & Covenants. They had published we think most of the book up to section 101, which is the Article on Marriage and that they're having piles of all these papers around the printing office. So we think, or at least I think and I think Michael agrees that the driver at that point was really that they just wanted to get official approval so that they could finish publishing the Doctrine & Covenants and so I don't think that they were trying to do something backhanded with Joseph. When Joseph came back there's no evidence that he really disapproved of what had happened. In fact he quotes or refers to the Article on Marriage two or three times later when he is performing marriages. He said this declares our church's belief which they had to have in writing in order for the elders of the church to be authorized by the state to perform state recognized marriages, so there were a number of things.... GT interrupts: So whether Oliver may have authored it or not, it didn't seem like it bothered Joseph at all and he was fine with it. Is that safe to say? Brian: I believe so. I'd have to refresh my memory, but what we do know is he could have had it rescinded but he also quoted it as authoritative and Michael Marquardt pointed this out to me I think he's even published that somewhere that Joseph did consider it after the fact to be the official declaration of the church at that time. Second, was it a response to polygamy that could have been happening in the Kirtland community? What does Dr. Brian Hales think about these things? Brian: The first accusation against the Latter-day Saints, they weren't called that then, against the Mormons, the Mormonites, that they had embraced some alternate form of marriage, came in 1831. It was in conjunction with the Law of Consecration and it was basically not only do they share everything, they share wives. That was the accusation that came up. Of course it's easily refuted. There's nothing to support that it was even thought of or discussed. So when people say they were talking about polygamy in Kirtland, I would really like to see the data on that, that this was really a response to polygamy because my research shows that there was, with respect to Joseph and Fanny Alger, discussion of adultery and that was the claim that everybody was worried about. I don't find anybody discussing polygamy during that period. Don't forget to check out parts 1 and 2 of our interview with Dr. Brian Hales. When do you think polygamy started being practiced among the Mormons? https://youtu.be/UqoLrao3vs8 https://gospeltangents.com/shop/transcripts/polygamy-rumors-declaration-marriage/

marriage law saints covenant rumors twelve doctrine mormon declaration gt consecration polygamy joseph smith quorum kirtland high council oliver cowdery brian you brian hales fanny alger michael marquardt

Episode 43: Genghis Khan

The Revisionists

Play Episode Listen Later Apr 22, 2017 30:50

The hilarious Christie Buchele returns to the show as we discuss Genghis Khan! In this episode, Zach gives us a backdoor glimpse of his writing process, Christie drops her celebrity for the hot young Khan, and Brian tries out a new slogan for war crimes. *Apologies for the uneven audio quality of this episode. My software was acting up and I didn't catch it until it was too late. -Brian* You can help us get new equipment over on our Patreon page: https://patreon.com/therevisionists

apologies khan genghis khan brian you christie buchele

Basic Pronunciation Practice #26 + Interactive English.

A Cup Of English

Play Episode Listen Later Nov 12, 2013 10:37

Brian: "What are all these boxes doing out in the hall?" Liz: "I decided that it was finally time for me to get organized?" Brian: "You? Organized?" Liz: "I'm not that bad, am I?" Brian: "Well, I don't know anyone else who loses her keys everyday, or her phone, or her bag." Liz: "That's called being normal." Brian: "Well, I don't lose my things everyday." Liz: "That's because you're not normal, ha, ha! I've made a resolution: I will be more organized. I'll use my iPhone calendar. I'll get rid of my junk, and become more efficient." Brian: "Wow! God help us all! An efficient Liz is hard to imagine." Liz: "That's because overly organized people like you lack imagination." Questions or comments? Email me at acupofenglish@hotmail.com. Feel free to join me on my Facebook page called Anna Fromacupofenglish. You're all welcome. Please rate my app or buy it by clicking the link. // // //

god english practice iphone basic interactive organized pronunciation brian you brian well

ドラマで英語を学ぼう (15) Adventure in Australia - Part 5

australia adventure fish flare takako brian you

Play Episode Listen Later Aug 2, 2011

毎月第１週は「ドラマで英語を学ぼう」の新作「オーストラリアの冒険（Adventure in Australia）」をお届けしています。広島大学に留学していた、オーストラリア出身のHelen Needsさんが作成したオリジナルドラマです。３人の登場人物（Takako, Carmel, Brian）と一緒にオーストラリアを旅しながら、オーストラリアの英語や文化に触れてみましょう。いよいよ最終回となりました。BrianとCarmel、Takakoは、いよいよ目的地に到着します。Takakoは二人と空港で再会してから、今までその場所を知らされていませんでした。このお楽しみの目的地はどこなのでしょうか。そしてTakakoはこの場所を気に入るのでしょうか。また、Takakoが嫌がっていた新しい仲間、ヒッチハイカーのNathanの行方は・・・？（聞き取りのポイント）・TakakoとCarmelは何を忘れてきたことを残念がっていますか。・Nathanはdidge（オーストラリアの楽器）の演奏を皆に披露するつもりでしょうか。・解説を聞き、会話で使われている次の語句の意味を考えてみましょう。(1) I'll let you off. (2) Can't take the country out of the girl, eh? (3) What a bummer! 毎回、ドラマの中で使われている表現をJoeとKanaがていねいに解説します。今回お借りした素材写真：Wikipedia Download MP3 （19:51　11.5MB　中級） "Adventure in Australia" (C) 2011 Helen Needs and FLaRE.Adventure in Australia Written by Helen Needs (#5: At a campsite) Brian: What'd ya reckon? It was a good surprise, hey? Takako: Yeah, I'll let you off. I love this place! I can't wait to go swimming in the morning! Brian: I can't wait to see if the fish are biting. Fish for breakfast! Carmel: This fire's pretty good too. Good job, Takako. You've still got it. Can't take the country out of the girl, eh? Takako: You said it! I love a fire. What a bummer you forgot the marshmallows! Carmel: Oh, sorry. I'm disappointed too. Takako: It's OK. We'll survive. Brian: Hey, where's your friend? The hitchhiker. Takako: Thank goodness he took himself off to the other side. I thought for sure he'd want to hang around all night. Carmel: He seems alright. He's friendly. Takako: I could have sworn we'd be hearing some didge by now. Brian: You can put in a request… here he comes! Takako: What?… Oh dear. Carmel: Hiya, you get yourself set up alright? Have you eaten? We've got plenty of sausages left over if you're hungry. Nathan: Oh, thanks. That's sweet. I ate already. Brian: Where's your didge? Takako thought you might be coming over to serenade us. Nathan: I don't play. I'm just the carrier. I make strange squawky noises and spit a lot when I try to play. I can't get the breathing right. I came over to see if you like marshmallows! It's no fun toasting them by yourself, you know. Carmel: Really? Takako loves marshmallows! Takako: Oh, no…

ドラマで英語を学ぼう (15) Adventure in Australia - Part 5

australia adventure fish flare takako brian you

Play Episode Listen Later Aug 1, 2011

ドラマで英語を学ぼう (14) Adventure in Australia - Part 4

Play Episode Listen Later Jul 5, 2011

毎月第１週は「ドラマで英語を学ぼう」の新作「オーストラリアの冒険（Adventure in Australia）」をお届けしています。広島大学に留学していた、オーストラリア出身のHelen Needsさんが作成したオリジナルドラマです。３人の登場人物（Takako, Carmel, Brian）と一緒にオーストラリアを旅しながら、オーストラリアの英語や文化に触れてみましょう。第四回目は、Brian、Carmel、Takakoがヒッチハイカーと話をしている場面です。（聞き取りのポイント）・Brianによれば、彼らの目的地はどこですか。・Nathanは興奮したらどのような癖があると言っていますか。・解説を聞き、会話で使われている次の語句の意味を考えてみましょう。(1) It’s good to boost the testosterone levels in here. (2) Should I hold my breath? 毎回、ドラマの中で使われている表現をJoeとKanaがていねいに解説します。今回お借りした素材写真：Wikipedia Download MP3 （18:00　10.3MB　中級） "Adventure in Australia" (C) 2011 Helen Needs and FLaRE.Adventure in Australia Written by Helen Needs (#4 Picking up a hitchhiker) Carmel: (whispering) Takako. He’s cute! Brian: Hey, mate. Can we give you a lift? But we’re not going far. Takako: We’re not? Brian: You’re welcome to get in ‘til Washpool. Takako: Oh, so you’ll tell him, but not me? Lucky! It’s Washpool! Nathan: Washpool? Really? That’d be awesome. You must be angels. I don’t usually hitch but I missed the last bus. I thought I’d make it on foot but the sun’s coming down so fast. I didn’t think I’d get there in time to set up my tent. And this damn didge is so heavy, I’ll never know why I agreed to carry it home for a mate. I’m heading back to Sydney after this. I’m a teacher. I’m having some personal days and the agency hasn’t had anything much for me. I took the opportunity while it was waving a big red flag in front of my face and shouting ‘pick me!’. Brian: So…that’s a yes, then? Nathan: Oh yeah, sorry. I talk a lot when I’m excited. Sorry, I won’t annoy you, I hope. Carmel: So, hop in. Hi, I’m Carmel. This is Brian. And this is Takako. Takako: Hi. Carmel: She’s a teacher, too. And she’s on holiday from Sydney. Road rage is her problem. Nathan: It’s bad, isn’t it? Sometimes I wonder what I’m doing there. I was born around these parts. They call me back every now and then. More now than then. I can’t thank you enough for the lift. I was starting to lose hope. And, my god, how good is the river at the moment? Just magic. Wish I had a canoe. Carmel: Is that so? Nathan: Sorry, talking again. Brian: It’s fine mate. It’s good to boost the testosterone levels in here. The girls were about to really start picking on me. I’m their favourite pastime on road trips. Carmel: No, we never. Brian, you really should go and see someone about these paranoid delusions of yours. Brian: See? Hey, you’re quiet, Takako. Takako: Yes, I am. I’m thinking of something particularly cruel to say. Give me a minute. It’ll come to me. Brian: Should I hold my breath? (Written by Helen Needs)

australia adventure picking flare brian it takako brian you brian so nathan it

ドラマで英語を学ぼう (14) Adventure in Australia - Part 4