Podcast appearances and mentions of Charles Lowell

This week we talk to Charles Lowell, a developer and consultant who has created a library called Effection. Effection is a library that allows you to write structured concurrency code in JavaScript. What is structured concurrency and how could it be useful for you? Find out in this episode!https://frontside.com/effectionhttps://github.com/thefrontside/effectionhttps://frontside.com/effection/contrib/https://frontside.com/Become a paid subscriber our patreon, spotify, or apple podcasts for the ad-free episode.https://www.patreon.com/devtoolsfmhttps://podcasters.spotify.com/pod/show/devtoolsfm/subscribehttps://podcasts.apple.com/us/podcast/devtools-fm/id1566647758https://www.youtube.com/@devtoolsfm/membership

Coté talks with Charles Lowell and Taras Mankovski about the Backstage work they've been doing with clients. First, they talk about what Backstage is in the first place. Obviously, it's an internal developer portal...thing...but what does that mean beyond, like, a wiki? Check out The Frontside, and also, Taras' talk at BackstageCon. Watch the video of this recording if you prefer that kind of thing.

Coté talks with Charles Lowell and Taras Mankovski about the Backstage work they've been doing with clients. First, they talk about what Backstage is in the first place. Obviously, it's an internal developer portal...thing...but what does that mean beyond, like, a wiki? Check out The Frontside, and also, Taras' talk at BackstageCon. Watch the video of this recording if you prefer that kind of thing.

When it seems like everyone around you has worked in the same field for a really long time, making a career pivot with confidence can be tricky. But not everyone's been coding since their early college days like Robbie and Chuck. Kara Luton started on track to become a professional ballerina. After college and a stint in music publicity, burnout prompted Kara to make a hard left and begin a career in tech.  With all the developer bootcamps and online resources now available, making the switch has never been more accessible. Not to mention, the skills Kara learned as a ballerina and a music publicist helped shape the developer she is today. From staying dedicated and detail-oriented, learning to write and learning from burnout, Kara wouldn't change anything about her unconventional path to software. In this episode, Chuck and Robbie talk with Kara about her experience learning and relearning Ember, why she loves the Ember community, her advice for those looking to switch careers, Kara's cool home office, and why every developer has something valuable to offer.  Key Takeaways [00:58] - A brief introduction to Kara.  [03:16] - A whiskey review.  [08:51] - Kara's non-traditional path to tech.  [15:57] - Kara's experience in a bootcamp and her thoughts on bootcamps as a developer launchpad. [17:34] - How Kara found Ember.  [23:10] - Kara's advice for people looking to make a career pivot. [28:44] - Why Kara's looking forward to contributing to open source projects.  [32:30] - How Kara's home office setup has evolved.  [37:57] - Kara's thoughts on NFTs.  [40:17] - Why Kara loves animals and a deep dive on her two pet dogs.  [47:48] - More of Kara's hobbies outside of the web and a chat about Marvel movies.  [58:48] - A soccer and sports-themed whatnot.  Quotes [15:20] - “Ballet, it's very detail-oriented and I feel like that's something that's really helped me in my career as a developer, like missing a semicolon or understanding the different syntaxes — it's really helped me a lot. I'm really really grateful for my time doing ballet.” ~ Kara Luton [29:37] - “Contributing to the framework that you use will give you such good knowledge of it, even if it's something small.” ~ Kara Luton [31:59] - “You never know if something you say, the way you phrase something, will just make it click for somebody in a way that they haven't understood it before. I really really recommend people writing blog posts.” ~ Kara Luton Links Kara Luton CrowdStrike Glimmer.js Three Chord Bourbon Strange Collaboration Nelson's Green Brier Distillery  Nashville Predators Joffrey Ballet School Summer Intensives Belmont University freeCodeCamp Codecademy  Ember.js Ryan Tablada Rock & Roll with Ember.JS Ember Octane Dev.to Ed Faulkner CodeNewbie Jessica Rose Alex Graul National Geographic Dependabot  Ember Learning Team The Ember Times  The Difference Between Regular Functions and Arrow Functions Introduction to CSS Grid: What You Should Know Kara's work from home setup Yeti Blue Microphones Amazon Quartet Whiteboards Lume Cube The Frame TV Damien Hirst's NFT Bored Ape Yacht Club Web3 is going just great Taras Mankovski  Whiskey Web and Whatnot: Alternatives to Relay, the GraphQL Stack, and Adulthood with Charles Lowell and Taras Mankovski Connect with our hosts Robbie Wagner Chuck Carpenter Ship Shape Subscribe and stay in touch Apple Podcasts Spotify Google Podcasts Whiskey Web and Whatnot Top-Tier, Full-Stack Software Consultants This show is brought to you by Ship Shape. Ship Shape's software consultants solve complex software and app development problems with top-tier coding expertise, superior service, and speed. In a sea of choices, our senior-level development crew rises above the rest by delivering the best solutions for fintech, cybersecurity, and other fast-growing industries. Check us out at shipshape.io.

Just because something is widely used doesn't always mean it's your best solution. Frontside Founder Charles Lowell and CEO Taras Mankovski, stumbled into an alt GraphQL stack simply because the nature of a product didn't mesh with Apollo. After happening upon two up-and-coming technologies, GraphQL modules and Envelope, a solution was born, as was a newfound flexibility with GraphQL stacks. In this episode, Charles and Taras talk with Chuck and Robbie about their accidental developer discovery, the drawbacks of UI libraries, what a Relay alternative looks like, what in the world Pact is, and why adulthood is vastly overrated.  Key Takeaways [00:48] - An introduction to the Frontside guys. [02:29] - A whiskey review. [08:46] - How Charles and Taras discovered a less-than-ordinary GraphQL stack. [18:39] - Why JSON:API doesn't always make sense. [23:11] - Taras' criteria for a valuable alternative to Relay.  [25:04] - What is Pact?  [28:30] - An NFT chat, and why adulthood is vastly overrated.  [41:45] - Charles' and Taras' hobbies outside of the web and the best way to bond with your baby.  [54:38] - A few last-minute mentions.  Quotes [21:04] - “Relay is complex, it's difficult, and it's not as magical as other things that I've used. So I actually don't think that the primary benefit is to the clients that consume it, ironically. I think the benefit is to the developers that are trying to understand.” ~ Charles Lowell [56:20] - “The combination of testing and simulation and the developer experience stuff, and the emergence of developer experience as an area of focus is exciting and interesting in the same way that web and Ember was when it started. Just that sense of, we're discovering something new and there are people who are actively trying to solve a problem.” ~ Taras Mankovski  Links Charles on Twitter Taras on Twitter Frontside  The Balvenie Doublewood 12  The Singleton of Glendullan Liberty GraphQL Apollo  Discord Envelope JSON:API runspired Whiskey Web and Whatnot: Discovering Ember, Adopting Orbit, and Unlocking Optimization with Chris Thoburn (runspired) Ember Data Orbit  Relay Pact Swach Blockchain Web3 No JS Rails The Guild Hive GraphQL CodeGen The Sandbox Snoop Dog Second Life Linden Lab  Duolingo Casamigos Deno GoJS Backstage Connect with our hosts Robbie Wagner Chuck Carpenter Ship Shape Subscribe and stay in touch Apple Podcasts Spotify Google Podcasts Whiskey Web and Whatnot Top-Tier, Full-Stack Software Consultants This show is brought to you by Ship Shape. Ship Shape's software consultants solve complex software and app development problems with top-tier coding expertise, superior service, and speed. In a sea of choices, our senior-level development crew rises above the rest by delivering the best solutions for fintech, cybersecurity, and other fast-growing industries. Check us out at shipshape.io.

"I don't care about networking...and load balancing." There's a lot of new concepts and stuff to learn when it comes to developing applications that will run on kubernetes. In this episode, Coté talks with Charles Lowell (https://twitter.com/cowboyd) about his experience. Also, we imagine measuring the humidity of mayonnaise. If you need some excellent app coding, check out Charle's company, Frontside (https://frontside.io/)! They also have a podcast (https://frontside.io/podcast) where they discuss recent programming frameworks and idea, and relating coding cool stuff. You may recall that Charles was the co-host of Coté's first podcast empire, DrunkAndRetired.com (https://archive.org/details/DrunkAndRetired). Special Guest: Charles Lowell.

Philippe De Ryck joins the show to talk all things security: the importance and why you should be taking active steps, how to do it in your codebase effectively, and what can happen during a breach. Philippe De Ryck: Pragmatic Web Security Resources: OWASP Top 10 OWASP Top 10 Proactive Controls Please join us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at contact@frontside.io. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer. This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. Transcript: CHARLES: Hello and welcome to The Frontside Podcast, a place where we talk about user interfaces and everything that you need to know to build them right. My name is Charles Lowell, a developer here at The Frontside. Joining me, also hosting today is Taras Mankovsky. Hello, Taras. TARAS: Hello, hello. CHARLES: And as always, we're going to be talking about web platforms, UI platforms, and the practices that go into them. And here to talk with us today about a pillar of the platform that I certainly don't know that much about, and so, I'm actually really happy to have this guest on to talk about it is Philippe De Ryck who owns his own company called Pragmatic Web Security. I understand you do trainings and are just generally involved in the small space. So, welcome, Philippe. PHILIPPE: Hi. Nice to meet you. CHARLES: Wow! I almost even don't even know where to start with this subject because I'm kind of like the hippie developer mindset where it's like, "LaÖlaÖlaÖlaÖlaÖ we're in this open land and nothing's ever bad going to happen and we're just going to put code out there," and nobody would ever take advantage of any holes or anything like that. And I think that a lot of developers share that mentality and that's how we end up with major, major security breaches. And so, like I said, this is something that I'm actually very eager to learn but I almost even don't know where to start. I need training, man. [Laughter] PHILIPPE: Well, that's good to hear. No, you're totally right about that. If you're not into security, it seems like this fast space for a lot is happening and you don't really know how or why and what really matters to you and should I even be worried about this. And let me start by addressing the very first thing. Yes, you should be worried because maybe you're not building something that somebody cares about but you always have something that somebody wants, even the simplest of attacks always targets a valuable resource. Just to give you a very simple idea today, cryptocurrency is all the hype and you have a taker that's just aiming to misuse your users' computers to mine crypto coins because it essentially saves them a bunch on electricity cost. So, there's always something to grab. Usually, it's data or services or worse. But even in the most minimal cases, you have hardware, you have devices, you have network capacity that somebody might want to abuse. So yes, security, I would say, always matters. CHARLES: What's the best way to get started? You said understanding that everything we do, we're holding onto resources that might be valuable, that someone might want to seize but I'm just getting started with my application. I don't know anything about security. Where do I get started on just understanding the space? And then before I even look at tools that I wantÖ PHILIPPE: You want the honest answer for that? [Laughter] PHILIPPE: The honest answer is probably hire someone who has security knowledge. I don't mean this in a bad way. I've come a very long way in my career doing what I do now. And if I look at that, if you are aiming as a developer with no knowledge about security to build a secure application, it's going to be very hard. There's a lot of things you need to know, intrinsic knowledge. These are not things you can simply read a small book in a week, you know all of these security things that you'll know what to do. So, if you have no previous experience at all, I suggest to find some help. CHARLES: Right. It's like saying, "Hey, you've never written a data layer before but you want to go out and you want to write a massively distributed system where you have all these notes talking to each other. You're not going to read the O'Reilly book 'How to Build Distributed Systems' in a week and go out and do the same thing." It's the same thing with security. You need to understand the entire context. And there's no substitute for experience. PHILIPPE: Sorry, I actually like that comparison because in a sense, you're right, it's like these other very complex topics you don't expect to learn that in a week or a month and right a functioning data layer. But the difference is if you fail at writing that data layer, your application is probably not going to work. While if you fail at securing the application or seeing potential vulnerabilities, it's still going to work just a bit more than you anticipated. It's going to result leaking all your data to an attacker or opening all doors so that they can gain access to your server, stuff like that. So, I would say that the consequences of not getting it right are, at least in the beginning, very invisible. It's only after things happened that it's like, "Oh, crap!" And you should pay attention to that. CHARLES: Yeah. And then you have these back doors and these leaks that are set in stone and may be very hard to change. PHILIPPE: Yeah, absolutely. And honestly, the worst part of the breach is for a company, it might be reputation damage. But what you really should be worried about is all that personal information that's being leaked to, most cases, publicly leaked to anyone but in other cases, it's sold on [inaudible] markets and actually abused by people looking for someone else's identity or credit card information or stuff like that. And the companies usually get away with some bad press, maybe a small dip in their stock price but eventually, they'll bounce back. But it's the users that suffer from these breaches for a very, very long time. TARAS: What do you see the kind of hot zones around concerns that companies have around security? Because I imagine it's hard to be concerned about everything, so they're probably thinking about specific things like this thing worries us. Like what kind of things do you see companies and teams need to worry about? PHILIPPE: That's an interesting question. You have all different kinds of companies and all different levels of security awareness and what they're worrying about. I would say if you have the companies that are not very good at or don't have very much security knowledge, they're probably not to worry about things because otherwise, they would have started investing in improving their practices. If you look at the companies that are at least very aware of the landscape, I'm not saying that anybody here is perfect, but some of the companies are actually doing quite a good job. One of the most interesting challenges today is dealing with dependencies. So, all of your packages, you depend on npm, Maven, Python packages, Ruby gems, and so on. All of them form a huge attack factor in most applications today. That's definitely a problem that a lot of companies struggle with and it's very hard to find good solutions. TARAS: GitHub recently, I saw their vulnerability alert service that I've been getting a lot of notifications from on some of the open source libraries that we use. They have a lot of dependencies. And a lot of projects have the same dependencies. So, the moment that one notification goes on, it like lights up on all of the GitHub repos that I have. So, I have been going through and like updating dependencies and all those libraries. PHILIPPE: Yeah, that's a very good example. Absolutely. A lot of the projects we build today usually start out by installing a bunch of dependencies. Even before you've written the first line of code, you already have a massive code base that you are relying upon. And a single vulnerability in a code base might be enough, it's not always the case, but it might be enough to compromise your application. And that leaves you, as a developer, in a very hard place because you haven't written any lines of code yet. You have built a vulnerable application and that starting point can be very terrifying. So, there's a lot of reports on this. And actually, if you want some numbers, 78% of the vulnerabilities discovered in existing applications like the ones you mentioned. If GitHub alerts you like, "Hey, there's a problem in one of your dependencies," it's often even an indirect dependency, meaning that you include a framework, if you include React or Express or whatever you're building, that one of your dependencies of one of those projects actually has a vulnerability. If you look at the trees of these packages, they get quite big that it's not dozens but it's thousands of packages that we're talking about. CHARLES: Yeah that's the other thing is how do you know how to interpret these security vulnerabilities because some of them, we get a lot of security vulnerabilities for node packages but we only use them for our development tools to build frontend. So, if we're building a React application and there's some security vulnerability in some node packages that we're using in our build tool, then that doesn't actually get deployed to the frontend. So, maybe it's not a concern but if we were actually using it to build a server, then it would be absolutely critical. And so, how do you evaluate because the same security vulnerability is not a vulnerability in one context, but might be in another or maybe I'm thinking about it wrong. You see what I mean? PHILIPPE: Yeah, sure. I totally get what you mean. Actually, I have observed the same things. I also get these security alerts on my projects, and sometimes it's devDependency, so it seems like we don't need to care about that. You're right in the sense that you have to assess the criticality of such a report. So, they will have a rating, a severity rating saying like, "This is a minor issue," or, "This is a major issue," that should be a first indication. And then a second thing to look at is, of course, how are these things used in practice. It's not because it's a devDependency that it's not exploitable because it all depends on what is the vulnerability. If there's an intentional malicious backdoor in the library and you're building that on your build server, it might give an attacker access to your build server. So, that might not be something you actually want to do. So in that case, it does matter. Of course, if it's only stuff you run locally, you can say like, "OK, this is less important." But usually, updating or fixing these vulnerabilities also requires less effort because there's no building and deploying to production servers either. So, it's a matter of staying up-to-date with these. And one of the things that people struggle with is handling this in a lot of different applications. You mentioned you had a lot of GitHub repos and the vulnerability starts popping up in all of them and you have to fix and update all of them. You can imagine that major companies struggle with that, as well, especially if you have quite a few different technologies. Managing all of that is insanely hard. CHARLES: Right, because you just usually look at it and you're like, "Oh, I've got to download this." And maybe, "I haven't used it this repo for a while. I've got to clone it up, I've got to update the dependency. I've got to make sure I run all my tests locally, then run all the tests in CI and make sure I didn't break anything by upgrading. I might have fixed closed security hole but broken my functionality." And so, make sure that that is all intact and then push it out to production. Even on the small, it's like I'm looking, "OK, maybe this is going to take me 30 to 45 minutes." But if you have four or five of those things, you're looking at half your day or maybe even the whole day being gone and that's if you have the processes in place to do those automated verification. If you have a very high confidence in your deployment pipeline which I don't think a lot of places have. So, it sounds like these are complementary, like you really need in order to keep a secure application, you have to keep it up-to-date because I think what I'm hearing is you should just evaluate all the threats. You should fix it if you can. The first part of my question is, am I kidding myself when I say, "Oh, I can ignore this one because it's just local or it's just a devDependency." PHILIPPE: The answer to that question is briefly, I would say they are less critical. CHARLES: That's cool. PHILIPPE: In general, the rule is update if you can. And actually some of the tools out there that monitor vulnerabilities, they will automatically create a pull request in your repo saying to upgrade to this version and then you can automatically run your tests if you have them, and you can very quickly see whether some conflicts are generated by updating that dependency - yes or no. And in most cases, if it's a minor version bump, it's going to work as expected and you can easily push out the new version without that vulnerability. So, I would say fix if you can. If it goes quickly, then definitely fix them. But I would focus on non-devDependencies first instead of devDependencies. CHARLES: Yeah. PHILIPPE: Second thing I wanted to add is you paint a very grim picture saying you have to spend a lot of time updating these issues and I can totally understand that happening the very first time you look into this. There's going to be some stuff in there, I can guarantee that. But if you do this regularly, the effort becomes less and less because once you have up-to-date libraries, the problem is bad but it's not like we have 50 new vulnerabilities every day, fortunately. CHARLES: Right. PHILIPPE: So, once you have done that, it's going to be a bit less intensive than you might anticipate at first glance. Of course, if you're using these projects, if you're reusing the same library, then you'll have to update them everywhere. That's the downside, of course. CHARLES: It's probably a little bit dangerous to be assessing the criticality of the security threats yourself if you're not an expert, and kind of in the same way, it's dangerous to be assessing an architecture if you don't have an expertise in our architecture, I guess is the thing, because you might not understand the threat. PHILIPPE: Yeah, that's, again, absolutely true. It again depends very much on how it's deployed and what it's used for. That's going to be one important aspect. Another thing that might be very useful is, how deep is the dependency that creates the vulnerability or has the vulnerability? Because for example, if you have your tree of dependencies, if you dependency is like five or six levels deep, the chances of malicious data are reaching that specific vulnerability, and that specific library is going to be fairly small. Because usually, libraries have a lot of features and you only use part of them in your application. So, the other one is address of the features is just sitting there and if it's never used and it's also not exploitable. So, that might play a role as well. I saw a presentation about a month or two months ago from how Uber manages these things and they struggled with a lot of those things as well. And they eventually decided that they really care about vulnerabilities going three levels deep. And something that goes deeper is considered to be less relevant or less urgent to update because chances of exploitability are going to be very small. CHARLES: That's actually really interesting. TARAS: One thing that got me thinking about something that is actually happening right now. A friend of mine has a WordPress site that was hacked. But what's interesting about WordPress, I think the fact that WordPress site was hacked is not really a surprise but I think what's interesting about that is that the frequency and the sophistication of these attacks has increased. The tooling has improved also in the WordPress ecosystem. But at the same time, I think there is actually more people that are aware of the kind of exploits that could be done. There are a lot of people going after WordPress sites, but it kind of makes me think that there's probably going to be a time when the vectors of attack for web applications are going to become pretty well known as well. Because of the architecture, there are a fewer of them. But as the awareness of the actual architecture becomes more common, I think the angles of attack are going to become more interesting. Like one of the things that I was reading about a couple days ago is that there are some researchers that found a way to attract users based on a combination of JavaScript APIs that are available in the browser. So, they are actually able to fingerprint users based on the kind of things that they're using, the application for [inaudible] extensions they have installed. I think people are going to get more creative. And that's kind of scary because we've seen this happen already in WordPress and people are greedy. So, there are going to be ways. I think there's going to be more people looking at how to get into and how to exploit these vulnerabilities. PHILIPPE: Yeah. That's actually a couple of very good examples that illustrate the underlying issue. So, this browser-based tracking of users, it's called browser fingerprinting and it's been going on for a while. Back when I did my PhD, I had colleagues working on those things and you have other people at universities doing research on this. And yes, you can use things like JavaScript APIs in the browser to identify a particular user with a very high probability. It's not perfect but it's usually enough to identify a user for ad tracking or those purposes. By the way, these things also have a legitimate purpose. So, they are also used to keep track of a particular user to prevent things like session hijacking or detect problem logins or stuff like that, so they can also have a legitimate use case next to tracking. But they very clearly show how security will always be a cat and mouse game. Tracking used to be easy. You just set a cookie in a browser and the cookie was there next time and you knew who the user was. And then, users became a bit more savvy. You had browser extensions trying to block listings because let's be honest, they're kind of shady. So, users probably don't want that. And then the attacker started moving towards other things and getting more advanced. And you see that in other areas of security, as well. So, I consider that a good thing because as we make things harder for attackers, they will have to get more creative and it will become more difficult to exploit or to take advantage of applications. That's the good side. The bad side or the dark side of that equation is that unfortunately, the vulnerabilities are not going away. It's not because we now have these somewhat more advanced attacks using advanced features or even CView-based vulnerabilities that the old things like SQL injection and [inaudible] have disappeared in applications. That's also not true and that means that it makes it just a bit more harder for everyone on the defensive side to build more secure applications. You're going to have to know about the old stuff and you have to learn about the new stuff. CHARLES: Again, we come back to that idea. It's all a bit overwhelming. Aside from the solution of like, "Hey, let's hire Phillippe. Let's hire some other security expert." We were actually in your training, and obviously, I don't want to divulge all the secrets or whatever. If we were to attend your training, what do you see is the most important thing for people to know? PHILIPPE: There's no secrets there. [Chuckles] What I teach is web security. I kind of like to think I teach that in a very structured and methodical way. But in the end, there's no secrets and I don't mind talking about this here on the podcast because I honestly believe that everyone should know as much as they can about security. What do I teach? I can talk about specifics but I can also talk about generic things. One of the general takeaways is that one of the best things in my opinion that a developer can do is realize when they don't know something and actually admit that they don't know something, instead of just doing something. Maybe having like a brief thought like, "Hmm, is this secure? Well, it's probably good. I'm going to deploy it anyway. We'll see what happens." That is not the right way of doing things. If you do something and you recognize like, "Hey, this might be security sensitive. We're dealing with customer information here. We're dealing with healthcare information. We might want to look at what plays a role here," and then you can go ask someone who does. You probably have a colleague with a bit more security knowledge, so you can ask him like, "Hey Jim, or whatever your name is, do you think that this is OK or should we do something special here?" Very much like you are doing, asking me questions right here. That's one important takeaway that I hope everyone leaves with after a training class because not knowing something and realizing that you don't know it allows you to find someone who actually does. That still leaves us with that point which you wanted to sidestep. CHARLES: [Chuckles] PHILIPPE: A second thing is to realize that security is not a target. It's not something you're going to hit. It's not a holy goal that after working really hard for two years, you're going to hit this security milestone and you're done. It's always going to be a cat and mouse game. It's always going to be a moving target but that's OK. That's how things are. And that's the same with all other things in the world essentially. It's an evolving topic and you'll need to be ready to evolve with that as well. TARAS: One of the challenges that I see into quite often in teams is that at the individual level, people really try to do their best, maybe the best of their abilities. But it's often, when it comes to being part of a group, it's often like they do best within the kind of cultural environment that exists. I'm curious if you've seen good or kind of environments or cultures for engineering teams that are conducive to good security. Are there kind of systems or processes the companies put in place that you've seen to be very effective in preventing problems? Have you encountered anything like this? PHILIPPE: Ideally, you have developers that are very well educated about security but honestly, it's going to be insanely hard to find these people because a developer not only has to be educated about security, they also need to know about UI design and JavaScript frameworks and other frameworks and all of these things. And it's virtually impossible to find someone up-to-date on all of these things. So, what most companies do today that seems to work quite well, even though it's very hard to judge whether it's working or not, is they work with security champions. So, you typically have a dev team and within a dev team, you would have a security champion, one or two or five, depends on how large your teams are, of course, that is knowledgeable about security. So, that developer has some knowledge. He's not an expert but he knows about common attacks and common dangers and how to potentially address them in the application. So, having that person embedded in the team allows the team to be security aware because when you have a team meeting like, "Hey, how are we going to solve this particular problem?" That person will be able to inject security knowledge like, "Hey, that seems like a good idea but if we're using SQL in the backend, we need to ensure that we don't suffer from SQL injection." Or if you're using a NoSQL database, it's going to be NoSQL injection and so on. And that already elevates the level of security in the team. And then, of course, security champions themselves are not going to be security experts. They're mainly developers just with a security focus. So, they should be able to escalate problems up to people with more knowledge, like a security team which can be a small security team within their organization that people can easily reach out to, to ask like, "Hey, we're doing something here and I know that this is security relevant and I'm not entirely sure what's happening here. So, can we get a review of this part of your application?" Or, "Can you guys sit on the meeting to see what's going on and what's happening there?" And I think that structure also makes sense. It's still going to be hard to build secure applications because there's still a lot of things to address, but at least, your teams get some awareness. And then of course, you can help your security champions to become better and they will get better over time. You can augment them with the security architects. You can train your security champions separately with more in-depth knowledge and so on. And that veteran or that setup seems to work quite well in many large organizations today. CHARLES: Yeah. I like that. It gets me to thinking, so having the having the security champions, having people who have this as part of, not their specialization, but at least part of their focus, being in the room, being part of the conversation because we try and do that and provide that service when it comes to UI but we also have a bunch of processes that kind of automate the awareness of quality. So, the classic one is your CI pipeline, your deployment pipeline. So, you're automating your advancement to production. You're automating your QA. It's still no substitute for having someone who's thinking about how to have that quality outcome but you still have some way of verifying that the outcome is quality. Are there tools out there that you can do to kind of keep your project on the security Rails. I'm thinking something that we we've done recently is having preview apps, so that we get a tight feedback loop of being able to deploy a preview version of your application that's on a branch but it's talking to a real backend. There's a lot of more software and services that are supporting this and it's kind of become an integral part of our workflow. So, testing automated deployment preview apps, there's this kind of suite of tools to make sure that the feedback loops are tight and that the quality is verified even though you have people, you also have people guiding that quality. It's just making sure that the standards are met. Is there a similar set of tools and processes in the security space so that we've got these champions out there, they're being part of the conversations. They're making suggestions but they can't be everywhere at once. And is there a way to make sure that the kind of the ways that they're guiding the application, just verifying that the application is going in that direction? Or an alarm bell has sounded. We mentioned one which is the automated pull request with the, "Hey, you got this dependency and there was a pull request." Are there more things like that, I guess, is what I'm saying. PHILIPPE: Yes, there are. But I would dare to say not enough. So yes, you have some security tools you can integrate in your pipeline that do some automated scanning and they tried to find certain issues and alert you of those issues. So, these things do exist but they have their limitations. A tool can scan an application. Some of the findings are going to be easy and fairly trivial, but it's good to have the check in place nonetheless. But some of the more advanced issues are very likely to be undetectable by those automated tools because they require a large amount of skill and expertise to actually craft and exploit to abuse that particular feature in an application. So, we do have some limitations but I like discretion because I do believe that we need to leverage these mechanisms to ensure that we can improve the security quality of our applications. A very simple thing you can do is you can run an automated dependency check when you build the application and you can use that to decide to halt deployment when it's a severe vulnerability or go ahead anyway when you consider this to be acceptable because if you automate all of those things, things can go wrong as well. We can talk about that in a second. So yeah, these things can be done. But what I strongly encourage people to do to ensure that they can kind of improve the code quality is to flag certain known bad code patterns. So if you're building an Angular or a React application, if you're using functions that output go directly into the template, that's going to be very dangerous. So, we know these functions in Angular, they're called bypassSecurityTrustHtml, bypass security should be kind of a trigger and this kind of security irrelevant. And in React, that property is called Dangerously Set innerHTML, also indicating like a 'developer watch out what you're doing'. So, what you could do is you could set up code scanning tools that actually flag these things whenever they appear in application because sometimes people make mistakes. You hire an intern and they don't really know the impact of using that property and they use it anyway which would cause cross-site scripting vulnerability. If you're code scanning to flag these things ensures that it doesn't get pushed to production unless it's a benign case which is actually approved to be in there, then you can definitely stop some of these attacks coming on for sure or some of these vulnerabilities happening. TARAS: I think the hardest thing to understand is when someone doesn't understand what they're doing that what they will create is so cryptic that I think any tool that tries to figure out what it is that person is doing I think will have a really hard time. The person making the thing doesn't understand what they're doing, then the system is not going to understand what they're doing which makes me think that one of the things that we think about a lot at Frontside is this idea of trying to understand the system from the outside as kind of looking at a system as a black box and wonder what kind of tools are available specifically for inspecting the application from the outside, like as if somehow understanding what the application is doing based on what's actually going on inside of the runtime and then notifying someone that there could be something off in the application, but through exercising the [inaudible] things like, for example, memory leaks is not something you can catch unless you have a test suite that has like a thousand tests and then you will see over time that your application is actually leaking memory. But if you run individual tests, you'll never see that. I wonder if there's anything like that for security where at runtime, there's actually a way to understand that there might be some kind of a pattern that's incorrect in the application. PHILIPPE: If only, if only. It depends on who you ask. There is such a concept that's called Dynamic Application Security Testing. Essentially, what you do there is you run the application, you feed it all kinds of inputs, and you monitor when something bad happens. And that means that you have detected vulnerability. So, these things do exist. But unfortunately, their efficiency is not always that good. It very much depends on what kind of security problems you're trying to detect. And they can, for example, detect some instances of things like cross-site scripting or SQL injection or things like that. But there will always be limitations. I've seen tools like that being run as an application where you actually know there's a vulnerability because it has been exploited. There is a manual written exploits and the tool still doesn't find any vulnerabilities which is not surprising, because these things are really hard to make an abstraction of to be able to find that in an automated way with a tool. If you would have such a tool that would be, I think, that [inaudible] would be a lot better. I think there's a lot of funders working on that. But at the moment, those tools are not going to be our savior to build more secure applications. CHARLES: Yes. I mean, it's kind of like linting, right? Or you can make tests. We've been through this kind of all the features or the aspects that we want our application to have, whether it be accessibility. There's certainly a very comprehensive suite of lint level checks that you can run to make sure that your application is accessible. You can run a suite of three thousand things and if it triggers any of these things, then yes, your application won't be accessible but it's not a substitute for thinking through the accessibility architecture. The same thing goes with code linting. You're not going to solve bugs with a linter that makes sure that it's formatted and that you're declaring your variables right and that you're not shadowing things. But you can definitely eliminate a whole classes of things that might be put in there just for maybe even you know what you're doing and you're just forgetful. PHILIPPE: Yes, these rules exist, as well. They're not extensive but there are linting rules for Angular used for security, for example. But the problem in linting is that they are very useful to find potential instances of security relevant features or security relevant functionality. But the linting rule alone cannot decide whether something is OK or not. Just to give you a very simple example, if you use the bypassSecurityTrustHtml function, if you give that function a static snippet of HTML, that's going to be fine unless you write your own attack essentially. But if you feed that function user inputs, you're going to be in a lot of trouble. And making that distinction with a linter is going to be difficult unless you have a static string in the arguments. But if once you start having that from variables to dynamically decide to have a different code path, then that's going to be very, very difficult to decide automatically. So, yes, you can use that to find the places in the application where you should be looking for, in this example, a cross-site scripting in Angular but the linting alone is not going to give you an answer whether this is OK or not. That still requires knowledge of how Angular handles this things, what happens, and how you can do things safely. TARAS: Sounds like we keep going back to nothing beats having knowledgeable developers. PHILIPPE: Yes. Unfortunately, that is true. However, with that said, I want to highlight that frameworks like Angular, well mainly Angular, make things a lot better for developers because yes, you still need knowledgeable developers but the ways to introduce a cross-site scripting vulnerability in an Angular application are actually very, very limited. It's not going to be one, but there's going to be maybe three or four things you need to be aware of, and then you should be set. While if you would have done the same for PHP, it's going to be 50,000 things you need to be aware of that are potentially dangerous. So, yes, frameworks and libraries and all of these abstractions make it a lot better and I really like that. That's why I always refer to abstract things away in a library so that you actually have the ability to look for this dangerous code patterns using linting rules in your code base and that you can, at least, inspect the go to see whether it's OK or not, even though you might not be able to make an automatic decision. You, at least, know where to look and what to approve or how to change in the code base. TARAS: I think that's one of the things that oftentimes is not taken into account that the frameworks are different. And I think of big differences in how much -- like right now, the most popular framework, I think, React. But it's such a thin layer, it's such a small part of the framework that you can hardly call it a framework. But it is something that companies rely on. But then when you consider how much of that code that you need to write, to make React into a complete framework for your company, the amount of code that your team has to write versus the amount of code that your team has to write when you use something like Angular or Ember, there's definitely a lot less parts of the framework that you need to write or a lot less parts of the framework you need to choose from what's available in the ecosystem. Like in Angler and Ember, and I'm not sure what the story is with the view, but the pieces, they come from kind of a trusted source and they've been kind of battle tested against a lot of applications. But I don't think that enters into consideration when companies are choosing between Angular or whatever that might be because they're thinking like what is going to be easiest for us. What is going be [inaudible] for developers? They're not thinking about how much of the framework are we going to need to put together to make this work. CHARLES: I can say it sounds, Taras, like almost what you're saying is by using the frameworks that have been battle tested, you actually get to avail yourself of code that actually has security champions kind of baked into it, right? Is that what you were saying? You keep coming back to 'you need developers who are knowledgeable about security', and if you're using kind of a larger framework that covers more use cases, you're going to get that. Do you think that that is generally true, Philippe? PHILIPPE: Yeah. I think it is and that's why I mentioned that I liked Angular before because Angular actually does offer a full framework. And because they do that, they made a lot of choices for developers and some of these choices have a very, very big and positive impact on security. On the other hand, if you make those decisions, you become an opinionated framework and some people don't like that. They actually want the freedom to follow their own paths and then a less full featured framework like React might be an easier way to go. CHARLES: But I think what happens is folks don't enter into that decision with their eyes open to the fact that they then now need to be their own security champion because they just don't even see it. We said the most dangerous thing is the things that you don't know. PHILIPPE: Yeah, absolutely. And I totally agree. That's something that's at least a couple of years and probably still today, many companies moving into this space struggle like, "Which framework do we choose and why do we choose one or the other and which one will still be there in three years because we don't want to switch to another thing in three years," which is risky to our developers. I like that you said that Angular has this security champion knowledge built in because in Angular 2 and every version behind it, but the new version of Angular essentially, they spent a lot of time on security and they learned from their mistakes in the first version because there were some and they took that and they built a more robust framework with security built in by design or by out-of-the-box. Angular offers, for example, very strong protection against cross-site scripting. It's just there, it's always on and unless you actively sidestep it, it's going to protect you. And that's one of the things I really like about Angular and how they did that. CHARLES: Yeah, that's one of the things that I really like too because I remember there was a blog post back, this is probably, I don't know, almost 10 years ago now, maybe seven or eight years, where someone was comparing why they were more interested in using, their servers were implemented in Ruby and why it was better to use Rails than just Sinatra which is just a very, very, very lightweight HTTP framework. And one of the things that he was pointing to was this new vulnerability was discovered and if you were using Rails, the middle way where the middle square stack is managed by the framework, you just upgrade a minor version of Rails. And now, by default, there's this middleware that prevents this entire class of attack. PHILIPPE: Was that a cross-site request forgery? CHARLES: I think it might have been. PHILIPPE: I think Rails was one of the first to offer built in automatically on support for that. So yeah, that was a very good early example of how that can work really well. CHARLES: And the advantage from the developers' standpoint, because the contrast that, if you'd been writing your application in Sinatra which is this is very, very low level based right on top of rack and you're managing the middleware stack yourself and there are no opinions, then not only do you have to like fix this security vulnerability, you have to understand it. You have to get to do a lot of research to really come up with what's going on, how is this going to affect my application and then I can deploy a fix. And that's like a huge amount of time, whereas you have the freedom to not even understand the attack. I mean, it's always better to understand but you can defer that understanding invariably knowing that you're kind of invulnerable to it. And I think for people who enjoy kind of pretending, not pretending, but that the security world doesn't exist and say, "Hey, I want to focus and specialize on these other areas and attain deep knowledge there." It's very reassuring to know that if a defense for a novel attack comes out, I can avail myself of it just by bumping a version number. PHILIPPE: Yeah, absolutely. If you have everything in place to actually upgrade to that version that fixes those, that's a preferable solution. Towards the future, I believe it's going to be crucial to ensure that we can actually keep things up-to-date because everything that's being built today is going to require continuous updates for the lifetime of the application. I definitely hope that the frameworks get better and more secure and start following these patterns of naming the potentially insecure functions with something that indicates that they are insecure. I think that's definitely a good way forward. CHARLES: Yeah. Can I ask one more question? Because this is something that is always something that I wonder about whenever you talk about any aspect of a system. And part of it is folks will not appreciate good architecture until they've experienced some sort of pain associated with not having that architecture in place. Their project fails because they couldn't implement a set of features without it taking months and years and they just ran out of runway, ran out of deadline. Those types of people who've been on those projects appreciate having a nimble system internally, good tooling. Folks who have experienced good tooling understand how much time they could save, and so, have a very low tolerance for bad tooling. A tool takes too long or is misbehaved or is not well put together, they just can't stand because they know how much time they're losing with security. Is there a way to get people to care about it without having some sort of breach, without having gotten smacked in the face? When you do your trainings, is it generally, "Hey, someone has experienced a breach here, and so they want to bring you in." Or is there some way to get people raise awareness of the problems they don't have to experience that pain but can just experience only the benefit? PHILIPPE: That's, again, a very good question and that's also a very good illustration of why security is so hard. Because if you get everything right, nothing happens. [Laughter] PHILIPPE: Or it might be if nothing happens, that nobody cares enough to actually try something against your application. So, there's no positive confirmation if you've done a good job. You can keep putting things off but eventually, there's going to be vulnerability and it's a matter of how you respond to it. We recently had a cross-site scripting in Google's homepage, one of the most visited pages on the web. And somebody figured out that there were some weird browser thing that could be abused and that resulted in a vulnerability on, let's say, such a simple page. So, even there, things can go wrong. So, what would be a good way to draw with some awareness about this is I would recommend following some simple new resources or some Twitter feeds. I have some security relevant articles there but plenty of other people in the industry have as well. And when you read such an article about security incidents, just think about whether this could happen to you or not. And that should probably scare the shit out of you. Simple examples like the Equifax breach, one of the biggest, most impactful breaches of the past few years happened because of an Apache library that was not updated. I think the Apache library, they had a known vulnerability in there. We knew about it. We had a patch, yet it took too long to install that patch and the attackers abused that vulnerability. This is something that probably can happen to each and every one of us because the attacks started, I think, 72 hours after the vulnerability and the patch had been published. So, ask yourself, "Would I have updated my servers in three days after I got that vulnerability report on GitHub?" Yes or no. And if the answer is no, then the same thing can happen to you. Other cases: Magecart is a very big problem, people injecting credit card skimming malware in the JavaScript library. Are you including third party JavaScript libraries? If yes, then chances are that this can happen to you. And there's nothing preventing someone from exploiting that. It's probably just because you got lucky that nobody tried to do that. And the same thing you see now with all these attacks npm packages where people actively try to get you to install a malicious package as one of your dependencies. And again, everybody can fall victim to these things. So, if you read the articles with that mindset, I probably guess that your security awareness will grow rapidly and you will start caring about that very fast. CHARLES: Yeah. TARAS: Lots to think about. CHARLES: Yeah, there's lots to think about because the next thing that occurs to me is how do you even know if you've been targeted. Because a good attacker is not even going to let you know. PHILIPPE: Yeah. CHARLES: It's just better to siphon off your blood, like you said, than to kill the -- you want to be a vampire bat and come to the same cow every night and just take a little bit of blood rather than the lion that kills the cow and then the cow's gone. PHILIPPE: I would say constant monitoring is going to be crucial and you need that data for all kinds of different purposes. You need to monitor everything that happens, first of all, for a post-mortem analysis. If something happens, you want to be able to see how bad it was. This user apparently got a full admin access and if you have decent monitoring, you will be able to retrace his steps to see what they did or did not get. So, that is one very good use case. A second use case is you can use that data to detect attacks. Usually when the attacks are noisy, it's an automated scanning tool but it might be an attacker trying to do things. Again, that may be something very useful for you to act on to see if there is a problem to prevent that user from connecting, or so on. And then, another very good use case of these things is actually inspecting the logs manually as an ops engineer or whatever, who is responsible for doing that, because that might again yield new insights. I've been talking to someone who said that they discovered an abuse of one of their APIs just by looking at the logs manually and detecting a strange pattern and looking and digging deeper into it. And the automated monitoring tools that they had installed that trigger on certain events like a mass amount of requests to the authentication and stuff like that, they did not catch this particular abuse. So, I would say monitoring there is absolutely crucial, for sure. TARAS: So, the takeaway is higher attentive knowledgeable developers who will learn about security. PHILIPPE: I would say the takeaway is security knowledge is essential for every developer. So, I encourage every developer to at least have a little bit of interest in security. I'm not saying that everyone should be a security expert. We should at least know about that the most common vulnerabilities in web applications, what they mean, what they might result in, and what to be on the lookout for. So yes, I think that's one of the crucial things to start with. And then within an organization, you should have someone to fall back on in case that there are security relevant things that you actually can talk to someone who does see a bigger picture or maybe the full security picture to decide whether these things are a problem or not. I think we're closing or nearing the end here, but one of the things we haven't talked about is how to actually get started in security. What if you are interested in security after hearing this podcast and you want to get started? I want to give you just a few pointers so that you actually know where to look. One of the first things to look at is OWASP. And OWASP is the Open Web Application Security Project. It's essentially a nonprofit that has the mission to improve the security posture or knowledge of developers, and they have a lot of resources on various different topics. They have a lot of tools available and things like that. What you want to start with as a developer is the OWASP Top 10, which is a list of the 10 most common vulnerabilities that exist in applications, just to open your eyes like these things exist in applications today and are definitely a problem. And then, there's a complementary Top 10 called the Proactive Controls and that's about how you, as a developer, can actually prevent these things. So, what should we know about implementing security, which guidelines should we follow. And these two documents are a very good place to start. And then there is a huge community that's actually mostly very eager to help people figure out the right way of doing things and solving these problems we have in our ecosystems. TARAS: Awesome. That's great. Thank you very much. CHARLES: Yeah. I'll take that in. That is really, really helpful. Well, thank you very much, Philippe, for coming on and talking about security. I actually feel a lot better rather than usually I'm thinking about securities kind of stresses me out. [Laughs] PHILIPPE: You can bury the problem but it's going to return in the future anyway, so you might as well get onboard and start learning. It's not that scary if you actually -- it's a lot of fun. So, you should learn about security. CHARLES: Well, I am looking forward to diving in. If anyone wants to get in touch with you, how would they do that on Twitter or Email? PHILIPPE: Yeah, sure. I'm on Twitter. I'm always happy to chat about security. You can reach me by Email as well. I'm very easy to reach. And I'll be happy to help out people with questions. Sure. CHARLES: All right. Thank you so much, Philippe. Thank you for listening. If you or someone you know has something to say about building user interfaces that simply must be heard, please get in touch with us. We can be found on Twitter at @TheFrontside or over just plain old Email at contact@frontside.io. Thanks and see you next time.

Luke Melia, Aaron Chambers, and Mattia Gheda john Taras and Charles to discuss all things deployment! Luke Melia: Luke has been working with Ember since it was under early development as Sproutcore 2.0. Ember.js powers a SaaS company he co-founded, Yapp, and they funded their business for a couple of years doing Ember consulting under the Yapp Labs moniker. They're full-time on product now, and his engineering team at Yapp (currently 3 people) maintains around 6 Ember apps. Luke helps to maintain a bunch of popular addons, including ember-cli-deploy, ember-modal-dialog, ember-wormhole, ember-tether, and more. He started the Ember NYC meetup in 2012 and continues to co-organize it today. Aaron Chambers: Aaron Chambers: Aaron is the co-author of EmberCLI Deploy and is currently an Engineer at Phorest Salon Software, helping them move their desktop product to the web platform. He's been using Ember for 5 years and maintains a number of plugins in the EmberCLI Deploy ecosystem. Aaron loves trying to work out how we can ship JS apps faster, more reliably and with more confidence. Mattia Gheda: Mattia is a Software Engineer, Ember hacker, Ruby lover and Elixir aficionado. Currently he works as Director of Development for Precision Nutrition where Ember, Ruby and Elixir power several applications. He loves meetups, organizes Ember.js Toronto and co-organizes Elixir Toronto. Resource: Immutable Web Apps Please join us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at contact@frontside.io. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer. This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. Transcript: CHARLES: Hello and welcome to The Frontside Podcast, a place where we talk about user interfaces and everything that you need to know to build them right. My name is Charles Lowell, a developer here at the Frontside. With me also co-hosting today is Taras Mankovsky. Hey, Taras. TARAS: Hello, everyone. CHARLES: Today, we have three special guests that we're going to be talking to. We have Aaron Chambers, Luke Melia, and Mattia Gheda who originally met collaborating on fantastic open source library that we, at the Frontside, have used many, many times that saved us countless hours, saved our clients hundreds of thousands of dollars, if not more. ember-cli-deploy. We're gonna be talking not about that library in particular but around the operations that happen around UI. So, welcome you all. LUKE: Thanks, it's great to be here. CHARLES: Like I said, I actually am really excited to have you all on because when we talk about the platform that you develop your UI on, something that often gets short shrift in communities outside of the Ember community is how do I actually deliver that application into users' hands. Because obviously, we don't want it to be working just on our laptop. We want it to be delivered to our users and there are myriad ways that that can happen and it's only gotten more complex since the last time we talked which must have been like three or four years ago. I kind of just have to ask, I think that what you all were talking about then was cutting edge is still cutting edge now but there must have been some pretty incredible developments like in the last three or four years. What have been kind of the new insights that you all have? LUKE: I think that what we realized as we got started with ember-cli-deploy and a project kind of came together as a combination of a few different open source efforts, something that Aaron was working on, something that our collaborator Mike was working on. We decided to come together under one umbrella, joined forces. And what we realized pretty soon is that deployment needs vary a ton between companies. And so, we are coming from this background in Ember community where we had this attitude where nobody is a special snowflake. We all kind of have the same needs for 90% of what we do. And that's true. I really believe in a lot of that Ember [ethos]. But when it comes to deployment, you know what? A lot of companies are special snowflakes or it's at least is much more fragmented than kind of our needs around on the JavaScript side. And so, what we decided to do was to try to evolve ember-cli-deploy into a platform essentially, an ecosystem that could let people mix and match plug-ins to do in their organization without locking them into an opinion that might simply be a non-starter in their org. CHARLES: It's hard enough to have opinions just around the way that your JavaScript code is structured but when it comes to rolling out your app, it really does encompass the entire scope of your application. So, it has to take account of your server. It has to take account of your user base. It has to take account of all the different processes that might be running all over, distributed around the Internet. Maybe somewhere on AWS, maybe somewhere on Legacy servers but it has to consider that in its entirety. So, it's having opinions that span that scope is particularly difficult. LUKE: Yes. And so, you mentioned a bunch of technical details which are absolutely forcing factors for a lot of words in how they do their deployments but what we found in talking to people that there are also people in political aspects to deployment in many cases. Engineers kind of own the JavaScript code that's running within their app, more or less. But when it comes to pushing the app into the world and a lot of companies, that means they're interacting with sysadmins, ops folks, people who have very strong opinions about what is an allowable and supportable way to get those deployments done and to have that stuff exist in production. And so, we needed to come up with an architecture that was going to support all these kind of varied use cases. And so, we came up with this system of essentially a deployment pipeline and plugins that can work at various stages of that pipeline. And that ecosystem has now grown quite a bit. It's actually, I don't know Aaron if you and Mattia would agree but I think it's probably the best decision that we made in this project because that ecosystem has grown and evolved without us needing to do a ton of work in maintenance. And it's been really great. I think Mattia, you pulled some of the current numbers there. MATTIA: Yeah. I pulled some numbers just yesterday and we have currently 150 different plugins attached to themselves, to different parts of the pipeline. So some of them are about how to build the assets, some of them are about how to compress them, some of them are about shipping. And they allow people to ship it with different ways like we are seeing [inaudible] with just simply Amazon APIs or Azure APIs and some of them even are about just how to visualize data about your deployments or how to give feedback to the user about what was deployed and represent the information. And then this is kind of a bit more detail, probably specific to us but we also added this idea of plugin packs. So, in order to help people define their deployment story, we created this ideal of plugin packs. Plugin packs are simply group of plugins. So, plugins grouped together. As a user, if you want to implement what we call a deploy strategy, you can simply install a plugin pack and that will give you all the plugins that allow you to deploy in a specific way. And that's kind of like an optimization that we added just to make it easier for people to share deployment strategies, share ways of deploying applications. CHARLES: Right. It's almost like an application within the framework. MATTIA: Yeah, exactly. But to stay on the community side, I think that the interesting part about what Luke was saying which was a great success for us is that all we maintain as the core team for this project is the core infrastructure, so the pipeline and a thousand of plugins. Everything else is community-based. And often, even in my day-to-day work, I end up using plugins that I didn't write and that I don't even maintain. But because the underpinning of it were designed especially by Luke and I are flexible enough. It just like has been very, very stable and very, very reliable for many years. So, I will say definitely the idea of like in the spirit of what Ember is, I guess, creating a shared ecosystem where people can add what they want and extend what was provided has been the one single biggest win of this project. CHARLES: One of the things that I'm curious about is we've talked about how you're allowing for and kind of embracing the fragmentation that happens in people's kind of the topology of their infrastructure. What do you see is the common threads that really bind every single good deployment strategy together? MATTIA: My biggest thing here, and we actually have some shared notes about this, but my biggest thing about this is the idea that building and deploying an application for me is divided into three parts. There's building part where you have to decide how to compile your JavaScript application and how to produce some sort of artifact. There is the shipping part where it's about deciding where you're going to put the artifact. And then there is the serving part which is how you show it and deliver it to your users. I think that these three are the underpinnings of any deploy strategy. What we did with this project is just acknowledging that and give each one of these a place. And so, the entirety of what we do in what Luke defined as a pipeline is simply give you a way to customize how you build, customize how you ship, and then customize how you serve. So yeah, I think that that's kind of the root. And the question that everybody that wants to deploy a modern JavaScript application have to ask themselves is how do I want to build it, how do I want to ship it, and how will I serve it to my users. And these things are completely independent one from the order in the sense that you can have something build it, something ship it and something serve it, and that's what we end up doing in most of our deployments, I find. CHARLES: It's good to think about those things as soon as you possibly can and make sure that you have all three of those bases covered really before you start adding a whole ton of features. TARAS: Sprint 0, right? LUKE: In Agile, we call Sprint 0 the phase the thing you do right in the beginning. You've got a skeleton of an app and then you get the deployment infrastructure going, you have the test infrastructure going, so that there is no task within your actual feature development where you have to do those things. And I think that can be a valuable concept to embrace. I would just add to Mattia's three points that for deployments, to me, some very simple qualities of a good deployments are repeatability. You need to be able to reliably and consistently run your deployment process. Sounds simple but there's plenty of operations that have run up way too long on manual deployments. So, we don't want to see those rollback capabilities if you have a deployment that you realize was a mistake right after it gets into production. I'm sure none of us have ever experienced that. CHARLES: That never happens to me. LUKE: You want to have a method to roll that code back. That's something that can be remarkably complex to do. And so, having some guardrails and some support mechanisms to do that like ember-cli-deploy provides can be really useful. But whatever your approach is, I think that's a necessary quality. And then I think we start to step into kind of more advanced capabilities that a good deployment architecture can provide when we start to think about things like personalization, A/B testing, feature flags, these kinds of things. And that requires more sophistication, but you cannot build that on a deployment foundation that's not solid. AARON: I think for me, one of the things I've been really thinking about a lot lately, it's a bit of a mindset shift, I think, to get to where the things Mattia was talking about separating those different parts of deployment. And so, I really start to realize the traditional mindset around deployments like I build some stuff and I ship it to the server and then the users get it. But if we can actually stop and actually split our understanding of deployment into two separate phases. One is the building and the physical shipping of the files; and the other one's actually making them available to people. You open up this whole world of our features that you wouldn't normally have. So to be able to actually physically put stuff in production but not yet have it active, as in users don't see it yet but you can preview those versions in production against production databases. And then at some point after the fact, decide, "Okay, I'm now going to route all my users to this new thing," And to be able to do that really easily is massively, massively powerful. And so, to me, the thing I've been thinking about lately is it is a small mindset shift away from packaging everything up and pushing and overwriting what's currently there to being something, again like Luke said, immutable deployments where everything we build and ship sits next to all the other versions and we just decide which one we want to use to look at it any time which leads into then, I guess, A/B testing, feature flags and things. So I guess deployment really is not so much about the physical shipping, that's one part of it. To me, deployment now is shipping of stuff, as in physical deployment and then the releasing it or enabling it or activating it to users. CHARLES: Or routing it. Sounds like what you're describing is an extraordinarily lightweight process. AARON: It is, yeah. CHARLES: To actually route traffic to those files. AARON: It is. It's incredibly lightweight. That's the amazing thing about it. When you think about it, you're building a few JavaScript files and CSS files and images and putting them on a CDN, and then you just need a tiny web server that basically decides which version of the app you want to serve to people. There's not much to it at all, really. CHARLES: I mean that's absolutely fascinating, though the capability that you have when you have the ability to have these versions, the same versions or different versions of your application sitting along next to each other and being able to route traffic. But it also seems to me like it introduces a little bit of complexity around version matching because only certain versions are going to be compatible with certain versions of your API. You have different versions of your API talking to the -- so the simplicity of having kind of mutable deployments, so to speak, is that everything is in sync and you don't have to worry about those version mismatches. Is that a problem or this could just be me worrying about nothing? But that's kind of the thing that just immediately jumps out to me is like are there any strategies to manage that complexity? LUKE: To me, what you're describing, I kind of think of as a feature not a bug. And what I mean by that is that it is very simple to have a mental model of, "Oh, I have a version of my JavaScript code that works with this version of my API." And as long as I kind of deploy those changes together, I'm good to go. The reality is that that's impossible. The JavaScript apps that we write today, people are using anywhere from two seconds at a time to two days at a time. It's not uncommon these days to have some of these dashboard apps. People literally live-in for their job eight hours a day, nine hours a day, keep the browser tab open and come in the next morning and continue. And so, obviously there are some mechanisms we could use to force them to reload that kind of thing. But at some point in most apps, you're going to have a slightly older version of your JavaScript app talking to a slightly newer version of your API for either the span of a minute or perhaps longer depending on their strategies. So to me, the process of thinking about that and at least being aware of that as an engineer thinking about how your code is going to get from your laptop into the world, I think it's an important step that we not paper over that complexity and that we kind of embrace it and say, "Hey, this is part of life." And so, we need to think about just like we need to think about how your database migrations get into production. That's not something that you can paper over and just have a process that it's going to take care of for you. It requires thought. And I think that this, in the same token, how different versions of your JavaScript app are going to interact with your API requires thought. An exact parallel also had different versions of a native mobile app that go into the app store. How did those interact with different versions of your API? So, I think you're right. There's complexity there. There's ways that we can try to mitigate... CHARLES: Keep repeating ourselves if we think that that [inaudible] actually doesn't exist even in the simple case? MATTIA: Yeah. I think that that's to reiterate what Luke is saying. That's exactly the point. You can pretend it's not there but it is and you have no way to avoid it. Once you ship something to a browser, you have no control over it anymore. And so, you have to assume that somebody is going to be using it. LUKE: Aaron, I think you too, I don't know if you can share it. But you recently told us some stories about kind of what you encountered in your work about this and of how long people were using versions and stuff. AARON: Yeah. Something that we hadn't sort of put a lot of thought into. But the last place I worked at, we had quite a long lived app and we're using feature flags and we're using launch [inaudible] something and it gives you a list of flags and when they were last requested. And there were also flags that we removed from the code and it was just a matter of waiting until all the users had the most current version of the app and weren't requesting the flag anymore. But this one flag just kept getting requested for months and we just could not work out why. It really sort of opened my eyes up to this exact problem that these long lived apps set in the browser and if you have someone that just doesn't reload the browser or restart the machine or anything, your app can live a lot longer than maybe you actually realize it is. So we're shipping bug fixes, we're shipping new features, and we're all patting ourselves in the back. We fixed this bug but have we really? If your users haven't reloaded the app and gotten the latest version, then you haven't actually fixed the bug for some number of people. And it's really hard to tell them as you think about this and put things in place, really hard to tell what versions are out in the world, how many people are using this buggy version still. CHARLES: Yeah, that's an excellent point. I haven't even thought about that. I mean, what is the countermeasure? AARON: We hadn't made it until we came across [crosstalk]. CHARLES: It's nothing quite like getting smacked in the face of the problem to make you aware of it. AARON: That's right. CHARLES: So, what's the strategy to deal with that? AARON: I guess for me, my learnings from that would be from very early on thinking about how we're going to encourage people to reload, to start with, and maybe even have the ability to force a reload and what that means but then that has gotchas as well. You don't want to just reload something when a user is in the middle of writing a big essay or something like that. But definitely thinking about it from the start is one of the things you've got to think about from the start. But I guess something that I'd like to implement and I've kind of thought through but not really explored yet, but the ability to see what versions are out there in the world and there are things I've been thinking about in terms of this little server that serves different versions. Maybe we can start having that kind of tracking what versions are out there and who's using what and being able to see because it would be great to be out to see a live chart or a dashboard or something that sort of shows what versions are out there, which ones we need to be aware of that are still there and even what users are using and what versions they can maybe even move them on, if we need to. But there's definitely a bunch of things that aren't immediately obvious. And I don't know how many people actually think about this early on, but it's critical to actually think about it early on. MATTIA: Yeah. I was going to share what we do which is very similar to what Aaron said just maybe for the listeners to have some context. The first thing you can do is basically what Gmail does, which is every time a web app sends a request, an [inaudible], it will send the version of the app with the request and the backend can check. And the backend can check if you are sending a request from the same version that is the most recent deployed version. And if it's not, this sends back a header and the same way that Gmail does, it will display a pop up that is like, "Hey, we have a new version if you want reload." And on top of that [inaudible] is that we have a dead man's switch. So if we accidentally deploy a broken version, a part of this process, the frontend application tracks in the headers. And if a special header is sent, it force reloads, which is not nice for the user but it's better and sometimes is critical to do so. CHARLES: Right. I remember that was that case where Gmail released something. They were doing something with broken service workers and the app got completely and totally borked. I remember that my Twitter blew up, I don't know, about a year ago I think, and one of the problems I don't think they had was they did not have that capability. MATTIA: I mean, you learned this the hard way sadly. But I think these two things are definitely crucial. And the third one, [inaudible]. I thought, Luke, you had the ability that Aaron was talking about like tracking versions in the world. And I think that's more useful for stats so that you know how often your users update. And then you can make the design decisions based on that and based on how much you want to support in the past. LUKE: Yeah. We haven't implemented that but it reminds me whenever there's a new iOS version, we do a bunch of mobile work in the app and we're always looking at that adoption curve that's published. A few different analytic services publish it and say, "OK. How fast is iOS 12 adoption? How fast are people leaving behind the old versions?" And that helps to inform how much time you're spending doing bug fixes on old version versus just telling people, "Hey, this is fixed in the new OS. Go get it." But if you are able to see that for your own JavaScript apps, I think that would be pretty hot. CHARLES: Yeah. Crazy thought here but it almost makes me wonder if there's something to learn from the Erlang community because this is kind of a similar problem. They solved 20 years ago where you have these very, very long running processes. Some of them there's some telephone servers in Sweden. They've been running for over a decade without the process ever coming down. And yet they're even upgrading the version of Erlang that the VM is running. And they have the capability to even upgrade a function like a recursive function as it's running. And there's just a lot of -- I don't know what the specific lessons are but I wonder if that's an area for study because if there's any community that has locked in on hot upgrade, I feel like it's that one. LUKE: That's a terrific analogy. I bet we could learn a ton. Just hearing that kind of makes me think about how kind of coursed our mental model is about updates to our JavaScript apps. We talked to Aaron, we're talking about kind of this idea of a mutable apps and you have different versions side by side. But the idea of being able to kind of hot upgrade a version with running code in a browser, now that's an ambitious idea. That's something I'd say, "Wow! That kind of thing would be a game changer." CHARLES: Yeah. AARON: Makes me feel like we got a whole bunch of work to do. CHARLES: We welcome them. I'm always happy to give people plenty more work to do. No, but how they manage even being able to do migrations on the memory that's running. I don't know if it's something that's going to be achievable but it sounds kind of like that's the direction that we're heading. LUKE: It does, as these apps get more complex and they continue to live longer. The idea, the work arounds that Mattia mentioned about kind of showing a message and having a dead man's switch, these are all certainly useful. And today, I would say even like the best practice but they were not what you would want to do if you could magically design any system. If you're taking a magical approach, the app will just be upgraded seamlessly as a user was using it. And they would be none the wiser, the bugs would be fixed. End of story. There's no interruption to their workflow. At least for me personally, I don't really think about that as a possibility but I love the Erlang story and analogy and to say maybe that is a possibility, what would it take? I would obviously take a collaboration across your JavaScript framework, perhaps even JavaScript language features and browser runtime features as well as your backend and deployment mechanism. But I think it's a great avenue for some creative thinking. CHARLES: I'm curious because when we're talking about this, I'm imagining the perfect evergreen app but there's also feels like there's maybe even a tension that arises because one of the core principles of good UI is you don't yank the rug out from underneath the user. They need to, at some point and we've all been there when the application does something of its own agency, that feels bad. It feels like, "Nope, this is my workspace. I need to be in control of it." The only way that something should move from one place to the other without me being involved is if it's part of some repeatable process that I kicked off. But obviously, things like upgrading the color of a button or fixing a layout bug, those are things that I'm just going to want to have happen automatically. I'm not going to worry about it. But there is this kind of a gradation of features and at what point do you say, "You know what? Upgrading needs to be something that the user explicitly requires," versus, "This is something that we're just going to push. We're going to make that decision for them." LUKE: Yeah, it's a great question. One of the things that I'm curious what you all think is when you think about the mental model that our users have of working in a browser app, do you think that there is a mental model of, "Oh, when I refresh, I might get a new version." Do people even think about that? Or are they just like your example about a button color changing as kind of a minor thing. I don't even know if I could endure stack. We've all been I think in situations where you do a minor redesign and all of a sudden, all hell breaks loose and users are in revolt. Take the slack icon. So, I think it's a fascinating question. CHARLES: I don't know. What's the answer? Do you always ask for an upgrade just observing? I don't have any data other than observing people around me who use web applications who don't understand how they actually work under the covers. I don't think it's the expectation that this code, this application is living and changing underneath their feet. I think the general perception is that the analogy to the desktop application where you've got the bundled binary and that's the one you're running is that's the perception. AARON: I'd say the difference there is that, and with all these new ways of deploying, we're shipping small things faster in multiple, multiple times a day or even an hour. So it's not the sort of thing you really want time to use. There has been an update, need to upgrade as well. And that's the difference between the desktop mentality. And if that's the mentality they have, it sits quite a bit of a shift, I guess. MATTIA: It makes me think that one of the tools that the users -- if you take a look at the general public, there's probably one tool that everybody can relate to which is Facebook. So, I think if there is a way to say what do people generally expect. There is a business user which I think we are often most familiar with but the general public, probably what they're most familiar with is what happens in Facebook. And I don't use Facebook almost. I haven't used it for a couple of years but I wonder how much of what people experience in Facebook actually impacts the expectations around how applications should behave. LUKE: I think that's a really good question. I do think your underlying point of you have to know your own users I think is an important one also. Obviously, some folks are going to be more technical than others or some audiences will be more technical than others. But I would even question, Charles, your suggestion that people think of it kind of as like a binary that it stays the same until you refresh. I think people have an idea that web apps improve over time or sometimes they get bugs but hopefully that they improve and change over time, and that there is a tradeoff there that means sometimes there's something new to learn but at the same time, you get new features. But I don't know that people necessarily associate that with and it happens when I hit reload or it only happens when I open a new browser, like I don't know that it's that clear for people in their head. CHARLES: Right. I can see that. But the question is if the evolution is too stark, I think people tend to get annoyed. If they're in the middle of a workflow or in the middle of a use case and something changes, then it gives it a feeling of instability and non-determinism which I think can be unsettling. LUKE: Definitely. We all value, as engineers, we value getting into that flow state so much of like, "Oh man, I'm being productive. I don't have any distractions." And you kind of owe that to your users also to be able to let them get into that state with your app and not be throwing up, "Hey, there's new stuff. Reload." "I'm in the middle of something. Sorry." CHARLES: Yeah. I definitely do the same thing. Sometimes, I let iOS be bugging me to upgrade for a month until I finally start to feel guilty about security and actually do the upgrade. LUKE: Right. CHARLES: Although once they started doing it at night, it actually made it a lot better. LUKE: That's an interesting idea, too. I think there's a natural tension between the lower integration risk that we have as the engineering teams of shipping very frequently. Aaron mentioned shipping a dozen times a day. We certainly have been there and done that as well. I would say on average, we ship a few times a day. But the reason that we do that is because we know the faster we get code into production, the faster we can trust it. So it passes all your tests, it passes your [inaudible], but you don't really know if you're being honest. You don't really know until there's thousands of people using it in production. And yet, this conversation makes me think about there is a tension between how frequently you do that versus your users' kind of comfort level and expectations. CHARLES: And maybe there is a thing where you can kind of analyze on a per user basis how often they're active in the application and try to push updates on times that are customized to them. LUKE: Like when a user has been idle for 30 minutes or something like that. CHARLES: Yep. Or even like track trends over months and see when they're most likely to be idle and schedule it for them. LUKE: Good point. CHARLES: Something like that. TARAS: I have an idea. We should introduce screen savers into web apps. And so when the user stops using the app, just turn on screen saver and do the upgrade. LUKE: I can see the VC patch. It's after dark but for the web. CHARLES: Enter install flying toaster. AARON: It does that to open up the idea as well of that automated checking that things are okay well after the fact because it's all right to sit there maybe activate something and sit there even for an hour and make sure there's no bug request coming in. But if no one's actually received your app, then of course it's not going to come in. And it's very easy to kind of move onto the next thing and forget about that. I guess it's not something that I've ever put a lot of time in and I haven't worked anywhere that's had really great automated checking to see is everything still okay. And I guess it's an interesting thing to start thinking about as well an important thing. CHARLES: Like actually bundling in diagnostics in with your application to get really fine grained information about kind of status and availability inside the actual app. AARON: I'm not exactly sure, really. I guess I maybe wasn't thinking about inside the app but I don't know what it looks like exactly. But there is that element of shipping fast and getting stuff out there. But are we really making sure it all works later on when everyone's actually using it. LUKE: Yeah. This by the way, I just wanted to say when we were talking earlier what are the essential qualities around deploying an app. And this reminds me that one thing that we didn't mention but is very simple is your app should have a version. And it should be unique and traceable back to what was the GitHub, git commit that was the origin of that code. It's just a very simple idea but if you're going to be analyzing errors in production when you have multiple different versions of your JavaScript app running, you're going to need to know what version caused this error and then how do I trace it back and make sure that the code that I'm trying to debug is actually the code that was running when this error happened. CHARLES: Do you only use just [inaudible] or do you assign like a build number or using SemVer? What's a good strategy? LUKE: In our case, we use git tags. And so, our CI deployment process for our Ember apps basically looks like this is we work on a PR, we'll merge it to master. If it builds, the master gets deployed into our QA environment automatically using ember-cli-deploy from our CI server. And then once we're happy with how things are on QA, we do git tag or actually I use an ember addon called ember release that does that tagging for me and I'll tag it either in patch minor or major, roughly [inaudible] although it doesn't matter that much in the case of apps. When there is a new tag that builds green on CI, that gets deployed automatically into production by ember-cli-deploy. And so, that's kind of a basic flow. That tagging, just to be clear, the SemVer tag is just going to be number.number.number. You can get more sophisticated than that and I think both Aaron and Mattia have a system where even in the PR stage, there's automatic deployments happening. So maybe one of you want to mention that. AARON: We're slightly different. Every time we push the pull request, that gets deployed to production. We're able to preview up pull requests in production before we even merge into master which we find super useful to send out links to stakeholders and maybe people that have raised bugs just to get them to verify things are fixed. And then at the point that it's all Google merged, the pull request to master which will automatically do another deploy which is the thing we'll ultimately activate, we activate it manually after the fact. We just do a little bit of sanity checking but we could automatically activate that on merge to master as well. But yeah, the being able to preview a pull request in production is super powerful for us. CHARLES: That is definitely a nice capability. It's hard. It's one of those certain workflows or patterns or tools that you remember life before them and then after them and it's very hard to go back to life before. I would definitely say kind of the whole concept of preview applications is one of those. AARON: Absolutely. It's a daunting concept if you're not there, previewing something that's essentially a work in progress and production. And there are some things you want to be careful with, obviously. But for the most part, it's a super valuable thing. As you say, it's a world where once you're there, it's very hard to step back and not be there again. CHARLES: So I had a question about, we talked about I think it was 162 plugins around the ember-cli-deploy community. What for you all has been the most surprising and delightful plugin to arrive that you never imagined? MATTIA: That is a good one. I'm pulling up the list. What I can tell you, for me, it's not about a specific plugin. The surprising part was the sheer amount of different strategies that people use for the shipping part. At least, I found that the build part is similar for most people, like most people want to do the things that you're supposed to do. So, you want to build your application and then you want a minify it in some way. And there's a bunch of options there from gzip to more recent technologies. But the way people deliver it to servers and the difference in the solutions, that I think for me has been the biggest thing, where people that ship [inaudible] are people that ship directly to Fastly, people that use FTP files, people that use old FTP, people that use our sync, people that do it over SSH. We have people that ship stuff directly to a database because some databases actually have great support for large files. So we even use it as a storage. We have people that do it in MySequel, people that do it in [inaudible]. CHARLES: It's actually storing the build artifacts inside of a database? MATTIA: Yes, I've seen them in that. It's kind of interesting like the solutions that people ended up using. And so for me, I think that that's been the most fascinating part. Because as we were saying at the beginning, I'm just seeing now, we even have one for ZooKeeper. I don't have an idea what this does but it's probably related to some sort of registration around the seven-day index. That, to me, I think has been the biggest surprise. Everybody ends up working in a different environment. And so, that flexibility that users need has been by far the most surprising one. AARON: I think that's also been one of the challenging things, one of the enlightening things for me. I think in the Ember ecosystem, addons and even just Ember itself, it's all about convention of configuration and doing a lot of the stuff that you do for you. I think people expect them to see a lot of point to do the same thing. But the key thing here is it just really automates all the things you would do manually and you need to understand exactly what you want your deployment strategy to be before use them to see a lot of [inaudible] could do for you. You need to decide do I want to install my assets on a CDN and do I want to install my index in Redis or in console or in S3 bucket. You need to know all these things and have decided on all these things and then ember-cli-deploy will make that really easy for you. And this is one of the educational things, I think we still haven't even nailed because there are always people that want to know why this doesn't work. But deployments are complex thing and as what you were saying Mattia, there are so many different variables and variations on doing this that there's no sensible configuration ember-cli-deploy could really provide out of the box, I guess. And so, that's why we ended up with a pipeline that gives you the tools to be flexible enough to support your strategy. LUKE: I think the closest that we come to the convention is that if any app is using ember-cli-deploy, you can run ember deployment targets or ember deploy prod and ember deploy QA and you can expect that that's going to work. What you don't know is how has it happened to be configured in this project. Charles, your question about kind of the most surprising thing that's come out of the ecosystem. For me, I would say -- Mattia mentioned plugin packs earlier which are groups of plugins that kind work together well. And so, we've seen some plugin packs like you might expect, like an AWS pack for deploying to AWS. But the more interesting ones to me that we've seen a lot of, companies open source their plugin packs. So what you naturally fall into as a company that's adopting ember-cli-deploy that has multiple ember apps is that you are going to develop your own plugin pack for internal use because generally speaking, companies follow the same deployment pattern for each of their apps. There's usually not any reason to vary that. So then the new thing that happen on top of that is people said, "Why don't I make this open source so other people can kind of see how we do it?" And that's been a really delightful part of the process to kind of get a peek into how other organizations are orchestrating their deployments. And if people are curious about kind of looking at that themselves, you can go on NPM and look for keyword ember-cli-deploy-plugin-pack and pull up all of those. And you can kind of poke around and see what different companies have open source there. CHARLES: I actually love all three of those answers because it really is for me when you have a constellation of people around a particular problem, it's the surprising solutions that emerge that are some of the most exciting that would have lain hidden otherwise. It would have been kind of buried beneath the source of company A or company B or company C but actually having it all out in the open so that you can inspect it and say, "Wow, where has this solution been all my life?" Something that you have never imagined yourself. LUKE: It's so funny that you mentioned that because that actually is the origin story way back, I'm talking like 2013 probably when we were very early Ember adopters and we were trying to figure out how do we deploy this thing. We're deploying it with our Rails app, like literally deploying the Ruby code and the JavaScript code together which took forever which is a disaster. And I heard through the grapevine, just exactly what you're saying Charles, where the good ideas are kind of hidden inside of a company. I heard through the grapevine that Square had this approach that they were using where they would deploy their JavaScript assets and then deploy their index HTML file, the contents of that file, into a database, it's Redis in their case, and serve then it out of there. And it empowered all of these interesting situations of like having multiple versions, being able to preview the release, et cetera. And so we then set out to copy that idea because there was nothing open source. So we had to create it ourselves which we did in Ruby which we made it inaccessible to many JavaScript shops in the first place. Then the evolution of that kind of over time and of Mattia and Aaron and Mike and the rest of the community kind of talking together has now moved this into the open source sphere where these ideas are more accessible and we've created an ecosystem encouraging these ideas to stay out in the open. It's so true that there's just gems of ideas that have been created by really brilliant engineers inside of companies that could be benefiting so many people, they just haven't seen the light of day yet. CHARLES: Yeah, that leads me to my next question. I would say most of the ideas that we've been talking about today really, except for the build part, how Ember specific are they? Obviously, the ember-cli is a great resource and has a lot of great opinions for actually building the JavaScript assets themselves. But the second two phases of the pipeline really can vary freely, if I understand correctly. And so, have you all ever thought about trying to maybe kind of abstract these processes and these plugins so that these same ideas don't remain not just inside of a company but inside a community that spans a set of companies so that it's available for a wider audience? How integrated is it with Ember and what kind of effort would that be? LUKE: That's a great question. Aaron or Mattia, one of you guys wanna talk a little bit about the history here? MATTIA: Yeah, sure. We've been thinking about this as well. In the past, a guy on the ember-cli-deploy team, Pepin, has actually started this effort and it kind of prototyped this very idea of separating the ember-cli-deploy part from Ember CLI itself and make it a bit more generic. And he started a project called Deploy JS which I can give you a link for the show notes later. I don't think that the project is currently maintained but definitely the start of the effort is there. And the funny thing is that it was surprisingly easy. I think that we didn't get there mostly because we just all use Ember at work. As you know, open source is mostly motivated by the needs that an individual or a group of people have. But if any of the listeners were very interested in this, I think they should definitely get in touch and we will be happy to talk to them and see what can be done here. AARON: And also if you look, there's actually a plugin called ember-cli-deploy-create-react-app and there's also ember-cli-build-view. So it can and is being used to deploy non-Ember apps which I think is super interesting because the only real Ember part of it is just using the CLI to discover the addons and plugins. And from then on, it's really out of the hands of Ember. But it sort of leads into a little bit, Luke mentioned this concept of immutable web apps. And I've been thinking a lot about this lately because a deployment strategy that ember-cli-deploy use as an example a lot and it's kind of become [inaudible] ember-cli-deploy in ways, the lightning approach which is this whole idea of splitting or putting your assets in CDN and your index HTML separately maybe in Redis and serving that. I've been trying to work out how I can talk about this to the wider JavaScript community in a non-Ember way and knowing full well that the concept of lightning deployment means nothing to anyone outside of Ember. Just by chance, I was just talking to some people and this terminology of immutable deployments kind of rose. I started searching around and I come across a website called ImmutableWebApps.org which was just scarily the same as what we've been talking about for the last three or four years with ember-cli-deploy. And a way to boil down at a framework-agnostic level, what are the key points that you need to consider when building a JavaScript web app to make it immutable. And it was just really amazing seeing it. This website was put up 3 weeks before I did my Google search coincidentally. And it's basically word for word what we have been talking about the last three or four years, So, someone else in the other side of the world's been coming up with the same ideas in their company like you were talking about earlier and we've reached out to him. I guess that, to me, is sort of the way forward that I want to sort of pursue to try and get these ideas out in a framework-agnostic way to the rest of community and say, "Hey, we thought about deployment in this way. Have you thought about building your app in this way to give you these sorts of capabilities?" CHARLES: I think the wider community could definitely benefit from that because most of the blog posts and talks I've seen that concern themselves with deployment of single page applications, it's still much more of the tutorial phase. Like, "This is how I achieve getting this deployment strategy." Not, "This is how I repeatedly encode it as a program," and leverage it that way. And so, definitely getting that message out to a wider audience, I think it's a -- what's the word? It's an underserved market. LUKE: Yeah. I really like this idea also. I think about this ImmutableWebApps.org, if you look at it. It's sort of a manifesto with conceptual description of what are the qualities that your app has to have to qualify as an immutable web app which I think is kind of a funny idea but one that people can start to get their heads around and compare that description to their own apps and say how do I hit this or fall short. And it reminds me a lot of kind of the idea of Twelve-Factor App which is an idea that I think came out of Heroku originally. And it's an idea of a backend app that is portable to be able to easily move across different hosts and easily be scalable to different instances of [inaudible] if your app obeys all of these things then it's going to do well under those circumstances, that will satisfy those needs. So, I think it's a great way of thinking and probably maybe even a better entree into the conversation with the wider community than a library, this is certainly a library called ember-cli-deploy. CHARLES: This is a fantastic discussion. It's definitely reminded me of some of the best practices that I haven't thought about in a long time and definitely opened my eyes to some new ones and some new developments. So often, we can be focused on how our apps work internally, like how the JavaScript code works that we can just kind of -- what's the saying -- lose sight of the forest through the trees or I can't remember. It's like too busy looking down the end of your nose to see past your face. I've probably mangled both of those adages, but maybe 60% of two mangled adages is at least equal to one. This is something that we need to be thinking about more, that everybody needs to be thinking about more. This is actually a very exciting, very useful problem space and I'm just really grateful that you guys came on to talk to us about it. So, thank you, Mattia. Thank you, Luke. Thank you, Aaron. MATTIA: Thank you so much. It was a great time. AARON: It was a pleasure. Thanks for having us. LUKE: Thanks so much. CHARLES: Thank you for listening. If you or someone you know has something to say about building user interfaces that simply must be heard, please get in touch with us. We can be found on Twitter at @TheFrontside or over just plain old email at contact@frontside.io. Thanks and see you next time.

Joe joins the panelists to talk about pull request etiquette. Joe Leblanc: Joe first learned to code on a Zenith computer his dad brought home from work. It had this built in blue LCD monitor and ran on 5 1/4" floppy disks. He used spreadsheets for work and Joe was interested. They spent about an hour going over macros together and he took off from there. Long after the Zenith died, the open-source content management system Joomla! landed in the center of his attention. Joe found himself writing a book about Joomla programming, authoring video tutorials about Joomla for lynda.com, giving Joomla talks, and helping organize Joomla conferences. Since his time in the Joomla community, he's picked up Node, Rails, React, and other frameworks. He's currently coding at True Link Financial and working on a few hobby-projects as well. Please join us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at contact@frontside.io. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer. This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. Transcript: CHARLES: Hello and welcome to The Frontside Podcast, the place where we talk about user interfaces and everything that you need to know to build them right. My name is Charles Lowell, a developer here at The Frontside. And with me also is Taras. Hello, Taras. TARAS: Hello, hello. CHARLES: Today, we're going to be continuing our conversation about platforms, as always, but in particular the pillar of your platform that has to do with how you collaborate on code. It's an important one. And so, we're spending some time on it. And with us to talk about this today is Joe LeBlanc who is a senior software engineer at True Link Financial in our very own beloved Austin, Texas. Hey, Joe. JOE: Hi. CHARLES: Thanks for coming on the show. We're going to be talking about collaboration and I was thinking we could kick off the discussion talking about pull requests because that's typically one of the ways that we collaborate on code. That's really where the rubber tends to hit the road. You have particular interest in the dynamics of a pull request. What experience kind of led you to that interest? JOE: My background has been doing a lot of work as a freelancer. And when you work as a freelancer or do agency work, a lot of times you are really the only software developer that is around or you're working with maybe one or two other people. And I didn't get a lot of opportunity for pull request reviews. Mine was in that space. And then when I moved to a full time job at a place where I was one among maybe half a dozen or a dozen engineers, that's when I really began to get interested in how I could be better at giving pull request reviews and also submitting pull requests that people want to review. CHARLES: What made you notice the need for this? JOE: What would happen is someone would submit a pull request and there would suddenly be just dozens and dozens of comments coming in that were just kind of difficult to keep track of and often were maybe talking about things that didn't necessarily need to be reviewed as a part of that pull request or addressed, maybe things that could be caught by a linter. CHARLES: Right. JOE: And other tools that are a little bit easier to receive feedback from. Like it's easier to have a tool tell you your spacing is off than have me tell that to you. CHARLES: Right. And really that seems like the spacing is off, that's kind of like if you need to deliver that feedback, that's kind of like not what you want to lead with. If you don't have linting in place like after all other issues are sorted out. JOE: Yeah. CHARLES: But it sounds like what you're describing is people who have kind of swarmed over a pull request each with their own kind of pet peeve issue. JOE: Yeah. And then you're just left with this long list of comments to go back and address and you're pushing up more and more commits. And by the end of it, you could have more than 100 comments on this PR. And you thought that you were going to get this done in a day or two, and then suddenly it's the end of the sprint and it's like, "Oh fine, then I get to merge this." CHARLES: And typically, it's actually, in my opinion, an anti-pattern when you have like 10 mergers that happen on the second last day of the sprint or the last day of the sprint. There's always issues with that, right? Ideally, you are merging code throughout the course of the sprint. It kind of like defeats the purpose almost. I guess you're doing your integration in shorter periods but even so, tons of stuff is bound to break when everybody's pushing and everyone's rushing. JOE: Yup. TARAS: So, what would happen in that situation, in your experience, when you have this pull request that a lot of people are commenting on and some of the comments could be addressed by linters. Would you guys go back and actually implement linting? Did that happen? Would you put linting in place so that you'd have consistent formatting on the projects to reduce that kind of feedback? What would be the resolution or a way to improve on a process so you could actually get better feedback? JOE: Yes, we did install a linter and that's something that you can either run locally or you can also run in your continuous integration environment. And it's really good to run it locally first because then you can just catch it before you even submit the PR. And that was something that definitely helped cut down on these sorts of comments and even debates over style. You just have this one thing that is maintaining that style or rather telling you when you're wrong for you, and then you don't have to have all these comments coming in as you submit your PR. And another thing that we would do is we would use, even beyond just things like whitespace and trailing newlines and those sorts of issues, sometimes you would have some of these issues that would come in that wouldn't necessarily be a trailing newline or whitespace issue or something like that but it would be purely someone's sense of style. And in those situations, we came up with a bit of a standard where if you wanted to make a comment that was strictly style that wasn't something that you necessarily wanted to block the pull request over, we would use emojis that had specific [inaudible] thing. So in our case, we would use either a beer emoji or the cocktail emoji to kind of say, "Here, take this with a drink." CHARLES: [Laughs] JOE: This is not something that's super serious. It's a suggestion, "Here, let's discuss this over beers." [Laughter] TARAS: It's a really nice approach to friendlify the actual pull request because people can get really -- I think another way people would say this is like when you say that it's not 'I believe it whole strongly'. How does that go, Charles? CHARLES: It's strong opinions, weakly held. JOE: Yes. CHARLES: Right. You're kind of making an assertion about the way things ought to be or often missing from that is how important it is to you. You've made this thing like it should be like this. That's a very black and white statement, very firmly cut between right and wrong. But how important it is to you if it goes the other way is often missing from the conversation. TARAS: So Joe, what would happen once you made this change? Did you see a shift in the kind of feedback you were getting? What did it look like after this? JOE: After we started implementing the ideas of putting on emojis to signify things that were not super serious, it became a lot easier to just picture all those comments and define the things that really needed to address. And it was a lot faster. I was just able to address these very specific problems and do that first. And then if I had time later, come back and address those style issues. CHARLES: Right. Was the kind of the good feedback there the whole time but it was just obscured by the noise of these kind of exterior issues or side issues? Or did you find that because people's thoughts were more given over to the heart of the matter, that you actually got more comments directly relating to kind of the meat of the implementation? JOE: I think it was really more that the good comments really were buried in there. A lot of times when you see critical comments that seem to be petty, that can really trigger your emotions. And so, I feel like a lot of the times the emotional state that we would get into would prevent us from seeing the good comments that people were leaving, and we would get a little too angry about things that really weren't that big of a deal. CHARLES: Would the tension kind of noticeably rise? JOE: Yeah. I actually remember several instances at my job where -- this is when I was working in office and in the same room as all my co-workers. And one of my co-workers would know that I was reading his review by the number of sighs that were coming out of me. [Laughter] JOE: Yes, I definitely had this reaction that people could tell. Implementing things like these emojis definitely helped. They didn't always fix everything, though. One thing that would happen too is even after filtering out kind of the noise comments and getting into the meat of the matter, we would still wind up in situations where I would want to use one design pattern and another engineer would come along and say, "No, you should do something different," and we would have these arguments. CHARLES: Right. And those are harder to resolve too because those are strong opinions strongly held. JOE: Yes. CHARLES: But it's still a step forward, right? You've eliminated the passionate arguments about the strong opinions weakly held. And so you're able to now grapple with 'how do you resolve these arguments' of these stronger opinions that are more strongly held. JOE: Yeah. Part of the way that we handled this was we came up with a way of being able to have a debate about something on a pull request, yet also keep in mind that we're working under deadlines and that ultimately, it might not be that big of a deal. TARAS: I don't know if you experienced this but there's this kind of a feeling where you know you have a deadline that you want to reach and this feedback is kind of not really helpful to that end goal. But at the same time, there feels like a trade off for you, like trading off quality by accepting something that people don't agree being fully ready. JOE: Yeah. TARAS: But then there is, I guess the tests seem to be like you have some tests and the tests are passing. So, thumbs up. [Laughs] CHARLES: Yeah. Well, here's the thing. This thought just occurred to me when I heard that is when we are reviewing code, probably the most important code to review is actually not the implementation but it's the tests. Because why do people get so care mad about the code being right? It's because they know how expensive it is to get it wrong. And we all have our different opinions of what's right and wrong and what's going to cost us more. But at the end of the day, we're all trying to save us, save ourselves and our team, pain in the future. The tool that we have to do that is the code review. We're trying to make sure the code is "good". But usually what ends up happening, and I know I'm certainly guilty of this, is I'm focusing on the implementation and I look in and say, "Oh yes, there are tests." I don't really look at the test and say, "Is this a good test?" Because a good test is going to lower that cost that we're so afraid of in the future, and that drives us to want to make sure the code is as close to what we conceive of as perfect as it's likely to get and still hit our deadline. It sounds to me like maybe what we really ought to be doing is reviewing the quality of the test because if you have an excellent test, then the shape of the code almost is unimportant or changing the shape of the code, more importantly, is very cheap. So if you do find that the implementation is suboptimal, the cost of rejiggering it into a better configuration is going to be very low. JOE: I've discovered so many times where you are running a test that was written previously either by somebody else or by yourself, and you realize that the test is testing the wrong thing entirely. CHARLES: Yeah, coverage is spotty. It covers 10% of your cases. JOE: It can also might have a stub or a mock in there somewhere that is hiding something that you weren't counting on. Or it could just be you pull up this screen and test then it saves but you're not really testing what it's saving. CHARLES: Right. Can I actually proffer a suggestion? At least I'm going to try and hold myself to going forward, is that when I review a pull request or some proposed change to a code base, I'm going to review the test first. JOE: Ooh, that's a great idea. CHARLES: I'm going to try and start with the test and avert my eyes from the implementation, no matter how curious I am how this person accomplished the solution. I'm going to hide from the solution and focus on the verification. And if I have faith, first and foremost, in the verification, then my faith in the implementation is secondary. TARAS: You know, Charles, I really like this. I think it'd be really interesting to try this out and see what kind of impact it creates. It makes me think that there is actually something really useful in here in terms of providing feedback because the challenge is that when you're giving feedback to someone on a pull request, ideally that feedback is going to be useful and it's going to give the person something to think about in regards to the work that they did. And I think a good way of pushing to work back onto the person in a way that allows them space to internalize what they need to do is identifying cases that might be missing because a lot of times, we test happy paths. We don't test the things that we didn't think about. And that's where the devil is in details. It's in the things that we don't test. Those are the kind of problems that we're trying to prevent with having good code. But really the best way to kind of flush those things out is to make sure we have the right test coverage for it. So yeah, I think if you add to that just when you're looking at a pull request is to go, is a test coverage missing the following cases and actually surfacing not. I'm actually thinking like a danger task. A danger is a tool for adding information to pull requests so that it's kind of like an automated mechanism to comment on pull requests. And so I'm thinking like having something that does what [JAS] does. [JAS] has this thing where they will check based on the last git commit. It will figure out what are the tests that have been added since the last commit and then it will actually show you. So when you run tests, it only runs the tests that you kind of impacted which is something that could be used to surface tests that were written in this commit. I mean, we also have that information in the actual commit itself but being able to see 'these are the tests that were written or added' and then you could use that and figure out what else is missing from this. This could be like a way to kind of add onto this process, something a little bit more automated so you could actually highlight this information very easily in the commit itself. CHARLES: It's funny, my mind was wandering off towards GitHub. When do you use danger and when do you use GitHub actions? GitHub actions have been kind of like brewing and fermenting in my mind. TARAS: One nice thing is that I mean, danger is a little awkward in that it creates one single commit at a topic. It creates a comment first time you commit and then it keeps pushing information into a comment which shows up at the top of your comment thread. CHARLES: Right. TARAS: But that's not how you read comments. You read like top to bottom. You don't read top to bottom to top. And so, I think the nice thing about the actions is that because it's using checks like GitHub checks that actually is part of a different area, it's part of the validation area as opposed to part of the actual comment conversation. This seems like a more appropriate place to put their information rather than the first comment. CHARLES: Yeah. So, do we decide on kind of what the ultimate resolution is for when you have these high order conflicts? JOE: Yeah. So that's one thing that we came up with that I'm really proud of. And so what we do and at least at that job that I was at, what we did in those situations was we said, "OK, if you've gone back and forth on this two or three times and there's still not an agreement as to what should be done or there is no compromise, what you should do is let the author of the pull request go ahead and merge that code," because you want to merge that code, get it out to production, make sure it's serving our users and our clients. And then what you want to do is take that conflict and record it somewhere to specify this interpersonal conflict not a code conflict or a merge conflict or anything like that. So you take this interpersonal conflict, the topic that was being discussed and you put that -- we were just putting it in a Google doc and leaving it there. And then every couple of weeks or maybe every couple of sprints, we would go through that Google doc and just talk about topics that had come up. And then really more often than not, what we would find is that we didn't care about the topic anymore. CHARLES: [Laughs] It's amazing how time has a way of cooling passion over things that really don't matter. JOE: Yeah. And for the things that did matter, we usually have like a really good discussion. Sometimes, we even came up with something different entirely based on just having more people in the conversation and thinking about this problem. TARAS: I like this because there's a bit of process around it. It's kind of like a retrospective on pull request passions. JOE: Yes. TARAS: It sounds like a healthy thing to do for a team, especially. And I think just allocating some breathing room to go through these kind of things could be really important cumulatively over time, especially. CHARLES: Yeah, definitely. It sounds almost like the pull request is then ongoing. You have this collaboration that happens kind of at the front of the change but then is rippling outwards and onwards and hopefully then has an impact on future changes, like if you're disagreeing. So, would you qualify that the types of issues that end up living in this Google document as architectural issues or just, 'hey this is the way we need to talk to each other and this was off and we need to fix this' or is it both? JOE: It was mainly more architectural decisions that was coming up in this, sometimes like code style issues as well. Not so much the interpersonal issues, like the interpersonal issues would come during retros. CHARLES: OK, because it sounds to me like it's not. Then you have the added benefit that your architecture does get to improve because clearly if there's some disagreement about this, then there's some sort of tension there. There's someone perceiving that some problem is not being resolved by a particular implementation. And so it's good to just at least surface this issue again and again. What was your experience with kind of revisiting the architectural issues? Was it mostly 'well, this wasn't really an issue' or is it 'wow, I'm really glad we took note of this because now we have these three ways of thinking about things' and it turns out that given three or four months more experience, this is something that we should be doing. JOE: Yeah, it is definitely a mix of both but I'm kind of leaning more towards the latter. Like we're just glad that we bring things up. Occasionally, there will be one or two topics that will come up that we still would resolve and we would have to agree to disagree on something. Or in some cases, I think we would allow one of the lead engineers to just make the final decision on something. But that was pretty rare. CHARLES: So, apart from appealing to either time, the passage of time or just kind of taking a brain dump of all the different architectural options or deferring to a more senior engineer and just kind of asking them to step in and make the call, are there any processes that you can put in place say to kind of mechanically come up with a decision? Like any set of values that you can use as a ruler to kind of line up a set of things to kind of attach weight to one particular solution? For example, one of the things that I always think of is when I have internal conflict about a decision, there's kind of a part of me that feels like this might be the right way and maybe another way is the right way. And so, a tool that I will use is what's internally consistent with the rest of the system. So, even if I've had an insight that something really ought to be framed,one solution should be framed in a certain way. If there's another solution that's more internally consistent with the way things are existing, then I'll use that as a discriminator to say, "OK, that's one thing that I can use to measure a solution against, that I'm not emotionally tied to." It feels more objective. And if you have those tools, you can kind of pile them up and see which pile ends up being higher for each solution. Do you see what I mean? JOE: I definitely lean more towards keeping things consistent with what's already there. Personally, I probably do that to a fault. Sometimes I need to branch out a little more and get comfortable with possibly breaking something. But you always need to weigh whether it's something that is acceptable to break. Like a lot of the systems I've worked with involve money and it's really important that people either have money when they need it or are charged the correct amount when you charge them with things. CHARLES: They get really mad when you charge them too much. [Laughs] JOE: Isn't it amazing? [Laughs] CHARLES: Yeah. JOE: It could be that you are OK with breaking this other end of it where you might charge them the correct amount but the fulfillment of that product or the service or whatever you're selling, you might have a little more wiggle room in that part of the code base. Where if you break something, somebody calls in and says it's broken, and then you're just able to fix it. CHARLES: Is there anything else? If folks are struggling with the way their pull requests work or their code reviews work and they're experiencing friction and tension with their teammates, is there anything else they can do? JOE: What I would suggest is definitely look at the pull request template feature that is in GitHub and make use of that. Because what you can do with that is come up with several sections of the pull request template and say, "These are the questions that we want to ask before we submit a PR." And if you have those questions and those sections right when you go on that template and people read through that as they're making their pull request, they can often catch problems before they get to another reviewer. I can't tell you how many times I have submitted a pull request, started to write the description and [inaudible] these questions and realized, "Oh, this is wrong," or it's broken or it's totally the wrong thing. CHARLES: Right. It's funny and I feel like you want to be asking those questions not when you're submitting the pull request but before you're even writing the code. JOE: Yes. CHARLES: But a lot of times, we don't do that until afterwards and you're like, "Oh man!" Just this process of forcing myself to think about the problem in this way or think about it holistically totally recasts my implementation. JOE: Yeah, things like, "How can this break after we merge it?" [Laughter] CHARLES: Yeah, that's actually a great question, that of one you're talking on pull request. JOE: Oh yes, definitely. CHARLES: Like imagining the [inaudible] scenarios. JOE: Yep. CHARLES: Man, it's almost like you want to read the -- what are the other questions that you have? JOE: So the ones that we have are, first of all, what is this pull request? How does it fulfill the requirements for the ticket? And then how can this break after we merge it? Are there any post-merge tasks that need to be run? If you need to get on a server and run a task or do something after it has gone to production, that's a place to document that there. And then the final question is, how is this tested? CHARLES: Right. That's a good one. And like I said, I think that's the one where I'm going to be starting when I'm both thinking about how my changes will be reviewed and how I review changes. JOE: Maybe I should consider moving that to the top position. CHARLES: [Laughs] I mean, it is like. I'm just thinking about it and it does seem like we go straight to the implementation because that's what's fascinating to us. And so we might have way too much bias for the importance of the implementation over the importance of the test because I have to say I am so guilty of when I'm reviewing. Just looking for the fact that it has a test and not looking for what the test does. JOE: Yeah. CHARLES: And only referring back to the test and trying to understand how the test accomplishes its task. If I don't understand the implementation, I'm like, "Oh, let me look at the test and see how this code is used to the test." That might be problematic but I know that's definitely a little personal goal that's coming out of this podcast for me. TARAS: Excellent. CHARLES: All right. Joe, do you have anything else that's coming up? Any talks? Any meet-ups that you're going to? Any announcements? Anything that you want to plug? JOE: I have a website at JLLeBlanc.com and that's where I am. CHARLES: All right. Well, fantastic. Thank you so much for coming on. JOE: All right. Thank you. CHARLES: Thank you for listening. If you or someone you know has something to say about building user interfaces that simply must be heard, please get in touch with us. We can be found on Twitter at @TheFrontside or over just plain old email at Contact@Frontside.io. Thanks and see you next time.

Sam joins the panelists to talk about frontend and backend team collaboration. Resources: Worse is better The Tao of Microservices by Richard Rodger Sam Joseph is a CoFounder of AgileVentures, a charity that helps groups of volunteers gather online to develop open source solutions for other charities all around the world. Sam's been mucking about with computers since the early 80s and followed the traditional education system through to a PhD in Neural Nets. Next he went all industry, researching mobile agents at Toshiba in Japan, going freelance and then swung back to academia to research peer to peer system and collaborative systems. He now spends the majority of his time trying to make AgileVentures a sustainable charity enterprise, with occasional moonlighting as a contract programmer. Check out his blog at nonprofits.agileventures.org. Please join us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at contact@frontside.io. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer. This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. TRANSCRIPT: CHARLES: Welcome to The Frontside Podcast, the place where we talk about user interfaces and everything that you need to know to build it right. My name is Charles Lowell, a developer here at the Frontside. With today also is TARAS: Mankovski. TARAS:: Hello, hello. CHARLES: Hey, TARAS:. Today we're going to be continuing our theme when we think about UI platforms and web platforms, continuing the theme of collaboration and with us to talk about this is Sam Joseph. Welcome, Sam. SAM: Hi, thanks for having me. CHARLES: We've already talked a great deal about how the way in which your team collaborates and the communication that happens between your team and between the different pieces of software, your system, form one of the pillars of the platform that you can't just take lightly. You need to actually be intentful about that. I was thinking we could kind of start today's discussion, kind of talking about some of those collaborations. One that we've probably all encountered, which is usually teams will be split into people who are focused on frontend, people who are focused on backend systems, kind of the services that make sure that all of the nodes that are running on our laptops and our desktops and stuff are running smoothly and error-free and obviously, those two groups of people can sometimes arrive with different sets of priorities and how do we resolve those priorities to make sure that that communication flows freely. TARAS:: What's interesting about these frontend and the backend teams is that our users are not seeing that separation. They only see one thing. They only touch one thing. They actually see as one group but there's tends of be this kind of split between the frontend and the backend. It's kind of interesting that how the user get into this. SAM: Yeah. Obviously in some teams, there's a very clear cut distinction between people at the backend and working with the components that are serving JSON over the API and there are some people who are very, very focused on the frontend and drilling CSS and a number of bits and pieces or even just staying explicitly on the design or UX design and there's a mythical full stack developer who is up and down the platform. It doesn't run exactly in parallel but there is this key thing which is almost how much sympathy or empathy can you have for another person who is not you, trying to use something that you set up. If there was a direct parallel, you'd say, "Obviously, all people who will be working on the frontend are more of that sort of person and perhaps, the people on the backend are not so much that sort of person," but actually, I think you can have people who are doing backend stuff and they're designing API is very, very thoughtfully or the kind of people that consumes those APIs and sometimes, you can have people who are very, very focused on the design and the aesthetics when not necessarily so plugged into how will someone else use this, how will it fits into their lifestyle, which might be very different from my own, so that's maybe another axis, if you know what I mean apart from this sort of pure technical [inaudible]. Does that makes any sense? TARAS:: Yeah. What's interesting is that everyone is trying to do a great job. Everyone is setting out to do something really good. What people's way of expressing good might be different, so if someone could be really focused on the quality of their code like they want to do their version of doing a really, really good job is doing the best code they could write. Sometimes that doesn't necessarily equate to the best user experience. I think everyone that I've met -- engineers who are writing code, almost everyone that I know is trying to do a really good job. If everyone is doing a good job, it doesn't necessarily always equate to the best user experience. CHARLES: I think we had a reference that everyone wants to make sure that their system is of the highest quality possible but quality in of itself is not an absolute value. It's relative. In other words, a good definition of quality is how well a system is fit to a purpose. If it fits very well to that purpose, then we say it's of high quality but if it fits very poorly to a particular purpose, then we say, "This is a system of low quality," but what constitutes something of high quality is relative to the purpose. The question is, what is the purpose of writing the frontend system? What is the purpose of writing the backend system? So it seems to me you're going to have a lot of dissonance if the purpose is divergent but if they both share the same purpose, then that is kind of standard quality on both sides. Does that make sense? SAM: That makes sense and what even more makes it tricky is that actually, purposes can evolve over time and so the thing with the frontend that needed to do with the business today is different tomorrow and the day after. The trick then is sort of the frontend and the backend, as people try to do a great their job, there's this question of how far ahead are we looking so you can talk about important things like sort of asking close modification about [inaudible] extension and there's a [inaudible] of different coding heuristics as best practice that say, you should kind of go in this direction. Sometimes, that will be the hill they want to die on. It's like, "This code needs to be this way because of this thing," and the idea is that it's sort of future-proofing them but I think the messy reality is that sometimes, the thing that you put in place to future-proof, the whole system gets canned. In two weeks, the business changes and what have you, so the extra effort that you are pushing in on one day to protect yourself against changes in the future actually gets lost and perhaps, if only you'd be able to deliver a shortcut pack, this one feature that that will got the next line of funding in, either one advocates cutting corners but... you know what I mean? Like --? CHARLES: Yeah, absolutely. I just wanted to chime in and vehemently agree with you that the purpose can change radically, especially if you're in an evolved environment. What constitutes the good code or the good quality solution can vary just as radically because it needs to fit that purpose. TARAS:: When we talk about frontend and the backend, it seems there's this relationship, which is kind of positional like there's the frontend and then there's the backend. One is in sequence from front to back. It's the direction with the frontend. It's closer maybe to the user where the backend is not. But I think in practice, it's probably more like the left and the right hand where when you have to do two things together, you actually need to do two things to coordinate together. What you describe about and this is I think where taking shortcuts sometimes can actually be a good thing is that if you say, "I'm going to spend the next few weeks on a backend and doing this specific thing," and then you find out that 10 days into the iteration that it doesn't fit, the better approach to synchronize would be to stop the action and realign. But if you persevere, then you might overstep and you actually be out of sync. It makes me think that the art of creating cohesive organizations from development perspective is to create context where the two kind of work in synergy together. I think that's where a good tooling that allows its synergy to exist. I think that's a lot of the work of actually creating systems where the user experience is kind of cohesive and integrated. CHARLES: Actually, there was something that you just mentioned in there but it's actually a little nugget that I'm curious to explore and that is if you're 10 days in to like a 14-day piece of work and you realize that it isn't right and it doesn't fit with the whole, right action is to throw it away. I feel like software organizations and this might be a little bit off topic but it's something that really is interesting to me, so please indulge me. I feel like we see the product being the kind of the code output, even in Agile environments and there is an aversion to throwing away code that has been written. There's the, I would say, incentive is to go ahead and persevere because developer time is so expensive. These are days out of people lives, so they've already invested 10 days. Why not just have four more days just so you have it. When in fact, you actually have more if you just throw that work in the trash because it's an obstruction that's not needed. It's a piece of weight. It's actually something that you now are going to own and it's actually going to be cheaper in the long run and it can be more beneficial to your organization to not own it. Do you all see that happen kind of play out where people become attached to software that they've invested and will make the decision to hold on to it? Say, we'll spend two more days to complete it, even though it's the wrong thing, like identifying which things need to be thrown away? I feel like we don't actually do that very aggressively -- to say, "You know what? We need to not complete this work." SAM: I think that is a really tricky one because I think whatever people might say, when you spend time working on something, you become emotionally attached to it. I would say strongly that how logical or rational or whatever sort of a person you are, my experience is being that people working on things want to see them used. I think in a situation with version control where you can keep things on a branch, they can be useful explorations, things don't have to be thrown away in their entirety. Our charity is doing work for the National Health Service in the UK where there's a lot of employees in Europe and I've noticed the user interface that we're supplying them and I was suggesting some sort of minor tweaks and one of the developers has sort of run with the entire, big, advanced search feature that partially solves the problem but brings in a lot of other bits and pieces. He does some good work there. It's a great work but I think he kind of ran further than I was expecting down that line and I think we have now found a much simpler solution that rather than bringing an entire cabinet, it's a little shading off the side of the existing one. But I think that doesn't have to be a loss and I was reassuring him that the work was valuable and that there is interesting learnings there and potentially, we might use it in the future version of the project and now of course, the way that stuff is moving on, just how it's going to branch, it doesn't mean you can then sort of magically lose some of the work. In that case of doing this 10 days out of 14 and so on, the real question is can you get any value from switching somebody that fast? If they spent 10 days in and they realized they don't need that thing, it's like can you switch them quickly onto something else where they'll do four days or might they as well, just finish up there and leave that on a branch. That's a pull request that gets closed and it's there to go back to. That's kind of depends team by team about how quickly you can repurpose people missed rigs, if you know what I mean. CHARLES: I absolutely agree but the key thing is leaving it on a branch and not integrating it into production, like not actually deploying it because I feel like when we see a feature, it's like we've got to get this thing done and it's done and we're just going to get it in versus now that we've learned how to do this, let's put that on a branch and let's rewrite it and let's take a different approach, rather than just being married to the idea that we're going to get it right the first time or that now is the time for this feature because we understand the complexity of what it actually takes. It is a tricky question. I'm wondering if our processes don't include enough implicit experimentation. We talked a little bit about this on a prior podcast that if they don't include some incentive to not deploy features just because you have them. TARAS:: I think from a business perspective, there's not a lot of model to evaluate a certain thing. We don't have really any effective way of evaluating learning. You can measure code by the numbers of lines of code that somebody wrote and how much of code was shipped but you can't really evaluate how much was learned and how much was persisted. I think a lot of this work around supporting effective collaboration is in building a cumulative systems of knowledge like why is a good Git history useful is because it has a built in mechanism for understanding history. It gives you a way to return back to time in your project and understand the context of that specific change and I think this is something that really good teams do this really well. They will respect this because when it's necessary, it is valuable to have this. When you don't have it and you are a year into the project and something happened and then, you are using the only thing you have, which is your troubleshooting skills or you are trying to figure something, as supposed to going back and relying on that history, on that knowledge that's built into the system, what I'm trying to get to is there is an element that is inherent to our development process, which is we don't have a way to quantify it. We have no way to really evaluate it. I think this is the problem that makes it difficult to throw away work because you can measure the amount of time they'll spend but you can't measure the amount of learning that was acquired. CHARLES: Right. Thatís true. SAM: I think if you have a positive team environment, if you have your team that is stable in there -- your contacts are coming in, the money is there, everybody is there and you've got the team, you don't necessarily be able to measure the learning because the output of the team will be good. They're kind of doing that in the background but I think individual comments are not going to want to pay for learning experience. It's certainly not at the rate of software developer's cost. Although, the throwing-aways, if you're familiar with the [inaudible] book, which I think is a [inaudible] is the author, you know, build one, to throw away, you will anyway. We have a frontend mob that we run weekly doing frontend stuff and CSS and so on and we've done like it's a mockup for the client and there's this question about in my [inaudible] and so Iím saying to other developers there, "Let's not get too attached to this. We may need to throw it away and it will be a great learning to sort of restart on that." The [inaudible] to look quite nicely and the client might get attached to it. They want to ship it and I'm like, "Well, actually there's a lot of other stuff that needs to happen," and so, it's a minefield. It really, really is. TARAS:: I think some of the business relationships that we have to this kind of client consulting company relationship, that relationship might not always create the fit to purpose team for specific challenge. You can have a company that has a lot of employees but their team and their team dynamics might not be fit to purpose to the problem they're trying to solve. If you're building a platform over a long time, you want to be creating that space where that learning gets accumulated over time. It doesn't matter what necessarily the relationship is between the companies that are participating in this process. I think what matter is what are you producing as a result. If you're working together with people from different companies and they're working together and they're building this kind of an environment where you can collaboratively build things together and then people learning from the output and that knowledge is being carried over and people are being able to stack their knowledge continuously over time, if you're creating that environment and you're building a large platform for example, then you are creating a build to purpose kind of technology team to fit what you're looking to accomplish and that'll include consulting companies. I think those team, they're kind of secondary but I think it's just [inaudible] to that. It's like building a quality development organization to fit the purpose of building whatever it is that you're trying to build. If you're building something small, you might have a small team that'll able to build that effectively. If you have something big, you could have a big team that is building that effectively but building that team in a way that is appropriate to what you're trying to accomplish, I think that's the real challenge that company struggles to do. CHARLES: Yeah. In the context of these real teams, I guess what can you put in place given that you have backend teams, frontend teams and each one of these needs to be specialized. This is kind of the power of the way that people work is that we're allowed to specialize and there's power in specialization. You can have someone who can be super focused on making sure that you have these high throughput backend systems that are resilient and fault tolerant and all these wonderful things, so that they don't necessarily have to have the entire context of the system that they're working on inside their head at a given time and the same thing someone working on CSS or working on a frontend system architecture, which has grown to be a very complex problem domain in its own right. But these teams can be working in a context with a purpose is whip-lashing around and changing quite drastically and so, how do you then keep that in sync? Because we've identified purpose as being actually something that's quite a dynamic value. How do you keep these teams keyed in and focused and so, that they're kind of locked in on that similar purpose of they're going to be adapting the systems for work that they're responsible for and specialties that they're responsible for to match that? SAM: It's a good question and I have my own bias view there -- CHARLES: I would love to hear it. SAM: I can't bear working on things where I don't understand in great detail how that end user has experienced or what is the effect overall that it's trying to be achieved. You know, I've been working in boards between the industry and the charity for like 30 years and I know people are commenting like, "You were the person who wants to understand everything." I think there were some people who are maybe quite satisfied who get ticket off the general board or what have you and work on that thing and if the API specs have filled in, so that's it. Go home and they weren't losing a sleep over it. But I think if youíre going to be dealing with these changes and sympathetic that the changes is coming down the path, I think you need to have some degree of empathy with the people who are driving the changes. Do you know what I mean? CHARLES: Oh, absolutely yeah. SAM: You know, as we've mentioned already, I think people get attached to what they build and I don't think you can do anything about that. They will get attached to what they build. CHARLES: Yeah, we've all experienced it. It's so true. It's impossible to [inaudible]. SAM: Yeah. We can try and target it but it's sort of part of our nature. I think there are sort of the tools of the design sprint, of the design jam, of these sorts of things where if you can build to some level is obviously that it can't be shipped but can actually tease out the needs of the user, the designs sprint, the Google team, they have an example with kind of like this robot for hotels where it's like the whole thing is they can have robots that can deliver toothpaste to your door if you run out of toothpaste. In this design, obviously, that would be a huge undertaking to make that as a sort of our production system but they used like a remote control of a robot to simulate all of the key touch points when the client of that hotel, will actually interacts with that robot. I think the same is true when you're building these systems if you can, again going to touch back on we don't fit a code in that way that I think an ever better thing is not be building the code in the first place and to build an almost a non-code system and then, maybe some of the backend are not going to be interested in the results of what people have learned through this kind of user experience trial before they [inaudible] forever but I think it's seeing other people try to use the end system that puts you in place where you can be empathetic. I argue for the bias point of view that I think people on the backend of Netflix, trying to optimizing the streams in that or the other, they need to spend some time watching Netflix or at least watching the comedy of Netflix in order to empathize of how their work relates to the end experience and then, when those end experience needs changed, it's important for them to make change on the backend. Do you know what I mean? CHARLES: Right, exactly. They have to watch through that kind of glass window where they can actually help the person. They can't give them any information. The only levers of control they have is through their own work, making the thing changes that they need to make on the backend, so that one of those users is going to have a good user experience. This idea reminds me of something which I believe is the case at Heroku where they rotate everybody in the company through pager duty, which I think is kind of a brilliant idea where the entire team is responsible for providing technical support to the end users. When a problem arises, you get to understand what it's like to be someone who's trying to use the system. You get to exposed to the satisfaction, whether an issue was resolved or when it's working correctly or your pager is quiet for your entire shift but you can also get exposed to the frustration that you're engendering in the users if something doesn't go quite according to plan. SAM: Yeah. I'm not [inaudible]. TARAS:: There is movement in this direction because there is actually a term that has been floating around. It's like a product engineer. It's someone who thinks about the product. These kind of people tend to have the highest value in Silicon Valley as someone who thinks about the product as the outcomes supposed to their code as the outcome. Even in the Agile space, there seems to be movement in that direction because I think one of the challenges, like to do that, you have to understand how you fit into the bigger picture. I could see it being really difficult. Imagine if you're working at a bank or something and being pager duty at a bank would be impossible, simply because their organization is either too big or too sensitive. The way that their company is dealing with that is they have these group of people where you have a product manager working closely with the frontend engineer and the backend engineer so there's this exchange that is happening where people get to understand the consequences of their actions a little bit more. They understand how things fit together. There is definitely a movement happening in that direction that I think just the more, the better because one of the big differences now is the things that we make are very palatable to the users and the quality of the user experience that users are now expecting is much higher than they were in the past. I think the world is changing. CHARLES: Yeah. We mentioned this when we were talking before the show but going to the University of Michigan as I did, I was in school with a bunch of mechanical engineers. Their goal was to go work for auto companies: Chevrolet and Ford and everything like that but their mindset was very much, I think the product engineering mindset. All of them loved cars and wanted to be part of building the coolest, most comfortable, most responsive cars. For whatever definition of a good car -- there are a lot of them -- they wanted to design good cars but they were into cars and they were into the experience of driving a car, so what's the equivalent then of that product engineer in software? I think that every conversation that I had with any of the mechanical engineers who were going into the auto industry, I'm sure there are some of them that were out there but it wasn't about carburetors or whatever. They really wanted to just get into the building of cars. In some aspects, I'm sure they all integrated in some way or another but it was a group that was very focused on that outcome. SAM: In the UK, I'm hearing this term, 'service designer' a lot that's coming up. Are saying that, Charles, you get the feeling that even getting into software, I'm not so excited about using their own? Like the car, I want this amazing car and then I can drive the car and I'll experience the car --? CHARLES: Right and other people will experience the car that I've helped design. SAM: Right and in some way, in software, it's almost like the software itself there's this kind of mathematical beauty of its own and it's like always been more important than the other software heads around me like the software that I wrote. I mean, the user? At the conference and all the software guys love it, that's the key objective, isn't it? Yeah. I work with a lot of learning developers who are rightly focused on wanting to improve their skills and wanting to level up in particular tech stacks that are the ones that will lead to jobs and the future and financial stability and so on but I kind of wish I could inculcate in them this desire that the user experience was the higher goal than which bit of code does this or the other. Maybe, I'm being unfair to them when I say that. Coding is hard. There's a lot strange concepts to grasp than sort of grappling with those concepts -- how does this work or why does this work or should I use this function or should I use this method or what have you. I don't know. I think [inaudible] bit a wall when I'm sort of saying like, "Let's think about it through the user perspective, what does the user needs here." It feels like they want the safer answer of what's the correct solution here, how should this be refactored because you need a skinny controller or a fat model or whatever happens to be. I don't know if that's -- CHARLES: Right, like what's going to make my life easier, in a sense of if I'm going to be maintaining this code and it's important. It's very important. You want to be happy in your job, you want to be happy in your work and so, you want to have the skinny control or the fat model because it's going to lower the risk of pain in your future, supposedly. I definitely understand but that can't be the only incentive. I think it's what I'm hearing. The thought just occurred to me, I actually don't know that much about game development. I know the game industry is certainly has got its problems and I don't know much about the development culture inside the game industry. I do wonder what it's like because it does seem to be somewhat analogous to something like the car industry, where if you're a developer in the game industry, you're probably extremely focused on the user experience. You just have to be. It's so incredibly saturated and competitive for just eyeballs and thumb on controllers. Like I said, I don't really know anything about the game industry when it comes to development but I'm wondering if there's an analogy there. SAM: Yeah. It sounds strong to me and as much of my cousin who works in the game industry and I sort of taught game programming and work through a lot of it. The game industry has got these sort of user testing experience baked in and it kind of like repeat, repeat, repeat, repeat. There's this sort of constant cycle of doing that and in the somewhat wider software industry, in a certain extent, pay lip service to that. CHARLES: The game industry, they live and die by that. The code lives and dies by how it actually plays with real users. SAM: Yes and it feels like in the general world, there's a lot more user interfaces that they just sort of struggle on whatever market dynamics people need to use, these existing vested interest, these banks, these supermarkets and what have you. I guess the market is too big, almost. It's not covered enough but do we really want our industry to be so cutthroat? I don't know. CHARLES: Yeah. We've kind of seen an example of a couple of industries: the car industry, the game industry which is kind of adjacent to where most software development happens but they have this concept of exhaustive user testing, kind of the pager duty if you will, where you get to experience that. So how do we, on our team and when I say our team, I mean anyone who happens to be listening, what's the equivalent of pager duty for the applications that we write? How can we plug ourselves into that cycle of user-ship so we can actually experience it in a real and repeatable way. SAM: In the work that we're doing with the NHS, they have a similar program. There's a lot of people who are working purely death by desk jobs but they do have a framework for us to go and observe in the hospitals and emergency rooms and so on. I guess the 'how can we achieve that outside of those clients who have those framework in place,' it seems like maybe we need a version of the cycle where a different person gets to be the product owner and tries to represent what the users are experience each week. I think almost though, it seems like we need more of that thing you mentioned, Charles which is like kind of being behind the one-way silvered mirror as some sort of framework that connects the loop between some of the individual developers are doing and their experience of how the users are seeing things. I think that's going to be difficult to introduce. Just to sort of back up a little bit with them, I see this kind of idea of Agile, which says, "Let's be using the software which is the thing that's kind of changing and evolving unless you already deployed and you are in the maintenance mode from the beginning and you're getting that thing out there so you have your one-week or two-week or three-week cycles where you keep on having touch points," and I think that's better than touching once every two years. As malleable as modern software potentially is, it's too sticky. It's like you were saying Charles before with that deploying or whatever, these are all gyros and so, you kind of need a really good mechanism for mocking out interfaces and having users experience and there's a couple of [inaudible], vision and there's something else that's sophisticated systems on the UX space where you can put together sort of a simulation of your interface relatively cheaply and you can run these things and then have sort of video capture of the users interacting with the system. I think the difficult part that we have in the software industry is we are in this thing, where there's not enough people wanting to be software developers, partly with I guess the cars and the games, is there's so many people want to be game designers that the industry can kind of set the terms but we're in this inverse situation here with software developers where software developers are so much in demand, they can kind of say, "Don't put too much pressure on me. I'll kind of like, I'll go off to different company." You know, we kind of have to like herd the cats, as they say into allowing these high powered software developers to go in the direction as they want to go in and so, it may be difficult to impose something like that, where you're getting software developers to really experience the end user's pain. I guess what one has to do is somehow, create a narrative to sell the excitement of the end user experience and the beautiful end user experience to software developers such that they're fully out of themselves and say, "Yes, I want to see how the users are using my software." Do you know what I mean? CHARLES: Yeah. TARAS:: Yeah. Because a lot of company had this process where there'll be a product manager, there'll be a designer, they'll design through mock ups and they'll just hand it over to developers to build. There could be some backend, some frontend. I think if you're doing that, there is no emotional engagement between the creators of the actual implementation and the people who are going to be using that. One way to do that is to try to add more of this actual 'however you do it.' You know, introduce more of the personal experience of the users to go along with the actual mock ups so that people understand like, there's actually be a person on the other end that are going to experience this and create some kind of emotional engagement between these two end. How you do that, I think that's a big question but I think if the organization is simply just throwing designs over to development, that's where the part of the problem is and trying to -- CHARLES: Well, actually, that's a fantastic point because ultimately, we've talked about emotional engagement and attachment to the software being a liability but the reason it exists and the reason it's intrinsic to the way we do things is that's how things get built. We get emotional attachment to it. It's the impetus. It's the driving force. It's literally the emotive force causes us to go forth and do a thing. It's why we're going to do it as a good job. Like you said TARAS:, if you just kind of throwing your tickets over the wall, there is no emotional engagement and so people will look for it wherever they can find it. In the absence of an emotional engagement, they will create their own which is good quality software that's 'clean' code because people need to find that meaning and purpose in their work. Maybe the answer is trying to really help them connect that emotional experience back to that purpose of the user experience, so that there is no vacuum to fill with kind of synthetic purpose, if you will. SAM: That's a great point. The charity in AgileVentures that I run, when we get things right, that's the thing that happens in that. We go for transparency at open source. We have these regular cycles where the charity client is using the software in a Hangout with us, with the developers who work on these things and the developers whether they're in a Hangout Live, whether they're watching the video a week later, they see the charity end user struggle with the feature that volunteer developers have been working on and they make that attachments. It's more than just 'I want to learn and level up in this thing.' It's like 'I want to make this feature work for this end user' and we're very lucky to have these charities who allow us to do that level of transparency. The difficulty often comes that I would see in our paid projects where I would love to be recording the key stakeholders using the system but for whatever political reason, you can't always get that. I think that process of actually connecting the developer with the person who using it and doing that reliably, so that they can have that empathy and then get that emotional connection. It's just tricky in the real world. CHARLES: It's very hard. One thing that we've deployed on past projects, which I think has worked fairly well is kind of putting a moratorium on product owners writing stories or writing tickets and actually having the developers collaborate with the product owners to write the tickets. In other words, instead of kind of catching a ticket that's thrown over the wall, really making sure that they understand what is the context under which this thing is being developed and then almost as in a code review sense, having the product owner saying, "Actually, the reason we're doing it is this," and having a review process where the developer is actually creating the story in support from the product owner. Because ultimately if they have that context, then coming up with the implementation is going to be much easier. It's really is about facilitating that upfront learning than being able to do the actual work. That's something but a lot of organizations are resistant to that because the product owners really want to say like, "No, I want it to go this way and I want to just hand this to a developer and I want them to do it." It's kind of wrestling in control and saying, "You've got veto power over this. You're the editor but we really want to make sure that you're on same page with the developer." In cases where we have been able to kind of have product owners make that shift where the developers are owning the story, oh sorry, or the primary authors of the story and more of the code reviewers of the story, if you will, implementation just seems to go so much more smoothly. The questions, the key points kind of come out of the front of the process and by the time you start actually working on the thing, the manager or the product owners has the confidence that the developer understands what is involved in making it work. SAM: Getting that done, what one will say something as tricky to do, I think in the ideal world, you have more of the developers involved in the design sprints or the design jams. The logistics of it are you can't really afford to have all those developers in all of those soft meetings where they are coding away. That sounds a great medium there. I think there's various organizations that do sort of their kick offs where their stories are kind of if not code designed, then there's cooperative voting on the complexities of the stories and making sure that they have folks understand the stories that they're working on and there might have some very different organization that are going to have different constraints on how much time that the organization feels, the different people can be allowed to spent on different sorts of activities. It's sort of tricky, isn't it? CHARLES: It is definitely tricky because any time that you allocate for people, that's the most expensive resource that you have, so you want to be smart about it. SAM: Which comes back to that issue then again of repurposing people. If you've got that feature that you go over 10 days, can you say to that person, "Can I switch you over onto this other thing for four days?" Maybe logically, that would be a better outcome for the sprint but maybe emotionally, that person is so attached to that. They are not fighting that front. The biggest trouble for me, I think over the last 30 years is I assumed that the logical stuff was tantamount. These things like the skinny controller and the fat model, we'd agreed on that that was correct and so, that's why I should pushed for, that's why I should fight for because it's the truth or whatever and actually, maybe I'll sling back around in another 30 years, you can choose your battles because the level of emotional pressure on people, then the whole thing just explodes and nothing gets done. You know what I mean? TARAS:: Sam, I have experienced something very similar and I think about this all the time. It's just how many perfect things are perfectly acceptable to people in using a different lens when I think about technology because the more skilled you are, I think quite often, people become more pedantic about how they approach things but in practice, the more things that you see that are not written but you realize how really imperfect they are and how in many ways, they fit the world very well, people use it. It's being used by many people in its imperfect state, so there is something like this hole between perfect implementation and the role of this perfect implementation in the world that I think that duality is really interesting. CHARLES: Yeah. There's a fantastic paper written by, I think it was Richard Garfield back in the late-90s or early 2000s called 'Less is More,' where he kind of talks about this exact tension and he calls it the MIT school of software and the Berkeley school of software and I think Garfield came from the MIT school but basically, the whole thing was saying the Berkeley School is actually right and the name of the paper is 'Worst is Better.' It's a really interesting essay but just talking about working things that are in people's hands will beat the best design every single time because those are the systems that get used and improved. There's a lot more to it than that. He talks about this exact fundamental tension and obviously, no system exists on either one of those perfect poles -- the MIT school or the Berkeley school. I think what he was trying to point out is that exact fundamental tension that we're talking about, the quest for the 'correct solution' and the quest for a solution. I'm actually have to go and re-read it. I remember it being an intriguing paper. SAM: I guess the thing that makes me think of is sort of related to philosophy value. I read recently about this, attachment to outcomes. I'm going to segue back but we've got this discussion in one of our teams about, this is sort of microservices versus monoliths and in reading this, I doubt microservices, which is actually pretty radical in some of its suggestions that they're talking about it in Greater Than Code Slack where it's sort of saying that actually, if you keep your microservices small enough, then a lot of the things like code quality become actually kind of irrelevant it sort of almost argues that a lot of the things that we like about the Agile and these things are our ceremonies that are only necessary because we trying to feed these monoliths. I'm not saying it is true. Iím just saying it's a pretty radical position that itís taking and it certainly, captures the minds of some people in our organization. Some things that I've been trying to make some simple changes just to smooth off some of the edges of the platform that we're working with and I had this respect of like, "No, we can't just make those simultaneous. We need to move to microservices." They're great and it will be perfect and they will allow expansion and so on and my immediate reaction is sort of, "I'm glad we don't have to build microservices because [inaudible]. It's all about attachment to outcomes, so am I attached trying to automate part of my week that I think will have some positive goals in the future? No one's got a crystal ball. That's only a guess on my part. If I've got some folks who are excited about doing this thing with microservices, maybe I should empower them in that and I should started a mobile microservices and we kind of playing with these set of framework and so on. It's interesting stuff as I'm working my way through the book. At the same time, while we've been doing that, I've actually delegated it to someone else to sort out this sort of thing and I've actually kind of addressed the problem what we would need the microservices for through a different mechanism but still, the microservices thing rolls on and I kind of think, "Well, is that all a complete waste?" But then actually maybe, in two years down the line, it will turn out and that will be a beautiful thing that will enable things. The further that I go on, the more I say, "I just don't know," and actually, if I can detach myself from caring too much about the outcomes one way or the other, I think it's both long and enjoy myself but it's a tricky thing when you got to pay the rent and pay the bills and so on. I don't see any resolutions to that. I love people being able to use things and get stuff done. I'm so excited about that and I guess, I will keep pushing all of the developers that I'm with towards trying to have a better empathy or understanding of their end users because I think software is more fun when you're connected to the end people who are using that. CHARLES: Absolutely. Microservices reminds me kind of the experience that we've had with the microstates library. Because we've kind of identified this one very small slice of a problem, a 'refactor,' it's basically a ground up rewrite. If you've got a library that is a couple of hundred lines of code but it presents a uniform API, then you can rewrite it internally. You can refactor it by doing a ground up rewrite and the cardinal rule or the cardinal sin is you're never supposed to do a rewrite. SAM: Yes and that's the microservices that they're saying. It's like you should rewrite everything all the time, basically. CHARLES: Exactly. You should always be rewriting it. SAM: Yeah. Should be able to throw it away because it's a microservice and it can be rewritten in a week and if you're throwing one away, it's only a week worth of work and you can keep on moving away. It is an amazing idea. CHARLES: I have to read that book. Anyhow, we're going to have you on again to explore -- SAM: Well, let's do another one on microservices. It was a great fun. Definitely, it's time to wrap up but I really appreciate you having me on the show and being able to discuss all these topics. CHARLES: Fantastic and I can't wait to talk about microservices. This might be the impetus that I need to finally actually go learn about microservices in depth. SAM: I'll recommend Richard Rodger's book, 'The Tao of Microservices.' CHARLES: All right. Fantastic. Anything we should mention, any upcoming engagements or podcast that you're going to be on? SAM: Iím always on the lookout for charities, developers, people who want to help, you can find us at AgileVentures.org. We're busy trying to help great causes. We're trying to help people learn about software development and getting involved in Agile and kind of like experience the real software in action, software development where you ideally interact with the end charity users and see how they're benefitting from the product. We love any support and help. You can get involved in that, if you want to give a little bit to open source and open development. We go for transparency. We have more mob programming sessions and scrum and meetings online/in-house every day, so just come and check out AgileVentures.org and maybe, see you [inaudible]. CHARLES: Yeah and if they wanted to say, reach out to you over email or Twitter, how would they get in touch? SAM: It's Sam@AgileVentures.org and I am at @tansakuu on Twitter. Hit me on Twitter or just at Sam@AgileVentures.org. CHARLES: All right. Well, fantastic. Thank you, Sam. Also, if you need any help with your frontend platform, you know where to get in touch with us. We're at @TheFrontside on Twitter or Info@Frontside.io. Thank you, TARAS:. Thank you, Sam and we will see everybody next time. Thank you for listening. If you or someone you know has something to say about building user interfaces that simply must be heard, please get in touch with us. We can be found on Twitter at @TheFrontside or over just plain old email at Contact@Frontside.io. Thanks and see you next time.

Jacob joins the panelists to talk about team collaboration based on his RubyConf 2017 talk, Code Reviews: Honesty, Kindness, Inspiration: Pick Three. Jacob Stoebel is a software developer living in Berea, KY. He spends his days writing web applications in Ruby, JavaScript, and Python, working with data, and leveling up as a software engineer. He works and studies at Berea College. You can find out more about Jacob at jstoebel.com. Please join us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at contact@frontside.io. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer. This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. TRANSCRIPT: CHARLES: Hello and welcome to The Frontside Podcast, the place where we talk about user interfaces and everything that you need to know to build it right. My name is Charles Lowell, a developer at the Frontside. With me today from Frontside also, is Taras Mankovski. TARAS: Hello, hello. CHARLES: Hello, Taras and today, we're going to be talking like we do every time about a piece of the platform that you used to develop user interfaces frontside at your company or organization or wherever it is that you build software. Today we're going to be talking about a piece of the platform that's very, very critical that often gets short shrift or is excluded entirely from what people think of when they think about their tech stack and that's how we as teams collaborate to build and maintain and produce the quality software that we can. With us today is Jacob Stoebel. Welcome, Jacob. JACOB: Hello. CHARLES: Now, what is it that you do, in your day-to-day? JACOB: I'm a full-stack developer for a little company called ePublishing and I mostly work in Rails and React. CHARLES: Rails and React and so, when we were searching for people to talk about how we collaborate these teams, Mandy suggested you because of a talk that you've given at a RubyConf, specifically about code reviews, which I think are actually a huge piece of the collaboration process because it's a major forum where team members get to interact with each other and it's the gateway for making sure code quality is maintained but more than that, I think it's a learning -- a place where we learn. I learned so much both as a reviewer and as someone who is submitting my work and so, it's actually a very important part of the software development process. You have a lot of great examples of how to not do code reviews. JACOB: Yeah, I think I may have been a little bit too indulgent in that talk. I had a lot of fun. I did some research from other people, mainly from anecdotes. I had research from talking to people about really, all the anti-patterns that come out of code reviews. It seems like every few weeks, I'll see a tweet that says something along the lines about how code reviews are broken. I don't really know about that and I have to say, I think I'm kind of lucky at my job that I think they're done in a way that really leaves me feeling pretty positive and that's certainly a good thing but I think what it comes down to -- I'm going to sort of talk about where these ideas come from in a minute -- is that we often have code reviews that for one -- and you can tell me how this is for you too -- often the code review is happening at a point so late in the process, where the feedback that you get may not be actionable. Have you experienced that? JACOB: Before I answer that question, just to kind of echo the sentiment and maybe I'm being presumptuous, I feel like the code reviews that we do are actually very positive, so I haven't got to experience firsthand. Although I have seen conversations on GitHub where it looks kind of like a Celebrity Chef, where you have someone doing the code reviews like Gordon Ramsay up there just screaming and someone has put this plate of food in front of them and kind of picking it apart. That one is extreme but this is actually something that I struggle with, what you were talking about, what is the appropriate point at which to get feedback. I agree that you want to get feedback as soon as possible and sometimes, when you've invested weeks and weeks into something or you're like at Mile 100 and they're like, "You know, at Mile 2, you were supposed to turn right," and now, you're off in the forest and you've been tracking 98 miles in the wrong direction. JACOB: Yeah and the work is due, right? This needs to get ships tomorrow. CHARLES: Right, so you've got massive pressure. This is something that I struggle with myself is when is an appropriate time to really try and be public about what it is that you're doing. JACOB: Yeah. I think that is a really good question and I think what you're getting at and I would agree is that, the sooner, the better and when you can tighten the intervals between feedback is probably better. I'll just take a step back and I'm going to take a longer route to go and get to my point. I am a career changer and before I was in this career, I was a high school theater teacher, so it was really different and I won't give answer why I changed other than this is the greatest [inaudible]. But one thing that I really struggled with is I was working with teenagers and I really wanted to see them grow and improve but at the same time, these were kids and they have fragile egos and I don't want to tear them down, so I came across this really interesting framework for feedback. It is called the Liz Lerman's Critical Response Process and I give credit to her in the talk. This comes out of the dance world. Liz Lerman is a pretty accomplished dancer and choreographer and what she found is that in the dance world and I think it's not too dissimilar from our industry is that the feedback received was often given in a way that was leaving people feel really torn down and mostly, not feeling inspired to go back to their work and make it better. It really felt like feedback is about giving you a grade. It's like, "I'm going to tell you how good of a job you do," and there's certainly a time and place for that but the inspiring question -- I guess the rhetorical question -- that she made is, "Shouldn't the big point about feedback be that it makes you so excited about the work you're doing that you just can't wait to go back to your keyboard and keep working on it," and she found or in the case, back to the dance studio, it really sort of structures this framework for how people can give feedback to a creator and that could be a creator of anything. You mentioned cooking. This could be about food as well that sort of set some guardrails for how we can give feedback that is useful, it's inspiring and it's kind. I'm going to really distinguish between kind and nice. Nice would be, here I'm going to say things that are only pleasant about your work but what I mean by kind is feedback that is really taking care of your team and making them feel like they are respected and cared for as human beings so they don't go home every Friday afternoon and cry on their couch and just drag and coming back on a Monday. That's kind of the basic idea of why we need, maybe a kind of a framework for getting feedback. CHARLES: I agree totally. It's almost like you need to conceive of your feedback is not a gate to quality but a gift to embolden somebody. It's like they've just been doing battle with this code, with this problem, they've been grappling with it and it needs someone to wipe their brow and maybe give them a stiff drink, so that they can get back into the ring and be invigorated. JACOB: Yeah. TARAS: What I'm hearing in this conversation so far is it's kind of like a tone or way of communicating to the person receiving the feedback but sometimes, no matter what your tone is, depending on how your team is set up, depending on the context of your actual code review, it can still kind of land in the wrong place? Have you experienced that where the team conditions impact your ability to actually provide feedback? JACOB: I think I know what you mean. I think what you're talking about is this sort of the organizational structures that are set up are sort of lend themselves to certain modes of feedback and discourage others. I will give this example that I've heard from numerous people and I think this is what you're getting at is feedback that's given at the end. Just like I said, feedback that is given the day before, it has to be shared. Or feedback that it's almost like if we work in an environment where sort of like the hair on fire environment where it's like everything was due yesterday, that's not an environment that's going to be conducive to slowing down, taking a step back and saying like, "Let's point out what we think this work is doing? What questions we have about it? Who has opinions about the direction that's being taken on it?" And really zoom out a little bit more. CHARLES: Do you have any concrete examples of how that early stage feedback has taken place at your work? Is it just over Slack? Maybe, the whole people need to step back from the idea that working on one-week iterations or two-week iterations and at the end of the iteration, everything is due and everything will get merged at that point. How do you kind of break up that structure to say, "No, we're going to try and have checkpoints and milestones?" What are deliverables that you can decompose the work into that fit inside the framework, that are inside the big deliverables, which is sensibly, the feature that you're working on? JACOB: I think first of all, it's the way this thing goes about. I don't think this framework works over Slack. I think it has to be immediate, I think it has to be either in personal or on Skype or Hangouts or something. One of the points of this type of feedback is really about checking in with each other as a team and I think what's great about Slack is that it lets you leave a message and walk away and the unfortunate thing about Slack is it's not meant for sort of attaching how people feel about things or what people's reactions were to them. I think you all have to be in the same space, hopefully with your camera on, if possible and really, just sort of checking in with each other as a team. I can point to times with my team that I think that's really worked out. I think you made a good point when it comes to really getting started with a project because there are times where I made the mistake of not trying to get feedback from my team early on when I was going to start a project and as you can imagine, did that really cost me? It's really getting feedback from my team and my supervisor about the direction this is going to want to go and I'll say from experience, I work on legacy codebases and as you can probably imagine, it's easy to paint yourself into a corner and what the thing about legacy code is that you don't know what pitfalls you're working into. You can get started and you can sort of find into the process, there is some reality about this code that you didn't know and it is really getting in the way of your work that had you known about it, it would have saved you a lot of trouble because you could have planned your way around it. Then the fortunate thing is drawing on the wisdom from people who they know about it because they struggled with it already if they've been around longer than I have. I think that's a really good point. This is a good thing to do. As you're getting started and you can be sharing this is my just initial idea of how I'm going to go about this. What my tech stack is or my understanding of the problem space, all of the above and to really sort of check your assumptions and see if they really check out. CHARLES: I want to circle back a little bit to something you'd mentioned before and that is you want to be kind in a code review and I would say, you probably want to be kind anytime you're giving feedback or interacting with your teammates but what's the process that you can go through if you find yourself struggling? How can I deliver this message that I want to deliver and have it come across this kind? What's a process or checklist that I can go so that I don't open my mouth up and say something that I can't take back or that is going to land wrong? JACOB: I'm glad you asked that and I think that this is a great way to introduce the guidelines for the critical response process. There's basically four steps to it and the way it's structured is you have the person who has created the thing. It could be the person or the team that has created the feature. You have other people who are responders to it. They could be people within your team or they could be others in the organization who are invested in your success. They're not here to tear you down or give you a grade. They're here because they want to see the project do well and then, there's going to be someone who is a facilitator. The way it works and I think this is directly addressing your question is there are guardrails that are going to help you not steer into territory that is going to leave someone feeling torn down. The first step to this process is all the responders are going to say statements of meaning about the work and what I mean by that is you're going to make things that stand out to you that are not attached with an opinion. You could say, for example, "This project is using React hooks. That's something I noticed." You're not going to say, "I think that's a good idea or a bad idea," but you're going to say, "This is what I'm noticing," and that is something that can get jotted down because that's going to be fodder for discussion later on like, "Let's talk about this new feature in React," and let's talk about it later. We can talk about if we think that's a good thing or not or what the implications are for that. The next step after that is that the person who has made the work of the team, they get a chance to ask questions of everybody else about what they thought. You've probably noticed that in a lot of pull requests, someone puts their work out there and then, the next thing that happens is everybody starts commenting on the work and saying what they think about it. This flips it. At first, you as the creator, start asking questions or you can say, "This thing over here, I think it's maybe a little wonky or it's a little hacky but I couldn't think of a better way to do it. What do you think? Do you think it's right? Do you think it's a bad idea?" Or this bit over here, "I'm pulling in the third party library. I think maybe, it's worth it but what do you think? Is it not worth it?" CHARLES: I like that because ultimately, the people who have created the thing are the most familiar with the problem space. They've spent a lot of time thinking about it and so, they can actually direct the conversation to the parts of the implementation that really are the most iffy and the most unknown because everybody is going to feel this need to comment but it's the classic case of bikeshedding really, the true experts or the people who've just been spent all this time implementing and so, people will comment up to their greatest point of familiarity with the problem, which could actually be not that great. Can you say like, "Can you really focus your mental energy on this?" I like that a lot. JACOB: Yeah and really, it sets a good tone. The assumption behind all of this is that the creator, like you said knows the work best and really ought to be in the driver seat when it comes to feedback, so it really sets the tone there and say like, "The creator knows what's most important. Let's give them the first opportunity to frame that," and I know plenty of times like if I'm asked to review a PR and I'm not given any kind of prompt or direction, I will probably scroll to the file. Maybe I'll find the file in that PR that I'm actually familiar with or I'll look for some pattern that's something that's like, "I understand what's going on here. I can give feedback," and if it's all that other thing over there, I'm completely confused, I'm going to ignore it. But the creator could illuminate me a little bit, then I would understand a little bit more and then, I'm in a position to comment on that thing that I otherwise wouldn't have understood. That's the second step. The third step is now the responders get a chance to ask their questions but the important thing about that is that there are neutral questions. The assumption here is that the responders need to better understand the context from which this code was written in order to be able to give their opinion. Here's an example coming from the Rails world. I could say, "Tell me your thought process for using Factory Bot," in this example. If anyone doesn't know that is somewhat of a contentious issue in the Rails community, rather than just coming out guns blazing and say like, "I think this was a bad idea." It's like, "Tell me your process because I want to understand the context you came from and then we can together evaluate if coming from that context makes sense or if it doesn't," so neutral question. They have to be neutral questions. By the way and we've probably all heard this before, this is not a neutral question, "What were you thinking when you decided to do blah-blah-blah-blah-blah." Everyone knows what that means. CHARLES: Right. I was going to say, you can be very, very passive-aggressive with questions. JACOB: Yeah, exactly and this is a place where having a facilitator can really help -- facilitator who is not one of the creators. Someone can just say, "Let's back up. Can you rephrase that without the opinion embedded? Can you just say that again?" and with that again, those are some of the guardrails that keep us on track. After the responders have been able to ask their questions how hopefully everyone has a better understanding of the context from which the code was written, then it's time for opinions with consent of the creators. The way it works is the responders can say, "I have an opinion about using React hooks in this codebase. Would you like to hear it?" The responders can say, "Yes, please. Let me know," or they can say, "No, thank you," because the responders, having the best knowledge of the context, might know that that feedback is not useful. Maybe, they have a really good reason for using React hooks or maybe, they know what's coming in the future or maybe, they know if there's no time to fix it now or it's not worth fixing now. They know the tradeoffs best and so again, those are guardrails and it puts the creator in the place of saying, "That's actually not useful feedback right now. Let's just not use it." They can say, "Yes, please tell me," or they can say, "No, thank you. Let's not talk about that." CHARLES: In the context of a pull request, the process you're describing could be played out in any number of media but in the context of a pull request, the creator is the person actually submitting the code, how do you handle the issue of who pushes the merge button if there's still some opinions that haven't been voiced? For example, a creator says, "No, I don't think that's helpful feedback," is the assumption of then the creator can go ahead and just push the merge button? Or is it basically saying, "I don't want to hear your opinion," relatively rare? JACOB: That is a great question. I think I tried to get at this in the talk. Before one thing, I am probably, like most people don't have the option to just say to my manager, "No, I don't want to hear your opinion." I get that. There is a contrast between the pure version of the feedback process and then reality. They have to balance. But teams can sort of work out the way they give feedback. I have example of an anecdote that someone shared with me once, when I was doing research for this talk. There was this sort of agreement with the manager that their manager wouldn't give feedback on Friday afternoon. They just wouldn't. Everyone preferred that. Everyone sort of wanted, say on Friday afternoons, I'm really going to be just focusing on winding down for the week and I don't want to get dumped on a bunch of feedback. The point being, even though you can't just blanket-ignore feedback, you can work out circumstances with your team for the best way that feedback can be given and circumstances under which, it can be politely declined. CHARLES: I see. TARAS: I'm curious about a different part of this because a lot of this is how to give feedback but I'm really curious about the why part. I think many of us take it for granted that good code reviews are very valuable because a lot of teams that I've encountered that are taking on really big challenges but didn't have a code review process and so, one of things I'm kind of curious is for people or for teams that don't have it in place, what kind of symptoms can they observe in their daily operations that would suggest that maybe code review is something that they need to put in place. CHARLES: Taras, you're talking about kind of the places where we've seen where the culture is just push everything to a branch, there's a 1000 commits there, open up a pull request and no description, no name. It's like, "Here's this thing. I'm taking comments for the next three hours and if everything goes well, let just merge." Are you talking about those kind of situations where really a big culture of pull request or just feedback around change is very, very nascent.? TARAS: Yeah, a lot of times, it's the 'ship it' culture, like this is getting in the way of shipping it. CHARLES: So you're saying like how you sell the entire idea? TARAS: Yeah and if someone is listening who is noticing that, a lot of people would know that we're not really doing code reviews but what are the symptoms that they could be observing that could say like really, this is we need to change, like this can't continue. JACOB: Yeah. One thing that occurs to me as if there's all high level of surprise when you read it, when pull requests are read, it's like, "Oh, you went in that direction." I think that could be an indication that maybe, we could have checked in at the halfway point or even sooner because there seems to be differences of perspective on where this is going or where it should end up that's why [inaudible]. TARAS: At what point do you think people would see this? Is this something that would happen kind of way down the road? Like actually, "How did this end up in the codebase?" CHARLES: That's surprising except deferred even further, right? JACOB: Yeah, who wrote this last few. In having that kind of feedback process, let's sort of take a look at where this project has come in the last few months and see if we can sort of learn from what went well and what didn't. CHARLES: This is related to the concept of surprise, if you see a proliferation of many different patterns to accomplish the same thing, that means the communication is not there. People are not learning from each other and not kind of creating their own code culture together. If that's missing and that manifests itself in all kinds of ways, in bugs, in weird development set up that takes me two hours to get this local environment set up and things like that, if you're seeing those things, chances are you need some way to come together in a code review culture is really, really good for that. JACOB: Yeah and I think in the ideal code review culture, everyone that's sort of brought on board is saying, "I am going to share responsibility in this patch, doing what it's supposed to do and not breaking anything or not burning down the world." You mentioned the 'ship it' culture, I could imagine toxic cultures where the person who shipped it, if it broke everything, it's on them to fix it and they have to get woken up or whatever and the idea about feedback is now we're sort of forming a community around what was done, so it's like the person that push the merge button isn't the only person involved. It's like if we have a culture where we say everyone that participated in the PR is collectively sharing the consequences and that will happen eventually and say like, everyone who's on this thread, it's on all of us if something goes wrong to fix it. I think that is certainly something that one would hope you'd see. TARAS: I'm curious, where do you see this kind of observations usually come from because sometimes, I can imagine there, being a developer, coming on the team and they're like, "Why wouldn't I do code reviews?" and they're like, "Well, we have always not done code reviews," but then there could be someone like a product manager or somebody who's like, "Why wouldn't I do code reviews? We should start code reviews." Have you seen ways of introducing these ideas to teams that have worked out well? JACOB: Yeah and I should probably say that, I'm not a manager, I'm certainly not an expert in how this works so I actually don't have a really great example, personally. I can churn example from working in a previous team, where everyone secretly wanted to be more collaborative but didn't know how to do it because if I'm the first person that puts myself out there and no one knows else knows how to collaborate with me, how to reciprocate, then what was the point? For the top 5% of teams that are just really have great energy together, they don't need a framework to do this sort of thing. They're just doing it naturally. I think for everybody else, we need some kind of guidelines to do this because for better or worse, this is the industry that we're in right now. It isn't built around us. We don't know how to function this way and it's no one's fault. It's just sort of that's the way we've all sort of learn to work in this industry and I suspect we're not the only industry like this but we need some kind of guidelines to do it. TARAS: Maybe it's a difficult question to answer. It varies probably from team-to-team. JACOB: Yeah. CHARLES: Yeah and maybe, we can kind of shift the question just a little bit because I have one that sits alongside not quite the same question but I think related and can bridge it and maybe we can find the answer there. We've talked about a little bit and there's definitely more to unpack there, how to give feedback that's kind and honest and I think to... What was the third? JACOB: Inspiring. CHARLES: Inspiring, that's right. What about from the flip side? This is something that actually comes up with us as consultants but I think it's something that other people will encounter in their jobs too, is what do you do when you are the creator and you're trying to present or share and ultimately solicit feedback from a stakeholder, the CEO of your company, one of your clients, one of your customers and you see them engaging in kind of a mode of feedback that's less than constructive. They're nitpicking on things that aren't really important and maybe, this could be completely and totally inadvertent. Their intention could be that they're trying to help you out but it's really not being helping at all and it's kind of like either tearing you down or just not being productive and it makes you feel... What's the word I'm looking for? Just brings an air of contention to the conversation that's not really helpful to producing the best result for everybody involved. How do you, as the creator actually engage with those people in a positive way and kind of help establish the guardrails and be that agent of change when those guardrails don't exist in their minds yet? JACOB: Yeah, how do you do it, right? It's probably rare that there's going to be an environment where someone's going to say, "We're going to do this framework for feedback," and everyone are onboard from Day 1. I think one of the things that you're probably getting at is the frustration that happens when people start giving feedback. You have this perception that they're giving you feedback that's unuseful and they're only giving it because that's the only thing they can think of or -- CHARLES: Exactly. They got to say something, they need to contribute their two cents to the conversation and some people are trying to be helpful and just aren't. Some people are just trying to appear smart. JACOB: Yeah and it's like, "Oh, boy. They're just really off." CHARLES: But you can derail the main narrative that you're trying to establish and get off in the weeds, kind of skirmishing with these people and after that happens, you're like, "Wait, no. I don't want to end up over there. I was trying to tell a story." JACOB: When I gave this talk once, one idea that someone threw out was when you make a PR, mark up your own code first with everything that you want to draw people's attention to. I think you were getting at this as like, "One frustrating thing is when people getting feedback seem to have less invested than you do." They sort of just flying by and dumping on you and they probably, actually couldn't care less and that can be frustrating. When you're engaging with people that maybe don't know how to get deeply involved in it yet or maybe don't have the time to, you can sort of gently nudge them to sort of like, "I want to talk about this part of it." It's almost like you're giving the answers to the test, which was like, "If you want to be part of the smart conversation, comment over here," and I think from the responders perspective, just speaking personally, I would appreciate that. It gives me an opportunity to feel like I'm actually being useful. We can all probably point to times where all the code review we have to do feels like homework that isn't the best use for our time. It's like, maybe the responder can make us feel like our opinions matter and they will be put to good use when you tell them, "If you comment here, it will probably be put to good use." I think the sort of the way to get started is the responder can just say, "I would really like feedback over here," and I can't speak for everybody but I would suspect that more people than you think would be more than happy to be gently guided in what feedback they should get. CHARLES: Right. I'm thinking how you would do this, for example in the context of a demo that you're giving to stakeholders because there's maybe the inkling of the idea to do that but it's often presented as an apology like, "This screen doesn't work," or, "Your handling isn't quite right. Sorry we're going to fix that." Look at the stuff that's over here and maybe, the way to frame that is a really hard problem based on the legacy architecture that we have for displaying errors and I'm not quite sure how to do it in a robust way and I could really use some feedback there. It's an area of exploration or something that we really need to focus on. You put that out there as I'm demoing this main functionality right now. JACOB: Yes. You sort of say like, "Here's something to watch out for and please, let us know what you think," and then after you could say like, "As a team, we thought up these three possible solutions but we didn't want to move on them because we wanted to get your feedback first, so please impart on us what is the right way to do it." You know, flattery goes a long way. TARAS: I have this imagery coming up in my mind as I'm hearing you guys talk about this that I keep seeing this kind of difference between a line and a triangle, where a line is kind of a tug and pull between, "Did I do this right? No, I didn't do this right." I think those are kind of a bad code review because it's very personal. It's not really where you kind of want to go and then the other way is like a triangle where the end goal that you're trying to get to is somewhere that is beyond both places where you let two people: the recipient of the code review and the giver of the feedback, the end goal that you want to get to is actually someplace else. When we deliver the code and we say, "My code is done," then it invites this kind of linear feedback where it's like, "No, you didn't do it right," but in the other way, "This is where I got to so far. Tell me where you think we could take it next. If you don't think we need to change anything, then we're done. We can merge this." JACOB: Yes. The very intelligent Jessica Kerr said recently on another podcast, there's no such thing as done but you can always make it better. I think you're getting at that point. That's a great question, by the way to ask of your responders -- where should we go next? Where do you see this going next? What will make it better? What would make it more robust? What would make us happier six months down the line when we're looking back on this? But I think of it as the firing range. That's the analogy I gave where it was like, "You put a pull request out and you assert that it is done and if no one can find fault with it, then it's done," and I just don't think that that makes sense. I don't think, really most people actually want to work that way. It's maybe not necessarily even healthy but I think everyone can find, at least I hope, a process that invites them to come into the process and share the way they see things. It can hopefully be enriching and just make everyone feel better along the way. TARAS: This sounds to me like the inspiration part of this conversation because one thing I really like about working at Frontside, I'm an to experienced developer but I know that Frontside is dedicated to doing something much greater than I can do as an individual. When I ask, "What do you think about this?" then I know the feedback is going to be because there something always that is just beyond the reach that could be a little bit better than what we have right now and it's not until I can actually get the feedback from Charles, from Jeffrey and hear, "What do you guys think? Is this it?" and they're like, "What about this?" and I know that the next step is going to be a little bit or maybe even a lot better than what I have right now. I think that's the part that inspires me. I know I have actually gone a little bit further beyond my personal ability to get us to that vision of like this being much better than we could do as individuals. JACOB: Yes and as opposed, "Oh, I screw it up and now, I have to fix it." The framework of find all the errors that I made, even though there may be errors. Let's be honest, there could be things that would be really bad, that would be a security problem but for a growth mindset, to think about it, it's like, "This is a way to make it better." CHARLES: One of the things that you can do to help that is try and find inside every change, try and see the potential that hasn't yet been realized but is enabled by this change, like what exciting paths does this change unlock in your mind, right? And then you can share that. That's a great way to 'inspiration is infectious,' so if you can find inspiration to change, then you might be able to share that with the creators. If something is very personally exciting to you when you see something, maybe spend a little time searching for what you find to be exciting about it. JACOB: Yeah, nice. That's great. CHARLES: Yeah, we might have touched on that. I just bring that up because so often, I'll see the changes that people on my team are submitting and sometimes, it can feel like a cloud burst of like, "We could do this or we could do this or we could do this," and it's great to get excited. JACOB: Yeah. I can share -- recently, there was an example. My coworker opened a pull request and it unlocked something for me where I said like, "You know what? This is making me think about this other thing that I've always hated doing with our legacy codebase. We should do that thing you're doing more because boy, would it make life easier?" and I wonder if it would be worth the time in refactoring X, Y, Z because then I would never ask you A, B, C again and I would be so much happier. TARAS: It makes me think that we need to revisit our pull request template like what would that like? What would be the sections on a pull request template that would facilitate this kind of way? CHARLES: I don't know. I know we're almost a time but before we even get into that, this is a question I have because we talk about pull requests templates, how do you match the amount of process with the scope of the change? Because it's probably a little bit heavy weight if I want to fix a typo and say, "Now we're going to have the creators lay out their case that they're going to ask questions of the reviewers and then the reviewers are going to ask questions and once everybody's been able to write down things that they observe the questions that they have, completely and totally divorced from opinion, now we can talk about opinion that is welcomed." If you're capitalizing one letter, that's probably a little bit heavy weight but on the flip side, it's probably absolutely warranted if it's a major feature that's going to be affecting a huge portions of your revenue stream and so, this is one of the problems I have with pull requests templates is they are one-size, fits-all. Sometimes, a pull request template feels like it's supporting you and sometimes, it feels like the epitome of busy work and I have to spend all this time deleting the sections for the pull requests template. I wish there were ways you could choose different pull requests templates. Maybe, there are. JACOB: So do I. CHARLES: That's a little bit of a quibble that I have is the amount of ritual and ceremony that you have to go through is fixed with a pull request template but it seems like you want to match the amount of process to the scope of the change. JACOB: Yeah, you do and the flipside is also joy. You don't want to give someone too little homework when you really needed the same work. Maybe there's a way to say, when a pull request is opened, the person who opened it or the manager or maybe someone else, has the option to sort of tag the pull request as saying, "This is going to be a bigger conversation. We cannot merge it until we have a face-to-face conversation first with these various stakeholders." Maybe, one of the questions, the first form that everyone gets for every PR gets is what level of PR is this? Is this a quick change? And what people that you just need some quick feedback on? Or is this the long form that you need where there needs to be a sit down with these people? TARAS: Actually, I was thinking what's the adjustment that could be made for pull request is to say, "Here's what a complete pull request looks like. You can opt into whatever section you want," so you decide based on your pull request, what the actual sections of this template are appropriate but then someone can, on the flip side say, "Look, I would really like to learn about the motivations of this pull request. Can you add motivation section?" and understand that from your pull requests. JACOB: Absolutely. You filled out Section A, please answer Section B and C. Yeah, cool. TARAS: Yeah. Because I think what part of the challenge is that we have people look to us, especially people who have a lot of experience, or developers look to us to know what is the right thing to do sometimes or often and I think it's helpful to be a little bit more gentle and saying, "You can decide what is the right amount of detail to set the right conditions for this code review but I will give you some directions for what are things you can consider in including. JACOB: Absolutely. CHARLES: All right. We're a little bit over time, so we should probably go ahead and wrap it up. You are absolutely right, Jacob. This is a topic that keeps on giving. We didn't really move much beyond the pull request and feedback and stuff but we moved a lot around within that topic because it's a big one. Jacob, is there anything that we should mention? Any upcoming talks, meetups? JACOB: No, I have a one-year old and it's all about work and family in this part of the life, so I have nothing to speak of at this point. You can come find me on Twitter as JStoebel. CHARLES: Okay, awesome. Well, thank you so much, Jacob for coming and talking to us. This is a very rich topic. We only really scratched the surface. That's it for our episode today and we'll see you next time. Thank you for listening. If you or someone you know has something to say about building user interfaces that simply must be heard, please get in touch with us. We can be found on Twitter at @TheFrontside or over just plain old email at Contact@Frontside.io Thanks and see you next time.

Sponsors Sentry use the code “devchat” for 2 months free on Sentry small plan Clubhouse CacheFly Host: Charles Max Wood Special Guest:  Charles Lowell Episode Summary In this episode of My Ruby Story, Charles hosts Charles Lowell, founder and  developer at The Frontside Software based in Austin, TX. Listen to Charles on the podcast JavaScript Jabber on this episode. Links JavaScript Jabber 337: Microstates.js – Composable State Primitives for JavaScript with Charles Lowell & Taras Mankovski Charles Lowell’s Twitter Charles Lowell’s GitHub Charles Lowell’s Frontside Bio https://devchat.tv/my-javascript-story/ https://www.facebook.com/DevChattv Picks Charles Lowell: Yousician App Charles Max Wood: Parade of Homes - St. George, Utah Vrbo.com

Sponsors Sentry use the code “devchat” for 2 months free on Sentry small plan Clubhouse CacheFly Host: Charles Max Wood Special Guest:  Charles Lowell Episode Summary In this episode of My Ruby Story, Charles hosts Charles Lowell, founder and  developer at The Frontside Software based in Austin, TX. Listen to Charles on the podcast JavaScript Jabber on this episode. Links JavaScript Jabber 337: Microstates.js – Composable State Primitives for JavaScript with Charles Lowell & Taras Mankovski Charles Lowell’s Twitter Charles Lowell’s GitHub Charles Lowell’s Frontside Bio https://devchat.tv/my-javascript-story/ https://www.facebook.com/DevChattv Picks Charles Lowell: Yousician App Charles Max Wood: Parade of Homes - St. George, Utah Vrbo.com

Sponsors Sentry use the code “devchat” for 2 months free on Sentry small plan Clubhouse CacheFly Host: Charles Max Wood Special Guest:  Charles Lowell Episode Summary In this episode of My Ruby Story, Charles hosts Charles Lowell, founder and  developer at The Frontside Software based in Austin, TX. Listen to Charles on the podcast JavaScript Jabber on this episode. Links JavaScript Jabber 337: Microstates.js – Composable State Primitives for JavaScript with Charles Lowell & Taras Mankovski Charles Lowell’s Twitter Charles Lowell’s GitHub Charles Lowell’s Frontside Bio https://devchat.tv/my-javascript-story/ https://www.facebook.com/DevChattv Picks Charles Lowell: Yousician App Charles Max Wood: Parade of Homes - St. George, Utah Vrbo.com

In this internal episode, Charles and Wil talk about testing issues and BigTest solutions. Pieces of the testing story are discussed, such as the start and launch application, component setup and teardown, interacting with the application and component, convergent assertions, and network. Then they talk about testing issues: the fact that cross browser and device-simulated browsers are not good enough, maintainability and when and when not to DRY (RYE), slowness and why (acceptance) testing is slow, portability and why tests are coupled to the framework, and reliability. Finally, they talk about BigTest solutions: @bigtest/cli to start / launch (Karma recommended for now) @bigtest/react, @bigtest/vue, etc for setup & teardown @bigtest/interactor for interactions @bigtest/convergence for assertions @bigtest/network in the future (Mirage recommended for now) Resources: Justin Searls – Please don't mock me This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. Transcript: CHARLES: Hello, everybody and welcome to The Frontside Podcast, Episode 115. My name is Charles Lowell, this episode's host and a developer here at the Frontside. With me today to talk some shop is Mr Wil Wilsman. WIL: Hello. CHARLES: Hello, Wil. WIL: How's it going? CHARLES: It's going good. I'm actually pretty excited to get to jump into this topic because we're going to be talking about some of the big things that are happening at Frontside and some of the things that we've been developing in almost for the last year. WIL: Yeah. It's been about a year now. CHARLES: It's been about a year and we've talked about it in various podcast but we're going to be talking about it again because there's just been so much progress that we've made, I think in a lot of clarity in kind of what we're going for here when we talk about BigTest and testing big and how we want to roll out the BigTest framework. We just have a lot more experience using it on a number of different projects, so we get to talk about that today. Before we get started, I just wanted to talk a little bit about what BigTest is, both in terms of the framework and also the philosophy. Wil, you're the one who works the most on BigTest. When you think about philosophically, what does BigTest mean to you? WIL: It's the size of your test, not a physical size like size and storage but how much your task actually does. The test itself can be very small as our test are but it tests the whole application from the user interacting with it down to the network requests. That's the definition of the philosophy of a BigTest to me. It's to tests your application from the biggest point of view. CHARLES: Actually, achieving that can be surprisingly difficult, especially in a frontend JavaScript application and there are a lot of solutions out there for testing and we've talked about them. One of the questions that arises is when we talk about BigTest, what exactly are we talking about? Are we talking about a product that you can download and install? Are we talking about the philosophy that you just outlined? Or are we talking about the individual pieces of software that make that philosophy real? I think the answer is we're kind of talking about all three but we want to take this episode to talk about where we're going with the product. What we've identified is the subcomponent pieces of that product. In other words, in order to get started testing big, what are the things that you need to think about? What are the things that you need to do? And then what are the component pieces? Because one of the things that I think is very important to us is that you be able to arrive at wherever you are in your project, whatever framework you are using, whatever current testing solution and be able to begin using BigTest. That means, you might be using some of it or you might be using a lot of it but we want to meet you exactly where you are, so that you can then, get onboarded and start testing big. WIL: Yeah. Definitely an important distinction that we get confusion about is what is BigTests and people just assume like this whole test suite is BigTest but we used the parts of it ourselves like we use Mocha, which is not part of BigTest. We use Chai, which is not part of BigTest. We use Mirage which is kind of part of BigTest but definitely it originate in BigTest and Karma and things like that. BigTest isn't your testing suite. It's not one thing to go-to to grab, to start writing tests. It is a small pieces that you can use in conjunction with other small pieces, just to make it really easy and flexible to test your application. CHARLES: Exactly. Because it turns out that there's a lot going on in the application. Maybe we should talk about what some of those pieces are that you might want to start using BigTest with or that you might need to test big, I guess I should say. What's a good place to start? Let's start with talking about some of the issues that you want to do when your testing big. Then we can talk about what pieces of the testing story that fit in to solve those issues. One of them is you need to test that your application works, like actually works. That means you need to be able to test on a multiplicity of browsers, for example. We're limiting to the domain of web applications. There are actually a shockingly large number of browsers. It's not just Chrome. It's not just Safari. There's Mobile Chrome, Mobile Safari, which are subtly different. There's Edge and I'm sure the Mobile Edge is slightly different too, so you want to be able to test cross browser, right? WIL: Yeah, absolutely and things like Nightmare and JS DOM and things that simulated browsers, we don't necessarily think those are the best tools for writing BigTest because we want to ensure that those browser quirks are caught and tested as well. CHARLES: This is not theoretical like sometimes you'll have a syntax, like the parser is slightly different and you have something that throws a syntax error in Safari or in the Internet Explorer and your whole app is completely busted. If you just take in the time, just even trying to load the app in that browser, you would have caught that. That's what I've been on many times. WIL: Yeah and what I just saw came up yesterday, which comes up frequently is not closing your CSS Selector and Chrome doesn't really care like web to browsers don't care too much but that will fail in Edge and depending on what you're missing, the failing is part of that too but mostly, Firefox and Chrome don't care about that kind of thing. CHARLES: Right. It seems like the majority of testing solutions are kind of focused around Headless Chrome or some variation of Electron. That entire class of really dumb errors has already been caught. Like I said, to actually catch it, it takes less than a millisecond of CPU time just to load it onto the browser and see that thing doesn't work. Unfortunately, they can be catastrophic errors but the problem is how do you actually do. We want to test like cross browser. This is something that we want to do. For me, I just can't imagine shipping an application without having some form of cross browser testing, some capability of being able to say, "I want to test it," like, "We want to work on these eight browsers and so we're going to test it on these eight browsers," but how do you actually go about doing that? WIL: Right now, we are working on the BigTest CLI which will help us launch browsers but that's not complete yet. It has some bugs on. For the meantime we've been using Karma, which is great. Basically, you just have this service that's able to find the browser binary on the system and just launch them pointing to local hosts with your app loaded up and your normal development server take care of loading the test up and running the test. Karma and the BigTest CLI is just there to capture output and launch those separate browsers. CHARLES: Yeah. I remember when I was first using working with Karma and I think Testim is another tool that's in this space. There's Testim, Karma and BigTest actually is we're developing a launcher because launching is something that you're going to need but it's such a weird problem. I feel like with the browser launchers, there's three levels of inversion of control because you're starting a server that then starts another process, which then calls back to your server, which then loads the app resources, which then loads the tests and then runs the test. There's a lot of sleight of hand that has to happen and – WIL: Including injecting the adapter that you use, like the Mocha adapter, the Jasmine adapter that ends up reporting back to the CLI. That's something that Karma and Testim and BigTest will handle for you. CHARLES: Right, so you're fanning out the test suite to a suite of browsers then collecting the results but basically, you need some sort of agent living inside the browser that's going to act on behalf of the test suite, to collect the results. I remember when I first came into contact with Karma and Testim, I was like, "This is so unnecessarily complex," but then, having used it for a while and I think there are some complexity that can be removed but if you want to do cross browser testing, that kind of level of ping-ponging is there's a certain amount of it that just necessary. It's something that's actually quite complex that you need to have in your stack, in your toolbox, if you want to truly test big. WIL: Yeah and all the solutions is mechanisms for detecting when the browser has launched and restarting the browser based on its health check, etcetera and things like that that you wouldn't think of actually loading up a browser but you need to think of when you're doing automated testing. CHARLES: What is it that sets apart, for example the launcher solution? We kind of call this class of solutions launchers, so Testim, Karma, the BigTest CLI. What is it that sets BigTest CLI apart from say, Karma and Testim? WIL: We're trying to be as minimal config as possible and just really easy to get started and going. Karma has a lot of plugins that you need to make sure you have installed and loaded in the options set for those plugins. Testim has some stuff bundled but it still requires this big config bulk at the beginning that you need to passing or that's all what you were doing. We're trying to avoid that with BigTest CLI and one of the ways that we're able to avoid that is by just letting your Bundler handle bunding the test. In Karma, you need Karma webpack or something. Testim has some stuff that it needs and really, we just want like in-testing mode. When you're in the testing environment, just change your index to point your tests, instead of your application and your Bundler will do all the work and we just serve that file and collect the results. CHARLES: Right, so it doesn't matter if you're using Parcel or you're using webpack or you're using Ember CI. WIL: Yeah, Rollup even. CHARLES: Or even just like low level Broccoli or Gulp or whatever. There's a preponderance of bundling solutions and that was always something that was just a huge pain in the butt with Karma. I know it's like just getting to the point where my tests are loaded and you look with Testim, most of my experience come with Testim comes through how it's used in Ember CLI like the histrionics that undertaken just to bundle all your tests assets and your application assets and your vendor assets and just kind of bootstrap that thing. It's a lot of work. WIL: Another thing with BigTest CLI that doesn't include in Karma and Testim does is a concept of a watcher because all these Bundlers, you have HMR -- hot module reloading, Rollup and things like that. It come with plenty of plugins. Parcel is always set out of the box, so if you're using your Bundler, your existing Bundler to bundle your test, you get that watch feature for free, so it's another complexity that the BigTest CLI kind of eliminates. CHARLES: What it means is we've hidden most of that complexity. Just let the Bundler handle it, right? The Bundler is the part of your project that bundles. WIL: Yeah. CHARLES: You should have your launcher actually doing that for you but we still do need to have some way to do that set up and tear down. When we have that testing endpoint, we have some way to say, "We're starting a test, not the application. We're ending the test, tear it down," so how do you abstract that away? WIL: That's kind of something that we can't really avoid. It is just like some sort of dependency on the framework itself, your application framework. It's like you need to mount a React app. You need to mount an Ember app, etcetera and there's different ways to mount those things. This is one of the things that can't really be decoupled as much as everything else can but BigTest has BigTest React and BigTest Vue and we want to eventually gets like BigTest Ember but really, the main export of all these packages is just a simple mount helper that will mount and clean up your application for you and your testing hooks, whether you're using before each from Mocha or before from something else like Jasmine. You know, no matter what you're doing, you just have a hook that mounts your application and then, cleans it up on the next mount. CHARLES: It's worth pointing out here that this is kind of a core concern of testing and testing big is being able to mount your application and tear it down with regularity and having hooks into that process. Whether you're using BigTest or whether you're not, can you still use BigTest React and BigTest Vue, even if you weren't using anything else? WIL: Yeah, absolutely. Like I said, they just export simple mount helpers. I don't even think they have any other inner BigTest dependencies. They just have pure dependencies on their frameworks. CHARLES: Right and so, you could use it, even if you wanted to roll everything else by hand or you wanted to get started somehow and you needed to do set up and tear down, again this is something that's key to being able to test big, so you should be able to use it independently, whether you use the CLI or not, whether you're using any of the other tools or not. All of the tools can be used independently. WIL: Then another feature of the BigTest React and BigTest Vue is the tear down that happens before set up, rather than happening after your test runs, having a separate tear down. This allows it. Whether your test passes or fails, you can look at it and play with it and inspect it and debug it much easier than if you had tear down. You have to disable at tear down or throw a pause in there to keep other or something. CHARLES: Yeah, I love that. When something goes wrong, you can just let the test case run and the last test that it runs, it just leaves at set up. It does the tear down right before the set up. WIL: Exactly, yeah. At the very end of the whole test run, there's an app there waiting for you to play with. CHARLES: If you focus in on a single test, we most commonly use Mocha, so you say a '.only' to run that single focus test, then you have the state of the application at that test case set up and ready to go. You can just play with it, you can inspect it, you can actually just use it as a starting off point and interact with the app normally as you would. WIL: I want to say, Cypress does this too. They do their tear down before they're set up as well. That's how you're able to play with Cypress test. CHARLES: Yeah, I like that trick. Now, we talked about launching, setup and tear down but we haven't actually talked about much of what actually happens in the test cases themselves. We talked about how to start and launch your test suite, how to do that across a bunch of different browsers, how inside of that, you have a separate concern as applications set up and tear down and how you want to lean on how you're actual app is actually bundled because that fits in with the philosophy of testing big. You don't want to use an external Bundler for your test suite. You want to use your real Bundler, how the asset is actually going to look. But when it comes down to actually writing the tests, you need to be able to interact with at the highest level as you possibly can. When I say highest level, we want to verify that the users, when they take certain actions, we'll see certain outcomes and so, we want those outcomes and we already talked about this to be reflected in a real DOM, in a real browser. But at the same time, the real interactions, we want those to be as high fidelity as possible, so you want to be sending events to the browser. You want real mount events, real key events, real interactions. WIL: Yeah, interacting with application. That's another core philosophy that we kind of talked about earlier that defines a BigTest. It's the user interacting with your application. We're not calling methods and expecting other callbacks or arguments to be passed or clicking on a button and expecting a message to pop up that says, "Form submitted successfully." These are user-facing things were starting on and acting on. CHARLES: Yeah and then, it can be really tricky because these things don't happen synchronously. They're happening inside of your browser's event loop. I click that button and then it goes off and there's some loading state and then, I might get an error message that pops up this thing that animates out and then, goes away. The state of the browser is in constant flux. It's constantly changing and so, it can be very difficult to put your finger and say, I want to be in this state if you are limiting yourself to only reading from the DOM. Some frameworks, Ember for example, you have kind of a white box where you can actually inspect the state of the Ember run loop and use that to do some synchronization but it can be very, very hard to coordinate these interactions. WIL: Yeah. You know, to talk about getting to the solution as a BigTest interactor, which is basically modern page components or page objects. If you ever heard of page objects, it's just a way to encapsulate interacting with big pieces of your pages. It's not a new concept. It's been around for a while but BigTest interactor has kind of a new twist on it where they're immutable, composable interactions that are also convergent, which we'll get into later, which basically means if your buttons not there, it won't click the button until it is there. They're really powerful and they're making really easy and fun to write these tests. CHARLES: Yeah, they're super powerful. I remember we talked about convergences last time when we talked about BigTest but interactors, I think are definitely a new development. I think we should spend a little bit of time there talking about, not just the power but also the ergonomics of interactors because they are like page components or page objects, except they're scope to the component. Not only do they have all this wonderful stuff where it'll make sure that the component exists before it starts to interact with it and things like that but their composable. If I have a button, then there are certain operations that are valid for that button. I can click it. I can hover over it. I can do all these things. They're the operations that make it unique to the button. Now, those might actually map to real events. WIL: Similarly, their assertions about that button as well, like as primary is secondary. If this button is repeated throughout your application, you might want to make sure that your form has a primary and secondary button. CHARLES: Exactly. It really encapsulates all the knowledge of how you can interact with both in terms of taking action and reading state from that button. It almost feels like an accessibility API. It would be easy to write a screen reader if you had these interactors for every single component on the page. WIL: That's kind of what it is. It's just like you're defining an API around how your user would interact with your application and what your user would expect in the application. That's the point of page objects and interactors as you're defining this user API, essentially. CHARLES: Yeah and so, really the step that interactor take is that they take the classic page object and it make them composable, so I can have, you kind of touched on this before, a modal dialogue interactor, which is composed out of two button interactors. One for the primary action, one for the secondary action and maybe, it's aware of its own title text, so you can assert on the title text but I didn't actually have to write the individual button interactors for that modal dialog interactor. Then I might have a second modal dialog interactor or a form that's on a modal dialog just composed of the modal dialog interactors and the individual form components, which appear on that particular modal dialog. WIL: It's essentially how we've been building applications lately with components but this is for page objects in your test if you want to mirror that. You don't have to have one-to-one mappings of an interactor to a component but if you do, it's really powerful. CHARLES: Yeah. I found that when we have one-to-one interactors, that's when it just feels the best. WIL: Yeah and on top of this, if you have a component library and your component library exports the interactor that it uses for the component test, like we said, this BigTest technology, they're sprinkled also. We don't have to use interactors in big acceptance tests. We can use them for smaller component tests too, so if we ship these component interactors with the component library, your application that's consuming this component library now can test those components for free, without having to write their own interactors. It can just compose the interactors exported by the library. CHARLES: Man, I almost want you to repeat that word for word again, just so it can sink in. It's so awesome. Because when you actually go to write your tests, you're not starting from ground zero like, "How do I do this?" They're like, "I'm writing some tests for this thing and I'm using these components and so, I've already got the prepackaged interactions for those components." It's like you start writing your tests. If your tests are a 10-story building, it's like you're starting on Floor 7 and you only have to walk up to Floor 10, instead of slogging up all 10 stories. WIL: One really helpful interactor that we work within the open source stuff we've been working on is a date-picker interactor because date-pickers can be really complex. Just having that common interactor and have a date-picker on multiple forms where we can just use that one interactor, we don't have to tell every single test how to interact with that date-picker. We just say pick date and pass the date. CHARLES: Yeah, it's so awesome. That is actually a great example. It doesn't feel scary to write a test for a page that has a date-picker on it or two. If you're doing like a date range or something like that, you're like, "Oh, my God. I don't write the selectors to test this." You just import your date-picker interactor, you set the date, it actually worries about all the low level events and there you go. It feels like you're operating at a much higher level. WIL: Yeah. The interactor API essentially, you're telling me the test what the user would be doing and what the user would be seeing. CHARLES: Yeah. It's worth pointing out again. We've identified starting and launching. We've identified set up and tear down but interaction is a core concern of BigTesting, no matter what tool you're using. One of the things that we found as interactors are something that you can sprinkle on literally any test suite if you're testing an interface and it makes it better. We've used it inside big acceptance tests. We use it inside Jest, doing just little component tests. There are people in the BigTest community who have used it to basically, write component tests against a JS DOM and while theoretically, philosophically, you want to make those tests as big as you possibly can, you can use that piece in your test suite. If you are using a simulated DOM and if you're running a node in a browser, these interactors will still work and you're going to get high fidelity test cases that are resilient to this asynchrony and are composable and if they do have a full-fledged test suite, you can reuse these interactors. They are a really awesome power up that you can bring into your test suite. WIL: And they are not tied to the framework at all. We use them in React for our stuff but we've also written some in Ember. Robert's written some in the Vue and ported some test and one of the beautiful things we've seen from this is that one interactor goes everywhere. You just write the interactor once and you can use it in Ember, in React, in Vue, in those test suites. If the rest of your test suite is framework agnostic, you have this test suite that you can jump frameworks in your test suites until it works and can test your application with high fidelity. CHARLES: Yeah, it's fantastic. I remember when we first tried using interactors inside an Ember test suite because Ember comes with like a big kitchen sink in testing set up but interactors just slotted right in and there's absolutely no issue. WIL: Yeah and there is actually a speed boost even because in most of the Ember test offers a hook into the Ember run loop and interactors are not. There is actually a good speed boos just using interactors. CHARLES: Yeah. This is a good point. It's a good segue because typically, we think of acceptance tests as being really slow and one of the reasons, even the people [inaudible] acceptance tests or testing big as they think like it's going to take a long time. We found that actually we've been able to maintain a happy medium of testing big but also, having those test be really, really fast. When you say you said a speed boost from using interactors with Ember, where is that speed boost actually come from? WIL: I mentioned the Ember test offers a hook into the Ember run loop and interactors aren't and the reason of this is because interactors are converging and they wait for things in the DOM to exist before interacting with them. Instead of waiting for the framework to settle, it just waits for the thing to appear and then interacts with it immediately. If you're starting something about a button toward the top of the page, you don't really care that another button at the bottom of the page has rendered yet, unless of course you have assertion about that but if they're converging, you don't need to hook into the wrong loop to wait for the entire page to load, to interact with just one piece of it. CHARLES: Right. You're just waiting and you say, "I'm expecting something to happen and the moment I detect it, no matter what else is going on, the page could be taking 30 seconds to load but if that button appears and I can interact with it, I can take my action then or I can make my assertion then." It's about kind of removing gates -- artificial gates. WIL: Yeah. Another common thing that's helped with is animations as most test that are hooked into the run loop, you kind of have to wait for some of these animations to finish before you can even interact with the element and that means if a model has a half second animation where it flies in and you have 30 tests around this modal, those tests are extremely slow now because you have to wait for that modal to come in, whereas -- CHARLES: -- Straight up flaky. WIL: Yeah, straight up flaky. Whereas in the actual DOM, that modal is inserted pretty immediately and can be interacted with pretty immediately. With interactors, they don't need to wait for the animation to finish. They can just immediately interact with that modal but of course, if you need to wait for the animation to finish, there are options for that as well. CHARLES: Yeah. If there's some fade in that needs to happen, you can kind of assert on any state and as long as it's achieved at some point, the interactor will recognize it and recognize it at the soonest possible time that it possibly could. I remember getting bitten on one project where the modal animations in particular were so brutal. Not only were they flaky, they just were slow because there was all these manual time outs. It wasn't even a paper cut. It was kind of like a knife cut, like there's someone sitting there and kind of slashing you with a pocket knife. It just was a constant source of pain in your side. WIL: Yeah and that's how you end up with things like waits and sleeps in your test suite. When you need to wait for the animation to happen or something, you just see a sleep for four seconds with a comment because we have to wait for the components to load in. That's kind of a code now. CHARLES: Yeah, that's just asking for trouble both in terms of slowness and in terms of it's going to get flaky again. That has been kind of one the most freeing things about working with interactors and working with convergent assertions on which they're based is you just don't ever have to worry about asynchrony. Really, really truly, most of the time, you're writing your tests, like it's all synchronous and that kind of makes sense because from the user's perspective, their consciousness is synchronous and they don't care about the internal run loop. It's just they were making observations in serial and at some point, they're going to observe something, so the interactor sits at that point and really observes the application the way that your user would. WIL: Yeah. We've mentioned a few times now the convergent assertions, which interactors are based on. A little caveat there if you're using interactors and you're making non-convergent assertions, they might fail or be flaky. That's because interactors wait for the thing to be there to interact with, so as soon as the buttons there, it clicks it but it doesn't wait for after that event has fired and your application has reacted to that event, that's your application is concerned. We need something there like our convergent assertions that can converge on that state and wait for that state to be true before it considers itself passing or in times out. CHARLES: Maybe we should dig a little bit into convergent assertions. I think the last time we had a public conversation on the podcast about this, this is kind of where we were, like we hadn't built the interactors, we hadn't built these other component pieces of the testing story. We were really focused on the convergent assertion. We've talked a little bit about this but I think it's worth rehashing a little bit because it's a unique way of approaching the system but it's also kind of horrifying when you see how it works under the covers. I think when we tell people about the fact that it's basically polling underneath the covers. The timeout is configurable but it's basically polling every 10 milliseconds to observe a state. I remember the first time being confronted with this idea and I was horrified and like my programmer hackles on the back of my neck, like raised up and I was like, "Wait a minute. This is going to be slow. It's going to be computationally intensive." WIL: Yeah. That was my exact thought too because this is going to be slow. If acceptance tests are slow and we're doing an acceptance test every 10 milliseconds, it's going to be really slow and that's actually not the case completely. It's actually the opposite. They're extremely fast. CHARLES: It is shockingly fast. You've got to try it to believe how fast it is, how fast you can run acceptance tests. WIL: Yeah, talking like 100 tests in just tens of seconds. CHARLES: Right. You basically gated by how fast your framework can render. Your tests are not part of the slowness. Your test -- WIL: And also, memory leaks can be costly too. We experience that recently where we had memory leaks that were slowing down our test but we fixed those up in test and put our backup. CHARLES: Yeah, because basically, running the assertion or running the convergence is very fast. It's just a very light ping. I kind of think of it is as it is light as the brush of a photon or something that was bouncing off of a surface, so that you can observe it. It's extremely light and most of the time, it's just waiting so the test and the convergence really just gets out of the way. Just because they can run a thousand times or a hundred times in a second, it's doesn't gun it up. But the thing is it means that your tests run as fast as your application will run. You get back to the point... Was it in React where the kind of the key insight is that JavaScript is not the bottleneck? Well, your tests are not the bottleneck. WIL: Yeah. CHARLES: I guess this is what it is. I don't know if there's anything else that you want to say about convergences. WIL: No. We pretty much summed it up there and that's what interactors are based on. That's how they're able to wait for things in a DOM. It basically polls the DOM until it exists and then it moves on and actually does the interaction. CHARLES: Once again, this is actually a very low level thing on which BigTest is based but this is once again, something that you can use independently. You can write your own convergent assertions. You can write your own convergences that honestly have nothing to do with testing your assertions. It's a free standing library that you can use in your test suite or elsewhere should you choose. WIL: That doesn't need to be a DOM for BigTest convergence there. I use BigTest convergence in BigTest CLI to converge on the browser being launched. Instead of waiting for the browser to report that, I can just kind of poll and see how that process is doing and the convergence waits for that process to start before moving on. CHARLES: Right. I guess the best way I've thought about it it's a way to synchronize on observations and not on callbacks. It's a synchronization mechanism and 99% of the synchronization mechanisms that we're used to, they've involved some sort of callback, a promise, an event-listener, things like that or even a generator where control is handed back explicitly to a piece of code when something happens. Whereas, this is a fundamentally different synchronization primitive, where you are writing synchronous code that's based on observations, so what I observe this, do this. When I observe this, do this. It's extremely robust. WIL: Yeah, very. CHARLES: It is a core piece. A fundamental thing that on which interactors are based on, which the CLI is based, I don't know if it's core to writing tests but -- WIL: It definitely helps. CHARLES: It doesn't helps. We couldn't have BigTest interactor without that. WIL: No, definitely not. CHARLES: Because that's what makes it fast, that's what makes it not flaky at all and having those things, I think it makes it easy to maintain because you can work at the interactor level or the level of user interaction and you don't have to worry about synchronization, so the flow of your tests are very natural. WIL: Yeah. We don't have to explicitly wait for request to be done for making an assertion about your app. That'll just come with convergences, just waiting for test date in application to true. CHARLES: Let's talk about one more piece of the testing issue because when you're testing big, when you're testing in the browser, there's always the issue of what are you going to do about your API. You got to have your API running. It's just always an issue and this is kind of interesting because this sits at the crossroads of testing big and also, getting the most utility out of your test because in an ideal world, if you're testing really big, you're going to be using a real API. You're not going to poke holes in reality. WIL: Yeah. One of the things that we avoid in BigTest is poking holes. We're not shallow mounting the components and testing the methods and the results. We're fully mounting these things and fully interacting with them through the full DOM API. CHARLES: Yeah, exactly, using real browsers. It just occurred to me the irony of us talking about reality being things that are still running inside of a computer processor. I think we've inherited this term from that talk that Justin Searls at AssertJS in 2017. It's a really, really excellent talk. I think he gave it at RubyConf. It's the 'Don't mock me.' WIL: Yeah, it's one of my favorite talks. CHARLES: Yeah, it's a great talk. In it, he talks about the value of a test is a balance of how many holes you poke in reality and sometimes, you encounter a test where all it is like holes in reality. Whether you're mocking this, you're mocking that, you're mocking the DOM, you're mocking the browser, you're mocking your network layer, you're mocking this external API and the more holes you poke, the less useful it's going to be. Network is one of those where it can be very difficult to not poke holes in that reality because it's a huge part of your application. Your frontend application is how it's going to interact with the server but at the same time, servers are gigantic pieces of software themselves, each with their own dependencies, each with their own set up and tear down -- WIL: Have their own concerns. CHARLES: Yeah, exactly. They might be in a different language. They've got runtime, things like they might need external C libraries and crazy stuff like that. They're their own beast. To get a true big end-to-end test, you going to have to stand up your server but the problem that presents is you want your tests to be also isolatable. If you're a developer, I can go to a repo, I can do an install of my dependencies and I can run the tests without having to do any external dependencies other than the repository and the language in which I'm working. This is one where we kind of have tried to walk the line of not wanting to poke holes in reality but also, have the test be containable to the actual application. In order to do that, you need something that presents a high fidelity version of the network. You can kind of try and have your cake and eat it too. You want to have something that acts like a server and really acts like a server but it's actually not a server. WIL: And still poke as few holes as possible and the application and how that's all set up, we don't want to be intercepting methods and responding with fake data. That's not a good way to mock that network. CHARLES: Right. We want to be calling actual fetches, calling actual XMLHttpRequest. Ideally, if you've got service workers, making actual service worker requests. WIL: Basically, as far as the application is concerned, it's talking to a real server on us. CHARLES: Yeah and that's kind of the litmus test for is it a hole in reality or is it just a really great illusion? WIL: Yeah and that's a good name for Mirage, right? It's a really great illusion. CHARLES: Yeah. It is a simulation of reality, so we use Mirage, which is something from the Ember testing world but something that we have extracted and made available as BigTest Mirage. WIL: Yeah. The main difference just being is that we've taken away the Ember dependencies and the run loop stuff. It's just plain JavaScript Mirage. It works exactly the same as you use it in Ember minus the auto imports and the file... Oh, man. I can't think of that word. Aside from automatically importing your files for your server config, you have to do that manually because Ember is what provides that but other than that, it's a form of Mirage. You define models and serializers and factories and all the good stuff. CHARLES: Right and then you can use those factories and you can use those models to really give a high fidelity server. If you are building something in whatever framework, you can use BigTest Mirage to simulate that network layer. Again, we've used it in a number of different scenarios but having that in place means that you're going to be able to have those high fidelity tests where your application is actually making XMLHttpRequest but it's all isolatable, so that it can be run in repo. This isn't really related to testing but it has a fantastic capability where you can prepopulate, you can use the factories to prepopulate your server with data, so that you can use the application without the actual server being implemented. WIL: Yeah. That's extremely powerful. That's we were talking about earlier and getting at is the scenarios which are setting up specific, essentially fixtures but you're generating these fixtures. Factories are essentially high level fixtures, network of fixtures. CHARLES: Yeah, higher order of fixtures. WIL: Yeah, so the scenarios are just setting up these fixtures for a scenario of your applications, like the backend is down or the list only responds with two items as opposed to 5000 items, something like that. You want to be able to, not only test these things but be able to develop against it and Mirage makes that really easy because you can just start your app with Mirage-enabled point to that scenario and you're there. You have that exhausted scenario to develop in. CHARLES: If you've never used Mirage, it is really hard to understand just how incredibly powerful it can be. We've used it now on at least four projects, where we did develop the entire first version of the product without any backend whatsoever. It's an incredible product development tool, even apart from testing, that then informs the shape of what the API was going to be. I know we've talked about this on the podcast before but it's really an incredible technology and it is available to you no matter what framework you're using. I think it's one of the best kept secrets in JavaScript development. WIL: Yeah. That's definitely great. That said, though it does have some fallacies. It's great but it can be a little slow sometimes, so we are eventually working on a BigTest network like another piece of the BigTest pie that you'll be able to sprinkle into your application but in the meantime, praise Mirage. CHARLES: Yeah. We are going to be offering an alternative or maybe collaborating for another version of Mirage but hopefully, we can make Mirage faster, we will be able to make this thing faster, so that it can use service workers and be used in a bunch of different scenarios. Just to recap, we've talked about a lot of different components but over the past year, a couple of years, these are the things that we've identified as being really key components as big part of your acceptance testing and really your testing stack. How you're going to start and launch these things? How are you going to set them up and tear them down? How are you going to interact with the application from a user, both in terms of making assertions and how are you going to take action on behalf of the user and still have it be maintainable, have it be resistant to flakiness, have it be performant? BigTest is the answer to that for those particular areas of the testing story and so, some were using we're using existing components like we use Karma, we use Mirage to date. Those, we did not develop but where we see kind of key pieces of that puzzle missing is where we started writing the BigTest solutions so things like the interactor. Eventually, we are going to make BigTest into a product that's you're going to be able to use kind of out of the box, just like you might install Cypress, where it's a very quick set up and we make all of the decisions about the components for you. But in the meantime, we're really trying to take our time, identify those pieces of the puzzle and build the software component that fits that piece of the puzzle at the absolute best so when they're polished, use them in a more comprehensive product. Things like convergence, things like interactor, things like BigTest React, BigTest Vue and very soon, BigTest Ember. These are things that you can use today, to make your tests just that much bigger and that much better, especially interactor. It's been an incredible journey this past year as we kind of develop these individual pieces and there's just going to be more goodness to come. WIL: Absolutely. Right now, I'm working on some validation type API for interactor that I'm hoping to land soon. That'll open up the possibilities of maybe hiding away those convergent assertions a bit more in your tests and just handling this automatically. It'll be pretty good. CHARLES: It's really exciting. Writing test has got more and more easy and more and more fun over the last year for us. I think we're already starting in a pretty good place. If you have any questions about BigTest, how would folks get in touch with us? WIL: We have a BigTest Gitter channel. You can find a link to that on the BigTest website: BigTestJS.io. Just ask us questions on Gitter and we'll try to answer them. CHARLES: And as always, you can ask us directly. You can send email to Contact@Frontside.io or reach out to us on Twitter at @TheFrontside or you can actually reach out to the BigTestJS Twitter account directly and just call us on Twitter at @BigTestJS. Thank you very much, Wil. WIL: Thank you, Charles.

Guest: Dillon Kearns: @dillontkearns | GitHub | Incremental Elm In this episode, Dillon Kearns joins the show to talk about techniques for experimentation with Elm, making those experiments safe, the concept of mob programming, why you would want to experiment with Elm in the first place, and how you too can begin to experiment with Elm. Resources: Grant Maki's talk on experimenting in your team "Types Without Borders" by Dillon Kearns @ Elm Conf 2018 Dillon's Elm GraphQL library How Elm Code Tends Towards Simplicity by Dillon Kearns The CSS as ByteCode Talk by Richard Feldman This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. Transcript: CHARLES: Hello, everybody and welcome to The Frontside Podcast, Episode 114. My name is Charles Lowell. I'm a developer here at the Frontside. With me today as co-host on the show is David. Hello, David. DAVID: Hey, guys. CHARLES: David is also a developer here at Frontside and we are going to be talking about something that we've been talking, I guess a lot about recently and we're talking about Elm. I think we first started talking about this several years ago and then it kind of simmer down a little bit but recently, it's been top of tongue. With us to talk about Elm today is Dillon Kearns. Welcome Dillon. DILLON: Thank you so much for having me. CHARLES: I understand that you are a full time Elm consultant. You have a background as a Lean and Agile coach but have recently transitioned to doing Elm consulting full time. Now, what exactly does that mean in 2018 to be an Elm consultant. DILLON: Actually a lot of my motivation for getting into Elm consulting in the first place is I kind of realize that Elm to me is just an extension of the things that I was passionate about with Agile and software craftsmanship. I'm trying to help teams have a better experience with their code, make it more maintainable, make it easier to change, make it easier to drive things based on customer feedback and I really believe that Elm helps people do that. I used a lot of the background and experiences that I've had with Agile and Lean coaching and a lot of those same skills, in order to help organizations adopt Elm. One thing I've seen a lot of teams struggling with is trying out a lot of different frameworks. I've encountered teams that have spent months, very painfully trying out different frontend frameworks and having trouble coming to consensus about that. One of the things that I think really helps address that is having an experimental and iterative approach, that you can really use the scientific method to focus on learning, rather than getting it right the first time. I think that there's really a need to help teams through that process of introducing a new frontend framework like Elm, so that that's why I've gone into full time Elm consulting. CHARLES: That's an interesting process. It sounds like you really need to be constantly sending out spikes, doing research on whether it's Elm or some other technology to help you kind of bridge the chasm to the next generation. How do you actually do that as an organization? My guess, this kind of a question independent of Elm but maybe we can talk about how you see that play out in the context of Elm. DILLON: Right and actually, for any listeners interested in that question, I would really highly recommend Grant Maki's ElmConf talk from this year. He spoke about exactly that topic and it was at ElmConf that it's relevant whether your team is considering Elm or looking at other frameworks. I think that the key is you need to get good at experimenting in a way that's low risk and in a way that you can be constantly learning and seeing how these different technologies fit in your codebase and fit for your team. There's a quote that I really like from Woody Zuill. Have you guys heard of mob programming before? CHARLES: I heard of mob programming from a paper by Richard Garfield a long, long time ago, almost 20... I don't know if it's the same concept. DILLON: Yes. It gained a lot of momentum these days. Mob programming is essentially pair programming but with more people involved. I've really enjoyed that process actually. I think it's actually a great way to experiment with different technologies because you get all of the minds together and it's a very good way to kind of transfer knowledge and explore things together but Woody Zuill talks about mob programming and he likes to ask the question, "Why did we begin doing mob programming for the team at Hunter Industries that originally started mob programming?" People would give answers like, "Because it cuts out code review from the process because you have lots of eyeballs on it in real time," or, "Because it reduces bugs," or, "Because it gives you better quality code. It gives all the best ideas into the product in real time," and all those things are valid points that are really good benefits of mob programming. But he says those things may be true but actually, they're not why we tried mob programming. The reason we tried mob programming was because the team wanted to try it. That's a really important point. The team needs to be experimenting with things that they're passionate about and they need to be exploring things on their own terms. But with that said, another lesson from that story of kind of his team at Hunter Industries discovering mob programming is that the team didn't discover mob programming in a vacuum. Really, the team discovered mob programming because the team became really excellent at experimenting and evaluating those experiments and then, they like to talk about this phrase that Kent Beck coined, 'turn up the good.' When something is working well, we often focus on the negative things and trying to eliminate those things but what happens if we take the things that are working well and 'turn that dial up to 11.' CHARLES: Yeah, I love that. I remember in the kind of the original layout of extreme programming, talking about how I really just wanted to turn up all the things that were working for 11 or to 11, so testing, refactoring, incremental releases and things like that. DILLON: Exactly. CHARLES: I actually had one question that's maybe a little bit of a diversion. This is actually the first time I've heard of mob programming. It's definitely not the same sort of mob programming I learned about in Richard Garfield's paper. I think it was more referring to massively distributed open source in the form which is really kind of commonplace now that happens on GitHub. I think it's maybe, an obsolete definition of mob programming but how many people would be in a mob like two, three, four, five, six, seven, 10? DILLON: That's a great question. Really, the answer is of course, it depends. That's a consultant's favorite answer but it really does. My rule of thumb is I find it usually three people is a very nice size for a mob. I find that mobs tend towards around three or four people but that being said, it's important to note that mob programming is all about this idea that what is the true cost of programming. I think that often we look at programming as the act of writing code, initially and that's a very limited way of looking at coding. Because of course, 90% of our effort is spent maintaining code, making decisions around code, reproducing bugs, fixing bugs, communicating with customers about bugs -- bugs are extremely expensive -- the farther out they get, until eventually they get to the point of a customer discovering them, bugs are in extremely expensive part of software. If we can minimize bugs, that's very valuable. When you look at programming on this bigger scale and look at the bigger picture of programming, then you realize that you may be able to get one person to write the code faster but then, that person needs to code review it. That person needs to go and ask somebody question down the line when they don't have context because they weren't involved in the decision making. For example, maybe there's a UX person who doesn't have context on certain choices that were made, so there's a lot of churn, so you can kind of eliminate that churn by getting all the relevant people involved right away and that's the idea. In my experience with mob programming, it works really well to keep kind of a core of around three people. Sometimes, somebody goes up to have a conversation with somebody, take a break or answer somebody's question, maybe somebody from another team has a question that type of thing and so, the team can keep coding as a pair or whatever. But ultimately, the idea is that you get faster because you're building up this shared context and you're not spending as much time down the line answering questions, doing code reviews and things like that. CHARLES: Right. I see. DAVID: That kind of matches with my experience. Mob programming on previous teams, the way we had it set up is there was a regular mob programming chat session that the whole team was invited to but it was optional. You can just show up if you wanted to and really, that sort of made it so that there was a set of people who regularly attend -- three to five people in a session -- and they were the core group, essentially. DILLON: Right. That's another great point. Invitation is a powerful technique. If you're kind of mandating the people try an experiment or work in a certain way, ultimately it's much more powerful to let the team experiment on their own and follow their passion and they'll discover great things. It's about experimenting, rather than choosing specific experiments. While we're on this topic of kind of the real cost of coding, I think this is a good point to talk about this quality in Elm because, I think that this is one of the things that really motivates me to use Elm myself and introduce it to others is that, I think that Elm really get something about programming where there's a sort of superficial ease of certain techniques that Elm kind of goes beyond and says, "Actually, let's optimize for a different set of things that we think make code more maintainable and more delightful to work with in the long run." CHARLES: I wanted to also transition between, we were on a little diversions on mob programming but do you use mob programming as explicit technique for introducing Elm when a team is considering adopting it? DILLON: That's a fantastic question. I absolutely do. Of course, I honor the ways of working in a particular organization or team. I think that's important to do but I do strongly encourage using mobbing as a technique for knowledge sharing and when I'm on-site with a client, I find it extremely powerful as a technique for knowledge sharing and also, let's say you do an experiment, somebody is off in a corner and they're trying out Vue.js or they're trying out Elm or they're trying a particular coding technique. Then they come back to their team and they say, "Hey everybody, I tried this great thing," and now they have to spend this time convincing everybody and saying like, "Wait a minute. You didn't try this, you didn't try it that way. It wouldn't actually work in our context because of this." I think that it's very powerful to have everybody kind of involved in that process so that you can evaluate it together as a team. CHARLES: Because the thing is like, when you experience win or you experience fail, it's a very visceral feeling and that's the thing that sells you or turns you away. You can argue until you're blue in the face but words have a very limited capacity to convince, especially when compared with like physical and emotional feeling. It sounds like you can get everybody to have that shared experience, whether for the good or for the bad, you're going to arrive at a decision, orders of magnitude more quickly. They have to rely in conviction of that decision spread around the team. DILLON: Exactly. I think that hits the nail on the head and you say that we have this sort of skepticism of arguments from theoretical conversations, rather than 'show me the money,' but it's actually, try solving a real problem in this and that's exactly as it should be. I think that's one of the big antidote from this problem that I've seen in a lot of environments, where there's this analysis paralysis, especially with the state of the JavaScript ecosystem these days. I think that one of the keys to improving that situation is to get good at trying things, rather than theorizing about things. We have a tendency to want to theorize and when we do that, then we say, "Can it solve this problem? Can it solve this problem? Can it solve that problem?" You can talk about that until the cows come home but it doesn't get you anywhere and it doesn't really convince anybody of anything. The key is to find very small experiments and what I really recommend and what I'm dead focused on when I'm initially working with a client is getting something into production. Now, that doesn't mean that you need to have a road map for turning your entire application into Elm. In fact that's the whole point, is that you're not trying to do that. The point is you're trying to get as realistic of an experience as possible for what problems might occur if we do this? Will the team enjoy working with this language? Will it work well with our built pipeline? Will there be any unforeseen issues? You don't know until you actually try it, so you've got to try it and you've got to try it in tiny, tiny steps and low risk experiments. CHARLES: Right but you've got to try it for real. You don't want to try it with a TodoMVC. DILLON: Exactly. It needs to be meaningful, to really have a good understanding of what it's going to be like. CHARLES: I would say that I tend to agree but I've definitely encountered the counterargument and I also think this counterargument makes sense or perhaps where the pushback lies is if I'm constantly experimenting, then what I'm doing is I'm internally fragmenting my ecosystem and there is power in similarity. Any time you introduce something different, any time you introduce one fragment, you're introducing complexity -- a mental complexity -- like maybe I have to maintain my Elm app and I also have to have my Legacy... Or not Legacy, I've got my other JavaScript tool kit that does it in one way. Maybe I've got a couple of more because I've run these other experiments. I'm not saying that there is one way but there is power in uniformity. There is power in diversity. Where do you find the balance? DILLON: Those are all excellent points. To me, I think really the key is it's about the scientific method, you could say. The thing with the scientific method is that we often forget the last part. We get really good at hypothesizing about things. Sometimes people leave it at that, which we kind of just discussed. Sometimes, people go past the hypothesizing stage and they actually run the experiment and that's great. But then, the majority of people, if they get to that point, will forget to do the last step which is to evaluate the results. I think the key here is you need to be experimenting and this is what it means for it to be a low risk experiment. It means that you're not setting yourself off in a direction where you can't turn back. You want to set it up in such a way that you can turn away from it with minimal cost. One of the things that is really helpful for that is if you build a tiny, independent, little widget in your application, try building that in Elm. Some people will do that with a little sort of login badge in the corner of their application. One of the teams where I've introduced Elm at a Fortune 10 company, actually where we introduced Elm, we started out with just a tiny little table in one page and if we wanted to back that out, it would have been trivially easy but we decided that we wanted to go in further and invest more. CHARLES: That makes a lot of sense. Effectively, you need to have a Plan B. Don't sync all of the available time that you have to invest in an experiment. Make sure that you have a Plan B and if you need to do this widget or this table in Angular or React or Ember or whatever, you are thinking about that -- how would that work. DILLON: Exactly and the thing with experiments is the purpose of an experiment is not to build something. It's to learn. I really like this kind of ethos of lean startup, which I think is really getting much more into the mainstream in the software industry, which is a wonderful thing. The idea of lean startup, the kind of core concept is this idea of validated learning. Basically, in an environment where there's uncertainty, which is pretty much most of the things you're doing in software, the main goal is you're not shipping a product like you would be if you're trying to manufacture cars as quickly as possible. The main thing that you're producing is what they call 'validated learning' and so, you want to minimize the amount of time it takes to validate or invalidate your assumptions about something and then, you want to make it as cheap as possible to move on from that. CHARLES: I like that. So if you're going to organize your development process around this principle or maybe not organize it but integrate it into development process, how do you know that you're conducting a healthy number of experiments, versus I may be conducting too many experiments? Is there a metric that you can look at? We need to have this many experiments running at all times or this is just too many or something else. DILLON: That's a really interesting question. I think I would tend to think about that more again, as looking at the way the experiments are run, rather than 'are there too many experiments?' That's just not a problem that I've seen there being too many experiments. The pain that we tend to really see in environments where experiments are hurting teams is the way the experiments are being done. It's hard to backtrack from those experiments and as you were saying before, you kind of put yourself down this path where you can't walk it back and you create this sort of rift in the way the code is being written, which makes it more difficult to work in that codebase. The thing with experiments is they can have really big payoff. Now, you want to make sure that you're not just going in and picking up every shiny object that you see. One thing that can keep you honest with that is if you're kind of coming up with a hypothesis before you start. If you're saying, "This is the value to our business and to our team if we attempt this thing and this is what will prove that it seems to have that value and this is what will tell us, 'Actually, it doesn't have that value and we should drop it and cut our losses.'" CHARLES: That's a great heuristic. As you're saying and imagining how that might have saved my bacon in the past because I've definitely made the mistake of playing with too many shiny objects and picking things because I didn't fully evaluate what I thought the value. I was explicit with myself about what is the value that is going to bring to this project or this business. I have a theory about it but I am not thinking what is my hypothesis and how am I going to validate or invalidate? I'm thinking, I've got a short term pain that I'm experiencing and I'm grasping for this thing, which I think will solve it and I'm not properly evaluating how it's going to affect me long term. DILLON: Right and that could be a great team practice to play around with is often, teams will kind of come up with action items out of retrospectives. One thing that I think can be really beneficial for teams is to kind of flip that notion of doing action items which again, it's really just doing the middle part of the experiment where you're conducting the test but you cut out the hypothesis part and the evaluation part. Try to bring that into your team's retrospective and try to have explicit hypotheses in the retrospectives and then, in the next retrospective, evaluate the results. CHARLES: All right. I will definitely keep that in mind but this feels like a fresh take on kind of how you manage software development that I haven't encountered too much, being more scientific about it. It sounds like science-oriented development. DILLON: Right. DAVID: I like that. DILLON: There are a lot of buzzwords these days in software development, in general and it's really becoming a problem, I think in the Agile community but really, what it boils down to is these basic elements and basing decisions on feedback is one of those fundamental unit. You can call the scientific method, you can call it lean startup and validated learning, you can call it agile, you can call it whatever you want but ultimately, you need to be basing things on feedback. I think of it almost like our nerves. There's actually a disorder that some people have, which can be fatal, which is that their nerves don't tell them when they're feeling pain. I think this is a great analogy for software because that can happen to companies too. They don't feel the pain of certain decisions not landing well. Because they're not getting feedback from users, they're not getting feedback from metrics and recording, they're not getting feedback from doing that final evaluation step of their experiments, so when you fall on the ground, a small cut could be extremely harmful because you don't know the damage it's doing to you. CHARLES: I think that is a good analogy. One of the things that I'm curious about is we've been discussing a lot of techniques for experimentation and how you can integrate that into your process and how you can make your experiments safe, so let's talk a little bit about -- first of all, two things -- why would I want to experiment with Elm in the first place? Because ultimately, that's why we're here and why we're having this conversation. What's compelling about it that would make me want to experiment? And then how can I begin to experiment with Elm? DILLON: I actually just published a blog post yesterday. It's called 'How Elm Code Tends Towards Simplicity.' To your question of why would a team consider Elm, I kind of talk a little bit in this blog post about a case study at a Fortune 10 company where I introduced Elm to a few of the teams there. One of the teams there, we had actually seen an Angular project that they had worked on and often, in an enterprise environment, you have projects moving from one team to another. I actually had my hands on this Angular project. It kind of moved over to another team and we were experiencing some major pain trying to make changes in this codebase. Even making the simplest change, we were finding that there were a lot of bugs that would be introduced because there's some global variable. There's some implicit state. Sometimes, it was even reaching in and tweaking the DOM and really, the topic of conversation at our team lunches was how afraid we were to touch this codebase. Fast forward a few months and this team was asking my advice on picking a new frontend framework and I introduced them to Elm. They took a run with it and it was pretty remarkable to see this same team that had really struggled with AngularJS and they didn't really have a strong sense of what were the best practices. They weren't getting any guidance from the framework itself and the tooling around it and they actually loved the experience of working with Elm because they were saying, "This is amazing. Maybe it takes a little time to figure out how to solve a particular problem on Elm but once we do, we know that we've done it in a solid way." One of the things that I think is most powerful about Elm is that it keeps you from shooting yourself in the foot. I think that's a really good headline kind of summary of what I love about Elm. For example, tweaking the DOM. Now, it might seem like a pretty obvious thing that we just won't tweak the DOM and that's fair enough. That might not be a problem for a lot of teams. People wouldn't even reach for that technique because they're disciplined about it. But at a certain point, you start taking on enough things and then go from kind of those basic things that are going to make your code more unreliable and unsafe like tweaking the DOM and you start getting into the realm of best practices. There's so much discussion these days in the JavaScript community about best practices, which is great. It's great to discuss that but my concern is that there's a new best practice each week and the team has to agree on it, you have to find techniques for enforcing it, people have to make sure that these best practices are being followed in code reviews. Then when you look at a given piece of code, you have to trust that those best practices are being followed, so it requires a lot of work to make sure, in your reducers, in redux that you are not mutating anything. With Elm, data is just immutable. That's just how it is. There are a lot of these kind of things that are baked into the language and the expressivity of the type system allows you to bake in your own constraints. One of the things that I find really compelling about Elm is its design really prevents you from shooting yourself in the foot and it gives you tools for making sure that you take it even a step further and it helps you enforce these best practices at a compiler level. CHARLES: Now what's interesting here is it's almost like the opposite tension of experimentation is a work, right? like here, we have an example of uniformity being the more powerful track but then inside the actual macroscopic process, you want a lot of experimentation and diversity. But at the microscopic level, inside your application, it sounds like you want less experimentation and you derive a lot of strength from that but -- DILLON: That's a great point. CHARLES: -- Experiments that are possible, yeah. DILLON: I think that there is a lot of pain these days in the JavaScript community. We hear people talking often about JavaScript fatigue and it's a real thing. It takes a lot of work to stay on top of the latest best practices and frameworks and that can be a lot of fun. I love learning about the latest new frameworks and tooling but ultimately as you're saying, we don't want that experimentation so much about the fundamentals. We want some dependable, solid fundamentals and then we want the experimentation to happen within there. I think that's exactly what we see in the Elm ecosystem. We have a single kind of data store or way of managing state in Elm. It's called the Elm Architecture. In fact, it's what Redux is based on and it worked extremely well and you don't have to experiment with different data stores in Elm because that's just what Elm code looks like. Now, if you want to experiment in Elm, then there is a lot of innovation happening. One of my favorite things about Elm is that the compiler and its expressiveness has sparked a lot of creativity. One of my favorite things about Elm is the library called Elm UI. Actually, a client that I'm working with right now, it's a really interesting case study. They are kind of a very small startup. They just kind of branched off of a larger startup. They're building some tooling for this ecosystem. They were engineers at a company called Procore that does cloud document management for construction companies. They wanted to get a product-ready for a big conference for their potential clients. The reason they brought me in to help them was because they wanted to reach this ambitious target of being able to do a demo of this brand new product at this conference and they wanted to iterate very quickly. One of the things that really drew them into Elm in the first place is this library Elm UI. Elm UI essentially, Richard Feldman gives a talk on it, where he uses the analogy of it being treating CSS and HTML as bytecode for your views. I think that's a really apt way to put it. If you break down this idea of CSS -- Cascading Style Sheets, it removes the cascading part of CSS and it removes the sheets part of CSS. What you're left with is a way of expressing style and it's a way of expressing style that is able to part ways with all of the baggage of the entire history of backwards compatible decisions that CSS has ever made. If you want to vertically aligned something, then you just say, "Align vertically," you know, center vertically. If you want to center something horizontally, you say center X. It creates a high level language for expressing views. My experience with Elm UI, this may not be the right choice for every team but I love it. I use it on all of the projects that I maintain personally. I love using it because it gives you that same sense of invincibility refactoring that you get with Elm, which is remarkable that you could have that feeling with managing views. CHARLES: It's definitely something that feels like a dark art and it can't be called science. It's an art. It's a science for some people but it's historically been a dark art and something fiddly to work with. In terms of being able to make the experiment with Elm, when we talked a little bit about why you might want to experiment with it in the first place, what the business case is, I guess my next question is or a question that immediately comes to mind is supposing that we have decided to experiment with this, how do you mitigate that experiment? We talked about lowering the cost, having a way to turn away from it, having a way to make it inexpensive. For example, one of the things that I think of when evaluating a new technology is how well can I use it with old technologies. I have a lot about best practices in my tool bag. We all do. We got our all favorite libraries and pathways that are just familiar to us. One of the things that I've noticed is when adopting a new technology, one of the things that makes it easy to experiment with is how well it works with the existing technologies. I know that, we talked about Elm UI, kind of rethinking style in CSS and your views and Elm itself as a completely different language within JavaScript, that can be both liberating but it can also be limiting in the sense that I can't reach back for my existing tool if no tool exists in this new space. The kind of experience that I've had where this is really worked is systems like JRuby or Clojure, where there's a very clear pathway to be able to use Java libraries from those environments, so you always have kind of an escape hatch. What's that like in Elm? DILLON: This is a really interesting conversation because it highlights, in some ways some of the most defining features of Elm. In terms of how do you kind of pull Elm into an existing application, there are a lot of different techniques for that. It's pretty straightforward to create a little Elm app. We usually don't call them components for reasons that we can get into if we want to but that's a whole can of worms. But if you've got a little Elm application that you want to use to render a widget on your page, then it's as simple as just calling Elm.yourmodule.init and rendering it onto the page there. That's quite straightforward and if you want to interface with your existing code there are several ways to do that. There's something called port in Elm, which is how you kind of communicate by sending these messages and data back and forth between your Elm app and JavaScript. Now, this is one of the decisions, I think that defines Elm as the language and the reason this is important is because Elm decided not to make the choice that a lot of other compile to JS languages do. For example, if you look at ReasonML or PureScript or a more extreme example, TypeScript. TypeScript is a superset of JavaScript, so it's trying to allow you to gradually introduce this to get some incremental improvements for your JavaScript code, so it's extremely easy to experiment with it, which we've talked about the importance of experimentation. Now, the challenge with this technique, the tradeoff here is it's great, that it then becomes very easy to transition into it and that's an excellent strategy for the goals of TypeScript. Elm has a different set of goals, so the things that elm is focused on giving you is a truly type-safe experience. When you're working with Elm, if your Elm code says that this data is a float, then it is a float. Either, it is a float or that code is not being run and so, that's very different than the experience in TypeScript where you have these escape hatches. This is an inevitable choice for any compiled to JS language. Are you going to have escape hatches or not? Elm is really the only language out there, I think that chooses not to have escape hatches and that is actually the thing that that I love about the language because that's the only way you can truly have guarantees, rather than, "Yeah, I'm pretty sure that these type guarantees hold." DAVID: Yeah, wishes and dreams. DILLON: Yeah. CHARLES: What does it mean to have no escape hatches? because you talked about ports. Does it mean like it's impossible to use an external JavaScript library? DILLON: That's an excellent question. You absolutely can use JavaScript libraries. It means that it's being explicit and upfront about the fact that there's uncertainty in these areas. That's what it comes down to. Take for example dealing with JSON. In a JavaScript application, what we get when we're dealing with JSON is you make a request up to the server, you have some callback that passes in the data you get back and then you start pulling bits and pieces off of it and you say 'response.users subzero.firstname' and you hope that none of those things are null, none of those types are different than what you expected. In a way, it's kind of letting you pretend that you have certainty there when in fact, you don't and with Elm, the approach is quite different. You have to explicitly say, "I expect my response to have this shape. I expect it to be a list of things, which have a first name and last name which are strings," and then Elm says, "Okay, great. I'm going to check your assumptions," and if you're right, then here you go and you're in a well typed-space where you know exactly what the types are and if you're wrong, then that's just another type of data, so it's just a case statement where you say, "If my assumptions were correct, then do something and if my assumptions were incorrect, then you decide what to do from there." CHARLES: Right. For me, it sounds like there is some way because ultimately, I'm going to be getting unstructured but I'm going to be getting JSON back from the server and maybe, I have some library that's going to be doing that for me and enhancing it and adding value to that JSON in some way. But then at some point, I can present it to Elm but what you just saying is I need to be complete in making sure that I handle each case. I need to do or handle the case. Explicit about saying if the assumptions that Elm wants to make, turn out to not be true, Elm is going to make me handle the case where those assumptions were not true. DILLON: Exactly. I think that TypeScript of any type is the perfect illustration of the difference. TypeScript of any type is sort of allowing you to say, "Don't type check this. Trust me here," and Elm's approach is more kind of just be explicit about what you want me to do if your assumptions are incorrect. It doesn't let you kind of come in and say, "No, I know I'm going to be right here." CHARLES: Right but there is a way to pass data structures back and forth. DILLON: There absolutely is and actually, there's a technique that's starting to gain some traction now, which I'm really excited about, which is rather than using this sort of JavaScript interrupt technique we talked about, which is again, it's very much like communicating with a server where you're kind of sending messages and getting data back -- getting these messages with data back. But there's an alternative to that which is using web components. Actually, there's quite good support for assuming that you don't need to be compatible with Internet Explorer. Basically in a nutshell, if you can wrap a sort of declarative web component around anything, it could be a Google Maps API, it could be a syntax highlighting JavaScript library, something that you don't have an Elm library for but you want to use this JavaScript library, it's actually quite a nice experience. You just render that custom element using your Elm code just as you would any other HTML in Elm. CHARLES: Yeah I like that, so the HTML becomes the canvas or composition with other JavaScript and the semantics are very well-defined and that interface is actually pretty thin. DILLON: Exactly and the key again is that you wanted to find a declarative interface, rather than an imperative one where you're kind of just doing a series of statements where you say, "Do this and then set this value and then call this and then set this call back." Instead, you're saying, "Render this Google Maps custom element," which is centered around these coordinates and has this zoom level on. You declaratively give it the bit of information that it needs to render a particular view. CHARLES: Okay. Then I guess the final question that I have around this area is about being able to integrate existing tools and functions inside of an Elm application. Because it sounds like you could theoretically develop large parts of your application, is there a way that you can actually have other areas of your application that are not currently invested in Elm still benefit from it, in the sense for kind of need of JavaScript APIs that Elm can make available. DILLON: Right, so you're kind of talking about the reverse of that Elm reaching out to JavaScript. You're asking about, can JavaScript reach out to Elm and benefit from some of its ecosystem? CHARLES: Exactly. I say that is that another potential vector for experimentation. DILLON: It's a really interesting thought. I haven't given it too much thought, to be honest but I actually have heard it come up before and my gut feeling is that it's probably more fruitful to explore the inverse, reaching out to JavaScript from Elm and the reason is kind of the main appeal of Elm is that when you're operating within Elm, you have this sense that if it compiles, it works. Because again, this central decision to not allow escape hatches is what allows you to have that sort of robustness, so you have this feeling of bullet proof refactorings and adding new features seamlessly where you change your data modeling to say, "Here's this other case that can be represented," and then suddenly, the Elm compiler says, "Tell me what to do here, tell me what to do here and tell me what to do there," and you do it and your app is working. That's the real appeal of Elm, I think and you don't really get much of that by just calling out to an Elm library from within JavaScript. That's my gut feeling on it. CHARLES: Okay, that's fair enough. On the subject of interrupt and using tools like JSON, you actually maintain a GraphQL library for Elm. You probably have a lot of experience on this. Maybe we can talk about that as a concrete case that highlights the examples. DILLON: Yeah. I think to me this is one of the things that really highlights the power of Elm, to give you a really amazing refactoring and kind of feature creating experience. A lot of Elm libraries are prefaced by the author name, so it's still DillonKearns/ElmGraphQL. I spoke about it this year at ElmConf. In a nutshell, what it does is it actually generates code based on your GraphQL schema. For anyone who doesn't know, GraphQL is just kind of a language for expressing the shape of your API and what types of data can return. What DillonKearns on GraphQL does is it looks at your GraphQL schema and it generates an API that allows you to query that API. using this library, you can actually guarantee that you're making a valid query to your server. Again, you get this bulletproof experience of refactoring in Elm where you can do something like make a change in your API and recompile your Elm code and see whether you've made a backwards incompatible change. All of this effort of doing sort of this JSON decoding I was talking about earlier where you kind of have to explicitly say, "These are my assumptions about the shape of the JSON that I'm getting back." When you're using this library, you no longer need to make any assumptions because you're able to rely on this sort of schema of your API and so you know when you're requesting this data, you don't have to run it, see if it works and then tweak it and run it again -- this sort of cycle of checking your assumptions at runtime. It moves those assumptions that you're making from runtime to compile time and it can tell you when you compile your application, it can say, "Actually, this data you're requesting, it doesn't exist," or, "It's actually called this," or, "This is actually the type of the data." CHARLES: Right. I love that. How do you do that? Because it seems like you've got a little bit of a chicken and the egg problem because the schema is defined outside of Elm, so you have to be able to parse and understand the schema in order to generate the Elm types to be able to compile Elm code against them. Maybe I'm not -- DILLON: That's exactly right. That's exactly what it does. Now, the nice thing is that GraphQL is really designed for these types of use cases. It supports them in a first class way. If you have a GraphQL API, that means you have built into it whether you know about it or not, a way to introspect the schema. All of the queries for kind of interrogating that GraphQL server and asking what types of data does this return, what are all your queries that I can run, it's built into it by the framework, so that comes for free. Getting up and running with this package I built is as simple as running a little npm CLI, pointing it to either your URL for your server or the JSON form of your schema, if you prefer and then, it generates the code for you. CHARLES: Wow, that sounds fantastic. This is the exact kind of thing that feels like it would be cool if I could just start using this library to manage the GraphQL of my application but I'm consuming that GraphQL from other JavaScript but it's the Elm code that's managing it. Do you see what I mean? DILLON: I hadn't considered that. I guess you could. You're right. Maybe I'm so smitten with Elm that it's hard to see an in-between but I guess, you could get some benefits from that approach. CHARLES: Right and as an experiment of course. DILLON: There you go. There you go. CHARLES: All right. With that, I think we'll wrap it up. Thank you so much, Dillon for coming on and talking with us on the podcast. DILLON: My pleasure. I really enjoyed the conversation. CHARLES: I actually got so many great tidbits from so many different areas of software development in Elm but also, just in kind of other things that I'm interested in trying. It was a really great conversation. DILLON: I had a lot of fun and I love discussing these things. For any listeners who are interested in this stuff, feel free to reach out to me on the Elm Slack or on Twitter. I'm at @DillonTKearns. I'm also offering a free intro Elm talk for any companies that are kind of entertaining the idea of doing an experiment with Elm. If you go to IncrementalElm.com/Intro, you can find out about some of the talks that I'm offering. CHARLES: All right. Well, thank you very much and we, as always are the Frontside. We build software that you can stake your future on and you can get in touch with us at @TheFrontside on Twitter or Contact@Frontside.io on email. Please send us any questions you might have, any topics that you'd like to hear about and we look forward to hearing from you and we will see you next week.

Panel: Aimee Knight Charles Max Wood Joe Eames AJ O’Neil Chris Ferdinandi  Special Guests: Charles Lowell (New Mexico) & Taras Mankovski (Toronto) In this episode, the panel talks with two special guests Charles and Taras. Charles Lowell is a principle engineer at Frontside, and he loves to code. Taras works with Charles and joined Frontside, because of Charles’ love for coding. There are great personalities at Frontside, which are quite diverse. Check out this episode to hear about microstates, microstates with react, Redux, and much more! Show Topics: 1:20 – Chuck: Let’s talk about microstates – what is that? 1:32 – Guest: My mind is focused on the how and not the what. I will zoom my mind out and let’s talk about the purposes of microstates. It means a few things. 1.) It’s going to work no matter what framework you are using. 2.) You shouldn’t have to be constantly reinventing the wheel. React Roundup – I talked about it there at this conference.  Finally, it really needs to feel JavaScript. We didn’t want you to feel like you weren’t using JavaScript. It uses computer properties off of those models. It doesn’t feel like there is anything special that you are doing. There are just a few simple rules. You can’t mutate the state in place. If you work with JavaScript you can use it very easily. Is that a high-level view? 7:13 – Panel: There are a lot of pieces. If I spoke on a few specific things I would say that it enables programming with state machines. 7:42 – Panel: We wanted it to fell like JavaScript – that’s what I heard. 7:49 – Aimee: I heard that, too. 7:59 – Guest. 8:15 – Aimee: Redux feels like JavaScript to me. 8:25 – Guest: It’s actually – a tool – that it feels natural so it’s not contrived. It’s all JavaScript. 8:49 – Panel. 9:28 – Guest: Idiomatic Ember for example. Idiomatic in the sense that it gives you object for you to work with, which are simple objects. 10:12 – Guest: You have your reducers and your...we could do those things but ultimately it’s powerful – and not action names – we use method names; the name of the method. 11:20 – Panel: I was digging through docs, and it feels like NORMAL JavaScript. It doesn’t seem like it’s tied to a certain framework or library platform? 11:45 – Guest: Yes, we felt a lot of time designing the interfaces the API and the implementation. We wanted it to feel natural but a tool that people reach for. (Guest continues to talk about WHY they created microstates.) Guest: We wanted to scale very well what you need when your needs to change. 13:39 – Chuck: I have a lot of friends who get into React and then they put in Redux then they realize they have to do a lot of work – and that makes sense to do less is more. 14:17 – Guest: To define these microstates and build them up incrementally...building smaller microstates out of larger ones. Guest continued: Will we be able to people can distribute React components a sweet array of components ready for me to use – would I be able to do the same for a small piece of state? We call them state machines, but ultimately we have some state that is driving it. Would we be able to distribute and share? 16:15 – Panel: I understand that this is tiny – but why wouldn’t I just use the native features in specific the immutability component to it? 16:42 – Guest: I’m glad you asked that question. We wanted to answer the question... Guest: With microstates you can have strict control and it gives you the benefit of doing sophisticated things very easily. 18:33 – Guest: You mentioned immutability that’s good that you did. It’s important to capture – and capturing the naturalness of JavaScript. It’s easy to build complex structures – and there is an appeal to that. We are building these graphs and these building up these trees. You brought up immutability – why through it away b/c it’s the essence of being a developer. If you have 3-4-5 levels of nesting you have to de-structure – get to the piece of data – change it – and in your state transition 80% of your code is navigating to the change and only 20% to actually make the change. You don’t have to make that tradeoff. 21:25 – Aimee: The one thing I like about the immutability b/c of the way you test it. 21:45 – Guest: There a few things you can test.  23:01 – Aimee: You did a good job of explaining it. 23:15 – Guest: It makes the things usually hard  easy! With immutability you can loose control, and if that happens you can get so confused. You don’t have a way to have a way to navigate to clarity. That’s what this does is make it less confusing. It gives you order and structure. It gives you a very clear path to do things you need to do. If there is a property on your object, and if there is a way to change it... 25:29 – Guest: The only constant is change no matter what framework you are working on. 24:46 – Chuck: We are talking about the benefits and philosophy. What if I have an app – and I realize I need state management – how do I put microstates into my app? It’s using Angular or React – how do I get my data into microstates? 26:35 – Guest: I can tell you what the integration looks like for any framework. You take a type and you passed that type and some value to the create function so what you get is a microstate. (The Guest continues diving into his answer.) 28:18 – Guest: That story is very similar to Redux, basically an event emitter. The state changes on the store. Maybe this is a good time to talk about the stability benefits and the lazy benefits because microstates is both of those things. Stability – if I invoke a transition and the result is unchanged – same microstate – it doesn’t emit an event. It recognizes it internally. It will recognize that it’s the same item. Using that in Ember or Redux you’d have to be doing thousands of actions and doing all that computation, but stability at that level. Also, stability in the sense of a tree. If I change one object then that changes it won’t change an element that it doesn’t need to change. 31:33 – Advertisement: Sentry.io 32:29 – Guest: I want to go back to your question, Chuck. Did we answer it? 32:40 – Chuck: Kind of. 32:50 – Guest. 32:59 – Guest: In Angular for example you can essentially turn a microstate... 33:51 – Guest: You could implement a connect, too. Because the primitive is small – there is no limit. 34:18 – Chuck summarizes their answers into his own words. 34:42 – Guest: If you were using a vanilla React component – this dot – I will bind this. You bind all of these features and then you pass them into your template. You can take it as a property...those are those handlers. They will perform the transition, update and what needs to be updated will happen. 35:55 – Chuck: Data and transitions are 2 separate things but you melded them together to feel like 1 thing. This way it keeps clean and fast. 36:16 – Guest: Every framework helps you in each way. Microstates let’s you do a few things: the quality of your data all in one place and you can share. 38:12 – Guest: He made and integrated Microstates with Redux tools. 38:28 – Guest talks about paths, microstates to trees. 39:22 – Chuck. 39:25 – Panel: When I think about state machines I have been half listening / half going through the docs. When I think of state machines I think about discreet operations like a literal machine. Like a robot of many steps it can step through. We have been talking about frontend frameworks like React - is this applicable to the more traditional systems like mechanical control or is it geared towards Vue layered applications? 40:23 – Guest: Absolutely. We have BIG TEST and it has a Vue component. 41:15 – Guest: when you create a microstate from a type you are creating an object that you can work with. 42:11 – Guest: Joe, I know you have experience with Angular I would love to get your insight. 42:33 – Joe: I feel like I have less experience with RX.js. A lot of what we are talking about and I am a traditionalist, and I would like you to introduce you guys to this topic. From my perspective, where would someone start if they haven’t been doing Flux pattern and I hear this podcast. I think this is a great solution – where do I get started? The official documents? Or is it the right solution to that person? 43:50 – Guest: Draw out the state machine that you want to represent in your Vue. These are the states that this can be in and this is the data that is required to get from one thing to the other. It’s a rope process. The arrow corresponds to the method, and... 44:49 – Panel: It reminds me back in the day of rational rows. 44:56 – Guest: My first job we were using rational rows. 45:22 – Panelist: Think through the state transitions – interesting that you are saying that. What about that I am in the middle – do you stop and think through it or no? 46:06 – Guest: I think it’s a Trojan horse in some ways. I think what’s interesting you start to realize how you implement your state transitions. 48:00 – (Guest continues.) 48:45 – Panel: That’s interesting. Do you have that in the docs to that process of stopping and thinking through your state transitions and putting into the microstate? 49:05 – Guest: I talked about this back in 2016. I outlined that process. When this project was in the Ember community. 49:16 – Guest: The next step for us is to make this information accessible. We’ve been shedding a few topics and saying this is how to use microstates in your project. We need to write up those guides to help them benefit in their applications. 50:00 – Chuck: What’s the future look like? 50:03 – Guest: We are working on performance profiling. Essentially you can hook up microstates to a fire hose. The next thing is settling on a pattern for modeling side effects inside microstates. Microstates are STATE and it’s immutable. 52:12 – Guest: Getting documentation. We have good README but we need traditional docs, too. 52:20 – Chuck: Anything else? 52:28 – Guest: If you need help email us and gives us a shot-out. 53:03 – Chuck: Let’s do some picks! 53:05 – Advertisement for Charles Max Wood’s course! Links: Kendo UI Frontside Redux Microstates Microstates with React Taras Mankovski’s Twitter Taras Mankovski’s GitHub Taras Mankovski’s LinkedIn Taras Mankovski’s Frontside Bio Charles Lowell’s Twitter Charles Lowell’s GitHub Charles Lowell’s Frontside Bio Schedule Once Ruby on Rails Angular Get A Coder Job YouTube Talks Email: cowboyd@frontside.io Working with State Machines Twitch TV BigTest Close Brace REEF The Developer Experience YouTube Video Sponsors: Kendo UI Sentry.io – 2 months free – DEVCHAT/code Get A Coder Job Picks: Aimee ShopTalk Episode 327 Professional JavaScript for Web Developers Technical Debt Stripe Taras Twitch Channel Big Test Frontside Charles Lowell Chalkboards Sargent Art Chalk Chris Close Brace LaCroix Water Chris’s Git Hub Joe The Developer Experience Bait and Switch Good Bye Redux Recording Dungeon and Dragons AJ UtahJS Conf Start with Why The Rust Book VanillaJS w/ Chris Zero to One Charles Podwrench.com -  beta getacoderjob.com

Panel: Aimee Knight Charles Max Wood Joe Eames AJ O’Neil Chris Ferdinandi  Special Guests: Charles Lowell (New Mexico) & Taras Mankovski (Toronto) In this episode, the panel talks with two special guests Charles and Taras. Charles Lowell is a principle engineer at Frontside, and he loves to code. Taras works with Charles and joined Frontside, because of Charles’ love for coding. There are great personalities at Frontside, which are quite diverse. Check out this episode to hear about microstates, microstates with react, Redux, and much more! Show Topics: 1:20 – Chuck: Let’s talk about microstates – what is that? 1:32 – Guest: My mind is focused on the how and not the what. I will zoom my mind out and let’s talk about the purposes of microstates. It means a few things. 1.) It’s going to work no matter what framework you are using. 2.) You shouldn’t have to be constantly reinventing the wheel. React Roundup – I talked about it there at this conference.  Finally, it really needs to feel JavaScript. We didn’t want you to feel like you weren’t using JavaScript. It uses computer properties off of those models. It doesn’t feel like there is anything special that you are doing. There are just a few simple rules. You can’t mutate the state in place. If you work with JavaScript you can use it very easily. Is that a high-level view? 7:13 – Panel: There are a lot of pieces. If I spoke on a few specific things I would say that it enables programming with state machines. 7:42 – Panel: We wanted it to fell like JavaScript – that’s what I heard. 7:49 – Aimee: I heard that, too. 7:59 – Guest. 8:15 – Aimee: Redux feels like JavaScript to me. 8:25 – Guest: It’s actually – a tool – that it feels natural so it’s not contrived. It’s all JavaScript. 8:49 – Panel. 9:28 – Guest: Idiomatic Ember for example. Idiomatic in the sense that it gives you object for you to work with, which are simple objects. 10:12 – Guest: You have your reducers and your...we could do those things but ultimately it’s powerful – and not action names – we use method names; the name of the method. 11:20 – Panel: I was digging through docs, and it feels like NORMAL JavaScript. It doesn’t seem like it’s tied to a certain framework or library platform? 11:45 – Guest: Yes, we felt a lot of time designing the interfaces the API and the implementation. We wanted it to feel natural but a tool that people reach for. (Guest continues to talk about WHY they created microstates.) Guest: We wanted to scale very well what you need when your needs to change. 13:39 – Chuck: I have a lot of friends who get into React and then they put in Redux then they realize they have to do a lot of work – and that makes sense to do less is more. 14:17 – Guest: To define these microstates and build them up incrementally...building smaller microstates out of larger ones. Guest continued: Will we be able to people can distribute React components a sweet array of components ready for me to use – would I be able to do the same for a small piece of state? We call them state machines, but ultimately we have some state that is driving it. Would we be able to distribute and share? 16:15 – Panel: I understand that this is tiny – but why wouldn’t I just use the native features in specific the immutability component to it? 16:42 – Guest: I’m glad you asked that question. We wanted to answer the question... Guest: With microstates you can have strict control and it gives you the benefit of doing sophisticated things very easily. 18:33 – Guest: You mentioned immutability that’s good that you did. It’s important to capture – and capturing the naturalness of JavaScript. It’s easy to build complex structures – and there is an appeal to that. We are building these graphs and these building up these trees. You brought up immutability – why through it away b/c it’s the essence of being a developer. If you have 3-4-5 levels of nesting you have to de-structure – get to the piece of data – change it – and in your state transition 80% of your code is navigating to the change and only 20% to actually make the change. You don’t have to make that tradeoff. 21:25 – Aimee: The one thing I like about the immutability b/c of the way you test it. 21:45 – Guest: There a few things you can test.  23:01 – Aimee: You did a good job of explaining it. 23:15 – Guest: It makes the things usually hard  easy! With immutability you can loose control, and if that happens you can get so confused. You don’t have a way to have a way to navigate to clarity. That’s what this does is make it less confusing. It gives you order and structure. It gives you a very clear path to do things you need to do. If there is a property on your object, and if there is a way to change it... 25:29 – Guest: The only constant is change no matter what framework you are working on. 24:46 – Chuck: We are talking about the benefits and philosophy. What if I have an app – and I realize I need state management – how do I put microstates into my app? It’s using Angular or React – how do I get my data into microstates? 26:35 – Guest: I can tell you what the integration looks like for any framework. You take a type and you passed that type and some value to the create function so what you get is a microstate. (The Guest continues diving into his answer.) 28:18 – Guest: That story is very similar to Redux, basically an event emitter. The state changes on the store. Maybe this is a good time to talk about the stability benefits and the lazy benefits because microstates is both of those things. Stability – if I invoke a transition and the result is unchanged – same microstate – it doesn’t emit an event. It recognizes it internally. It will recognize that it’s the same item. Using that in Ember or Redux you’d have to be doing thousands of actions and doing all that computation, but stability at that level. Also, stability in the sense of a tree. If I change one object then that changes it won’t change an element that it doesn’t need to change. 31:33 – Advertisement: Sentry.io 32:29 – Guest: I want to go back to your question, Chuck. Did we answer it? 32:40 – Chuck: Kind of. 32:50 – Guest. 32:59 – Guest: In Angular for example you can essentially turn a microstate... 33:51 – Guest: You could implement a connect, too. Because the primitive is small – there is no limit. 34:18 – Chuck summarizes their answers into his own words. 34:42 – Guest: If you were using a vanilla React component – this dot – I will bind this. You bind all of these features and then you pass them into your template. You can take it as a property...those are those handlers. They will perform the transition, update and what needs to be updated will happen. 35:55 – Chuck: Data and transitions are 2 separate things but you melded them together to feel like 1 thing. This way it keeps clean and fast. 36:16 – Guest: Every framework helps you in each way. Microstates let’s you do a few things: the quality of your data all in one place and you can share. 38:12 – Guest: He made and integrated Microstates with Redux tools. 38:28 – Guest talks about paths, microstates to trees. 39:22 – Chuck. 39:25 – Panel: When I think about state machines I have been half listening / half going through the docs. When I think of state machines I think about discreet operations like a literal machine. Like a robot of many steps it can step through. We have been talking about frontend frameworks like React - is this applicable to the more traditional systems like mechanical control or is it geared towards Vue layered applications? 40:23 – Guest: Absolutely. We have BIG TEST and it has a Vue component. 41:15 – Guest: when you create a microstate from a type you are creating an object that you can work with. 42:11 – Guest: Joe, I know you have experience with Angular I would love to get your insight. 42:33 – Joe: I feel like I have less experience with RX.js. A lot of what we are talking about and I am a traditionalist, and I would like you to introduce you guys to this topic. From my perspective, where would someone start if they haven’t been doing Flux pattern and I hear this podcast. I think this is a great solution – where do I get started? The official documents? Or is it the right solution to that person? 43:50 – Guest: Draw out the state machine that you want to represent in your Vue. These are the states that this can be in and this is the data that is required to get from one thing to the other. It’s a rope process. The arrow corresponds to the method, and... 44:49 – Panel: It reminds me back in the day of rational rows. 44:56 – Guest: My first job we were using rational rows. 45:22 – Panelist: Think through the state transitions – interesting that you are saying that. What about that I am in the middle – do you stop and think through it or no? 46:06 – Guest: I think it’s a Trojan horse in some ways. I think what’s interesting you start to realize how you implement your state transitions. 48:00 – (Guest continues.) 48:45 – Panel: That’s interesting. Do you have that in the docs to that process of stopping and thinking through your state transitions and putting into the microstate? 49:05 – Guest: I talked about this back in 2016. I outlined that process. When this project was in the Ember community. 49:16 – Guest: The next step for us is to make this information accessible. We’ve been shedding a few topics and saying this is how to use microstates in your project. We need to write up those guides to help them benefit in their applications. 50:00 – Chuck: What’s the future look like? 50:03 – Guest: We are working on performance profiling. Essentially you can hook up microstates to a fire hose. The next thing is settling on a pattern for modeling side effects inside microstates. Microstates are STATE and it’s immutable. 52:12 – Guest: Getting documentation. We have good README but we need traditional docs, too. 52:20 – Chuck: Anything else? 52:28 – Guest: If you need help email us and gives us a shot-out. 53:03 – Chuck: Let’s do some picks! 53:05 – Advertisement for Charles Max Wood’s course! Links: Kendo UI Frontside Redux Microstates Microstates with React Taras Mankovski’s Twitter Taras Mankovski’s GitHub Taras Mankovski’s LinkedIn Taras Mankovski’s Frontside Bio Charles Lowell’s Twitter Charles Lowell’s GitHub Charles Lowell’s Frontside Bio Schedule Once Ruby on Rails Angular Get A Coder Job YouTube Talks Email: cowboyd@frontside.io Working with State Machines Twitch TV BigTest Close Brace REEF The Developer Experience YouTube Video Sponsors: Kendo UI Sentry.io – 2 months free – DEVCHAT/code Get A Coder Job Picks: Aimee ShopTalk Episode 327 Professional JavaScript for Web Developers Technical Debt Stripe Taras Twitch Channel Big Test Frontside Charles Lowell Chalkboards Sargent Art Chalk Chris Close Brace LaCroix Water Chris’s Git Hub Joe The Developer Experience Bait and Switch Good Bye Redux Recording Dungeon and Dragons AJ UtahJS Conf Start with Why The Rust Book VanillaJS w/ Chris Zero to One Charles Podwrench.com -  beta getacoderjob.com

Panel: Aimee Knight Charles Max Wood Joe Eames AJ O’Neil Chris Ferdinandi  Special Guests: Charles Lowell (New Mexico) & Taras Mankovski (Toronto) In this episode, the panel talks with two special guests Charles and Taras. Charles Lowell is a principle engineer at Frontside, and he loves to code. Taras works with Charles and joined Frontside, because of Charles’ love for coding. There are great personalities at Frontside, which are quite diverse. Check out this episode to hear about microstates, microstates with react, Redux, and much more! Show Topics: 1:20 – Chuck: Let’s talk about microstates – what is that? 1:32 – Guest: My mind is focused on the how and not the what. I will zoom my mind out and let’s talk about the purposes of microstates. It means a few things. 1.) It’s going to work no matter what framework you are using. 2.) You shouldn’t have to be constantly reinventing the wheel. React Roundup – I talked about it there at this conference.  Finally, it really needs to feel JavaScript. We didn’t want you to feel like you weren’t using JavaScript. It uses computer properties off of those models. It doesn’t feel like there is anything special that you are doing. There are just a few simple rules. You can’t mutate the state in place. If you work with JavaScript you can use it very easily. Is that a high-level view? 7:13 – Panel: There are a lot of pieces. If I spoke on a few specific things I would say that it enables programming with state machines. 7:42 – Panel: We wanted it to fell like JavaScript – that’s what I heard. 7:49 – Aimee: I heard that, too. 7:59 – Guest. 8:15 – Aimee: Redux feels like JavaScript to me. 8:25 – Guest: It’s actually – a tool – that it feels natural so it’s not contrived. It’s all JavaScript. 8:49 – Panel. 9:28 – Guest: Idiomatic Ember for example. Idiomatic in the sense that it gives you object for you to work with, which are simple objects. 10:12 – Guest: You have your reducers and your...we could do those things but ultimately it’s powerful – and not action names – we use method names; the name of the method. 11:20 – Panel: I was digging through docs, and it feels like NORMAL JavaScript. It doesn’t seem like it’s tied to a certain framework or library platform? 11:45 – Guest: Yes, we felt a lot of time designing the interfaces the API and the implementation. We wanted it to feel natural but a tool that people reach for. (Guest continues to talk about WHY they created microstates.) Guest: We wanted to scale very well what you need when your needs to change. 13:39 – Chuck: I have a lot of friends who get into React and then they put in Redux then they realize they have to do a lot of work – and that makes sense to do less is more. 14:17 – Guest: To define these microstates and build them up incrementally...building smaller microstates out of larger ones. Guest continued: Will we be able to people can distribute React components a sweet array of components ready for me to use – would I be able to do the same for a small piece of state? We call them state machines, but ultimately we have some state that is driving it. Would we be able to distribute and share? 16:15 – Panel: I understand that this is tiny – but why wouldn’t I just use the native features in specific the immutability component to it? 16:42 – Guest: I’m glad you asked that question. We wanted to answer the question... Guest: With microstates you can have strict control and it gives you the benefit of doing sophisticated things very easily. 18:33 – Guest: You mentioned immutability that’s good that you did. It’s important to capture – and capturing the naturalness of JavaScript. It’s easy to build complex structures – and there is an appeal to that. We are building these graphs and these building up these trees. You brought up immutability – why through it away b/c it’s the essence of being a developer. If you have 3-4-5 levels of nesting you have to de-structure – get to the piece of data – change it – and in your state transition 80% of your code is navigating to the change and only 20% to actually make the change. You don’t have to make that tradeoff. 21:25 – Aimee: The one thing I like about the immutability b/c of the way you test it. 21:45 – Guest: There a few things you can test.  23:01 – Aimee: You did a good job of explaining it. 23:15 – Guest: It makes the things usually hard  easy! With immutability you can loose control, and if that happens you can get so confused. You don’t have a way to have a way to navigate to clarity. That’s what this does is make it less confusing. It gives you order and structure. It gives you a very clear path to do things you need to do. If there is a property on your object, and if there is a way to change it... 25:29 – Guest: The only constant is change no matter what framework you are working on. 24:46 – Chuck: We are talking about the benefits and philosophy. What if I have an app – and I realize I need state management – how do I put microstates into my app? It’s using Angular or React – how do I get my data into microstates? 26:35 – Guest: I can tell you what the integration looks like for any framework. You take a type and you passed that type and some value to the create function so what you get is a microstate. (The Guest continues diving into his answer.) 28:18 – Guest: That story is very similar to Redux, basically an event emitter. The state changes on the store. Maybe this is a good time to talk about the stability benefits and the lazy benefits because microstates is both of those things. Stability – if I invoke a transition and the result is unchanged – same microstate – it doesn’t emit an event. It recognizes it internally. It will recognize that it’s the same item. Using that in Ember or Redux you’d have to be doing thousands of actions and doing all that computation, but stability at that level. Also, stability in the sense of a tree. If I change one object then that changes it won’t change an element that it doesn’t need to change. 31:33 – Advertisement: Sentry.io 32:29 – Guest: I want to go back to your question, Chuck. Did we answer it? 32:40 – Chuck: Kind of. 32:50 – Guest. 32:59 – Guest: In Angular for example you can essentially turn a microstate... 33:51 – Guest: You could implement a connect, too. Because the primitive is small – there is no limit. 34:18 – Chuck summarizes their answers into his own words. 34:42 – Guest: If you were using a vanilla React component – this dot – I will bind this. You bind all of these features and then you pass them into your template. You can take it as a property...those are those handlers. They will perform the transition, update and what needs to be updated will happen. 35:55 – Chuck: Data and transitions are 2 separate things but you melded them together to feel like 1 thing. This way it keeps clean and fast. 36:16 – Guest: Every framework helps you in each way. Microstates let’s you do a few things: the quality of your data all in one place and you can share. 38:12 – Guest: He made and integrated Microstates with Redux tools. 38:28 – Guest talks about paths, microstates to trees. 39:22 – Chuck. 39:25 – Panel: When I think about state machines I have been half listening / half going through the docs. When I think of state machines I think about discreet operations like a literal machine. Like a robot of many steps it can step through. We have been talking about frontend frameworks like React - is this applicable to the more traditional systems like mechanical control or is it geared towards Vue layered applications? 40:23 – Guest: Absolutely. We have BIG TEST and it has a Vue component. 41:15 – Guest: when you create a microstate from a type you are creating an object that you can work with. 42:11 – Guest: Joe, I know you have experience with Angular I would love to get your insight. 42:33 – Joe: I feel like I have less experience with RX.js. A lot of what we are talking about and I am a traditionalist, and I would like you to introduce you guys to this topic. From my perspective, where would someone start if they haven’t been doing Flux pattern and I hear this podcast. I think this is a great solution – where do I get started? The official documents? Or is it the right solution to that person? 43:50 – Guest: Draw out the state machine that you want to represent in your Vue. These are the states that this can be in and this is the data that is required to get from one thing to the other. It’s a rope process. The arrow corresponds to the method, and... 44:49 – Panel: It reminds me back in the day of rational rows. 44:56 – Guest: My first job we were using rational rows. 45:22 – Panelist: Think through the state transitions – interesting that you are saying that. What about that I am in the middle – do you stop and think through it or no? 46:06 – Guest: I think it’s a Trojan horse in some ways. I think what’s interesting you start to realize how you implement your state transitions. 48:00 – (Guest continues.) 48:45 – Panel: That’s interesting. Do you have that in the docs to that process of stopping and thinking through your state transitions and putting into the microstate? 49:05 – Guest: I talked about this back in 2016. I outlined that process. When this project was in the Ember community. 49:16 – Guest: The next step for us is to make this information accessible. We’ve been shedding a few topics and saying this is how to use microstates in your project. We need to write up those guides to help them benefit in their applications. 50:00 – Chuck: What’s the future look like? 50:03 – Guest: We are working on performance profiling. Essentially you can hook up microstates to a fire hose. The next thing is settling on a pattern for modeling side effects inside microstates. Microstates are STATE and it’s immutable. 52:12 – Guest: Getting documentation. We have good README but we need traditional docs, too. 52:20 – Chuck: Anything else? 52:28 – Guest: If you need help email us and gives us a shot-out. 53:03 – Chuck: Let’s do some picks! 53:05 – Advertisement for Charles Max Wood’s course! Links: Kendo UI Frontside Redux Microstates Microstates with React Taras Mankovski’s Twitter Taras Mankovski’s GitHub Taras Mankovski’s LinkedIn Taras Mankovski’s Frontside Bio Charles Lowell’s Twitter Charles Lowell’s GitHub Charles Lowell’s Frontside Bio Schedule Once Ruby on Rails Angular Get A Coder Job YouTube Talks Email: cowboyd@frontside.io Working with State Machines Twitch TV BigTest Close Brace REEF The Developer Experience YouTube Video Sponsors: Kendo UI Sentry.io – 2 months free – DEVCHAT/code Get A Coder Job Picks: Aimee ShopTalk Episode 327 Professional JavaScript for Web Developers Technical Debt Stripe Taras Twitch Channel Big Test Frontside Charles Lowell Chalkboards Sargent Art Chalk Chris Close Brace LaCroix Water Chris’s Git Hub Joe The Developer Experience Bait and Switch Good Bye Redux Recording Dungeon and Dragons AJ UtahJS Conf Start with Why The Rust Book VanillaJS w/ Chris Zero to One Charles Podwrench.com -  beta getacoderjob.com

Guest: Philip Poots: GitHub | ClubCollect Previous Episode: 056: Ember vs. Elm: The Showdown with Philip Poots In this episode, Philip Poots joins the show again to talk about the beauty of simplicity, the simplicity and similarities between Elm and Ruby programming languages, whether Elixir is a distant cousin of the two, the complexity of Ember and JavaScript ecosystems (Ember helps, but is fighting a losing battle), static vs. dynamic, the ease of Rails (productivity), and the promise of Ember (productivity, convention). The panel also talks about the definition of "quality", making code long-term maintainable, and determining what is good vs. what is bad for your codebase. Resources: Michel Martens mote Learn the Elm Programming Language and Build Error-Free Apps with Richard Feldman Worse is Better: Richard P. Gabriel Gary Bernhardt's Destroy All Software Screencasts Zen and the Art of Motorcycle Maintenance: An Inquiry into Values The Calm Company It Doesn't Have to Be Crazy at Work This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. Transcript: CHARLES:: Hello, everybody and welcome to The Frontside Podcast, Episode 113. My name is Charles Lowell. I'm a developer here at the Frontside and with me today are Taras Mankovski and David Keathley. Hello? DAVID:: Hey, guys. TARAS: Hello, hello. CHARLES:: And we're going to be talking with a serial guest on our serial podcast, Mr Philip Poots, who is the VP of engineering at ClubCollect. Welcome, Philip. PHILIP: Hey, guys. Thanks for having me on. CHARLES:: Yeah. I'm actually excited to have you on. We've had you on a couple of times before. We've been trying to get you on the podcast, I think for about a year, to talk about I think what has kind of a unique story in programming these days. The prevailing narrative is that folks start off with some language that's dynamically typed and object oriented and then at some point, they discover functional programming and then at some point, they discover static programming and they march off into a promised land of Nirvana and no bugs ever, ever happening again. It seems like it's pretty much a straight line from that point to the next point and passing through those way stations. When I talk to you, I guess... Gosh, I think you were the first person that really introduced me to Elm back at Wicked Good Ember in 2016 and it seemed like you were kind of following that arc but actually, that was a bit deceptive because then the next time I talked to you, you were saying, "No, man. I'm really into Ruby and kind of diving in and trying to get into Ruby again," and I was kind of like, "Record scratch." You're kind of jumping around the points. You're not following the preordained story arc. What is going on here? I just kind of wanted to have a conversation about that and find out what the deal was and then, what's kind have guided your journey. PHILIP: There was one event and that was ElmConf Europe, which was a fantastic conference. Really, one of the best conferences I've been to, just because I guess with the nature of early language, small conference environment. There's just a lot of things happening. There's a lot of people. Evan was there, Richard Feldman was there, the leading lights of the Elm community were there and it was fantastic. But I guess, one thing that people have always said to me is the whole way track is the best track of the conference and it's not something I really appreciated before and during the breaks, I ended up talking to a guy called Michel Martens. He is the finder of a Redis sourcing company and I guess, this was just a revelation to me. He was interested in Elm. He was friends with the guys that organized the conference and we got talking and he was like, "I do this in Ruby. I do this in Ruby. I did this in Ruby," and I was like, "What?" and he was like, "Yeah, yeah, yeah." He's a really, really humble guy but as soon as I got home, I checked him out. His GitHub is 'soveran' and it turns out he's written... I don't know, how many gems in Ruby, all with really well-chosen names, very short, very clear, very detailed. The best thing about his libraries is you can print them out on paper. What I mean by that is they were tiny. They were so small and I guess, I just never seen that before. I go into Ruby on Rails -- that was my first exposure to programming, that was my first exposure to everything -- unlike with Rails, often when you hit problems, you'd start to dive a bit deeper and ultimately, you dive so deep that you sunk essentially and you just accepted, "Okay, I'm not going to bend the framework that way this time. Let's figure out how everyone else goes with the framework and do that." Then with Ember when I moved into frontend, that was a similar thing. There were so many layers of complexity that I never felt like had a real handle on it. I kind of just thought this was the way things were. I thought it's always going to be complex. That's just the nature of the problem. That's just the problem they're trying to solve. It's a complex problem and therefore, that complexity is necessary. But it was Elm that taught me, I think that choosing the right primitives and thinking very carefully about the problem can actually give you something that's quite simple but incredibly powerful. Not only something quite simple but something so simple that it can fit inside your head, like this concept of a program fitting inside your head and Rails, I don't know how many heads I need to fit Rails in or Ember for that matter and believe me, I tried it but with Elm, there was that simplicity. When I came across this Ruby, a language I was very familiar with but this Ruby that I had never seen before, a clear example was a templating library and he calls it 'mote' and it's including comments. It's under a hundred lines of code and it does everything you would need to. Sure, there were one or two edge cases that it doesn't cover but it's like, "Let's use the trade off." It almost feels like [inaudible] because he was always a big believer in "You ain't going to need it. Let's go for that 80% win with 20% effort," and this was like that taken to the extreme. CHARLES:: I'm just curious, just to kind of put a fine point on it, it sounds like there might be more in common, like a deeper camaraderie between this style of Ruby and the style encouraged by Elm, even though that on the surface, one is a dynamically typed object oriented language and the other is a statically typed functional language and yet, there's a deeper philosophical alignment that seems like it's invisible to 99% of the discussion that happens around these languages. PHILIP: Yeah, I think so. I think the categories we and this is something Richard Feldman talks. He's a member of the Elm community. He does a lot of talks and has a course also in Frontend Masters, which I highly recommend. But he often talks about the frame of the conversation is wrong because you have good statically typed languages and you have bad statically typed languages. You have good dynamic languages and you have bad dynamic languages. For all interpretations of good and bad, right? I don't want to start any wars here. I think one of the things that Elm and Ruby have in common is the creator. Matz designed Ruby because he wanted programming to be a joy, you know? And Evan created Elm because he wanted programming to be a delight. I think if you experience both of those, like developing in both of those languages, you gain a certain appreciation for what that means. It is almost undefinable, indistinguishable, although you can see the effects of it everywhere. In Ruby, everything is an object, including nil. In Elm, it's almost he's taken everything away. Evan's taken everything away that could potentially cause you to stumble. There's a lot to learn with Elm in terms of getting your head around functional mindset and also, working with types but as far as that goes, people often call it like the Haskell Light, which I think those are a disservice to Elm because it's got different goals. CHARLES:: Yeah, you can tell that. You know, my explorations with Elm, the personality of Elm is 100% different than the personality of Haskell, if that is even a programming term that you can apply. For example, the compiler has an identity. It always talks to you in the first person, "I saw that you did this, perhaps you meant this. You should look here or I couldn't understand what you were trying to tell me." Literally that's how the Elm compiler talks to you. It actually talks to you like a person and so, it's very... Sorry, go ahead. PHILIP: No, no, I think the corollary to that is the principle of the surprise in Ruby. You know, is there going to be a method that does this? You type it out and you're like, "Oh, yes it is," which is why things like inject and reduce are both methods in enumerable. You didn't choose one over the other. It was just like, "Let's make it easy for the person who's programming to use what they know best." I think as well, maybe people don't think about this as deeply but the level of thought that Evan has put into designing Elm is crazy, like he's thought this through. I'm not sure if I said this the last time but I went to a workshop in the early days in London, which is my kind of first real exposure to Elm and Evan was giving the workshop. Someone asked him, "Why didn't you do this?" and he was like, "Well, that might be okay for now but I'm not sure that would make so much sense in 10 years," and I was kind of like, "What?" Because JavaScript and that ecosystem is something which is changing like practically hourly and this is a guy that's thinking 10 years into the future. TARAS: You might have answered it already but I'm curious of what you think is the difference, maybe it just comes down to that long term thinking but we see this in JavaScript world a lot, which is this kind of almost indifference to APIs. It almost doesn't really matter what the API is for whatever reason, there seems to be a big correlation between the API that's exposed with the popularity of the tool. I think there are some patterns, like something that's really simple, like jQuery and React have become popular because of the simplicity of their APIs. What the flip side to that? What other ways can APIs be created that we see in JavaScript world. Because we're talking about this beautiful APIs and I can relate to some of the work that Charles has been doing and I've been doing microstates but I wonder like what would be just a brief alternative to that API, so it's kind of a beautiful API. PHILIP: I don't know if anyone is familiar with the series of essays 'Worse is Better' like East Coast versus West Coast, from Richard Gabriel. The problem is, I guess and maybe this is just my understanding over my paraphrase of it, I'm not too familiar with it but I think that good APIs take time and people don't have time. If someone launches a V1 at first and it kind of does the job, people will use that over nothing and then whenever they're happy with that, they'll continue to use it and develop it and ultimately, if she's market share and then that's just the thing everyone uses and the other guy's kind of left behind like, "This is so much better." I guess this is a question, I think it was after Wicked Good Ember, I happened to be on the same trend as Tom Dale on the way back to New York and we started talking about this. I think that's his big question. I think it's also a question that still has to be answered, which is, "Will Elm ever be mainstream? Will it be the most popular thing?" aside from the question of whether it has to be or not. For me, a good APIs good design comes from understanding the problem fully -- CHARLES:: And you can understand the problem fully without time. PHILIP: Exactly and often, what happens -- at least this is what happens in my experience with the production software that I've written -- is that you don't actually understand the problem until you've developed a solution for it. Then when you've developed a solution for it, often the pressures or the commercial pressures or an open source is [inaudible] the pressures of backwards compatibility, mean that you can never refactor your way to what you think the best solution is and often, you start from scratch and the reality is people are too far away with the stuff you wrote in the past about the thing you're writing now. Those are always kind of at odds. I think there are a lot of people that are annoyed with Elm because the updates are too slow, it relies on Evan and we want to have a pool request accepted. All of the things that they don't necessarily recognize like the absence of which make Elm an Elm, if you know what I mean. The very fact that Evan does set such a high standard and does want everything to go through his personal filters because otherwise, you wouldn't gain the benefits that Elm gives you. The attention is very real in terms of I want to shift my software now and it becomes easier then. I think to go to a language like JavaScript, which has all of the escape hatches that you need, to be able to chop and change, to edit, to do what you need to do to get the job done and let's be quite honest, I think, also with Elm, that's the challenge for someone who's not an expert level like me. Once you hit a roadblock, you'll say, "Where do I go from here?" I know if I was using JavaScript, I could just like hack it and then clients are happy and everything's fine and you know there's a bit of stuff in your code that you would rather wasn't but at the end of the day, you go home and the job's done. DAVID:: Have you had to teach Elm to other people? You and I did some work like I've seen you pair with someone and guide them through the work that they needs to get done. If you had a chance to do something like that with Elm and see how that actually happens, like how do developer's mind develops as they're working through in using the tool? PHILIP: Unfortunately not. I would actually love to go through that experience. I hope none of my developers are listening to this podcast but secretly, I want to push them in the direction of Elm on the frontend. But no, but I can at least make from my own perspective. I find it very challenging at first because for me, being a Ruby developer and also, I would never say that I understood JavaScript as much as I would have liked. Coming from dynamic language, no functional experience to functional language with types, it's almost like learning a couple of different things at the same time and that was challenging. I think if I were to take someone through it, I would maybe start with a functional aspects and then move on to the type aspects or vice versa, like try and clearly breakdown and it's difficult because those two are so intertwined at some level. Gary Bernhardt of Destroy All Software Screencast, I watch quite a bit of his stuff and I had sent him an email to ask him some questions about one of the episodes that he did and he told me that he done the programming languages course, I think it's on Coursera from Daniel Grossman, so [inaudible] ML which is kind of the father of all MLs like Haskell and also Elm. I find that really helpful because he broke it down on a very basic level and his goal wasn't to teach you ML. It was to teach you functional programming. It would be a very interesting exercise, I think. I think the benefit that Elm gave you is you get to experience that delight very quickly with, "Oh, it's broken. Here's a nice message. I fix the message. It compiles. Wow, it works," and then there's a very big jump whenever you start talking about the effects. Whenever you want to actually do something like HTTP calls or dealing with the time or I guess, the impure stuff you would call in the Haskell-land and that was also kind of a bit weird. CHARLES:: Also, there's been some churn around that, right? PHILIP: That's right. When I started learning, they had signals, then they kind of pushed that all behind the scenes and made it a lot more straightforward. Then I just mastered it and I was like, "Yes, I know it," and then I was like, "All right. I don't need to know it anymore." This is the interesting thing for me because at work, most of our work now is in Elixir and Phoenix. I'm kind of picking a little bit up as I work with them. I think Elm's architecture behind the scenes is kind of based, I believe on Erlang's process model, so the idea of a mailbox and sending messages and dealing with immutable state. CHARLES:: Which is kind of ironically is very object oriented in a way, right? It's functional but also the concept of mailboxes and sending messages and essentially, if you substitute object for process, you have these thousands and thousands of processes that are sending messages back and forth to each other. PHILIP: Yeah, that's right. It's like on a grand scale, on a distributed scale. Although I wouldn't say that I'm that far with Erlang, Elixir to appreciate the reality of that yet but that's what they say absolutely. CHARLES:: Now, Phoenix and Elixir is a dynamically typed functional language. does it share the simplicity? One of the criticisms you had of Rails was that you couldn't fit it in your head. It was very difficult. Is there anything different about Elixir that kind of makes it a spirit cousin of Elm and the simple Ruby? PHILIP: I think so, yes. Absolutely. I don't think it gets to the same level but I think it's in the right direction and specifically on the framework front, it was designed specifically... I mean, in a sense it's like the anti-type to Rails because it was born out of people's frustrations with Rails. José Valim was pretty much one of Rails top core committers. Basically, every Rails application I wrote at one period, at 80% of the code written by José Valim, if you included all the gems, the device and the resourceful and all the rest of it. Elixir in many ways was born out of the kind of limitations of Ruby with Rails and Phoenix was also born out of frustrations with the complexity of Rails. While it's not as simple as say, Michel Martens' Syro which is like his web framework, which is a successor to Cuba if people have heard of that, it is a step in the right direction. I don't understand it but I certainly feel like I could. They have plug which is kind of analogous but not identical to Rack but then the whole thing is built out of plugs. I remember Yehuda Katz give a presentation like 'The Next Five Years' and essentially about Rails 3.0. This is going way back and Phoenix is in some ways the manifestation of his desire to have like the Russian doll pattern, where you could nest applications inside applications and you could have them side by side and put them inside each other and things like that. Phoenix has this concept called umbrella applications which tells that, like Ecto is a really, really nice obstruction for working with the database. CHARLES:: I see. It feels like, as opposed to being functional or static versus dynamic, the question is how do you generate your complexity? How do you cope with complexity? Because I think you touched on it at the beginning of the conversation where you thought that my problems are complex so the systems that I work with to solve those problems must necessarily also be complex. I think one of the things that I've certainly realized, kind of in the later stages of my career is that first part is true. The problems that we encounter are extremely complex but you're much better served if you actually achieve that complexity by composing radically simple systems and recombining them. To the commonality of your system is going to determine how easy it's going to work with and how well it can cope with complexity. What really drives a good system is the quality of its primitives. PHILIP: Absolutely. After ElmConf, I actually invited Michel to come to my place in the Netherlands. He live in Paris but I think he grew up Buenos Aires in Argentina. To my amazement, he said, "Yes, okay," and we spent a couple of days together and there he talked to me about Christopher Alexander and the patterns book, where patterns and design patterns actually grew out of. One of his biggest things was the code is the easiest part, like you've got to spend 80% of your time thinking deeply about the problem, like literally go outside, take long looks. I'm not sure if this is what Rich Hickey means with Hammock Driven Development. I've never actually got around to watching the talk. CHARLES:: I think it's exactly what he means. PHILIP: And he said like once you get at, the code just comes. I think Michel's work, you should really check it out. I'll send you a link to put in the show notes but everything is built out of really small libraries that do one thing and do it really well. For example, he has a library like a Redis client but the Redis client also has something called Nest, which is a way to generate the keys for nested hashes. Because that's a well-designed, the Redis client is literally just a layer on top. If you understand the primitive then, you can use the library on top really well. You can embed Syro applications within Syro applications. I guess, there you also need the luxury of time and I think this is where maybe my role as VP of engineering, which is kind of my first role of that kind, comes in here which is when you're working on the commercial pressure, try to turn around to a business guy and say, "Yes, we'll solve this problem but can we take three weeks to think about it?" It's never going to happen -- CHARLES:: No. PHILIP: Absolutely, it will never going to happen. Although the small things that I tried to do day to day now is get away from the computer, write on paper, write out the problem as you understand it, attack it from different angles, think about different viewpoints, etcetera. CHARLES:: I think if you are able to quantify the cost of not thinking about it for three weeks, then the business person that you're going to talk to is their ears are going to perk up, right? But that's so hard to do. You know, I try and make like when we're saying like, "What technologies are you going to choose? What are the long term ramifications in terms of dollars or euros or whatever currency you happen to be in for making this decision?" I wish we had more support in thinking about that but it is kind of like a one-off every time. Anyway, I'm getting a little bit off track. PHILIP: No, not at all. This is a subject I love to talk about because we kind of had a few a bit of turbulence because we thought, maybe we should get product people in, maybe we should get them a product team going and what I find was -- and this is maybe unique to the size of the company -- that actually made things a lot more difficult because you got too many heads in many ways. Sometimes, it's better to give the developer all of the context so that he can think about it and come up with the best solution because ultimately, he's the only one who can understand. I wouldn't say understands the dollars and cents but he understands the cost implications of doing it in efficient ways, which often happens when you're working in larger teams. TARAS: One thing I find really interesting about this conversation is the definition of good is really complicated here. I've observed Charles work on microstates and I work with him, like I wrote a lot of the code and we got through like five or six iterations and at every point, he got better but it is so difficult to define that. Then when you start to that conversations outside of that code context and you start to introduce business into the mix, the definition of good becomes extremely complicated. What do you think about that? How do we define it in a way? Are there cultures or engineering cultures or societal cultures that have a better definition for good that is relevant to doing quality work of this? CHARLES:: That's a deep question. PHILIP: Wow. Yeah, a really, really deep question. I think often for business, like purely commercially-driven, money-oriented good is the cheapest thing that gets the job done and often that's very short term, I think. As you alluded to Charles, that people don't think about the cost of not doing the right things, so to speak in our eyes and also, there's a huge philosophical discussion whether our definition of good as programmers and people who care about our craft is even analogous to or equal to a good in a commercial context. CHARLES:: Yes, because ultimately and this is if you have read Zen in the Art of Motorcycle Maintenance, one of the things that Pirsig talks about is what is the definition of quality. How do we define something that's good or something that's bad? One of the definitions that gets put forward is how well something is fit to purpose. Unless you understand the purpose, then you can't determine quality because the purpose defines a very rich texture, a very rich surface and so, quality is going to be the object that maps very evenly and cleanly over that surface. When it comes to what people want in a program, they're going to want very different thing. A developer might need stimulation for this is something that's very new, this is something that's going to keep my interest or it's going to be keeping my CPU max and I'm going to be learning a whole lot. A solution that actually solves for that purpose is going to be a high quality solution. Also, this is going to be fast. We're going to be able to get to market very quickly. It might be one of the purposes and so, a solution that is fast and the purpose fits so it's going to be good. Also, I think developers are just self-indulgent and looking for the next best thing in something that's going to keep their interest, although we're all guilty of that. But at the same time, we're going to be the ones maintaining software, both in our current projects and collectively when we move to a new job and we're going to be responsible for someone else's code, then we're going to be paying the cost of those decisions. We both want to minimize the pain for ourselves and minimize the pain for others who are going to be coming and working in our code to make things long term maintainable. That's one axis of purpose and therefore, an axis of quality. I think in order to measure good and bad, you really have to have a good definition of what is the purpose of that surface is so rich but the more you can map it and find out where the contours lie, the more you're going to be able to determine what's good and what's bad. TARAS: It makes me think of like what is a good hammer. A sledgehammer is a really good hammer but it's not the right hammer for every job. CHARLES:: Right. TARAS: I think what you're saying is understanding what is it that you're actually doing and then matching your solution to what you're actually trying to accomplish. PHILIP: Yeah, absolutely and in my experience, we have a Ruby team building a Rails application. That's our monolith and then, we have a couple of Elixir teams with services that have been spun out of that. This isn't proven. This is just kind of gut feel right now and it is that Elixir is sometimes slower to develop the same feature or ship it but in the long term it's more maintainable. I haven't actually gotten dived into to React and all of the amazing frameworks that it has in terms of getting things up and running quickly but in terms of the full scale application, I still think 10, 11 years on, Rails has no equal in terms of proving a business case in the shortest time possible. CHARLES:: Yeah. I feel very similarly too but the question is does your development team approach the problem as proving a business case or do they approach the problem as I want to solve the set of features? PHILIP: Yes. Where I'm working at the moment, I started out just as a software developer. I guess, we would qualify for 37 signals or sorry... base camps definition of a calm company -- CHARLES:: Of a what company? PHILIP: A calm company. Sorry. They just released a new book and called 'The Calm Company' and 'It Doesn't Have to Be Crazy at Work.' I was given in my first couple of months, a problem. It was business oriented, it had to be solved but it had to be solved well from a technical perspective because we didn't want to have to return to it every time. It was standardizing the way that we exported data from the database to Excel. You know, I was amazed because it was literally, the first time that I'd been given the space to actually dive in on a technical level to do that kind of stuff. But I think even per feature, that varies and that sometimes challenging when handing the work on because you've got to say, "This fit. Literally, we're just trying to prove, whether if we have this feature, the people will use it?" versus, "This is a feature that's going to be used every day and therefore, needs to be at good, technical quality." Those are the tradeoffs that I guess, keep you in a job. Because if it was easy, then you would need anyone to figure it out but it's always a challenge. What I like is that our tools are actually getting better and I think, with Elm for example, it's kind of major selling point is maintainability and yet, with Elm, there haven't been that many companies with Elm over a period of years that exists, that can live to tell the tale. Whereas, we certainly know with Rails applications have done well like Basecamp and GitHub. For sure, they can be super maintainable but the fact that it took GitHub to just moved Elm to Rails 5.0, I belief, the fact that it took them years and years and they were running off at fork of Rails 2.3, I think it shows the scale of the problem in that way. You know, Phoenix also went through a few issues, kind of moving architectures from the classic Rails to a more demand driven design model. I think we're getting there slowly, zig-zagging towards a place where we better understand how to write software to solve business problems. I guess, I was really interested in microstates when you shared it at Wicked Good Ember because that to me was attacking the problem from the right perspective. It's like given the fact that the ecosystem is always changing. How can we extract the business logic such that these changes don't affect the logic of our application? CHARLES:: Man, we got a lot to show you. It has changed quite a bit in the last two years. Hopefully, for the better. TARAS: It's been reduced and it's almost a quarter of its size while maintaining the same feature set and it's faster, it's lazier, it's better in every respect. It's just the ideas have actually been fairly consistent. It's just the implementation that's evolved. CHARLES:: Yeah, it's been quite a journey. It parallels kind of the story that we're talking about here in the sense that it really has been a search for primitives and a search for simplification. One of the things that we've been talking about, having these Ruby gems that do one thing and do it very, very, very well or the way that Elixir being architected has some very, very good primitives or Elm, the same kind of thing being spiritually aligned, even though on the surface, it might share more in common with Haskell. There's actually a deep alignment with a thing like Ruby and that's a very surprising result. I think one of the things that appeals to me about the type of functional programming that is ironically, I guess not present in Elm, where you have the concept of these type classes but I actually think, I love them for their simplicity. I've kind of become disenchanted with things like Lodash, even though they're nominally functional. The fact that you don't have things like monoid and functors and stuff is kind of first class participants in the ecosystems, means you have to have a bunch of throwaway functions. Those API surface area is very large, whereas if you do account for those things, these kind of ways of combining data and that's how you achieve your complexity, is not by a bunch of one-off methods that are like in Lodash, they're all provided for you so you don't have or have to write them yourself. That is one level of convenience but having access to five primitives, I think that's the power of the kind of the deeper functional programming types. PHILIP: And Charles, do you think that that gives you the ability to think at a higher level, about the problems that you're solving? Would you make that link? CHARLES:: Absolutely. PHILIP: So, if we're not doing that, then we're actually doing ourselves a disservice? CHARLES:: I would say so. PHILIP: Because we're actually creating complexity, where it shouldn't exist? CHARLES:: Yeah, I think if you have a more powerful primitive, you can think of things like async functions and generator functions, there's a common thread between async functions, generator functions, promises arrays and they're all functors. For me, that's a very profound realization and there might be a deeper spiritual link between say, an async function and an array in the same way that there's a deep spiritual link between Ruby and Elm, that if you don't see that, then you're doing yourself a disservice and you're able to think at a higher level. Also, you have a smaller tool set where each tool is more powerful. PHILIP: You did a grit, I think it was a repository with a ReadMe, where you boiled down what people would term what I would term, the scary functional language down to a very simple JavaScript. Did you ever finish that? Did you get to the monads? CHARLES:: I did get to the monads, yeah. PHILIP: Okay. I need to check that out again. I find that really, really helpful because I think one of Evan's big things with Elm is he doesn't use those terms ever and he avoids them like the plague because I think he believes they come tinged with the negative experiences of people trying Haskell and essentially getting laughed at, right? CHARLES:: Yes. I think there's something to that. TARAS: But we're doing that in microstates as well, right? In microstates documentation, even though microstates are written completely with these functional primitives, on the outside, there's almost no mention of it. It's just that when you actually go to use it, if you have an idea, one of the thing that's really powerful with microstates is that this idea that you can return another microstate from a transition and what that will do is what you kind of like what a flat map would do, which is replace that particular node with the thing that you returned it with. For a lot of people, they might not know that that's like a flat map would do but a microstate will do exactly what they wanted to do when it didn't realize that's actually should just work like that. I think, a lot of the work that we've done recently is to package all things and it make it powerful and to access the concepts that it is very familiar, something you don't need to learn. You just use it and it just works for you. CHARLES:: Right but it is something that I feel like there's unharvested value for every programmer out there in these type classes: monads and monoids and functors and co-functors or covariant functors, contravariant functors, blah-blah-blah, that entire canon. I wish there was some way to reconcile the negative connotations and baggage that that has because we feel kind of the same way and I think that Evan's absolutely right. You do want to hide that or make it so that the technology is accessible without having to know those things. But in the same way, these concepts are so powerful, both in terms of just having to think less and having to write less code but also, as a tool to say, "I've got this process. Is there any way that could it be a functor? If I can find a way that this thing is a functor, I can just save myself so much time and take so many shortcuts with it." PHILIP: And in order to be able to communicate that, or at least communicate about that, you need to have terms to call these things, right? Because you can't always just refer to the code or the pattern. It's always good to have a name. I'm with you. I see value in both, like making it approachable, so the people who don't know the terms are not frightened away. But I also see value in using the terms that have always existed to refer to those things, so that things are clear and we can communicate about them. CHARLES:: Right. definitely, there's a tradeoff there. I don't know where exactly the line is but it would be nice to be able to have our cake and eat that one too. We didn't get really to talk about the type versus dynamic in the greater context of this whole conversation. We can explore that topic a little bit. PHILIP: Well, I can finish with, I think the future is typed Erlang. Maybe, that's Elm running on BEAM. CHARLES:: Whoa. What a take? Right there, folks. I love it. I love it but what makes you say that? Typed Erlang doesn't exist right now, right? PHILIP: Exactly. CHARLES:: And Elm definitely doesn't run on BEAM. PHILIP: I don't know if I'm allowed to say this. When I was at this workshop with Evan, he mentioned that and I'm not sure whether he mentioned it just as a throwaway comment or whether this is part of his 20-year plan but I think the very fact that Elm is designed around like Erlang, the signal stuff was designed around the way Erlang does communication and processes, it means I know at least he appreciates that model. From my point of view, with my experience with Elixir and Erlang in production usage, it's not huge scale but it's scale enough to need to start doing performance work on Rails and just to see how effortless things are with Elixir and with Erlang. I think Elm in the backend would be amazing but it would have to be a slightly different language, I think because the problems are different. We began this by saying that my story was a little different to the norm because I went back to the dynamic, at the dark side but for example in Elixir, I do miss types hugely. They kind of have a little bit of a hack with Erlang because they return a lot of tuples with OK and then the object. You know, it's almost like wrapping it up in a [inaudible]. There are little things and there's Dialyzer to kind of type check and I think there are a few projects which do add types to Erlang, etcetera. But I think something that works would need to be designed from the ground up to be typed and also run in the BEAM, rather than be like a squashed version of something else to fit somewhere else, if that makes sense. CHARLES:: It makes total sense. PHILIP: I think so. I recently read a book, just to finish which was 'FSharpForFunAndProfit' is his website, Scott Wlaschin, I think. It's written up with F# but it's about designing your program in a type functional language. Using the book, you could probably then just design your programs on paper and only commit to code at the end because you're thinking right down to the level of the types and the process and the pipelines, which to me sounds amazing because I could work outside. CHARLES:: Right. All right-y. I will go ahead and wrap it up. I would just like to say thank you so much, Philip for coming on and talking about your story, as unorthodox as it might be. PHILIP: Thank you. CHARLES:: Thank you, Taras. Thank you, David. TARAS: Thank you for having us. CHARLES:: That's it for Episode 113. We are the Frontside. This is The Frontside Podcast. We build applications that you can stake your future on. If that's something that you're interested in, please get in touch with us. If you have any ideas for a future podcast, things that you'd like to hear us discuss or any feedback on the things that you did here, please just let us know. Once again, thank you Mandy for putting together this wonderful podcast and now we will see you all next time.

Guests: Amanda Hickman: @amandabee | GitHub Amberley Romo: @amberleyjohanna | GitHub | Blog In this episode, Amanda Hickman and Amberley Romo talk about how they paired up to get the safety pin, spool of thread, and the knitting yarn and needles emojis approved by the Unicode Committee so that now they are available for use worldwide. They also talk about how their two path crossed, how you can pitch and get involved in making your own emojis, and detail their quest to get a regular sewing needle approved as well. Resources: Unicode Technical Committee Draft Emoji Candidates The Unicode Consortium Members Sewing-Emoji Repo Proposal for Sewing NEEDLE AND THREAD Emoji This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. Transcript: ROBERT: Hello, everyone. Welcome to Episode 112 of The Frontside Podcast. I'm Robert DeLuca, a software developer here at the Frontside and I'll be your episode host. With me as co-host is Charles Lowell. Hey, Charles. CHARLES: Hello, Robert. Good morning. ROBERT: Good morning. This is an exciting podcast. Today, we're going to be discussing writing a proposal to the Unicode Committee, getting it accepted and rejected. This is basically making emojis which I think is really awesome. We have two guests today who have an amazing story, Amanda Hickman and Amberley Romo. Thank you both for joining us. You two have an amazing story that I would really love to dive into and we're going to do that today. It's about basically creating your own emoji and getting that accepted so everybody can use that and I think that's super, super cool, something that I've always kind of wanted to do as a joke and it seems like that's kind of where your stories began, so you two want to jump in and start telling? I think Amanda has a great beginning to this. AMANDA: Sure. I mean, hi and thanks for having me. I don't know where to begin and really for me, this starts with learning to sew my own clothes which is an incredibly exasperating and frustrating process that involves a lot of ripping stitches back out and starting over and Instagram was a really big part of me finding patterns and finding other people who are sewing their own clothes and learning from the process. I wanted to be able to post stuff on Instagram and it started to drive me absolutely crazy, that there's emojis for wrenches and nuts and hammers and there are no textile emoji. The best I could find was scissors which is great because cutting patterns is a place where I spend a lot of time procrastinating but that was it. I knew a woman, Jennifer 8 Lee or Jenny who had led a campaign to get the dumpling emoji into the Unicode character set. I knew she'd succeeded in that but I didn't really know much more about how that had worked. I started thinking I'm going write a sewing emoji. I can do this. I can lead this campaign. I started researching it and actually reached out to Jenny and I discovered that she has created an entire organization called... What was that called? She's created an entire organization called Emojination, where she supports people who want to develop emoji proposals. CHARLES: Before you actually found the support system, you actually made the decision that you were going to do this and you found it. You know, from my perspective, I kind of see emoji is this thing that is static, it's there, it's something that we use but the idea that I, as an individual, could actually contribute to that. I probably, having come to that fork in the road would have said, "Nah, it's just it is what it is and I can't change it." What was the process in your mind to actually say, "You know what? I'm actually going to see if I can have some effect over this process?" AMANDA: It definitely started with a lot of anger and being just consistently frustrated but I knew that someone else had already done this. It was sort of on my radar that it was actually possible to change the emoji character set. I think that if I didn't know Jenny's story and it turned out I didn't know Jenny story at all but I thought I knew Jenny story but if I didn't know that basic thing that that somebody I knew who was a mere mortal like me had gone to the Emoji Subcommittee of the Unicode Consortium and petition them to add a dumpling emoji, I am sure that I wouldn't bother. But I knew from talking to her that there was basically a process and that there were a format that they want proposal in and it's possible to write them a proposal. I knew that much just because I knew Jenny. I think at that point, when I started thinking about this, the Emoji 9 -- I should be more of an expert on that actually, on emoji releases but a new release of emoji had come out. There were a bunch of things in that release and it got a little bit of traction on Twitter. I knew that the Unicode Consortium had just announced a whole new slate of emoji, so I also was generally aware that there was some kind of process by which emoji were getting released and expanded and updated. ROBERT: That's interesting. Do you know when that started? Because it seems like Apple started to add more emojis around like iOS 7 or something but it was pretty static for a while right? Or am I wrong? AMANDA: I actually am tempted to look this up but the other piece that is not irrelevant here is that at the time, I was working at a news organization called BuzzFeed that you may have heard of -- ROBERT: Maybe, I don't know. It sounds kind of familiar. AMANDA: I do feel like people kind of know who they are. I was surrounded by emoji all the time: in BuzzFeed, in internet native of the highest order and we had to use emoji all the time and I had to figure out how to get emoji into blog post which I didn't really know how to do before that. I can put them on my phone but that was it. I was immersed in emoji already. I knew that there was a project called Emojipedia, that was a whole kind of encyclopedia of emoji. One of my colleagues at BuzzFeed, a woman named Nicole Nguyen had written a really great article about the variation in the dance emoji. If you look at the dance emoji, one of the icons that some devices use is this kind of woman with her skirt flipping out behind her that looks like she's probably dancing a tango and then one of the icons that other character sets use and other devices use is a sort of round, yellow lumping figure with a rose in its mouth that you sort of want to hug but it's definitely not to impress you with its tango skill. She had written this whole article about how funny it was that you might send someone this very cute dumpling man with [inaudible] and what they would see was sexy tango woman. I think there was some discussion, it was around that time also that Apple replaced the gun emoji with a water gun. There was some discussion of the direction that the various emoji's face. One of the things that I learned around that time was that every device manufacturer produces their own character set that's native to their devices and they look very different. That means that there's a really big difference between putting a kind of like frustrated face with a gun pointing at it, which I don't really think of it as very funny but that sort of like, "I'm going to shoot myself" is very different from pointing the gun the other way which is very much like, "I'm shooting someone else," so these distinctions, what it means that the gun emoji can point two different ways when it gets used was also a conversation that was happening. None of that answers your question, though which is when did the kind of rapid expanse of emoji start to happen. ROBERT: I feel like the story is setting in the place there, though because it seems like there's a little bit of tension there that we're all kind of diverging here a little bit and it's sort of driving back towards maybe standardization. AMANDA: There's actually, as far as I know, no real move toward standardization but the Unicode Consortium has this committee that actually has representatives from definitely Apple and Microsoft and Google and I forget who else on the consortium. Jenny 8 Lee is now on the consortium and she's on the Emoji Subcommittee but they actually do get together and debate the merits of adding additional emoji, whether they're going to be representative. One of the criteria is longevity and I tend to think of this as the pager problem. There is indeed a pager emoji and I think that the Unicode Consortium wants to avoid approving a pager emoji because that was definitely a short-lived device. CHARLES: Right. I'm surprised that it actually made it. Emoji must be older than most people realize. AMANDA: My understanding is that very early Japanese computers had lots emoji. There's a lot of different Japanese holidays that are represented in emoji, a lot of Japanese food as well are represented in emoji, so if you look through the foods, there's a handful of things that haven't added recently but a lot of the original emoji definitely covered Japanese cuisine very well. ROBERT: I definitely remember when I got my first iPhone that could install iPhone OS 2, you would install an app from the App Store that then would allow you to go toggle on the emoji keyboard but you had to install an app to do it and that's kind of where the revolution started, for me at least. I remember everybody starting to sending these things around. AMANDA: But if you look at Emojipedia, which has a nice kind of rundown of historical versions of the Unicodes, back in 1999, they added what I think of as the interrobang, which is the exclamation/question mark together and a couple of different Syriac crosses. Over the years, the committee has added a whole series of wording icons and flags that all make sense but then, it is around, I would say 2014, 2015 that you start to get the zipper mouth and rolling your eyes and nerd face and all of the things that are used in conversation now -- the unicorn face. ROBERT: My regular emojis. AMANDA: Exactly. CHARLES: It certainly seems like the push to put more textile emoji ought to clear the hurdle for longevity, seeing there's kind of like, what? Several millennia of history there? And just kind of how tightly woven -- pun intended -- those things are into the human experience, right? AMANDA: Definitely. Although technically, there's still no weaving emojis. CHARLES: There's no loom? AMANDA: There's no loom and I think that a loom would be pretty hard to represent in a little 8-bit graphic but -- CHARLES: What are the constraints around? Because ultimately, we've already kind of touched on that the emoji themselves, their abstract representations and there are a couple of examples like the dancing one where the representation can vary quite widely. How do they put constraints around the representation versus the abstract concept? AMANDA: You don't have to provide a graphic but it definitely kind of smooths the path if you do and it has to be something that's representable in that little bitty square that you get. It has to be something representable in a letter-size square. If it's not something that you can clearly see at that size, it's not going to be approved. If it's not something you can clearly illustrate at that size in a way that's clearly distinct from any other emoji and also that's clearly distinct from anything else of that image could be, it's not going to be approved. Being able to actually represented in that little bitty size and I don't know... One of sort of sad fact of having ultimately worked with Emojination on the approval process is that we were assigned an illustrator and she did some illustrations for us and I never had to look at what the constraints were for the illustration because it wasn't my problem. ROBERT: Sometimes, that's really nice. AMANDA: Yes, it's very nice. I ended up doing a lot of research. What made me really sad and I don't want to jump too far ahead but one of things that made me really sad is we proposed the slate and the one thing that didn't get approved was the sewing needle and it also didn't get rejected, so it's in the sort of strange nether space. That's kind of stuck in purgatory right now. I did all this research and learned that the oldest known sewing needle is a Neanderthal needle so it predates Homo sapiens and it's 50,000 years old. CHARLES: Yeah. Not having a sewing needle just seem absurd. AMANDA: Yeah. We have been sewing with needles since before we were actually human being. ROBERT: That's a strong case. AMANDA: Yes, that's what I thought. If I sort go back to my narrative arc, I wanted to do a sewing needle and started researching it a little bit -- CHARLES: Sorry to keep you interrupting but that's literally the one that started this whole journey. AMANDA: Yes, I wanted a sewing needle and I really wanted a sewing needle. I did a little research and then I reach out to Jenny and to ask her if she had any advice. She said, "You should join my Slack," and I was like, "Oh, okay. That's the kind of advice." She and I talked about it and she said that she thought that it made more sense to propose a kind of bundle of textile emoji and I decided to do that. She and I talked it through and I think the original was probably something closer to knitting than yarn but we said knitting, a safety pin, thread and needle were the ones that kind of made the most sense. I set about writing these four proposals and one of the things that they asked for was frequently requested. One other thing that I will say about the proposal format is that they have this outline structure that is grammatically very wonky. They ask you to assert the images distinctiveness and they also ask you to demonstrate that it is frequently requested. I found a couple of really interesting resources. One, Emojipedia which is this sort of encyclopedia of emoji images and history maintains a list of the top emoji requests. I actually don't know how they generate that list or who's requesting that and where but I think it's things that they get emailed about and things people request in other contexts and sewing and knitting, I've done on that list and I started compiling it in 2016. ROBERT: To be a part of the proposal process, to show that it is requested, without that resource, you just start scouring Facebook and Twitter and history and shouting to people like, "I really want this emoji. Why it didn't exist?" That seems pretty hard. AMANDA: Actually our proposals all have Twitter screen shots of people grousing about the absence of knitting emoji and yarn emoji and sewing emoji. I know that Emojipedia, they do a bunch of research so they go out and look at based what people are grousing about on Twitter. They look at places where people are publicly saying like, "It's crazy that there's no X emoji," and that's part of their process for deciding what kinds of emojis people are asking for. Their research was one resource but we took screenshots of people saying that they needed a safety pin emoji and that was part of making the case. One of the things that I found as I was doing that research was that, I guess at this point it was almost two years ago, when the character set that included the dumpling emoji came out, there was a bunch of grousing from people saying, "Why is there not a yarn emoji?" There was a writing campaign that I think Lion Brand had adopted. Lion Brand yarn had put in this tweet saying like, "Everyone should complain. We needed a yarn emoji," but it doesn't matter how much you yell on Twitter. If you don't actually write a proposal, you're not going to get anywhere. I had been told that the Emoji Subcommittee, they're really disinclined to accept proposals that had a corporate sponsor, so they weren't going to create a yarn emoji because Lion Brand yarns wanted them to create a yarn emoji. ROBERT: Right, so it was like counter-peer proposal. AMANDA: Right. But as I was digging around the other thing I found was this woman in... I actually don't know if you're in Dallas or Austin but I found Amberley, who also put a post on Twitter and had started a petition, asking people to sign her petition for a yarn emoji proposal or a knitting emoji. I don't remember if it was a yarn emoji or a knitting emoji but I found her petition and reached out to her to ask if she was interested in co-authoring the proposal with me because she had clearly done the work. She actually had figured out how the system worked at that point. I think she knew who she was petitioning, at least. I reached out to Amberley and we worked together to refine our proposal and figure out what exactly we wanted to request. I think there were a bunch of things that were on the original list like knitting needles, yarn and needles. I think crocheting would have been on the original list. We were sort of trying to figure out what was the right set of requests that actually made sense. ROBERT: So then, this is where Amberley stories comes in and it is interesting too because she has entirely different angle for this. Maybe not entirely different but different than outright. This kind of ties back to the word software podcast mostly. It kind of ties back to the software aspect, right? AMBERLEY: Yeah. I think, really they're kind of separate stories on parallel tracks. My motivation was also two-fold like Amanda's was, where I started knitting in 2013 and I had a really good group of nerd friends with a little yarn shop up in DC, like a stitch and ditch group -- ROBERT: I love it. AMBERLEY: It was a constant sort of like, where's the insert emoji here, like where's the yarn emoji? Where's the knitting emoji? And we would sort of sarcastically use the spaghetti emoji because it was the most visually similar but that was something that was in the back of my mind but it teaches you a lot about yourself too because I was like, "Oh, this is like fiber art, not really an emoji. It's kind of technical, like on a tech space," and I didn't really connect that it was relevant or that I might have any power to change it. It just didn't occur to me at the time. ROBERT: Interesting. I feel like a lot of people are in that similar situation or maybe not situation, even though you can make change on this. AMBERLEY: Right, so my brain didn't even make it like, "Why isn't this a thing? let me look at how to make the thing." When that happened for me, Amanda mentioned using emoji and everything in the BuzzFeed space. I love how you explained BuzzFeed a while ago, it's my favorite description of BuzzFeed I ever heard. Something similar that happened for me was I was a software developer and in 2016, the Yarn package manager was released and that kind of turned something on in my head. That was like I'm seeing all these software engineers now be like, "Where's the yarn emoji?" and I'm like, "Welcome to the club." ROBERT: "Do you want to join our Slack? We can complain together." AMBERLEY: Right. It has been like a pretty decent amount of time, I'm semi-seriously ranting and complaining to my coworkers who were primarily male software engineers. I remember I went to [inaudible] in the Frost Bank Tower after work and was just like, "I'm going to figure out how this happens," and I spent a couple hours at the coffee shop. I found the Unicode site and I found their proposal process and their structure for the proposal and everything and I just started doing the research and drafting up a proposal specifically for yarn. Maybe it was a bit naive of me but to me it was like, "Okay, here's the process. I follow the process. Cool." I mean, you have to make a case and it has to be compelling and has to be well-written and it has to be supported and all that and that to me it was like, "Okay, there's a process. At the same time, I did read about the dumpling emoji but I didn't connect it to Emojination and they had started the Kickstarter. We should talk about this later but I think the sort of idea the issue of representation on the committee and who gets to define language is really interesting but I saw that they had done the Kickstarter and there was a campaign aspect to it, so I ended up just building up this simple site so that if anyone Google, they would find yarn emoji. It's still up at YarnEmoji.com and that was how Amanda found me. I got this random email, I sort of like had this burst of energy and I did all the research and I wrote the draft, sort of piecemeal, filling out the different sections of the way they have it outlined on the Unicode site and then I feel like a month or two went by and I had kind of not looked at it for a bit and then, I get this random email from this website that I almost forgot about. It was like, "Hey, I'm working on this series of proposals. If you're working on knitting or yarn or whatever, maybe we could work together," and I was like, "Well, that's sweet." Then she opened up this whole world to me. There's this whole Emojination organization, sort of 100% devoted to democratizing the process of language formation through creating emojis and so then, I got really into that. My primary motivator was yarn. CHARLES: So what's the status of the yarn spool, those emoji right now? AMBERLEY: The yarn, the spool of thread and the safety pin, they're all approved emoji for the 2018 released. Amanda and I are actually at the end. Amanda, a couple of months ago when I saw someone used the spool thread emoji for a Twitter thread -- you know how people will be like all caps thread and have a thread of tweets -- I saw someone do that just out of the blue. I was like, "Oh, my God. Is it out?" and the thing about these individual vendors, it sort of gets released piecemeal, so at the time Twitter have I think released their versions of this series of new emoji but others hadn't. CHARLES: How does that work? Because you think the Twitter would be kind of device depending on what browser you're using, like if you're on a Windows or a Mac or a Linux Box, right? ROBERT: -- Emoji set, right? I know Facebook does this too. AMBERLEY: I'm painfully aware that Facebook does it because I can't use the crossed finger emoji on Facebook because it actually gives me nightmares. ROBERT: I have to go look at this now. AMBERLEY: Because it's so creepy-looking. CHARLES: Okay. Also like Slack, for example is another. It's like a software-provided emoji set. AMANDA: Right. AMBERLEY: I'm not totally sure that Slack actually adheres to the standard Unicode set. I think it's kind of its own thing but I might be wrong about that. AMANDA: Sorry, Slack definitely supports the full Unicode set. They also have a bunch of emoji that they've added that aren't part of the set. AMBERLEY: Slack emojis? AMANDA: Yes. CHARLES: Yeah and then every Slack also has its kind of local Slack emoji. AMBERLEY: Right. CHARLES: But how does that work with --? ROBERT: Okay, this crossed-finger Facebook emoji is... yes, I agree with you, Amberley. AMBERLEY: Thank you. I had yet to find someone who disagrees with me about that. AMANDA: I have never seen it before and I'm now like, "What is going on?" CHARLES: Yeah, so how does it work if a vendor like Twitter is using a different emoji set? How does that work with cut and paste, like if I want to copy the content of one tweet into something else? Are they using an image there? AMANDA: They're using an image. I think it's doesn't happen as much anymore but for a long time, I would often get texts from people and the text message would have that little box with a little code point in it and you were like -- AMBERLEY: More like an alien thing? AMANDA: Yeah. Definitely, if you don't have the emoji character set that includes the glyph that you're looking at, you're going to get that little box that has a description of the code point and I think what's happening is that Twitter is using JavaScript or generally programming. There were air quotes but you can't see. Twitter is using their software to sub in their emoji glyph whenever someone enters that code point. Even if you don't have the most up to date Unicode on your computer, you can still see those in Twitter. If I copy and paste it into a text editor on my computer, what I'm going to see is my little box that says '01F9F5' in it but if I get it into Twitter, it shows up. I can see them on Twitter but I can't see them anywhere else. AMBERLEY: Damn, you really have the code point memorized? AMANDA: No, I -- CHARLES: Oh, man. I was really hoping -- AMBERLEY: Oh, man. ROBERT: You live and breathe it. AMANDA: No, I'm not that compulsive. AMBERLEY: We definitely have our emojis on our Twitter bios, though. AMANDA: Absolutely. ROBERT: If you see Amanda's bio, it's pretty great. AMANDA: They started showing up on Twitter and I think that somebody in Emojination probably told me they were out and that was when I first started using them. Amberley might have actually seen it. It sounds like you just saw it in the wild, which is kind of amazing. AMBERLEY: I saw it in the wild with this tweet thread and yeah, it's just [inaudible]. I was like, "Amanda, is it out?" CHARLES: Yeah, I feel like I saw that same usage too, although I obviously did not connect any dots. AMANDA: This last week, October 2nd -- I'm also looking things up. I'm just going to come to the fact that I am on a computer looking things up so I can fact check myself -- after they actually released their emoji glyph set, so by now any updated iOS device should have the full 2018 emoji, which in addition to a kind of amazing chunky yarn and safety pin, there's also a bunch of stuff. There's a broom and a laundry basket. There's a bunch of really basic, kind of household stuff that certainly belongs in the character set alongside wrenches and hammers. AMBERLEY: I think one of the big ones too for this year was the hijab? AMANDA: No, the hijab actually came out with a dumpling. Hijab has been available -- AMBERLEY: It's been up, okay. ROBERT: So did it come with iOS 12 or 12.1? I don't know for sure. I just know -- AMANDA: I'm looking at it and it's 12.1. I really feel that I should be ashamed that I have used the internet and search for this. AMBERLEY: I would say, I have no idea what their release numbers are. AMANDA: [inaudible] as it appeared for the first time in iOS for 2018 with today's release of the iOS 12.1, Beta 2 for developers. ROBERT: That is amazing. Do you get some kind of satisfaction -- like you have to, right? -- from people using the emoji and it's starting to make its way out there? AMANDA: So much. Oh, my God, yeah. AMBERLEY: I didn't really expect it, like saying that random tweet using this spool of thread for a tweet thread. I just thought and I just got so psyched. For me, I'm a knitter. I have knitter friends and it started with yarn and then really, Amanda and through Amanda, Jenny really sort of broadened my idea of what it all really meant. To think someone using it in the wild for a totally different application than I had ever thought of was like, "That's legit." AMANDA: I definitely have a sewing emoji search in my tweet deck and sometimes, when I'm feeling I need a little self-validation, I'll go look over there and find people who are saying things like, "Why is there no sewing emoji?" and I'll just reply with all the sewing emoji, like it is part of my work in this life to make sure that not only do they exist but people know about them. ROBERT: That is awesome. I would do the same thing, though to be honest. You'll be proud of that. AMANDA: Totally. ROBERT: Were there any hitches in the proposal process? I know we're kind of alluded to it but the thing that you started off one thing, Amanda didn't make it. Right? AMANDA: I know. ROBERT: So how did that process happen from you two meet each other and then going through the actual committee and the review process and then being accepted. What would that mean? AMANDA: The process is actually incredibly opaque. We wrote this whole proposal, a bunch of people edited it, which is one of the other nice things about collaborating with Emojination. There was a bunch of people who are just really excited about emoji and the kind of language making that Amberley was talking about. There's a whole bunch of people who just jumped in and gave us copy edits and feedback, which was super helpful and then, there was a deadline and we submitted it to the committee and it actually shows up in the Unicode register which is also a very official kind of document register. I was a little excited about that too but then they have their meeting. They first have a meeting and there's like a rough pass and the Emoji Subcommittee makes formal recommendations to the Unicode Consortium and then the consortium votes to accept or reject the Emoji Subcommittee's recommendations. It's a very long process but unless you're going and checking the document file and meeting minutes from the Unicode Consortium meetings, you'll never going to know that it happen. AMBERLEY: -- You know someone connected through there because one of the things in our first pass, it wasn't that it was rejected. It was that we needed to modify something. We do have art for knitting needles with yarn because at one point, I think we weren't totally sure that a ball of yarn would be visually distinct enough in this emoji size to look like yarn and so, we had put it with sort of knit piece on knitting needles. AMANDA: Oh, that's right. There was a tease of a little bit of knitted fabric. AMBERLEY: Right and I think that, probably through Jenny or the people actually in the room, the feedback I remember is that there is a crocheter in the room who was like, "Yeah, why isn't there a yarn emoji but knitting needle?" so there was a little bit of like that was how I think we ended up from knitting needles with a fiber piece to ball of yarn, maybe. AMANDA: I think that sounds right. I'm actually sure of that. It's just all coincide with my recollection. There were some things that they had questions about and that happened really fast because I feel like we had a couple of days and they have stuck to our guns and said, "No, we're only interested in knitted bit of fabric." Also, we worked with an illustrator and went back and forth with her because the initial piece that she had illustrated, I feel like the knitting needles were crossing in a way. That was not how knitting works and so, there was a little bit of back and forth around that as well. But then once they decided that the they like the thread, yarn and safety pin, we're going to move to the next stage. I actually had to go back and look at the minutes to find out that the two reasons that they didn't move the sewing needle on to the next stage is when they thought it was adequately represented by the thread, which I wholeheartedly disagree with and they thought it wasn't visually distinctive. That's so much harder because a sewing needle, which is really just a very fine piece of metal with an eye at the end, you get down to a really small size and it is maybe a little hard to know what you're looking at. But I think there's such a big difference between the static object which is the spool or the thread which represents a lot of things and is important and the needle, which is the active tool that you use to do the making, to do the mending, to do the cobbling. CHARLES: Yeah. I'm surprised that it almost isn't reversed when certainly in my mind, which I think is more culturally important in terms of the number of places which it appears, it's definitely the needle as being kind of... Yeah. AMANDA: Yeah and I think that the thread and yarn, they're important and I think that the decision to have a ball of yarn rather than a bit of knitting makes sense because there's a lot of things that you can use a ball of yarn that aren't just knitting and they think that -- AMBERLEY: And it's the first step too that doesn't exclude anyone in the fiber art community. AMANDA: But there's so many things like in sutures and closing wounds, you're not using a little spool of cotton thread for that or polyester thread and stuff like embroidery and beadwork, you might be using thread or fiber of some sort that started on a spool but you might not. Embroidery floss was not sold in a spool and there's all these places where we use needles and all kinds of different size and you don't always use thread. Sometimes, you're using yarn. Sometimes, you're using leather cord. Sometimes, you're using new bits of, I would say Yucca. You're using plant fibers to do baskets and in all of these different practices, that process of hooking it through the eye and sewing it is how it's actually made. It still sort of mystifies me why they haven't accepted it but they also didn't reject it, which is really interesting. I don't know how many other emoji are sort of sitting in this weird nether space because sometimes they just reject them outright. I think there was a proposal for a coin that they just said no. ROBERT: They were a like, "A coin?" That would be [inaudible]. AMANDA: Oh, God. ROBERT: They have to add one for every -- AMANDA: [inaudible]. CHARLES: Literally, the pager of 2017. AMANDA: Exactly. CHARLES: So what recourse is now available to you all and to us, by extension, to get the sewing needle? AMANDA: I'm actually working on a revised proposal and I've been trying to figure out what are all the arguments that I'm missing for why sewing and the needle are not adequately represented by the thread and yarn. A bunch of things that a friend of my named, Mari who's half-Japanese, half-American but lives in Guatemala and does all this kind of arts in textile work, pointed out that there's a whole holiday in Japan devoted to bringing your broken needles and thanking them for their service. I thought that was really cool. I've been trying to formulate what are all of the arguments for the necessity of both a needle and a spool. If anybody has interesting ways to phrase that, I would love for arguments. CHARLES: Yeah but it's hard to imagine the arguments is just anything being more compelling than the arguments the you just laid out that you named about seven context: shoemaking, medicine, different fibers where the needle operates completely and totally independent of the thread. It's looming so large in kind of our collective conscious like holidays, being dedicated to them, except I think the Cro-Magnon pager, which is made out of stone, I believe, the being the artifact that pre-dates... AMANDA: There's the idiom landscape as well. Things like finding a needle in a haystack, that has a very specific meaning -- ROBERT: And for puns. I've been resisting saying a pun this whole time. AMANDA: Oh, share your pun with us. AMBERLEY: Yeah, you have to say it. ROBERT: Well, you could say that trying to get this through the committee is like threading a needle. Butchered but -- AMANDA: There's a biblical quote about getting into heaven -- a camel through the eye of a needle. I forget actually how it... CHARLES: To thread a camel through the eye of a needle than for a rich man to get into heaven. AMANDA: Exactly and there's this sort of do-re-mi, saw, a needle pulling thread. There are all these places where it's about the needle and somebody had -- CHARLES: It's primarily ancient. AMANDA: I know. CHARLES: It is the prime actor. Maybe, this is a good segue into kind of talking about the makeup of the committee and the decision making process and these kind of what seem like very clear arguments might not be received as such. AMANDA: I certainly don't want to say anything bad about anybody on the committee. CHARLES: No, no, I don't think that there's anything bad. I think that being receptive to things which are familiar to us versus with things that aren't is a very natural human thing and it can be interesting to see that at work and at play. AMANDA: The Unicode Consortium is also evaluating all of these requests for whole language glyphs sets. Lots of languages and lots of character sets that are kind of obvious, like there has to be a sort of like character set like there has to be an Arabic character set but there are a lot of languages that have been left out of that because they're very small minority languages or they are historical languages, where the actual writing is no longer written the same way but there's historical reasons to be able to represent those characters. One of the reasons why the Emoji Subcommittee cares about what gets into the formal character set is that everybody has to accommodate it and there's already been, I think some grousing. People start to moan and groan about how there's too many emoji, then it's too hard to find things. CHARLES: And there's no take backs. AMANDA: There's no take backs. You can't undo it. The committee is made up of representatives from a lot of tech companies primarily, although there's a couple of other kind of odd additional folks on there. I do try to find the committee list and I can find it right now. AMBERLEY: I have it from Emojination. I don't know if it's up to date but Oracle, IBM, Microsoft, Adobe, Apple, Google, Facebook, Shopify and Netflix. The other voting members -- ROBERT: Shopify? AMBERLEY: Yeah, right? The others being the German software company, SAP and the Chinese telecom company, Huawei and the Government of Oman. AMANDA: Yeah, the Government of Oman is a fascinating one. I don't think they're the ones that are biting us on this. Especially for those tech companies, every time the emoji character set adds 10 or 12 emoji, they don't have to accommodate it on their devices. They have to put illustrators on it, they have to deal with everyone saying that the crossed fingers emoji in Facebook looks like I-don't-even-know-what. AMBERLEY: Hey, Amanda. AMANDA: It's all your fault. There's a whole process and there's non-trivial work associated with every single new emoji, so wanting to put the brakes on a little bit and be intentional about where and when they apply that work, it doesn't seem crazy to me. I just want them to approve the thing that I want. AMBERLEY: I like the way that Emojination captures it. I looked at their website earlier and actually, they take it down but their goal quote "Emojination wants to make emoji approval an inclusive representative process." There has to be a process. There's overhead involved but looking at the makeup of the decision makers are not a trivial question. CHARLES: Right. This is a great example like [inaudible] metaphor but these little artifacts, these emojis are literally being woven to the fabric of a global culture and certainly, everybody uses them and they become part of the collective subconscious. It does seem like very important to be democratic in some way. It sounds like there is a process but making sure that everyone has a stake. AMBERLEY: Yeah. ROBERT: What was the reason that they gave for not accepting the needle and thread? Was it like a soft no? You said it's like just hanging out, not really rejected but not accepted. We're going to drop a link in the show notes for the proposal and your GitHub and everything. I'm looking at the PDF that was put together and it seems like it was all a package deal like we talked about. How do they just draw or they just take like a lawyer would, just like draw or cross it out like, "Well, no but we'll take the other ones." AMANDA: Yes, basically. What they did is they need to discuss and I don't know how long they've been meeting but they need to discuss all of the proposals that have been supplied by a particular deadline and -- ROBERT: That sounds painful. AMANDA: Yeah, I mean, it's -- ROBERT: Just imagine the power of thinking about emojis. AMANDA: One of the things that they rejected, I think because there's the smiling poo face. Somebody wanted a frowning poo face and they rejected that. There's a bunch of things that actually do get rejected. I don't know if they've been really care about a smiling poo face versus a frowning poo face. ROBERT: What about an angry one? AMANDA: We got all the feelings of poo. ROBERT: We got important work to do here. AMANDA: But they go through when they're trying to figure out. I think to some degree, you want to get them when they're not tired but I think the status that it's listed right now is committee pushback, so they've set it aside until we have some concerns. We're not going to reject it outright but we're not really sure why this isn't adequately represented. Then their most recent meeting, they just kind of passed on reconsidering it, which is fine because I think I was traveling and my proposal is not done. I really want to make sure that I have consolidated every imaginable argument in one place so -- ROBERT: And make it strong as possible. AMANDA: Yeah. If people want to help the other thing that would be amazing is any and all idioms that you can think of, especially ones that are not in English or European languages, idioms in Central European languages, idioms in Asian languages that refer to needles, either translations of the kind of classic, 'finding a needle in a haystack,' but also any idioms that are kind of unusual and specific to a culture outside of what I have experience with would be amazing for making the case, so this is an international need. ROBERT: Do they need any specific or actionable feedback or do they just say, "We're going to push back on this. We're just not quite sure?" AMANDA: The two things that we're in the minutes -- there are minutes and they publish the minutes to Unicode.org -- were it was not visually distinct, which is not totally crazy. We actually worked with an illustrator to get a different image. The first image was almost at 90 degrees. It was kind of straight up and down and it is a little hard to see and the second is -- ROBERT: Especially, because it's thin. AMANDA: The second image is actually a kind of stylized needle because it's fairly a little fatter and the eye is bigger but it's much more distinctively a needle. I'm hoping that that will also convince them but you have to be able to tell at a very small size that it's a needle. The other thing that they said was that sewing was already represented by the thread, that we didn't need thread and needle but it was literally one line in the minutes that referenced that and then it sort of like, "Did you have somebody in the room or not?" and so, if there is somebody on the committee who is willing to tell you really what their concerns were, then you have some sense of what they're looking for and why they're pushing back. When you can very much see in the earliest emoji character sets that I have a hammer and I have a wrench and I use them but there's these very conventionally male tools. We have all of the kind of office supplies but all of homemaking and housekeeping and textile production, none of them were there until very, very recently. I think it does reflect the gender of the people who've been making these tools, that sewing and knitting weren't important enough as human practices to be included in this glyph set. AMBERLEY: I guess, that's non-trivial to mention because that wasn't an argument that I made in my original yarn draft and Amanda and Jenny sort of pushing to open it up to this whole slate of craft emoji. I didn't realize until they brought that up. I took a stroll through pretty much the whole slate of emoji and you can count on almost one hand the number that represented the creative endeavors or sort of more traditionally known as creative things like camera or painting palette and stuff like that. It was extremely limited. AMANDA: I think they have stuff like that. I think there's a few different variations on the camera and then there's painting palette and that's it. AMBERLEY: Oh, there's the theater mask. AMANDA: Oh, that's right. There is the theater, the happy and sad -- AMBERLEY: And I don't know it exactly and I haven't read the minutes like Amanda has but I think and I hope that that was a particularly compelling piece of that argument. AMANDA: I think they definitely heard it. AMBERLEY: Yeah. CHARLES: Opening it up then, what else is coming in the way of craft? It sounds like this is historical but these pieces are being filled in not only with the work that you all are doing but by other emoji which you're appearing. AMANDA: Yes. CHARLES: And are you in contact with other people who are kind of associated with maybe craft and textiles and other kind of what you're labeling historically creative spaces? AMANDA: I don't think there are anymore with a possible exception. Someone's working on a vinyl record proposal which I think is great. CHARLES: Yeah, that's awesome. ROBERT: Antiquated, though. AMANDA: Maybe not, I don't know. AMBERLEY: Take a stroll through the Emojination Slack and people discussed that. AMANDA: Yeah. If you click at Emojination.org, the whole Airtable database is on there. There's not a lot of other creative ones. A friend of mine got really bent out of shape about the lack of alliums and wrote a whole slate proposal for leeks and scallions and garlic and onions. ROBERT: Oh, there is a garlic one, right? AMANDA: No. I mean, there is -- AMBERLEY: Actually, I'm looking at the Unicode page for current emoji candidates. They first get listed as... I forget the exact order. They become draft candidates and then provisional candidates or vice versa but I don't see any pending further creative ones but garlic and onion are on there. AMANDA: Yes. ROBERT: That makes my Italian a little happy. AMANDA: I think there's some prosthesis, the mechanical leg and the mechanical arm, a guide dog -- AMBERLEY: Ear with hearing aid, service dog. AMANDA: Yeah, there's a good chunk of interesting things that have been left out. I guess they've been approved by the subcommittee but are still waiting on final approval by the Unicode Consortium. ROBERT: Okay. What are the next steps that we can do to help push the thread and needle proposal through it. You mentioned a couple things like coming up with idioms that are in different languages and whatnot but how can we contact you and push this effort and help? AMANDA: That's such a good question. I don't even know. I mean, I am Amanda@velociraptor.info and you're totally welcome to email me if you want to help with this and I will -- ROBERT: That's a great domain, by the way. AMANDA: Unfortunately, there's no information about velociraptors anywhere on that site. ROBERT: That's the way it should be. AMANDA: But also, if you're excited about working on emoji proposals, Emojination is an incredibly great resource and folks there, including me actually will help you identify things that are on other people's wish lists that you could work on if you just want to work on something and we'll help you refine your proposal if you know what you want and we'll help you figure out whether it's worth putting the time in or not and how to make it compelling. You can definitely check out Emojination.org. I think there's a path to get on to the Slack from there. AMBERLEY: Oh, yeah. The Slack and the Airtable. AMANDA: Yeah. ROBERT: It sounds like there's a whole community that was born out of this, where everybody is trying to help each other and collaborate and get their shared ideas across. AMANDA: Definitely and there's a woman, Melissa Thermidor who is fantastic, who actually is a social media coordinator. It's her actual title but she works for the National Health Service in the UK and was tasked with getting a whole series of health-related emoji passed. There's a bunch of things that she's -- AMBERLEY: Is she's the one doing blood. AMANDA: She's doing blood. AMBERLEY: That's a good one. AMANDA: Because there's a lot of really important health reasons why you need to be able to talk about blood and getting blood and blood borne illnesses and -- AMBERLEY: That one was listed on the emoji candidate page or blood donation medicine administration. AMANDA: Yeah. ROBERT: That's really interesting, so she works for the government, right? and that was part of her job to do that? AMANDA: Yes. ROBERT: That's awesome, actually. I love that. AMANDA: Yeah, I think the drop of blood, the bandage and the stethoscope are the three that are in the current iteration, which is interesting because the existing medical emoji were the pill and that gruesome syringe with a little drops of fluid flying off of it, which do not do a lot to encourage people to go to the doctor. ROBERT: No, not at all. AMANDA: So a few more, we're welcoming medical emoji. ROBERT: You have a GitHub. Is that where you're still doing for the follow up and the prep work for the sewing emoji? AMANDA: Yeah, that's probably the best place. I do have a Google Docs somewhere but that's probably a better place to connect even than my ridiculous Velociraptor email. The GitHub -- ROBERT: But it's still awesome. AMANDA: It is awesome. I won't lie. I'm very proud of it. I am AmandaBee -- like the Bumble Bee -- on GitHub and the sewing emoji, the original proposals are there and I will make sure that there is information about how to plug into the revised needle proposal there as well. You guys are a tech podcast, so if people want to just submit suggestions as issues on that repository, that's awesome. We'll totally take suggestions that way. ROBERT: That would be pretty rad. Well, I appreciate you two being on the podcast. I love hearing your stories and how it ended up converging in parallel tracks but it end up achieving the same goal. Still unfinished, right? Let's see if we can help push this over the finish line and get it done because I would really like to see a needle. I could definitely use that in many of my conversations already now, making all kinds of puns. Thank you, Amanda for coming on and sharing your story. AMANDA: Thanks for having me. ROBERT: And thank you, Amberley for also coming on and sharing your story. This was super awesome. AMBERLEY: Yeah and thank you for connecting us to finally have a voice conversation. AMANDA: I know. It's great to actually talk to you, Amberley. CHARLES: Oh, wow, this is the first time that you actually talked in audio? AMANDA & AMBERLEY: Yeah. ROBERT: We're making things happen here. The next thing we have to do is get this proposal through and accepted. AMANDA: Yes. CHARLES: You've converted two new faithful sewing and needle partisans here and I'm in. AMANDA: Awesome. ROBERT: I know you've already gotten, what? Three through accepted? AMANDA: Yeah. ROBERT: We talked about that, it's got to be really awesome. I think I want to try and jump in and get that same satisfaction because a lot of people use emojis. AMANDA: Exactly. CHARLES: It definitely makes me think like you look at every single emoji and there's definitely a story. Especially for the ones that have been added more recently, there's a lot of work that goes into every single pixel. That represents a lot of human time, which I'm sure you all know, so thank you. AMANDA: Thanks for having us on. AMBERLEY: Yeah, thank you guys. ROBERT: Cool. That is the podcast. We are Frontside. We build UI that you can stick your future on. I really love this podcast because it wasn't necessarily technical but had a lot of interesting conversation about how to work with a proposal and probably make a bigger impact than any of us with software, just because the sheer reach that emojis have are insane and the fact that you can influence this process is new to me and really cool, so I hope a lot of other people learn from that too. If you have any feedback that you would like to give us on the podcast, we're always open to receive feedback. We have our doors and ears open, so if you like to send an email at Contact@Frontside.io or shoot us a tweet or DM us at @TheFrontside on Twitter. We'd love to hear it. Thank you, Mandy for producing the podcast. She always does an amazing job with it. You can follow her on Twitter at @TheRubyRep. Thanks and have a good one.

In this episode, Robert, Charles, and Wil talk about the whys and hows of accessibility, as well as what makes single page applications special, why they are they harder for accessibility, and frameworks that can do this for you. Resources: #SkyQ app on #iOS from a #VoiceOver user's Perspective Rob's Routing Doc Wil's PR Single Page Apps routers are broken Greater Than Code Episode #92: A11y Ally with Rob DeLuca This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. Transcript: ROBERT: Hello everyone. Welcome to The Frontside Podcast. This is Episode 111. I'm Robert DeLuca, a software developer here at the Frontside and I'll be your episode host. Today, we're going to be discussing accessibility in single page apps. With me as co-hosts are Charles Lowell. Hey, Charles. CHARLES: What is up Robert? ROBERT: And Wil Wilsman. Hey, Wil. WIL: Yo, yo, yo, yo. ROBERT: Sounds like we're ready to drop a disc track. We're not going to be dissing anybody here. We're going to be talking about helpful things with accessibility in single page apps. Before we get into the nitty-gritty of accessibility in single page apps because we're getting into some deep stuff, I think I want to cover a lot of 'how' because I know accessibility things are usually about why you should be doing it and then they touch on things like, "You should be using alt attributes for your images," but for single page apps, I think we need to go further. CHARLES: It always ends up like, "Then draw the rest of the accessibility owl." ROBERT: Yeah, here's your two circles and the alt-tags are your circles and then the rest of the freaking owl is focus management and everything else that comes with it. Before we get there, what is accessibility? I guess, if we trim the giant umbrella down a little bit from everything that is accessibility that can be physical space things, like wheelchair accessible ramps or things like that, what about just technology? WIL: People need assistive tech to interact with technology such as switches or keyboards and obviously screen reader is a big one but when it comes to the software itself, you could even talk about colors and people who are color blind, so red might not be red to everybody. ROBERT: Speaking from experience there? WIL: Yeah. I'm colorblind. Red is brown to me. CHARLES: Things like hearing and all of it, right? It really is like just designing it in such a way that it can be used by as many people as possible. ROBERT: Right. WIL: And that includes your mom who may not be the best with technology but she still needs to pay your bills online or something. ROBERT: Exactly, yeah and in for context to listeners that might not know, my mom is 100% blind, so it's kind of where it comes from. CHARLES: But my mom is not but she has all kinds of problems. WIL: Yeah, same with my mom. CHARLES: That also falls under the category of accessibility, right? ROBERT: Absolutely. CHARLES: Right, accounting for age and culture. ROBERT: We're blending into the why of accessibility, which is perfect. One of the things that it's such a good segue because what people are starting to realize and I think why accessibility is really starting to catch wind and get some traction is because a lot of people that grew up in the technology age were open in using technology a lot. Our parents probably did not use technology heavily. That's definitely the case for my dad. He still has a flip phone and he says, he's a low tech man living in a high tech world and just refuses to pick up technology but those are the people that just didn't use technology. But now we have a lot of people that grew up with technology and use a lot and they're aging into more disabilities and they're going to need that accessibility, which I think is really interesting to think about because that's a lot of buying power if you're just going to start moving in that needs in accessibility, right? CHARLES: That's true. I know I may need glasses pretty soon, so colors and fonts were going to be heck a lot more to me in the next five years than they have over the last 20. ROBERT: Yup, exactly and that's going to be huge for that and that's one piece of the why, so what are the other reasons that you'd pick up accessibility other than people saying that it's morally correct. I don't like starting off conversations for accessibility because it's the thing that you should be doing. WIL: I think it goes even to my user experience like power users that don't like using the mice or mouse. That's me. I really prefer to just use my keyboard for everything. When the new Firefox browser came out and I couldn't navigate through the menus the way I was used to, I went back to Chrome. ROBERT: That's interesting. CHARLES: I still have not found a good workflow for navigating tabs with the keyboard without just kind of twisting my wrist all out of shape. You have to share that with me but again, that's another thing. That's an impediment that sits between me and the application, that actually -- ROBERT: Which is really interesting, you're getting into a keyboard navigation and focus management, which is really the crux of accessibility for screen reader users and switch users. CHARLES: What I'm hearing is that in this case, including good keyboard and focus management in your application, e.g. making it accessible to screen readers, you at the same time, enabling your power users. I think a great point is that by introducing these very low friction workflows, you're actually going to be enabling other parts of your customer base and not just catering to one, that there are ripple effects throughout your system. ROBERT: It may set it for everyone. WIL: Yeah, not only people who need it but people who don't know they need it. ROBERT: Yes, exactly. Think about the WCAG spec as user experience guidelines. They're not telling you how to implement a thing specifically for a screen reader. They're telling you how to implement it in a way that works for everyone, regardless of what ailment they have. It could be a temporary ailment. It could be a permanent ailment, where they have to use a screen reader or any kind of ailment that you can think of. They're not considering just one use case. It's a broad thing that shows how you can make your application better for everyone. I think that's a better way to look at the WCAG spec than I need to read through this and make sure that this auto complete works for a screen reader. If you look at the guidelines, they're not telling you just first screen reader. They're telling you how to make it work for a switch, someone who is colorblind or who is using a dictation software. I kind of tend to look at that as UX guideline, which really helps me build a better app overall because when you nail down that user flow, everyone benefits from it because it's pretty well thought out at that point. CHARLES: I like that too and I think that thinking of it as user experience and making sure that you have a complete user experience is a good way to think about it because it kind of separates the concerns of, "I've got HTML but is my application really HTML or is it a set of workflows and the data over which those workflows operate?" It really forces you to think my application is not a set of React opponents or web components or Ember components or what have you but really, there's a deep structure to it and it makes you kind of shine a light on that deep structure and try to map its surface. Then, if you really, really know it and you capture it, then you can represent it in any medium. I think it's just a win not just for one niche group of users but also for all of our users and then also for your future users that you don't have because your application is designed better and is going to work. Who knows? Maybe there's some new interface or some new medium, some new device that comes out that hasn't even been invented yet. But if you really have a strong internal representation of what your application is, you're going to be able to be the first to move to that market. ROBERT: Right and like I said earlier, people are starting to aid in the needing accessibility thing. If you need a dollar to justify it, that's going to be a big reason coming up. As a part of the new WCAG 2.1 spec, there is a zoom. I forget what this access criteria number is but the new criteria says your app basically needs to be responsive. That kind of maps directly to what you said earlier, Charles which is like you're going to need glasses soon and fonts and being able to zoom. It's going to be really important. We see that just pop up everywhere. To give a counter example of not just for accessibility like you need it for glasses but the other side of that could be like when we give demos on low resolution projectors or screencast or anything, when we zoom the screen, the apps just still be usable and you should still be able to demo it and that's just something that you have both sides to that, where accessibility kind of works out for everybody. People on the call probably don't need glasses but to see that tiny screen that's being shared, you should be able to assume that. CHARLES: Right. I'm just kind of restating what you said but I want to make sure to call it out explicitly because it was a definitely an aha moment for me, that you basically if you build yourself an accessible website, you build it so that it works on projectors and mobile devices too, so you kind of killed two birds with one stone and you don't have to make a special effort because zooming in the screen is tantamount to viewing it on a phone or viewing it on a tablet. WIL: Yeah, we mentioned it earlier about accessibility and physical space and one of those things is being able to access something from anywhere no matter the device that person is using. ROBERT: Yes. That's a lot of the 'why' of accessibility and I think we did a pretty good job of staying away from the more argument because morally we don't know if it's the right thing to do but a lot of the times it's not in front of you, it's hard to do and I want to make sure that you don't feel bad if you're not building accessibility in your apps. It's not easy. I don't sort of like people get up and say that, "Accessibility is easy. Just do this," because it's not -- WIL: If it was easy, everybody would do it. ROBERT: Exactly. I always come back to accessibility being just like UX -- user experience -- because if it were easy, everybody would have a really great user experience too. It's a hard thing to boil down and to simplify it, right? CHARLES: Right and like every other aspect, the kind of nonfunctional requirement, I say nonfunctional requirement that's a little bit of a contradiction terms but it becomes very hard if you haven't done it from the beginning. But if you didn't start with an accessible app, you weren't thinking about that or you inherited the app or it just wasn't on your radar for whatever reason. If you got a codebase that's two years old, going back and trying to make it accessible, it was extremely hard. It's expensive, expensive, expense. WIL: Yeah, it's going to cost more in terms of money and time to added it after the fact. ROBERT: Exactly. I spent a year on helping on Visa Checkout and there were some accessibility considered in the designs but a lot of the time my feedback wasn't just like, "Yeah, sprinkle some ARIA attributes on there and you're good." It was like, "Do we really need a carousel here to represent your list of carts because it's really hard to navigate around that?" A lot of the times, it ends up boiling up to is this the best way we can represent this data? Is this the best way we can navigate and build this user flow? A lot of times -- CHARLES: And if the answer is no, it's so painful, right? ROBERT: Yeah, exactly, so all that work that was put into building that carousel and all the components that built that carousel together is thrown out because it's just not a pattern that should be used there. Doing after the fact is really hard and really expensive and usually ends up in refactoring anyways. CHARLES: I just want to point that out because a lot of people find themselves in that situation, where they're staring down at a pretty big cost. Now, the reasons why your app may not be accessible or many and good and you shouldn't feel bad, if you're finally in that situation. ROBERT: I gave a talk at Nodevember a couple of years ago called Accessibility Debt and it's just like any technical debt. Your apps going to have it. It's okay. Don't get beat up about it, especially if anybody is trying to beat you up over it, don't listen to them. It really is just like any other kind of technical form of technical debt. It's something that you'll have to deal with and it is just something you have to work through. It's not world ending. It's just another problem to work through. What are some things like everybody usually talks about like the things you can do, the basics for making your app more accessible like using all its attributes and instead of doing a div with an on-click handler, just use a button or don't overuse ARIA attributes. Are there any other things that I missed there like the basics of accessibility? WIL: The biggest is the HTML structure. Screen readers and other assistive tech were built with the standards in mind, so if you're doing nonstandard things like putting divs in H1 and adding ARIA attributes within there, you're not going to have a great time. ROBERT: Right or splattering ARIA roles all over the place, probably not a good idea. That will be harder to debug. Also fun fact, ARIA roles, while you can implement directly to the spec, you may still have bugs across the different screen reader combinations or assistive tech combinations, so that's fine. CHARLES: Keep it super simple is what I'm hearing, like use semantic markup. If you're going to introduce a custom button, still make sure that it's a button. ROBERT: Use the platform. CHARLES: Yeah, use the platform. Don't fight the platform. Probably the best example of that is people implementing their own select boxes. That's the classic example. ROBERT: Wil and I, our lives has centered around that for a little while. It's so true. Usually, it's the first thing people go to grab to reimplement because selects are just ugly. I think Firefox has the ability for you to style select options now like you can change the color and the font but you can't style that. The pop up that comes, usually that's the system dialogue, which a lot of designers don't really like. That's usually the first thing that people go to implement and that's usually actually the first thing that stop somebody from signing up. A lot of sign up forms that I see, if your date of birth is in a select format, that probably will hinder somebody that uses assistive tech from signing up. CHARLES: Yeah. You just basically bounced that entire person. The thing is people don't appreciate the cost. This gets into the whole concept of accessibility that is how much money would that person or those group of people have actually brought into your site versus the cost that you spent redoing that select box. You might be thinking, "It only took this developer actually two weeks," but when you actually look at the actual cost over the course of your application, you're not factoring that into the decision to go with custom select box. Just in our experience, it's just the truly low cost option per quality of experience, that tradeoff there is almost invariably going with platforms select, right? ROBERT: Yeah. Your secret sauce and the best UX that you're going to provide is not going to be nicely styled select box and seriously, a battle that I had to fight a lot was if you really want to implement a custom widget, decide if this is what you want to spend your time on because custom widgets aren't just quick and easy things you implement. You're not going to implement the select that's fully accessible across all the AT combos in a couple of days. CHARLES: It's a lot of work. ROBERT: Yeah. You're going to fall down on a huge rabbit hole. CHARLES: Yeah. It is just you're committing to that work over the lifetime of your application. ROBERT: Exactly, if you can maintain that now. If you implement custom check boxes and custom radios and custom input that of content edible for some reason, I think -- WIL: I think what I see is like custom date pickers -- ROBERT: Oh, I just had a rant about that. CHARLES: Date pickers are hard and there's not really a good option. You just have to open your eyes to the true cost. ROBERT: Right, exactly. That's what I always try to explain, just like you have ownership over this now and you now have to maintain this and you can't regress. CHARLES: Right and if you do regress, it's your neck that gets choked. ROBERT: Yes. A good way to put it. We've talked a lot about things that are just general accessibility but nothing specific to single page apps. I do want to say like a lot of other things that people recommend is like if you're using React, use like the JSX ES1 plugin to help you analyze if you're writing any JSX that might not be accessible or use like HTML_CodeSniffer or aXe to statically analyze the DOM that you have. Those tools are great. I'd see a lot of people [inaudible] those things as like, "Look, this automated checkers says I'm inaccessible so I'm inaccessible," and that's not the case, especially in a single page apps. You can have 100% passing automated checkers but your app also could be 100% broken and why is that? WIL: I don't know. ROBERT: I'm wobbling the router question here. WIL: Yes. I guess that would just be due to routing. In single page applications, they handle their own routing, whereas static web sites and whatnot, the routing is handled despite URLs and the browser and reloading pages. ROBERT: Right. There are probably other major differences between single page apps but the biggest thing is the routing is managed by the client, the JavaScript and on a server-rendered page, you click a link, it's going to rerender the entire page for that next page and the screen reader and the browser know exactly what to do with that, to put the focus back to the top of the page and start working through the page for you. But with single page apps, you're just replacing a piece of the DOM and the screen reader has no idea of what's going on. An automated checker cannot check for this. They cannot tell you if your routes are accessible. The reason I'm bringing this up is because this has never talked about. I don't see in accessibility talks and this is the thing that actually is most broken and makes your app pretty much unusable to anybody that's using assistive tech. There's ways that people if they're savvy can navigate around it but if they don't know what's going on, they're going to think, "All right, I pressed this button and nothing happened, so I'm just going to leave now because it's not working." CHARLES: There's a great video that you point to me, I believe a guy from the UK who has recorded a bunch of his experiences using websites and apps -- ROBERT: Yes, I want to dig it up and put it in the show notes. CHARLES: Yeah. Those are great to watch because you'll really get it. ROBERT: Yeah and that's a really unique case because he's really savvy. I believe, I could be wrong but I think he might have said that he definitely work in tech somehow -- CHARLES: Right. He knows the workarounds and he knows these things but in some of the cases it's like, "If I didn't work in tech, there's no way that I would be able to use this website." ROBERT: Yeah. This is kind of the crux of like if you ever listen to anybody that is an accessibility consultant, they'll say, "You will never ever be able to automate accessibility," and this why I tie accessibility so much to user experience, would you ever have a user experience test that can tell you in a binary fashion? Yes or no, that your app has a great user experience? No, because it's pretty subjective. CHARLES: Of how does your users feel? right? ROBERT: Yeah. CHARLES: You can rate, "My users feel great." ROBERT: Accessibility is like that because there's a lot of context that you have to carry around. It's all about context. When I transition from this page, the next thing does the user have enough context of where they're coming from to where they're going to be able to operate on that page. Is there enough information to achieve the task they want? That is pretty much the crux of why there is no binary yes or no for that because it's contextual. It varies from person to person but you do your best to make sure that that works and that you provide enough information to do something. That's kind of a single page app as a crux. This is why we have a philosophy of testing as a whole. We don't test components individually because again, you can make all of your components individually accessible there but as a whole, they might not work together because you're not providing enough context on an entire page. We did this in one of the apps that we work on, which is open source, so we can link to the PR that Wil wrote for this and I wrote an entire routing documentation around our philosophy and the different things that we tried. How did we manage the focus in that application? I'm kind of just going to lob it over to you Wil since you did the work. Do you want to give context of the holdings and how that all kind of came together? WIL: Yes. One of the features of the app is these panels. It's like this three panel system. When you click an item in a list, a third panel pops up and this goes back to the context thing where if you're using a keyboard, you can't see the screen. You click an item in a list, how do you know that third panel popped up? The solution isn't for a component. The component can't be responsible for this. The list can't focus the item. It opens or vice versa, so this is definitely an application concern, where we needed to check against the route and see whether or not, the pane is opening or was already open or wasn't open before and when this pane opens for the first time, it will just focus it and that gives a lot of the context that the user needs when they click it, like they click an item in the list, it focuses this third pane and they're on a third pane. ROBERT: Right. We didn't even just focus on the entire div. We focused on the heading of the thing that you selected. WIL: Yeah because focusing the div, it might read something off but not all the time. The main thing we're focusing on that third pane opens is the heading to let them know that the item you click, you are now on that page and reading that heading. This is the same thing that would happen if you loaded up that page statically and the screen readers would usually just focus on that first heading. ROBERT: Right. That helped a lot. For a little bit more context, the middle pane is like kind of a master detail thing going on here and in the middle pane there that we have, it's an infinite scrolling list. You have different things there and one of them is like a package. If you select the package and without the focus management and focusing on that heading, you would have to go through every single package that's in that list which could be a thousand of them before you actually get over to the pane that you just opened because source order. You have to go through each one of those. WIL: And it's the same thing is true for power users, not just screen readers. It's like if they want to use tab once that pane open, they have to tab through the entire list. ROBERT: Exactly. It wasn't keyboard accessible and it made it really hard to navigate around with a keyboard because the focus just wasn't being managed. There was a lot of work that we did there. I want to focus on the routing situation there because if you can't navigate around with a keyboard in your single page app, like you click a link and it's not selecting the next thing that should be focused and you provide the right amount of context, your app probably won't be usable without a lot of trial and error to a user, which depending on what your product is, they may not have a lot of patience for trial and error, right? WIL: Yeah. ROBERT: It's really important to try and nail down the routing situation and there are some frameworks and things out there that can do it. In the app that we're talking about, it's React app and we use React router but we don't actually really hook into the router to handle that and that's because React router doesn't provide very much information. WIL: Yeah. You can think of React router as more of an outlet system, where your routes can render anything, anywhere on the page. That's kind of dangers of accessibility and that's kind of the reason that React router can't handle accessible things very well because at any point, the route can just change the button on the page to look like something else. CHARLES: Yeah. It just pop in and pop out. There's no deeper model, right? There's not -- WIL: Yeah, there's no tree. CHARLES: Right, there's the internal state -- WIL: Like a component tree, yeah. CHARLES: The internal state of what is happening is completely opaque. You can only analyze the second and third order effects of the React tree. ROBERT: Right and especially if you have nested route components, it's really hard to determine. One of the things that I've seen people do is focus on mount, which is a very naive approach because what if you have three nested routes, they're all going to focus on mount and the last one that mounts wins and that's a focus for which nobody wins. CHARLES: You all tried that, right? ROBERT: Yes. That's part of the document that I wrote up. We tried three different approaches and we ended up landing on something because we were using React router that was more of like an application state thing, so we were checking props because we knew what the user flow was. We knew what the user, when they come to this page is trying to accomplish, so we're able to kind of figure out from where they came from or where they're going through props and focus the right things for them. WIL: One of the examples, like we talked about the third pane opening and focusing the heading inside but what you know what happens when you close that third pane, you kind of lose contexts again, so we have to focus the previous list item that was active because if you focus back at the top of the list, they've essentially lost their place and the list of results. In that case, we couldn't use on mount. That list item is already mounted. We have to listen to props and we have to look at the route through these props to determine if that third pane was open. If it just closed and if an item in the list is active, it should have focus. It's a lot of logic going on. You have to really understand your app in order to make it a very good accessible app. You can't just sprinkle in ARIA attributes and focus on mount everywhere and think it'll work fine because you're accessible. You have to really know the flow. CHARLES: Right. All those signposts point towards having a deeper application state, a deeper understanding of your application than just the render tree. At that point, it's too late and so by that, I'm definitely lobbying a Reach/Ember router. WIL: Yes. We talked a lot about the React router and we can't really be too accessible with it but to create a React router to go out and there is now Reach router. Robert, have you heard any good things about that? You're the accessibility expert here. ROBERT: I haven't played with it myself, so [audio glitch] things that were lost from the React router three to four transition, I think and also, it's actually accessible routing which is nice because it comes out of the box and you know how to implement it. I haven't played with it. I know Gatsby has implemented that for their V2, that's their default router now, which makes me really happy because a lot of static sites that were being built with Gatsby were very inaccessible and broken which made me sad but now, they're not with V2. I haven't played with Reach router but one of the things that I think it provides which was missing from React router was transition hooks. It has a concept of like where you're going from and to for your routes, which really helps figure out what you need to focus. The one thing I will say about Reach router and I'm sure it's got to be configurable somewhere but I haven't really looked and the demos that I saw, they were just focusing the div of the content that's being rendered, like it kind of just wraps with a generic div or the tabindex="-1" and then focuses it. If you have a lot of content that's inside of that div, it probably will be confusing but at least, it manages the focus somewhere. If you're now off in a no man's land, you have no idea what's going on, at least it focuses that. But if you want to really nail the experience to be better, I would recommend trying to figure out what you should focus inside of that route that you just transition to, what is the best thing. That's for React. There are other things out there for other frameworks, like I am on Ember accessibility team that's out there and we have Ember A11y, which just provide you a focusing outlet for you to be able to just drop in your app and then when the route changes, it does the same thing. It focuses the wrapping div of that outlet that was just rendered. I want to emphasize more in talking about e-holdings work that we did that we were focusing the heading because that told you exactly where you're at and what package you're on or what thing you're on. You know the name of it and now you know how you can go and navigate through that. CHARLES: But it's conceivable, so how would you do that with the Ember router? Would you just introduce some way to delegate down to a particular component? Like when I'm rendered into an outlet, it focus on this component? ROBERT: That would be interesting. I actually don't know. It's been a long time since I've messed around with the Ember router and Ember accessibility. It's definitely a great first step and that's where it kind of came from. I think there needs probably some work done to help implement on what you want to focus. That's kind of where I was going with when we were first exploring the React router stuff and e-holdings was I wanted to have this like high order component that knew of your routing tree and it knew where you're coming from or where you're going. Then from there, it would just tell you, this is the route that we're going to and there's need to be focus management done, like if this prop exists, then you can pick inside of that component that what you want to focus. It leaves it up to the implementor of what they need to focus but it could be probably just like a fallback to the general div because that's not bad. It's a good first step. There needs to be a little bit of work there to get that done probably. CHARLES: Yeah. But it is worth pointing out that by starting from a position of having an externalized application state via the route structure, in an Ember application, you're starting 95% of the way there. An Ember application, at any point, you know where you are and during your transition, you know where you're starting and where you're going to end up and you know when you leave and you know when you got there. ROBERT: Yeah. Having an Ember route pivot handler there, like where you're pivoting from and to is just so nice and it kind of made it click together a lot easier. CHARLES: Right. Reach router looks interesting but as I understand, it's still couched in React components and it feels to me like this is a problem that ought to be solved, that ought to be framework agnostic. Because if I use something like Reach or I use some other routing library for another framework, some other tool might come out that I want to use and I shouldn't be locked in -- ROBERT: Right like what if I really like Redux little router, then I have to make a choice between Redux state that I like or accessibility. CHARLES: Exactly and that feels like a false dichotomy to me. What I would want to be looking for is a platform independent or a framework independent routing library that really just helps you represent your application state and the concrete state in which your application is in at any moment and then, also be able to represent the transitions between those states fully and completely. Then if you have that, you could embed that into any framework. ROBERT: Right. That would be really nice to have. Just like pull it off the shelf and help you out there. CHARLES: If anybody is listening who wants something to do for the next 18 months, no one will thank you for 18 months but you had to get on that. We'll give you lots of thanks then. ROBERT: [inaudible] compare with you. That would be awesome. CHARLES: I would love to be part of that. ROBERT: Yeah, that would be awesome. To kind of tie back to other things, I don't know too much about Angular but I'm sure there are solutions out there to help with Angular. I know Marcy Sutton used to do a lot of work in the Angular world, so I'm sure there's something out there that helps with that. I just don't know off the top of my head right now. I wrote a Medium 'think piece'... No, I wrote a little medium blog post about how all of single page out for routers are broken. I was pleasantly surprised by Vue. Vue brings its own router and their router has a concept of before each, so before each route transition you can run some code and that really helped with implementing the live, the announcer pattern where you use ARIA live to announce something but even beyond there, if you wanted to dig in there, you could probably figure out where you're transitioning from and to and give that next route some kind of attribute that says, "This needs to be focused. Figure out what you need to focus there and inside that route, you can focus wherever you want." I thought that was a really awesome. That's kind of the crux of what I was missing from React router. I wanted the concept of knowing where I'm coming from or where I'm going and I would help with everything but unfortunately, that kind of doesn't exists because they're just components. For better or for worse, they're just components. CHARLES: The world is so much more than just components. ROBERT: Yeah, a little bit off topic, I think it's kind of funny how React kind of just shoved everything into the Vue layer, just to make it all a component. CHARLES: Yeah, it's very easy. ROBERT: Yeah, until it's not. CHARLES: Yeah, exactly. It's easy but it's not simple. ROBERT: That's a lot of talking about how routes need to be done and what you can kind of do to manage focus. It's really about managing the context and how you can provide the most contexts. For somebody to operate on that information, can they complete this thing? What can you do to make sure that you've done this properly? What steps can you take to make sure that you actually are accessible and that your routes work? WIL: Manually testing those screen reader is probably the biggest thing, you know? CHARLES: Yeah. I think the biggest thing is really watching someone who uses assistive tech on a regular basis use your application and then trying to use it yourself. WIL: Yeah. ROBERT: Right. Yeah, I'll definitely -- CHARLES: If you can get a little bit of this yourself but then it's kind of like someone who is never used a mouse before and trying to learn something new. What really helps is seeing someone who's good at it and see how their expectations are either being met or not being met. ROBERT: Yeah, it's definitely the best way. If you can find somebody that actually relies on assistive tech, there's nothing that beats that kind of feedback. If they get to your app and they're really confused, I see some people that just dismiss it because they just don't understand. That is the best feedback you can actually get. If they don't understand what's going on, you have -- CHARLES: That's on you. ROBERT: Yeah, that is going to be what happens for everybody that uses that. Well, maybe not everybody because everybody has different experiences but it's probably going to be a thing that pops up everywhere. But if you don't have access to people that are actually using assistive tech regularly and are pros at it, WebAIM provides really great tutorials for how to use a screen reader. If you're using a Mac, you have a screen reader built in and you can use that called VoiceOver. If you ever want to turn it on or turn it off, its command F5. WIL: You might have to have that shortcut enabled, though. ROBERT: Really? I'm pretty sure it's quite default. WIL: I thought I had to enable mine but I could be wrong. ROBERT: It's interesting. CHARLES: Yeah. They got great tutorial too. It's like it notices the first time you turn it on, so it tries to help you navigate bullet lists and select boxes and input fields and check boxes and all kinds of good stuff. ROBERT: Yeah. It gives you a little bit of a boot camp but WebAIM also helps with specific to website and stuff because one of the things that we ran into while working on the e-holdings project is they're transitioning from a native app to a web app and there were just things that you can do on a native app that you cannot do on a web app. Just keep that in mind also when you're testing, there's just things that will behave differently. Like you're not going to have a lot of crazy key shortcut commands like you're not going to press command F to get to the search box if your app has a prominent search box because that is going to clobber a bunch of other assistive tech key commands. WebAIM is really a good help with giving you the tutorial of how to test a web app and many different screen readers, so they have it for JAWS, they have it for NVDA, they have it for VoiceOver iOS, they have it for TalkBack, which is the Android screen reader. There's a lot of really good resources there for you to start using as screen reader and test with your app. I highly recommend using it with a screen off. You gain a lot of context by looking ahead of your cursor and -- CHARLES: Yeah. It's true. ROBERT: -- The example that I usually give to people is like, "I worked on Visa Checkout. I went through that checkout flow pretty much every day for a year and eight months in, even then turning off the screen, I was still lost." Even though I knew what each screen was and what each component was there, I would find myself confused of like, what just happened and it's because sometimes, you'll get into something like you have a dropdown that sets the focus inside the dropdown and then the dropdown disappears from the DOM and your focuses in nowhere. You're like, "What is going? what am I doing. I don't even know where I'm at," and I turn the screen back on and I'm like, "Oh, now, I know what's going on." CHARLES: Right. It really is a lot like interacting with your application as though you were interacting with Siri or Alexa but with a keyboard instead, instead of voice commands. That's an excellent point. We would understand if where would Amazon be if Alexa couldn't successfully navigate those situations. The counterpoint or even the flip side of that is if you model your web application in such a way that it can handle that type of serial interaction, instead of the highly parallel environment to being able to perceive huge amounts of information concurrently, like you can on a screen, if that were effectively serialized, that means you could represent your application through nothing but voice commands. ROBERT: Yeah. Did you provide enough context for me to build this mental map is really what I'm going on to? CHARLES: Yup. ROBERT: Which I always thought was really interesting because I always wanted to know how my mom visualizes what a website looks like because it's wildly different than any of us. She's never been able to see what a website looks like. Does it look like a bunch of nodes and graphs and webs connecting to each other and how things pieced together? It's just a different way. But it's not only about screen readers, right? You can use a keyboard to navigate and that's definitely what we did with e-holdings is like can we tab through this [audio glitch] list, hit enter and go into the detail record, edit it, close it and go back and edit another one. Is that something that we can do with just a keyboard, not even a screen reader? WIL: And with the keyboard, we're not talking about shortcuts to hit edit. We're talking about like tabbing over and hitting enter like people with accessibility issues would have to do. ROBERT: Right and that's kind of a good segue into creating use cases. If you want to know if your app actually works, if your screen reader users or if your keyword users or if your dictation users are going to be able to navigate this app, create use cases. Things like actual user flows like how would somebody actually going to use this app? What task are they going want to complete? In that case, e-holding was like an electronic holdings management system for libraries. They probably want to get in there, add a couple of books or whatever you might have to their library and get out. A use case could be like, "Can I search for this thing? I'm going to search for something specific. I'm going to go through the list, find the thing that I want. I'm going to add it, close the pane, go back and then remove one thing and can they complete that flow successfully without running into any issues or any blockers or any showstoppers." I can tell you before we did the routing management stuff, you would hit search and that was it. There was nothing else that you could do. CHARLES: Yeah. They wouldn't even announce that anything can happen. ROBERT: Exactly. WIL: Yeah, and with these librarians, it's not necessarily a matter of they can't see the screen. It's just that they don't use the mouse because they're power users. ROBERT: Right. They don't have any disabilities but that was essential to the workflow. CHARLES: Yeah, exactly. Do that change. You just lowered the activation energy of that workflow by... What? three orders of magnitude? WIL: Yeah, at least. ROBERT: Let me click here right now. I can tab, now I can tab. Right now, let me click here and now I can tab, I can tab, I can tab. It's not as nice as just being able to completely do it through a keyboard. Through us making it super keyboard accessible, that also became super screen reader accessible and the people who use dictation were able to work through and get through the app and use it now, which was really cool. It really helps when you go for those things and create use cases to really figure out how a user is actually going to work through this app. That's the best way. Just get right through it. With Visa Checkout, we're like, "Can somebody buy something?" or if they don't have an account, can they sign up and buy? those were some of the use cases that we had because it turns out, those are actually pretty important to the business. That all have been said, you also can test your components for accessibility individually because even at a smaller level, some of your components might have to manage focus. The best one I can think of is models. When you open a model, you should trap the focus inside of that model and you should be able to hit escape to close the model and when you close the model, it should go back to what triggered the model to be opened. These are all things that are individual that can be tested also. But just because your components are individually accessible, it does not mean your application as a whole was accessible either. I don't want to paint the picture like, you don't have to care about your components accessibility individually because you do. It really does help but I think a lot of people miss the whole of the application, rather than individual with components. CHARLES: Right. The components are the individual stitches but you have to follow the thread throughout the entire garment. ROBERT: Right, exactly. CHARLES: Man, what I really want to do is I want to find out how your mom visualizes website navigation and use that as a visualization technique. ROBERT: That might be a fun webcast or something. CHARLES: Yeah and then see like could we actually use it as a tool because I have to imagine, it's probably pretty simple. It's much simpler than what we think of when we think of a website because it has to be really condensed down to its essence. ROBERT: Right. Yeah and [inaudible] users, they're not dumb. They have different ways. They know the gotchas. They know things that happen. They know there are different ways of getting trolled like my mom knows about focus jumping and she gets irritated what happens but she knows generally of what to do and it depends on her patience level. Like if it's focused jumping, she's like, "I don't need to use this thing. See you," and it's just not worth the frustration but there's different ways to navigate around like if you're a real power user, you might be able to recognize like the routing is inaccessible and you can navigate by headings or by regions or different landmarks. There's many different ways for users to navigate but to use those different navigation methods, you need to have a real strong coherent document structure. Your H1s have to actually be H1s and you have to have some things that they can navigate around to kind of work around those things. It's interesting and basically, do everything you can to help those situations. If you can provide semantic markup and give it a proper structure. If you can do the focus management, it's going to help everybody. It really will. I did see when we went through the use cases and defining those things, we actually learned a little bit more about our product because we had to put ourselves in a different seat and think about it because you get real close to it. You go through that same flow nine million times and you pick up real power user things that you can do like, "I'll work on that. I got this. I'll click this button. All right, we're good." Somebody that's going through it for the first time and you put yourself in that seat, it kind of opens your eyes a little bit and makes the experience better for everyone. I think that's kind of the underlying tone there. That's the message. WIL: Yeah, accessibility makes things better for everybody. ROBERT: That was a lot of content thrown at you there. We covered what is accessibility and why you'd want to do that and then kind of like more the basics things and how automated tools are really helpful and they can help you pick out things like using improper roles and nested things but they're not going to be able to tell you if your application is truly accessible or not and never will, unless we get something like a headless screen reader, where you can write automated tests for in that fashion but you're not to get something that will just run over you app and go, "Yup, you're good." CHARLES: Even so, it's a matter of user experience and that's not something you can get a thumbs up or a thumbs down to. When you as a user, it comes with application, you know when you see it but I would say until we have androids that accurately simulate human beings, I don't think we're going to actually have automated testing. ROBERT: Yeah. There's a joke for accessibility consultants. It's like if you put four accessibility consultants in a room and tell them to give you an alt attribute for an image, you'll get four different alts. We talked about the automated checkers, right? They'll not going to get everything for you and we talked about single page apps specifically in the routing and how we handled the routing in a React app and then how you can probably do it in an Angular app and how you can do it in an Ember app and Vue and different methods of how you can kind of attack that. We're definitely giving the link to the document that I wrote and the PR so you kind of see the real 'how' of how we did it because that would probably be helpful. I think there was a lot of good information there, so I would like to thank both Charles and Wil for being awesome co-host on this and -- WIL: Thanks for being an awesome host. CHARLES: Yeah, thanks for being an awesome host. I should say, you're welcome. ROBERT: I tried, I tried. We are The Frontside. We do accessibility consulting and training. If that's anything that your team needs help with, we're more than happy to jump on a call with you to kind of figure out what your needs are and what you need to do. If you need a WCAG support statement, if you need to audit, if you just need to figure out what's the next steps for you to even do, we're more than happy to help you sort through that. To reach out for that, you can contact us at Contact@Frontside.io or Sales@Frontside.io or Info@Frontside.io. You can contact us at any different ways and we'll be more than happy to help you. As always, thank you Mandy for producing the podcast. You're awesome and the next episode, what we're going to have is also still accessibility related and I'm really excited about this. It's about writing a proposal to the Unicode Committee and getting it accepted so basically, writing a proposal to get an emoji added and that's with Amberley and Amanda. They wrote three or four Unicode specs and actually got them accepted for, I believe it was sign language and deafness. That's really cool and I'm super excited for that because they'd be the first people that I've ever talked to that have actually created an emoji and gotten it accepted. WIL: Yeah, it's pretty cool. CHARLES: Yeah, that must feel great. ROBERT: Yeah, it's going to be awesome. That's our next episode. If you have any ideas or comments or anything, you can tweet us at @TheFrontside on Twitter or you can contact us through any of the emails that I talked about earlier. We're always open to hearing feedback. Thanks for listening. Take it easy, everyone.

Panel: Charles (Chuck) Max Wood Lucas Reis Special Guests: Charles Lowell & Taras Mankovski In this episode, the panel talks with two special guests Charles and Taras. Charles Lowell is a principle engineer at Frontside, and he loves to code. Taras works with Charles and joined Frontside, because of Charles’ love for coding. There are great personalities at Frontside, which are quite diverse. Check out this episode to hear about microstates, microstates with react, OM, Redux, and much more! Show Topics: 2:32 – Chuck: Why do we need it (microstates) and why do we need another state library? 2:42 – Charles answers Chuck’s question. Charles goes to explain that if you need to increment the number, you don’t need to do it with microstates. 3:41 – Another suggestion is given on this topic. 5:13 – The application isn’t hard in-of-itself. 6:45 – Chuck makes comments, and asks: It seems to be more like object-oriented approach? 7:44 – Objects compose much more easily. When you are dealing with pure functional code you are de-structuring and restructuring. Check-out LENSES. 9:53 – Taras makes comments. What were your inspirations for microstate? 10:27 – Charles: The personal journey it started for me started back in 2015. When I was working primarily in Ember.” Charles makes a reference to OM, check it out! 15:40 – Charles: “We had a goal in mind, and we kept that goal on mind and kept ‘dipping into the candy jar.’ We had to learn about the functional mumbo-jumbo. The goal was never to use those things. Whatever tools we needed from the functional world, we borrowed from freely.” 16:50 – Chuck asks a question. 17:00 – Taras answers chuck’s question. 19:58 – Charles (guest) keeps the conversation going and goes into detail about how to handle different scenarios with different tools. 21:00 – Question: How do you think microstate enters into this situation? 21:45 – The design of microstate is that it gives you a solution that is flexible. Other options aren’t as comprehensive like where you can use it; for example Redux. 23:49 – Another way to say it is...check-out this timestamp to hear other ideas about this topic. 24:53 – Digital Ocean’s Advertisement 25:28 – Conversation is back into swing. Question: There is a very interesting design with people who are not developers. What are the benefits or do they play together? 26:41 – As a frontend shop, there is a very clean mapping between state machine and type. The type corresponds to the state transitions, among others. For every state you have a class, and you have a method for every transition. It’s a great design tool. 29:07 – We don’t talk about states very often, right now, but in the near future we will. The valuable goals for us are to give people tools that will work correctly for them. To help people be more productive that is a great goal. One thing from people, I’ve learned, is to ask yourself ‘what needs to change?’ 33:03 – Now you are touching on the subject of teaching. What about mentoring with microstates? 33:26 – Success (to one of the panelists) is defined of how confident a person is with X program or tool. If they have ease, then they are on the right path. With mentoring in microstates the design speaks its purpose, the transitions are clear, so the panelist feels that he doesn’t really have to go into a lot of detail explaining the features. 36:25 – In the React community...  39:12 – Curious: Would we really be able to distribute state like how we distribute components? What is out-of-reach now, is that we have the state machine for the autocomplete component. 40:27 – Chuck: Is there a way to test microstates? 41:28 – Shameless plug...check it out! 42:31 – Anything else? Microstates and Microstates with React. 42:48 – If anyone is interested in this, then we are interested in talking with these people and/or companies. 43:29 – Let’s go to Picks! 43:31 – Advertisement for Charles Max Wood’s course! Links: Kendo UI OM Frontside Redux Microstates Microstates with React Taras Mankovski’s Twitter Taras Mankovski’s GitHub Taras Mankovski’s LinkedIn Taras Mankovski’s Frontside Bio Charles Lowell’s Twitter Charles Lowell’s GitHub Charles Lowell’s Frontside Bio Schedule Once Ruby on Rails Angular Get A Coder Job Sponsors: Kendo UI Digital Ocean Get A Coder Job Picks: Charles (Chuck) Framework Summit – Chuck will be talking at this conference in UT. Ebook – Finding a Job. Prelaunch in August. Final version launches on Labor Day. Lucas Take care of your health! Martial Arts and Jujitsu Nutrition Charles (guest) Fantasy Land JS - Tom Harding Funcadelic.JS Taras (guest) BigTest

Panel: Charles (Chuck) Max Wood Lucas Reis Special Guests: Charles Lowell & Taras Mankovski In this episode, the panel talks with two special guests Charles and Taras. Charles Lowell is a principle engineer at Frontside, and he loves to code. Taras works with Charles and joined Frontside, because of Charles’ love for coding. There are great personalities at Frontside, which are quite diverse. Check out this episode to hear about microstates, microstates with react, OM, Redux, and much more! Show Topics: 2:32 – Chuck: Why do we need it (microstates) and why do we need another state library? 2:42 – Charles answers Chuck’s question. Charles goes to explain that if you need to increment the number, you don’t need to do it with microstates. 3:41 – Another suggestion is given on this topic. 5:13 – The application isn’t hard in-of-itself. 6:45 – Chuck makes comments, and asks: It seems to be more like object-oriented approach? 7:44 – Objects compose much more easily. When you are dealing with pure functional code you are de-structuring and restructuring. Check-out LENSES. 9:53 – Taras makes comments. What were your inspirations for microstate? 10:27 – Charles: The personal journey it started for me started back in 2015. When I was working primarily in Ember.” Charles makes a reference to OM, check it out! 15:40 – Charles: “We had a goal in mind, and we kept that goal on mind and kept ‘dipping into the candy jar.’ We had to learn about the functional mumbo-jumbo. The goal was never to use those things. Whatever tools we needed from the functional world, we borrowed from freely.” 16:50 – Chuck asks a question. 17:00 – Taras answers chuck’s question. 19:58 – Charles (guest) keeps the conversation going and goes into detail about how to handle different scenarios with different tools. 21:00 – Question: How do you think microstate enters into this situation? 21:45 – The design of microstate is that it gives you a solution that is flexible. Other options aren’t as comprehensive like where you can use it; for example Redux. 23:49 – Another way to say it is...check-out this timestamp to hear other ideas about this topic. 24:53 – Digital Ocean’s Advertisement 25:28 – Conversation is back into swing. Question: There is a very interesting design with people who are not developers. What are the benefits or do they play together? 26:41 – As a frontend shop, there is a very clean mapping between state machine and type. The type corresponds to the state transitions, among others. For every state you have a class, and you have a method for every transition. It’s a great design tool. 29:07 – We don’t talk about states very often, right now, but in the near future we will. The valuable goals for us are to give people tools that will work correctly for them. To help people be more productive that is a great goal. One thing from people, I’ve learned, is to ask yourself ‘what needs to change?’ 33:03 – Now you are touching on the subject of teaching. What about mentoring with microstates? 33:26 – Success (to one of the panelists) is defined of how confident a person is with X program or tool. If they have ease, then they are on the right path. With mentoring in microstates the design speaks its purpose, the transitions are clear, so the panelist feels that he doesn’t really have to go into a lot of detail explaining the features. 36:25 – In the React community...  39:12 – Curious: Would we really be able to distribute state like how we distribute components? What is out-of-reach now, is that we have the state machine for the autocomplete component. 40:27 – Chuck: Is there a way to test microstates? 41:28 – Shameless plug...check it out! 42:31 – Anything else? Microstates and Microstates with React. 42:48 – If anyone is interested in this, then we are interested in talking with these people and/or companies. 43:29 – Let’s go to Picks! 43:31 – Advertisement for Charles Max Wood’s course! Links: Kendo UI OM Frontside Redux Microstates Microstates with React Taras Mankovski’s Twitter Taras Mankovski’s GitHub Taras Mankovski’s LinkedIn Taras Mankovski’s Frontside Bio Charles Lowell’s Twitter Charles Lowell’s GitHub Charles Lowell’s Frontside Bio Schedule Once Ruby on Rails Angular Get A Coder Job Sponsors: Kendo UI Digital Ocean Get A Coder Job Picks: Charles (Chuck) Framework Summit – Chuck will be talking at this conference in UT. Ebook – Finding a Job. Prelaunch in August. Final version launches on Labor Day. Lucas Take care of your health! Martial Arts and Jujitsu Nutrition Charles (guest) Fantasy Land JS - Tom Harding Funcadelic.JS Taras (guest) BigTest

Guest: Saron Yitbarek: @saronyitbarek | bloggytoons | CodeNewbie | @CodeNewbies In this episode, Charles and Sam talk to Saron Yitbarek about her idea of mentorship, ideas for distributed learning for businesses to promote individual and company growth, and why it's important to take "digital sabbaths" on the regular. This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. Transcript: SAM: Hey, everyone. Welcome to Episode 110 of The Frontside Podcast. My name is Sam Keathley. I'm a developer here at the Frontside and I will be your episode host. Today, we're here with Saron Yitbarek, discussing mentoring. She is the founder of CodeNewbie and the host of the CodeNewbie Podcast. Also with me as a co-host is Charles Lowell, who is also a developer at Frontside. Welcome Saron and welcome Charles. How are you guys doing? SARON: Thanks for having me. I'm doing pretty well. CHARLES: Hello. SAM: Today is going to be an interesting take on the mentoring talk. I mostly want to know first off, Saron, how do you feel about mentoring? What are your opinions on the mentor-mentee relationship or the value there? SARON: Yeah. I have lots of opinions on this topic. I think that the traditional structure of mentorship was usually looks like someone with less experience going to someone who has a lot more experience and say, "Will you be my mentor?" kind of like the children's book, 'Are You My Mother?' like 'Are you my mentor?' and then that person, that mother figure, that mentor looks after them and checks in on them and they have regular coffees and lunches and kind of steers them in the right, usually career-related direction. I don't think that's very realistic, to be honest. I think about why that might be. There's many different reasons. I think the fact that we're so, so, so networked and there's just so many different ways to get in contact with people and build relationships is a big reason but I think that that traditional mentorship model, that kind of one directional way of doing things is just not really needed and kind of overrated. I think that mentorship nowadays looks more like a mutually beneficial relationship, where I might reach out to someone who has more experience than me, for example in drawing, in art, not something I want to get to do but I know a crap-ton about podcasting, so I help you, you're my mentor in this specific area, in this specific topic but then, I get to be a mentor in this other thing that I'm really good at. I think it's those types of very focused topic-oriented, ideally two-way relationships that are more accurate and frankly, a more effective way of doing mentorship. SAM: Yeah, I actually agree with that. I am pretty new myself to development, only really been in this career for about a year now and I always kind of consider that mentoring relationship as a regression back to school days, where you have these -- SARON: Yeah, yeah. SAM: -- considered superior over you in some way, well, that's really not true in this community of developers and no matter where you are in development, it's all about working together and pairing. Working here has really showed me that value in paring, rather than like I have to look up to someone and regress back to feeling like a teenager in high school, like this person so good at this thing and they're the only one who can teach me. I definitely share that view of mentoring. I went to a boot camp one day when they were telling you like, "Oh, you got to find a mentor. You got to find a mentor," and I was like, "Well, but why? Why do [inaudible] together?" SARON: It's also a huge responsibility, a huge burden on the mentor too. Having to be, in a lot of ways, responsible for someone's career and trajectory and direction, that's a big responsibility. We don't have time for that, you know? On both ends, whether it's feeling like you're back in school or feeling like you have this huge responsibility, I think the traditional model isn't really the best model for either party. I think this idea of let's all learn together, let's be really focused on topics and problems that are very particular to what I'm doing, what I'm learning, what I'm trying to do. I think that works out. It feels healthier for both people. SAM: Absolutely. CHARLES: I wonder also too, it seems like it might be a little bit of a throwback to the days when people would spend 30 years in a single company or 30, 35 years in a single career where you have these people who are really these reservoirs of this intense tribal knowledge. It seems like people move around a lot more in their careers, not only in the company that they work for but also in the things that they're literally doing. I might be podcasting one day and producing a bunch of content and then, I might move into music or writing or other things like that. The careers seem to be broken apart a little bit more is one of the reasons why older models of advancement in those careers might not be as good a fit as they once were. SARON: Yes, absolutely. I'm obviously very biased because I'm in tech but when you think about the different roles that people have in tech, I feel like we're always wearing so many different hats. We have to, obviously code and be technical in that sense but we have to be really good communicators, a lot of us are speakers, we host podcasts, we organize conferences, we go to meetups, we're bloggers, we do so many other things, that this idea of, "I'm going to have a mentor who can help me in my role of being a developer," is just too big. You kind of meet people who are good at those individual pieces and those individual skills to get me to where I want to go. SAM: The whole idea of mentoring to me always reminded me of that whole Mr Miyagi relationship where you have this master of something and you're trying to learn from them. But in software development, I've learned that really no one is a master of anything because it's just changing so much and everything is so different. SARON: Yup, absolutely. CHARLES: The question is obviously, it seems like the idea that you're going to find one person who's going to represent the ideal confluence of every single skill set that you could hope to want, to be at some point in your career. That's looking more and more ludicrous. Is there a model where you can try and distribute those things, where you single out a large range of individuals? I guess the kind of what you were hinting at the beginning is that it is a lot more broken up, a lot more distributed but how do you pursue that, even if it is in a distributed manner? SARON: Yeah, that's a great question. One of the moments where I realize that the distributed model is really the only way that makes sense is, I think it was three years ago maybe. We thought about doing some kind of mentorship program in CodeNewbies, some type of way for people to link up and find people who can be there and guide in a way and we had people fill out this survey that basically said what do you want out of a mentor, what do you look for, what do you hope for to achieve but also asked how do you see yourself. Do you see yourself as a mentor, a mentee or both? It was really surprising that most people checked off both, which I thought was so interesting. It was so interesting to me that the same people who said, "I need help," with the same people who also said, "I also have help to give," and to me, that was such an amazing moment because I said, "Wow, it really isn't about this idea of I am the guide and I'm going to guide you." It's really about, "I have information expertise in one area and not in others." What I realize is there really isn't a mentor model. I think it's more of a culture of being helpful, which probably sounds really cheesy but it's true. I really think it's about saying, "I know how to do a thing. I'm going to go on Stack Overflow and answer questions. I'm going to go on Twitter and answer questions. I'm going to write blog post and share with the world." I think the real model, the real distributed mentoring model is us as individuals saying, "I just learned how to do a thing," or, "I just figured out how to do a thing well. Let me capture that. Let me capture it in a response, in an answer, in a forum, in a post and let me share that," and the more we do that as individuals, the more we have this huge amazing aggregate of knowledge that can serve as mentorship for all of us. SAM: I like the touch on that. That's kind of the idea behind Codeland Conference, where everybody thought they might be new to development and everyone is just sharing their knowledge. You might feel you really knew about it but you know a lot more than you think you do and you could still help. SARON: Yup, exactly and that's the whole idea, even the Twitter chats. When we first started, I don't have all the answers, I don't claim to, I don't really want that responsibility but I know there are a lot of people who do and I know a lot of people who have resources and opinions and who can help out. One thing that people do is they'll DM us and say, "I'm having a hard time with this." Sometimes, it's a very technical problems. Sometimes, it's a general 'I'm having a hard time getting a job,' which I do like high level question and I never answer them. I always say, "Tweet us and we'll retweet it and we'll get the whole community involved and we can have a rich conversation around it," and that's really been our motto, our philosophy. I see what they do with mentorship. If you have a mentor, you're not obligated, I guess to listen to them but the idea is kind of that you should. There is one person and you should listen to them because they know more than you. I think that's just not really fair and instead, I like to think that there are lots of different ways to do things and it's up to you to decide what's best for you and in order for you to decide, you have to have a lot of options. With Codeland, with the Twitter chats, no matter what skill level you're at, no matter how confident you feel in your coding abilities, you do have something to offer. Let's pull that out. Let's put it on the table and let's see who you can help today. SAM: That's a perfect way of introducing someone to the idea of helping or getting help from a community, rather than an individual because you never want to take one person's word as gospel on something you just know nothing about. SARON: Yeah, exactly. SAM: You can sound confident talking about something and it can be the total wrong answer but if you're seen as someone superior or someone who knows what they're doing -- SARON: And you can sell it. SAM: Yeah. You can be the best snake oil salesman in the world. SARON: Absolutely. For the CodeNewbie podcast, we do short questions at the end of each episode and one of the questions -- my favorite question -- is what's the worst advice you've ever received. I love that question because I started by saying, I think that people love giving advice. I think people love asking for advice and the assumption again is if I give you advice, then I probably know more and you should probably listen to me but that's not always true. I want people to be comfortable making their own decisions and deciding for themselves. "This may sound like it could work and this may sound like a good idea, generally speaking, but for me, it's probably not a good fit. It's probably not what's best for me," and to be comfortable rejecting advice and that was kind of the reason why I started that question, it's my favorite question and it's so interesting how much advice is good. It's kind of generic but it's a good advice. It's an advice like, "Don't quit. Keep going," which maybe a good idea and maybe you do need to quit, so being open to rejecting advice, I think is really important and that one of my favorite questions. SAM: In your experience, what is the best way to reject advice? Because when you're a new developer, when you're new at anything and you're seeking advice from a community or from a person, you don't want to come off as rude or maybe you're feeling... I don't know, not very confident but you think the advice that you were given is just bad advice. What would your advice be? I guess, what would your advice be on this advice? For your perspective, what would be a good advice to reject advice that you think is just wrong? SARON: Number one is when I ask people for advice, I try to think about and try to know upfront, do they have the same values that I do? Do they have the same worldview? Do they have the same goals? Because that's the thing too. Oh, my God, I got so much unsolicited advice. It's amazing. I've actually stopped just saying to people my ideas or what's on my mind because I know as soon as I say, "I'm thinking about this," I'll get a whole slew of unsolicited advice and I'm like, "I didn't ask for that." What I've learned is first to kind of figure out are we even on the same page because if my goal is to be a developer, if your goal is to be -- the only thing I could think of is a juggler, I don't know why -- a juggler and I ask you for a career advice, I'm going to go ahead and safely assume that what you have to say is probably not as applicable to me and so, number one is kind of identifying that. But number two is if they say something that just I know is not going to work or I've already tried, I'll just nod and say, "Thank you very much," and kind of go about my day. I don't think you have to declare whether or not you going to take it. I think just acknowledging and I think the people that give advice, I think they're trying to be helpful, they have good intentions, usually. Usually it's, "I'm trying to save you from making mistakes that I made and I'm trying to help you get to learn a little faster." I always appreciate it but knowing that I can say, "Thank you. Thank you for your time. Thank you for your perspective," but know that I don't have to go off [inaudible]. SAM: That's actually very similar to my tactic. Just not like, "Thank you. Thank you so much. Don't talk to me ever again." No. SARON: And disappear, yeah. SAM: I'm like the wind. With this pressure that either new developers or seasoned veterans are feeling about the mentor relationship, because I feel like a lot of senior developers that I've spoken with or people who've been in the business for a long time, feel like they should be mentoring or they need to take on that responsibility but there's always that hint of dread in their voice when they say about like, "Oh, I should be doing this." I always feel like it's okay to not do that. I never really understood why it was so high value to have this one-on-one relationship with an individual when you're not in school because it just feels so like... Not childish but childish. SARON: Yeah, that's one of things, frankly that I love about the tech community and also do not understand about the tech community. It's such a giving knowledge sharing community, whether you do it in a tactful way or not in a tactful way but the idea of giving back and paying it forward is just so deep. It's so, so deep that even when I've been coding for only a couple of months, I still felt this, I don't want to call pressure because pressure kind of sounds a little negative but I definitely felt this expectation that I was supposed to be blogging, supposed to be sharing and shouting and helping and doing these pay it forward type things. I don't really know where that comes from. Maybe that comes from the culture of open source, maybe that's where it kind of penetrate. I'm not sure but there's this huge need, desire, idea that we're supposed to be giving back and for that, I am very, very, very grateful. But I think that acknowledging that if you're someone who wants to be a mentor that you can do it simply by being available, literally being available, the going to be hashtag is super, super active even when we're not doing our Twitter chats and people use it to ask for help, they used it to ask questions. If you're feeling particularly giving or extra helpful that day, go on Twitter and check the hashtag and see what questions people are asking. Things like that or just super helpful and don't require a huge amount of time and effort. There's a lot of small ways to help out. That may not seem like a big deal to you but for the person who's asking for help, who has been banging their head against the wall, it's hugely valuable. SAM: Yeah, absolutely. Up until I went to my first conference this year, I didn't realize how supportive and important Twitter is. You know I always kind of considered it to be like another social media platform that I don't understand because I'm 84 years old and I just don't get it. It's been so uniquely helpful and in ways like Stack Overflow or even issues in GitHub, it just aren't. You can get so many more perspectives, so many different perspectives from people. SARON: Yeah, absolutely. Twitter has been amazing exactly for that. It's just an efficient way to crowdsource opinions and crowdsource perspectives and when you get your question answered and someone else answers it, it doesn't only benefit you the way it would if you email someone but it benefits anyone else who comes across that page, so yeah, it's hugely valuable. CHARLES: I remember the first moment I had kind of like that, a light bulb went off in my head where I was working on some really weird project that was using some strange wiki for its content storage and I was getting frustrated and I just tweeted about it and then the CTO of that company just immediately answered my question and I was like, "What?" You know, it was years ago and definitely, I was like, sound of explosion, that is where my mind exploded. I was not seeking help. I was just literally being kind of a jerk and venting frustration and lo and behold, the answer for my problem descended from the Twitter clouds. It was incredible. SARON: The Twitter clouds are the best clouds, usually CHARLES: Twitter is very... What's the word? It's very split down in the middle. SARON: Noodie? Yeah, there you go. SAM: There's this live feedback, so there's no buffer of emotion there, you know? SARON: Yeah. SAM: It's like, "Oh, this thing that you said, it made me mad. I'm going to tell you about it right now." Charles, I know that you had mentioned before that you think mentoring would be a good idea for Frontside and then, after all this discussion, have your views stayed the same? What are you feeling about that? CHARLES: As kind of the person who's like the grizzled veteran in the software world, it's definitely something that I've kind of whipped myself over the back. It's like feeling like it's something that we should do but I think it comes from the idea that people come here and we want to make sure that they're getting access to the learning that they need and the ways, in which they can level themselves up that they need, that's the kind of the prime motivator there. I always perceived mentorship as some vehicle through which to achieve that. It's something that I've heard. Obviously, we don't have a mentorship program at our company. It's something I felt that we should always be investigating. I've always felt maybe a little bit bad that we didn't have it but it's also something that I really struggled with in my career because I can't really say that I've ever had a mentor, so I don't really know what that relationship would look like but I do have a lot of people that I learned a lot of critical things from. I can look at it as kind of these seminal moments in my career, where like light bulbs went off and a lot of the time, they're associated with an individual and that individual and the thing that they taught me or multiple things that they taught me, are still with me. I have those experiences, which have been phenomenal and critical to my development as a software developer, so I guess it's just part and parcel of that impulse that you're describing to pay it forward, to realize that when you walked into the building, the lights were on and the walls were standing and the air was at a comfortable climate temperature. As you live there, you realize that there are people involved in actually, doing that maintenance and providing the building for you and the space for you to become aware of your world and then, when new people walk into the building, you want to provide them the same experience that you had. I guess that's my take on. That's the kind of thing that I would want to provide, so the question is like what does mentoring or mentoring 3.0, as we're maybe talking about in this conversation, how does that fit into that? How would you implement something like this, some sort of distributed learning in a company? I don't know. Maybe, it's not worthwhile. Maybe it is. SAM: I think it focuses more on that pair programming because when you're thinking back and you have all these people, like you have names of people that taught you something, it's multiple names. It's not just this one guy taught me all of these things. Actually, within a company, that mentoring just comes from pairing with your coworkers, seeing what different hats everybody wears and then, trying them on every now and again but not necessarily taking that individual's word as gospel, you know? CHARLES: Right and hopefully, that's not something that we've advocated for. Maybe, it's how do you introduce structure around that, to make sure that the proper ferment is happening, so that you have novel pairings and make sure the ideas that are flowing are flowing around the entire company and not just through certain set channels. SARON: Yeah. The other part of that is if you create structure around what it looks like to share outside. You know, pair programming is interesting because it's kind of one-on-one and it's hopefully, I have something important or bright to say in our pairing session. Maybe, I don't. Maybe I'm having a dull day. Maybe, I'm having a bad day but this really give you the same opportunity to take a moment and say, "What do I know? What do I want to share? What do I want to put together?" It does really give you a chance to prepare and gather yourself. It's kind of in the moment. I think having another opportunity where you can gather yourself is important and so, that might look like brown bag lunches, where everyone takes a turn and has to do a little lightning talk. That's usually the opportunity to say, "I have five minutes. I'm going to share something." Everyone has a turn, which means that the company's literally saying everyone has something to say. It's only five minutes, only a few minutes, so hopefully it won't be too terrifying if you're not big on public speaking and it's your co-worker so hopefully, it won't be terrifying because it's people you know and not total strangers. You know, a format like that, where there's structure but the company is saying, "We're going to give you time to think about what you're good at and what you know and to share that is good." I think another way to do that is by setting time aside for blogging. If your company can say, "Thirty minutes out of the week, we're going to take some time to write down five things I learned, to write down one cool thing I learned, post it publicly, post it internally about this expectation that everyone should be writing, everyone should be sharing, which also says everyone has something to share." I think those are two ideas and two ways that we can create a culture of sharing and a culture of distributed mentorship, where everyone has an opportunity to find the thing that they're excited about and specific ways to share it. SAM: Yeah, that's excellent. CHARLES: I hope you're grinning as much as I am, Sam because at least in the first half, you basically described our Lunch and Learn process. It's a little bit more than five minutes but I think another thing that clicked for me there is making sure that having a variation of expectations of quality or something like that because I feel like where we do really well is in the Lunch and Learn thing. We have a process very similar to what you describe but where we're not so good is maybe with blogging. I wonder if part of that is we just hold ourselves to such a high standard of what it is. The idea that we could throw together a blog post in 30 minutes, I love that idea but it means you really have to be willing to just say, "You know what? We're going to get it out there and we're going to make it bite-sized and the expectation is that we're not going to be writing some gigantic essay that's going to shake the industry to its core every Friday at three o'clock." There are things and if we can, I shouldn't say a reduction in quality but maybe a reduction in scope so that you can say like, "We're going to carve out 30 minutes or an hour and we're going to pick a topic that scope-appropriately for that." SARON: Absolutely and I think that knowing that you only have 30 minutes or maybe it's an hour -- an hour is probably a little bit more realistic -- but knowing that you only have an hour also forces scope. You can't write a book about JavaScript in an hour. You just can't, so the fact that by the end of that hour, you have to have something to turn in is a great way to force people to focus and do something really small and really bite-sized. The other thing is I don't think that after the hour, you need to publish it publicly. It could be, you need to turn it into someone else, to edit and look over for you and give you feedback. It's not quite ready yet but it's a solid first draft and the ideas that after that first person edits it, then it's on its way to being published. I think there's different ways to manage a scope without also making this scary thing where I have to say something for the Earth to shake. There's different things that we can do around that. CHARLES: I'm wondering what other forms of sharing that we can fit into our workday. I guess, the other baseline is just making sure that you're always... What is it? ABT -- always be tweeting. It's so easy to be write-only, not even engaging conversations but just throwing ideas out into the void. SAM: I think holding a discussion with your coworkers like if you're in an environment where you can work face to face or are constantly online, if you're more of a remote worker, I think just having a conversation with anybody, rather than just putting your idea out there, putting it out there with someone who can actually provide real time feedback in a more friendly way, then I think some people on Twitter could be. Because they're your coworkers and they're not going to call you rude names, you know? CHARLES: Right and you're also going to know, hopefully the whole trust and intent and 'are we on the same page?' that question is answered even before the text is written. SAM: Right. SARON: Yeah, absolutely and with those conversations, this actually mean [inaudible] that we do. We do like a show and tell every week and the idea is that we're both always learning random things, usually related things but sometimes, totally random but still very interesting and it will take 30 minutes to just share what we've learned. Sometimes, they'll also turn it into a blog post. Sometimes, it's just a knowledge sharing opportunity. I'm not really sure but it's a good window of opportunity to say, "We are learning and we're sharing and we have something of value to bring." The great thing about something like a show and tell is that it doesn't necessarily have to come from my brain. It doesn't have to be like, "I have a great idea that I'm going to share." It could be, "I read about this cool idea." It can be, "I heard about this cool thing that we can try and we can apply," but I still kind of credit, you know? Like I get credit for being the value bringer but the burden isn't on me every week to come up with the idea. That's a nice balance, to kind of create space to share and to promote this knowledge share and if it comes from you, it's great but if not, you're still helping other people. CHARLES: I have a question that may or may not be related in here. This has just kind of occurred to me because this is definitely something that I experienced, where I get into a mode where I become overwhelmed by the ideas that people are sharing and so, what's the balance? Because usually, we're out there searching for ideas and we're searching for novel things so that we can include them in our work, in the things that we want to do and accomplish, whether be that in tech or elsewhere. What's the balance of being heads down and being like, "You know what? I'm going to be closed to new ideas right now." Because they can be distracting, right? The [inaudible], that's actually a phenomenon and so, how do you protect yourself from sharing? What's the balance? SARON: My solution was to move to San Diego. That was my solution to that. I actually moved from New Jersey and I worked in New York City... How long has it been? Was it only been a year? Oh, my goodness, and now, I'm in San Diego and it was so interesting because that ended up being a really nice side effect. I didn't move specifically for that reason but when you are commuting in New York City every single day, there's so much going on all the time. There's just so many ideas and events and meet ups and companies and people. It's just so much. I think that it was a great place to be in my early 20s when I really just wanted to soak up everyone else's ideas and I didn't really have opinions of my own at that point and it's a great way to just kind of absorb and be this awesome sponge in the big city but after a while, I kind of realized, maybe I have my own ideas and my own thoughts, so moving to San Diego, which is a much, much, much quieter place, has been a really great way to reflect and sit with my own thoughts and feelings and opinions and just kind of focus on that. I understand that everyone can move to San Diego, although I highly recommend it but I think in that way of carving out like... What do they call it? Do they call it a tech sabbath? A digital sabbath? Am I saying that right? CHARLES: Yeah. Maybe. It sounds about right. SARON: Yeah. I came across that term recently but this idea of one day out of the week, I'm not going to be on the interweb. I'll just not going to do it. Maybe, even the whole weekend, oh, my goodness and saying, "I'm just going to stay away from things. I'm just going to create a little space for me to think and reflect." I think when I don't have the whole day and what I need is just like a moment, I find that writing things down is a great way -- a really, really good way of doing it -- even if I'm taking notes or from doing a strategy session or if I'm trying to make a decision. Usually, I'll start typing and what I've started to do recently is to say, "I'm not going to type. I'm going to plot in a notebook and I'm just going to write things down," and because writing is slower than typing, it forces you to just think. It forces you to be alone with your thoughts, for better or worse and it forces you to really just to think about what you're doing or what you're saying and reflect. It's a really, really great meditative exercise that I found. You know, finding little ways to build in an escape from the noise, ideally on a regular basis, I think is a really healthy thing to do. SAM: Yeah, I definitely agree with that. My normal method of sort of closing everybody off is I just sketch out ideas because I'm an artist. I come from that sort of perspective as I can make a drawing out of feelings more so than I can write them out. If I'm feeling really overwhelmed or if I'm trying to make sense of some information that people have given me, I just sketch it out and I think that's something that doesn't really get talked about a lot in the whole community because sometimes it's always about eat, breathe, live, code, you know? We have different skills. You have other things that you're good at that's not just code. When I was at React Rally, a lot of people were in the music and then finding a way to separate yourself from the entirety of it is definitely one of the better ways to make sense of your situation. SARON: Yup, absolutely. CHARLES: Yeah. Sleep? It's a great way to take a break. SARON: Sleep is so good. SAM: On my top three things: sleep. CHARLES: Yeah. SARON: Yeah. As a kid, I never ever thought I would look forward to my bedtime ever. You know, it's my favorite part of the day. CHARLES: Yeah but like sleep, writing, drawing, I always forget. Like when I'm caught up in a problem, I always forget that engaging in some other activity -- I like to walk in the place next to my house. I like to play my ukulele. Engaging in those activities is almost on the critical path to solving the problem and I always forget it. I think we always forget it but sometimes, you can be so frustrated and you can just shut it off, be completely alone and there is some magical process that's probably going on in your brain. I don't know exactly what it is but you come back and the answer is just sitting there, waiting practically on your desk. SARON: Absolutely. One thing, we recently moved a couple of weeks ago to a new place and it's a little bit bigger and so, I had the opportunity to unpack boxes and buy furniture that we didn't really need before or just trying to make it more of a home. I've been using it as a really great opportunity to be productive and to feel productive but not be in front of a screen. Sometimes, I even like save tasks for myself like, "I'm going to wait till the evening to put together this shelf because I'm going to need a moment to just be away from my computer." I have a plan around it so it ended up being such that, I think about every day, there's one little home activity thing that I can do, whether it's cleaning or cooking or assembly or movie or something with the home, that allows me to, because I'm not really a hobby person. I don't really do hobbies because it just feels, like no judgment on people who do. I know people get really into their hobbies but it just feels like, "Why?" You know, like, "Why?" like, "For what?" But when I put a shelf together, it's like, "I'm going to use this for books." You know what I mean? Like you're very purposeful, so home activities has been my way of carving out that space away from the screen but also feeling like I'm doing something productive, I'm getting things done. SAM: Something that I would recommend to anybody who doesn't really have that, I'm going to step away from the computer and do this thing. Something that's kind of helping me was the Pomodoro Technique, which if you're not familiar with that, it's basically a time management thing where you set your timer for work like for 25 minutes, 30 minutes or whatever and then, when the timer goes off, you take an allotted break for five minutes, 10 minutes, just so you're not doing the thing that you've been doing for the last 30 minutes. SARON: Yes, absolutely. That's a great one. SAM: My advice is to implement that if you don't have a thing, that you use for your cleaner, I guess. SARON: Oh, you want to hear some really terrible but effective advice? SAM: Yes. SARON: And this is what I found out very accidentally is if you have a really, really crappy office chair that hurts -- CHARLES: Oh, man. I'm sitting in one right now. It's literally a rocking chair from the 1800s. SARON: That sounds awesome. CHARLES: Yeah, it does sounds awesome. SARON: But if you have a crappy office chair, that can be a really great way to get away from your screen because after about an hour, your thighs will hurt and your back will hurt and you will be forced to get up and walk around. It's so funny because I have a pretty crappy office chair and my back has been hurting for so long and I just thought, "That's life." That's like my [inaudible] for myself. I'm like, "That's just how life is, my backs hurt," and then it got to a point where I was like, "I need to go and talk to someone," and as soon after that, I spoke to a conference and I was basically up and down away from any type of chair for like four or five days straight and magically, my back pain went away and I was, "Oh, my God. It's that freaking chair," and ever since I realized it, I started noticing it. I would sit down and I would go, "I'm nearing the one- Oh, there are my thighs. There they go. They're in pain." There's another great cheap life hack: get a crappy chair, after about an hour, everything will hurt. You will be forced to go do some jumping jacks for about five, 10 minutes and there's your additional sabbath. There you go. SAM: In Charles' case, probably a haunted office chair. Yeah, that's an excellent advice. I never really noticed that. Now, I'm going to. CHARLES: You know what funny is, I actually didn't even notice it but I do. I never used to get up and go on walks before or spend as much of the day standing as I do and I'm actually completely and totally oblivious to it but I think, I realize like I can attribute a lot of that to the terribly, uncomfortable chair, in which I sit on every day. SARON: I have this awesome habit of sitting cross-legged, which apparently is a very, very bad idea, my physical therapist told me, so I make it worse for myself. If you don't feel like you have enough screen time, just sit cross-legged and that will accelerate the pain process. SAM: As you said that, I am currently sitting cross-legged in my desk chair. SARON: Yeah! SAM: That's just how I sit in chairs. My legs are too short to reach the floor. SARON: Me too. I just find it so comfortable. I don't know and I'm so happy to hear that you also do this because I'm like, "Am I just weird?" because I love sitting cross-legged. It's so comfortable. It makes me really happy. SAM: No, I verified 100%, that's how I sit in all chairs. SARON: Yeah, the same. CHARLES: I can do one leg. SARON: Only one? Man, you need to work on that. SAM: -- [inaudible]. That's going to be the end of our podcast. Again, thank you Saron for coming on and having this awesome conversation about mentoring. Definitely check out the website CodeNewbie.org. We are the Frontside. We build software that you can stick a future on. Hit us up if you're looking for help building your next big thing and also, hit us up on Twitter. Hit Charles, myself, Frontside and Saron on Twitter. SARON: Thank you so much for having me. This is awesome. This is fun. SAM: Also, I want to give another thanks to Mandy, our producer for producing this lovely episode and all of our episodes. If you have any questions for us, hit us up on Twitter. Any ideas for future topics, any thoughts you have on this great conversation, let us know and we'll talk to you all later.

In the last episode, we spoke a lot about the "why?" behind microstates. This time we wanted to cover how ideas in Microstates map to different patterns used to build JavaScript applications using frameworks like React, Ember, Vue and Angular. We discussed what you need to know about Microstates if you prefer component state architecture or global store architecture like Redux, as well as how setState in React can be refactored to use Microstates. We closed off with the comments about the trade offs that component heavy frameworks make by overemphasizing the view layer at expense of other aspects of the MVC pattern. This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. Upcoming Conference Talks: Manhattan.js - August 8th (Taras) ReactJS Austin August 6th (Charles) Transcript: CHARLES: Hello everybody and welcome to The Frontside Podcast, Episode #107. My name is Charles Lowell and we are going to be following up with our Episode 106 with the exciting conclusion of 'Microstates, the Podcast.' With me today to wrap this subject up, at least for the near term, obviously we're going to be talking about it a lot in the days and months to come, are David and Taras, also co-developers here at Frontside. Hello guys. TARAS: Hello, hello. DAVID: Hi everyone. CHARLES: Before we get into that, we'll just make a few quick announcements. First off is here at Frontside, we are going to be having some availability at the end of August. If you or anyone on your team is looking to level up in the area of testing, especially testing single page applications, acceptance testing single page applications or if you need React or general JavaScript consulting, please get in touch. We would love to work with you and love to do some of the great things together. Second, we finally released this and this is germane to the topic at hand. We released a major version of microstates this week that's based on a new and simpler architecture. That's really exciting. It doesn't really change the content of the conversation because the API hasn't changed that much. It just means that the library, which was already very small is even smaller. I think we shaved almost 40% of the size off and it's just much simpler and it's much more harmonious with the underlying JavaScript runtime. It's even more just simple JavaScript objects. Unless there's any other news that you want to cover, we'll jump right into it. TARAS: All right. Let's do it then. Let's do microstates. CHARLES: I Love to talk about microstates. This is obviously the second podcast that we're doing on microstates, just because we ended up, I think it was two weeks ago and we'd been speaking for almost an hour and really, we're just laying out the problems that microstates solve -- the problems of state management and why you often run up against complexity when you have a single state management tool that doesn't account for a bunch of different use cases. We got into microstates a little bit but we left it as kind of a cliffhanger. We talked about transactionality and laziness and immutability, ease of composition, simplicity of API, performance, memory footprint, things like this. Those were all the problems and then we're like, "Yeah, microstates. It's awesome." We're going to talk a little bit more about actual microstates proper and what is involved like the adjustments in mindset that you need to make or don't need to make when adopting a tool like microstates. TARAS: Actually, it's been really interesting for me because I just gave a talk at Toronto JS on microstates and it was pretty cool to see. There was a panel at the end. I think the people that are representing just using component state as a way of architect implication. It was managing state and application and it's representing redux. It's really interesting to see, first of all, like how curious people were. Most of questions were directed to the discussion around microstate, simply because it is a new tool but it was also just interesting to see how curious people who really loves redux, he was like, "I really love redux but I really like that fact that you guys have types," so I was like, "Oh, that's cool." Even though you really liked your solution, you actually found something in microstates interesting but at the same time, I think he was kind of missing that what aspects of Microstates overlap with redux. I think one of the things that we can do today is talk about what microstates has and what API does microstates provides and how similar they are to what people already know. I think that's one of the things that I was thinking about this conversation is that there are some things around microstates API like how to use microstates but the architectural concepts that power a microstates, they are already part of the development process and development architectures for most people that building single page applications. I think we can do is try to map these ideas not what people really know and patterns that they use to ideas in microstates, just to show how close we actually are. I think that's a good way for us to go. CHARLES: I guess we can start with, maybe one particular mode of managing state and then, how that maps to uses with microstates. TARAS: Let's start with components state because that was kind of our starting point. One of the original starting points for microstates because so much of our work was in Ember and component state is where most of Ember development happens. I think that's a good place to start. If you're someone who uses components state, if you're using Ember or Vue without something that is redux-ish which is pretty common these days, if you use components state, then one of the features of -- CHARLES: And also, if you're using React, right? Ember, Vue and React pretty much are all the same in this regard. TARAS: Yeah. Well, Ember and Vue are a little bit different in that behalf. Their particular pattern is you have data coming into the component, then you're doing some work with the data so that in Ember and Vue, you might use computed properties to derive state on that and then you might pass that state further down into other components. In React, you would do something a little bit different because there is no memorized computed properties by default, so you, a lot of times, write your computations that you pass into other computers that are written as expressions in JSX. If you're using this pattern, then one of the challenges around using this pattern is the process of lifting state becomes quite complicated. Because if you have like a bunch of state that lives in the component, it's attached to your derivation of state and the properties that you can invoke are attached to a component object. In React world, it's called lifting state. In Ember world, it'd be essential refactoring it into the parent component and passing that state in. in microstates, you're essentially moving this state into the microstate and now, what you have is this object that instead of having the state live on the component, the state now lives on the microstate object and in all frameworks, when you do that in microstates, you can use the getters to get the memorized computed properties behavior on the microstate. If you use your computed properties, computed properties are available for you on the microstate. You can do it the same way. CHARLES: It's kind of like a wrap up of that idea, not a wrap up but a summary of that idea is that the microstate essentially is a substitute for your component properties. If you're working with a component, a component has state associated with it, so you're setting properties on a component and then using the computed properties of that component. Microstates is you have all those benefits where you can set properties directly. You can define methods that like set properties as part of a transaction and you can then have simple JavaScript getters, which are just like computed properties except there's no enumeration of dependent keys. It's just, that's all they do for you. TARAS: Then the next question that arises is like, "Do I make like one huge microstate or a bunch of smaller microstates?" I think the question is really depends on the role of your component and what's nice thing about microstate is in microstates, there's two things that are kind of cool. One is when you represent your status in microstate, it makes it very easy. If you need to lift that state from the component up into the parent, you can lift that type. You can take that type and you can compose that type into the parent's microstate, if you have to and in that case, you would pass this new object that's created from that type. You'd pass the state for that component to the component that would otherwise have its own state. Essentially now, the parent has the state that would otherwise be in a child component and now, the parent is passing a state from the parent to the child. There's two kind of benefits to that. One is it gives a parent a way to control the state of the child and it also gives you a really easy way to serialize and deserialize state. You start off with having ability to serialize state for a particular component and all of his children, now you can represent that serialized state as part of the components state, so you have an easy way to restore the state of the component tree at the parent component. CHARLES: Right and that's possible because a microstate is really just encapsulating a value within a type but the value is just a simple serializable plain old JavaScript object, right? TARAS: Yeah. CHARLES: I'm just not trying to say like this is the feature that microstate provide, so when you're creating a microstate, you just pass it a POJO and it off to the races with that POJO and you can access that POJO at any single point. TARAS: Yeah. This is part of the architecture that I think are helpful for people that are using components. Is there anything else for people who are using components state that we think would be helpful? CHARLES: I think we're just realizing that you get the benefit of the laziness and immutability and the transactionality and all kind of things that we talked about last time but actually, the mental model is very similar. I think the thing to realize is that if that's the way that you used to working with state, the bridge is not too far. It's actually quite a short one. TARAS: Yeah, that's good. CHARLES: It feels very natural. It's not a huge mental shift. It's just more about a very small mental shift, a very small shift in mechanics for a very large payoff. DAVID: For instance, say you're in an Ember application using component state, what I'm used to is you're going to be passing actions around between components. Whenever you're trading that out for microstates, is it more that these actions are just bundle in with the microstate as the transition? TARAS: Yeah. One nice thing with the microstate, because the transitions that you can perform on any data type are intrinsic to the actual data type, the part of the data type, when you instantiate a microstate, you get these transitions for free. When you pass the object into a component, you can now invoke transitions on that object, that are part of that object's type. It's hard to imagine how awesome that really is because the closest pattern that you would see for that would be like, for example, if you're using Ember, you have Ember object or Ember model that you pass into a component and then, you can work a transition and because it goes through the Ember data store, your component is going to update. With microstates, there's no observation of any kind but what you're doing is when you are invoking the transition, transitions going up to the top of the root and gives you the next microstate, which updates your component tree. You get that functionality of 'data-down, actions-up'' built into the entire system of the microstates and all of that is hooked into these transitions that you can invoke on the objects that you pass into your components. CHARLES: Right, maybe David and Taras, you all could provide a concrete example of what that looks like. Because I think it is something that when I was giving talks on microstates at Ember meetups, one of the first things I like to show is you've got all these actions that you're writing either on your router or your controller but they're really actions manipulating the same type of data. It really comes down to like you're pushing and popping things off of arrays, you're toggling Booleans, you're incrementing counters, you're setting this property on this object. These are actions that we're writing and in the kind of microstates world, that's boilerplate. Because the transitions are intrinsic to the data type of the microstate that you're working with, maybe we could provide an example, like what's an action that you describe that you would pass around. TARAS: I think an easier one would be like if you have a model for example, you're composing a model into your application state or you're compose a model into your components state, so you have this model type that gets instantiated and you pass it into component that represents a model. Off that model, you might have open status like is it open or not that gets consumed by the model component, to know whether the model should be visible. Then along with that is model is open state, there is a toggle transition that you can find inside your component that is going to automatically flip the visibility of open. What will happen when you bind that transition to some kind of action handler that you invoke inside your component, when you invoke that action, it will then trigger transition at the top of that microstates and then, it will create the next state, push it through which will cause your model to change the visibility. David, that does answer the question? DAVID: Yeah, it does and that's actually the example that I have come up with whenever asked. A very simple sort of Boolean toggle. CHARLES: In a couple of my earlier demos, I should dust off that talk that I gave to the Austin Ember meetup where I was able to create an input with a dropdown menu with basically, some pretty advanced mouse behavior, all without having any component state or like storing it all in microstates and a lot of pushback was, "Aren't you putting logic in the template?" and the answer is, "Absolutely not." The logic is in the models but what I'm doing is I'm composing the actions that operate on those models inside the templates but at the template level, the action is data. If you're thinking about, we always want to have, we always want to lift state and then push that state down to the application, what this is really saying is that the actions are part of the data. It is implicit to the value that you've got. TARAS: I think that's really powerful because we do think about actions as being something that you can invoke. Like with closures, it's an action that you can invoke but considering that operation, that piece of data that invokes a transition in microstates and is derived from the type of the data, I think it's a cool concept. That's probably, one of the things that is kind of new for microstates but how you use it in your application should feel pretty much the same as you would if you were creating action handlers on your components. CHARLES: Yeah, just like kind of bundled action handlers for free. TARAS: Right. CHARLES: Maybe the way that you wrap up on this one when talking about kind of what microstates has to offer for the Ember developer, the Vue developer is really, I would say someone who's used to working with like MobX. Would be another good example is when I first started using Ember back in 2012, Ember Object solved a whole lot of issues in a really profound fundamental way and I really, really was drawn to that as the best API to be working with state. What kind of became apparent was perhaps not the best implementation. This is not like bag on Ember Object. I think it was actually amazing technology for seven years ago when most of it was written and probably, even came from before like SproutCore. It's almost 10 years old, which is incredible but it's still under heavy use in 2018 and it speaks well of how well it was constructed. I actually don't have much of a problem with Ember Object API. I just think that the way in which it's implemented that API means that there are some problems that require a different paradigm, so very much I kind of see microstates as heavily inspired by those types of APIs but with, I want to say a modern implementation but an implementation that is designed to solve the problems that have come to light over the last five years of developing single page JavaScript application. TARAS: I want to add that, when you describe with Ember Object, it exists in every framework that uses immutability. In MobX, the observation introduces the necessity to wait for a bunch of things to resolve. In Angular zones, I saw a similar purpose to ensure that if there are kind of cascading or streams or all of that stuff gets settled in into kind of restful state before you can start to assert on what's going on and with Vue, their computed properties has something similar. I think because of the complexity of having to track lots of objects and then recomputing things based on the result, that complexity in microstates is simplified by the fact that you describe you transitions. When you write them, you describe exactly what changes, so we don't need to wait for things to settle down. When you invoke a transition, that transition is explicit and based on the path of where that transition is invoked, we know exactly what needs to change, so we need to only perform one operation to compute the next state. There is no other things that we need to wait. There's no other side effects that we need to resolve before we know what the final status. I think that simplicity can be applied to all the frameworks that are using immutable APIs and derived computations from those immutable APIs. CHARLES: Uhm-mm [affirmative]. TARAS: Should we jump into talking about what microstates has to offer for people familiar with redux? CHARLES: Yeah, okay. Let's jump right into it. For folks who are familiar with immutable APIs like Ember, like Vue, like other ecosystem using MobX, a shift to microstates might feel like what you can expect. What about people who are just using often React-land and they're just using like set state. TARAS: Set state is a little bit tricky because it does get complicated over time and then you get this kind of funky things going on after a while where you start seeing things like, "I'm going to set state on this component and then once the change happens, I'm going to change some states some other state," so this kind of cascading state changes. The other part that I find particularly more challenging in set state world than it is in, I think in regular components state, like what you have with Vue and Ember. I feel like the way that the transitions, the state change handlers become part of the component, I find that part particularly kind of fragile. When you start doing refactoring, when you need to lift up state, it's like -- CHARLES: What's an example of this? TARAS: When you start off and your component owned old state and now, you need to have that state being controlled by the parent, we get into a situation where like, "What am I going to do? Am I going to do something with props as they're coming into the component or I should probably just flip that state from the child to the parent?" and now, you're refactoring the internals of a component to lift up that state so you can then combined those operation with the parent's state transitions. But then, you have this kind of added complexity of the fact that you're working with immutable data in that place. You've got these three things going on: you're refactoring your child component, you are moving these things into the parent components, you modify a parent component and now, you're also managing the complexity of forming immutable objects. I think the fact that people make it work is kind of a testament to human resilience. The people are able to solve such challenging problems and this is not a super hard one but when a component is complex enough and when stakes are high enough, these changes can become fragile. This is what I think microstates simplifies is that by taking care of the model, it makes components much simpler and makes it possible for you to just render a lot of components that are functional components without their own state. CHARLES: It actually reminds me and like I said, we're talking about what it's like to use them, not so much the implementation but that's the exact same sentiment that the author expressed in his book that was very helpful in writing microstates, which is Brian Marick. He has this book called 'Lenses for Mere Mortals' and he's talking about the practical use cases of using lenses, which microstates leans very heavily on. What he says right in the introduction is it gives you by abstracting the location and compose it the way in which your data structures are composed, it makes them very refactorable. You can say, "I want to change where the state lives. I want to change the structure of this object and I want to move it somewhere else," and it's sounds weird to say this, because the structure of the object is not coupled to the structure of the object, it means that it's very malleable, so you can move and you can say, "You know what? I don't want this address to be embedded in this person. I want this address to be inside this address book and then my person has an address book entry." Being able to make those refactors make those changes is very easy because your method of accessing the data is abstracted. It's something that applies beyond even user interface as a component state but it's something that this is a problem that's very salient when you are authoring complex UIs and compose components. It's something that you can benefit very greatly from. DAVID: Taras, you were telling us earlier about a friend of yours who is learning React for the first time over the past few months and he's mainly been using set state as you would just starting out in this world. You said that he didn't really get why you might have to go and learn some other way to manage your state and I think Charles said it, whenever your application starts to get more complex and you've got a lot of different moving parts, that's really where microstates will come out and shine for you. TARAS: Yeah. My friend kind of feel bad because he's spent the last three months learning how to work with React and how to use set state, "And now, you're telling me, I need to learn something else? Like I'm going to start from scratch?" I kind of reassured him that especially when you're a beginner, when you're learning, you learn how to address very specific challenges but you don't know how complicated these things can get when your UI gets really intricate. In those intricate scenarios is when you have to leverage your experience and you'll be able to solve these problems but if you're learning, you encounter these challenges head on and your tool does not really, like set state. Although it can be used to build really complex UIs very effectively but the complexity that over time increases as you start to manage the state, increases much more later on than does early in the beginning. Basically, with microstates, even if you have some basic proficiency with set state, when you start working with microstate, the things that you can do with microstates, you will be able to do more sophisticated things easier when you need to use them than you might realize when you start. Because microstates API is so consistent, there are a very few concepts to actually learn and they cover a broader range of use cases so when you actually starts using it, you'll encounter these challenges. What you already know it will just work for you and always just continue to work for you. That was part of kind of fundamental design on what Charles and I have spent, a lot of time putting into place from a point when we started two years ago. CHARLES: I think, and you touched on this and this is kind of what I was speaking to earlier is that when you're working with a simple system, it's simple, it's easy to work with and then the complexity starts to grow, it's never a good thing when you have to reach for a more complex tool to manage the complexity. It's a hallmark of a good system and that it can scale with you from a very simple and easy use cases to actually being able to handle very large and complex use cases but your API doesn't have to change and your usage doesn't have to change. If you have a tool that can actually scale with you from a one liner to a hundred thousand liner, that reflects very well on the design of the tool. I think you see that with things like Ruby, you see it with things like Clojure. You see it with, I would say there's even people writing like Haskell, shell scripts now but not so much with like C++. I think the analogy is very similar with microstates in the sense that when I'm looking for evaluating a language, it's not the only thing I look for but I really am looking like how is this going to work for me as a one liner? How's it going to work for me as a hundred thousand liner? If the answer is pretty consistently, then that's something that is going to get a lot of bang for your buck, so to speak. If I invest the time in learning it, I'm actually able to reap the reward of having a tool that's got my back on a whole bunch of different use cases. We talked about, in React, if I'm using set state and that's kind of a sub case of component state because I would say that in the previous systems we talked about -- Ember, Vue -- they have components state but the component state is a little bit more rich in the sense that you've got computed properties and what have you. But if we look at a system that externalize a state like redux, where you have a global state atom in your application, at least that's the way most of the redux applications that I've encountered behave, what does it look like for you? TARAS: I think it's a little bit challenging when talking about redux is redux conflate a few different things together. I think it's helpful to split those things up, so we can talk about them separately. One of the parts is how the state is delivered to components. The way that redux does is the instance is this created so every time you use connect, you essentially wired together. You can connect through the context or... I'm not sure. I think they have an observation mechanism that's internal to redux as well but they essentially connect components to the store and then they view that to deliver the state. It's kind of worth pointing out that for that like microstates bindings for React actually give you something similar out of the box through context. For those who really like the ergonomics of connect, I think it would be pretty easy to make that available for microstates. Usually, we don't even know why they wouldn't be available. CHARLES: But I do think that connect can be problematic like you can encourage you to not make components that are reusable and have isolated state. It's very easy to hitch yourself to the redux store and now, you don't have components that are shareable. TARAS: I think I would personally prefer to pass microstates instances through props because of the stability that's built into microstates and structural sharing, you can get some free optimizations and allow it to use functional components in more cases like out of the box but some people who are really like redux, they really like connect. Although it might be my personal preference, there's no reason why that wouldn't work if somebody wants me to connect function and make it available for people. CHARLES: Right and I think, there's a happy medium there too, right? TARAS: Yeah, I think -- CHARLES: -- You can connect components and then fan out that state to a set of functional components that are not connected. TARAS: There are some places where microstates and redux are very similar. When you're using in redux, you have this dispatch mechanism, where you essential saying, "I'm going to dispatch this action and your action creator is going to create an action for you with the payload," and then you're going to match that action to a reducer. One of the things that I kind of hear redux people really enjoy about redux is the one with data flow that dispatch the action and then, it reduces this next state and pushes that through and you have this kind of ingress point where everything is going through this one point. I think what's really interesting with microstates is you essentially get that. That's exactly how microstates works, in a way. The only difference is that the API is different. Any action that you invoke is going to go through a single entry point, which is going for you. Because we know the structure of the data, we know how to transition that state for you, so you don't need to reducers, so you're just defining your transitions or use the built-in transitions and then when you invoke them, we know the path. We're essentially forming for you. The path where you invoke the transition, conceptually, it kind of forms the name of the action that you are invoking. The path refers to a place where the state is going to be transitioned, then you have your transition, which is the actual reducer for that part but it's contextualize, so you don't have to think about how you need to transition that state in a great, global redux state. CHARLES: There's no matching. The matching is automatic. TARAS: Yeah, the matching is automatic, so you get that same single ingress and one directional dataflow. You get those mechanics except the APIs that you use, instead of writing the actions yourself. Instead of writing, we just use yourself. You get to use microstate types. I've heard some people who use redux who are like, "I really love the fact that microstates has types," but other people don't like types for whatever reason. Microstates comes with 'from,' which allows you to take a POJO and from that POJO, it creates a microstate and then you can invoke transitions the same way as if you had a type microstate. The only thing you don't get is you don't get to create your own transitions. You have to use the transitions that are provided for the primitive types. CHARLES: I think that there's a couple of benefits that you'll realize for free. There's laziness, reducers by default, or eager. When you dispatch an action, it will run against every reducer in the store. If the reducer matches, you're going to run the computation that's associated with that reducer. Microstates by contrast is lazy. Basically, until you try and read the property that is affected by that reducer, the reducer won't run. There's some ways that you can get around this. When you're using redux and first of all, you can actually have your reducers return objects that have getters on them. You can realize some of that laziness but again, it's work that you have to explicitly put in. I think, didn't you actually say that there is a package of plugins? There are plugins. There's basically a set of libraries that you could bundle together, which would give us an experience similar to using microstates. TARAS: Yeah. If you wanted to combine redux and reselect and immutable JS together, you can get some of the benefits, except this benefits are not integrated that well because they're still separate systems that you are essentially using together. Also, like microstates, it's four times smaller than redux and reselect and immutable JS combined together. If size matters to you and ergonomics matter to you, you actually can get a lot of ease out of using microstates while still maintaining the benefits of having redux. CHARLES: But if those things, those packages, like reselect and immutable JS, are things that are familiar and you naturally gravitate for it, then you'll probably absolutely love microstates. Because honestly, one of the ways that I think about microstates is like, what if you could have immutable JS, if there was no cost for composing the types like list and record. Immutable JS has come a long way since I've last used it or it's evolved since I've last used it but I think there's still only about four or five basic types and actually, making your own new immutable structure, your own custom type with its own custom methods that still get the benefits of structural sharing and laziness that you have on immutable JS is not something that you can do. But you could think one way to think about microstates is an immutable JS where you can make any type that you want. You're not just constrained to the record types and to their list types and set types or map types. TARAS: It's worth pointing out that at the moment, microstates doesn't map perfectly to immutable JS simply because immutable JS has certain optimizations for managing lists that kind of a great value of immutable JS and microstates doesn't have some of those pieces but -- CHARLES: They're definitely on the road map. TARAS: Yeah. Because microstates is abstracted high enough, that we can actually change internals and some people who are using microstates now, they will get benefits of ergonomics and there's already performance benefits from the stability that microstates offers but there will be a time when by upgrading to newer versions, you will not basically, need to change anything in your op but you will get the benefit of improvements to performance that we will introduce over time. Some of those improvements might come from what we will learn from immutable JS. CHARLES: Right. I have a couple of thoughts before we wrap up. What are the ramifications no matter who you are? What kind of development background, some of the benefits that you'll experience with microstates that might be a pain point or something you hadn't thought about where you are currently? A couple of things that I have jotted down is first, and this is what brought it to mind is talking about stability. Something that you see in a bunch of frameworks is having to manually track the keys that are associated with data. If I've got a list or I've got an object, being able to say, if that object changes, then I need to actually have some sort of key object, which effectively amounts to a hashing function to say, "Did it really change?" Because no matter what system you're in, you need to know how you're going to re-render. If the reference to this object changes, then the default thing needs to be, "I need to re-render it," right? You see this in Ember, in Vue. In React, there's this ability to pass a key or ask the question, "Should this component actually update?" and with microstates, that's much less of an issue because basically, it keeps the key tracking instability for you. If you are using a model with microstates, if you think of an object as a graph of nodes, no node in the graph will change unless it absolutely has to, at least that's the goal. We still actually have some work to do when you're running queries against the state of a microstate. We can cover that later but that's most largely the way it is now and definitely, the way it's going to be going forward is you don't have to do any extra work as a programmer to figure out what has actually changed. TARAS: That opens up some interesting opportunities. Imagine if you had a rendering engine that did not expect side effects to be significant and you could just say, "If I know whence they'd changes, I will then re-render that." That will be really interesting exercise, seeing like what would that look like for a performance perspective, if you have a very clear picture of what has actually changed and what part of the DOM as a result, need to be updated without having to do diffing. The [inaudible] you could actually do a diffing at much higher up. Actually, [inaudible] to diffing because you know what's changed but you can push a lot of assumptions higher up on the architecture stack. CHARLES: Right. That's actually one of my favorite thing about microstates and one of the unwritten values is like triple equals used to work everywhere and by and large, it does. It's usually simplifying but when you don't have to manually tell the computer what equivalence looks like, you can just say, "Look, are they the same object? They're not the same. If not, then they're not," and keeping that consistent is huge. TARAS: I think this is a good segue for us to kind of bring this conversation to a close and also, kind of set up potentially a third full-on conversation, which we could talk about actual architecture of microstates and design decisions because for people who just want to use microstates, they don't need to know all these details but for people who are curious, they might actually want to understand what are the considerations that remain when we were designing microstates, so maybe in a next conversation about microstates, it could be about architecture and the pieces in microstates today and then where we are going with microstates and what it could give us long term. CHARLES: Yeah, I like that idea. It is a plannable subject that we've been talking about internally for the past two years, so it makes sense that there would be plenty to speak about on the podcast. There is one other thing that I did want to bring up and that is, I think enabling to have a state solution that is composable because it allows you to think about your state first. Because really, if you do have a functional UI, where your view is a pure function of the model, that your view follows the model and so, if the view follows the model, then really, the thing that you should be thinking about first is the model that's going to be required to drive your view. I shouldn't drive. I should say derive your view because that's the primary artifact of which the view is nothing more than a function. It's a reflection onto a surface. I don't think that we have a state management solution yet, that enables that mode of thought, where I'm thinking about my prime artifact first and working forward rather than thinking about my secondary artifact and trying to kind of wishy-washy way, work backwards and reconstruct the primary artifact. I think that we've talked about all the development ergonomics and I think there's a mechanic of thought there that's enabled by this that I hope to see in more and more applications. TARAS: I think that's a really well put. I think that's something that I've been thinking about as well, as how do you convey this shift that microstates allows in terms of how we're thinking about architects and the application. For some people that value, the model, like they'll find that shift easier but regardless, I think that making that shift has a potential of simplifying your view dramatically and I'm very excited about exploring this further and having more conversations about this. CHARLES: Yeah, that's where we really kind of open up the conversation about state machines, which is also central to the conceit of microstates and using state machines as an incredible design tool but anyway, we can all get into that later. You heard it here folks, Episode 3 is coming out, although probably not for a while. We're going to be mixing up and we will be talking about microstates at least for a while. I understand that next time, we actually teased it but we based on how much material there was on microstates, we ended up packing in a second episode. We teased it last time, we're going to be talking next time about running an online conference with Twitch, so definitely look for that. Thank you, Taras. Thank you, David. TARAS: Thank you. DAVID: Yeah, it's been great. CHARLES: This is a wonderful conversation and as always, we are Frontside. As I mentioned at the top of the show, we have availability coming in August, so if working with us is something that you would like to do, we have a range of services, please get in touch. You can get in touch with us at @TheFrontside on Twitter or Contact@Frontside.io. That's it for now. I guess we should also mention that Taras that you are going to be giving a talk on microstates at ManhattanJS. When is that? TARAS: On August 8th. CHARLES: I will be giving a talk on microstates at ReactJS Austin on Monday, the 6th, so that is right around the corner. I'm excited about both of those talks, especially following so closely on the heels of the TorontoJS meetup talk, which I understand is... Is that posted online yet? TARAS: It's recorded but it should be coming out soon. We'll definitely tweet it out. CHARLES: Okay. All right. Look for that and we will see you next time.

In part I of The Frontside's microstates series, Charles Lowell, Taras Mankovski, and David Keathley talk about state management that's easy and fun and transactionality. This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. Upcoming Conference Talks: Toronto.js - July 30th (Taras) Manhattan.js - August 8th (Taras) ReactJS Austin August 6th (Charles) Transcript: CHARLES: Hello, everybody and welcome to The Frontside Podcast, Episode 106. My name is Charles Lowell, a developer here at Frontside and I'm going to be hosting today's episode and we're going to be talking about microstates with fellow Frontside developers, David Keathley and Taras Mankovski. Welcome you all. DAVID: Hello. TARAS: Hello, hello. CHARLES: I'm really, really excited that we finally get to talk about this but before we jump into this, just wanted to let everybody know that last week, we published our roadmap for Big Test, which is a JavaScript acceptance testing framework that we've been building right here, in-house at Frontside, which can work with both Ember applications and React applications, Vue applications. It's in alpha phase and we're looking for feedback but we're really excited about it, so we're going to leave a link to that in the show notes and please go check it out. All right. Finally, the moment has come that we actually get to talk about this publicly. We've been publishing things about microstates for a while now but we feel that it's ready to share with the world and that's really, really exciting. We should probably like wind the clock back a few years because that's about how long we've been working on this and talk about, kind of the why and the what of what microstates are. It's kind of a weird word. What are we doing here? TARAS: Yes. That's interesting question because we were working for so long and after all this time, what is this specifically we're working on. I can speak personally from my personal motivations because we have conversations over the last two years to why we were doing this and I think for me personally, it's always been that I've been mentoring building complex applications for almost five years now and one of the things that I find consistently is that there are patterns for how to build complex stateful UIs that the required solutions, that are fairly reliably consistent. I can teach people certain patterns and then they can use patterns to build complex applications and those patterns scale really well, the challenge is that, there's not an easy way to express them and it's different for every framework. The way that I would teach somebody how to do it, for example in Ember versus how they would do it in React, even though the pattern itself is the same but the implementation of the pattern is different but it's different in such a way that it's very difficult to see where the consistency is, what is the same about these two patterns. It shows me that there is room for improvement. In the same way that if there is an opportunity that in the future, components will get unified under one umbrella or what component spec. The fact that we do states differently across every implementation, across every opinion, it suggests that what we might be missing is something that would unify across frameworks, how we actually do statement management. CHARLES: Yeah, that makes sense because as much as people try to buck the trend of MVC, it can't keep coming back in the rear view. Hopefully, not like a horror villain but like a caring friend, like a reliable pattern. I remember when I was in writing Java applications back in the day, the most important thing when you were writing a swing application was making sure that you had your models right. If you were modeling either a form or a dialogue or a set of pages, the most important thing was to have those models. Back in the day, we modeled those things with event listeners, very similar to where the way like a Backbone application used to work or if you're familiar with Ember Object, the way that it worked basically, adding observers to some model so that when that model change, you were able to react to that. Your representation was able to be 100% dependent on this model. That was like a Java Swing Application, which was then inherited from this pattern from Smalltalk but we keep on seeing this again and again and again. There is this piece, this state, that if you're going to be representing something, then you need to be able to get at the meat of it. That's ultimately what the model is in the MVC pattern. I remember when React first came out, everybody was like, "Oh, MVC is dead," but now in terms of state management and all of the state management solutions that we see, what that really is, is the model trying to reassert itself. Because it's such an important piece of your application, it's going to rear its head and you cannot escape it. I make it sound like a problem. I guess, it is a problem but there's a proliferation of solutions out there that are attacking the problem in order to represent something you have to know what that something is. DAVID: I think when you only work with the one framework, your perspective might be very much shaded by your experience. If you're really good at building React applications and you have subscribed to POJO is king and you don't need anything beyond that, then you're going to get good at handling POJOs and you might not realize that there is actually limitations to what you're doing. I think the same applies to people who use Ember and they're using Ember Object and they have computer properties and now, in Angular, there is observables and subscribing to streams. What was really interesting with the work that we've been doing with microstates is that and one of things that have notice with other state management solutions is that a lot of them emerge as, "Oh, I had this insight. I spend a couple of days working on it. Here's my proposal, in a way, for managing state." I think that's great because insight and understanding is really important but sometimes, taking the time to design something can be really helpful. I think that one of the things that are really interesting with microstates is that because we've been iterating on it for so long, they've been testing it for so long, we've been able to do something in microstates that's really difficult to do when you have a production application that you're like, "I want something different. Something is not enough but I don't have the time to really make it better," or, "I have an idea. I want to release this thing but I don't have any other collaborators that I can talk to and really flush out these ideas," so what you get is a solution but it doesn't always touch on a lot of the angles that you want to touch on. It's really interesting when you think about -- CHARLES: I'd say, we should talk about those angles, like what are some of those angles? DAVID: Yes. You could think about this from a few different perspectives. I would say, from Ember perspective because that's been my personal starting point. For anyone who was listening. Ember and Vue, for all intents and purposes, in this scenario are very similar because they have the same primitive, which is the computer property. The computer property in Ember, the computer property in Vue is almost identical from the way that you actually consume. In Ember, you would use computer properties and in Vue, it's just computer properties to create derived state from data that's passed into the component and then, you would use is derived state to basically, decorate information that comes with the props, so you could present it in your component. This pattern is very nice. This pattern is being used to build LinkedIn, to build applications at Apple. There are huge implications being built by Ember community using this patterns and I'm sure something similar has happening now with Vue but this pattern doesn't really exist in React in the same way. You might opt in to start using this pattern if you start using [inaudible] but it's not quite the same. Writing computer properties in Ember and in Vue is essentially free because it's so effortless, where in React, if you're using [inaudible] to create cache computer properties and memorize computer properties, you are really opting into doing a particular way of computing this properties and writing selectors that's not trivial. I think that's one thing that is available in Vue and Ember and it's a really effective pattern and you can do it in React but it's actually not possible in Angular, unless you're using, I'm guessing something like ngrx, where you can do similar things to what you would do if with redux in React. Creating derived state that you can consume in your component is one of the things that is possible but it's not possible consistently across all frameworks. That's just one of the things. CHARLES: I agree. I think that was one of the things that drew me to Ember at first coming from Backbone as I did and honestly, from the models in Java was that, in order to compute anything, you had to install a listener and then eagerly make that computation and store it somewhere, where as those frameworks, it made you feel effortless where you can just decorate some state and derive it and the information is there, the computation is there when you need to reach for it but you don't have to do expend any extra effort aside from say, "This state is derived off this other state." I think another case that I came across was immutability. Immutability is a means but the end is to have consistency inside your application. I first really started running up against this when I was working with forms and since then, I've realized that actually, a lot of the pain that I was feeling was because things were not being immutable but this is where the fact that things weren't immutable ended up causing me a lot of pain and headaches and I was having the code into being a lot more complex. Essentially, when I wanted to make something transactional if you're editing a form or something like that, then you need to essentially store off a copy of your current state. If you're editing some object, I want to say, open up a dialogue and I'm going to edit it and then I'm going to commit the changes back to that dialogue. What I'm modeling there is a transaction so I kind of need to shave off a copy or make a Xerox copy of the object and then make the changes inside to that object and then somehow, try and merge those changes upstream and then, get them back into the main lines. Really, like a very Git-like operation. It wasn't just at the object level. When you're doing things like dirty checking, you need to make what is the original value of a field versus what does the input currently have. I might be doing any number of transformations in between the actual physical representation of the field, maybe it's a date but the user interacts with it in as a string, so there has to be this kind of parsing serialize thing sitting in between that value and you need to basically, keep a copy of that field as a mini transaction within your macro transaction because you want to say like, "Is it dirty? is it really the same object?" because if you can just do an object comparison, you have to get into all that hairy like equality checking and it gets really complex really fast but it turns out that a very clean solution to this is if you just never actually make the changes in place and whenever you are making changes, you're generating new information without destroying the old information because when a form is the case where we really come up against this, where we're modeling the same object over time, so a form explicitly models the change of an object. Change is part of what it is and so, the definition of change is being able to compare something in its prior state to be able to compare it to its current state. If you're making that change by destroying the prior state, then you're going to run into a lot of trouble. It just turns out that when you're working with forms and it turns out there are a lot of use cases like this but this was the one where I first really just couldn't even... Without immutability, you want to model your change as always rolling forward and not ever destroying prior states but being able to pick and choose and can always be able to look back to where you were and compare where you are now to where you were. That's where immutability comes when you're modeling change. It's actually much easier to model change when you have explicit states that represents what was and what is. But when you start doing that, then things get a little bit messy. You have to do the compromise. There's a tradeoff, like when I say, "Set this property on this object," that is easy and that's something that I think that we can all understand. We're not idiots. That's why immutable models are very conceptually easy to grasp, to wrap your head around. If I'm modeling myself and I'm saying, "Set hand position two feet up in the air and now I'm raising my hand." That's easy to understand and it turns out that when you're changing a data structure immutably, then you have to, for example model a simple set operation as duplicate and swap. Or if say like you're swapping out one property in an array, you actually model that as a map, where you swap out the one element at that index, rather than just saying, "Just set the thing at that index," or if I'm changing a property of an object, I want to copy over all the fields except for that one which I'm going to change. If you start to do that then, you realize the benefit of always being able to look backwards but you've now introduced overhead and complexity in your code, so you've made a tradeoff. I think that's a lot of people look at saying, "I want this to be immutable," then they actually have to come to the grips with the complexity that's going to introduce. There are libraries like Immutable.js that do make that a lot better but even they have the problems. I can talk about those but one of the cases is like being able to reason about a series of states for your object, rather than just only having one copy of it ever and being stuck with having to deal with it as it is. DAVID: Yeah and I think that really important too is being able to mentally track where you've been because in my experience on other large Ember applications, I've run into so many different bugs that I could really track down to people or rather, places in the code where things are just reaching in and mutating your model, where it wouldn't make sense, so as you're writing your code, you're in a completely different state than you might have expected. CHARLES: Yes and it's easy, right? But essentially, when you have some model, you've basically got a global object. There's a lot of recomputations that needs to happen when those things change. As a result, if you look at the actual code that supports computer properties in Ember and I'm sure other frameworks as well, some of the most hairy and complex. If you look at the chain watchers and the chain nodes and all the stuff that's required to support things like computer properties, it's amazing that it works as well as it does. I've tried to actually open that up and understand that code on a number of occasions and every single time, I had to walk away in defeat. TARAS: This problem exists in React applications and Angular applications because if you think of the challenges of working with set state, I think one of the problems with working with set state in React is that your complexity increases very quickly. It starts off kind of simple when you're using immutability with set state in React. They're kind of very light and then, as you start to add features, your complexity starts to grow very quickly in a way that you really start to run into limitations. You start to confront your personal limitations of your abilities to work with immutability, so you're limited to doing immutable structures that only a few levels deep because your ability to use destructuring to create new immutable object is limited by expressiveness of JavaScript. It's gotten a lot better with the destructuring but if you do it a few levels deep, I think everybody's familiar with what happens. It starts to get really hairy. You kind of loose track of where you are. CHARLES: Yeah. The signal-to-noise ratio increases because after probably, two or three levels, the majority of your code has to do with destructuring and restructuring and very little of the actual change that you want to make. TARAS: And this is kind of interesting because people talk about declarativeness versus imperativeness but at certain point when you have a complex immutable state change, if you're doing with destructuring your code, even though it looks declarative but there is so much processing that you're doing, it's actually kind of losing the benefit of its declarativeness. It's actually starts to look more like imperative code than it does what you would expect a declarative system to look like. I think this touches on the other aspect. It kind of compromise that when you work immutably, when it compromises, you make a serialization. Your ability to represent your state as a POJO becomes restricted by the fact that you have this complex system that's wired together and you have systems like zones in Angular and in Ember Object that are able to keep track of changes in these objects but you don't have a way to restore those objects. You don't have a way to do more sophisticated things that you might want to do, especially in situations where if a service feeding you, what do UIs going to look like. In that situation, it's really helpful to be able to say, "Here's a POJO that I got from the server. I'm going to use this POJO to build this component tree that the user is going to interact with and then, as the user interacts with it, I'm going to then, capture that state, serialize it and put it back in the server." When you're using something like Ember Object or if you're doing this kind of stuff in Angular or even if you're doing this stuff with React but without using something like redux, you essentially end up doing so much wiring to accomplish that. By the time you finished writing your application, you've written a ton of code just to handle this particular use case and if you have to do this again in another application, you just rewritten that kind of code in a new application as well. CHARLES: It reminds me of the concept of a Smalltalk image, like there's no way to really get at the state of a Smalltalk thing. It's almost like you're dealing in docker containers and not actually being able to write that state down into JSON or something like that. I'm trying to casting about for an appropriate analogy. Maybe that's not a good one but what's actually happening cannot be made orthogonal. It can only exist in that one run time that you're currently running. If something wrong manifests itself, reproducing it can be extraordinarily difficult, right? TARAS: Yeah. CHARLES: Imagine if there's some render cycle that's making a bunch of mutations and there's this process that you stood up and it runs in completion, some signal comes in and those effects are like ripple through the system, there's no way at any point to have any other representation of that system than the running system itself. TARAS: There's another element to this, which I find really interesting. When you're thinking about how to architect complex UIs, it's actually helpful to get really clear about what kind of changes are happening. A lot of times when I want to see beginners are writing, especially if you task someone who is a fairly junior at building a single page applications, a lot of times what will happen is because they don't have a clear mental model of what is going on in regards to state. They end up setting a lot of properties. Every operation, every time you have an event handler because they don't have a clear model of what's going on, they end up setting like five or six or seven properties. That kind of signals that they don't have a clear picture but what that also does is a lot of times, they usually comes together with cascading state changes. Usually they're not representing a single operation as a single state change because there might actually be a bunch of things that are happening because what they're doing is they're massaging the system into submission. Not like they're not in control of the state transitions, so they use, essentially time and their dedication to kind of sort it out and make it work and eventually, would that ends up looking like that if it works for most of the cases that they are able to test for or that they able to manually see. But then they AR, not accounting for problems that they're not able to understand right now, they become discovered by users when users start to interact with the system and with the components or with that application and the application is there to get into some funky states. The tools that we have, they don't prevent that from happening. They just -- CHARLES: Right. They don't force you. I think what you're saying is that ideally, you want to model your UI as a set of transactions on your state, that you want transactionality to your state so that I basically am saying, "I'm not going around and setting seven properties in reaction to this one event." I'm saying, "This event triggers this transaction and that transaction clearly bundles up every single operation that needs to happen and the tools don't enforce that." Is that a fair --? TARAS: Yeah and then, the problem is we're working with component trees. You start off with having a set of requirements and over time, the requirements change. As the business unit understands your application better, they give you more direction of how accounting should work and then you find out that there's more interconnect at stake but then what's happening now is that the cost of refactoring those state that spread throughout the components, whether that be with set state, whether that be with the actions in Ember or even in Angular. What you end up doing is you start to change the system but change is not trivial because the actual process of changing where that state lives is not linear. It depends on the complexity of the code that you wrote and it just gets really hairy very quickly. That's where companies end up losing a lot of time. A developer could start off with a requirement, you build something and then a new requirement comes in and instead of it being a simple change, it turns into a week or two weeks refactor because you now understand the state ownership should be different. The state that you have should be in a different place. You have to manually make that change. You have no obstruction to help you express that in an easy way. CHARLES: Another thing, because we are doing a kind of a roundup of all the things that you need to account for when you actually embark on managing your state. Another is actually constraining the amount of computation that happens. If your system is based on listeners and large chain reactions of things where it's like, "This property changes so I need to notify these other 10 dependent properties that change." You can do a lot of unnecessary computation, especially if nobody is going to render. That's kind of the thing that you have to do if you're going to be immutable. You have to eagerly walk those change to see which objects are affected so that you can then invalidate those caches. A system like Ember Object, I don't know exactly how Vue works, it mitigates this somewhat by the fact that the computer properties are lazy but you still have to walk all of those chains. That can actually get out of hand. They're eager, not lazy. Then the other concern that you have, where you normally have to make a trade off around is around composability. One of the things that's really nice about immutable systems is they're very composable. If I've got some object that does one thing, I can then just set that object onto another object just by mutating one of its properties and I've effectively composed them. I can then install listeners onto that thing or I can compute properties off of that property and they can post pretty well. That's something that you get but then of course, you're losing all of the benefits of immutability, so things like Immutable.js don't really compose very well or redux doesn't really compose very well. The concept of taking a redux store and embedding it into another redux store, you just don't see that. I would never distribute and I think ultimately, the litmus test there is would I be able to share it on something like npm. Nobody shares an npm package that's just a redux store that you can dispatch actions to and observe and use it with your other redux stores. When it comes to a system like Immutable.js, that does make transitions a little bit easier over lists and arrays and maps but you still run into the exact same set of problems that you have when you have lists of maps of records and you don't really get any help there, so you have to make this tradeoff between immutability and composability, whereas a system like MobX or Ember object actually quite composable. Before we start talking about microstates, I want to say that you just throw those in there because there is just a lot of concerns out there, a lot of edge cases that actually build up but through the course of a real application, you will encounter them all. You might be making tradeoffs at the beginning that you're not realizing that you're going to need or are going to get you into trouble later on. DAVID: This actually happens in the Angular community as well because there's something really great that's happening with observables in the Angular community. I think everyone's embracing them wholeheartedly and I think that's really been pretty great to see but observable streams of composable, but objects that have on them observable stream providers of some kind, like if you have something that you can subscribe to and that is part of a property in a class, composing multiple classes together and consuming properties from those classes, there is no mechanism for composing that. That kind of composition has to be done manually. Again, you're kind of manually wiring together a bunch of objects and the big challenge is that you are manually subscribing to all those streams and unlike what you have with components. Components have lifecycle hooks. When your components is being torn down, you know you can perform some operations. If you need to remove an event listener, you have a hook where you can do that when you have a class instances like JavaScript class instances that have on them properties that have observables that you subscribe to. There are no tear down hooks for class instances, so there are no obstructions from managing unsubscribing from those streams. You essentially end up having the foundation that you can use to build complex reactive systems and you can subscribe in there really fast but wiring those things together at a bigger scale is simply not there. It's something that you have to create and enforce yourself. CHARLES: Right and I think that's probably a perfect segue into talking about microstates, which is the project that we've been alluding to for the past 30 minutes, that is I think in attempts to solve these problems and make sure that you don't actually have to compromise on those things, so you can reason about things locally but have those things be composed into a greater state. But also have them be immutable so that you can look at past states and reason over a data structure over time as opposed to just in one instance. Also, have an intuitive interface that when you're making these changes, doesn't look like half of your code is unpacking some data structure, flipping some bit in it and then repackaging it back up again. That's the context. Should we start talking about what microstates is and how it addresses those? TARAS: That's a good next step and when I start working in the ReadMe, I end up actually, I think I wrote about 40 pages. One thing that's interesting about microstates is that and this was part of the design of microstates from the work that we've done is that we intentionally wanted to make the number of ideas that you haven't microstates very little, so when you use microstates, the number of concepts that you need to remember in your mind is very few. It is a conceptually a different way of thinking about organizing your state in the same way that shifting from managing DOM elements directly to having an obstruction like component that declaratively applies changes to your DOM tree. In a same way, microstates is kind of an abstraction that allows you to declaratively describe how your state will change and it will manage the transitions for you and allows you to give the state transitions names and it allows you to give your states names as well, so you can actually name things. You don't get a POJO that has a shape but has no name. You actually get to give things names. CHARLES: So, why don't we start? I have a list in my mind. I should probably write it down of the things that we just talked about but I think the things that we talked about are ease of representation, like conceptually easy, transactional, basically serializable and immutable, lazy and composable. Those are like five or six things. But I think there are kind of aligning principle around which we gathers that the state management should feel easy. It should feel fun. One of the things that is awesome about working with components, whether you're using web components or React components or Ember components is when you get it right, you're just snapping these things to feel together and it feels great. It's like I'm just passing properties and render blocks down the tree and the framework is just doing all of the grunt work for me and I'm just operating at a very high level. That's what organizing principle with microstates as it needs to feel easy. Maybe we should start there and just say, how does that easy and fun line up with each one of those kind of unique problems around which we typically have to make tradeoffs? We could start with the interface of making a change. TARAS: I'll go back to the starting point. I remember what got me first interested in microstates is Charles, when you said that, when you have a number, there are certain operations that a number can do. We really don't need to be writing an increment operation for every... Like if you a have a number, you can increment it and decrement it. CHARLES: Honestly, every time I see state management tutorial and they tell you how to increment and decrement a number with it and you write the increment code and you track the thing and you store it back into the store, at this point I'm still annoyed with those tutorials because I'm like, "It's a number. We know we can increment it. Just show me where to plug in the code. I should not have to be writing an increment method." TARAS: Yes. And that's the kind of starting point. There are certain operations that you can perform with the primitive types. If you need to add a number or if you increment a number, we already know how to increment the number. It's part of microstates. But that in itself is nice but that's not that important. I think what's really important is that when you need to put a number into another data structure, let's say you have a nap and you're like, "I need to..." I don't know -- CHARLES: Let's say, like a click tracker that has a number of clicks. TARAS: Right. By itself, you can increment the click tracker but if you need to put a quick tracker into another app, essentially you can compose it in and you don't need to figure out how to wire the actual mechanism of how to make sure that you can update the property, like it's part of another class, for example, you don't need the way you would increment the number. When it's a part of another class versus how you would do it when you're working with it by itself is an approximately the same. The amount of work that you need to do to actually perform that operation is the same. Your complexity doesn't increase as you compose one data structure into another. CHARLES: Right. You can just say, "This is an app. It's got a click tracker and this property is a click tracker and I have to do nothing else. I can register clicks on that thing. It doesn't increase the complexity of application at all." TARAS: There's no wiring. Now, you added some new state, that state is very explicit and it's really clear that it is not impacting other parts of the state. You can operate with this thing. If you change it, it's going to work properly with all the other things that are in the type that you are adding this counter to. Those things are just going to fit well together and it's not going to break if you need to transition one more thing. All of the other transitions will work the same way. I think that kind of consistency is really meaningful, over time especially when you start to increase the amount of state that you manage in your application. CHARLES: Just the ability to work with types and just have kind of those implicit operations and have those things compose, kind of indefinitely. Moving down, we talked about easy and the other thing I would put on that is that the way in which you express those transitions, for example if microstates comes bundled with numbers and Booleans and strings and arrays and objects and kind of the stable of types that you would expect in any JavaScript application but those types are expressed using this way for expressing types, essentially. When you actually do make a transition, it feels very object-oriented, I would say, even though it's not. It feels like you're mutating but you're actually doing a transition. Does that make sense? TARAS: I think for anyone who is familiar with what it's like to write queries for GraphQL, if you're not familiar with it, it's fine. You can get a sense of that from microstates but if you're familiar with the ease of just writing a query and if your backend knows how to retrieve the data, then your queries will just give you the data that you want. That feeling is really powerful and just being able to write the query and just gives you what you want. Microstates is kind of like that. Actually, the inspiration came from experiences with GraphQL, which is that sense of ease is what we wanted to have in microstates and so you get that seems sense of like, "I can just do what I want and it just going to work and with this other thing and it just going to work," and you're just like flying through, like adding states to your application and it's just working for you and working for you and working for you and you don't have to do monkey work like gluing things together. It'll change how you are working before because you have a way of opt working with these things at a higher level. CHARLES: Right. Let's talk about transactionally or should we talk about immutability? How does this make immutability easy and fun? TARAS: I think one thing is that you don't have to write reducers and you don't have to do destructuring by hand. I think you have a way of expressing. Thinking about this, if you have a component tree and let's say you have redux and then a bunch of components like your parent component, your root component has some state using sets date and then components further down the tree also have state. You could actually express that as a microstate. What you would do is, essentially the parent component state would be the root and then the children's component states would be composed into it. The nice things about doing that is that at the root level, you have access to transition state of the children declaratively. You know where the states for those children is on the route type and you can write transitions that are going to declaratively perform multiple operations on the children state and I suppose it'll restructure to what happens with components but if you don't use this, you might have multiple sets date operations. The process of wiring data from the root down to the children is kind of complicated, where here, you have a way to represent that and perform a lot of transitions in the way that is going to be just easy at whatever level you need to operate at. CHARLES: Right. I think, for people familiar with redux, in redux you act globally and then you react locally, if that makes sense, so you dispatch an action to the entire store and every single reducer can see that action. There's ways to manage that but effectively, you have this one atom and then you have the reducers that kind of act on local state, whereas with microstates, you're basically acting locally. You're reacting locally but the effect is global. TARAS: You're participating globally. CHARLES: Yeah, participating globally but you never have to consider the context that is above your own, so you never have to be mindful or cognizant of the context in which you're enclosed because from your perspective, it just doesn't exist. DAVID: Every microstate has a set transition which is the basic transition that you can invoke, essentially in any type, so what's interesting is that it's amazing how powerful -- CHARLES: So, we should break it down really simple. Basically, when you create a microstate, with a type, you say like, "I want to create a number with the value five," and then I can just say, "That returns a microstate," and I can say, "microstate.set 10," and that will return a new microstate who's also a number but the value is 10 and that's available on all microstates. DAVID: Yes. If you have a tree of components and your state is presented by a microstate at the root level, then what you can do is you can invoke the transitions on any part of the microstate and it will just know how to properly create the next microstate for you. The example that Charles you gave of one number so that number can be inside of a class that represents state for a particular component and then that can be a part of another class or represents a state for another component but then, when you invoke a transition on one of the leaf nodes, an equal sets state on one of the leaf nodes or equal set on one leaf nodes, it will respond locally but it will actually reflect the changes globally. At the root level you're going to get a new object that causes the components to update. CHARLES: You know what? I have another concern that actually just popped into my head, which is something that I've certainly struggled with in every single application of notable complexity is stability of value. We should put that on the list. We're almost out of time to talk about this. We spend too much time... Well, not too much time, of the issues of state management, which I think you can't spend enough time talking about but I did want to pile one more on there is when you're making that transition, where you're acting locally but you're participating globally. For things that are unaffected by your action, remain unchanged. That is a super power. It's actually very hard to do with a lot of state management, especially when you're cloning a bunch of stuff, being very judicious about what you don't want to clone. Where this really comes into play is if I want to re-render something. A lot of times you have to jump through a bunch of hoops to tell, did my model really change or did only referentially change? With microstates, when you make that local change, if you're embedded in a very large graph of objects, obviously all of the objects above you are going to be changed but what about things that are off to the side of your siblings. They're outside of that scope of that change. They shouldn't be cloned. They shouldn't be copied over. They should remain the same. If you're doing a re-rendering based on the changes that are happening, that's going to be a key feature because you're not going to have to write, basically any hooks to say, should I have to re-render my component. You can always rely on triple equals. DAVID: This quality is going to describe the structural sharing and some of the other tools that are available but I think that's how it's accomplished. I think one thing that's a little bit different with microstates is that when it comes to structural sharing, it's not that difficult to do if you just do structural sharing and value. Meaning that you can do structural sharing on a value using something like lenses and not a lot of people are familiar with lenses in JavaScript but it's actually only three functions that you can use and it can give structural sharing on complex POJOs. It's pretty easy to use relative to how little people know about it but what that doesn't do is it doesn't allow you to graph of objects that have their own operation that you can invoke and that will perform structural sharing. That part, I know that is not available in any solution, I think at the level of completeness and luxury that microstate provides when you write things. CHARLES: Right because every piece of the tree kind of comes bundled with its own things that you can do with it. DAVID: I don't think you should jam everything into this podcast because there's a lot to talk about it. I think one of the things and we've talked about this a lot, which what we want to do is create an implementation for an idea of what it would look like. What would it be like if we had a composable state primitive that we could use to describe state and share state solutions in the same way that we share components like react-virtualized or whatever your particular frameworks or popular component may be. What it would look like if we had solutions to state problems that we could share and we could -- CHARLES: I think the litmus test of an awesome solution is like you look at this current crop of MVC frameworks and what's so awesome about it is they're sharing. That can happen, right? If I'm writing a React component, I can publish it on npm and other people can use it. If I have an Ember add-on, I can publish it and people can use it. They can consume those components. That's awesome and I think it's the hallmark of a great system. What would it look like if I wrote just the state piece of a file upload and I could publish it npm and then anybody, in any framework, could actually use it with their framework without paying any penalty. What would it look like if there was some transactional data store that could be built and shared and the hooks into any framework were minimal. The possibilities are really exciting around that idea, whether it is realizes microstates or not. But clearly, we feel this is something you should be able to do. DAVID: I'll add one more use kind of use case that I personally find really motivating is that there's a lot of companies that are investing into building single page applications and a lot of times, what you see happening is they're building a very similar application to what they had before because their business hasn't changed. The technology has moved on so solutions have improved. The demands for better user experience have increased but the actual business and how the things that people have to do on day to day within their company hasn't changed. What we're seeing right now is we're seeing the same, like whatever was written before in jQuery or an AngularJS is now being rewritten in React or Angular or whatever you might Choose. But whatever you like to see is it has a situation where the domain specific logic of your business is represented as a data structure that knows how to, potentially, in the future talk to the server and retrieve data from the API because that's likely not going to change. But you can use that like that's been tested and published as an npm module within your enterprise and then you can then consumed that in any framework and it's actually easier to do it this way, than to implement it in a framework-specific version of their state management. That's the part that I find the most exciting. I think one of the things, just to connect to the goal is that, we would like to keep this conversation going. If you're interested and I think this is a kind of a call to our audience that if you're interested in this, we would love to have in our podcast to talk about these things because I think there's a lot of things that microstates really is a beginning of a conversation. It's not meant to be a statement. It's meant to be a proposal that we can just talk about this. CHARLES: I agree and that's one of the reasons we're keeping it very small at this point. The core library of microstates is not setting out to accomplish too much. In the core library, there aren't even any side effects. It's actually impossible to have side effects. It means that it cannot be used for anything except for the model but that's very liberating and it let us focus on what would a system like this actually look like. DAVID: It's really exciting because this has been the biggest metric but we have microstates in the JavaScript weekly and that was great and then it got circled around when people know and it's like 800 stars now, which is not a really big deal. It's funny because somebody commented like, "How can you put something in production that only had 100 stars?" CHARLES: I think it's just important to realize that this really is the beginning of a conversation. There's a really exciting set of things to come. We haven't even talked about how we're going to model side effects, although we're going to use microstates to do it. We haven't even talked about what the various framework integrations will look like and what are the best practices for using this to organize state in your application. We've had some lively discussions internally about what that looks like. There's still a lot of questions but it's going to be a really, really exciting and edifying experience to get to answer them. DAVID: Yeah, it's pretty exciting. I'm excited too. There's been a lot of interest from people in microstates, so it's going to really great. I'm looking forward to meeting people and having conversations about how we can use microstates because I'd love to have someone create a really great solution that I could just take off the shelf and just use and not have to implement them myself. CHARLES: All right. Well, I think we could talk about microstates for at least the next three hours but we have to give everybody an opportunity to, at least like go to the bathroom or something. Microstates will return but if you're interested in learning more about microstates and you happen to be in one of the many places on which we're going to be presenting on microstates in the future, who knows? Maybe you can come in and join the conversation in person. Taras is going to be speaking at Toronto.js on July 30th. He's also going to be presenting at Manhattan.js on August 8th and then, yours truly will be presenting on microstates at React.js Austin on the 6th of August. Come out and see us. We'll drop those in the show notes and it's guaranteed to be a good time and we'll have that conversation. Until then, we are the Frontside. We lead with the why, the how and then the what, if you're interested in working with us and that helps us that we guarantee the lowest total cost of ownership for your application. We're always looking for feedback. If you have news items that you'd like to see at the head of the show or just any feedback or questions, we would be happy to answer them. Thanks today to Mandy Moore for producing our show and next time, we'll be talking with Kristian Freeman about what it's like to run an online conference with Twitch, so I'll be looking forward to that. Bye David. Bye Taras. DAVID: Yeah, thanks for having us. TARAS: Bye. CHARLES: Yup, and bye everybody. See you next time. Next Time: Running An Online-Only, Free Conference on Twitch with Kristian Freeman

A rare episode with Charles Lowell, back to reprise some DrunkAndRetired.com style rambling. We discuss, oddly, actual retirement, GI Joe vs. transformers, and talking about just one thing. Oh, and "a dry heat." Robert will be back next time. Special Guest: Charles Lowell.

A rare episode with Charles Lowell, back to reprise some DrunkAndRetired.com style rambling. We discuss, oddly, actual retirement, GI Joe vs. transformers, and talking about just one thing. Oh, and "a dry heat." Robert will be back next time. Special Guest: Charles Lowell.

Special Guests: Brian Douglas and Bex Warner of GitHub. In this episode, the panelists talk about automating GitHub with Probot. The origins of Probot are discussed, as well as making GitHub apps with the GitHub API, automating workflows with Probot, must-have Probots for every repo, and GitHub's V4 GraphQL API. References: Microstates README Probot github.com/integrations/slack github.com/marketplace/pull-reminders platform.github.community/c/integrations probot.github.io/apps/unfurl-links/ probot.github.io/docs/deployment/ probot.github.io/docs/extensions/#scheduler probot.github.io/community This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. TRANSCRIPT: ROBERT: Hello everyone and welcome to Episode 105 of The Frontside Podcast. I'm Robert DeLuca, the director of open source here at the Frontside and I'll be your episode host. Today, we're going to be discussing automating GitHub with Probot with Brian Douglas and Bex Warner. I'm really excited about this topic. The idea of automating GitHub workflows with bots is amazing. This is something that I've been wishing the GitHub have the platform support for since I even started using GitHub for open source. Just being able to have a bot to take care of certain things like somebody doesn't leave enough of a PR description and they open up a PR, you can have a bot that just responds to it and saying, "Can you provide more information?" It's pretty awesome. With me as co-host today is Charles Lowell, who is also a developer here at the Frontside. Hey, Charles. CHARLES: Hey, Robert. ROBERT: Before we get into the discussion, I like to make a tiny little announcement. We've been building a composable and an immutable state container called Microstates. I'm sure Charles can talk about this more at length, then we will in the next podcast episode -- 106, but I would like to make a small announcement that Taras who is an awesome developer here just wrapped up a month's worth of work, creating a new ReadMe to describe the vision of Microstates and what you can do with them and everything about Microstates. If you're interested in that, I highly recommend checking out the ReadMe. I'll drop a link in the show notes for you that are interested. CHARLES: If I can add, it really is [inaudible] because it isn't like any other state management solution out there. ROBERT: No, absolutely not. I've been building something with it in React Native over the weekend of the 4th of July and it's amazing. But enough about that, you'll hear about that next episode. For this episode, I want to talk about Probot with Brian and Bex. Hi are you two doing? BRIAN: I'm well. BEX: I'm good. Thanks for having us. ROBERT: No, thank you for joining. This is really exciting. Like I said in the intro, I've been really excited about this project. I do a good amount of open source, I would say and this has been really helpful in all of our repos. We have, I think like 78 open source repos on the Frontside. We have Microstates, like we just talked about and Big Test and all of those repos use some combination of Probots that people have built and it's really nice, especially with the new Checks API that has just come out. You can integrate Probot into that, right? BEX: Yes. I, actually am currently working on shifting one of our bots from using the commits Statuses API to the Checks API. ROBERT: That's awesome. Before we go too deep into it because I want to come back to that because that sounds really cool and what the integration of that is like and what changes because I'm not even really that familiar with it. I just know it was released. I kind of want to go from the beginning here. Where did Probot come from and can we get a little bit of a history for everybody that might not know what Probot is? BEX: Sure. Probot originally started out as this simple idea to make GitHub scriptable. The original idea was you have a single file in your repository that would be like a JavaScript file and it would essentially spell out how the bot would act on your repository and the goal was to make GitHub apps accessible to people because if you ever look through our GitHub apps documentation, I think it can be a little tough to get started. There's, honestly, a lot of nonsense that you have to go through in order to get set up. For one thing, the way our GitHub app authentication works is it requires a JSON web token followed by using that JSON web token to request an installation access token and that process would be really tough for new people to get started. ROBERT: Yeah, it sounds like it. BEX: Yeah, so Probot was created to abstract all of that away and handle all of that authentication automatically and simply leave you with the payload that you get from listening on web token events and in authenticated GitHub client to make authenticated API requests while authenticating as an app. ROBERT: Cool, so that's where it started like a flat JavaScript file in the root but today, you use like EMO files and a .GitHub folder. How do that kind of progress? BEX: Originally, their use case was much simpler and it quickly became clear that a single JavaScript file in the GitHub repo was not scriptable enough and not easy enough to understand. The goal was to make like an API that could make that JavaScript file really, really easy to customize for every API of GitHub and it quickly became clear that that was not really a feasible thing to do. as time went on, it turned into this way to build Node JS applications and essentially, what the configuration files you're referring to are the way in which we make it customizable because right now, there's no way to be officially supported GitHub apps channels to pass secrets because it means you're a [inaudible] and the owners of GitHub apps, so that was just a way to kind of stop that problem. ROBERT: Gotcha, okay. BEX: The actual code for GitHub apps still lives in a Node JS module basically and the configuration file just specifies how that module runs. ROBERT: Right, so they're deployed like Heroku instances, if you want, like anywhere you can host a node app. BEX: Yup. Heroku, Now, yeah. ROBERT: Interesting. BRIAN: As a reason to that, some explorations of doing serverless deployments for Probot, I think there's a couple of issues of them. I'm not sure if anybody's shipped anything like the way they at but it's pretty much it's possible to. BEX: Just a week ago, we even released a new version in which we update our core from Node JS to TypeScript and now that things are typed, we have big plans for serverless. ROBERT: Nice. That's awesome, so then you'll be able to deploy to a Lambda and off to [inaudible]. BEX: Exactly. CHARLES: Can I actually interject here, as kind of a person who doesn't really know the relationship between GitHub apps and the GitHub marketplace and what exactly a Probot is before we hear the origin story. I would love to hear a very high level view of how this ecosystem fits together. BRIAN: I think a lot of people are pretty familiar with interacting with the GitHub API and OAuth integrations. I think I've just spent a lot of time at different companies previously to GitHub, just like making calls, either to cURL or through Node JS or more recently, [inaudible]. GitHub apps itself are a way to take all the things that you had to do to make an integration to GitHub much easier. It has a lot of cool things like OAuth, scopings, so you no longer have ask for all your repos ask access whenever someone logs in with GitHub and the connection between like, "Now have gone from OAuth to Now to GitHub apps," there was a lot of, as Bex mentioned earlier, ceremony that happens to getting set up with GitHub apps and integrations that Probot is like this tool to speed up the process of getting to the point where you just want to script some automation or some sort of workflow and it gives you all that bullet play for you. I don't know if that was a good high level for you Charles. CHARLES: Yeah. I've kind of witnessed this second hand with Robert installing a bunch of things here, so let's use an example, like you did some sort of automation on our repos, Robert, where when someone files a ticket, there's this workflow that automatically adds a triage label, so that we know that this thing hasn't even been dealt with, so we really need to address that issue. It doesn't need to be as a high priority. It doesn't need to be closed as a duplicate of something. One of the different aspects that you described there, how do they fit in terms of serving this workflow onto the end user? Or was that a good example, even? BRIAN: One of the cool thing about GitHub apps and what Probot does for you is that normally, if you want to add a label to an issue, either you Charles or Robert, would have to be admin or maintainer on the team for the Frontside and you could add labels. But somebody who opens up an issue, doesn't have that ability to have write access to your content, which is adding a label. What a GitHub app does, it actually takes a spot as if you would have another user on your platform, instead of creating a dummy account or a dummy user. Probot is basically building a bot for you to then, give you the ability to add that issue. That's sort of workflow that normally would have to happen through an actual real human could not happen through a bot without taking up a spot of like, "I guess, I probably shouldn't speak so ignorant about our platform and what we actually pay for nowadays for GitHub," but I know we used to have like a limited amount of seats for organization, like that seat no longer has now taken up and now, it could be just be used a bot can do something that normally us would take. ROBERT: Right. You no longer have to create a user to do these things. BRIAN: Correct. BEX: [inaudible] within GitHub. It's sort of built in a way that apps can take a lot of power in your repositories. CHARLES: So then, what is the relationship between Probot and an app? BEX: Probot is essentially the framework for building an app. You can definitely make the equivalent of any Probot app outside of Probot. It abstracts away all of, basically, the horrible parts and leave the easy part. CHARLES: Now, I think I'm ready to participate in this discussion. ROBERT: That was perfect, though. That's a great intro because I actually didn't have a total grasp or understanding of the relationship between GitHub apps and Probots. That's really good. BEX: Yeah. Additionally, going back a second. You mentioned the marketplace before. One thing to note that is that there actually are several Probot apps on the marketplace right now. The marketplace is essentially the home for any larger, usually third-party companies that have made apps and Probot is essentially supporting some of those. ROBERT: Interesting, so then my question would then be, do you know anybody selling their Probots. Does the marketplace charge? I'm going to assume it does. BEX: Yes. ROBERT: Okay. Is there anybody charging for their Probot? BEX: Yes. There is a quite a few, in-fact, charging for it. Recently, a pretty popular example is the GitHub Slack integration, which is if you open new issues, you can have them appear in your Slack channel. That whole application was recently rewritten by GitHub. It was previously owned by Slack and that was built on top of Probot. CHARLES: And I actually remember, we upgraded to that version. It's actually way, way, way better. BEX: I'm glad you feel that way. CHARLES: I didn't know the story behind there. I was like, "Oh, I just got a lot of... Awesome," you know? Although I don't know what's the costing. BEX: Yeah, I think that integration is actually free, so that wasn't the best example. I think it's for open source projects, at the very least. BRIAN: Brandon, one of the maintainers for the Slack integration and work at GitHub, also did a really cool talk at the SlackDev Conference a couple of weeks ago, so if you're interested what were the behind the scenes. That integration is all open source as well, so if you have request or you have features that you would like to add to the Slack integration, you can pop into the repo that hopefully will show up on the show notes because I'm not sure if it's like GitHub/Slack, but I guess we'll find that out in the show notes later on. BEX: It's Integration/Slack. BRIAN: But for an example of a paid app of a non-third party, we're not talking like Travis or Circle or another one with the big names but rather, a solo dev created. It's Pull Reminders, which is on the marketplace as of today and essentially, this gives you reminders of your pull quest, so you can actually ping inside the comments and tell Pull Reminders to say, "Tell me about the pull request like next week because it's Friday and I don't have time to look at this." ROBERT: That's awesome. I've also seen the one that's kind of related, that is like you can set your out of office at GitHub, which is actually kind of a neat concept. BEX: Was that the one where we are already changing that profile photos to have the overlay or the one where is just auto-replying to messages because I've seen a couple of -- ROBERT: I think, it's just auto-replies. BEX: Okay. CHARLES: So, it can change like your profile pictures and really, not just related to repo and history related activities but everything? BEX: Anything that you can access via the GitHub API, you can almost access via GitHub apps. There's a list of end points that I specifically enable for GitHub apps because there's something such as delete a repository that there's basically, a very few circumstances under which you want to give that permission to an app. Also, to things very specific like your profile or your personal page. About a year ago, there was an official internal audit of all of the API endpoints because there are lots of inconsistencies over what was and what wasn't enabled for GitHub apps, so they went there and kind of decided, what endpoints should be enabled and what endpoints actually get enabled. Now, that list is much longer than it was a year ago. Now, it's much more comprehensive. ROBERT: That's awesome and is this for the Rest API and the GraphQL API? BEX: Yes. Probot does support both. The Rest API is the one that specifically had all of these endpoints audited. The GraphQL, since it's a bit newer, we sort of built those and more. ROBERT: Cool. I really like working with the GraphQL API with GitHub. It makes it easier than trying to do a bunch of Rest calls. BRIAN: Yeah, there's a community form, it's like a discourse form that the API team actually manages and sort of pipes in there. Again, going back to like, if there's not something in the Slack integration that you would like to have, the form, that community is actually in there, if there's something not in the GraphQL API, that you would like to see. No promises on shipping it within an x amount of time but if enough people are requesting it obviously, there's going to be some resources [inaudible] at. ROBERT: What do you mean? We're doing open source. It has to be done yesterday. BRIAN: Yeah, exactly. And that form is at Platform.GitHub.Community, just a URL to get there. ROBERT: Awesome, that will be helpful to look through and get some recommendations in there. One of my favorite things I was going to say about the new integration for Slack and GitHub is the fact that I can highlight line numbers, paste that linked in and then it just expands it and the chat in Slack. That is so nice and I use it all the time. BEX: Yeah, I love that they built that feature. Actually, the original feature that was built on GitHub to allow those line expansions in the first place, like on GitHub itself, was actually built last summer by some folks who were also a part of my intern class at GitHub last year. ROBERT: Hey, intern power. That's awesome. BEX: Yeah. ROBERT: Everyone there is doing amazing work. I'm also following along with somebody that is also an intern and it's building a weekly digest program. BEX: Oh, yeah. That's actually a Google Summer of Code student. ROBERT: Oh, interesting. BEX: So, being sponsored through Google Summer of Code by Probot as an open source support. ROBERT: Is there anything more to unpack there? That sounds really interesting. BEX: Essentially, we submitted an application for Google Summer of Code because we thought it'd be a cool way to get more people, more students, a mentorship opportunity for the maintainers, basically and we were honestly overwhelmed. We got like almost 100 applications and it ended up being a huge of a deal but we're -- ROBERT: That's a great problem. BEX: Yeah, definitely a good problem but we were really happy. We, initially wanted to accept more students but Google limited us to only two students, so we have two Google Summer of Code students working on projects and one team of women from Rails Girls Summer of Code working on Probot. ROBERT: That would be awesome. What do they working on? BEX: I'm not sure yet. They actually just started a couple of days ago but the other Google Summer of Code student is working on a background checks API to eventually do sentiment analysis of comment history of someone new to your repository. ROBERT: That's interesting. That sounds like there will be some machine learning in there. I might just throwing out buzzwords? BEX: Most likely, I think they're just using some sentiment analysis API, like the perspective API. I don't think they're actually doing that themselves. ROBERT: Okay. CHARLES: Actually, I have a couple questions. Back on the subject of Probot. How does this square with the classic mode of integration because there was a lot out there? I think the first one that I remember that stuck in my mind was like Travis and I don't know if there had to be like a special relationship between the Travis developers and the GitHub developers, that's like, they was able to make that integration happen so many years ago. I don't know how that happened. I just remember it popped up and I was like, "Woah. This is incredible," and we see kind of the integrations gets more and more rich. For someone who's got, like you mentioned a couple of the big names, is the idea that eventually those would be able to be completely supported is GitHub apps or is it they're always going to be kind of a separate track for kind of the really deep integrations? BRIAN: I wasn't around when Travis first integrated with Lyft GitHub and I think that's a really cool integration and I know they have a very nice sized team that's able to do that. I think if we zoom back out like Probot, the way to get started with Probot is that we have the CLI command, which is to create Probot app. I believe it was intentionally copied off of create React app and the cool thing about create React app and create Probot app is that they abstract all the ceremony and boilerplate to get started really quickly. It was like, what developers or smaller teams can get started with integrating with GitHub apps. I highly doubt that Travis is going to rewrite their entire application with something like create Probot app but they're definitely going to be moving towards the new API calls, which would have been like GitHub apps. Part of the Checks API that we had launched at the end of May, Travis had blog post on how their integration with the Checks API works. They're making, though they have a lot of what Legacy endpoints and a lot of Legacy integrations in the way they integrate with GitHub, they are actively moving towards a GitHub app. I don't know if I could actually comment on their status of where they are today, to be honest but actively, we want all new apps and new integrations to follow the model of being a GitHub app, so that way, out of the box, you have access to all the newer features. You have all the access to all the newer GraphQL endpoints, if you want to use GraphQL and that way, we can serve one market, as opposed to everybody who had a GitHub integration from five or six years ago, that was all piecemeal together and sort of duct tape, like we run move away from duct tape everything together. CHARLES: I see. BEX: I definitely agree that I don't think Travis is going to switch to using Probot anytime soon and I don't think most of the large companies will be doing that but I do think, there will be shift towards GitHub apps in general. For those companies that don't already have the buildings of the GitHub app started, I think that Probot could be, in time to free some of them. BRIAN: In addition to that too, Travis and Circle and all the CI integrations, they're doing a really good job. I think the cool thing about GitHub apps is what you take away all that ceremony of getting your checks to work, now we can start opening up the door of like what's the next sort of CICD thing like? There's another term or another, I guess category of applications that can now be built to improve GitHub. CHARLES: The most amazing thing about having a great platform is the apps that you don't foresee, like it just come completely out of left field and you're like, "Woah. I can't believe that's actually a possibility now." When you have started to see some of those, some Probot or GitHub apps, you're like, "Man, I didn't see that coming. That's awesome." BEX: A hundred percent. I think it's the most exciting part of Probot because I think GitHub as a platform, we all know GitHub is the largest developer platform in the world and I think the idea that developers can build on top of this platform is the most exciting idea right now. I have honestly already seen apps that really excites me. The other day, I saw this app that was definitely not near completion but it was essentially updating and issue a comment box over and over and taking response through like checking a box and then listening on that common edit, in order to specify your coffee order. ROBERT: Woah. BEX: I was like, "Do you want an ice coffee or regular? Do you want milk or sugar and cream?" and it was going one at a time. It didn't actually order you your coffee at the end but it was super exciting to watch that. You're just editing the comment. I had never seen that before. ROBERT: That's pretty slick and that's taking the API pretty far. I'm sure there were some parsing in there and each Webhook response are like, "Was this box edited or not." That interesting. CHARLES: Yeah. Actually, now that we're having this discussion is kind of like changing my mind a little bit. Robert and I were actually talking yesterday about trying to standardize on our release management and our plan was basically to have some software that was going to run inside of our CI provider and have kind of a shared library, just a little ntm package that was shared by all of our repos but I'm thinking now, man, we should really explore doing this as a GitHub app. ROBERT: Yes, please. I've had three ideas that I really want to build out as a Probot. I'm just going to list them off and then we can build them all together and take equity and you know. I'm kidding. But the two that really excite me, that I kind of want to do is one concept that we work on this open source project for our clients and if somebody from the outside that doesn't have commit bits to be able to push to master, it would be really cool if we had a Probot that after it had an approved on the PR, from the maintainer, that the person that open the PR could then tell a Probot say, "This is approved by somebody that manages this project. Can we merge?" and then the Probot would then actually merge. I don't know if that's possible. That's something that I definitely wanted to explore. Then the other one, which is less cool, would just be like if we have a couple branches on some of our projects that we want to continue and we're not ready to put it back into master but we want to continuously run the test suite against it, so the idea there would be to have a Probot that would watch for changes on master and rebase as needed and continue to run the test suite and see where you're at. Those are the two things that I'm really excited about to do with Probot but I just want to automate everything with GitHub now. CHARLES: Right. BEX: Yeah, definitely, that first idea was actually pretty viable. I'm curious to know like how you actually get those commit links -- is that what you called it? ROBERT: Commit bits are more like commit permissions, I guess. BEX: Oh, I see. ROBERT: An outside contributor. CHARLES: Yeah, we want to push responsibility to the person who is the maintainer who can approve it but actually, the way we do it at Frontside is the person who actually is making the change is responsible for merging it. Once you get approval, you still have to hit the go button and that's just going to make sure that you're taking responsibility for saying it's done but that doesn't work for open source because people coming off the internet are going to have the right to push but we would like to give it to them, maybe via an app, if there is a maintainer who's approved it. BEX: Yeah. That's definitely something you can do. I've seen quite a few apps that, essentially add outside collaborators to the repo. Are you familiar with the... I forgot what it is called, like the all contributor section, where you cite everyone in your repo and everything and who's worked on it. There was a GitHub app that would add someone automatically after they merge their first change. CHARLES: That's awesome. ROBERT: I may have seen that on React State Museum but I'm not sure. It's a repo that we've contributed to and it has all the contributors at the bottom. It seemingly just kind of popped up there. BRIAN: There's an app that, I would like to mention too that I'm pretty excited about, that it sounds trivial too and it's almost similar... Not similar but it's sort of related to what you were talking about, Rob, with your first app, which is the WIP bot, which is the work-in-progress bot. This is a pattern of whenever I open a PR and I might not ready for a merge but I want to share my code so I can get feedback earlier on, I'll type in WIP so that append to my title of my PR. What this engineer did was every time you do WIP, it's going to go into the GitHub API and actually block the PR for merging, which is a feature available to GitHub. It's nested in your settings but the cool thing about this it actually blocks the PR for merging, so you don't have to worry about getting your, sort of like show and tell code merging the master without being ready. ROBERT: That's one of the first bots that I installed on all of our repos and then you can correct me if I'm wrong, it didn't always have the ability to block the PR from being merged but with the new Checks API, is that something that was introduced? BEX: Not exactly. The way that blocking of merging works is if you set it as the required status, so you can install any sort of CI on your account and have it not being required and ignore it whenever you feel like it, so it's really up to you to make it required. Otherwise, it just isn't checked and that's true for anyone who uses the Statuses or the Checks API. ROBERT: Okay, so that's a Statuses API. Okay, sorry. BEX: Yes. ROBERT: Also, the cool thing about that that I noticed when that was rolled out was I was now able to pick and choose and use workflows on Circle CI and each workflow is broken out as a different status check. I am now required like linting and the build and the test have to pass for these browsers before it can merge, which is really cool to be able to pick and choose. BEX: Yeah. It's awesome. I know personally on some of my repos, I have a few checks that I just don't require because I know I have to make them pass. ROBERT: Yeah. Speaking specifically about the work-in-progress bot, do you know how that works? It's open source, so I am sure I can go look. I think we want to go make a PR. We had some back and forth about this, Charles. CHARLES: I actually just [inaudible] we disagree. ROBERT: Yes. Charles opened a PR and one of his first commits in the PR had work in progress and the title had work in progress and we have this this Probot on our website and it was a blog post. You know, you make a couple more commits and you're further down, you move the work in progress in the title but the PR were still blocked because the first commit on a PR have work in progress in it. I think if it's the most recent commit or if it's in your PR title with work in progress, it should block but otherwise, it should not and Charles feels differently. CHARLES: I have about six commits and the very first one have WIP in the title or in the commit message and it blocked the whole thing but I kind of felt like it actually made me go back and I had to squash it down to two commits because I actually feel that your commit history should tell the story of the development, not like it should an absolute one-to-one journal of what happens but what you are intending. I actually felt that it could help me out because there's six commits that we're kind of all over the place and just kind of slapdash together have made me kind of go back, rethink it and tell a coherent story. I think it did me a service but it was not obvious. I definitely agree with that but I was like, "Why? Why were you still blocking?" ROBERT: Do I really [inaudible] admin privileges? BEX: I would say, I am friends with the creator of the web app. His name is Gregory Mantis and he is actually got a huge work in progress PR shifting work in progress over to using the Checks API and one of the features that he's using with the Checks API is essentially this mark as now work in progress button that will add the special line, like feel free to merge or something like that into your original PR description at the bottom. If that is there, the work in progress app will no longer be blocking. It's essentially like a hard override and honestly, that's the power at the Checks API versus the Statuses API. That's really exciting. ROBERT: Because I have seen the work in progress bot to get into a weird state, where I did remove the work in progress from the title but it didn't quite update and I'm still blocked. It's okay for me because I have admin privileges but other people on the team maybe not and they might be blocked from something that's actually work in progress. It's a lot like that hard override will be probably pretty helpful. BEX: Yeah, definitely. I think sometimes, there's some confusion with that just because of the way what perks work on GitHub and the way our pages are rendered, that you may need to refresh the page before you actually see it take effect. ROBERT: Right, yeah. Overall though, I love that bot. I go weekly, probably to the Probot apps listing and just go shopping. BEX: Wow. I'm actually the person who approves all the Probot apps to the listings so that's pretty motivating there. ROBERT: It's really nice. I am not even joking when I say shopping, I go through and I open up a bunch of tabs, I read through them, "Oh, this could be useful," that kind of thing. BEX: The first app you mentioned, which was like the one that requests more info is actually one that I built, so that was kind of funny. I guess you got that from the Probot apps too. ROBERT: Yup. That one, we definitely use on a couple of our organizations and repos. It has yelled at me a couple of times because of a blank PR. BEX: It yells at me all the time. I think I get yelled at more than people who are actually doing it wrong. ROBERT: I'm a little embarrassed like, "I should do better. I need to set an example." BEX: Definitely. ROBERT: Cool. I'm curious what both of your favorite Probot app is. This ought to be interesting. BRIAN: The app that I'm really impressed with so far, that I actually only use on a junk project at the moment, is the weekly digest one and it's mainly because I built something for this in my previous role at the company but then we shift it, which is basically go through every single repo. I worked at a company called Netlify previously and we had way too many repos to maintain... Oh, sorry, to keep track of and I was moving further and further away from the backend at the time so I was unable to keep up to date with all that was changing. I built a Lambda to watch Webhooks and then give me a digest of what was shipped like issues and PRs closed. It was way over-engineered and I never actually shipped that to actually make it work. But then the weekly digesting came out maybe a couple of weeks ago and it blew me away because I was like, "This is exactly what I needed," and I was trying to make it overly complicated through like a Lambda and like a bunch of Webhooks and this person, with only a few weeks, has the scaffolding of what I needed. That's the one thing I'm pretty excited about. It was already mentioned earlier too, as well. BEX: I guess, I would say one of my favorite ones is the unfurl a link app. I think that one it so simple but so nice. I don't know. I think having that unfurl link preview is just beautiful. Essentially what it does is it listens on issue comment creation or pull request comment creation or issues your pull request or whatever and read through the text or whatever was that issue or pull request and looks for links and then, essentially unfurls them so you can get a really nice preview of what you're going to. I think that's really beautiful and just so simple. ROBERT: Yeah. I love that one too. I have that added to all of our repos. BEX: It's so much nicer. Why would you not unfurl your links when you could unfurl your links? ROBERT: Exactly. CHARLES: I actually have a question. I think it's been touched on, probably at least twice throughout the conversation. I want to actually create a Probot, how do I actually go about deploying it? What does that look like? What does it look like to deploy and maintain it? BEX: We have a page on our docs about deployment and essentially the TL;DR is you can deploy it on any normal cloud hosting service that you wanted to deploy it. There are a few things you need to specify. For example, GitHub gives you a private key that you need to create your JWT and that private key means to be passed into your hosting service however you do that and then, there's a few bits of information that need to be pass in. We have pretty intense docs about it. Honestly, I'm not a deployment person. I usually try to let other people do that and I have never had a problem going through our docs and just getting it working immediately. BRIAN: It's also mentioned that there are examples like Heroku and Now and a couple of other ones. If you have a service that you already like, it's possible it's already in the docs, like steps to how to get that deployed. BEX: Yup and any other services are more than welcome to be added to the docs. Pull request are welcome. ROBERT: Sweet. It sounds like we need to set up a hack date to create a Probot, Charles. CHARLES: Seriously, my mind is brewing. ROBERT: I guess it's not directly related to GraphQL but there's something that I've always wanted to build. For prior history to everybody [inaudible], then the podcast, Brian and I used to work at a company called IZEA and one of the things that we built and I worked on a lot was we would create a collect metrics on people's social accounts that they're connected and do that and graph it over time. This idea came from when I was building up that feature all the way back in 2013, I want to graph the change in GitHub stars. Is there an API available for me to see like weekly GitHub stars or is that something that I still have to manually store and track? BEX: There's definitely an API endpoint to get the amount of stars and I don't see why you couldn't just do that on weekly basis and compare but I don't think there's any track that change API. ROBERT: Gotcha, like a history of it. I could do this by just stealing and looking at what the weekly digest Probot is doing because there is a change in stars section in there. I was just curious if there was now an API that was available. BRIAN: Yeah, that's more unlikely. I'm going to say no without looking at all the reference documentation. I think as far as that database, it's something you'd probably have to collect on your own but it's also a good candidate for a GitHub app, where you build a service that you can actually track stars once you've installed it and then if you want to monetize it, you can actually pay for private repo or whatever stuff like that, if you wanted to. But it sounds like a great opportunity to see this in the GitHub/Probot listings. BEX: I actually just look this app really quick in our docs because I was curious but apparently, you can receive the star creation timestamps. That could be doable through timestamp usage. ROBERT: Oh, and then I just kind of loop through back and build your graph in there. BEX: Yeah. ROBERT: Interesting. All right. Well, [inaudible] I was going to do today. BEX: Yeah. But I think it's exciting to bot the weekly digest and then what you could extract from that into stargazing is that Probot scheduler, which is essentially this all Probot extension we made that triggers a Webhook on a scheduled time period because right now, the way GitHub apps works are so centered around Webhooks. It can be difficult to find a way to trigger an action on something outside of a Webhook, like on a schedule basis. ROBERT: Yeah, that would be really helpful. I can definitely see how that would be a problem, if it's very, very central to reacting to Webhooks and events that happen on the system. BEX: Exactly. ROBERT: You're just hoping that somebody comes through and creates an event at a specific time. CHARLES: Can I ask you a question about, it's definitely on topic of extending GitHub but currently, just a question about, where the line is between what you can and cannot extend? You mentioned, for example in the rewrite of the WIP bot, being able to throw out a big button that says override this merge. Are there any plans to be able to actually extend the UI in novel ways? Everything there right now is happening with API calls, with I assume, UI elements that are related but the UI elements are static. If someone wants to put a novel piece of the UI, that button is going to require an extension of the GitHub UI by GitHub itself. Are there any plans to be able to, I know it's a dangerous waters, perhaps at a limited fashion at first but maybe more so, add different interactions and the actual application. BEX: I think this is actually the most exciting future of GitHub as a platform. In the past, GitHub APIs have only specifically supporting things that you can do through the command line or you can do through GitHub's UI itself. The Checks API introduced the very first non-integration specific UI element essentially and the merge button that I was referring to in WIP is exactly that. It's essentially this button that you can change the text of it to be whatever you want and you can listen on that action and then you can do as an integration or an app, anything that you want based on that. I think that's the most exciting direction for GitHub. Because if you look at Slack, Slack is a platform that has sort of really impressive integrations in that response. Your apps on Slack can really do all of these things, use custom UI elements, so I think the most exciting features for GitHub as a platform is all of this customization and giving the power to the apps. ROBERT: Yeah, that sounds an awesome way to be able to extend GitHub without having to try and throw the feature on to GitHub developers. BEX: Exactly. I feel that a lot of the struggle right now is that there aren't these nice ways of communicating via apps because I feel lot of the apps and bots end up just commenting on issues and pull requests and taking up a ton of screen real estate as a result and I just think that that's not the way that bot should ideally interact with the GitHub platform. They should have their own space to exist and that's the feature I'm most excited for. CHARLES: Yeah. I can think of having like progress bars for CI checks and your various appointments. It's too exciting. I'm glad. That's definitely the response I was hoping to hear. BEX: Yeah. We're excited for it too. ROBERT: Basically, you all have a massive community of a bunch of developers that would want to do this and are willing to get their hands dirty on it. Enabling that community is probably the root of all Probot is about. That's super awesome. BEX: Yup. CHARLES: That's a good place to end, because gosh, it's going to be so exciting to have the millions of developers on the planet, just like surgeon to the APIs that you're developing. BRIAN: One thing to add to that too, about the whole million developers, there's a number that's been thrown out from Stack Overflow and also, some other people who are saying like there's 50 million developers, there's 24 million developers. As far as GitHub, our public user number is 28 million, the cool thing about Probot and GitHub apps is that there's a good chance that all those people that are using GitHub today are not actually developers. They're like PMs or designers and what's really cool about this, like having interactions with that kind of platform in this way is that you can now enable all the non-developers to be able to interact with your GitHub repos and start bringing more designers and PMs onto to the GitHub platform to interact with the developers. ROBERT: That is an interesting point. That is awesome and something that I'm always looking for is a different ways to collaborate with non-developers on my team because... I don't know, developers tend to think everything is always centered around code but it's not. The shifting at work that are awesome, needs a lot of collaboration from non-devs and non-dev skills. That would be really interesting to see. I'm excited for that to play out. BRIAN: Yeah. There's a blog post that was published a month ago, I think about where the design team, design system teams rather, built the integration to Figma to update their icons effectively. I just posted that in the chat to look into but they also built this as a Probot app as well. ROBERT: That is awesome. BEX: Yeah, that one is super exciting. You would have the app comment, the diff between what the old icon versus what the new icon look like and it's just such a beautiful design change to be able to see that shift. ROBERT: Man, I'm happy that this is happening. The future seems super bright. Where can we direct people to get resources to contribute, to get involved and start really going at this? BEX: Basically, Probot.GitHub.io has all the Probot stuff, /app has all the listings for apps you can install today, /docs is where the docs are, if you want to get started and hopefully from there, we link up to the necessary things that you need to do. BRIAN: Also, what I mentioned too via Probot Slack channel, there's a Slack channel as well and they do a weekly call. I think, it's weekly or bi-weekly call to actually chat with the Probot community. If you have questions, you can actually bring your questions to the team. BEX: Yeah, we call it 'Office Hours' and it's once a week and it's under our community page, where we also have a link to our Slack. We have a link to another podcast we run and basically, how to get involved in the Probot community. ROBERT: Those are really helpful resources. I do remember seeing that Office Hours. It's on Thursdays, right? BEX: Yes. ROBERT: I was going to drop in for one and then, I actually forgot. Actually, it might be going on as we talk right now in this podcast. BEX: It starts in half an hour, I think. ROBERT: That's awesome. Cool. Well, thank you Brian and Bex for having a conversation about Probot. This is really awesome. Is there anything that you would like to plug for yourselves? How people can get in contact with you? BRIAN: Yeah, I am BdougieYO on Twitter. Everything you need to know about me is there and I am happy to say hello. I'm also helping with the GitHub developer program, which is sort of getting a soon-to-be announced rebranding. If you go to Develop.GitHub.com/Program and you want to have more conversation about the API and GitHub apps on the GitHub side, you can go there to sign up. BEX: And I am HiImBexo on Twitter. You can ping me in any Probot stuff. I'd be happy to look at any Probot code. I've been looking at it for a while now so I'm happy to do that. ROBERT: That's awesome. Thank you all for having a conversation with us. This was really fun. I'm so excited about everything you can do with Probot. This is a really fun project. I'm happy that this is happening and I will make a Probot in the future. CHARLES: I'm looking forward too. Robert has been excited for quite some time and he definitely talks a lot about it and now, I have some insight as to what -- ROBERT: It's happening, I'm telling you. Well. Thank you for being here and we are the Frontside. We build UI that you can stake your future on. We are specializing in JavaScript. We can build anything that you want throw at us. We do functional programming, React testing, Vue, anything in JavaScript, we specialize in. As always if you want to suggest anything for us to have on the podcast or talk about, you can reach out to us at Contact@Frontside.io and like I teased earlier in the podcast, next episode is going to be all about Microstates, the immutable and functional state container, composable model system that we've been building, it's controls as a brainchild for the past two years. That is next episode and I'm really excited about that. It's a really fun API and expressive to build models with. Thank you, Mandy for producing our podcast and we'll see you next episode.

In this episode, Harry Kaplanian of EBSCO joins the show to talk about the FOLIO project: a community collaboration to develop an open source Library Services Platform (LSP) designed for innovation. He discusses the why behind the the decision made to embark on this project, the benefits it brings to the market, and what makes it different from current offerings and other open source projects. Harry also shares where he sees FOLIO going in the future, challenging and unexpected outcomes of making the choice to go open source, and advice for other businesses that are considering embarking on a similar journey today. Do you have opinions on this show? Want to hear about a specific topic in the future? Reach out to us at contact@frontside.io or on Twitter at @thefrontside. This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. TRANSCRIPT CHARLES: Hello everybody and welcome to The Frontside Podcast Episode 102. My name is Charles Lowell. I'm a founder and developer here at The Frontside and I'm going to be today's host. So today, we're going to be talking about the business of open source with Harry Kaplanian. Harry's someone that we've had the pleasure of working with over the past year and collaborating with on a very unique project, which we are going to get into later. With me is Robert DeLuca. Hey, Robert. ROBERT: Hello. CHARLES: And before we start out today, we just wanted to take care of some quick news. Just so you know, we released BigTest React. This is actually something that grew out of the work that we've been doing with Harry. It's a set of React helpers to acceptance test your React applications so that you can make sure that you know that your application is actually working on actual browsers in actual life. So, that's exciting. You can go check it out at BigTestJS.io. And with that said, let's get on with the project. So first if all, welcome, Harry. HARRY: Thank you. CHARLES: I teased this in the intro, but we'll go ahead and say it outright. The project that we've been working on with you and with EBSCO is a project called FOLIO. And it's one of the more unique projects we've ever had the pleasure of working on. So, maybe you could just give us a brief overview of what exactly FOLIO is and how it came to be. HARRY: Sure. So, FOLIO I think can best be described as an open source platform for services. Our focus has always been the library market and so, currently this platform exists in the library space. That said, there's nothing about this platform that is library-specific. It can actually be used anywhere, anytime, for anything. The platform itself, all communication occurs basically via HTTP. It's microservice-based. It's language-agnostic. And it is really the core of what we're building. However of course, a platform on its own is not very useful. And at least for right now, libraries are overall expecting to get to a point in terms of features, functionality, apps, and services that are being built. So FOLIO can actually take over the enterprise-wide day-to-day operations of what it takes to actually run a library. CHARLES: And these are libraries like university libraries, the Library of Congress, the library at Alexandria. I mean really, when you think of library, like library [3:00 inaudible]. HARRY: Yes. However, we decided to focus on a particular market, at least initially or I guess a subset of the library market. And so, that subset we've actually chosen to focus on is the academic library market. And so, really mainly colleges and universities all around the world. That said, there is no reason why others can't use this software. And it is all open source. So, if any changes are ever needed, modifications or additional applications are needed to support those additional markets, that can certainly happen at any time. CHARLES: Historically, it's a very complex problem, right? There is a lot that goes on at these libraries. But historically, the software that's driving it has not been open source. HARRY: That is correct. So, it's sort of an interesting history. And I guess you're leading into what I think was maybe your second question [Chuckles] which was: What are the circumstances or the thinking that lead to the creation of a project such as this? And really, libraries have been pretty much operating the same way, to be honest, for probably about the last 40 years. And I'm sure we either all know or have heard, most libraries are facing shrinking budgets. They're really challenged in terms of space and storage requirements. And just in general, users have changed dramatically in the last 40 years, the last 20 years, maybe even the last 10 years. Most people don't really ever expect to walk into a library and actually conduct research anymore. They expect that to happen elsewhere. And so, in many ways, they're really facing what amounts to Google and others, because that's where people expect to go. The vendors that typically created the systems and software that supported these libraries, the day-to-day operations over the years, many of them have consolidated. And there are just actually very few options left. These are really old, monolithic systems. They were built initially with the best intent. And for the first couple of years, many features and functionalities were added. But over time, these systems grew to be so large. There really was never any one person anymore that fully understood it. They find themselves in situations where if they add a little feature here or modify something there, it breaks something somewhere else within the system. And so really, they've gotten to a point where libraries have generated lists of features and functionality that they need that are frankly years long. And the vendors just can't deliver any of it. They just don't have the ability anymore. The systems are too old. In addition, really they're sort of building these walled gardens where they really try to take over all the day-to-day operations of the library but are actually starting to try and expand out in other areas of the university as well. And as they do this, they really tend to lock the system down. They're not really open to integration with other systems or other vendors. And they really see it as they are the one company you should go to that can completely manage and operate everything that goes on in your library. And when we talk about innovation in these systems, really they've been operating the same way, as I mentioned earlier, for the last 40 years. Really, the only major, major innovations is there's maybe one company now that's offering these services on the cloud. One size fits all. And the real issue there is, all of these libraries we talked to, all of them see themselves as being unique in some certain and specific way or sometimes multiple ways. All of them believe they're bringing something special to the world in terms of knowledge, research, understanding and learning. And they tend to cater or specialize often to the institution they're a part of. And so for example, if a library belongs to a medical school, of course their collections are tailored that way. A lot of their workflows and operations tailor to students that conduct research in that field. And oftentimes, even to a hospital that may be associated as well. And this will oftentimes be very different from what might exist for example in an engineering library. And so, for most of them, they look around at what's available to them. Not only has nothing changed, not only has there been no innovation, but on top of it, they're forced to choose essentially a single system. And that's not really getting them what they want or where they want to be. So I, as well as some others, spent quite a bit of time looking around at the market and trying to assess and understand all of this. And one of the things we felt was, as we look at all of this, really we're living in a world, these libraries are living in a world of closed systems, lack of transparency, lack of change, lack of innovation. How do we change that? How do we spark that innovation? And it seemed like what we really need is some sort of an ecosystem, a sustainable ecosystem or really, openness and transparency. And it seems like the only way to get that really is open source. And in fact, we actually conducted a survey and out of that survey, 100% of the libraries we spoke to really believe in open source. And when you think about it… ROBERT: Whoa. HARRY: Open source really in many ways aligns itself really nicely with the mission of the library, right? The library is there to not only promote learning but to make sure information and knowledge that normally you may not have access to is accessible to everyone. They're really leveling the playing field. And that really is a really nice analogy for what open source is doing in the marketplace as well. And so, this seemed like really, and at least, an initial ideal path to head down. And so, we did. CHARLES: Yeah. And so, you did. And I'm curious to explore more about that decisions. Because – now this is a little bit of a dated quote – but Steve Balmer, when he was the head of Microsoft, called open source communistic and un-American, then how do you reconcile that? Because obviously you represent a business that ultimately needs to be profitable and is engaging in this in order to make a profit. And so, where does the balance lie and how do you reconcile that? And obviously you all perceive this as a strong business move. And so, yeah, how do you balance those two concerns? HARRY: I think from our perspective, we looked at it as if a single vendor or maybe a group of two or three vendors end up locking up the entire market with systems that operate the entire library that are not open and they're not willing to integrate with other systems and services that exist out there, in essence that's an attack on what is the basis of our business as a company. We provide services and we provide content that libraries use and ingest and make available to their users. And essentially, it looks like they're trying to lock us out. And how does that really help in terms of creating an open and fair marketplace where people get to make choice? It doesn't. And so, what can we do to disrupt that? And building a system that allows everyone to play at least puts us on a level playing field where if an organization or a library chooses to adopt it, we're now back in a very competitive situation where really, the vendor or the organization or the individual that provides the services that best fit that library's needs wins. And that's what we really want to happen. And so, that's in essence how we win. We believe we offer better services than most others in the industry, if not everyone else in the industry. We think there's a sizable percentage of the market that truly believes that. And we just need to make sure that the mechanisms or the systems are in place that allow us to compete in that manner. And in the end, if what we end up with is a platform that allows 3, 5, 10, or 20 companies to actually compete in a reasonable manner fairly, really it's the customer in the end that wins. It's these libraries and it's the users and everyone else conducting research. ROBERT: So, I'm still kind of reeling from 100% of libraries said that they would like to have an open source platform. How many did you survey? HARRY: So to be truthful, it wasn't a large number. ROBERT: That's still insane to me. [Laughs] HARRY: Right. We focused on, however, surveying different segments of I guess really representation within the library. So, the people that do the work, the middle managers, and then really, the folks at the director level that are really responsible for running and organizing a library. ROBERT: The ones that will experience that pain of being locked in, right? HARRY: Right, right. And trying to get decision-makers at many different levels. And we tried to also cover both very small libraries to very large, enormous libraries as well. And all of them saw a benefit. One of the pieces where things sort of maybe went in the other direction as far as the statistics are concerned, if you ask them though “How many are willing to adopt open source?” the numbers aren't quite the same. In fact, they're rather dramatic. And so, at that point it's roughly about 33% said they'd be willing to adopt. CHARLES: So, why is that? HARRY: So, we asked. And interesting question. And what most of them came back and stated was, “Well, we love the idea of open source. We fully support it. We're willing to do what we can to help it and make sure this happens. However, we ourselves don't believe we've got the staffing, the resourcing, or the knowledge to host this ourselves.” And so, this leads to another really interesting thought or idea as far as open source is concerned, is so this really implies that there really needs to be sort of an ecosystem of organizations that are providing these types of services. And to be honest, the faster we can make that happen, the better off the market is. Libraries are now getting to choose vendors that they get to get their services from, which is rather interesting. Because for the first time, not only do they get to choose the vendor but if they're not happy with the vendor, they should be able to switch vendors but keep the software and the data the same. Today, in the library market, if a library chooses to switch vendors, that means essentially they're completely starting over in their library. They completely have to migrate to a fully new system which means enormous amounts of training, enormous amounts of data transfer, and all the pain that goes along with it. I had a librarian tell me once, “The next time we migrate to a fully new system, I either want to die or retire.” I just want [14:30 inaudible]. ROBERT: Man, that sounds painful. HARRY: That's how painful it was. CHARLES: Yeah, wow. Because the complexity intrinsic to these systems, it does, it boggles the mind sometimes. Just even the little slices of the ecosystem in which we're operating, there's just a lot at play there. ROBERT: Yeah. Like trying to work with other vendors and be agnostic so anybody can adopt these things that we're building. And since we had that vendor lock-in, it's hard for you to adapt the two systems, because they were built in complete silos. So, trying to merge that is an interesting challenge. HARRY: And I think everything I'm saying here, I'm talking about it from the library perspective. But if you look at other industries, I don't think it's really any different. If you look at any of the enterprise-wide management systems that are used in corporate organizations today, most of these corporate entities will stick with the system they chose for a very long time. Because it's incredibly painful, it's disruptive, and it's expensive to switch from one system to another. And so, the idea of you're not happy with a vendor and hey, I can go to somewhere else and maybe keep the system I have in place is really kind of interesting. And even better, if the vendor's not able to provide some of those features that I need but for the most part I like the system, wouldn't it be great if I can hire someone to do that work and to have those pieces built for me? And those are some of the things that open source offers. CHARLES: Right, right. So now, I understand that for example to kind of put a concrete bow on this, [Chuckles] to move away from abstraction of what it means and provide an example, so we have this open source platform, this FOLIO platform. Which if I'm a library, I could in-house, I could either use it on premises on my own servers or I could run them on AWS or I could run them on Google Cloud – there are all these different deployment options and some libraries are doing that, but that's something that they might not want to do. So, I understand that EBSCO is actually going to be for example one of the services that y'all are going to be offering around FOLIO, is actually hosting the platform so that the overhead of managing all of the servers and provisioning accounts and doing all that stuff is going to be taken care of for you. So, I can just sign up. HARRY: Yes. So, that really fed into the results of the survey. For those libraries or organizations that felt they couldn't do it themselves, where would they go? And who could provide those services? So, we're absolutely going into the business of providing these services for libraries that choose to. But at the same time, we're really strongly encouraging other vendors and organizations to provide these services as well. Because for instance, you may represent a series of libraries in let's say Hungary. You may have a vendor there you enjoy working with in the past. It's provided great services for you. There really ought to be no reason why you should not choose that vendor to provide those services for you. And again, it's an open system. There's nothing there that's excluding me and the organization I work for to provide additional services, to provide content or what have you. It's an open system. And so, we should strongly encourage that. And the reality is, if we're going to have a disruption in the market, it seems we can either have one organization or one company trying to usher this disruption along. But what if we could get 10? Or what if we could get 20? Or more companies out there ushering this disruption along simultaneously worldwide. It feels like we've got a lot more of a disruption happening in that case. And so, that's what we're strongly encouraging. CHARLES: Right. But at the same time, I can see it from if I were considering making a play like this, one of the things that would worry me is, “Okay, so we're going to disrupt the market by allowing a large tent where lots of vendors could compete.” But then, I immediately experience fear of, “Well then, how do I differentiate myself?” Because ultimately I want to be competitive. How do I make it so that I'm not a commodity in this new market? HARRY: So really, I think that's key for any vendor that chooses to compete in a market like this. Do you believe as an organization you have not only what's needed to compete but those key differentiators that at least a certain segment of the market would be interested in you and the services you offer? We believe we do. Another company that just recently announced that they're going to be supporting FOLIO and providing services around it is a company called ByWater Solutions. They've actually been in another segment of the library market for many years providing services to libraries as well. Longstanding, actually, with open source codebases. And so, this very much appeals to them. They believe… ROBERT: Oh, interesting. HARRY: They have a niche that they can provide and they are going to be doing that. And we strongly encourage it. What's again great about it is if they set up a library with FOLIO, they're not building the walled garden. They're building or providing the open platform that we can connect our services to. So in many ways, yeah there are some areas we're going to compete. But there are actually many areas that we're also going to be complementary. And I think that's what's really, really interesting to us. CHARLES: So, it sounds like you're also counting on just by having the ecosystem in place and having this sea as it were filling the ocean to enable trade, one of the bets too is that you're going to open lines of business that weren't even possible before. HARRY: Correct. So, another way to look at it – we've got our iPhones or Android devices. Apple built a platform is really what that phone is. When it was released, you could make voice calls. You could do some simple text messaging, some basic email, and I'm sure there are a couple other of things in there that I've forgotten. I do not believe that Apple had any idea that they'd actually be able to create a marketplace that had hundreds of thousands of people and vendors providing features and functionality for that platform. I am pretty sure no one at Apple had ever imagined the guitar tuner would be released. I don't know, a compass app, flashlight apps, whatever. You name it. And not only that, as a user of that platform, you're not happy with the standard email app? Well you know what? I can get a different one. In fact, I've got a choice of many and I can find the one that suits me best. And that's fine. It's an open marketplace. And so, with FOLIO today, we're in early stages. We're building out that platform. We're building out some of the basic functionality for that library that they need to operate as a library. But once this gets released, it's at that point when we believe the more interesting things are going to happen where we build a cataloging app for FOLIO. Well, we know already that there's other organizations that are starting to look at and think about, “You know what? We've got a whole other way or a whole other interest in terms of how we'd like to support those workflows on the FOLIO platform.” Or even better, librarians starting to think about, “Hey, if I've got this open platform just like Apple does, this means I now have the option to build features and functionality that take me where I want to be five years from now, 10 years from now. And I've never had that ability on the existing systems that we use today.” And so, one of the other I think advantages or benefits that something like FOLIO or a platform like this brings to the market is the ability to create that marketplace very much like Apple has done and Google has done with Android as well. ROBERT: So, we're pretty far in, but you mentioned FOLIO a couple of times. Do we want to unpack what FOLIO actually means? HARRY: Yes. FOLIO actually stands for the Future Of Libraries Is Open. ROBERT: Right there in the name. I love it. HARRY: Absolutely. CHARLES: So, we've talked about this platform. We've talked about this marketplace. We've talked about the business incentive of why you would want to do this, why it makes a lot of sense. Clearly, this takes an enormous investment. So, we think, I think, a lot of people think of open source as a bunch of wild developers running around. ROBERT: Pushing commits wherever they want. CHARLES: Pushing out code and doing stuff. And then it kind of organically grows into something that someone then picks up. ROBERT: And then you burn out. CHARLES: Yeah. [Chuckles] But that's I think a lot of people's experience with open source. But an initiative like this takes a staggering investment both in terms of capital resources but also in terms of will, in terms of management, and putting all these enormous forces in motion. There's developing the software, developing the awareness, and I think right now there are hundreds if not thousands of people working on this right now. How do you even go from zero to 6,000 like that? And I'm not talking about people. I'm talking about velocity. [Laughs] HARRY: Right. Early on, when we were conducting research, market research, one of the things we did, we spent some time looking at what we believed were successful open source projects. And I think what was interesting was in many cases it took a single individual or a company to actually create what amounts to that first piece, that first core building block that others could start to expand upon. And very often, they literally just created it themselves, made it available out there as open source, and basically told the world, “Here. Have at it. Have fun. Do with this whatever you wish.” So, we actually thought we'd like to do that. However, building a system on this scale is not that easy. Understanding the operations of a library day-to-day is not that simple. We need help. And so, what we decided early on was: could we kick off and start this project while at the same time [25:42 inaudible] that community as early as possible? To get people excited, interested, and to get this project in a place where we're not the only contributors, where there are many others contributing as well. And when I say ‘contribute' I don't necessarily mean software developers. But that is certainly one aspect of it. But one of the key pieces of building software is gaining access to subject matter experts as well. And it's absolutely key and critical. And so, our goal of course was to build this community, have that community start to provide that in-depth knowledge, those subject matter experts that we need so we can determine what it is we need to build, how we need to build it, and really go from there. And by including those people as early as possible, I think one of the things we find that happens with this project which is really sort of incredible is the excitement that starts to build. The word of mouth advertising, marketing, as librarians, libraries, vendors and individuals start to talk to others and really spread the word about this project. In many ways, it starts to compound or snowball and build on itself. But that was really challenging. And it was really actually kind of slow going in the beginning. Because we started with nothing. And one of the issues you face is, “Well, you want me to contribute my time to this project, yet I see nothing.” Right now it's just a dream. And so, one of those early, early issues was, “Can we build a team small enough, focused enough, where we can start to build some of the basic, basic, basic core pieces, to really prove to the world that this project is real? This project is actually moving forward and this project is actually delivering.” And now that we're in a state where people can actually see working software, the excitement is just starting to expand and compound rapidly. And we see it everywhere. CHARLES: So, do you find libraries or people in the target market, there's that key point where they start to feel empowered? It's real and then the thing that I think that you're going for is to have the realization dawn that not only is it real, I can put my hands on the wheel and I could control my destiny. And I can contribute now because there's this critical mass. ROBERT: You have the power to make change, which never existed in the library world before. HARRY: Or at least, not in this specific area. Because to be truthful, I think libraries as a whole believe they're making change around the world all the time. But it's really related to content. But this is actually making changes in the actual operations of the library. And they are actually empowered to get involved, to contribute, and to help us get this done. And it seems to really resonate with folks. Because it's something they haven't been able to do previously. And it seems genuinely exciting. And we announced this really, well the market learned that this was happening I'd say a little over two years ago. Maybe more like two and a half years ago. And it was sort of a gentle drip in terms of a roll out. People started to learn because we started talking to them. There was no real active marketing going on. But of course, those people started to talk to other people. And so, it happened. And when we were able to provide those first demonstrations, no matter how rudimentary it was, that's when things seemed to really kick into a much higher gear where people started talking to each other. Librarians started talking to each other and say, “Hey, this is real. This is exciting.” ROBERT: It's not vaporware. HARRY: And this is happening. Right. It's not vaporware. And we need to get involved. And so, they have. CHARLES: Yeah. And the energy and involvement just in the course of the time that we've been a part of the project is apparent. Every kind of gathering is larger and the diversity of topics that are discussed is increasing. And yeah, the conversations have burned from “What can we do with it?” to the last time the project got together as a group, it had much more of a conference-y feel. And the topics being discussed were like, “How do we actually deploy this to our real systems?” Which has been fantastic to see. And so, just to actually feel the traction is just, it's so gratifying. So, I guess one of the questions that is on my mind is – so you mentioned that you had been at this for about two and a half years. What is in your mind the most pleasant, unexpected outcome of this project that you didn't foresee two and a half years ago that you're experiencing now? HARRY: So, I think one of the one's that's surprising is I've never actually personally been involved in an open source project before. And I think one of the fears constantly working for organizations that really have been closed source – in fact, I've been that my adult working life – and you sort of walk into this with some [trepidation]. Because oh my gosh, everything I'm doing, everything I say, everything that gets documented is going to be out in the open for everyone to see? Including competition and everyone else? And I have to say, one of the most interesting things, or one of my favorite pieces here, is: it is so freeing. It's a release. It's actually amazing to not care about what your competition or anyone is doing or thinking. And it's even more amazing whenever you've got a question and someone's out there asking you for access to documentation, you can simply point them to the website, to the URL and say, “It's there. It's all there. Anything you want is there. Go take a look.” I think that's pretty amazing. The other one of course are the relationships between many, many different people who I've just never really been – or I never would have had the chance to work with before. And really hearing all these different perspectives and points of view as far as what people think are right, correct, or what we should be doing – it's great to see. And it's great to see this wildfire word of mouth message that seems to be moving everywhere. We're hearing back from people not just in North America, not just from the EU, but from the Middle East, from South and Central America. There's really just people interested everywhere. And it's amazing to me, because I don't think I spoke to anyone over there. So, it's happening. And that's just an exciting thing to see. CHARLES: Yeah, yeah. ROBERT: It's really cool that open source can connect you with the rest of the world like that. It's just so powerful. HARRY: It does. And it's an amazing thing. ROBERT: And the amount of collaboration that happens, I've never experienced it in any other way. It's really cool. [Chuckles] I don't know how else to put it. It's just, it's almost mind-blowing that you are able to across timezones, across the world, collaborate with somebody on something that you both feel passionate about and you're pushing it forward in an open manner. HARRY: Right. And it's also amazing when you have these meetings and there's people you've been collaborating with for months if not a year or more. And getting to meet them face-to-face for the first time. That's really a pretty amazing experience. ROBERT: Yeah. And it's pretty awesome. At our last gathering that we had at WolfCon, I had the absolute pleasure of chatting up a bunch of different librarians. And hearing their experiences and what they're looking for out of an experience, it's just really cool to be able to talk to those people and see how they work and help build this into the platform that they're wanting to use. HARRY: Right, right. CHARLES: Yeah. I would say on the whole, there's been a lot of those wonderful, unexpected outcomes. Were there any that presented a particular challenge or wasn't quite so much a walk along the path of roses? HARRY: So, you know I think the biggest one is – I mentioned earlier we embarked on this project a little differently versus what we saw as normally successful open source projects, right? We did not have that core, beautiful, shiny object to unleash onto the world and let them know, “Hey, have at it.” And so, it wasn't so hard in terms of getting interest. But what's been hard is the amount of interest has generated enormous numbers of organizations that are interested in contributing to the project. Which also implies then that now, not only are they coming to us asking us, “Hey, how can we help? What can we do here?” but then there's a certain aspect, because this is a first version of, “How do you manage the building and construction of that very first version of this software project when the reality of the matter is, you're not really responsible for most everyone working on the project?” And how do you get everyone aligned and organized? How do you get everyone onto that same path along with those same milestones that we've all agreed on? And even worse, how do you get agreement on those milestones so we can all walk that same path and deliver that first version? And so, I get asked sometimes by folks that are involved in other open source projects, “How do you handle this situation? Or how do you manage all the work that goes on?” And when we tell them, it's kind of surprising because it's not what they're used to seeing. And a lot of it is because of the way we started, the interest that was generated early on, and the fact that we did not have an initial version. And I don't want that to sound like, “Oh, this is a huge problem,” or, “If I were to do it again, I would not do it this way again.” I in fact would do it this way again. I think there are a lot of benefits and it's been a great and interesting way to go. It's just the management aspects of it are definitely challenging. And I think more than maybe we had initially anticipated. Going into this next time, though, I definitely know what I'm facing and I'll be prepared. CHARLES: Yeah. So with that in mind, given that you've been through this, do you have any advice to offer someone who might be considering embarking on a similar journey? And undertaking a similar Herculean task. HARRY: Well, I think you'll find it's harder than you think. And be sure to plan for that. But I guess at the same time, I think my biggest advice or my best advice would be: you need to keep an open mind. And when I mean an open mind, you need to be open to other thoughts and ideas. And I think you need to put yourself in the perspective of what if you were one of these other people that are working on the project or trying to contribute to this project? What are the things that you're doing or rather the things that you're doing now, the way you're acting, the way you're reacting, how does that look to others on the project? Because you're not the only one contributing. You're not the only organization contributing. And are the things you're doing reflecting badly in terms of others' perception or optics as to what this looks like as an open source project? And I think for me and others, there was a huge learning experience. Because your first thought is, “I'm going to tackle this like any other software project,” and it's not. It's not like that at all. CHARLES: I think that's… ROBERT: That makes sense. CHARLES: Yes, that makes sense. That's excellent. That's excellent advice. And I think that's one of the things that makes open source so powerful, is because there's that aspect just baked into it from the get go, you have to be mindful of a bunch of different perspectives. And that ultimately results in a solution that's going to be workable for a bunch of different perspectives, that's going to be flexible to accommodate for all the different use cases that all the different participants in the community might bring to the table. HARRY: And you have to be very mindful of how your actions are perceived. Because normally, your actions are perceived by the company you work for, what tends to have a particular culture. But on an open source project, you're actually dealing with many different cultures. CHARLES: Yes. HARRY: And so, that's a tight line that you have to walk. CHARLES: That was a powerful note to end on. Thank you everybody for listening. We are The Frontside and we build software that you can stake your future on. If you want to get in touch with us, continue the conversation, you can reach out to us on Twitter at @TheFrontside or drop us an email at contact@frontside.io. Thank you so much Harry, for being on the show today and talking to us about FOLIO. HARRY: And thank you very much for having me. This has been great. CHARLES: And if you want to, if there's something that you want to learn about or a topic that you want to discuss, please, please let us know. We always are open for your feedback, especially when it comes for things that you would want to hear. And if you want to get involved or learn more about the FOLIO platform, you can just head on over to FOLIO.org. There are resources for librarians, developers, and anyone who is curious about becoming a community member. Thanks as always to Mandy, our producer. And we've got a really great podcast coming up on the 14th of June. We're going to have Michael Jackson on the podcast to talk about separations of concerns in React. So again, thank you Robert. Thank you Harry. And we'll see you all next time.

In this episode of The Frontside Podcast, panelists Charles Lowell, Rob DeLuca, and Sam Keathley, discuss how much the distinction between frontend, backend, and fullstack developers matter in both personal and professional senses. The differences in mindset between these categories are also discussed, for example, how does a fullstack developer solve a problem vs how a backend or frontend developer? And also, how clear (or fuzzy) is the line that separates them? What are the skills a frontend or backend developer needs to consider themselves fully fullstack? And finally, is there any sort of tribal separation between the factions? Do you have opinions on this show? Want to hear about a specific topic in the future? Reach out to us at contact@frontside.io or on Twitter at @thefrontside. This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. TRANSCRIPT CHARLES: Hello everybody and welcome to The Frontside Podcast Episode 101. My name is Charles Lowell. I'm a developer here at The Frontside and I'll be hosting today's episode. Today we're going to be talking about the different developer profiles that are out there. You might have heard mention of them on Stack Overflow, on job boards, on the Twitterverse. Things like frontend, backend, full stack, half stack, short stack. [Chuckles] Things like that. With me to talk about these are Rob and Sam. Rob is our director of open source here at Frontside and Sam is a software engineer here at Frontside. So, hey y'all. SAM: Hey. ROB: Hey yo. CHARLES: Alright. So, before we get into the meat of the topic, there are a few announcements that we wanted to make. First and foremost, this is some really exciting news. This week, Taras Mankovski officially joined the team at The Frontside. And we are really, really excited about that. [Cheers] ROB: That's exciting. CHARLES: Yeah. It's super exciting. Not only is Taras an exceptional engineer, he's got amazing code skills. He's extremely empathetic and a great person to work with. He's great working with clients and he's also fantastic at really working with developers who are climbing that ladder and helping them level up. So, he's been involved in mentorship literally since I've known him. I think that was the first time that we ever came into contact with him was through his mentorship work in the Ember community. And we've been collaborating with him for years on several open source projects. And so, we just are so excited and know that he's going to come on and do some amazing things by helping us be better developers, helping us be better community emissaries. And so, lots of good stuff coming out of that. ROB: Absolutely. CHARLES: Yeah, yeah. No, that's really exciting. So, definitely wanted to throw that out there. ROB: Too bad he didn't join a little bit earlier. We would have had a good time with him on this podcast. CHARLES: I know. Well he's actually, I think he's been on the podcast twice. ROB: [Laughs] That's true. CHARLES: But yeah, there's definitely a couple of topics where his input would have been absolutely critical. I think it would have been great to have him along when we were trying to run our own apprenticeship programs. Those would have worked out I think a lot better. Well, not to say that the outcomes haven't been stellar in some cases. But I think having his expertise would have made it a lot smoother. That said, before we jump in then, just wanted to let everyone know that as always, if you want to work with us you can get to us at contact@frontside.io or send a shout-out on Twitter @TheFrontside. Okay. Y'all, let's jump into it. Like what is even a stack? ROB: [Laughs] So, do you want to kind of set up where this came from, Charles? I think it came from you and I going back and forth on Twitter. And I was like, “You know what? We need to talk about this on the podcast.” CHARLES: Yeah. I think I was actually on vacation. ROB: [Laughs] That's right. You were trying to ski. CHARLES: [Laughs] I was on the ski lift. And I saw you tweeting and I was like, “No, no, no. I got to respond to that.” Probably because there was maybe a little bit of disagreement but also because I just needed something to do. Looking out the majestic scenery of the Rocky Mountains wasn't enough. So yeah, what's the backstory there? ROB: So, I can't remember what I was actually tweeting about. But I kind of feel like there is a distinction between a frontend developer, a backend developer, and at this point I don't know if a full stack developer truly exists. I mean, it does. But it's used more often than it really implies what they're doing. Usually when you hear of a full stack developer, it's like, “Man, I can sling some code on the backend and I can do a little bit of architecture stuff on the frontend. But CSS, keep that away from me.” And in my opinion, if you're a full stack developer, you need to sling CSS, too. CHARLES: Really? ROB: Yeah. CHARLES: No, no. I certainly agree. Because I was going to ask, is it fair to say that full stack developer is code for like, startup developer? ROB: Yeah. That's a good – like a utilitarian, like a jack of all trades but not a master of any? CHARLES: Right, exactly. ROB: I can take that. So, I guess the way we can start to dig into this one is like, how much distinction is there between a frontend, a backend, and a full stack developer? And does that matter in a professional sense? CHARLES: Right. And so, I think my take was that the skillsets that you need in both places are actually much more convergent than you think, than people might think. In other words, the skills that you utilize on the backend actually translate very well to the frontend. I think my stance is that what makes a frontend developer and a backend developer is more the problem sets that you gravitate towards naturally, the things that interest you. I actually was a backend developer. And now I consider myself to be primarily a frontend developer. But the skills that I had developed writing backend systems in Java translated shockingly well into frontend development with JavaScript. But it was working on the portion of the program that interfaced with the human was, that was the thing that fascinated me. And so, it's what I gravitated towards. And so, I guess in my [6:15 inaudible] or the way that I have been thinking about it is that what makes a frontend developer and a backend developer is more what gravitational pulls you respond to, rather than one particular skillset. But I know that you've got a perhaps more refined take on that, or a different take. ROB: Yeah. Before I do, I want to know what Sam's thoughts are on this. This would be a good one. SAM: Where I come from in the understanding of frontend, backend, and full stack is I – background on me: I did a bootcamp before working here. So, I did a development bootcamp where they were really excited to tell you at the end of it that, “Oh, you're a full stack developer because we've introduced you to everything. Like, touched on every sort of little topic. You're full stack. Tell everyone you're full stack.” And it's like, “Well, you're not really a specialist in any of the sense.” So, what makes a difference to me, I'm going to specifically talk about full stack, but it's like before you can say you're full stack, having knowledge of frontend and at backend, you need to have expertise in at least one of them first, or understanding of that mindset first. Like if I say that I like frontend work more then I need to understand frontend work more before I can say that I understand backend work, you know? ROB: Yeah. CHARLES: So, do you feel like the bootcamp was doing you a disservice? Or do you think they got… SAM: I don't think they did a disservice at all. So, they did a really good job. So, bootcamps are really for introducing you to everything. And then whatever you take from that is your own. So, they introduce you to all the little things like database working and then JavaScript, Ruby, whatever it is. And then it's your job to figure out in those introductions, what you resonate most with. So, I think they did a really good job in that. I think that they shouldn't tell you what you are, like tell you that because they touched on whatever it was, that you're full stack. CHARLES: Right. SAM: So, I don't think they did a disservice at all. I learned a lot. But I wouldn't say I'm full stack at all. CHARLES: So, if you feel as though you're not full stack, what do you feel that you are? SAM: Definitely frontend. CHARLES: Definitely frontend. SAM: I gravitate more towards frontend. ROB: Why would you feel that way? Is it because you're more pulled towards the UI of things? Or is that where you feel like you've gotten most of your experience now? SAM: Actually a little bit of both. I tend to like the UI of things a little bit more. It's interesting. Whenever I was going to the bootcamp, I thought I was going to be more backend because I thought I really like working with the behind the scenes sort of. You don't see this work but you know it's there. But I definitely like the UI of frontend a lot more. Just personal. ROB: I see that. SAM: And it's where my skillset is, because this is all frontend work that I'm doing here at The Frontside. So, it's what I know most at this point. CHARLES: Yeah. I'm curious too. Isn't it true that there's also a conception in the wider tech world that frontend is limited to CSS and HTML? ROB: Ooh. Yeah, that could be. CHARLES: Like when you see job posting for frontend developers, typically it's like, we want someone… ROB: Yeah. It's kind of morphed from – JavaScript developer is now a single-page app developer instead of frontend. Yeah. Yeah, I see what you're saying. It's now, it could be considered as HTML and CSS and not JavaScript anymore. [Chuckles] CHARLES: Which is so bizarre to me. SAM: [9:42 inaudible] a lot of places when a frontend developer who is also a UX designer. Someone who has the knowledge of designing things from the ground up and then only working in things that look good, rather than logic. ROB: Yeah. This can be probably an entirely other topic. But that's like the frontend of the frontend and the backend of the frontend, is like [10:03 inaudible] call it. Because there are [10:06 inaudible] developers. And single-page apps have gotten so complex these days, between your data layer and your testing and managing state on the frontend. There is a backend, an architecture, that you need to consider when you're developing a single-page app. And I've worked on teams where there are developers that are working on the frontend but they still don't like CSS. They don't do CSS things and they don't consider layout or any design. They just take the ticket and they get the data going. Get the data from the server, request it, display it on the page. And then another person would come through and style it up to the spec or the mock. But yeah, that's interesting. So for my take on this, and where this actually really came from, is we were going through some hiring. And we were trying to figure out how we can place somebody or hire someone that can be effective in the role that we immediately needed. And all of our immediate needs were in frontend-specific things. And we can actually talk about if they're frontend-frontend or frontend of the backend. But where this kind of came from or was born was: if we're hiring somebody that needs to fill a frontend role and they're primarily a backend developer, there are gotchas and things you have to know to be an effective frontend developer. Like quirky browser things that happen or like how to set up a Babel transpilation setup. And what are the gotchas here? Kind of the history of the frontend web, because there's a lot of things that are there that can cause a lot of headache if you don't know the backstory. And I know from your perspective Charles, you think – and this is isn't wrong – but you think that if somebody has a really talented backend developer, it's really just concepts that you can apply to the frontend. And I agree with that. CHARLES: Right. But you have to want to do it. [Laughs] ROB: Yes. That is also a key. CHARLES: You have to be gripped by the problem set of the frontend. You want it to inspire you to leverage those solutions. ROB: Yeah. So, would you say there's a different mindset between a frontend, backend developer, and full stack? Does a full stack developer solve a problem that's different from how a backend developer or a frontend developer would? CHARLES: That's a subtle question and probably one that might offend people, my answer to it. So, I'll go ahead and answer it. [Chuckles] ROB: Hashtag [12:23 inaudible] CHARLES: Actually, maybe not. Just in my experience historically – I want to underline the fact historically – I think that it is people who gravitate towards the outside-in mindset, in other words the very first thing they're thinking of is, “How is this going to feel to a person who's going to use this?” If that's your first question that you ask, then you probably are going to gravitate towards the frontend. Then in terms of the backend, I'd say historically – and like I said, I want to underline that, and I'll get to that later – I think it's people who are more drawn to: I want to attack the most complex problem that exists. Like distributing state over 10,000 nodes, managing huge deployment infrastructures that drive these massive websites that happen at this mind-boggling scale. And so, there you really are thinking computer to computer and, “What's the ideal interface for doing that?” And then on the frontend you're thinking more like human to computer, because that's where the interface lies. Now the reason I say… ROB: I feel so let down. Where's the controversy? CHARLES: Well, okay. Well, because I think that basically – I think the controversy is saying like people who are more naturally empathetic. I don't know. That would be the controversy thing. But I want to say historical. ROB: Yeah. I can see that. [Laughs] CHARLES: So, I want to say historical too, because I feel like one of the things that's been magical about the last two decades of software development is the dawning realization that no software you write exists in a vacuum and that it's all interconnected. And so, I think that I for example feel very strongly while I try to think about the user experience first. And the developer, I'm also thinking about developer experience first. The way that you want a system to feel is going to inform the design of the UX workflow and that's going to affect the architecture of your frontend. And if you're interfacing with it through different media, like websites, phone, I don't know, even text messages, Slack, what have you – that's going to inform then the next level of how your APIs are shaped, your public-facing APIs. Which then inform the structure of your internal deployments. So, recognizing that the decisions you make in one end of the stack and ripple through completely and that they're not what we hold as dear these concepts of separation of concerns and complete and total isolation of layers. That really is false. But what it does is it enables us to change the individual layers to more – ironically the reason you want to have separation of the layers is so that it more easily lets you change them provide a more integrated experience. So, I think that's a long way of saying that I think frontend developers are more aware of what's going on in the backend and that they might be drawn into the backend. And the same thing goes for backend developers, realizing that the decisions they make are affected by the frontend and affect the frontend. And so, I think there's been a dawning more of a collective responsibility for design, for operations, for developer experience. And I think it's great. ROB: Yeah. And to [think back off] that. So, I was kind of wondering where the controversy was, because that's kind of my exact answer, too. Where do you gravitate towards on? The difference of mindset is like, if you've got a problem and let's say you're a full stack developer, so whatever that manes, but if you're given a task where you have to implement the frontend and the backend, where you gravitate towards first I would say is how you would attack solving a problem. For me, I would immediately go to the frontend and see how a user would immediately interact with it and then work out that way. Mostly also because I like having a tight feedback loop. And the frontend provides that nicely. I can change something and immediately see the difference there. And in the backend you can spend a little bit of time, unless you're TDD which you should be – you can spend a little bit of time piecing together things and figuring out the architecture. And then you'll have something that you could show for. It's just nice to see UI get thrown on the page. And it's real and you click a button and something happens. That's a really nice feeling. And that's kind of where I gravitate towards. And if yeah, if I had to attack the problem, I'm going there first. I get the endorphins from it. [Chuckles] CHARLES: Right. So, if we want to add controversy to the other side, because you got to always be controversial, right? So, we said the really blunt horrible oversimplification is that people who are more naturally empathetic gravitate towards the frontend. You could also say that from the other perspective, people who are more internally validated will gravitate to the backend. Because if you need to get those endorphins from working on the frontend, basically what you're saying is, “I want to put it in front of people and get the feedback from those people and know whether it's good or bad.” Whereas you could say, “Actually, I don't need validation from other people. I've got this concept of this architecture that is going to be the ideal thing. I know it. I've got this vision and I'm going to chase it, regardless of what's going on.” I think you can more readily do that on the backend because you're not as open to the feedback of a whole bunch of different users. And like I said, I think those are gross oversimplifications. ROB: Yeah. I agree. And so, as a devil's advocate towards the backend developer that is less empathetic, I think actually some of the best backend developers I've ever worked with were the most empathetic people ever. Because they know that software is for humans. And humans are going to be consuming the API that they build. And they take that into consideration. CHARLES: Absolutely. And I actually think that empathy pervades good software development just wherever you find it. Because yeah, someone's going to be using your APIs whether it's software as a service or it's a library. If it's software as a service, it's got to be easy to work with. It's got to be performant. It's got to have a nice command line. And so, you have to be thinking at all times, “What is it like to use this software? What is it like to consume it? What is it like to link it into my library? What is it like to call it from a web client?” So yeah, I think you're absolutely right. I think it's actually one of the great virtues of great software developers. SAM: Yeah. So, something that I've always kind fo had a question with, especially coming from that bootcamp setting: is there any sort of tribal separation between what you would consider a frontend or a backend developer or even full stack? Like, “I'm better because I understand data layers than I understand how a button fits on a page,” that sort of thing. CHARLES: I would say absolutely yes. If you at the, just do a quick poll of the Twitterverse, it seems like people tend to hang out in little circles of similar interest. So, there's definitely a bunch of people that I follow that are always tweeting about backend stuff. ROB: Yeah. Twitter is ‘build your own echo chamber'. CHARLES: Yeah. And there are people who I follow that are tweeting nothing but mostly frontend – when I say frontend, I mean frontend of the frontend. Well basically, from CSS down to nothing deeper than React components. And then there's people who are talking primarily about the backend of the frontend. So yeah, I do think that people – there are clearly, there are different areas of the stack and I think that people do tribalize around them. So, the question is: Who are the border agents? Because always cross-border trade. Any time you have borders, there's an exchange of ideas and information and things that happen at those layers. And actually, one of the things I'm really curious about is: How does that happen? ROB: Yeah. You might have the best perspective on this because you did – you were using Java Swing back in the early 2000s building UI. And that's like backend things. And then you had a lot of experience with Rails and you've moved into the JavaScript world. And the thing that I've noticed with frontend is, we're applying a lot of concepts that existed on the backend. And we're moving all that complexity into the frontend. And that's kind of where that fuzzy line of, “Are you the frontend of the frontend? Are you the backend of the frontend?” comes from. And it's interesting that – like this is going to be another podcast. We'll end up talking about this – but we have – MVC lives on the frontend. Like in React, your C is a container component. The controller is a container component. The model is probably your Redux layer if you have it or if you're using micro-states. And the view is your components, your view layer components that you're rendering down. So, these concepts are moving from the backend to the frontend. We're just kind of renaming them and making them our own concept. But they're there. So, how does that knowledge sharing happen? Is it really just a mass exodus of backend developers interested in frontend developers now? CHARLES: I don't know. I think it goes both ways. So, I can only speak to my own experience. And that was when I first started doing frontend development was back in 2003 I think. So almost 15 years ago. We were building a touchscreen interface for an electronics vendor in the UK. And it was just so much fun, because we were getting to build these interfaces that literally looked like nothing else. And you could touch them and you had support for rudimentary gestures and there was no keyboard. And basically the only interface was your fingers and like a price scanning gun. And everything, all the entire UI had to be developed out of that. And so, it was such a novel system and it was so fun to implement that I just gravitated towards it. I think the thing that was particularly compelling was it was very interactive UI. It was high-touch, literally. This is a clerk who's sitting there using this thing as rapidly as they can to sell stuff and move people through the line. So, it was like a unique problem. But the thing that was cool about it was I realized so many – we kind of already touched on this in this conversation – so many things that I had learned on the backend applied there. And there was in order to provide that experience, there had to be a pretty complex machine behind it. And that was fascinating, that machine. And so, we were able to apply a lot of the concepts. And so, in that time on the backend I'd just come off working off a backend that had basically an event bus – we would probably call them microservices now, although we didn't call them that then – were coordinating based on all these events. And that translated over into the frontend really, really well. And I remember using a lot of the techniques for exception handling – doing it on the frontend and doing a lot of the multithreaded stuff that we were doing on the backend, trying to reconcile that with the frontend and really understanding. So, there were all these analogous concepts. And it could be – so, there was an original exodus of backend development, for me personally. And so for me, it was like I felt like I moved onto the web pretty much exclusively in 2006. And it was a good – I would say 2013 maybe is when web UI development finally caught up with Java Swing. Like it was like, that whole time I was like, “I'm using the web because it's an awesome deployment platform. And it's got all this great stuff,” but man, the developer experience is not as good as like a stateful fat GUI was back then. And now, I would say it's actually surpassed quite significantly. But now, I think there are a lot of people maybe who either – I think there's a casting back of people who are in frontend development and casting back to the web. So right now, my love affair with immutability and functional programming started by using a Clojure web framework which borrowed a lot of the ideas from Clojure. So, I guess there was an exodus there – people from Clojure wanting to develop apps, wanted to have their goodies. But then I found that and I was like, “Whoa, that's awesome.” But then that really set the hook for me. And so now, I try and go back to the well, the backend well or the shared computational well, as much as I can. So, all of that stuff of basically discovering all this functional programming stuff, this highly formalized functional programming, all came from saying, “Wow. I got a taste of that. Let me get more.” So, it's very much like trying to run through the village of the backend and ransack the stores and take as much as I can and bring it back, because I know that it's good. ROB: So earlier in the podcast, you said that you were a frontend developer. You described yourself as a frontend developer. And that's funny, because I actually consider you a full stack developer. It's because you can jump in on the backend and sling any kind of code as well as anybody could, and then you can also do the same thing on the frontend. So, the question I have here is: Do you think actually someone that truly is a full stack developer – and we can define that in a second – but do you think they're at an advantage here, because they have all that experience? You can answer yes. But why? Why do you think they have an advantage? CHARLES: I think they have an advantage in the same way that a person who's bilingual has an advantage. So, if you are living in North America, it behooves you to speak Spanish because you are now open to an entire set of markets that were previously closed to you. Well, not entirely closed but like you can trade with Mexico and you can trade with South America. You can buy and sell goods. You have access to yeah, emerging technologies that might – something special might be happening in Mexico City, or in Medellín, Colombia, and you're going to have first access to that. And so, if you don't, if you're not bilingual, then you're going to have to go through an intermediary who is. And so, there's a premium: you get to be the middle man so to speak. Or you get to be, maybe that has kind of a negative connotation, but you get to be the broker of new ideas and new technologies. If you can, if you are fluent in backend so to speak, then you can go into backend-landia and you can talk with the developers there and see what kind of cool stuff they're doing. And then you can take it across the border into frontend-land and sell it. And so, that's – I think we're very much first to the gun on – not first to the gun but we're ahead of the pack in terms of functional programming. Because we've seen that. We've seen it proved out and we've now actually started to integrate it into your day-to-day routines. And we're ahead of the pack on that, right? And so, I think that's kind of a keystone analogy for me that I think really, really captures what is the advantage in being a full stack engineer. ROB: So, well how do you define a full stack engineer or developer? What things do you have to possess to actually claim that title? In order to be a full stack developer I personally believe you have to know, you have to be comfortable with CSS. You can hate it. You can yell at it. But I think you have to be able to put together a layout if a designer gives you a comp in order to claim full stack. And in the same token, if you're a full stack developer and you mainly came from the frontend, you should be able to implement backend features. I don't actually have – so, I'm strong in the frontend, not so strong in the backend. What would be the analogous of CSS in the backend? Would it be like mocking your controller actions? [Chuckles] CHARLES: I would say you should have a familiarity with HTTP and REST, would probably be the equivalent. Kind of like a foundational technology that just every single technology is oriented around it, or most. Regardless of the protocol you're using – there are things like GraphQL which are not really RESTful, although it's a blurry line there – but they're still delivered over HTTP. And so, understanding things like CORS, understanding the things that are going to be universal to APIs. ROB: I think a lot of people try to claim full stack. I try to claim it. And I know I'm not. I can put together a pretty crummy Rails API on my own for personal projects. But I'm not going to be the one that's setting up indexes on a database to optimize a database or anything. I think that does come in time, but you have – borrowing from Brandon Haye's talks about career development – if you're a full stack developer, I think you end up having to be in the industry for a long time. Longer than what we consider a senior developer these days, I think. Because you could be a senior developer and be specialized. And we see that all the time, and that's okay. But in order to amass that knowledge and experience, I think you have to be in the industry for a long time and done those things a couple of times to really know. CHARLES: Yeah. I agree. So, can I add something there? To continue the analogy of being full stack is like being bilingual or multilingual. I go back to those analogies a lot because that's what I – linguistics is what I studied in school. But when I was studying linguistics, one of the things that was going on there was they were kind of redefining what… ROB: That makes so much more sense of your vocabulary. [Laughs] CHARLES: What it means to be bilingual. There was kind of a reorientation of that definition that was going on while I was in school. And that was being bilingual doesn't necessarily mean you're able to converse about poetry or like deep technology or give speeches in another language. It really is, it's as spectrum of… ROB: Ooh, that's quite interesting. CHARLES: The definition of bilingual is like: Do you use another language in your life? So, if you are let's say someone who is a recent immigrant to a country and let's say you've got less than 1,000 words but you're using them to buy groceries, to pay bills, to do things like that, then you are bilingual. Bilinguility is not an exclusive club. It's really, are you actually using a language? So, if… ROB: Ooh, so that actually gets my wheels turning. Does that mean that you can have a junior full stack developer? Because like, if you just merely speak the both languages, technically by that definition, I jive with that. You can speak two languages. You are bilingual. You can write in two different languages. You might be full stack. But does that mean in the software industry you are a junior full stack developer and then as you go on and get tasks that are full-stack-like, you get better on both sides? Or is it as an industry, we really have to specialize? CHARLES: To start on the first point, I think that if you – let's just restrict it to one language – if you're doing JavaScript on the frontend and using Node.js on the backend, if you're writing Node code you are I would say by definition, and certainly by the definition of bilingual that I just proffered, you would be considered full stack, a junior full stack developer like I was saying. And I think that it's just been my experience that as you go on in your career, you will cross multiple layers of the stack just because you can't keep your hands off. If you have an ownership mentality of, “Here's this thing that needs fixed,” and it's on the backend, you know what? I'm going to go ahead and learn a little bit of how to do this, because I need it to work with the thing that I'm working on. ROB: Yeah, that makes sense. CHARLES: You will just naturally be drawn over borders throughout your career just because someone's got to do it, to do the thing that… ROB: You got to solve a problem. CHARLES: Right, when you've got to solve a problem. So, I think that people become more and more full stack over the course of their careers. But that said, I do think that there are clearly areas where functional specialization is absolutely a requirement. Like if you say, “You know what? We've got this API and we need to support 600 queries per second and we've got this huge, crazy deployment…” ROB: I will not be your person to do that. CHARLES: Yeah, exactly. That's something you're very much – that is something that you're very much hiring for. And you want to be hiring for like you said, someone who has the skill and someone who, that's what they want to do. ROB: Yeah. If you need better state management and rendering performance and testability on the frontend, I'm your person. [Chuckles] If you need me to scale your API, to handle hundreds of thousands of requests a second, nope. [Laughs] So yeah, I agree with the specialization. So Sam, I want to know. Has this conversation swayed you any way? Are you more interested in being full stack or are you leaning to a side more? [Chuckles] SAM: So, I think full stack is, it's as much about skillset as it is about personality. So, like you were talking a little bit earlier on how someone who's frontend might have more empathy towards the user. And someone who's backend might have more empathy towards the computer and the developer, rather than the end user. I think someone who's full stack has to have a wider range or empathy so they can empathize with all rather than just one or the other sets. I think personality-wise, I'd be a fine full stack developer. But I think professional-wise, I do gravitate towards the frontend because that's just how I am. As a person, I like to see the visual rather than the concept. I do a lot of painting. I do a lot of drawings. So, I'm a very visual person. So, it's really helpful to me, especially for someone who's junior and who's still learning and who came from that bootcamp experience, to be able to see what I'm changing. So, I think it does kind of go a little bit into experience as well. So, I think over time when I start touching on maybe some more backend work and I see some stuff that interests me, definitely I'll gravitate towards it, just because I want to learn. But I don't know that it's like an intrinsic quality, you would want to be full stack or be backend or be frontend, you know? ROB: That's interesting. Yeah, I agree with that. CHARLES: Yeah. That is actually a really interesting point. Because I think what I heard you say there is that when you have – software is a hidden world in many ways. You talked about it in the beginning of the conversation, the areas of the site unseen. Like you know it's there but it's hidden from view. So, there's a certain amount of vision that needs to develop. So, you kind of internalize what software, like this hidden software, looks like. So what does a deployment of some load-balanced microservice clustered thing look like? Most people would not be able to answer that question. But the more time you're exposed to it, the more it becomes burned into your inner vision. ROB: Mental model? CHARLES: Yeah. You develop a mental model. Your brain literally wraps around that. It takes time. But on the frontend, especially if you're a visual person – but I would say even if you're not – I think the same would apply to someone who's using assistive technology. It's still something that you can feel with your sense and you don't have to develop a sixth sense to perceive it. So, there's literally a problem of perception there. And so, maybe frontend work is a great on-ramp for everybody, because it's so perceptual. ROB: That's exactly why I picked it. I was trying to do iOS dev before. And I could not do it for those reasons. I didn't have that nice tight feedback loop of even though it was UI, in Objective-C you had to construct your UI and buttons out of Objective-C. Unless you wanted to use Apple's Interface Builder and that was, meh. Everybody built it procedurally. And what I loved about HTML and CSS, was I could instantly throw something on the page, attach some CSS to it, and see it change immediately. And that felt so good. [Chuckles] CHARLES: Yeah, yeah, no. And it's important to get those really tight feedback loops, especially when you're starting out. ROB: So, if we had to tie this back together, did we decide that there is a distinction between frontend, backend, and full stack? And if there is, or isn't, why? SAM: I think there's like… CHARLES: I don't know if we've… go ahead. SAM: Kind of a distinction. But it's also really fuzzy, I would say. So, if I'm going to explain what the difference is between frontend and backend to someone who maybe isn't a developer, I would always go back to a video game. So, when you see your guy… ROB: Ooh, this is interesting. SAM: Yeah. [Chuckles] When you see your guy running around and you're like, “Oh, this game looks really good. I'm really into these graphics,” but do you ever think about what it takes to save the game or what is actually being saved when you go to save it? So, if you're more interested in making the guy run and making the guy and the environment look good, you might gravitate more towards frontend. But if you're really interested in saving the data, like well what is being saved when I hit ‘save to this slot' or whatever? Then you might gravitate more towards backend. ROB: Or like the network layer of the online multiplayer? SAM: Yeah, yeah. ROB: [Laughs] That's interesting. I've never used video games as an analogy. I always go like, you know, if you're a frontend developer, you know that button that you click? That's the button I create. And then the action that happens is usually what the backend developer does. I think I like the game analogy better. [Laughs] SAM: I think the game analogy is something that most people can grasp. And most people can grasp. Like I'll tell people the difference between frontend and backend if I'm talking about Facebook. Like what I do for a job is I'll show you everything that's on your Facebook page. And when somebody, or the backend is somebody who makes all of that come into play, kind of, you know? But I think video games are easier to conceptualize that abstraction of data being saved rather than trying to explain the intricacies of Facebook to somebody. ROB: [Chuckles] CHARLES: Yeah. No, I like that. I like that. So, I guess if we achieved consensus, would we say that these do exist as broad functional categories? And a full stack developer is someone who will move in between those functional categories through the course of their career. And that generally, the trend is that the longer the career goes, the more you will do that. ROB: Yeah. I can get on board with that. I'm going to use your career as like the guiding post of that. The way you just explained it, it kind of made something click for me. You describe yourself as a frontend developer. And I think in our industry with software development and the way teams work, I think you end up specializing no matter if you call yourself a full stack developer or not. Because I do think you're a full stack developer. But you've mainly been working on frontend for the past what, three years, four years? So, I think it's okay to call yourself a frontend developer. But you are a full stack developer. But as you specialize around and amass that knowledge, that's where you become that full stack developer, right? And so, at one point in your career, you were a backend developer. And then now at this point in your career, you're a frontend developer. But now you have those experiences and you can draw on both of them and apply them across the fields, right? That's super interesting to me. So, maybe it is that you have to specialize in order to achieve the full stack. And you're never truly a full stack developer in your role. But it's possible, depending on the team and the team dynamics. It's just interesting that you can be a full stack developer, also at the same time be specializing as a frontend developer at that current time, right? Yeah, that's interesting. CHARLES: Alright, so it sounds like consensus achieved. Let's all virtually hug. [Chuckles] ROB: Send the hug emoji. CHARLES: Alright everybody. That is our show for today. We are The Frontside and we build software that you can stake your future on. We would love to hear from you, especially if you have an idea for a future podcast topic. Any news that you think is something that we should discuss, even if it's not the theme of the whole episode. And we will see you next time. As always, you can give us a shout on Twitter at @TheFrontside or just drop us an email at contact@frontside.io. If you enjoyed today's podcast, it was produced by Mandy Moore, the inimitable @TheRubyRep. So thank you, Mandy. And see you all next time.

Frontside alum and original podcast host, Brandon Hays, makes a special guest appearance to talk with Charles about the evolution of The Frontside as a company: where it's been, where it's going, and more hopes, dreams, and goals for the future! Transcript CHARLES: Hello everybody. Welcome to The Frontside Podcast Episode 100. Here we are. Episode 100. My name is Charles Lowell. I'm a developer here at The Frontside and I think it's safe to say, your official podcast host. With me to celebrate the 100th episode, he was also here a few episodes ago but also was here on our first episode I believe, is the [inaudible] Hays. Hello Brandon. BRANDON: Hi. CHARLES: Welcome back to the podcast. BRANDON: Actually, are you going to light your trainee badge on fire now in a bucket, in a ceremonial pyre? CHARLES: I live in New Mexico, so I think I'm going to just after this, grab my shotgun and give myself a 21 gun salute. Just in my front yard. BRANDON: There goes old man Lowell again, with the shotgun. CHARLES: I'm just going to [gun shot sounds] in my own honor. BRANDON: I was at the Alamo this weekend, actually. And I don't know if it was just because it was fiesta in San Antonio but they had a demonstration, like a musket firing demonstration where those things are basically little cannons. They're just small cannons. It's very interesting. They're very loud. CHARLES: Yeah. They're small, handheld cannons, yeah. So wait, were you – what is fiesta? Now, as someone who grew up in Central [inaudible], I feel like I ought to know this. BRANDON: I don't know. We found out by accident because we were planning a weekend to go hang out and get drunk on the riverwalk and we took our families down with some friends and then they're like, “Oh, it's fiesta,” which is like a 10-day celebration of the history and establishment of San Antonio – which I did not know is a 300-year-old institution. So, it's like one of the oldest things in this entire western United States. So, it's pretty neat. It's different. It's weird. It's like 90 minutes from Austin. There's nothing in Austin that's older than six months. Every six months we must demolish something and then build a condo skyscraper in its place. So, it's kind of neat to be in a city where it has – walking around the Alamo, I'm realizing, “Wow. Setting aside any of the historical significance of Texas independence or whatever, this is just like a really interesting very old building. This is hundreds of years old in an area where there's nothing that's hundreds of years old.” So yeah, it was pretty cool. It was a good weekend and we got to see muskets being fired. And we saw a doctor gross my kids out by talking about the medicine of the day, in full costume and showing all of the procedures and threatening my kids with amputation. And it was a good time. We all had a good time. My nine-year-old thought it was the coolest damn thing he'd ever seen. CHARLES: Really? Did the have bloody saws and everything? BRANDON: Oh, yeah. CHARLES: Was it like a reenactment of 300-year-old surgery? BRANDON: It wasn't a full reenactment. But it was a graphic description using the tools of the time. CHARLES: Wow. BRANDON: Highly recommend, check out the Alamo. Super fun. CHARLES: That does sound really cool. BRANDON: I did not expect to have a good time and it was a good time. CHARLES: Yeah. Yeah, I know the whole reenactment with the musket firing is fun. And it is, it's actually an incredible building. Although there's been a big kerfuffle about something about how they're going to preserve the lawn. But I haven't really followed that too much. BRANDON: Yeah. Yeah, I don't care about the lawn. I care about – no offense, lawn, if lawn is listening. This is not weird, how Stanley broke our brains with the word ‘lawn'. CHARLES: That's true. BRANDON: Yeah. He broke us real good. CHARLES: Yeah. I can't see a lawn without a beard. BRANDON: So yeah. So, life has been pretty good, man. Let's see. I left Frontside September, October. CHARLES: 2016. BRANDON: 2016. CHARLES: So, it's been months. BRANDON: 18. Yeah, thereabouts, right? So, I assume that nothing happened since then and if I came back to The Frontside now, everything would be exactly as I left it. My posters are still up in my room. My Bon Jovi poster. You left my bed just as I made it, like kind of unmade. Everything is just preserved as a shrine to me. CHARLES: Pretty much. I mean, we did give away the mics to Goodwill. BRANDON: No. CHARLES: We actually did not give away those mics. BRANDON: I never even got to use them. CHARLES: I know. Well, you know part of the problem is we don't even get to use them that much either. It looks really cool and it plays really well, like our podcast studio. But you know, I'm now spending 75% of my time in Corrales, New Mexico. And at any given time, people are either working from home, or working remotely. So, a lot of times the podcast room tragically does not get used. But it looks so cool. People come in there and they're like, “Wow, you guys must be really smart and technical people.” BRANDON: I realize this is probably a rote stereotype at this point, but I am assuming the only reason that you moved is that you are dabbling in the production of meth. CHARLES: Pretty much. BRANDON: It's like, I want to learn a new trade. Programming, it's just – programming, how interesting does it stay honestly for 25 years? CHARLES: Right. Yeah, and you know, we've got some good techniques. Continuous integration, deployment, things like that. Test-first. These are things that can be applied to different verticals. And I was looking… BRANDON: [Laughs] We ship meth to production on the first day. CHARLES: Right. [Laughter] Exactly. So, I figured it was a market ripe for disruption. BRANDON: [Laughs] It's probably true. So yeah, I wanted to ask you about that. You all kind of scattered to the four winds in some ways. You have Elrich in Boston and you're in New Mexico most of the time. CHARLES: Joe is in [inaudible]. BRANDON: Oh yeah, Joe moved to New York. CHARLES: Yup. And honestly, the traffic is so bad in Austin that I'd say 50% of the time, people stay home rather than drive into our centrally-located office. So, that's actually something that we're struggling with right now because the bulk of the team is still in Austin. But the office space is underutilized. Our team size now, we have eight engineers. And five of them are in Austin. Our other staff is also in Austin. So, what do we do with the office? It's a big question. BRANDON: And that's quite a cultural change, too. Because when I was there, we would tell people, “We want to be able to do remote someday. But we just don't know how to get into that culture to change the way that we do our meetings and change the way that we do standups and coordination and communication.” I didn't feel like we had the tooling at the time. So, something – I knew that at some point there would be probably a forcing function to basically catalyze something to allow that to work. And I'm curious to know what that process was like there. CHARLES: I wish I could say that there was a process other than experiencing the force of the forcing function and then being forced into it and then just kind of dealing with it. I have not taken a poll of the other remote employees of which now I am one, at least for the time being. So, I don't want to speak for them. But it was less painful than you might imagine. And the reason is because – and it's one of those things you actually gave me this analogy back, probably three or four years ago and I love it – is sometimes you're hanging off of a precipice and you don't realize that you're toes are two inches off the ground. And then all you can perceive is the precipice and you feel the weight of your own body concentrated on your fingers gripped to the ledge. And you don't focus on the fact that you're actually, the fall is only two inches long. And that's kind of what we experience with the remote culture. Now, I don't want to say we were Pollyanna about it and didn't realize that this was the step that we were taking and making sure to check in with the remote employees. But one of the things is our communication styles were already very asynchronous both for our client work, for our internal work, using mostly Slack and GitHub pull requests and issues – certainly for the development portion, very little changed. What we didn't realize is that because of our involvement in open source, we were already acclimated to a distributed work style. We just didn't really realize it. We didn't have to change much. I think where we have a lot more work to do is kind of integrating people socially and making sure that conversations don't happen that aren't available for other people to consume asynchronously. So, if you're having some architecture problem and you're sitting next to somebody, you'll take that avenue rather than let it play out in chat or over email. And there is definitely a certain portion of that, but I think we still do a lot of pair programming. That's still our major mode. I'd say 75% of our code gets written as people collaborating. And so, while those in-office discussions do happen, the ramifications circulate rather quickly. And most of those are in the context of people pairing inside the office. Does that make sense? BRANDON: Mmhmm. CHARLES: So, I don't think the office and the physical space were as much of a bottleneck as we thought they might be. And so, because of the – a lot of people did work from home already because of the traffic. And we were involved in open source. And our communication with our clients is usually – we don't currently have any clients in Austin. So, that's all to say that the transition was actually quite natural. And I think there's some strong analogies between collaborating in open source and having a remote culture in your office. I think what we need to get better about is making sure that we get the team together at least twice a year, everybody together. Making sure that people are able to understand their priorities and get to circulate around and get introduced to a bunch of different people. And yeah, I don't know. There's definitely a lot of work to be done on the non-development front. BRANDON: It's interesting. The agile approach to things is to try something. I'm starting to think the agile and the scientific method are related where it's like, “Here's a hypothesis. Here's the experiment. Here's what we think we want to learn,” and then you learn it and you take the next step based on that information. And that failure is an option. I think that's the point of agile, is to make failure safe because it's small and you're guaranteed to learn from it. Like, the point is to learn. And so, I really, I'm starting to think that those are just basically the same thing. That agile is like the application of the scientific method to product development. And it sounds like you're being agile or experimental about your work. And the trick is, like any scientific discovery, the trick is in coming back around to it and analyzing it and deciding whether this was successful or a failure based on feedback and finding what the measurement was that you were trying to improve. So, the lesson there was, “Oh, people become disconnected from each other. We need to gather everybody for an all-hands periodically.” We didn't use to have to do that because all-hands was every week, at least. CHARLES: Right. Yeah, everybody was constantly – there was a constant chatter and you could just kind of, the context was just all sitting at that one table, in that one room on 38th Street. And all you needed to do was dip your ear into that pool of context and you're set. Whereas that's just not an option right now. So yeah. I think the danger with agile is not being concentrated in your experimentation. I think what gave us our fear about saying we're going to do remote work – because I remember we always talked about it. We danced around the issue – was are we going to lose who we are? We have a set of way that we do things. And there is power in kind of sticking to the framework of the way that you do things. Because you understand it and you know it. So, when you're pushing and you're experimenting, being able to say, “We're going to – push and we're going to focus on this one area and we're going to iterate on it and we're going to keep everything else static,” it's going to be the wall that we can walk along. But we are going to push in this area. And so, I think the dangers of you doing that in all the areas of your business or all the areas of your project, you're iterating and refining, nothing ever gets done. And so, it's kind of like once you get to some ground that's solid, when you do start iterating it, you start introducing instability. So, when you go remote you have to start thinking about remote work, whereas we didn't have to think about that before. We were essentially, the feature of saying that we were a one-office company and an on-site company is we didn't have to think about that problem. BRANDON: One thing that you were just taking about is this idea of concentrating so that your experiments are happening one or two or maybe three at a time instead of trying to run five experiments at a time. And yeah, there's another danger I think in agile of seeking local optimization where you're basically like – it's like taking a bacteria and running it through many, many, many iterations that's targeting one thing and it mutates into this weird thing that only does this one thing. Or a dog breed that the whole – did you see that, I don't know where this came from but there was some scientific findings that there was a dog that was bred in ancient prehistoric times that was bred to turn a spit to roast meat over. So, they bread a dog that the whole point of this dog was to turn a spit so that people could roast meat and go to sleep and let their dog serve it, cook for them I guess? CHARLES: Wow. BRANDON: That's pretty impressive. CHARLES: I would say like their dystopia is in the past. Or certainly canine dystopias. I guess we live in a canine dystopia. BRANDON: Not in my house. CHARLES: Not at your house. BRANDON: This place is known as a canine paradise. So yeah, I think that's a really interesting point though, that limiting the number of concurrent experiments so that you can actually respond to them in a meaningful way instead of just being like, “Wow, we learned a bunch of stuff we're doing wrong. Anyway, back to the grind.” CHARLES: [Laughs] Yeah. BRANDON: Back to sucking at everything. CHARLES: Right, right. BRANDON: That kind of feeds into a lesson that I have learned very, very, very recently in the interview process for looking for my first real job in over a decade. And that process is very humbling. And one of the humbling experiences was being rejected for a job from a very notable larger former startup here in Austin. And their interview process is really buttoned up. I got really deep into the interview process and at the end of it they're like, “Oh, you're not technical enough.” And it was really, it was like, I don't know. It was hard for me to process at the time but it's super easy now to look back and go, “Oh, I was definitely not a fit for that type of job if being able to write JavaScript on a whiteboard without the aid of Google to solve problems and refactor code is like a fundamental part of what is valued in a manager there.” That's just not going to be me. But one thing I – and it wasn't a colossal waste of time. There was a ton of time and energy I invested into that specific process, but I actually derived a ton of value out of it. Because every person I met there was focused on the same thing: their culture of making experimentation inexpensive so that everything there is framed in terms of an experiment. What's the experiment here? What's the hypothesis? What's the expected outcome? How soon can we get to a place where we can validate that outcome? So, it's kind of like everything is really lean. And yes, it does – like I asked, “What's the dark side of that?” and it can lead to optimizing for a local maximum. So, you have to pause every once in a while and reflect at a larger scale. But it changed my attitude about a lot of stuff. I tend to walk around fearing failure. That's more my speed. I'm afraid of failing because failure can be catastrophic. But that's because I take big swings at stuff. When I go give a conference talk, it better be the best conference talk of my life. When somebody's like, “Oh, that was the best conference talk I have ever seen,” I'm like, “Ah. I'm so glad you said that because if you'd said literally anything else I would have collapsed internally.” You know? The stakes are so high for everything. And making it safe for yourself to fail by treating things like an experiment and working with my teammates. And so, two or three scenarios over the phone in a week when I was managing the team at my last company, somebody would bring something to me and I'm like, I instantly went to all the reasons this probably won't work. “Here's the problem with this.” And I thought, and I immediately turn around and went, “Wait a minute. Bring me a hypothesis and the experiment and how we can experiment with this thing.” And he's like, “Well, we could try this next week and we'll know whether or not this is a good idea.” And we tried it the next week. It was like organizing an architecture team because we were waiting to hire an architect. And the results were mixed for reasons I won't get too deep into. But the fact was, it gave us the freedom to try things. And I'm trying to carry that spirit around with me now. It's been really eye-opening. So, completely like, just a 4% alteration in the way that I think about problems, but it has the ability to dramatically alter the trajectory of how I solve things in the future. CHARLES: So, do you include now inside the planning process experiments? Like, a certain number. BRANDON: Absolutely. CHARLES: So, the typical “enterprise” development is we have our features, we're going to do them in this order because they're this priority. And then agile comes along and it's like, “You need to take these things and you need to break them up into small chunks so that they can be accomplished in small time slices,” so that you don't basically bark up wrong trees. Or explore [inaudible]. BRANDON: Yeah, but that's almost like a stupider version of waterfall. CHARLES: But exactly. That's exactly my point. Whereas the problem is, there's no avenue for experimentation in there. Rather than saying the entire team is marching in this one direction that meanders around and focuses in on the local maximum, which hopefully is relative to the market landscape is the absolute maximum, saying, “We're actually going to be marching in one major direction but we're going to be sending out scouts at all points.” If you were actually – I've actually been reading a lot of ancient military history. And It's just insane that an army, or even a detachment, would go all in one clump. They're constantly sending out people. Information is really, really, really important. BRANDON: That's an extremely, extremely good point. I've actually – it's so funny, because I've used a very similar description where we are trying to chart a course to this ocean of opportunity somewhere. And we can't just send the whole team in a direction hoping that the ocean is in that direction. We have to have our Lewis and Clark. Somebody has to be the cartographer. Somebody has to be the explorer. And that means that there has to be a little bit more freedom for those explorers. I don't yet know how to translate that into software terms. I just know that that's a collaboration usually at most companies between product and development. That product is doing some of the exploring of the space and then development is doing some of the exploring of the technical capabilities and possibilities there. CHARLES: So, you see it. What's interesting is you see it in product planning, kind of in the large, with the waterfall. You see it in huge organizations. They have a research and development department. And I wonder if agile kind of saw the Balkanization of your feature set into very small component parts. Can you take the exact same principle and Balkanize your research and development and integrate it into micro-iterations? We have this R&D but we're going to integrate it into our day-to-day and week-to-week process. BRANDON: I think that is a really noble goal and I think I see some people making progress toward that. The company I interviewed with does it almost to a pathological degree where there is a point of diminishing returns where you're sort of bound to this process of experimentation. And at a certain point you can only achieve incremental results. CHARLES: Some of these problems, you just need to be able to think about them for a long, long time. I actually didn't read, I actually didn't see the talk. But everything from the title, Rich Hickey's ‘Hammock Driven Development', just that title resonated with me so much. I was like, “Yes,” because sometimes you just need to be in the hammock for six hours at a time. Or in the shower. Or hiking. Or doing whatever it is that you need to do to put yourself in a zen state where you're just, your brain is slowly turning its wheels. And it can follow every lead to its conclusion without any interruption. And sometimes that process can take hours. Sometimes it needs to take weeks. BRANDON: Right. I want to kind of pivot on that. Because that's actually one of the biggest things that I've learned in the intervening time since leaving Frontside, which is creating space instead of trying to maximize – one thing that I did when I was at Frontside and then did again at my next place and I'm realizing is really has long-term negative implications is cram as much into a work day, as much output out as possible. I'm very output-oriented. I want to jam as much into my day as possible. I want to jam as much software out the door as possible. And people describe working at Frontside while I was there as one of the most intense work experiences they'd ever had. Literally, I can project that, literally just from my own intensity of trying to cram all that stuff. And providing that space for developers to ruminate on hard problems, on some of the harder problems they encounter, providing space for managers, I've learned that a big chunk of what it is to be a manager is to be available. And so, I actually want to write a sign – I was on the fence about doing this but I think I'm actually doing to do this – I have an office and I'm going to write a sign and put it up on the door that says, “If I look busy, interrupt me and remind me I'm not doing this right.” So, creating the space to ruminate or to be available for discussion, people that protect their breathing room sometimes are made fun of, especially in American corporate culture. I walked in and they were just reading a newspaper. What the heck are you doing at work if you're just going to read a newspaper? Like no, this is actually really important time. CHARLES: I think it's, yeah, it's something that I think about a lot. And I know I've shared this analogy with you before. I don't know if I've done it on the podcast. But I saw and I can't take credit for it. I actually saw it at DevOpsDays I think in 2013. There was a woman giving a talk and she was just talking about managing developers. But one of the things that she was saying was that if you looked at a microservices architecture or you looked at just even your operating system, and if your CPU was constantly pegged, you were squeezing out 100% of every time slice, instructions were just flowing through that, you're going to have a very unhealthy, very brittle, very prone to failure software system. If our microservices were not available to actually service requests, and service excess requests, and service spikes of requests, then something is fundamentally wrong. BRANDON: I want to add to that a little bit, because the thing that I noticed in managing a team where I received a ton of pressure to peg everybody out at 100% – and it jived with my philosophy at the time of, “Hey, I'm 100% guy. Everybody I work with is 100% type people. And then, let's peg everybody at 100%. This is a startup. Let's get everything going,” and I realized very, very quickly that if you don't preserve a little buffer, 20% buffer in that level of intensity, there is no ability to share resources. Everything is now a silo. So, if you're going to peg all your CPUs out, part of that thrashing is that there's no time for people to share things with each other. And people become very protective over their little silo all of a sudden. And it causes us – it's actually like the first stage of a catastrophic cultural collapse if everybody's pegged out at 100%. And literally, just dialing down the intensity is often the only thing that's necessary to get people to feel comfortable sharing some of their time with each other. You do a really good job of that with the lunch and learns. You mentioned that y'all are doing better thoughtful lunch and learns and stuff like that. It's one of those forcing ways that you can force that and say, “Hold on. Stop the development and do some stuff where you're actually sharing things with your teammates.” CHARLES: Yeah. And we do that. My biggest concern is that that actually increases the intensity. So, one of the things we've done is we used to actually be very formal about our lunch and learns. It's like, “We've got to generate content and put it out on the web so that people can see us.” We backed away from saying – we're not going to do them as often and make sure that people can actually do them. Yeah, making sure that people don't feel overwhelmed by, “I've got a lunch and learn coming up.” The point is to share something that you're passionate about and maybe introduce some really cool ideas to ferment in people's head. Rather, that's kind of the goal. There are certain things that we do very much feel interested in generating content. But I think, we've kind of been dancing around the ideas of distributed computing and IoT and what are some of the others? BRANDON: If you say blockchain, I'm going to just virtually punch you in the face. CHARLES: [Laughs] I actually didn't. Did I say blockchain? BRANDON: No. I just was waiting for you to say it. CHARLES: Okay, no. I haven't. Well, because that's – but it is distributed computing in Web 3.0, right? These problems – and we're actually going to be podcasting about this next, so in two weeks you can tune in to listen to us talk about blockchain but in the context of distributed computing – and one of the things that we're seeing is now we're starting to pay the price of outsourcing all of our lives to these central services like Facebook and Google and Amazon. And I think now they're starting to build a credible and more mainstream movement to wrestle back that control and say, “What would it mean to have software as a service that wasn't actually dependent on some central thing?” What would it look like to have Slack where it's Slack that looked like email? Where everybody had their own email server, maybe not a bad example. But you've got an email at Gmail or Microsoft or Yahoo or your company-run that's big enough its running its own Outlook client or something like that. Email is actually a really great example. Now probably people are going to crucify me for saying this, but I think it's actually a good example of a distributed system that's worked well. I own all of my email. All the messages that you send to me, I own, and all the messages I send to you, I also own. But you also own the messages that I send to you. Information is duplicated. And it's fine. If I send you an image, yes it's on your hard drive or it's on your Google Drive. You send a message to me, it's got an attachment, I also have that attachment. But the point is that we can each own our email and we each own our email service. And we can change it up. That's not possible with Slack. That's not possible with Facebook. That's not possible with all these other sharing platforms. All of them are controlled by this one thing. And so, I think that that's something that we've been exposed to through the lunch and learns and I'm actually certainly very excited about it. It's not something that we're going to be investing in immediately. We're kind of dancing around that idea. But that's something that's come out of that. So yeah, we've kind of refocused it on, what is something that you feel good about? But back to the original point, I think that this is something that applies on all fronts. If you have a business where you can't actually take opportunities because you don't actually have people – so there's maxing out at the individual level, filling up people's workspace with client work or filling it up with what have you or having them work nights and weekends. There's individual maxing out but then there's like maxing out of your business. So, if you have – we're a consultancy – if you have 100% utilization or you're shooting for 100% utilization, that everybody is placed on a project, that is a brittle and unsustainable system. BRANDON: I wish you would have told me that 18 months before I left there. There were like two years where we were at 100% for two solid years. CHARLES: Yeah, yeah. We're still at 100%. BRANDON: Yeah. I wonder what would have happened if we'd had a little, if we had figured out how to build in space. CHARLES: Part of the problem – so, here's the thing though. Space, nice space costs nice money. BRANDON: Yeah. CHARLES: And so, that's the thing, is you have to charge more. And you have to say, “We are going to be more expensive than other people.” You have to be dedicated to be at the forefront of a cultural battle, essentially. In the same way that people were with testing, where it was very [controversial]. BRANDON: Yeah. You were with CI. CI is a given now, right? CI is… CHARLES: Yeah, like [inaudible]. BRANDON: This idea was semi-revolutionary when you and I were talking about this in 2012, 2013, that we ship to production on the first day. We don't even start building software until the CI system is set up. The first thing we do is set up Jenkins and tests and get everything, the pipeline working. And now, that's just what people do. By and large, that's how software is expected to be built. And the tooling has really come up around that. But that was an expensive way to sell software five years ago, that, “Hey, this is going to cost more than bringing in Cowboy Bob and having them come jam in your console for 40 days and ship a bunch of stuff that then will most likely collapse and you won't know about it and Cowboy Bob has ridden off into Juarez, Mexico.” CHARLES: Right, with his saddlebag stuffed with your cash. BRANDON: Yup. CHARLES: Yeah, no. So, you have to – the problem is, you know when you pick these battles, you need to be prepared to fight the war of attrition of they're not going to be able to perceive the value for six months, a year, right? You're going to have to ask your clients to bet on this strategy. And it's a bet. And you're going to have to say, “It's going to pay off in six months. It's going to pay off in a year.” And you're really going to start raking in like five years. That's when… BRANDON: Yeah. Try making that pitch to a startup founder that is borderline, that is on the verge of an anxiety attack, and you can kind of just figure out what my last year was like. And the… CHARLES: So, that's one of the reasons we don't really work with startups anymore. They have a five-year plan, but not really. BRANDON: Yeah. CHARLES: They're fighting for their survival. And they're fighting for the opportunity to have a legitimate five-year plan. And so, in that sense, it's maybe not a good fit for the way that we develop software, because you either need an extraordinarily prescient founder who has been through this before, knows the true costs of software development, and is pretty well-funded so that they can actually – because we're more expensive upfront, like a lot more expensive upfront and so sometimes they flat out don't even have the cash. And that's something that you can make a quick, “It's not a good fit,” but then there also needs to be this understanding and an acknowledgment that what you're really shooting for is your five-year dividend. BRANDON: Yeah. It is really interesting, the turn that occurs when a company finds product/market fit. By then it's too late to fix the problems. So, it's really tricky to find the balance of: how much energy do you put into the success case for a company before they have product/market fit? How much time and energy do you invest in betting that this is going to be successful versus betting that if it is successful, hopefully we'll have the time, money, and resources to redo a bunch of the things that we are going to have to apologize for later? And I think that's what makes… CHARLES: Right. Like, where do those two lines cross on that graph? BRANDON: Yeah. Because you and I have both seen startups completely sunk by somebody who was overly focused on building a scaleable architecture in a company pre-product/market fit. That is a common story where an engineer that doesn't understand the business value of what they're doing and only focused on “quality” will absolutely torpedo, they'll chew up your first million and a half of funding and leave the place in just a smoldering pile of ashes at the end. So, it is tricky. It's totally a difficult thing. But I think coming back to your point of being sort of a vanguard of cultural, the tip of the spear on somebody's cultural changes – DevOps would be one. People that were really investing in DevOps culture in 2010, 2012 saying, “Hey, this, automation, is the future of how software gets shipped, maintained, observed, supported.” And so, now it sounds like, so what is your big bet for the future? CHARLES: Boy. That's a great question. There are two bets. One you're going to like, one you're going to vomit. BRANDON: [Laughs] CHARLES: But that's okay. BRANDON: Yeah. I don't work for you. CHARLES: You need to serve, what is it? You need to serve the spiny urchin with the yellow tail. BRANDON: Is that a Sonic the Hedgehog reference? CHARLES: It's just a sushi reference. BRANDON: Oh, okay. CHARLES: Some people don't like urchin. Or maybe they don't like eggs. What it like, the roe that come with sushi. But they're on the same plate. So, I would say the first one that I've been thinking about a lot is optimizing for capacity and being able to handle spikes and not being at 100% both for people and for utilization. I think that's something that is – I don't see how you could have a healthy software development process if people are completely spiked on delivering, heads down delivering features for product. That is something that I'm betting on. Essentially, you could call it the 25% time but it's really about having excess capacity to exploit opportunities as they arise. And then being protective of that excess capacity. Because you can exploit an opportunity. Your CPU has a spike load up to 100%. But then make sure you [inaudible] down to 50% at some point, or 75%. And so, I would very much like to see Frontside have a bench where people can rotate out and they're working on different stuff that are not even client-related. They can recharge their creative tanks. They're not going to be idle. BRANDON: Yeah, I've really come around on – and I really hated this at the time – but I've actually come around to the thoughtbot style of working on a product where – because owning and managing a product and developing it as a side quest, the goal is not necessarily for that product to catch fire and become the world's next big thing and to replace your consulting revenue. The goal is to give people a sense of – think about all the stuff that you've learned in your side projects that you went back and brought to your work. And some of my biggest gains as a developer have come from having a side gig of some kind, some side project that that's how I learned Ember. That changed my life. And I would never have gotten to try it if I was waiting for somebody at work to tell me it was okay to do it. So, it's about taking that permission back for yourself and giving yourself permission to try stuff. So, it could be something like that, or it could be the content stuff that y'all do. Or it could be conference talks. It could be whatever. But the goal isn't necessarily to produce things that have a direct return. It is to create the space to allow people to flex some muscles of creativity that you may not get in your day-to-day work. And that's very difficult to offer to people in any company. Now having explored startups and larger companies, but I would say especially in a consultancy where the exchange rate is dollars for days. It's sort of like when I was freelancing. I could feel every vacation I took draining both real money and opportunity money out of my bank account. That's such a hard, difficult thing to do. And so, you actually have to create the budget ahead of time and say, “This budget is allocated to these things and it's already spent.” Anyway, that's really tough to do. CHARLES: It is hard. BRANDON: If you can exercise the discipline necessary to do that and create the environment for that, I would say you're ahead of 90% of companies in the industry. CHARLES: Yeah. Yeah, so that's something I definitely want to bet on, because I think that's where the best things come from. BRANDON: Okay. So, what's the thing I'm going to hate? CHARLES: Functional programming. BRANDON: Oh, Charles. Okay, I have to stop you. Do you know what I'm doing? Did I tell you this yet? That I am participating. When I told them this, I was like, “Charles is going to have a field day with this,” but I am participating in a Haskell study group. CHARLES: No way. BRANDON: And I'm like four exercises into this thing. I have to do four more for next week. And I'm like, “This is bizarrely easy, actually,” after as much JavaScript as you and I did in sort of a functional style and then learning Elixir. And I was like, “Wait a minute. The case statement is, Elixir just stole Haskell's case statements.” So like, so far I'm not finding functional programming to be onerous. Or anyway, but we'll see when we get to the static typing. But so far, I'm not getting any of that in the earlier lessons of the book. CHARLES: Yeah, the static typing. But the thing is, you can do – it's not 100% necessary. It isn't in Haskell, for sure. But I'm surprised. What inspired you? BRANDON: We have an architect at the office that was like, “Hey, I want to do sort of a functional programming book club.” So, we have a Slack group for FP study group. CHARLES: Are you doing ‘Haskell: From First Principles'? BRANDON: No. That one was a little actually intimidating. CHARLES: Really? BRANDON: Yeah. It gets into the lingo a little early. And we're doing one called ‘Get Programming with Haskell' that is a little more – ‘Haskell: From First Principles' is kind of math-oriented. So, for somebody with a math background but not necessarily a programming background, it's perfect. But for somebody with a programming background that is just trying to understand functional programming principles using Haskell, ‘Get Programming with Haskell' is actually a really great option. CHARLES: Okay. Actually, I have not heard of that one. BRANDON: The stuff that I'm looking at looks just like Elixir. So, it's early. But it's very comfortable so far. CHARLES: Yeah. So, this is the thing. It's all a matter of messaging and marketing. Because I really feel – so, it is like there are a lot of behaviors that you see sometimes in currently entrenched functional programming communities that I think are, well I think they're objectively repulsive. But I think they're also pragmatically repulsive and that they repulse potential community members. But I think a lot of it too is people talk about these things that are, they use abstruse terminology. And they're kind of chattering back and it's very jargon-oriented. And there's just – people operate with a different set of concrete things. So, when you and I are talking, for example we might talk about a Rails controller and that's a very concrete thing. You know exactly what I'm talking about. It's something that you have held in your hand, literally. Remember when we got that Rails codebase that came as a thumb drive? BRANDON: Yes I do. CHARLES: But the point is you knew that this had a Rails codebase on it. There were any number of controllers. And when I say controller to you, a controller is an abstraction, but not really. Once you work with an abstraction long enough, it becomes concrete. And so, part of the problem is just a mismatch in language where people are talking in their world about concrete things, things that you can touch and you can feel and you can exchange and they're very relatable. But from another person's perspective, they're talking about something that's totally abstract and totally opaque and totally what have you. And so, I feel like yeah there's a huge mismatch there. And that's been one of the big bets. The other big bet that I'm making is on this trying to make what is currently abstract to JavaScript and Ruby developers be concrete. And I think that we're going to see type classes like functor and monoid and semigroup and all these things, they're abstract to you now, become concrete over the next five years. And so, that's something that I'm betting on. BRANDON: Check out this – and I know that you have a good relationship with the people that did the other book, but it really does tend to come from more of a mathematical background. And this one actually does speak to people with JavaScript, Ruby, Python experience. Like, “Hey, here is how you will perceive these things.” And so, it's much more approachable. I'm still in the first unit of the book. But having sort of tasted it a little bit, it's like, “Wait a minute. This is actually extremely familiar and not super intimidating.” CHARLES: Exactly. And that was kind of – so, I read the other book. And I think I was also aided by the fact that I tried to learn Haskell probably for five times in the past. And so, I also had the benefit of jumping against the wall with the velcro suit and bouncing off four times. And fifth time, it stuck. So, I had just temerity on my side and a general feeling. But that's definitely – the lesson that I actually came away from reading that book was like, “Oh, there's a mismatch in concrete concepts.” It's using concrete concepts that are concrete to people with a CS background or mathematics background, or people who are brand new. Honestly, people who are brand new to programming who don't actually have JavaScript or Elixir or Ruby or any other thing to lean on, I think that the First Principles book is actually pretty decent for them, too. Because they don't have anything to compare to. BRANDON: They don't have anything to unlearn. CHARLES: Yeah, they don't have anything to unlearn whereas one of the things I took away was I was like, “Oh, man. I'm using semigroups all the time. This is something that I do constantly.” When I'm coding, I might do it eight times in a day. I just didn't have a name for it. BRANDON: Right. They're like design patterns, just at a micro level. CHARLES: Yes, micro-design patterns. Yeah, it's like a RESTful architecture for your code. In REST you only get five verbs. There's five methods, man. That's all you got. BRANDON: Okay, so those are two bets. And I want to cover one more thing because I know we're super overtime. But the last thing I want to be able to say about talking about what we've learned since I left Frontside but I want to put a bow on that. So, the two things that you're betting heavily on are functional programming as a basis for solid architectures in the future, like the work that you all are doing. And… CHARLES: I would also like to say, and this is something – let me just add one more thought. What I don't understand, and this is in no way like, I don't understand people who do the, “Saying goodbye to framework X.” That's not me with object-oriented programming. BRANDON: Often abstractions are like oversimplifications but they're really useful, sort of like Rich Hickey's Simple versus Easy. Like, “Hey, there's a lot of promise with that metaphor. It's a leaky abstraction but it's a useful abstraction.” And Gary Bernhardt's ‘Functional Core, Imperative Shell' is a leaky abstraction but it's a useful abstraction. If people haven't seen or experienced that, it's pretty good. The subtlety is that these are tools that are suited to certain situations a little better. And those same situations can exist in the same codebase, can exist in the same program. CHARLES: Yeah. I still, I love Ruby. I adore it. And in some ways, I've been researching functional programming and it's been going on for the last four years. So many times, people are like, “Oh, I just can't stand this tool anymore.” And I'm like, “Man, I still love Java.” I don't understand how learning to love something decreases your love for something else. BRANDON: That happens the first two times that you fall in love, is that you feel like you have the old thing less in order to love the new thing. And then you start realizing, “No, you are allowed to fall in love with new things without falling out of love with the old things.” I would almost use that as an interview question. Is there some way to use that as a way to gauge somebody's actual real concrete maturity as a developer? Because that is a mark of maturity. CHARLES: Yeah. I mean, you could say, “What's some tool that you no longer use that still informs your day-to-day routine?” BRANDON: Yeah. I guarantee you, people that were doing Smalltalk in the 80s think about it all the time. CHARLES: [Laughs] Yup. Yeah, exactly. Exactly. BRANDON: Alright. So, I want to cover one last thing. CHARLES: It's part of growing, right? If you're going to grow as a developer, you can't be shrinking at the same time as you're growing. Otherwise, you're like the same size, just in a different place. BRANDON: However, you don't get any Medium think piece points. Nobody does the one clap, two clap, forty, for blog posts that are like, “Why I'm still using some programming language but using one a little more than I used to use and this one a little less.” CHARLES: [Laughs] Zero claps. BRANDON: Yeah, zero claps on that think piece. I just want to cover one last thing before we wrap this up, and it is the fact that Frontside, the biggest gift that Frontside gave me was the mission for the next 20 years of my career. I think it could change, but I'm pretty confident about this, at this point. Being approximately 20 years into my career, I feel like I kind of have a feeling for what the next 20 years is about. And the Frontside really drilled that into me and helped me focus it and helped me dial it in. And it is this idea that there is an incoming generation of programmer that thinks about things differently than the previous generation in a pretty radical way. Because the previous generation all came out of the same schools. They all look the same. They all have a similar shared set of values in general. They created the Sil- – you know, I'm not actually going to be overly critical of the Silicon Valley culture that exists now. It is a result of the type of people that came out at the time that value innovation over almost anything else. People talk about ripe for disruption. The fact is, that has been an engine of economic growth and progress for society in a lot of ways that has a lot of costs that weren't factored in by a bunch of people who all thought the same way. And now, with people coming through code schools and people coming from different backgrounds and people coming from different environments, they're looking at programming and software as either an economic opportunity or something they didn't see that they could possibly do. Those doors were not open to that group of people before. There is a natural influx of people but many of them are bouncing out because they're not finding that group of people, they don't have a shared enough set of values that the people that are new are coming in and finding job opportunities, finding promotions, finding leadership positions. And so, I know now that my mission over the next 20 years of my career is to create those opportunities for people that have different backgrounds from me and different experiences. The career tracks, the promotions, endorsing and supporting and kind of sponsoring this incoming group of freshmen into our industry that come from different places, different backgrounds, different problems that they care about solving. They want to figure out how to solve the Flint Michigan water crisis instead of delivering socks to people in Silicon Valley, you know? So, I feel like we're at the beginning of a seed change in the value system potentially of our entire industry. But that's going to require training up the next generation of technical leadership. And I felt like the best thing I could do right now is learn to be a better manager, because I really like that job. And it provides the opportunity to find, hire, sponsor, promote and encourage those people to move into their own leadership positions. There are lots of other things that a person, you could be a VC and care about that stuff. You could have lots of different positions and put yourself in a position to do that. You could be a consultancy owner. You know what I mean? There are jobs that you can do that you can accomplish that goal. But it gives me such a sense of direction that when I'm looking for a job, I was looking for a home for that mission rather than just the thing that I felt like doing. Like okay, this job is important to me because I need it to house me and this mission so that I can support my family but have enough emotional overhead to participate in community stuff, but enough ability to lead within an organization, enough influence to actually push that agenda. So that the next generation of people are making better companies. So anyway, all of that came out of my time at Frontside where you and I sat around talking about: how do we build a place that is like a monastery? These were your words. You remember this? We want a monastery for code where people can just focus on becoming better developers. And underneath that though was the sense that this was a place of opportunity for people that might go somewhere else and stagnate as a developer. This will be a place to accelerate them. And so, that kind of spun me out and accelerated me into my mission. So anyway, I just wanted to point out that that was like, with a bullet, is the most important thing that came out for me in my time at Frontside, was that it clarified for me what I was trying to accomplish with the next couple of decades of my career. CHARLES: Wow. Well, that's fantastic. You definitely did a lot of that both here at Frontside and I mean you're continuing to do that. I definitely want to see more public speaking from you. Maybe some [inaudible] perfect. [Inaudible] at EmberConf was actually fantastic. But I mean, you're also able to help people find their mission, too. Like the talks you have at Keep Ruby Weird and even really, the first talk you gave at LoneStarRuby about moving Ember. It's always, how do I adapt what I'm feeling to my overall mission and then relate that back to technology? Man, I just can't wait. I can't wait. When are you going to hit the road again? BRANDON: I think this is the year. I'm going to start thinking about this stuff. I'm looking at the stuff that I wanted to talk about on this podcast and I was like, “Oh no, wait. That's like a dozen podcasts.” Like, no. Absolutely not. Not possible. I will say, I miss so much, this time that I spend with you. I don't want to let it go. I really miss working with you. I really miss having these conversations whenever I want. This has been a very, very special privilege for me to be able to do this with you today. And congratulations on Frontside continuing to thrive and grow and become more of its own entity and more of its own special flavor. And it makes me really happy to see the people coming out of there, that it's still doing its mission of making great software by making great developers. It makes me real happy. CHARLES: Yeah, yeah. Hopefully we can keep on keeping on. I do miss working with you. I miss the conversations that we would have in the kitchen which are basically an extension of this podcast. But I also, man, I really, really, really, really like working with the group of people that are here today. I've just seen them producing just some absolutely amazing things. And honestly, there's a selfish aspect to it, too. I get stimulated. My own thinking and learning is stimulated by the people that I work with. And like I said, the whole side note we had about distributed systems and IoT and just a constant ferment of things. So, I still really, really, really enjoy it. BRANDON: That makes me happy. CHARLES: And I'm really glad that we got to kick it today. BRANDON: Yeah, me too. CHARLES: I thought you were going to say that your 20-year mission was to have your perfect Emacs initialization setup. BRANDON: Oh my gosh. Some of these days, I'm going to figure out RuboCop. CHARLES: Actually, do you want to pair on that? BRANDON: Yeah, let's do that. CHARLES: Alright, everybody. I'm going to sign off. If anyone wants to continue the conversation, obviously you can get in touch with Brandon. He is misspelled @tehviking on Twitter. T-E-H-V-I-K-I-N-G. Always come at him. BRANDON: Don't @ me. CHARLES: [Laughs] BRANDON: I work for a really cool company and if you ask me about it on Twitter, I'll tell you all about it. CHARLES: Awesome. And we of course are Frontside. You can get us on Twitter at @TheFrontside or just drop us a line to contact@frontside.io. And we would love to talk to you more about this podcast and all the wonderful things that we do here, which includes building custom software that you can stake your future on, that's going to be good for the five-year outlook. So with that, goodbye Brandon. Goodbye everybody. And we will see you… BRANDON: Bye Charles. I love you. CHARLES: Me too.

Taras Mankovski: tarasm In this episode, Taras and Charles talk about a project that they work on together: Funcadelic - a Functional Programming and Category Theory for Everyday JavaScript Development. Funcadelic takes the simple idea that a single function can operate on many different data structures (typeclass oriented programming) and brings it to JavaScript. Yes, there are a lot of FP libraries out there, but this one is geared towards unlocking the magical powers of functional programming while always maintaining a tangible and accessible experience for JavaScript developers. Because if you're a JavaScript developer today, you're already using most of the structures in funcadelic! Transcript: CHARLES: Hello everybody and welcome to The Frontside Podcast Episode 99. My name is Charles Lowell, developer here at The Frontside and your podcast host-in-training. And with me today is Mr. Taras Mankovski. Welcome. TARAS: Thank you, Charles. It's a pleasure to be here. CHARLES: Yeah. So, you are ubiquitous in the JavaScript world. You do a lot of stuff with mentoring and you are involved in a bunch of different interesting projects. I think you're one of those developers who's difficult to classify, which is – that's definitely one of my favorite kind of developers. I wanted to have you on the show today because there's been a project that we've been collaborating on. And there have been some interesting ideas to come out of that and solidify through that project, at least in my head. And yeah, I thought we could maybe just talk about that a little bit. TARAS: Yeah, sounds good. It's going to be fun. : The thing that we are going to be talking about is a project called Funcadelic. It's more than really just a library, a JavaScript library on GitHub. It's kind of a different way of thinking about your code. And so, I know for me, where this really became part of my workflow was, when was it? It was about three months ago or something like that? Six months ago? TARAS: Oh, I think yeah, I think it's probably more six months ago. I think it's probably what, two months, I think probably December maybe? CHARLES: Okay. But it's hard now to imagine working without this tool on my workbench. It's been probably the biggest game-changer for me in the last, I don't know, definitely in the last several years. TARAS: Yeah, it's pretty impressive how little, how small of a library can have such a big impact in what we do day-to-day. Because it definitely makes me think differently about how I can solve problems that I solve on a daily basis when I work with React. So, it's been pretty interesting. I think for me, having worked with this library, I think what I'm getting is an understanding of how things work in a way, and a perspective on how React works, in a way that I don't think was available to [inaudible] Funcadelic. The funny thing is it's not a React library, right? It's not designed for React. It's just that… CHARLES: I don't even think that – it helps you think about React, but I don't even think it's the way that the React developers think about React, right? TARAS: Yeah, I don't think so, either. I think a lot of people are on the spectrum of understanding functional programming. And I think a lot of people use, people learn how to use React, but they don't really – I don't think a lot of people have traveled very far. I'm talking about general, majority. There's definitely people who know functional programming really well. And there's a lot of really good libraries in the JavaScript space for doing functional programming in JavaScript. But I don't think the general public, the general people that on a daily basis go into – write a render function and do ‘this.' or like ‘product.map' and then return an array of components. I don't think those people necessarily think about or get the context within which they use this tool. CHARLES: Right. And I think that's actually kind of one of the reasons I think a library like Funcadelic is so important and fills kind of a missing piece in the ecosystem, is because it really is predicated on the idea that programmers use these concepts all the time. They really are, they're foundational. But we only kind of see them out of the corner of our eye, in our peripheral vision, as being like a formal concept, like mapping. And giving a name to that. You know what I mean? Like you do the mapping, but you're not thinking about: how do I generalize over it? And I think that that for me, certainly in my journey with functional programming, I thought that it was mostly about functions. Not to say that it isn't, but that was kind of the end of the story. It's like, keep your functions pure so that the outputs are only dependent on the inputs. And away you go. And understand closures and higher-order functions, functions that return functions or take functions, and that's it. But I really feel that that's only half the story. TARAS: Part of it I think is that for people, even if you look at the kind of content that's available around functional programming, it tends to be – trying to kind of [reach] people into this idea of thinking of map, filter, reduce, kind of operations. And I think that's a place where everybody starts. But I think what happens is that you really are missing – and I think for most people. And it wasn't for me, it wasn't until you wrote the readme for Funcadelic and then I read it – up until that point I didn't really, I was also the same. I didn't know how these things were related to each other. Because there's this history and wealth of conceptual depth that connects all these things together. And these things are well-understood by people who don't – they're probably not writing JavaScript on a daily basis. They might be like Haskell programmers or Lisp programmers or ClojureScript or something like it. In other worlds, not JavaScript world. So there is all this understanding about how functional programming works but I don't think it's really leaked to the general masses of the JavaScript community. CHARLES: Yeah. TARAS: You know? And it wasn't until I started reading the readme – I'm like, “There's so many answered questions that I didn't even know these questions were asked.” You know? CHARLES: Yeah, yeah. Yeah, no. And I think you're absolutely right. It isn't accessible to the general JavaScript community. And part of that is because one person's – like when you read about these things, when I would go read about these kind of higher-order concepts, of basically classifying functions, not just of saying, “Yeah, what is the essence of a map operation? What's the essence of an apply operation?” you know, “What's the essence of concatenation?” things like that, I go read the documentation in Haskell or in Clojure. And first of all, it's hard to distinguish when you're not programming in those day-to-day, am I reading reference documentation or explanatory documentation? But even in the explanatory documentation, they're using what seems like incredibly self-referential and abstract examples. And I don't think that's necessarily a knock against those communities. I think what it is, is what's concrete to one person is abstract to another. And it's like, if you're working with those things, you're working with those sets of analogies, and then you're working with those abstractions every day, then they're concrete to you. In a sense that once it clicks in your mind and your mind kind of accepts it and rationalizes over it, then it moves from being, “Yes, it's an abstraction. But it's a concrete abstraction,” in the sense that you can have a conversation with somebody and use that abstraction as an example, as a counterpoint, and a method triangulate and reveal other abstractions. But if you're talking to somebody for whom those abstractions haven't clicked yet, then it's just, it's opaque. And it's not helpful. And so, I think that one of the things that I realized is like we are using these abstractions, we just don't have names for them. And so, I wanted to give them names and put them in the hands. And the names are weird, but they are really useful. And so yeah, maybe we could talk about some of those right now. Because I think that maybe now's a good time to actually introduce some of those abstractions. So for example, if you don't know what a functor is, it's worthless to talk about a monad, in my opinion. So, that was critical piece of information for me. Because that is like missing in every monad tutorial you ever read. At least, I must have read a thousand monad tutorials. And they kind of glossed over functor or didn't mention it. Whatever. Maybe they did but I wasn't looking. And that needs to be put front and center, that there is a natural sequence to these things. It's like, some of these abstractions are built on other abstractions and you have to – you can't skip to monad. You have to start with functor. Again, I realize that's probably gibberish gobbledy-goop to a lot of people. So really, this is what Funcadelic is about, what this conversation is about, is just saying – talking through it in real-world examples to make those abstractions concrete, so it doesn't feel strange anymore. TARAS: Yeah. It's definitely giving names to things. I think it's really helpful. One thing I really like about Funcadelic is that you're not giving names that you made up. These are names that existed for 50 years. They're historic. And so, when you talk to somebody who is familiar with functional programming and if you say ‘functor', all of a sudden, I feel much smarter. And we're actually referencing the same thing, because we are – because alternatively, you can say something like, “Something that is mappable,” right? Like a functor is essentially describing something that you can map over. CHARLES: Right. That's all it is. TARAS: Right. But you know, having a name for it, it allows you to just describe it exactly as it is. CHARLES: So yeah, there are tons of things that you can map over, right? Most of the time, we think about arrays as something that we can map over. TARAS: One thing I found really interesting in starting to use Funcadelic is that when you start thinking about things as they are like abilities – like you know with an array you can map over an array. It's something we've all been doing for a while – but then something that you end up doing a lot of. When you get familiar with an array, being able to use object map, mapping an object, becomes something you want to do at some point. Most of the time, what happens is that you're like, “Oh, I don't actually – well, I have to write this thing. How do I write this thing?” and then by the time you do this, you're like, “I'm just going to go to Lodash and I'll just get the map thing that will map an object.” At the end of the day, it doesn't quite necessarily feel right because a lot of these libraries – like, Lodash has map but it feels like there is always some kind of a compromise with how these things are implemented. It's not consistent. CHARLES: Right. TARAS: And I don't remember exactly what the problem with Lodash map was. I know for a fact there are, like there's different ways that you can map things. There are different functions available for mapping different things in Lodash. CHARLES: There's ‘map to' and ‘map in' and blah, blah, blah, blah, blah. TARAS: Yeah. All those different variations. But I think it's been really interesting. We've been using Funcadelic on a project we've been working on, on microstates. And just being able to use one function map that is going to map an array or is going to map an object and it's going to do it the same, use the same function, and it's going to do the same thing. CHARLES: Yeah. You have one function that maps over the values. And that's the thing, is you realize you can map over arrays. You can map over objects. You can map over observables. You can map over promises. You can map over trees. You can map – there's literally thousands of things that you can map over. And realizing that all of these fragmented interfaces can be coalesced into a single interface. And so, it really is, I think the biggest thing is like the power of polymorphism for a function. Because that's basically the problem that I think – it's not I want to say basically the problem. I think it is a problem that a lot of the functional programming libraries suffer from, is that the only polymorphism is object polymorphism, which is kind of the native polymorphism in JavaScript. Whereas in systems like Haskell and like Clojure, you can have a function be polymorphic. And so, one function can operate on many different kinds of data, provided it has an interface. So, when we're working in microstates, we use literally one function: map. The same, the actual same function, reference to the same function we use everywhere when we map. And we're just mapping a bunch of different things. So, I think that that's one of the reasons that I prefer Rambda, for example, over Lodash, is because it has a form of polymorphism. Most of the things like maps and lenses and applicatives and stuff, almost everything works on both objects and arrays. That's actually kind of nice. So, Rambda has a basic polymorphism. But I think one of the other things that is really empowering about Funcadelic is that it allows you to make the map function work on any data structure that you happen to come up with. Anything that you want. You can make it mappable and map will work on it. TARAS: Yeah. I think for people, it's probably quite abstract, what that actually looks like. I think one thing that's interesting is that – so for listeners, the way that would look is you have a class, you have an ES6 class, which gives name to a certain piece of data that you have in your application. And then what you can do with Funcadelic, you can then say, “This is how you would implement – if you were to map this kind of an object, if you had an instance of this object, if you wanted to map that object, you can specify: what is the function you would use?” So, even though you would use the same – you would import map from Funcadelic and you would use that map to map whatever that object is and whatever type it is, but there's a way for you to say, “For instances of this type, I would like to use this specific implementation of a map to map that object,” but use one function to do it. So, it's going to use one function that you call. But you can specify, under the hood it can specify how the actual, what is actually used to map that instance. CHARLES: Right. And then that's nice, because then you can – anything that uses a mappable object, there's a couple of reasons that that's nice. Any time you can have some sort of higher-order mechanism that just requires that something be mappable, that it be a functor. And then it's really nice because then you can have higher-order operations that they just need something that's mappable. But you don't have to use that one shot of actually having a map function as a property of the object. You can actually, you can kind of define wrapper classes or whatever, that then introduce a unique way that this object can be mapped. So, you can have the same structure and map it three different ways. Whereas you're kind of constrained by that using normal OO inheritance because you have to have a map property on your object. TARAS: Yeah. There's something else actually, when you start thinking about this. For me personally, I think the first step when we start working with objects, having mappable objects, it was the first thing that was really helpful. But then I think really quickly, right after that, I think my second thing that I started using and I think is probably my favorite feature now, is append. I think it's actually – yeah, so append is an implementation of a semigroup but I think it's simpler. A simplest metaphor would be something like object assign. So, object assign is an implementation of a semigroup, except that object assign mutates the object that you pass into it. Right? CHARLES: Right. TARAS: It doesn't create a new object for you. CHARLES: Right. So again, getting to this idea that there are some universal operations. Like there's this idea that you can take two things – I don't want to say object because that's an actual concrete type in JavaScript – two things and you can smoosh them together. And I think there was actually – wasn't there literally a big controversy about this? TARAS: Yes. CHARLES: About like, array smoosh? I think this is nice – smooshing, right? But you can smoosh things together. And with addition, I can smoosh two integers together or two numbers together and get a third number, right? Or with objects, I can take two objects and smoosh them together by merging their keys. Or I can take two strings and I can smoosh them together and I can end up with a string that's concatenated. Or I can take two arrays and smoosh them together and I've got now an array that's been concatenated. So, what's interesting is these are all very, very different types that I've just described. And yet, there is some fundamental operation about them. And I think this is actually something that's bad about JavaScript is there's five different names for all those operations that we talked about, but it's really one unifying concept. And that might seem like a small thing, but when you have five different interfaces for one fundamental construct, that leads to fragmentation and you can't treat that data uniformly. And it ends up like, paper cuts, paper cuts everywhere. Whereas if you can unify all of these into a small, one thing, which is like we can append two objects, and then we're going to have an implementation of append for array. And behind the scenes it's going to call smoosh. And we've got a universal – we've got an implementation of append for object, which is going to assign the keys in an immutable way and return a new object. Or we're going to have an implementation of append for string which is going to concatenate the strings. You might even, I don't know a hardcore FP nerd would have to probably correct me because this is just totally conjecture, although I know it's probably a solved problem – is maybe you could say we append two functions together. And that returns a function which is composed, right? That might be a way that you could say – what would it look like? Is function a semigroup? Can we append two functions together? And maybe you end up with like a pipe or a function composition or something like that. But I think that highlights, when you have these universal interfaces, because literally I feel like most of the stuff that we have been working with, there's literally five, there's like five interfaces. And everything is one of these five interfaces. And it kind of flips you on your head, because the classic programming wisdom that I have certainly have espoused for at least the last 10 years is that you don't want to race to find abstractions. That's dangerous. Because you can get locked into the wrong abstraction, right? Wait. Let the abstraction emerge. And I'm a lot less bullish on that concept now that I've discovered these things, because that wisdom is cultivated in a world where there are [billions] of abstractions, if you're giving unique names to everything and the combinations between them. But if you are coming up in a world where there's five basic abstractions, then it actually pays off to ask the question, “Is this thing a functor? Is it a semigroup? Is it a monad? What would that look like?” It's a nice thought experiment that doesn't require that much investment. You can think about it for a couple of minutes. And usually, you can come up and say, “Yes, yes. It totally is,” and I can start using it. And now I've introduced this really powerful abstraction. Or you say like, “No, no it's not.” And then that information is just as valuable. And so, it's very low-cost to experiment with abstractions. And so, I kind of think of it as – I know I'm on a little bit of a rant here but this has kind of been a major revelation for me – is that when you have very few abstractions which you compose in myriad ways versus having a whole bunch of abstractions that can't be composed very much, the cost for experimenting with abstraction and making the wrong abstraction is several orders of magnitude lower. And so, you don't have to be as cautious. And you can actually use trying on abstractions as a tool, rather than a very, very high risky undertaking. And just to kind of close that thought out, I think that – I don't know if anybody else but me remembers the world before we decided we were going to make all of our web services RESTful – like when people first started building all these web services, we were just going crazy with the endpoints. And there was no rhyme or reason. Kind of weird arbitrary levels of nesting. Sometimes, you'd throw in an ID as a query param, sometimes you'd throw it in as part of the path. And then I definitely remember, it was probably around, I don't know, 2010 for me where I listened to a podcast where James Edward Gray was talking about S3 and ‘Was it a RESTful interface?' and the O'Reilly book. And it really clicked for me. And realizing that if you constrain yourself to thinking about your API at least as these fundamental operation of manipulating resources, and you were constrained to four verbs and everything, you want to have ID-based URLs and resources and as flat as possible, those constraints actually are very enabling for consumers of your API and for actually authoring an API. And I think it's the same principle at work here. Anyway, so I'll end that rant. [Laughs] TARAS: Yeah. I think it's, I think people could probably, for those who haven't been in programming as long as Charles has been, it's probably easier – Charles I think people could probably relate to what's happening with components now, I'd imagine. Because having components essentially look the same across every framework, they all have props and they all render, return some DOM, or some variation of that. But it's kind of the same thing. You take some data and then you return something that is going to become DOM. And I think having that as a rule for what a component is, you can then make really complicated applications using these fundamental building blocks. And then you don't have to – there's not really much thinking on, “Well, how am I? What interface is this component going to have?” Okay, well you know it's going to accept props. And you know it's going to render some DOM when you actually render it, right? CHARLES: Right. TARAS: That simplicity I think is really helpful. And I think it's one of the things that – I think one of the things about Funcadelic that I really like is there's kind of a really small set of rules that are really helpful. And these rules are actually, they make it predictable. Because one thing that I find really challenging with using Lodash or using Rambda is that because there are so many functions, it's difficult to know what is actually going to happen when I do something. So, a good example would be like if you use omit from Lodash, Lodash omit will then, it will actually – one thing you can do is you can materialize your getters. So, if you have getters in your objects, those getters can become values on the new object that's created. So, that's one thing that could happen. Or if you use omit, your symbol, if you have values… CHARLES: What does omit do, by the way? I'm actually… TARAS: Omit is a pretty popular way to exclude functions. Basically like a filter equivalent for an object. It's usually used to remove some props that are coming into a component. But it can do some – it can actually change the type of the object. One thing you know for sure is if you use something like omit or if you use assign, if you have an instance of something, guaranteed, working with that object in a mutable way is going to cause some really strange things. I think with omit, if you were to have an instance, it would definitely not give you an instance of the same type, like an object of the same type. It will give you just a regular [inaudible]. And there is no real way – you could create an instance. I don't know what assign would do. I'm guessing that it would just take an instance and would put things on it. But it's really not – I think this kind of ambiguity doesn't work very well when you're trying to build something, when you really just need to know exactly how your tool behaves. I think that's one thing that with Funcadelic, because it's such a small API and because you know for a fact that the library is designed to be immutable and it's designed to preserve type information, then you know that if you use one of the operations, you will get most likely the same kind of object. And you're not – well, you will the same kind of object and it's not going to mutate that object. And so, there are some of these guarantees that are actually really helpful and [inaudible] is liberating, I think. CHARLES: Right. TARAS: Especially when you're trying to do more challenging things, not trivial, just copy some properties from one object to another. But when you're actually doing more sophisticated things, in those use cases, having these kind of rules is extremely powerful. CHARLES: Yeah. And I've definitely resisted and I think will continue to resist expanding the API surface area that much. Because it is, I think there's only five or six functions in there. But what you get for those functions is extremely powerful and extremely predictable. I think it might help to give a concrete example. Like when you were talking about object assignment. Like if you have a car class in your application and you want to do an immutable object assign, well the kind of stereotypical way to do that or the typical way to do that is you assign. You create an empty object and then you assign one car to it and then you assign the next car to it. And now you've merged these two car things, right? But then the problem is, you're going to get an object out of that, not a car. It's just going to be a vanilla object. Now, it's going to have all the properties of both, but it's not going to be a car. And that could be a problem if you've got any custom methods on the prototype, any computed properties on the prototype. It's going to be a problem. Whereas Funcadelic, you can append two cars together and you're going to know that it's going to have both of the properties. You're going to know that it's going to be of type car. And you don't have to worry about running the constructor or anything like that. It's just going to have – the properties are going to be carried over properly. If you're using a library like Lodash or Rambda or something that doesn't account for type, because in order to append two things or map something, the implementation actually lives with the type, not with the function. The function is polymorphic, but the implementation lives with the type. You can then actually, you can always return the proper type. Because it's ultimately – like if your map operation or your filter operation or your what have you operation doesn't take type into account, then there's no way to actually preserve type. But because we delegate in Funcadelic, it's core to the concept. And so, it's actually a very trivial thing to do, which is why you get that repeatability. TARAS: One of my favorite things about append and semigroup implementation for object is that you can overload getters on an existing instance. So, what you get is you get a new instance with the getter that you passed to append applied to that instance. And this is kind of trippy but it's actually really powerful because – so, let's say you have an instance of an object and that instance has some getters on it. And those getters use some properties of this object to compute their value. And so, when you want – if I need to create a copy of that object in such a way that the getters still continue to work properly but I need to override how one specific getter works for one specific instance, one specific scenario, one specific use case in my code, then I can just use append. So, the first argument is the original instance, second argument is an object that has the getters that I want that I want to overload the getters on the original object. And then append will squish those things together, smoosh those things together, and give me a new instance that has the getters that I passed in, in the second argument. And all the same things that the first object had. And that object will work. This new object will be a fully-functional object just like the original object that I still have a copy of, that I can use. But this is really interesting because one thing that I'm finding with having these kind of tools in my toolset is that I've had features that I needed to implement on a project. And the people that I work with are really technical. So, they know where problems are going to be. And so, the would write requirements for how something – for a feature that needs to be implemented. And knowing the problems, they're like, “Oh, by the way. You're going to have a problem in this area when implementing this kind of specific functionality.” And for me, I'm like, having this toolset, I'm like, “I don't see it as a problem at all.” It's so easy for me. Because I know that if I need to implement – so if I have something that has expected behavior, but I need to create something that behaves very similarly but slightly different in one particular use case, I can always just copy that object and then overload its behavior. And it's still going to be a fully-functional object. And I think that alone is just not something that you can do usually. It's not something that's available. CHARLES: Yeah. It's a technique that I think was discovered. Maybe it's not original to us. But it's just, the tools enabled it in the sense that it's like having a flashlight in a dark room. It's like, that technique was always there. It's just when it becomes so concise that it's so easy to just append one more computed property to a thing, then you just wouldn't have thought to do it otherwise. TARAS: What's interesting about this too, for me, is that – and this goes back to the context conversation that we had earlier – is that I think React brought into our lives functional programming kind of, in a big way. Because part of programming React applications is working with functional components and working with functional concepts. And a lot of the things, like Redux, a lot of these things are powered by functional programming. And they work together. They compose well together because they're all from the same paradigm. But the problem is that there are all these concepts developed together over time. And they've been tested together and they've been formulated together in languages like Haskell. But the ideas that make all of these stuff work really well together, they haven't really become available in the JavaScript community. And it feels like, it's like we're all using a language that we don't fully understand. And it's like we're all – a lot of people in the JavaScript world who works, when it comes to functional programming, it's kind of like having English as a second language. It's like, you can use the words but you don't understand the humor. And it's kind of like, you can't make a joke. It's kind of like that. You can't really express yourself correctly when you don't have full grasp of the language. And I feel like how we use functional programming in JavaScript is a little bit like that. And by starting to bring these ideas, moving the wealth of knowledge from the source into the realm where we actually need to use it now, we can actually start to take advantage – leverage all these insights that actually enables all of this. It's not like – so, the idea of how to write React and think in a reactive way or think in a functional way, those things are not just owned by the React core team or they're not owned by an elite group of developers who really understand how functional programming works. It's just available to everyone. And all you need to do is just learn some of these concepts that glue all these ideas together that are fundamental pieces of how functional programming works. CHARLES: Right. That's a great point. And it points back to kind of the original reason that I wrote Funcadelic and then started and then continued to work on it with you, is that it really was – it was actually meant as a – it started out as an educational exercise. What would these things look like if they were translated into JavaScript? And it turned out that it rapidly became core to my workflow and way of thinking. And so, it really is, there are weird names to these universal concepts. It is true. It is a foreign language. I really like that analogy. But foreign languages sound weird, and when people are talking in a foreign language, you can feel excluded. And the really, the reason that we wrote Funcadelic and the reason it's there is to make them accessible, these things accessible to you. So that those abstract foreign words can over time turn into concrete concepts that you're completely comfortable with, just like any word in any language. At some point, you approached it having no idea what that sound represented. And so, it really is trying to – the emphasis there is not to noodle about and dwell on the names of the concepts but to take the real things that you are actually doing, give them names and formalize them, to enable you to participate in this new functional world that you're describing. Because I love that sentiment. It does not belong to the React core team. It doesn't belong to an elite set of developers on this project or that project. It literally is a universal tool that is 100% achievable. And people don't even realize how close – if they've been working with JavaScript for a couple of years, how close they actually are. TARAS: Yeah. I think they're really – for most people, if they were to read the readme and then – well, I think one of the problems, it kind of works with the language metaphor, is that you need an immersion, right? I think one of the reasons why Funcadelic really stuck and functors and semigroups and [filterable] and all of these things really stuck for me, and I'm thinking about how using monads and monadic operations or applicatives and all that stuff – the reason why it all stuck for me is because I've been able to talk to you about it. And I think for people, finding a network that will allow them to practice immersively, to think about functional programming, not just occasionally – I mean, you could do it occasionally as well, it just takes much longer. But once you really, once you have a few conversations where you try to dissect a problem in a functional way and you think about what these pieces are made of, it becomes very natural, very quickly. CHARLES: Yeah. I think that's actually a really great point. And the thing is, you can immerse yourself incrementally. So you can just say, “You know what? I'm just going to start using append.” Anywhere that I would concat two strings or I concat two arrays or I would do object assign – screw that. You can even just say, “Instead of using object.assign, I'm going to use append,” and start there. Or to say, “I'm just going to start with mapping.” I think also the thing that's nice about it too, is the buy-in can be incremental. But you're right. You do need immersion. You do need practice. You need to actually use, you need to use the functions and you need to be able to use them one at a time, so that your mind can close over them. So then, you can kind of move onto the next one. TARAS: Yeah. CHARLES: So, that might be one way, is to say – because you know, I don't think I really understood. I was already using append and map ubiquitously before I really understand applicative/apply. So, you don't have to grok it all at once. You can definitely bite it in chunks. And the best way to do that is to start with one concept and really just attack it mercilessly. And then also understand that there's a natural sequence there. TARAS: I would add a little bit of a caveat to that. I think there's a thing about using – doing something for learning purposes and there's another thing about shipping things. What's interesting with Funcadelic and what's interesting about a lot of these ideas from functional programming is that I think they give you benefits that you might have not previously thought. Like for example, if you're going to concat two strings together, doing it with append is probably the most robust way of doing it, relative to just being able to use [inaudible]. CHARLES: Yeah, that's true. You wouldn't want to use a string [inaudible], wouldn't you? TARAS: Yeah. But there are areas when using – there are times when things are just not possible otherwise. Like for example, if you wanted to treat an instance of a class in an immutable way, this is simply not possible in any way. So, if you're going to say, “I'm going to work with a bunch of these instances of ES6. And I want to keep them as instances, because they have certain behaviors that I want to have. I want to have a getter, or I want to have a method on it. And I want to keep these things fully full instances, not broken, not be turned into objects. I want them to be normal instances but I want to work with them immutably. When I need to make a change, I want to get a new object and not modify that object.” So, if you set yourself that goal and you say, “This is what I'm going to do,” then you really are not left with very many options. You only really have, you have to use append from Funcadelic. And because alternatively, you're going to implement something yourself. You might as well just use append. And [I think] that's a good place. I think if you're starting to, if you need to make something lazy, if you need to delay an execution of something – so, instead of pushing that execution, instead of using object assign and then computing everything ahead of time, you can use append. And you can create a getter and you can delay the computation of that value until the point when the user actually reaches for that value. If you want to start doing that kind of stuff, you really are not left with very many options. And append is the way to do it. But that's the thing, is when you start to set these kinds of standards for yourself, you level up in a way that is very significant. I think it's like a natural progression of learning. You start off learning and anything goes, as long as you can make this website work, it's like, “I'm happy.” And then over time you get better and better at it. And then when you get good at building applications, your next step might be like, “What if I was stricter about it? What if I could actually – what would that open up for me? What would that allow me to do?” I personally think about it that way. CHARLES: Yeah. I think that's a good way to think about it. You mentioned having a network for discussing these concepts and trying to internalize them. Let me first and foremost offer myself as somebody. If this is a hill that you are interested in climbing, and I think it is a very worthwhile hill to climb because of the perspective that you will gain from its summit, please reach out to me. You can contact me at cowboyd on Twitter or cowboyd@frontside.io. I'd be happy to discuss these kinds of things, because I think that these tools are just incredibly powerful and will improve you. So, if folks want to get in touch with you, Taras, where would that be? TARAS: I'm tarasm@gmail.com and tarasm on Twitter. CHARLES: Alright. Well, thank you everybody for listening. And as always, if you want to get in contact with us at Frontside, you can just email us at contact@frontside.io or give us a shout on Twitter at @TheFrontside. Thanks everybody. We'll see you all next time for episode 100.

This Frontsider panel episode explores what virtues go into making quality software, such as having tests, making sure software is performant and accessible, and why you should try to avoid technical debt. Transcript: CHARLES: Hello everybody and welcome to The Frontside Podcast Episode 98. My name is Charles Lowell, developer here at The Frontside and your podcast host-in-training. With me today, we're going to have a round table, a Frontside round table. With me today is Elrick. ELRICK: Hey. CHARLES: Joe. JOE: How you doing? CHARLES: And of course, Will. WIL: Hello, hello. CHARLES: Welcome, y'all. We're going to be talking today about some of the things that we do around here, aside from trimming the shrubs and making coffee and snacking on Altoids. Like, way too many of them. Yeah. I was thinking we could talk a little bit about software qualities of relative things, like this software has these qualities. And I think that that kind of lofty goal of software quality is comprised of having a bunch of little qualities. The quality of having fewer bugs or the quality of having these things. And so, talking about all these things that we do and kind of what we do to make sure that we continue to do them. Or the ways that we can ensure that our software has these things. So yeah, we can just start really anywhere. WIL: Yeah. So, one core thing is obviously tests. CHARLES: That kind of falls under we want to have – really, there's two qualities there that we want, right? Is we want to have… WIL: Maintainable software. CHARLES: We want it to be maintainable. We want it to be resilient to change. And we want it to work properly, right? Yeah, so we put tests in place to make sure that that happens. JOE: Tests also inform design in a really positive way. A lot of the time, anyway. WIL: Another thing that we like to include in our apps is responsiveness. CHARLES: Yeah. And just making sure that you have – that it works on a multiplicity of devices, right? WIL: Yeah. And not just the devices, but browsers as well. CHARLES: Yeah. And it turns out it's actually really hard to do that after the fact. WIL: Right. JOE: Yeah. CHARLES: Making sure that lots of browsers, lots of devices. Because yeah, sometimes you have some weird screen width that is on some weird device, and making sure that that works. I guess there's some overlap with testing there, too, isn't it, right? Like you want to be running your tests on those devices at those resolutions to make sure that they're going to work. This is something that we aspire to but I don't think we're quite there yet. It was making sure that our applications are accessible. WIL: Yeah. JOE: I'm very excited to learn more about this as we get into this, yeah. CHARLES: Right, right. And asking the question, how is it that we actually can ensure our applications are accessible? We have very paved roads for making sure that our applications are resilient to change and that they have low bug rates and that they're well-designed via testing. But what is the analog of testing for accessibility? What's the way that you can put those guardrails in for accessibility? I have no idea. And that's an ongoing conversation here at Frontside. JOE: So, I guess I'm curious as to what technologies are actually involved in accessing a web application in – would it be reasonable to say a non-traditional way? I know there's such things as screen readers, but is that all we're talking about? Or what is the ecosystem that we have to consider supporting? CHARLES: I'm certainly not an expert on this. We'd have to get Rob in here to chew our ears off this. JOE: Yes. CHARLES: But from what I've picked up from him and from our conventions with Marcy Sutton and some other folks that we've had on the podcast, it's a big umbrella. So, it's anyone using an application in a non-traditional way. So, whether that can have to do with limited vision, hearing, movement, range of movement, cognitive ability, it's a gigantic whale of a domain. WIL: Yeah. The topic of accessibility can definitely be several podcasts on its own. CHARLES: Yeah. One thing that we've talked about is it would be great if you could drive your test suite through a screen reader or something like that. What would that even look like? There are a couple of open source ones out there, but they're Windows-only. I think it was NVDA was the big one. And then you have a screen reader that then drives the applications in your operating system, so it's going to vary per operating system. So, making sure that it's accessible on Windows, at least as I understand it, is very different from making sure that it's accessible on a Mac. JOE: Yeah, it's like a whole other layer. And it's like BrowserStack outside of the browser. CHARLES: Right. WIL: There are things that you can do from the beginning that will make it easier when you get to that point. It's just like using semantic HTML, knowing when and how to use proper aria labels. All these things, if you do it from the beginning, it's not as big of a task as bolting it on afterwards. CHARLES: Right. And I think we do have a leg up when it comes to web applications. It's within our power to change. There are cross-platform of those technologies. But as you said, it's important to put them in from the beginning. Because as we've seen, for each one of those categories, you're accumulating debt if you don't address it. So, there's technical debt. But I think that technical debt can [inaudible] into a bunch of different areas. So, there's technical debt in terms of the internal quality of your architecture, the way your software components talk to each other. And I think that that's what people mostly think of when they talk about technical debt. But I think in terms of responsiveness debt, there's a slice of the technical debt pie that has to do with making your application responsive. And so, if you don't address making your application responsive, you're accumulating debt and you might not know it. And if you're not making your application accessible, then from the beginning you're accumulating debt. So that if you have to go and try and figure out your accessibility story six months, a year, two years, you might actually uncover and say, “Whoops. I've been swiping the accessibility credit card. And holy crap, with all this. All my fines and penalties and compounded interest. Now I'm accessibility bankrupt.” And that can be scary, right? WIL: Yeah. And a lot of people don't realize with all this debt after the fact is they think they're going in and adding things like responsiveness and accessibility and tests. But really, you're also taking away previous work that's already there, things that need to be refactored. If you put these things off, you're not just adding a few hours of time. You're inflating your time exponentially. CHARLES: Right. Right, exactly. It can be intimidating but I think it's also empowering, because technical debt is like a scary subject. But if you're like, “Oh, we can actually slice our technical debt into a bunch of different categories and address them individually,” just knowing that this is an area where debt can accumulate, that's half the battle. Because the worst thing is debt you don't even see. ELRICK: Yeah. WIL: I mean, [inaudible] is big. That's a big part of accessibility, that, is most people don't think of accessibility. So, that is a huge debt that a lot of companies don't see. JOE: What about something like internationalization where I feel like I've never been in an application where that wasn't punted on to some degree. That's kind of a well-known problem, but it still takes a back burner. Do you think that if accessibility had more exposure as a concern, would it actually get the attention it deserves or is it kind of destined to, “Oh, we'll get to those yaml files later. We'll send those off for translation later,” that type of thing. ELRICK: I don't know. Sometimes I feel as though people feel as though they're trading speed away when they're building applications when they go to implement these things. Like, “Okay, well we're not really going to touch on these right now because that's going to slow us down from pushing out features.” Which is not really true. Because if you don't settle on these things early, you're not really building a solid foundation for your application in the long haul. So, I think people are like, “Oh, we'll just do it later.” CHARLES: Right. ELRICK: And, “We'll just ship features now.” CHARLES: Right. I think that's exactly right. It has this kind of secondary effect where not only do you develop the debt but you develop a culture of accumulating debt, right? Like when it comes to people getting a hold of their finances, the first thing that they have to change is they have to change their spending habits. And that can be the hardest thing. It's not just balancing the equation. It's like saying, “I need to readjust my thinking about this.” ELRICK: Yup. CHARLES: So that I'm not consistently put in this situation again. JOE: So, there's an operative word there, right, in personal finance in that usually if a company is addressing technical debt especially down the road, something that they've punted on for a while, it's far from personal. There's a board of directors or there's a special interest group involved. There's people who want features that are putting money into it. There's a lot of pressure as the company grows and more people are involved. Priorities are more likely to be lost, I guess. CHARLES: Are you saying it can be hard when your culture is spread over that many people, it can be hard to shift? JOE: Absolutely, yeah. And I guess to keep with the dash-first thing, ideally were we starting a company, we would want to start a culture for this company. A culture that recognizes the vulnerability that we all have to technical debt as applications grow. We want that upfront. But the reality is, you know, startups are eager to get things out. Companies that have been around for a long time have high-paying clients that they depend on that want certain things. And yeah, I guess I'm just saying that it has to come in from the beginning. CHARLES: Yeah. And I think that – I don't want to completely disparage technical debt entirely, because technical debt like actual debt, like financial debt, is a powerful tool that you can wield. But it's also, it's like a table saw. You can also easily slice your finger off. It doesn't mean that it's not a useful tool, right? If anyone's bought a house, it's really great that you can borrow money to buy a house. It's great that businesses can borrow money and get small business loans to get bootstrapped. And that benefits us all to have that community. I don't think that – yeah, startups definitely, they need to have technical debt as a tool that's available to them. But they just need to understand the consequences of it and be able to get a hold on it. JOE: That's a super interesting take. I never considered it that way before. CHARLES: Yeah. It's definitely not my take. I actually think the person who coined the term ‘technical debt', that was the original idea. But then people realized that technical debt can also get way out of hand. WIL: It's just like real debt. If you're not paying down a certain amount every so often, it's going to keep growing. CHARLES: Yup. You're going to have to declare bankruptcy at some point and throw out the piece of software if you don't pay a down. And that's going to be more expensive. ELRICK: Yup. That's definitely true. So, I have a question. And we see this all the time repeating itself at various companies, whether it's a startup, a large company, where they put off testing and mobile-first, user-first, accessibility-first. Like all the firsts, they just toss it to the side. Why do you all think that that happens so frequently? CHARLES: I think it comes into people not understanding that if you don't address it from the start, it won't happen naturally. There is a prime motivator that has to happen. If you don't imbue something with those qualities when it's tiny, when it's a tiny seed, a tiny crystal, you're going to have to drill through layers and layers and layers of core to put it at the crystal to begin with. I like to think of software as kind of like a tree. And we eat the fruit of the tree, and that's the features that users use. And we can tell that a fruit is delicious merely by placing it in our mouths. And we can tell what fruit is bad. But we can't really look at the fruit itself to say what caused this fruit to be good, what caused this fruit to be bad. We have to look at the tree. And I think that that's what people miss when they're developing software, is that what you really want to do is you want to build a tree that builds good fruit. You can't just take the fruit off the vine and say like, “Hey, I've got this peach but it doesn't have enough sweetness. So, I'm going to take a syringe and I'm going to inject glucose around it and make it less tart.” You say, “I want a sweet fruit,” right? ELRICK: Yeah. JOE: You could probably actually do that. CHARLES: You could. And that might be a strategy. And we see a lot of software that has those qualities of, “Oh, we're going to make this accessible,” or, “We're going to try and make this beautiful.” I happen to think that pigs are adorable animals and look great in lipstick. But that [laughs]… you could put lipstick on a pig but people can tell. And you can say, “Oh, this peach needs to have softer fruit,” and you can whack it with a mallet to actually make the meat more tender. But people are going to be able to tell. So, what you really need to do is you need to care for the peach tree rather than worry so much about the fruit. Because if you have a healthy tree, then you will have healthy fruit, right? ELRICK: Yeah. So, you want to plant good seeds. JOE: Yeah. WIL: Back to you question, Elrick, about what motivates startups and other companies to put off these things. I think the biggest thing is just time and money. They have this misconception where they're saving a little time and saving a little money now just to add it back later. But in reality, it's going to cost them tenfold time and money for adding it later, versus just spending that little bit of time and money and all that to begin with. CHARLES: That's true. JOE: It could also boil down, as far as just personal intimidation. Not so much like a business side of a thing but maybe just, think of all of the things that you listed, Elrick. It was almost a dozen dash-firsts in there. If you're sitting down at a startup that you started with three friends and just approaching these things for the first time, that's a lot to tack on right upfront. It's intimidating. CHARLES: It is intimidating. I think my message to those people is I've felt intimidated by that. I think my message to those people is like, the nice thing about it is if you attack those, if you tack all of those things from the get go, the features will take care of themselves and feel more effortless as you go on. You say like, “Oh, well actually, I don't worry about a high rate of bugs.” I want to say recidivism, but that's not the right word. A high rate of return, not on money but on – or high rate of bouncing your users. You don't want that. And if you bake that in from the beginning, parts of the software development cycle that were stressful before just aren't stressful anymore. So, if we say, “We want to have a system that is easily maintainable, well let's put that in from the very beginning.” We say that a lot. We deploy to production on day one. But what that means is, we say we have this value that we want the system to be easily maintainable. And so, we're going to do it from day one. That means that we actually – it's not something that we worry about so much on down the road. Whereas that used to be very stressful. I don't know. I remember when I started my career, there were these long release cycles where every six months, you'd release software. And the last month was just absolutely terrible as you try to stand this thing up and get it into production and then realize it's not monitored. There's no one checking the health of this thing. So, it's pissing off users at one in the morning. And… WIL: Beepers. CHARLES: What's that? WIL: Beepers. CHARLES: That's actually a great – there's a story there. The one time I got a beeper, I went canoeing in the canals of London and I tipped over my canoe and I dropped both my cellphone and the beeper that they've given me. ELRICK: What? CHARLES: I never got put on pager duty again. [Laughter] JOE: I'm going to use that next time [inaudible] with an on-call position. That's a good move. CHARLES: I remember, I definitely remember how sour my manager's face was when I turned [inaudible] the cellphone that was like, dripping with water. JOE: He was eating bad fruit, probably. CHARLES: Yeah. [Laughs] So, the other thing is we like to build beautiful applications, right? So, you have to – that match the user experience. You have to spend that time on design and beauty upfront. You will not have a beautiful application after the fact. You just need to bake it in. ELRICK: And accessible design. CHARLES: Exactly. ELRICK: Don't forget that one. CHARLES: Don't forget that, right? A responsive design. WIL: Yeah, accessibility-first in design. Yeah, responsive and all that starts in design phase, yeah. CHARLES: Yeah, all that, right? So, you want a great experience. You want an accessible experience. You want a responsive experience. You want a quality experience. You want a performant experience. That's another quality that you say. Like, “We're going to make sure that this is performant.” If you want that – and that's something that we're not always great about, right? We don't actually put in benchmarks for our software from the get go. But maybe we should. But there's perhaps a hidden cost there that we might be actually accumulating performance debt that we don't even know about. JOE: That's true. ELRICK: Interesting. JOE: So, things that pop up that are new. Like, accessibility wasn't probably always a thing in computing. Internationalization probably wasn't always a concern. Beautiful certainly wasn't a concern if you look on Wayback machine. You will see that to be true, right? [Laughter] JOE: So, all code is tech debt, I would argue. Or at least has the potential to be. And yeah, as the ecosystem as a whole evolves, being responsive to that, having plasticity in that respect, sort of like meta-first. CHARLES: Right. JOE: That could be the real challenge. WIL: Yeah, Charles is mentioning all these experience things. And so, I was thinking X-first is simply experience-first. You want you users to experience a certain quality of your app. That experience needs to start in the conception phase. CHARLES: Yeah. ELRICK: That's true. And even your developers coming in, developer experience. JOE: Yeah. CHARLES: Right. And I think the core of that X-first, that experience-first, is you need to pick which experiences. Because you can't have everything. JOE: Right, yeah. CHARLES: One, there is going to be too much. You have to say, “I'm going to sacrifice on knowing that this is a performance thing. I'm not going to include that in the core DNA of my application.” And there's just going to be things that you don't know about yet that are just unsolvable problems or that don't necessarily work. And you can say, “You know what? Hypothetically, I'm not going to make this an accessible – I'm not going to focus on accessibility.” But then you need to own that. And you need to know that you're accumulating a huge amount of debt around that. And then I think that is a particularly bad trade-off because someone's always going to come along and you're going to have to know that your application is accessible. I think once we clamp down on that, that's going to be something that we have a strategy for and we include at the beginning on every single application, right? ELRICK: Yeah. CHARLES: But I think you need to have, almost like holding the cards in your hand, say, “These are the cards. These are the X's that I'm going to have in my hand. And they are going to be core to my app.” And they're going to be part of the DNA of that tree. So that I know that the fruit is then going to have those qualities. JOE: And then you as an engineer, that goes through an iterative process as well. Just starting out, you have no idea what that DNA should look like. And short of learning from people who are wiser than you who are around you, and reading blog posts and whatnot, really the only way to know the pain of strong-arming internationalization for instance into a 15-year-old Perl application, is to go through it. And then, you know, future trees will not have this DNA. CHARLES: Right. Right. And that's the other thing. Is if you are going to include, if you are going to try and splice something into the DNA, there's a lot of work. And you just need to go for it. You acknowledge that it's going to be a lot of work. And you need to, you just need to own it and go for it. And pay that expense of actually getting it deep, deep, deep into your application's core values. So that then, you don't have to worry about it anymore. Otherwise, you're going to be paying – you're just basically signing up for a lifetime of debt. Right? WIL: Yeah. And then to make the debt analogy even more, it's like people don't understand the total debt. The end debt. People get a $30,000 loan with a 4% interest and they think they're paying back that $30,000 loan. But really, they're paying back $36,400 after all the amortization of their interest. The debt is higher than you can see, always. CHARLES: Right. WIL: And it's true in tech debt, too. React is the new hot thing now, but in 10 years we're going to be on React debt that we're migrating away from. JOE: I hope so. [Laughter] CHARLES: Maybe less, I think less than 10. WIL: Yeah. The debt is always there. And people don't realize how much they have to pay on top of what's visible. JOE: Yeah. It's an invisible vig. CHARLES: What's a vig? JOE: It's interest, in the mafia. CHARLES: Oh. JOE: Sorry. Yeah. CHARLES: I forgot you're Italian. JOE: Yeah. ELRICK: So, for people that are listening, they might be in a situation where they need to advocate to the powers that may be these X-first values. What do you all think that some of the approaches that they should take to say to whomever it is that, “We need to do this first”? Because there's times where you might say, “Hey, we need to do this first,” and people just look and say, “Oh, maybe not.” Then you need to push back on that. CHARLES: In my experience, I find that the tech debt argument is a good one. Because I think it can be, it's both limiting and empowering. Because sometimes it really is the right call to pull out your credit card and put something on it. If you need to buy water and you need to buy food and you don't have any other means, man, put it on the credit card. Right? Seriously. Even if you have no idea how you're going to pay it back. Like, whip that sucker out and stick the chip in. And it doesn't matter how much it costs. And so, sometimes that is the right call. But I think draining it of a moral or a value as a human person thing, and approaching it from a business decision and saying, really trying to attach a cost to it. Because then I think if you can drain out the emotion of it, because people really want something. They're striving to go get it and trying, give them tools to think about it rationally. That I think is a good strategy, to just say, let them know that there is a debt that's being paid here or that's being accumulated here. And it's really large. And maybe even say, “Look, if we were to put this off by six months, this might cost not twice as much. It might cost ten or even a hundred times as much.” So, by saving $5,000 now, you might actually be accumulating $50,000 worth of debt. It's [bigger] than you think. But I do like – so, I think that's one important tool. But I think then also the other important tool is to say, “If we are going to attack this, let's drive it home. Let's put it at the core. Let's make this a value that we hold so that the tree can take care of the fruit itself.” So, if we say that we're going to put in accessibility – because not all projects are greenfields. JOE: Absolutely not. CHARLES: So, what's the message to them? Sorry. You're just SOL. I think if you're a year into a project, two years into a project, and you realize, “Oh no. We need to do internationalization,” recognize that that might be something that's – that's a pillar of your architecture. Or, “We're going to make this application accessible,” don't half-ass it. WIL: Weave it in. CHARLES: Say, “We're going to transform this. We're not going to add accessibility. We're going to transform what we have into an accessible application.” Or, “We're going to transform what we have into a beautiful application.” Otherwise… WIL: Yeah [inaudible]. CHARLES: I would say leave it ugly and focus your efforts elsewhere on things where you do have your values straight. Because you're never going to have everything in line. JOE: No. WIL: Treat software like immutably. You don't add something to it. When you want to add accessibility, you're creating a whole new accessible app. ELRICK: Ooh. That's deep. CHARLES: Yeah. JOE: So, having seen – I don't know. I think it was very apt, looking at it as a business decision. I've seen it go the other way. Because at least among engineers and people on the technical side of it, this can become a very strong moral issue that people feel very strongly about. CHARLES: Because we have to live with the consequences quite honestly, right? JOE: Exactly. And that's a hard thing to translate to say an executive board that may be three levels abstracted away from you and is making those decisions. I've seen people attack or approach this I guess with that emotion built in, with the, “This is the right way to do it. Everybody else is doing it wrong.” It gets nowhere, basically. What needs to happen I think, so you talk about having this beautiful tree. But that also requires beautiful gardeners. And so, where the moral thing or the interpersonal thing comes in is there needs to be kind of an inclusive and encouraging environment that is fostered among the people tending to the tree. And that's a totally separate thing than selling the business value of it. Those things should be completely divorced. CHARLES: Yeah. It's funny. It's always hard to reconcile those two things, right? Because on one you have, “You have to take care of the raw consumption of material and the output of product.” But then also trying to – so, there's some baseline math that has to happen but making sure that that goal, it doesn't slice people. And can enable them to be happy and feel like they're doing good work. And that the things that they're doing is having meaning. It's probably an insoluble problem that we're going to be dancing around for as long as people are around. If there's one thing that we've come to recognize around here, and we've stated it many different ways from a bunch of different angles through the course of this conversation, and I would say through the course of this podcast, but that is if you want to see something in your software, make sure that you attack it from the get go. ELRICK: Intertwine it in your DNA. CHARLES: Exactly. And then you can actually experience the fruit, rather than trying to always, always trying to jam it and change it and get it into the taste you want after the fact. So, I guess that's it. Thank you so much y'all, for this conversation. I really, really enjoyed it. For those of y'all listening, if you want to continue the conversation, you can get in touch with us we are @TheFrontside on Twitter. Or you can drop us an email. We're contact@frontside.io. So, thanks Elrick. Thanks, Joe. Thanks, Will. JOE: Thank you. WIL: Thank you. ELRICK: Yup. It was great. JOE: It was fruitful. [Chuckles] ELRICK: Frontside-first. CHARLES: And well, we'll see y'all around.

Erich Gamma: @ErichGamme Dirk Baeumer: @dbktw Show Notes 01:11 - The Design Patterns Book 02:45 - The Eclipse Project 09:24 - Language Server Protocol: Overview 15:16 - What can you do with a server that implements the LSP? Incremental usage? 20:12 - Keeping the Tools in Sync and Refactoring Support 24:33 - Keeping it Performant 29:41 - What kind of proliferation of codesmart tools are there that implement the LSP? 34:51 - What are the challenges encountered trying to build abstractions that work for 40 different languages? Resources Visual Studio Code Transcript CHARLES: Hello everybody and welcome to The Frontside Podcast Episode 97. My name is Charles Lowell. I'm a developer here at The Frontside and your podcast host-in-training. And with me today, we have two very special guests. They have been working on technologies that have run very parallel to my entire career as a software developer. And we're going to talk about that. So with us today are Erich Gamma and Dirk Baeumer who are developers on the team developing VS Code, which if you're in the frontend space is taking that area of development by storm. It's just amazing, some of the things they can do. Lots of people are using it every day. Lots of people are trying it. And so, we're going to talk about the technologies that underlie that and the story of how it came to be. So, welcome Erich and welcome Dirk. ERICH: Hello from Zurich. CHARLES: Alright. Zurich to Albuquerque. Here we go. As a first start, I would have to say my first contact with this story, I at least have to mention it because – and this is for Erich – you wrote a book that was very, very instrumental in my formation as a young developer. I think I was about 22 years old when I read ‘Design Patterns'. And I don't know. I still carry a lot of those things with me to this day, even though a lot of things have changed about the way that we do development. I still carry a lot of those lessons, I think especially things like the state pattern and the strategy pattern, and stuff like that. I want to move onto other things, but I was hoping that we could talk just a little bit about, what are the things that you find still kind of relevant today? ERICH: Well, now as you said, some of the things are kind of timeless and we're lucky to have found these things. And I still love all the patterns. But I must say, things have changed, right? So, at that time, we thought objects are very cool. And as we have evolved, all of a sudden we think, “Oh, functions are actually very cool, too,” right? Closures and so on. So, I think we got more broader and of course if you use functional programming, you have many more patterns available as you program. So, I feel some of the object thinking still applies. But that's not the only thing that counts anymore. Today it's functions, stateless, immutability, and all those things within functional programming which is [straight] and which [inaudible] in our team. CHARLES: Yeah, yeah. I would love to see an update to how do these concepts transfer into functional programming. But anyway, just wanted to say thank you for that. And it was about the same time that, a few years after, I don't know the exact same timing, I want to wind back. Because we're going to talk about VS Code but before VS Code, there was a project that both of you all worked on called Eclipse, which I also used. Because at the very beginning of my career, I did a lot of Java development. And it really opened my eyes into a level of what tooling could do for you that I didn't see before. And I was wondering how did you arrive to there? Because before that, I was using Emacs and Vim and Joe's Editor and things that were editing the text files. And how did you kind of arrive at that problem? Because I feel like it's very similar to the one that VS Code solves, but this was what, 15 years ago? ERICH: I think it's older, right? CHARLES: Really? DIRK: It's 17, 18, yeah. Yeah, yeah. It was end of the millennium, right? So to be honest, Eclipse wasn't the first development tool we worked on. Then, we worked on the company ObjectTechnologyNational. They worked on Smalltalk tools. And of course, Smalltalk had a great IDE experience, right? So back then, Java became popular. One idea was, how can you preserve the great Smalltalk coding experience? [Inaudible] CHARLES: Ah, okay. DIRK: [Inaudible] and find all references, method-level history versioning, and so on. So, that was the input that got Eclipse kicked off. And one idea we had at that time, Eclipse is our opportunity to make everything right. And as we have seen now, when we did VS Code, we could even improve what we have [inaudible] at that time. So an example, in Eclipse we thought plugins are very cool and we have kind of a microkernel. And you load all of the plugins in the same process, they have a rich API, and so on, which is great. But we found over time, if you have lots of plugins and they do bad things and they run in the same process, it's not the best thing. CHARLES: Ah. Right. And so… DIRK: [Inaudible] have a different architecture. We believe now in isolation, separation. So, we now run extensions in separate process that communicates through RPC with the IDE so that we are in full control. And we can always say you can save the tool, save the document, no matter how bad a plugin behaves and decides to do an endless loop. Because in a separate process, the hope is still one CPU is open, available for you, that it can be safe from the other process. So, that's some example, right? Eclipse has done many things right, but the multi-process architecture I think is a major switch. And the other major switch is at Eclipse time you think Java is cool. Everything has to be in Java. CHARLES: Right. DIRK: No longer think like that, and that brings up this other topic of then the language servers that we can also talk at some point. CHARLES: Right, because that's the thing, is VS Code – now I've primarily been exposed to it through JavaScript and TypeScript development. But it really, it's designed to support all kinds of different languages. So, the C++ support is really good. The C support is really good. And I assume the Java support is really good. Is it safe to say? Because I only ever used Eclipse in the context of Java. Did Eclipse gain kind of a wider acceptance further beyond Java and C++? ERICH: Yeah. I think it's fair to say Eclipse has a rich ecosystem. Yeah. I think with all the tools. And it will be interesting to see that you can close the loop, because for Visual Studio Code, when you do Java development, you actually run Eclipse behind the scenes. That's how we kind of smiled at each other, Dirk and I, when he said, “Now we close the loop.” We started with all JavaScript and then we integrate Eclipse using this language server protocol and that's how we close the loop. CHARLES: Ah. DIRK: So maybe one thing I would like to add is that when you look at Eclipse and the tool and framework landscape that existed in the Java time, at that point in time when we started with Eclipse, it was very well-defined. There was Java. It was a well-defined set of libraries you were using and frameworks you were using. And if you look at the programming and tool landscape you have today, in months you see a new framework for JavaScript popping up or there's something else or another cool X, Y, Z thing. So, the tooling you build today has to be a lot more open to these new inventions, especially since they occur in a higher frequency than they did in the past. And that had influence on how we architected Visual Studio Code to give people a lower barrier of integrating their stuff into Visual Studio Code than you typically have in Eclipse. In Eclipse you needed to program in Java. With the LSP you can program in any programming language. In Eclipse, if you really want to try to do something nice with code complete and stuff like that, you had to hook up a lot of stuff. So, we raised that to another abstraction layer where we more talked about what people provide on data and we do a lot more for them in the user interface than compared for example to Eclipse, which lowers the barrier for people to integrate languages in Visual Studio Code than the barrier you had to integrate something in Eclipse. And so, [inaudible] for that one was that there are a lot more tools and programming languages out there that have importance than 10 years ago. ERICH: I'll give you an example. So, when we did C support in Eclipse, and it was also the team that seeded it. Of course, it took over and has now a great community behind it in Eclipse. But you wrote the C tooling in Java. And of course, that means you built the parser in Java and then of course, there are great C parsers around, C frameworks. But also it means you cannot dogfood what you write. You write Java but you don't program in C++. I think which is what makes VS Code so appealing is we are a very aggressive dogfooder. We want to use ourself and of course [inaudible]. That's why [inaudible] is very good. The C++ guide, they programmed C++ and they write in C++ so that's how they make it very good, that you have this feedback loop. CHARLES: And so, what's an example? We've talked about this low barrier of entry. So, if I were wanting to say, I do mostly programming in JavaScript. Let's say I wanted to add, I know all of this already exists, this infrastructure already exists, but let's say I wanted to add smart editing to JavaScript source files. What would that process look like for me as a JavaScript developer? DIRK: To be fair, whenever it comes to language services, it's never easy. But [inaudible] lower the bar. A language always means you have to do parsing, you have to do [9:59], type bindings. You have to make it fast, scale high up, and so on. So, this is never easy. But I think if you think about the different steps you can do, the first thing, let's not take JavaScript. Let's take a new language. CHARLES: Okay. DIRK: Your new cool language. CHARLES: Or maybe we take a Lisp or something where writing the parser is very easy. DIRK: Even that, you have to resolve symbols and so on. CHARLES: Okay, okay. DIRK: Even the parsing [inaudible]. But yeah, let's take a fancy language like Lisp or whatever. So, the first level I think is you want to get some nice coloring. That's the first level. CHARLES: Yes. DIRK: So, you get some coloring. And what we do there actually in VS Code is we tap into the community from TextMate. So, we use TextMate grammars to support colors in languages, which gives us access to a long [10:51 tail] of languages. So, to change the [10:54], if your language is not too exotic, you will find the grammar that describes how to color, what the tokens are in your language, and then you can get your language colored. That's step one. The next step is of course you want to get smarts like IntelliSense and so on. Ideally of course you can say, “Well, maybe there is something already around that has abstracted the parser and you can use this library.” CHARLES: Right. Because there actually are a bunch of JavaScript parsers written in JavaScript. I know I keep coming back to JavaScript, but let's assume with this language that we've got. I may not have to write a parser but I've got one. ERICH: You've got one, exactly. You've got one, right, and then technically it's not in the same language as the tool. So, that's why I don't want to go too much into JavaScript because for instance VS Code is written in TypeScript, which [transpiles] to JavaScript, which moves a little bit, makes it not as convincing as it could be. So, let's say it's a different language. Your fancy language is written, has a parser in your fancy language, which is different than the language of VS Code which is JavaScript. CHARLES: Right. ERICH: So, then the next level is to say, “Okay, well you have your code you encapsulate it in a server that you can talk to through some protocol.” And now the challenge is what protocol do you talk to? Typically in the language, the library you get, it will use some ASTs, symbols, type bindings. And what Dirk mentioned with lowering the bar is that assuming you have those ASTs, the way you talk then with our tool is through a protocol that is not at the level of the ASTs but at a higher level. CHARLES: A higher level than the ASTs. ERICH: No, yeah. A higher or simpler level. Let's give you an example. You want to find the definition of a symbol in your fancy language. The way the protocol works is you only tell it, in this document with the URI, at this position, I want to find the definition of the symbol that is this position. The request goes over the wire to the other process. Document URI, and the textual position. And what comes back of course now in the server you used AST, you find the symbol, you find the binding of the symbol which means it gives a definition for it. Of course you use your AST to analyze it. But then what gets back to send over the wire is yet another document, the reference, and the position. CHARLES: I see. So, you're really like pinpointing a point in just the raw bytes of the document. And you're saying, “Look, what is here?” And you just want to delegate that completely and totally to this other process. So, the IDE itself doesn't know anything about the document? ERICH: It knows about the document, right? CHARLES: I mean, it knows about the textual positions of the documents and the stream of characters, but not the meaning. DIRK: True. The smarts are in the server. And you talk to the smarts at the level of documents and positions. And the [good thing is] it's a protocol, is at this level it makes it easy to integrate into one editor, which is VS Code, but also into other editors. So, that's why we came up with the idea to have a common language server protocol which allows to provide a language not only for one editor but also for many editors. That was a challenge we had in VS Code. Remember when we started, we were kind of late to the game. We said, “VS Code should be in between an IDE and an editor.” But what we liked from an IDE is of course code understanding, IntelliSense. Go to definition, find all references. But how do you get that for a long tail of languages? We cannot do it all ourselves. So, we need to get a community to tap into. [Similar to] like TextMate grammars are kind of a lingua franca for coloring. So, we are looking for the lingua franca for language smarts. And that's what the language server protocol is, which means you can integrate it in different IDEs and once you've written a language server you can reuse it. CHARLES: I guess I've got two questions. What are the kind of things that I can do with a server that implements the language server protocol? And then I guess the – so we've talked about being able to find a reference. And is there a way you can incrementally implement certain parts of the protocol as you go along? ERICH: Yeah. DIRK: Yeah, basically you can. The protocol on the server and the client side talks about capabilities. The server can for example say, “I am only supporting code complete and go to definition and find all references.” And for example, something like, “Implementation hierarchy or document symbols or outline view is not supported.” And then the client adapts dynamically to the capabilities of the server. CHARLES: Okay. DIRK: That's one thing. And the set of capabilities is not fixed. So, we add them. We just added four or five new capabilities to the protocol last week. So of course, we listen to requests that come from other IDEs, what they would like to see in the protocols that we see in Visual Studio Code, we would like to extend. And that's the way we move the protocol forward. CHARLES: Okay. DIRK: It's capability-based and not so to speak version-based. So, [inaudible] versioning at the end of today. CHARLES: Right. You can incrementally say, “I'm going to have,” if I'm starting to write a server, I can say, “Well, I'm going to only start with just find definition at point.” And that's the only thing that my server can do. ERICH: Well, there are some basics, right? Keep in mind you have two processes. And once the user opens an editor, the truth is in the buffering memory on the one process. The basic thing you have to in a language, so you have to support the synchronization of [inaudible]. Once you open a file in the editor, then the truth in the buffer, and then you have to sync it over. CHARLES: Right. ERICH: [Inaudible] close the truth on the file system and you also have to tell this to the server. Because the server has to know where the truth is. DIRK: That's correct. These two open/close handshake methods and change methods, this is the minimum you have to implement. But for example, for Node itself, we provide libraries that help you with this. And the protocol is not very complicated. It's a buffer. Then it's change events. Either it's an insert, a delete, or an edit. CHARLES: So, let me try and get this straight in my head. I think I understand. The problem is that the VS Code, or your code editor, it's actually making changes to the buffer, and it needs to communicate those changes to the server. Or does the server actually make the changes itself? DIRK: The editor does make the changes. So, the protocol is spec'd in a way that as soon as an editor opens a document, the ownership travels from the server for the content to the tool. And the server is basically not allowed to read the state of that content from disk anymore, or get it [inaudible]. CHARLES: Aha. DIRK: Therefore, the client guarantees that everything the user does in that document is notified to the server, so that the server can move the document forward. CHARLES: Okay. DIRK: [Inaudible] we see the close event, that basically with the close, transfers the ownership of the document back to the language server. And it is allowed to re-read that content from disk if it wants. CHARLES: Okay. ERICH: Here, the protocol is really data-driven. Dirk mentioned that earlier, right? So, basically what flows between the server and the tool is data. So, what do we mean by data? You ask for IntelliSense or completions at the line. What follows is just the data. A list of completions that flows then from the server to the client. And then the client decides what to do with this data and decides to modify the document by inserting the completion proposed that the user selected. CHARLES: Right. And then if it decides to make any updates, it needs to send those to the server. DIRK: Exactly. CHARLES: So, if I actually insert the method that I want to call there, I'm going to be inserting nine characters, and I need to tell the server, “Hey, I just inserted nine characters to this document,” something like that? ERICH: Exactly. CHARLES: Okay. And so now how, because I remember now one of the coolest things about the class of tools of Eclipse that I hadn't really seen in the more lightweight editors – I went from Java, like so many of my generation went from Java to Ruby and then to JavaScript – once I moved out of the Java world, one of the things that I had come to expect from my tools was that they would help me make modifications to my codebase at a very high level. So, I would be able, if I had some class that was imported into say five modules in my codebase, I could say, “I want to change the name of this class,” and then it would find the references and then make the updates to those things. So, how do you manage that? So, if I have a class called ‘Person' that I want to change to ‘User', if I change it to ‘User' then it's going to break in those five different places unless I rename it to ‘User'. That's something that was very doable in the Java world. How do you keep the code editor, the tool I guess is what you were calling it, in sync? Like the server is going to make that change or does it just come back with data and says, “Here's the references if you wanted it to change”? ERICH: Yeah, yeah, yeah. So, two things you mentioned, right? Java and JavaScript or course. Java is a typed language which means you have better understanding of the code and what the reference is. In JavaScript which is typeless, you cannot know it as much, so that's actually why we developed also, we're using TypeScript. VS Code is actually [written] in TypeScript which allows you to do these kinds of things like refactorings. But if you look at the language server protocol, it has support for rename. And the way how rename is done is again it just documents positions. You say, “At this position, I want to rename the symbol with this other name.” And then you tell this to server and the server will handle the rename by giving you back a list of positions that need to be updated. CHARLES: Ah, okay. So now, I'm starting to understand what you're talking about when you say data-driven. It's literally just telling the tool – the tool proposes, “I want to do this rename.” And then the server provides all of the information that is required to actually do the rename. But it doesn't actually do the rename itself. It just provides the data. DIRK: A couple of reasons for it. The data effects, at the end of day, it's again, edit, and it's more or less the same edits the client sends to the server when the user types in the document. This is the protocol. On top of it, something that you can create a file or rename a file, this comes as a result back to the client. And then there, since it is a client/server architecture, the whole process is async. So, we have to give the client the change to revalidate if that edit structure that comes is still valid. If it is still valid, the client basically applies it. And by applying these edits to these documents, they will automatically flow back to the server until the client either closes these documents again or saves them. So, the reason being is that some of the tools may even show you a preview. You can only select some of them and apply them. So, there's always an interaction in these refactorings and to make that possible, as Erich mentioned, the whole protocol is data-driven. We don't go the server and say, “Okay, do that rename,” and he writes that back to disk. It computes a set of transformations to bring the current state of the workspace into that new state after the refactoring. CHARLES: I see. ERICH: [Inaudible] be fully transparent. Actually, no. Refactorings, Dirk [inaudible] refactorings for Eclipse so we can go deep on that. What we don't support right now in the protocol, we support edits in the buffer but when you want to rename a class in Java, you also want to rename the file. And that's something we're currently working on to support in the specification of the language server protocol. So, we don't have that yet. But we support code actions, quick fixes, that you like from Eclipse probably. And you can use then to do refactorings like extract method, extract constant or extract local variable, things like that you can do at the level of the language server protocol. CHARLES: Wow. That is… ERICH: I think [inaudible] right now. Let me go back to the Java thing. The Java language server actually has the support for refactorings. And there is now a language server protocol implementation of this Java provided by Eclipse. So, all the support you had in Eclipse for Java or most of the support is now also enabled in VS Code. CHARLES: Right. ERICH: [We don't] really have to reimplement it because you can reuse. And that's the big thought we have. You want to reuse language smarts as much as possible because they are so hard to implement. CHARLES: Right. And so, you can do that because you're providing this abstraction between the tool and the actual smarts, which is really, really cool. I do have to… how do you make it fast? Because you're describing this tool, this client and this server, and they're syncing. They're keeping this distributed state in sync and you know, how do you keep that from coming too chatty? Or is it something that you have to consider? Or is it just, maybe I'm overthinking it because I haven't dealt with it? DIRK: So, at the end of the day, it is chatty. But it is made performant in the way that it's very incremental and partly event-based. So for example, if you type in the document in the editor, you can either decide to [inaudible] sync the full content of the document, which we do not recommend but for some basic exploration, that is something people do. And we have [inaudible] the delta-encoded mechanism. So, we sync the buffer once and then after that you only get the edits the user does. These are chatty of course since the user types them, we debounce them and collapse them on the client side and only send them if we know that the server really needs to know them because we have another request we are asking the server or after a certain timeout. So, there are smarts behind it. But the protocol is kept performant by making it an incremental protocol at the end of the day, and not sending too much data back and forth. ERICH: Right. We don't serialize ASTs. We serialize positions, a list of items for completions. And actually, the transport is just JSON RPC. CHARLES: Okay. ERICH: And actually, someone, there is different usage now for language server protocol. And there is one host, Eclipse J, which brings it again back to Eclipse. They actually run language servers remote. CHARLES: Interesting. ERICH: And if you use it, you can run it on the browser, you get IntelliSense, and of course I guess it depends on how far away you are from the server. But it seems to work, according to feedback we've heard. CHARLES: Really? ERICH: The feedback we heard from them [is pleasant]. So, they use many of the language servers. CHARLES: So, is this a product that they have where the language server is running in the cloud and you send – your entire codebase essentially goes over to the language server and you can export the smarts to the cloud? ERICH: It's one step at a time. So, Eclipse J is kind of, they have what they call cloud workspace, which means the workspace is in the cloud. And [inaudible] code smarts of the workspace in the cloud, they can run the language servers in the cloud. It's a [inaudible]. One user has one workspace, has one language server. CHARLES: That sounds amazing. And if they can make it performant. ERICH: We have done cloud IDEs, right? If you look at the history from Visual Studio Code, you also had our stuff running in the cloud at some point. That's how we started. Before we pivoted to VS Code, we built – our exploration was, that's why the project is six years old. The first two years, we explored how far you can get coding done in the browser. CHARLES: Right. ERICH: And we had some [inaudible] there. CHARLES: So, I've played around with a lot of cloud IDEs and I've found them to be neat, because every few years it comes along. But yeah, it does seem that there are certain challenges that it's nice to have a client running and just be able to have the files locally. And is that a performance thing or if VS Code is written in TypeScript, theoretically it could run in a browser, right? ERICH: Of course. The [inaudible] there still runs in the browser. Then it's used by many tools that run in the browser. Like actually, if you want to edit your source code in the browser, there it's using the same editor that's running VS Code. So, that's how we started. Cloud IDEs, yeah we were at this point. We had our cloud IDE. We could edit websites in the browser, source control them, have a command line, deploy them. What we found is it's great for some scenarios like code reviews or doing small tweaks to files. But when it comes to really development, you use so many other tools. And you want to just have them. And [inaudible] a long tool chain problem. So, as a developer, you just want to use other tools as well. And that's why you can't have them all in the cloud. CHARLES: Right. ERICH: And [inaudible] we said at some point, it was a great lesson we had that you can program in the browser. But now we want to go to have a really [seven by 24] coding, you want to have a desktop experience. So, what we then did, we moved over the code we had run in the browser using a shell, the Electron shell, and can run it on the desktop. CHARLES: But there's theoretically, you could be running your language server for example in the cloud, but everything else on the desktop. ERICH: Yeah. Some people do that. DIRK: Right. CHARLES: Okay. Wow. It's crazy. It's heady stuff. We've talked about the barrier to implement the code smarts is much lower than it has been in the past. What kind of proliferation of code smart tools are there now that implement the language server protocol? Like how many different languages would you say have airtight…? DIRK: So now, [inaudible] time where we don't count anymore. You tell us a language and I can look it up, whether it's supported. Tell me a language and I can tell you whether – no, we have a website. CHARLES: Okay. DIRK: And when I look at it, we have about 40 languages. CHARLES: Wow. That's probably about, pretty much every mainstream language. DIRK: Yeah. I cannot find what isn't there. CHARLES: Yeah. It almost kind of begs the question, is this going to be the new bar for a language? Because I remember when I was starting out, really you just needed to have some interpreter or some compiler to have “a language”. And nowadays, it's not just the language. You need to have a command line tool for managing your dependencies. And you need to have a package system with a public repository where people can publish reusable units of code. And what's become expected out of a language to succeed has upped. Is having a language server implementation going to be part of the bar, the new bar, for “Hey, I'm thinking about creating a language”? I haven't really arrived until I have a package manager, I have a command line for resolving dependencies, I have documentation, and I have a language server. DIRK: I personally think that is our dream at the end of the day, to get there. We know about languages that do so. So, a lot of these language servers come for example from the people that developed the language. For example, the WASP guys, they do the compiler and they actively work on their language server as well. So, at the end of the day, the advantage of that approach since the WASP language server is written in WASP and runs in WASP, they can reuse so much code that they already have written in WASP. That's easy for them to package that up in the server and basically the people that maintain the compiler, at least the same team, maintains the language server at the end of the day. ERICH: And that's why we call [those] a win-win for the language provider. Because if you implement the language server using the language server protocol, then it can be integrated easily by the tool provider. And it's a win for the tool provider since there is a common protocol across all these languages you have to support. You can write an implementation once and again benefit and support many different languages, which makes the matrix problem one language support for each tool into more a vector, right? It reduces the matrix into a vector. You only write language servers that get integrated into different tools. CHARLES: Right. DIRK: And [inaudible] especially I think appealing for new languages that come out, because it lowers the bar for them to get into existing tools. Because if they write a language server speaking the language server protocol integrating that at the end of the day in Visual Studio Code is basically packaging up an extension for Visual Studio Code and writing 20 lines of code. CHARLES: Yeah. DIRK: And same [inaudible] for other IDEs that exist where people implemented the language protocol client side for the tool, for example. For vim or for Atom. CHARLES: Yeah. DIRK: So, new languages I think definitely, we see that trend go onto the language server protocol because that gives them an entry point into a large tool community. CHARLES: Yeah. I'm really excited about it. I'm actually an Emacs user. And that's actually how I found out about LSP, was in my Emacs newsfeed I saw that someone was starting on LSP support, and got digging into it. And I think that one of the problems that has plagued not only Emacs but all these editors is what you're describing where for example the JavaScript support was really great – is really great – in Emacs. There's refactorings. There's IntelliSense, code completion, all that stuff. But that's because someone wrote an entire JavaScript parser and code smart system in Elisp, which is just an absurd hurdle to jump over, to expected. And so, what you expect out of your editing experience, like when I went to try – if I were to go to try Python, well it's not nearly as good as what I'm expecting. And so yeah, I think it's exciting to hear what you're describing where with having some shared set of abstractions, you can offload all of that code smart onto the community that's building these new tools so that they're really easy to integrate into your environment. I think it's really exciting. Although it does make me ask – and I think we've got time for one more question – is we've been talking about all these different languages. Java, C++, JavaScript, TypeScript, Ruby, Python, et cetera, all these, the 40 languages that you talked about that have this implementation. What are the challenges that you've encountered trying to build abstractions that work for 40 different languages? All with their different syntax, all with their different conventions. It sounds like when aside from the fact that you've actually done it, I would say it's impossible. So, I'm curious. What were the unique challenges to solve there? DIRK: I think we already touched that at the beginning, the appealing stuff of the LSP is that it's not talking about the programming language itself. It's talking about things I can do with source code. For example, requesting code complete, go to definition, find all references. And the data that flows between the client and the server is not in terms of the programming language itself. It's about editor abstractions. We talk about documents and positions. We talk about edits that are applied to documents. We talk about snippets and stuff like that. And these abstractions, since they are programing-language-neutral, are a lot easier to implement for different editors. And the [inaudible] where the [inaudible] would speak AST nodes and symbols and functions and classes and methods, that at the end of the day, would not work. Because if I ask go to definition, the result is not a function or a variable definition. It's simply a position in the document with a hint which range to select. CHARLES: Okay. Yeah. ERICH: [Inaudible] places. In only a few places, we have to really abstract across languages. Like for instance, completions. When you do completions, you don't know, is it a variable? Is it a function or a method? That's where we have to abstract. But that's one of the few places. But again, it's an enumeration. DIRK: Yeah. And that's only to present an [icon]. ERICH: Yes. DIRK: It's only to give you a nice icon in front, because when you insert it, what comes back for completion item is basically a textual edit or a bunch of textual edits that when you select that completion item, we take these edits and apply them to the document buffer. And whether you edit a functional programming language or some other stuff, Prolog or whatsoever, it does not matter at the end of the day. CHARLES: Yeah. That simplicity, and treating it at that simple of a level is what unlocks all those superpowers. ERICH: It unlocks lowering the bar. But of course, if you look at some [of the demands], refactorings, whatever, they cannot easily be funneled. Not all of them can funnel to this low-level abstraction. Then of course, the criticism of the LSP protocol is that if you have already a very rich language service, you might not get it all through the LSP. DIRK: That's true. ERICH: And the [inaudible], that criticism we see of the LSP. But it's a tradeoff, like so many things in software. DIRK: Yeah. But what we learned there looking at different types of refactorings, it's more the set of input parameters that vary much between languages. The result of a refactoring can for every programming language that is at least document-based, [inaudible] in that lingo the LSP speaks. Because at the end of the day, it's textual edits to a document, right? ERICH: So, many people like LSP but there are people that don't like it. And people that have rich language services like IntelliJ, [Cool Tool], and [inaudible], even with LSP we would only get 20% of [our cool] features. Which is a little bit downgraded and not really true. But you see, it's a tradeoff. CHARLES: Right. ERICH: And if you want to [inaudible] language available broadly, I highly recommend it packaged as a language server. Your chances that it gets used, supported by different tools, is much higher than anything else. CHARLES: Right, right. So, it's kind of like, what's the UNIX thing? The universal text interface and how it seems counterintuitive but it actually just means you can literally compose anything. Because so few assumptions are made. ERICH: I would just recommend, [inaudible], go to the website that we have about the language server protocol. I'm pretty sure it will be in the introduction or whatever. It's microsoft.github.io/language-server-protocol and then you see the implementations, all the implementation of languages, who integrates language servers, and also what kind of libraries are available, if you want to implement your language server. DIRK: And a full specification. ERICH: And the specification is there as well. Yeah. CHARLES: Yeah. If you want to go ahead and do it yourself. Well, thank you so much, Erich. Thank you so much, Dirk, for coming on the show to talk about the language server protocol. It's very exciting to me and I think it's exciting for development in general because I just think by having – even if it's 20, 30, 50% code smarts for ever single language, just the billions and billions of hours that you are going to save developers over the next, over the coming years, it's a great feeling to think about. So, thank you for all your work and thank you for coming on the show. ERICH: You're welcome. DIRK: Yeah. It was fun talking to you. ERICH: Yeah. [Inaudible] CHARLES: Yeah. If people want to continue the conversation, is there a good way that they can get in touch with you? DIRK: Usually GitHub Issues. So, where the language protocol is, it's a project on GitHub. Simply find issues. We accept pull requests. I think that's the way we communicate. CHARLES: Awesome. Again, if you want to get in touch with us, you can get in touch with us at contact@frontside.io or you can reach out to us on Twitter. We're @TheFrontside. So, thank you everybody for listening. And we will see you next time.

In this Modern Web Podcast This Dot Labs CTO, Taras Mankovski @tarasm, discusses State Management with guests... Charles Lowell @cowboyd (Frontside), Michel Weststrate @mweststrate (Mobx), and Tim Dorr @timdorr (React, Redux Router)  Topics covered: - MobX tracking changes in objects- Redux changes to accommodate async rendering in React- Best practices around batch operations in Redux- Immer.js   To learn more visit www.thisdot.co Follow us on Twitter @moderndotweb

Katharine Beaumont: @katharinecodes Show Notes: In this episode, we hit the topic of machine learning from a 101 perspective: what it is, why it is important for us to know about it, and what it can be used for. Transcript: CHARLES: Hello everybody and welcome to The Frontside Podcast, Episode 94. My name is Charles Lowell, a developer here at The Frontside and your podcast host-in-training. Today I'm going to be flying it alone, but that's okay because we have a fantastic guest who's going to talk about a subject that I've been dying to learn about. But you know, given the number of things in the world, I haven't had a chance to get around it. But with us today is Katharine Beaumont who is a machine learning consultant. And she's going to talk to us, not surprisingly, about machine learning. So welcome, Katharine. KATHARINE: Hello. Thank you very much for having me. CHARLES: No, no, it's our pleasure. So, I guess my first question is, because I'm very much approaching this from first principles here, is what is machine learning as a discipline and how does it fit into the greater picture of technology? KATHARINE: Okay. Well, if you think about artificial intelligence which is one of those slightly undefinable fields because it encompasses so much, so it encompasses elements of robotics, linguistics, math, probability, philosophy, it has six main elements. So, a really basic definition of machine learning is getting, and this comes from Arthur Samuel in 1959, it's about getting computers to learn without being explicitly programmed. And that's hugely paraphrasing. But machine learning is an element that sits under the wider discipline of artificial intelligence. Artificial intelligence is one of those tricky to define fields because people have different opinions about what it is. And obviously philosophers can't agree what intelligence is, which makes it slightly complicated. But artificial intelligence as a broad brush is a discipline that borrows from philosophy, math, probability, statistics, linguistics, robotics, and spawned subfields like natural language processing, knowledge representation, automated reasoning, computer vision robotics, and machine learning. Machine learning is the, in a sense, the mathematical component of artificial intelligence in that from a basic point of view, even though you're looking at it from the perspective of computer science, you're utilizing algorithms that a lot of mathematicians will say, “Look, we've been doing this for years. And you've just stolen that from us,” that try and find patterns in data. And that pattern could be as basic as mapping, say, the square footage of a house to the price that it will sell at and making a prediction based on that for future examples, or it could be looking for patterns in images. CHARLES: Okay. You mentioned something that I love to do. I love stealing ideas from other disciplines. It feels great. KATHARINE: Who doesn't? CHARLES: Yeah. It's like free stuff. And the best part of ideas is the person who had it still has it after you've lifted it off of them. KATHARINE: Yeah. You just have to reference and then it's not plagiarizing. CHARLES: Yeah. So, how did you actually get into this? KATHARINE: Well, a few years ago, I was desperately bored in my job. CHARLES: So, what was that job that you were working on that was so desperately boring? You don't have to name a company. KATHARINE: Oh, I won't name the company but I will – I have to make a confession now which links back to something that we were saying off recording earlier, which was that it was doing web development. So, I'm sorry. And that's not to say that web development is boring at all. It's just that I wasn't particularly engaged, which is not a reflection on web development. CHARLES: No, no, no. I actually came – I was doing, before I got into web development, I was actually doing backend stuff for years. That was all I did. KATHARINE: Yeah, me too. I would have described myself as a server-side Java developer who then cross-trained into Ruby. And I thought I'd be doing exciting backend things in Ruby. But unfortunately, it was more, “We'd like you to move this component from this part of the page to this part of the page.” And I didn't really connect with that. And I started to wonder if I even should be a developer. CHARLES: Wow. KATHARINE: Larger forces than myself were at work to try and push me into management or analysis. And as happens, I think, after a few years. So, I started doing, in my spare time, looking at a website (and I'm sure you've heard of it) Coursera. CHARLES: Yeah. KATHARINE: So, this is the birthplace of the massive online, I can't remember what the second O is, MOOCs. Massive Online something learning. Maybe a Q in there. I'm not sure what. Do you know what the acronym is? CHARLES: I actually don't know. KATHARINE: Well, MOOCs anyway. Massive online learning courses. And there was one offered Andrew Ng from Stanford on machine learning. So, I took that and I just loved it. I really enjoyed it. And I really connected with the programming. I really enjoyed the programming. It was very fulfilling. So, it grew from that, really. And now, I've decided to go back to university. So, I'm a mature postgraduate student and I'm just currently weighing up my PhD options. So, whether to sacrifice four years for the greater good and the pursuit of knowledge or go back into an employment. So, we'll see. We'll see. And I'm quite enjoying not being employed, I have to admit. Or being employed on a freelance basis. It's wonderful. CHARLES: Right, right, right. Now, a couple of things stuck me when you were talking about – so obviously, you're studying a lot of the mathematics behind it. And you said that machine learning involves a lot of the – it's the mathematical component of artificial intelligence. But what strikes me is learning, to me, implies a lot of statefulness where you're accumulating state. Whereas my experience with mathematics is usually you're solving equations. You start from some set of facts and whether it's a dataset or some other thing, and you derive, boom, boom, boom, boom, boom, you get your answer. Whereas with learning, at least when I think about school learning, like spelling or, I don't know, paleontology or something, you're accumulating facts over a very long time. And the inferences that you make are not necessarily – they're drawn from all fo the sources that you got over all that period of time rather than some one set of facts that then you make this logical argument and poof, presto, you've got your answer. How does that square? I guess it's just a little bit off from my experience with mathematics. KATHARINE: So, I am being a bit reductionist. So probably, one way to explain it is that essentially, behind a lot of the machine learning algorithms, you're inputting numbers. And that might be the percentage of red, green, blue in a pixel for example. Or it might be the diameter of a wheel, for example, if you're looking at a component of a car. Or it might be a binary configuration if you want to input the configuration of a control panel, for example, and you're looking for anomalies. And you're running these numbers through an algorithm. And what you're getting out is either a continuous value, if you're looking at a problem with continuous data like house prices, or you're getting a probabilistic output like 60% certain these pixels together make a cat, for example. So, I am simplifying by saying it's math because what you're really doing is looking for patterns in data but a way to get a computer to understand it is to somehow input it as numbers, essentially, and to get numbers out of it. CHARLES: Oh. KATHARINE: Yeah, it's more algorithms, really. And I shouldn't have said that it was essentially math because I'm sure I'm going to get shouted at on the internet. CHARLES: Well, I certainly don't want to get you in trouble. But maybe that's a point that we should shy away a little bit, the high theoretical stuff, and bring it back. If I'm excited, not even if I'm excited, why should I be excited about it? I've heard that it's a hot topic. I've heard that a lot of people are excited about it. Is there a way that I as someone who has no specialization in this might actually be able to bring some of these techniques to bear on the problems that I'm working on? Perhaps even without understanding them first, like understanding how it works. What are some problems that I might be able to attack with these techniques? KATHARINE: Yeah, absolutely. So actually, and one of the things about machine learning that I should say is don't think, “Oh, it's not for me. I'm rubbish at math. I don't understand these concepts. I'm not willing to get my head around an algorithm,” because there are so many pre-configured APIs available from big companies like Google and Amazon and Microsoft and IBM, and I'm sure many, many more. And I'm not paid by any of them, I should say. So, you don't need to understand the inner workings of an algorithm to use it. So, one example is speech-to-text. So, if you imagine that you're working on a website and you want to make it accessible, maybe you could have a component to your navigation bar that allows users to record their voice and say, “I want to navigate to the shopping cart,” for example. And machine learning would be behind that processing. So maybe, behind it you'd have an API, I've used a few of them before just to play it, where you make a call to, say, IBM service and it returns you the text. And in your program you match on keywords like shopping cart and then change the menu bar for them. So, that's one really simple way you could do it. Another more complicated way to do it is to implement something like a recommender system. So, say you have a website where you offer customers products of some description. And the most famous example of this is Amazon and Netflix. Amazon, the shopping site, rather than now the big, big corporation. And you see what other customers like you bought, or Netflix, what you might enjoy. And that's based on taking your information, comparing your viewing habits to other people's viewing habits, and then drawing some kind of correlation between the programs you watch and trying to find programs that other people have watched that you haven't, that you might enjoy. That's more complicated, to be honest. CHARLES: But that is an example. Machine learning is what underlies all that. KATHARINE: Absolutely. And at the heart of some recommender systems, the mathematics behind it is finding a way to quantify people's preferences and measuring distances between them. But you don't need to understand that to understand the basics of how a recommender system works. CHARLES: Okay. And so, how does a recommender system work? KATHARINE: So, imagine me, yourself, and Mandy each read four books. And we rate them. But I read four books, you read three of the same books, and one different one, and Mandy reads three of the same books as me and one different one, for example. So, we've got a little gap but we're not really sure what the other person will think. And we know the genres of the books. And you can compare the genres and the ratings. So, you might rate sci-fi 6 out of 10 and romance 7 out of 10. And I might rate sci-fi and romance in equal ways. So then you might say, okay, there's a similarity between our preferences. So for this book, that Charles read, Katharine might like it. CHARLES: That makes so much sense. KATHARINE: Yeah. And maybe Mandy only likes romance, only rates it 0.3 for example. So we think, “Okay, well Mandy might not be able to recommend a book to Katharine and Charles.” CHARLES: Right, I see. Implicit in this though is there's this step of the actual learning, I guess. Or the actual teaching. How do you actually teach? Again, and this is kind of me trying to wrap my head around the concept, is I've got these set of facts and I'm inferring and I'm pattern-matching and I'm trying to draw conclusions with some certainty from this set of data. But is there this distinct actual teaching phase where you have to actually teach the computer and then it takes new facts and gives stuff? How does that work? How does it incorporate the different – I guess what I'm saying is, are there distinct phases? Or is it… KATHARINE: Yes, and it's not the same for every algorithm. So, I'm going to try and give you two examples of training. So, there's something called online learning where as a new example comes in, for example it might be – I'll just explain what a classifier is. So, this is a brief diversion. So, a classifier is a machine learning process where you're trying to put information in and you're trying to get discrete information out. So, discrete meaning like it's a cat or it's a dog or a weasel or a minion or something like that. Or, it's cancer or it's not cancer. Whereas continuous output might be the price of a car. So, in a classifier you're trying to work out what type something is. So, a really good example is there's a Google project where they've clustered artworks. So, they've taken lots of different artwork and their algorithms, which I won't explain now, have determined, “This is a ballet dancer. So, we're going to group all of these ballet dancer paintings together. This is a landscape, so we're going to group all the landscapes together.” So, online learning, you might get a bit if information in, like a picture, and you will classify it and you add it to your existing information. Whereas another type of learning is you take all of the information you have, you train the algorithm, and then you make predictions. So, you either make predictions as you go along with online learning, or you do all of the work upfront. So, one algorithm – have you heard of decision trees? CHARLES: No, I haven't. KATHARINE: So, do you ever read those rubbish teen magazines where you have a flowchart and it starts at the top like, “Do you like cats? Yes or no?” CHARLES: Oh right, yeah. KATHARINE: “Do you likes dogs? Yes or no,” and it tells you what kind of a person you are or what make up you should wear or something like that. CHARLES: Right, right, right, yeah. KATHARINE: Yeah, so decision trees are kind of like that. One example is you might get information about, it's a famous toy dataset, information about passengers on the Titanic about gender and age. And we all know the techniques on the Titanic, women and children first. I've lost my use of normal English. I'm sorry. CHARLES: Right. Maybe like a trope? KATHARINE: Yeah. So, women and children first. So, you put this data in a decision tree. And what happens is at the beginning you have all of this data. So, person A is a male. They're in their 50s. This is the type of ticket they had. And this is their income. I don't think income is one of them, but just as an example. And then you have person B, person C. So, you might have a hundred people. And the decision tree algorithm goes, “Okay, if I just looked at one of these features like gender, would that differentiate the people the most?” So, it already has the answer as to whether or not they survived or did not survive. And it's looking for the one feature that gives it the most information. And then it will split on that feature. So, you go form your thing at the top and the first question might be, “Were they male or female?” And then the decision tree will split down a level. And then your algorithm will go, “Alright, what's the next feature?” Maybe the next feature is, “Were they under the age of 30?” for example. And it works down. And you end up with this sort of flowchart. And once it's trained, you then get a new example you have, person X. and you basically just work your way through the decision tree to make the prediction of whether they survived or did not survive. CHARLES: I see. And so, do you do it with some sort of certainty? Because there's going to be variation, right? You're going to have some people who are a poor fellow in his 70s who survived. Like, it's not certain that he went down but there's some probability at the end? KATHARINE: Yes. Because you're using it to make a prediction, there is always a probabilistic element. And it depends on the decision tree algorithm. So, there are some decision tree algorithms that really don't work with contradictory data, for example. CHARLES: I see. KATHARINE: There's an element of picking your algorithm. If you're approaching a machine learning problem, you have three elements. The first one is choosing your feature and your algorithm. Then you have evaluating it, so you need a way of saying how good or bad the algorithm's doing on your data, how accurate is it for example. And then you have optimizing it, which is the dark art of machine learning. CHARLES: Right. That's the wrap across the knuckles. It's coming up with wrong numbers. KATHARINE: Yeah. That's – oh no, this took two weeks to run and I need it to take 20 seconds. Or, this is only 60% accurate and I need it to be more accurate. And it's easy, just as an example I'm just working on a course that I'm giving in a few weeks' time. And I just took a day to set up a website called Kaggle, K-A-G-G-L-E, for wine quality. So, it's got acidity, citric acid, residual sugar, chlorides. They're the features. They're the components that you're going to put in. And it has a quality score. So, what you're trying to do is you're trying to find a relationship between the features and the quality. And it's very easy to get it to work. So, I now have it working. But I have it working with a really low accuracy. So, it took maybe five minutes to get it to work and it's going to take me about half an hour to make it more accurate, and that's the optimization element. CHARLES: I see. How stable are these processes? Is it finicky and fragile so that if you get new types of features it just throws things way off? Or are there ways you could control for that? KATHARINE: Well, that's a particular type of problem and that all links in with the evaluation and optimization. And that's to do with something called overfitting. So, decision trees is an algorithm notorious for overfitting, depending on the data, that is. So, overfitting is when you get the algorithm to perform really well on the training data. And then you feed it in a new example and it might grossly misclassify it because it hasn't learned to generalize beyond the examples that you've given it. CHARLES: I see. Okay. So, it's just too concrete. It hasn't recognized deep patterns. It's only recognized something superficial. KATHARINE: Yes. So typically, if you have a finite amount of data, you only train your algorithm on a certain percentage. And then you test it on the rest. CHARLES: I see. KATHARINE: So, you hold back some data. But back to our conversation earlier about when does the training happen? Another example is something called K-nearest neighbors. You could just imagine that means three nearest neighbors, for example. So, we're trying to find who we're most similar to in a room. So, it's a room full of people. And all the people are standing next to already similar people, for example. So, you might have a room where marketing's in one corner and the software developers are in another corner and the project managers are in another corner. And you go it and you're looking for the three people who are the most similar to you and you're going to go and stand in that group, for example. So, in that type of machine learning, the training is happening as each new sample comes in, rather than upfront. And there are drawbacks to both methods and there are positives to both methods. And really, in a horrible, unsexy way, it's to do with the data. And that's normally where most people because it's the most boring part when you're talking about machine learning. CHARLES: Is the data? KATHARINE: Yeah. It's this backlash from data scientists, from the golden age of data scientists where it was the hottest job on the internet to now everyone cringing going, “Oh, I really don't want to deal with my data.” But with any machine learning problem, you can't just go, “Okay. Here you go, Charles. Here's a dataset. Learn something from it.” You need context. You need to understand it. You need to have an idea of what you're looking for. So, you're getting the machine to learn but you're using it as a tool to complement your knowledge, really. And you're feeding in your knowledge to it. And part of that are the decisions that you make on the algorithms to use. CHARLES: Okay. KATHARINE: And what you're looking for. CHARLES: So, here's something that I'm wondering is related, and again I have no idea – for some reason I always associate when people talk about neural nets as being related to teaching a computer something. Is that part of the discipline of machine learning? Or no? KATHARINE: I would say it is. But it has its own cool and trendy title of Deep Learning. But it's very much powerful machine learning. So, let's go back to this. This is a classic example of machine learning. It's probably the first example you'll come across if you do any course. House prices. So, imagine that you have a piece of paper and you're going to draw one line at one side and that is the price of a house, and you're going to draw a line at the bottom which is the size in square feet, and you're going to plot examples that you have. And you might find that there's a linear correlation between the two. So, you'll draw a line and that line of best fit is the human equivalent of doing linear regression on a computer, for example, where you're just trying to find a linear correlation between two things. And a lot of the principles in linear regression, which is a very simple learning algorithm, are found in some examples of neural networks, so some basic examples of neural networks. But instead of having one input, the size in square feet, you might have 20. And you might be repeating that process of trying to find the line of best fit with different combinations of features in different places again and again. And it scales up in complexity very quickly. But it's very similar basic principles. I'm hesitating to say ‘very similar' because they're notoriously more complicated. CHARLES: Yeah. I guess I'm not really divining what exactly, what makes it – why is it called a neural net? What makes it special? It sounds like if I'm just comparing the regressions of house prices, I'm comparing those datasets over and over again, how is that different from just a loop? What are you getting out of it? KATHARINE: Historically, neural networks come from a very simplified idea about how the brain works. So, in the early 20th century people had performed autopsies and divined the inner workings of kidneys and hearts and livers. And the brain was still a bit of a mystery. And then 2 men jointly won the Nobel Prize for Physiology, and I'm going to pronounce these names wrong, so I'm sorry. I think it's Santiago Ramón y Cajal is one and Camillo Golgi is another. And they completely disagreed about the brain but they used a staining technique from Golgi to look, using silver nitrate, at the cells in the brain. And the idea of the neuron doctrine was borne out of that, that the simplest unit to look at the brain at in order to understand it is the level of the neuron, this cell in the brain. And from there, several – well, everybody was a polyglot really, back then, so I don't want to say computer scientists. So, you had people like Frank Rosenblatt with perceptrons, Marvin Minsky and so many other people looking at a very simple idea which is that a neuron either fires or doesn't. And then you're linking boolean algebra to this cell. So, you're saying it either fires or it doesn't. And from that principle, people started drawing similarities between neurons and a basic function machine. And when I say function machine, I mean imagine when you were in elementary school and you're learning how adding up works. You might have a box with a plus on it and your teacher says, “I want you to put a three in a box and a four in a box and I want you to add them together. And what do you get out?” And obviously the answer is seven. And you can think about that little box with a plus as a function machine. So, now you could think of a little mathematical function machine where you put in some inputs and there something happens in the box. And then you'll either get a one or a zero out of it. CHARLES: And so, that's like your neuron, is the little box? KATHARINE: Yes. CHARLES: Okay. KATHARINE: Yes. So since then, the neuron doctrine is pretty much contested. There are several other elements of the brain that compose thinking and the circuitry. So, any cognitive scientist listening to this will say, “That's really not how the brain works.” You have to say it with a lot of disclaimers. But the whole idea of neural networks was borne out of this idea of thinking of a neuron as like a function machine. CHARLES: Right. And also, it doesn't actually discount the usefulness of neural networks. There are a lot of things where people didn't find what they set out to find but what they found was useful. KATHARINE: Absolutely. Yeah, and they are incredibly powerful, especially with multilayer networks which are deep networks or deep learning. CHARLES: Okay. So, I didn't want derail you from your explanation. So, you've got these little function boxes and those are the kind of neurons inside the neural network? KATHARINE: Yes. So, what happens is you might have a layer of 10 of them and you might have another layer after that of another 10. And each of them are connected in a simple neural network. And then you might have an output layer of five because you're looking to classify, I don't know, an apple into five different types of apple, for example. CHARLES: Now, when you're talking about a later, you're talking about, I've got the outputs of one layer of the network are the inputs to the next layer. KATHARINE: Yes. And in different neural network architectures they'll be connected differently. But in a simple neural network, you can assume that every neuron is connected to every neuron in the next layer. So, if you have two 10-layer, two layers each with 10 neurons in, the top neuron in one layer is going to have 10 connections going out of it into the next layer. CHARLES: Oh really? Wow, that's interesting. KATHARINE: Yes. And each connection has its own sort of configuration. CHARLES: So, you're like cross-wiring all the – okay. Wow, that's kind of… KATHARINE: And yeah, that's where the complexity comes in. CHARLES: Yeah. I was thinking it was like a simple exponential fan-out. But it's even more complicated, the number of combinations you can get. KATHARINE: Yes. But in theory, it's very simple because each neuron is like this function machine with inputs coming in and something going out. It just might go out to several different locations. But it scales up in complexity very quickly. So say we're classifying apples. So, we have five different attributes of an apple like its color, its texture, its weight, acidity. I don't know how else you would measure an apple. How shiny it is, for example. And we're putting all of that information in. And each one of those things that I've just listed is a feature. So, we might have an input per feature and we'll link them all up to the neurons in a layer and we'll move that information onto the next layer. And the really important thing is what's happening on those connections between them. Because on the connections you have weights, which is a way of changing the input from one neuron to another. So, the weight might be like 0.5 for example. So, it squashes the input. Or it might magnify it. And then you get the output. Now, once you have the output you can compare the output with what you know the right answer is. And then you have this idea of an error. So, you might be like, “You got this so wrong. It's not a Braeburn apple at all. It's actually a Granny Smith apple. And what you do is with each example that you train in, you use your information about the error to train the network. Because what you're trying to do is get the error to be as small as possible. And one of the techniques of that is called back propagation. And it's notoriously difficult to understand because it involves partial derivatives and a large element of calculus. But essentially, what you're doing is comparing the right answer with the answer that the network gave it and asking it to go back and change it, change those connection weights. CHARLES: And so, do you make them fluctuate at random or is there some – is there a method to the madness of changing the weights? KATHARINE: There is a method to the madness and it's called back propagation. And the reason I linked linear regression in early, so our really simple map of house size in square feet, and the price, is because it uses a similar technique called gradient descent which is an algorithm for looking at the error and changing the weight, those numbers on the connections, to try and get it to a minimal point. So, if we go back to our house price problem, I just want you to imagine in your mind that we've got this one axis going up which is the price, and one going across which is the size in square feet. And you've got line drawn, a diagonal line. If you just imagine now in your mind moving that line down till it's completely flat at the bottom, and then moving it up so it's vertical, so it's aligned with either axis in every point in between, what you could do is you could take the error on each of those lines. So, if we imagine we have these two axes, we have the price of a house on one side and we have the size in square feet in another. And we're going to draw a line from the top axis and we're going to sweep it down and draw a line at each point as it goes down until it's aligned with the bottom axis. And at each point we draw that line, we measure the error. So, what we'd do for that is all of the little points, all of the example data, we'd measure the difference between them and the line and we're going to, say, add them all up. So, what you'd end up with is you'd end up with a graph mapping the error against the gradient at all fo those different points. And it would look like a bowl. You'd have a lowest point. You'd have a point for some gradient where the error is the lowest. CHARLES: Right, yeah. Okay. I'm seeing it. I think I'm seeing it. So, you want to take that error function and you want to, what is it? Now this is – boy, I'm going back to high school math. You would take the derivative and find out where the tangent is and that's your min point? That's the root of the equation and that's the point where your error is lowest? KATHARINE: Yeah, exactly. Or there's an algorithm called gradient descent that does it automatically by taking little steps. So, it looks at the tangent of the gradient at a point and says, “If I move the gradient down is the error going to decrease. And if so, move in that direction.” So automatically, it tries to take steps to get to the bottom. And optimization problem with that is that you can configure the step size. So, you could take tiny, tiny steps and take forever to get to the bottom or you could take massive steps and completely miss the bottom. So, you can imagine it like walking down a hill. If you're a minion then it will take a really long time because you're tiny. And if you're a giant, you might never get to the valley. CHARLES: Right. You might just leap right across the chasm. KATHARINE: Yeah, just miss it completely. So, that's linear regression where we're looking. And that's really, even though we have a two-dimensional graph that's a one-dimensional problem because we just have the one input feature. Now, when we're looking at neural networks and we're looking at gradient descent in neural networks, each one of those connections is something that we're trying to configure to reduce the error. So suddenly, you have maybe a hundred-dimension landscape and you're trying to get to the bottom of a hill. And there might be several local optimas and one deepest valley, but you might have lots of other valleys that you could get stuck in. So, it becomes a very difficult problem. Does that make sense? CHARLES: Yes. No, that does make sense. I'm just trying to let it sink in. KATHARINE: It's completely impossible to visualize a hundred dimensions, yeah. CHARLES: I actually had to sit back and kind of close my eyes and stare up at the ceiling. KATHARINE: I think the trick is not to think about it. I heard someone say, there's another fantastic course on Coursera by a famous computer scientist studying neural networks called Geoffrey Hinton. And his advice in one of the videos on visualizing these multidimensional landscapes, say it's 15 dimensions, is to close your eyes and shout, “15!” And that hasn't worked for me, but I'm sure it's worked for some people. CHARLES: But it certainly probably makes you feel better. KATHARINE: Yeah. I think it's one of those things that's just beyond comprehension. But we can just quietly accept it. CHARLES: Right. It's just – yeah, what's nice about I guess math is you just don't have to understand it. I mean, you do. You just understand that there really is no mapping to our physical experience. And that's okay. And just let that go. It's just like, this is just some… KATHARINE: Yeah. CHARLES: We had some numbers that existed in the domain of understanding, that we can understand. And there are just some rules that we follow and if you look at the intermediate steps, well the points don't really exist in that domain of physical experience and understanding. And that's okay. We just accept and let it go. We just hope that at some point, we can translate that model back into the domain of ‘we can understand it'. KATHARINE: Oh, completely. There's a lot of faith. CHARLES: Yeah. KATHARINE: And also, I remember coming into machine learning and thinking, “This is like magic. It's amazing.” And then you study it a bit and you're like, “This is so easy. This is just basic – this is just functions. These are just glorified function machines.” And then you look into it some more and you're like, “Nope. It's definitely magic.” CHARLES: Yeah. It's a phase of, every time you come up against the wall, right? And then you realize, “Oh no, it's actually something that I can close my mind over,” until you come to the next hurdle of magic. KATHARINE: Yeah. You think you've got a grasp on things and you think, “I know the landscape,” and then you suddenly realize how much more there is to learn. And that sinking feeling that you'll never learn it all. CHARLES: Yeah, yeah. Yup. Unfortunately, it seems like in tech that's like, that's just the condition. KATHARINE: Yeah, like all of my sad, unread books. CHARLES: If I wanted to get into just really start experimenting with this stuff and start saying, “Maybe I can utilize some of these techniques for some of these problems that I'm encountering,” Where would be a good place to get started? What libraries? What online resources? What people are good to follow and ask questions of? KATHARINE: Okay. Let's start with websites. So for a start, this website called Kaggle which I mentioned earlier, and that is K-A-G-G-L-E dot com. And that has a lot of dataset resources. It also has a community of people discussing how they use the datasets. It has competitions. And it has a lot of links. I discovered recently as well a really good blog. It's on Medium and it is called ‘Machine Learning for Humans'. And that's really well-written. I really like it, actually, and it has a good section on resources called ‘The Best Machine Learning Resources'. And I should probably plug my own blog, but this one's so much better. CHARLES: But please do. KATHARINE: No, no. I have to actually write stuff for it. But there's a lot of things there about, well, if you want to learn linear algebra, what if you want to learn probability and statistics, calculus, and then just go straight to machine learning and pick up the math on the way. I would say go on Coursera, because there are courses like Andrew Ng's course on machine learning from Stanford. And Geoffrey Hinton from the University of Toronto. But there are also courses there on things like calculus and probability and statistics if you want to level up your math. If you don't want to do anything to do with the math, I would say go to the vendor websites, like AWS, Google. If you're into Java, go on deep learning for J. And a lot of them have tutorials that complement their products. Deep learning for J is one of my favorites at the moment. It's an open source Java library. It's pretty plug and play, actually. You don't need to understand a lot of it to get started with it. But it helps. And obviously then there's TensorFlow although personally, I find just other Python libraries like SciPy a lot easier than using TensorFlow. And I think naturally you'll find the resources and the people to follow on Twitter from that. But the crucial thing I'd say is don't get hung up on which language to start playing around with. So, a lot of people say, “Oh, I must need to know Python. I must need to know math.” And really, you don't. It just depends on what level you want to approach things at. So, if you want to write your own gradient descent algorithms, then Python is probably more for you, or Matlab or R or something like that. But there are libraries where you can do it in Java. I've heard rumors that there's a JavaScript library and I wouldn't be surprised. So, I would just have a look at what's out there. But try and get a grasp of the fundamentals just from an intuition point of view, because it will make your life so much easier. You might for example realize that you're using the completely wrong algorithm for the problem that you're looking at. And that's invaluable. CHARLES: Yeah. Knowing what not to do certainly is. Alright. Well, thank you so much for that, Katharine. Thank you for being on the show. Thank you for curing us of at least a small portion of our ignorance. And if people want to get in touch with you perhaps and continue the conversation, or follow you, what's a good place to get in touch? KATHARINE: Sure. Probably tweet me on Twitter. I'm @KatharineCodes but it's spelled like Katharine Hepburn. It's K-A-T-H-A-R-I-N-E codes. Because when I joined Twitter, I didn't have much of an imagination. I still don't. So, it's not particularly clever. But it's there. CHARLES: Alright. Well, fantastic. And for everybody listening at home, you can also get in touch with us. We're @TheFrontside on Twitter. Or you can just drop us a line at info@frontside.io. Thanks for listening and we will see you all next time.

Julie Moronuki: @argumatronic | argumatronic.com Show Notes: This episode is a follow-up episode to the one we did with Julie in September: Learn Haskell, Think Less. We talk a whole lot about monoids, and learning programming languages untraditionally. Transcript: CHARLES: Hello everybody and welcome to The Frontside Podcast, Episode 93. My name is Charles Lowell, a developer here at The Frontside and I am your podcast host-in-training. With me today from The Frontside is Elrick also. Hello, Elrick. ELRICK: Hey. CHARLES: How are you doing? ELRICK: I'm doing great. CHARLES: Alright. Are you ready? ELRICK: Oh yeah, I'm excited. CHARLES: You ready to do some podcasting? Alright. Because we actually have a repeat guest on today. It was a very popular episode from last year. We have with us the author of ‘Learning Haskell: From First Principles' and a book that is coming out but is not out yet but one that we're eagerly looking forward to, Julie Moronuki. Welcome. JULIE: Hi. It's great to be back. CHARLES: What was it about, was it last October? JULIE: I think it was right before I went to London to Haskell [inaudible]. CHARLES: Yeah. JULIE: Which was in early October. So yeah… CHARLES: Okay. JULIE: Late or early October, somewhere in there. CHARLES: Okay. You went to Haskell eXchange. You gave a talk on Monoids. What have you been up to since then? JULIE: Oh wow. It's been a really busy time. I moved to Atlanta and so I've had all this stuff going on. And so, I was telling a friend last night “I'm going to be on this podcast tomorrow and I don't think I have anything to talk about.” [Laughter] JULIE: Because I feel like everything has just been like, all my energy has been sucked up with the move and stuff. But I guess… CHARLES: Is it true that everybody calls it ‘Fatlanta' there? JULIE: Yeah. [Laughs] CHARLES: I've heard the term. But do people actually be like “Yes, I'm from Fatlanta.” JULIE: I've heard it a couple of times. CHARLES: Okay. JULIE: Maybe it's mostly outsiders. I'm not sure. CHARLES: [Chuckles] JULIE: But yeah, it's a real cool city and I'm real happy to be here. But yeah, I did go in October. I went to London and I spoke at Haskell eXchange which was really amazing. It was a great experience and I hope to be able to go back. I got to meet Simon Payton Jones which was incredible. Yeah, and I gave a talk on monoids, monoids and semirings. And… CHARLES: Ooh, a semiring. JULIE: Semiring. So, a semiring is a structure where there's two monoids. So, both of them have an identity element. And the identity element of one of them is an annihilator. Isn't that a great word? It's an annihilator… CHARLES: Whoa. JULIE: Of the other. So, if you think of addition and multiplication, the identity element for addition is zero, right? But if you multiply times zero, you're always going to get to zero, so it's the annihilator of multiplication. CHARLES: Whoa. I think my mind is like annihilated. [Laughter] JULIE: So, it's a structure where you're got two monoids and one of them distributes over the other, the distributive property of addition and multiplication. And the identity of one of them is the annihilator of the other. Anyway, but yeah, I gave a history of where monoids come from and that was really fun. CHARLES: Yeah. I would actually like to get a summary of that, because I think since we last talked, I've been getting a little bit deeper and deeper into these formal type classes. I'm still not doing Haskell day-to-day but I've been importing these ideas into just plain vanilla JavaScript. And it turns out, it's actually a pretty straightforward thing to do. There's definitely nothing stopping these things from existing in JavaScript. It's just, I think people find type class programming can be a tough hill to climb or something like that, or find it intimidating. JULIE: Yeah. CHARLES: But I think it's actually quite powerful. And I think one of the things that I'm coming to realize is that these are well-worn pathways for composing things. JULIE: Right. CHARLES: So, what you encounter in the wild is people generating these one-off ways of composing things. And so, for a shop like ours, we did a lot of Ruby on Rails, a lot of Ember, and both of those frameworks have very strong philosophical underpinnings that's like “You shouldn't be reinventing the wheel if you don't have to.” I think that all of these patterns even though they have crazy quixotic esoteric names, they are the wheels, the gold standard of wheel. [Laughs] They're like… JULIE: Right. CHARLES: We should not be reinventing. And so, that's what I'm coming to realize, is I'm into this. And last time you were talking, you were saying “I find monoids so fascinating.” I think it took a little bit while to seep in. But now, I feel like it's like when you look at one of those stereo vision things, like I'm seeing monoids everywhere. It's like sometimes they won't leave me alone. JULIE: In ‘Real World Haskell' there's a line I've always liked. And I'm going to misquote it slightly but paraphrasing at least. “Monoids are ubiquitous in programming. It's just in Haskell we have the ability to just talk about them as monoids.” CHARLES: Yeah, yeah. JULIE: Because we have a name and we have a framework for gathering all these similar things together. CHARLES: Right. And it helps you. I feel like it helps you because if you understand the mechanics of a monoid, you can then when you encounter a new one, you're 90% there. JULIE: Right. CHARLES: Instead of having to learn the whole thing from scratch. JULIE: Right. And as you see them over and over again, you develop a kind of intuition for when something is monoidal or something looks like a semiring. And so, you get a certain intuition where you think, “Oh, this thing is like a… this is a monad.” And so, what do I know about monads? All of a sudden, this new situation like all these things that I know about monads, I can apply to this new situation. And so, you gain some intuition for novel situations just by being able to relate them to things you already do know. CHARLES: Exactly. I want to pause here for people. The other thing that I think I've come in the last three months to embrace is just embrace the terminology. JULIE: Yeah. CHARLES: You got to just get over it. JULIE: [Chuckles] CHARLES: Think about it like learning a foreign language. The example I give is like tasku is the Finnish word for pocket. JULIE: Right. CHARLES: It sounds weird, right? Tasku. But if you say it 10 times and you think “Pocket, pocket, pocket, pocket, pocket.” JULIE: Yes, yeah. [Laughs] CHARLES: Then it's like, this is a very simple, very useful concept. JULIE: Right. CHARLES: And it's two-sided. There on the one hand, the terminology is obtuse. But at the same time, it's not. It's just, it is what it is. And it's just a symbol that's referencing a concept. JULIE: Right, right. CHARLES: It's a simple concept. So, I just want to be… I know for our listeners, I know that there's a general admonition. Don't worry about the terminology. It's… JULIE: Right, right. Like what I just said, I said the word ‘monad'. I just threw that out there at everybody, but [chuckles] it doesn't matter which one of these words we'd be talking about or whatever I call them. We could give monads a different name and it's still this concept that once you understand the concept itself, and then you can apply it in new situations, it doesn't matter then what it's called. But it does take getting used to. The words are… well, I think functor is a pretty good word for what it is. If you know the history of functor and how it came to mean what it means, I think it's a pretty good word. CHARLES: Really? So, I would love to know the history. Because functor is mystifying to me. It sounds like, I think the analogy I use is like if George Clinton and a funk parliament had an empire, the provinces, the governors of the provinces would be functors. ELRICK: [Laughs] JULIE: Yes. CHARLES: But [Laughs] that's the closest thing to an explanation I can come up with. JULIE: I might use that. I'm about to give a talk on functors. I might use that. [Laughter] ELRICK: Isn't that the name of the library? Funkadelic? CHARLES: Well, that's the name of the library that I've been… JULIE: [Could be], yeah. ELRICK: That you'd been… CHARLES: That I'd been [writing] for JavaScript. ELRICK: Yeah. CHARLES: That imports all these concepts. JULIE: [Laughs] ELRICK: Yeah. JULIE: Yeah. ELRICK: So awesome. JULIE: Yeah. Yeah, I have… CHARLES: So, what is the etymology of functor? JULIE: Well, as far as I can tell, Rudolf Carnap, the logician, invented the word. I don't know if he got it from somewhere else. But the first time I can find a reference to it is in, he wrote a book about… he was a logician but this is sort of a linguistics book. It's called ‘The Logical Syntax of Language'. And that's the first reference I know of to the word functor. And he was trying to really make language very logically systematic, which natural language is and isn't, right? [Chuckles] CHARLES: Right. JULIE: But he was only concerned with really logically systematizing everything. And so, he used the word functor to describe some kinds of function words in language that relate one part of a sentence to another part of a sentence. CHARLES: Huh. So, what's an example? JULIE: So, the example that I've used in the past is, as far as I know this is not one that Carnap himself actually uses but it's the clearest one outside of that book… well the ones inside the book I don't really think are very good examples because they're not really how people talk. So, the one that I've used to try to explain it is the word ‘not' in English where ‘not' gets applied to the whole sentence. It doesn't really change the logical structure of the sentence. It doesn't change the meaning of the sentence except for now it negates the whole thing. CHARLES: I see. JULIE: And so, it relates this sentence with this structure to a different context, which is now the whole thing has been negated. CHARLES: I see. So, the meaning changes, but the structure really doesn't. JULIE: Right. And it changes the whole meaning. CHARLES: Right. JULIE: Not just part of the sentence. So, if you imagine ‘not' applying to an entire sentence because of course we can apply it just to a single word or just to a single phrase and change the meaning just of that word or that phrase, but if you imagine a context where you've applied ‘not' to a whole sentence, to an entire proposition, because of course he's a logician. So, if you've applied ‘not' to an entire proposition, then it doesn't change the structure or the meaning of that proposition per se except for it just relates it to the category of negated propositions. CHARLES: Mmhmm. JULIE: So, that's where it comes from. And… CHARLES: But I still don't understand why he called it functor. JULIE: He's sort of making up… well, actually I think the German might be the same word. CHARLES: Ah, okay. JULIE: Because he was writing in German. Because he's looking for something that evokes the idea of ‘function word'. CHARLES: Oh. JULIE: So, if you were to take the ‘func' of ‘function' [Laughs] and the, I don't know, maybe in German there's some better explanation for making this into a particular word. But that's how I think of it. So, it's ‘function word'. And then category theorists took it from Carnap to mean a way to map a function in this category or when we're talking about Haskell, a function of this type, to a function of another type. CHARLES: Okay. JULIE: And so, it takes the entire function, preserves the structure of the function just like negation preserves the structure of the sentence, and maps the whole thing to just a different context. So, if you had a function from A to B, functor can give you a function from maybe A to maybe B. CHARLES: Right. JULIE: So, it takes the function and just maps it into a different context. CHARLES: Right. So, a JavaScript example is if I've got an array of ints and a function of ints to strings, I can take any array of ints and get an array of strings. JULIE: Right. CHARLES: Or if I have a promise that has an int in it, I can take that same function to get a promise of a string. JULIE: Yeah. CHARLES: Yeah. I had no idea that it actually came from linguistics. JULIE: Yeah. [Laughs] CHARLES: So actually, the category theorists even… it digs deeper than category theory. They were actually borrowing concepts. JULIE: They were, yes. CHARLES: We just always are borrowing concepts. ELRICK: I like the borrowing of concepts. JULIE: Yeah. ELRICK: I think where people struggle with certain things, it's tying it back to something that they're familiar with. So, that's where I get… my mind is like [makes exploding sound] “I now get it,” is when someone ties it back to something that I am… CHARLES: Right. ELRICK: Familiar with. Like Charles' work with the JavaScript, tying it with JavaScript. I'm like, “Oh, now I see what they're talking about.” JULIE: Right. CHARLES: because you realize, you're using these concepts. People are using them, just they're using them anonymously. JULIE: Right. ELRICK: True. CHARLES: They don't have names for them. JULIE: Right. ELRICK: True. CHARLES: It's literally like an anonymous function and you're just taking that lambda and assigning it to a symbol. JULIE: Yeah. CHARLES: You're like “Oh wait. I've been using this anonymous function all over the place for years. I didn't realize. Boom. This is actually a formal concept.” ELRICK: True. And I think when people say like “Don't reinvent the wheel” it's a great statement for someone that has seen a wheel already. [Laughter] ELRICK: You know what I'm saying? If you never saw a wheel, then your'e going to reinvent the wheel because you're like “Aw man. This doesn't exist.” [Chuckles] JULIE: Yeah. ELRICK: But if people are exposed to these concepts, then they wouldn't reinvent the wheel. CHARLES: Right. JULIE: Right. Yeah. CHARLES: Instead of calling in some context, calling it a roller. [Chuckles] It's a round thingy. [Laughter] JULIE: Right. Yeah, so that's a little bit what I tried to do in my monoid talk in London. I tried to give some history of monoid, where this idea comes from and why it's worth talking about these things. CHARLES: Yeah. JULIE: Why it's worth talking about the structure. CHARLES: So, why is it worth the… where did it come from and why is it worth talking about? JULIE: Oh, so back when Boole, George Boole, when he decided to start formalizing logic… CHARLES: George Boole also, he was a career-switcher too, right? He was a primary school teacher. JULIE: Right, yeah. CHARLES: If I recall. He actually, he was basically teaching. Primary school is like elementary school in England, right? JULIE: I believe so, yes. CHARLES: Yeah. I think he was like, he was basically the US equivalent of an elementary school teacher who then went on to a second and probably, thankfully a big career that left a big legacy. JULIE: Right. Although no one knew exactly how big the legacy was really, until Claude Shannon picked it up and then just changed the whole world.[Laughs] Anyway, so Boole, when he was trying to come up with a formal algebra of logic so that we could not care so much about the semantic content of arguments (we could just symbolize them and just by manipulating symbols we could determine if an argument was logically valid or not), he was… well, for disjunction and conjunction which is AND and OR – well, disjunction would be the OR and conjunction the AND – he had prior art. He had addition and multiplication to look at. So, addition is like disjunction in some important ways. And multiplication is like conjunction in some important ways. And I think it took me a while to see how addition and disjunction were like each other, but there are some important ways that they're like each other. One of them is that they share their identity values. If you think of, it's sort of like binary addition and binary multiplication because in boolean logic there's only two values: true or false. So, you have a zero and a one. So, if you think of them as being like binary addition and binary multiplication then it's easier to see the connection. Because when we think of addition of just integers in a normal base 10 or whatever, it doesn't seem that much like an OR. [Laughs] CHARLES: Mmhmm. No, it doesn't. JULIE: [Inaudible] like a logical OR. So, it took me a while to see that. But they're also related then to set intersection and union where intersect-… CHARLES: So can… Let's just stop on that for a little bit, because let me parse that. So, for OR I've got two values, like in an ‘if' statement. This OR that. If I've got a true value then I can OR that with anything and I'll get the same anything. JULIE: Right. CHARLES: So, true is the identity value of OR, right? Is that what you're saying? So, one… JULIE: Well, it's false that's the identity of OR. CHARLES: Oh, it is? JULIE: Zero is the identity of addition. CHARLES: Wait, but if I take ‘false OR one' I get… oh, I get one. JULIE: Right. CHARLES: Okay. So, if I get ‘false OR true', I get true. Okay, so false is the identity. JULIE: Yeah. CHARLES: Oh right. You're right. You're right. Because… okay, sorry. JULIE: So, just like in addition, zero is the identity. So, whatever you add to zero, that's the result, right? You're going to get [the same] CHARLES: Right. JULIE: Value back. So, with OR false is the identity and false is equivalent to zero. CHARLES: [Inaudible] ‘False OR anything' and you're getting the anything. JULIE: Right. So, the only time you'll get a false back is if it's ‘false OR false', right? CHARLES: Right. Mmhmm. JULIE: Yeah. So, false is the identity there. And then it's sort of the same for conjunction where one is the identity of multiplication and one is also the… I mean, true is then the identity of logical conjunction. CHARLES: Right. Because one AND… JULIE: ‘True AND false' will get the false back. [Inaudible] CHARLES: Right. ‘True And true' you can get the true back. JULIE: Yeah. CHARLES: Okay. JULIE: And it's also then true, getting back to what we were talking about, semirings, it's also true that false is a kind of annihilator for conjunction. That's sort of trivial, because… CHARLES: Oh, because you annihilate the value. JULIE: Right. When there's only two values it's a little bit trivial. But it is [inaudible]. So… CHARLES: But it's [inaudible]. Yeah. It demonstrates the point. JULIE: Right. CHARLES: So, if I have yeah, ‘false AND anything' is just going to be false. So, I annihilate whatever is in that position. JULIE: Right. CHARLES: And the same thing as zero is the annihilator for multiplication, right? JULIE: Right. CHARLES: Because zero times anything and you annihilate the value. JULIE: Yeah. CHARLES: And now I've got… okay, I'm seeing it. I don't know where you're going with this. [Laughter] ELRICK: Yeah. CHARLES: But I'm there with you. ELRICK: Yup. JULIE: And then it turns out there are some operations from set theory that work really similarly. So, intersection and union are similar but the ones that are closer to conjunction/disjunction are disjoint unions and cartesian products. So we don't need to talk about those a whole lot if you're not into set theory. But anyway… CHARLES: I like set theory although it's so hard to describe without pictures, without Venn diagrams. JULIE: It is. It really is, yeah. So anyway, all of these things are monoids. And they're all binary associative operations with identity elements. So, they're all monoids. And so, we've taken operations on sets, operations on logical propositions, operations on many kinds of numbers (because not all kinds of addition and multiplication I guess are associative), and we can kind of unify all of those into the same framework. And then once we have done that, then we can see that there's all these other ‘sets'. Because most of the kinds of numbers are sets and there are operations on generic sets with set theory. So, now we can say “Oh. We can do these same kinds of operations on many other kinds of sets, many other varieties of sets.” And we can see that same pattern. And then we can get a kind of intuition for “Well, if I have a disjunctive monoid where I'm adding two things or I'm OR-ing two things…” Because even though those are logically very similar, intuitively and in terms of what it means to concatenate lists versus choosing one or the other, those obviously have different practical effects. CHARLES: So, I'm going to try and come up with some concrete examples to maybe… JULIE: Okay, yeah. CHARLES: A part of them will probably be like in JavaScript, right? So, to capture the idea of a disjunctive monoid versus a conjunctive monoid. So, a disjunctive monoid is like, so in JavaScript we're got two objects. You concat them together and it's like two maps or two hashes. So, you mash them together and you get… so, for the disjunctive one you'd have all the keys from both of the hashes inside the resulting object. You take two objects. Basically we call it object assign in JavaScript where you have basically the empty object. You can take the empty object and then take any number of objects. And so, we talked about… JULIE: That would become a disjunctive monoid, right? CHARLES: That would be a disjunctive monoid because you're like basically, you're OR-ing. Yeah. JULIE: You're kind of, [inaudible] CHARLES: Hard to find the terminology. JULIE: Yeah. CHARLES: But like object assign would be a disjunctive monoid because you're like mashing these two objects. And the resulting object has all of the things from both of them. JULIE: Right. So, it's like a sum of the two, right? CHARLES: Right, right. Okay, so then another one would be like min or max where you've got this list of integers and you can basically take any two integers and you can mash them together and if you're using min, you get the one that's smaller. Basically, you're collapsing them into one value but you're actually just choosing one of them. Is that like… JULIE: Yeah. CHARLES: Would that be like a conjunctive monoid? JULIE: No, that's also disjunctive but that's more like an OR than like a sum. CHARLES: Okay. JULIE: Right. So, that's what I said. It's hard to think of disjunctive monoids I think because there's really two varieties. There's some underlying logical similarity, like the similarity in the identity values. But they're also different. Summing two things versus choosing one or the other are also very different things in a lot of ways. CHARLES: Right. Okay. JULIE: And so, I think the conjunctive monoids are all a little bit more similar, I think. [Chuckles] But the disjunctive monoids are two broad categories. And we don't really have a monoid in Haskell of lists where you're choosing one or the other. The basic list monoid is you're concatenating them. So, you're adding two lists or taking the union of them. But for maybe, the maybe type, we do have monoids in Haskell where you're just choosing either the first just value that comes up or the last just value that comes up. So, we do have a monoid of choice over the maybe type. And then we have a type class called alternative which is monoids of choice for… so, they're disjunctive monoids but instead of adding the two things together, they're choosing one or the other. CHARLES: Okay. JULIE: Though we have a type class for that. [Laughs] CHARLES: [Sighs] Oh wow. Yeah. JULIE: Mmhmm, yeah. CHARLES: I'l have to go read up on that one. JULIE: That type class comes up the most when you're parsing, because you can then parse… like if you found this thing, then parse this thing. But if you haven't found this thing, then you can keep going. And if you find this other thing later, then you can take that thing. So, you allow the possibility of choice. The first thing that you come to that matches, take that thing or parse that thing. So, that type class gets mostly used for parsing but it's not only useful for parsing. CHARLES: Okay. JULIE: So yeah. That's the most of the time when I've used it. CHARLES: Is this when you're like parsing JSON? Or is this when you're just searching some stream for some value? Like you just want to run through it until you encounter this value? Or how does that…? JULIE: Right. Say you want to run through it until you find either this value or this value. I've used it when I've been parsing command line arguments. So, let's say I have some flags that can be passed in on my command line command. There are some flags that could be passed in. So, we'll parse until we find this thing or this thing. This flag or this flag. So, if you find this flag, then we're going to go ahead and parse that and do whatever that flag says to do. If you don't find that first flag then we can keep parsing and see if you find this other flag, in which case we'll do something different. CHARLES: Okay. JULIE: It'll take the first match that it finds. Does that make sense? CHARLES: Yeah, yeah, yeah. It does. But I'm not connecting how it's a monoid. [Laughs] JULIE: How is that a monoid? Well, because it's a monoid of OR-ing CHARLES: What's the identity value or the empty value in that case? JULIE: Well, the empty value would be… let's say you have maybes. Let's say you have some kind of maybe thing, so you're parser is going to return maybe this thing, maybe whatever you're parsing. Like maybe string. CHARLES: Yeah, yeah. JULIE: So, it's going to return a maybe string. So well, nothing would be the empty. CHARLES: Okay. JULIE: But nothing is like the zero because it's a disjunction, logical OR. So, only when you have two nothings will you get back a nothing. Otherwise, it will take the first thing that it finds. CHARLES: Okay. I see. JULIE: Yeah. So, the identity then is the nothing, like false is the identity for disjunction. CHARLES: Mmhmm. Okay. JULIE: Yeah. CHARLES: [Inaudible] JULIE: Yeah. If you have nothing or this other thing, then you return this other thing. Then you return the maybe string. If you have two nothings, then you get in fact nothing. Your parsing has failed. CHARLES: Right, because you've got nothing. JULIE: Because you've got nothing. There was nothing to give you back. CHARLES: So, you concatenated all of the things together and you ended up with nothing. JULIE: Right, because there was nothing there. CHARLES: Right. [Laughs] JULIE: You found nothing. So, it's useful when you've got some possibilities that could be present and you just want to keep parsing until you find the first one that matches. And then it'll just return whatever. It'll just parse the first thing that it matches on. CHARLES: Okay, okay. JULIE: Does that make sense? CHARLES: Yeah. No, I think it makes sense. JULIE: I'm not sure. Because I feel like I kind of went down a rabbit hole there. [Laughs] CHARLES: Yeah. [Laughs] No, no. I think it makes sense. And as a quick aside, I think… so, I was, when we were talking about min and max, are min and max also like a semiring? Because negative infinity is the annihilator of min and it's the identity of max. and positive infinity is the annihilator of max but it's the identity of min. JULIE: I guess. I don't really think of min and max as having identities. Is that how [inaudible]? CHARLES: I'm just, I don't know. Well, I think if you have negative infinity and you max it with anything, you're going to get the anything, right? Negative infinity max one is one. Negative infinity/minus a billion is minus a billion. JULIE: Yeah, okay. CHARLES: I don't know. Just off the cuff. I'm just trying to… annihilators sound cool. And so… [Laughter] CHARLES: And so I'm like, I'm trying to find annihilators. JULIE: Yeah, they are cool. CHARLES: [Laughs] JULIE: One of my friends on Twitter was just talking about how he used the intuition at least of a semiring at work because he had this sort of monoid to concatenate schedules. So, he's got all these different schedules and he's got this kind of monoid to concatenate them, to merge the schedules together. But then he's got this one schedule that is special. And whenever something is in this schedule, it needs to hard override every other schedule. CHARLES: Right. JULIE: And so, that was like the annihilator. So, he was thinking of it as a semiring, because that hard override schedule is like the annihilator of all the other schedules. CHARLES: Yeah. JULIE: If anything else exists on this day or whatever, then it'd just get a hard override. So, there's a real world use. [Laughs] CHARLES: Yeah, a real world example. That's the thing that I'm finding, is that all these really very crystalline abstractions, they still play out very well I think in the real world. And they're useful as a took in terms of casting a net over a problem. Because you're like… when I'm faced with something new, I'm like “Well, let's see. Can I make it a functor?” And if I can, then I've unlocked all these goodies. I've unlocked every single composition pattern that works with functor. JULIE: Right, right. CHARLES: And it's like sometimes it fits. It almost feels like when you're working on something at home and you've got some bolt and you're trying on different diameters. So you're like, “Oh, is it 15 millimeter? Is it 8 millimeter?” JULIE: Right. [Laughs] CHARLES: “Like no, okay. Maybe it'll work with this.” But then when it clicks, then you can really ratchet with some serious torque. JULIE: Right, right. Yeah. CHARLES: So, yeah. Definitely trying to look for semirings [Laughs] is definitely beyond my [can] at this point. But I hope to get there where it can be like, if it's a fit, it's a fit. That's awesome. JULIE: Right. Yeah, it's kind of beyond my can too. Semirings are still a little bit new for me and I can't say that I find them in the wild as it were, as often as monoids or something. But I think it just takes seeing some concrete examples. So, now you know this idea exists. If you just have some concrete examples of it, then over time you develop that intuition, right? CHARLES: Right. JULIE: Like “Okay, I've seen this pattern before.” [Chuckles] CHARLES: yeah. Basically, every time now I want to fold a list, or like in JavaScript, any time you want to reduce something I'm like “There's a monoid here that I'm not seeing. Let me look for it.” JULIE: Yeah. Oh, that's cool, yeah. CHARLES: Because like, that's basically, most of the time you're doing a reduce, then like I said that's the terminology for fold in JavaScript, is you start with some reducible thing. Then you have an initial value and a function to actually concatenate two things together. JULIE: Right. CHARLES: And so, usually that initial state, that's your identity. And then that function is just your concat function from your monoid. And so, usually anytime I do a reduce, there's the three pieces. Boom. Identity value, concatenation function, it's usually right there. And so, that's the way I've found of extracting these things, is I'm very suspicious every time I'm tempted to… JULIE: [Laughs] CHARLES: A fold. I'm like “Hmm. Where's the monoid I'm missing? Is it [under the] couch?” Like, where is it? [Laughs] Because it just, it cleans it up and it makes it so much more concise. JULIE: Oh yeah, that's awesome. CHARLES: So anyhow. JULIE: Have we totally lost Elrick? ELRICK: Nope, I'm still here. JULIE: Okay. [Laughs] ELRICK: I'm sitting in and listening to you two break down these complex topics is really good. Because you guys break them down to a level where it's consumable by people that barely understand it. So, I'm just sitting here just soaking everything in like “Oh, that's awesome.” Taking notes. Yes, okay, okay. [Laughter] JULIE: Cool. ELRICK: So, I'm like riding the train in the back just hanging out, feeling the cool breeze while you guys just pull the train ahead in… [Laughter] ELRICK: In the engine department, you know? It's awesome. CHARLES: Yeah. ELRICK: I don't know if they're related. But you were talking about semirings and I heard of semigroups or semigroups. I have no idea if those two things are related. Are they related or [inaudible]? JULIE: They're kind of related. So, a semigroup is like a monoid but doesn't have an identity value. CHARLES: What is an example of a semigroup out there in the wild? Because every time I find a semigroup, I feel like it's actually a monoid. JULIE: Well, you know I feel like that a lot, too. We do have a data type in Haskell that is a non-empty list. So, there is no empty list CHARLES: Ah, right. Okay. JULIE: So then you can concatenate those lists, but there's never an identity value for it. CHARLES: I see. JULIE: Yeah. So, that's a case. There's actually a lot of comparison functions, greater than and less than. I think those are semigroups because they're binary, they're associative, but they don't have an identity value. Like if you're comparing two numbers, there's not really an identity value there. CHARLES: Right. Well, would the negative infinity work there? Let's see. Like, negative infinity greater than anything would be the anything. Well, okay wait. But greater than, that takes numbers and yields a boolean, right? JULIE: Yeah, CHARLES: Right. So, it couldn't be… could it be a semigroup? Don't semigroups have to… Doesn't the [inaudible] function have to yield the same type as the operands? JULIE: Yes. CHARLES: But a non-empty list, that's a good one. Sometimes it's basically not valid for you to have a list that doesn't have any elements, right? Because it's like the null value or the empty value and it could be like a shopping cart on Amazon. You can't have a shopping cart without at least something in it. JULIE: Right. CHARLES: Or, you can't check out without something. So, you might want to say like the shopping cart that I'm going to check out is a non-empty list. And so, you can put two non-empty lists together. But yeah, there's no value you can mash together, you can concat with anything, that isn't empty. JULIE: Right. CHARLES: So, I guess going back to your question Elrick, I don't know if it's related to semiring. But semigroup is just, it's like one-half of monoid. It's the part that concats two values together. JULIE: Right. Well, yeah. And so, it's supposed to be half a group, right? But I don't remember… CHARLES: [Laughs] JULIE: [Inaudible] all of the group stuff is, all the stuff that these types have to have to be a group. And similarly, I forget what the difference between semiring and ring is. [Chuckles] Because a ring and a group I know are not the same thing. But I forget what the difference is, too. So, I kind of got a handle on what semigroups are, and I know all my Haskell friends are going to, when they hear this podcast they're going to tweet all these examples of semigroups at me, especially my coauthor for ‘Joy of Haskell', Chris Martin. He's really into semigroups. And so, I know he's going to be very disappointed in my inability to think… [Laughter] JULIE: To think of any good examples. But it's not something that I find myself using a lot, whereas semirings are something that I have started noticing a little bit more often. So, how a monoid relates to a group is something that I can't remember off the top of my head. And I know how semirings relate to monoids, but how monoids then relate to rings and groups, I can't really remember. And so, these things are sort of all related. But the relation is not something I can spill out off the top of my head. Sorry. [Laughs] CHARLES: No, It's no worries. You know, I feel like… ELRICK: It's all good. CHARLES: What's funny is I feel like having these discussions is exactly like the discussions people have with any framework of using one that we use a lot, which is EmberJS. But if you could do with React or something, it's like, how does the model relate to the controller, relate to the router, relate to the middleware, relate to the services? You just have these things, these moving parts that fit together. And part of… I feel like exploring this space is really, absolutely no different than exploring any other software framework where you just have these things, these cooperating concepts, and they do click together. But you just have to map out the space in your head. JULIE: Yeah. This is going to sound stupid because everybody thinks that because I know Haskell I must know all these other things. But I just had to ask people to recommend me a book that could explain the relationship of HTML and CSS, because that was completely opaque to me. CHARLES: [Laughs] Yeah. JULIE: I've been involved in the making now of several websites because of the books and stuff like that. And I have a blog. It's not WordPress or anything. I did that sort of myself. So, I've done a little bit with that. But CSS is really terrifying. And… CHARLES: Right. Like query selectors, rules, properties. JULIE: Yeah. ELRICK: [Laughs] CHARLES: Again, might as well be groups and semigroups and monoids, right? JULIE: Right, right. ELRICK: Yeah. CHARLES: [Laughs] ELRICK: That is really interesting. [Chuckles] I've never heard anyone make that comparison before. But it's totally true, now that I'm thinking about it. JULIE: Yeah, yeah. CHARLES: Yeah. In the tech world we are so steeped in our own jargon that we could be… we can reject one set of jargon and be totally fine with another set. Or be like, suspicious of one set of concepts working together and be totally fine with these other designations which are somewhat arbitrary but they work. JULIE: Right. CHARLES: So, people use them. JULIE: So, it's like what you've gotten used to and what you're familiar with and that seems normal and natural to you. [Chuckles] So, the Haskell stuff, most of it seems normal and natural to me. And then I don't understand HTML and CSS. So, I bought a book. [Laughter] CHARLES: Learning HTML and CSS from first principles. JULIE: Yes, yeah. I just wanted to understand. I could tell that they do relate to each other, that there is some way that they click together. I can tell that by banging my head against them repeatedly. But I didn't really understand how, and so yeah. So, i've been reading this book to [Laughs] [learn] HTML and CSS and how they relate together. That's so important, just figuring out how things relate to each other, you know? CHARLES: Yeah. ELRICK: Yeah. That is very true. JULIE: Yeah. ELRICK: We can trade. I can teach you HTML and CSS and you can teach me Haskell. JULIE: Absolutely. ELRICK: [Laughs] CHARLES: There you go JULIE: [Laughs] ELRICK: Because I'm like, “Ooh.” I'm like, “Oh, CSS. Great. No problem.” [Laughter] ELRICK: Haskell, I'm like “Oh, I don't know.” JULIE: Yeah. CHARLES: Yeah. ELRICK: [Laughs] CHARLES: No, it's amazing [inaudible] CSS. ELRICK: Yeah. CHARLES: It is, it's a complicated system. And it's actually, it's in many ways, it's actually a pretty… it's a pretty functional system, CSS is at least. The DOM APIs are very much imperative and about mutable state. But CSS is basically yeah, completely declarative. JULIE: Right. CHARLES: Completely immutable. And yeah, the workings of the interpreter are a mystery. [Laughs] ELRICK: Yup. JULIE: YEs. And you know, for the Joy of Haskell website we use Bootstrap. And so, there was just like… there's all this magic, you know? [Laughs] ELRICK: Oh, yeah. CHARLES: Yeah. JULIE: Oh look, if I just change this little thing, suddenly it's perfectly responsive and mobile. Cool. [Laughter] JULIE: I don't know how it's doing this, but this is great. [Laughs] CHARLES: Yeah. Oh, yeah. It's an infinite space. And yeah, people forget what is so easy and intuitive is not and that there's actually a lot of learning that happened there that they're just taking for granted. JULIE: I think so many people start from HTML and CSS. That's one of their first introductions to programming, or JavaScript or some combination of all three of those. And so, to them the idea that you would be learning Haskell first and then coming around and being like “Oaky, I have to figure out HTML,” that [seems very] strange, right? [Laughter] CHARLES: Yeah. Well, definitely probably stepping into bizarro world. JULIE: And I went backwards. But [Laughs] CHARLES: Yeah. JULIE: Not that it's backwards in terms of… just backwards in terms of the normal way, progression of [inaudible] CHARLES: Yeah. It's definitely the back door. Like coming in through the catering kitchen or something. JULIE: Yes. CHARLES: Instead of the front door. Because you know the browser, you can just open up the Dev Tools and there you are. JULIE: Exactly, yeah. CHARLES: The level of accessibility is pretty astounding. And so, I think t's why it's one of the most popular avenues. JULIE: Oh, definitely. Yeah. ELRICK: It's the back door probably for web development but not the back door for programming in general. JULIE: Mm, yeah. Yeah. CHARLES: Yeah. It seems like Haskell programming has really started taking off and that the ecosystem is starting to get some of the trappings of a really less fricative developer experience in terms of the package management and a command line experience and being able to not make all of the tiny little decisions that need to be made before you're actually writing ‘hello world'. JULIE: Right. ELRICK: Interesting. Haskell has a package manager now? CHARLES: Oh, it has for a while. ELRICK: Oh, really? What is it called? I have no idea? Do you know the name off the top of your head? CHARLES: So, I actually, I'm not that familiar with the ecosystem other than every time I try it out. So I definitely will defer this question to you, Julie. JULIE: This is going to be a dumb question, I guess. What do we mean by package manager? CHARLES: So, in JavaScript, we have npm. The concept of these packages. It's code that you can download, a module that you can import, basically import symbols from. And Ruby has RubyGems. And Python has pip. JULIE: Okay, okay. CHARLES: Emacs has Emacs Packages. And usually, there's some repository and people could publish to them and you can specify dependencies. JULIE: Right, yeah. Okay, so we have a few things. Hackage is sort of the main package repository. And then we have another one called Stackage and the packages that are in Stackage are all guaranteed to work with each other. CHARLES: Mm, okay. JULIE: So, on Hackage, some of the packages that are on Hackage are not really maintained or they only work with some old versions of dependencies and stuff like that, so the people who made Stackage were like “well, if we had this set of packages that were all guaranteed to work together, the dependencies were all kept updated and they all can be made to work together, then that would be really convenient.” And then we have Cabal and we have Stack are the main… and a lot of people use Nix for the same purpose that you would use Cabal or Stack for building projects and importing dependencies and all of that. CHARLES: Right. So, Cabal and Stack would be roughly equivalent then to the way we use Yarn or JavaScript and Bundler in Ruby. You're solving the equation for, here's my root set of dependencies. Go out and solve for the set of packages that satisfy. Give me at least one solution and then download those packages and [you can] run them. JULIE: Yeah, yeah. Right, so managing your dependencies and building your project. Because Haskell's compiled, so you've got to build things. And so yeah, we have both of those. CHARLES: And now there's like web frameworks and REST frameworks. JULIE: Oh there are, yeah. We have… CHARLES: All kinds of stuff now. JULIE: We had this big proliferation of web frameworks lately. And I guess some of them are very good. I don't really do web development. But the people I know who do web development in Haskell say that some of these are very good. Yesod is supposed to be very good. Servant is sort of the new hotness. And I haven't used Servant at all though, so don't ask me questions about it. [Laughter] JULIE: But yeah, we have several big web frameworks now. There are still some probably big holes in the Haskell ecosystem in terms of what people want to see. So, that's one thing that people complain about Haskell for, is that we don't have some of the libraries they'd like to see. I'd like to see something… I would really like to see in Haskell something along the lines of like NLTK from Python. CHARLES: What is that? JULIE: Natural language toolkit. CHARLES: Oh, okay. JULIE: So yeah, Python has this… CHARLES: Yeah, Python's got all the nice science things. JULIE: They really do. And Haskell has some natural language processing libraries available but nothing along the lines of, nothing as big or easy to use and stuff as NLTK yet. So, I'd really like to see that hole get filled a little bit better. And you know… CHARLES: Well, there you go. If anyone out there is seeking fame and fortune in the Haskell community. JULIE: That's actually why I started learning Python, was just so that I could figure out NLTK well enough to start writing it in Haskell. [Laughter] JULIE: So, that's sort of my ambitious long-term project. We'll see how that goes. [Laughs] CHARLES: Nice. Before we wrap up, is there anything going on, coming up, that you want to give a shoutout to or mention or just anything exciting in general? JULIE: Yeah, so on March 30th I'm going to be giving a talk at lambda-squared which is going to be in Knoxville and is a new conference. I think it's just a single-day conference and I'm going to be giving a talk about functors. So, I'm going to try to get through all the exciting varieties of functors in a 50-minute talk. CHARLES: Ooh. JULIE: So, we'll see how that goes. Yeah. And I am still working with Chris Martin on ‘The Joy of Haskell' which should be finished this year, sometime. I'm not going to… [Laughter] JULIE: Give any more specific deadline than that. And in the process of writing Joy of Haskell, I was telling him about some things that, some things that I think are really difficult. Like in my experience, teaching Haskell some places where I find people have the biggest stumbling blocks. And I said, “What if we could do a beginner video course where instead of throwing all of these things at people at once, we separated them out?” And so, you can just worry about this set of stumbling blocks at one time and then later we can talk about this set of stumbling blocks. And so, we're doing… we're going to start a video course, a beginner Haskell video course. I think we'll be starting later this month. So, I'm pretty excited… CHARLES: Nice. JULIE: About that. Yeah. CHARLES: Yeah, I know a lot of people learn really, really well from videos. There's just some… JULIE: Yeah. [Inaudible] for me, so I'm a little nervous. But [Laughs] CHARLES: Yeah, especially if you can do… are you going to be doing live coding examples? Building out things with folks? JULIE: Yeah. CHARLES: Yeah. Well, you just needn't look no further than the popular things like RailsCasts and some of the… yeah, there's just so many good video content out there. Yeah, we'll definitely be looking for the. JULIE: Cool. CHARLIE: Alright. Well, thank you so much, Julie, for coming on. JULIE: Well, thank you for having me on. Sorry I went down some… I went kind of down some rabbit holes. Sorry about that. [Laughs] CHARLES: You know what? You go down the rabbit holes, we spend time walking around the rabbit holes. JULIE: [Laughs] CHARLES: There's something for everybody. So… [Laughter] CHARLES: And ultimately we're strolling through the meadow. So, it's all good. JULIE: [Laughs] Yeah. CHARLES: Thank you too, Elrick. JULIE: It was nice talking to you guys again. CHARLES: Yeah. ELRICK: Yeah, thank you. CHARLES: If folks want to follow up with you or reach out to you, what's the best way to get in contact with you? JULIE: I'm @argumatronic on Twitter and my blog is argumatronic.com which has an email address and some other contact information for me. So, I'd love to hear questions, comments. [Laughs] Yeah. I always [inaudible]. CHARLES: Alright, fantastic. JULIE: To talk to new people. CHARLES: Alright. And if you want to get in touch with us, we are @TheFrontside on Twitter. Or you can just drop us an email at contact@frontside.io. Thanks everybody for listening. And we will see you all later.

Sam Cates: @SamCates | GE Ventures Show Notes: 02:01 - What Corporate Investing Looks Like 03:48 - Presenting Ideas For Funding 09:01 - Democratizing Venture Capital 10:17 - ICOs and Cryptocurrency 13:53 - Evaluating Companies to Fund 21:09 - Investing in Potential Competitors 24:42 - Looking For Funding as a Company 28:04 - “Mentoring” Ideas/Companies 30:07 - Monitoring/Evaluating Company Metrics 32:47 - Putting Together a Basic Business Plan 36:05 - Making Choices: Investor and Company-wise Resources: Resin.io Series A, B, C Funding Angel Investor Seed Money Initial Coin Offering (ICO) AngelList Crunchbase Fred Wilson's Blog: (AVC.com) Transcript: CHARLES: Hello everybody and welcome to The Frontside Podcast, Episode 92. My name is Charles Lowell, a developer here at The Frontside and I am your podcast host-in-training kicking it off in 2018. [Inaudible] of our first episode. We've got Elrick also joining us. Hello, Elrick. ERICK: Hey. How you doing, Charles? CHARLES: I'm doing well. I'm doing well. You having a good new year so far? ERICK: Yeah, it's great. There's a snowstorm passing through today. So, I'm going to break in the New Year shoveling. CHARLES: Let us know if we need to parachute in some shovels for you. ERICK: [Laughs] CHARLES: And then with us today, we have Sam Cates on the show who is… a lot of times we have developers on the show. He's actually a venture… what would you describe yourself as? SAM: Yeah, I'd say I'm a venture investor with GE Ventures. So, on the corporate investing side. CHARLES: Okay. Now, I didn't even know that GE actually had a corporate investing side. Is that pretty common for a large company? SAM: You know, it's becoming increasingly common. I think in 2015 there was actually a peak of activity coming from corporate venture capital groups. And I've only seen the number of firms escalate since then. Although the dollars invested stays pretty consistent. But if you look at a lot of big companies, particularly in the common tech world like Cisco, Google, Intel, they have historically had large venture firms inside of themselves. And then GE and a lot of other industrials have since followed suit. We've been at it for about five years and we see it increasingly. CHARLES: And so, have you been with them since the beginning? SAM: Yeah, just about. I've actually been with GE for about nine years now. So, I was on the operating side in a number of the industrial businesses before I joined GE Digital and then GE Ventures. And so, it was just after GE Ventures got kicked off. CHARLES: Oh, that's exciting. So, what is it… now, we actually got connected to you through one of the companies that you actually invested in. It's something that we use and we're very interested in. Why don't you tell us a little bit about what your job looks like on a day-to-day basis and what companies you invest in? SAM: Sure. I really focus a lot of my time on Internet of Things companies. So, that's a really big trend that GE has been a part of and a leader in over the past few years. And so, we spend time investing in companies that are directly working with GE or playing in similar spaces to us. And so, Elrick and I actually met at a hackathon for one of those companies. And I always like to use that as an example because it's a good one, to demonstrate the kinds of investments we make. And that's Resin.io. I know you guys have done an episode or two talking with them. But that for example was a ‘Series A' investment that we made about two years ago. And then company essentially helps developers build connected products. And so, that's something that GE cares a lot about. We had people inside the company who found the product and loved it and that's actually how we met. CHARLES: When you say ‘Series A', can you give a brief overview of what the different stages of funding of a startup might be? SAM: Yeah, yeah, certainly. So, maybe if I take a step back and answer your original question on what I do on a day-to-day basis. A lot of my job is meeting with all kinds of new companies, whether they be early stage, usually things that would be seed funding – and we'll go into what some of those things mean – all the way through the late stage which would be companies that are maybe on the border of going public or are already profitable. And so, if we go into what kinds of investors there are, I think that's probably an interesting subject to talk more about. But they're a whole wide variety. When I said ‘Series A' I just meant a company that was at what we would call the ‘Series A' stage, and the letters act just like you'd expect. So, there's ‘Series A', ‘Series B', ‘Series C', and so on. And they all, they tend to look similar at those stages in terms of sizes and progress. But there is a range, and no two company is the same. ERICK: In today's world, it's very easy for people to create a startup. They can write some code and they can either come up, get a Raspberry Pi or some microcontrollers or whatever it is, and either do an IoT startup or a software startup. Now, when you get to the point where you have an idea and you kick it off initially, how do you go about then saying, “Let me get some funding.” How do you even get funding? SAM: Sure, yeah. And to your point, there's a huge range of technologies that are making it easier to start almost any kind of company. It's a great time to be an entrepreneur, whether it be 3D printing for hardware products, all the technologies that you were mentioning, AWS, all this stuff is contributing to reducing the cost to allow companies or people to create companies. And so, once people have gone out and experimented with some of these things and built what they think is a product the market wants, often if they require more money which may be for acquiring customers through things like Facebook Ads or simply doing further product development to make sure the product is somewhere that more customers could use it, often they can't finance it just through their own revenue. And so, there are typical stages and types of investors that people go approach looking for money. ERICK: Okay. What are those Series? I remember you mentioned something like a ‘Series A' investment. So, initially when you're looking for an investment, is that where you would… category you would be in as a startup looking for investment? They would consider you a ‘Series A' startup? SAM: Well, I want to caveat and just say every company is different. So, I see companies that… ERICK: Gotcha. SAM: Start out at a much later stage because they're able to bootstrap to that point. And bootstrap is the word that I use for a company that funds its own investment. They get paid by customers and they use that money to continue building the product. But if I talk about the range of types of financing a company may go for, I think the way that most people categorize this, first people often raise from friends and family or angels. And so, it's just money to get off the ground and maybe to pay the rent while you're doing some of that experimenting we were talking about. And then commonly after that is a seed round. And a seed round tends to be a little more institutional. So, it's maybe a more formal set of funds who exclusively invest in companies that are often pre-revenue but they have a product, or at least the beginnings of a product. And so, that's a really common category of investors. And then you get to ‘Series A' and the letters can escalate from there to the point where… ERICK: Gotcha. SAM: There can be some later rounds when they'd be ‘Series F' or even beyond, I guess. CHARLES: Right. So now, what are generally the terms on these? So, for my angel investments or my seed investments, I assume what distinguishes these is essentially how much ownership of the company you're getting for how much money. And those kind of, those change as the product solidifies. SAM: Yeah. CHARLES: And the potential becomes more visible. SAM: Yeah, it's a wide… and again, these are all… the venture is a world of ranges. There's a really wide difference between the two ends of any spectrum. So, I'll just talk in generalities though. So, I think the latest report that I've seen at least for an annual basis was PitchBook's 2016 report. And they were laying out some of the medians. So, for seed stage deals I believe it was something like one and a half million dollars raised was the median on a pre-money valuation of six and a half million. And that just means the company is worth, investors say the company is worth six and a half million dollars today. And we're going to give you a million and a half dollars invested at that price. CHARLES: So roughly, a sixth… they would take a sixth of the company then in return? SAM: Yeah. CHARLES: Okay. ERICK: Ah. CHARLES: Okay. ERICK: Okay. CHARLES: I see. That makes sense. So now, back to Elrick's original question. If I'm, I've got my product. Or I've got this idea. I've written some code. I've turned it into a prototype product. Maybe I'm moving through these various stages. What type of VC am I going to be looking for? How do I actually find the right type to be talking to? I guess what types are there even? SAM: Yeah. And one part… we mentioned a lot of the technologies that are making it easier to start companies. One part that also makes it easier is the proliferation of financing options, whether it be even more investors in these traditional structures we talked about like seed and A. And then there are other options that are emerging, things like you see a lot of people raising through what they call Initial Coin Offerings or ICOs. And then there are also things like AngelList which are attempting to democratize the investing process, make it more accessible. So traditionally, a lot of the seed A, B investors, they tend to be network-based, which can be a challenge for a lot of people that are maybe not in Silicon Valley or not a part of that network already. And so, one thing you can do is obviously go search databases that are on the web, things like Crunchbase. It's a free resource. It has a lot of deal history for investments that people have made. And it's a great resource for knowing, “Okay, this investor cares about these things.” And then in addition to that, there are also platforms that people can put their companies on. Like I mentioned, AngelList. And that's somewhere that you can list your company, you can meet investors, and they actually have some backend to actually support the investing process as well. CHARLES: So, there were two acronyms in there, or two specific technologies. [Chuckles] CHARLES: You talked about ICOs which I assumed that you said it was Initial Coin Offering. Not like insane clown offering. [Laughter] CHARLES: Which I would love to see. And then AngelList. So traditionally, these had been very network-based which brings to mind the capitalists of Old England or whatever where there's a bunch of people with cigars in a room and I realize it's not actually like that. What are each of these things? The AngelList and the ICOs? And how do they democratize that process? SAM: It's funny you should mention the old times. I think a good example of that is there are a lot of stories about the founding of General Electric. It's a 126-year-old company and back then it was largely, it was Thomas Edison working with I believe was JP Morgan to get it off the ground. And so, today there's still a bit of the network piece you're mentioning. But I think of AngelList as a place that you can essentially market to investors. If you think about the types of people that are on there, it's people that are looking to invest money in early stages in startups. And I'm not a big user of AngelList because I tend to be investing a little bit later. So, I really recommend anybody who's interested, just go check it out. It's I believe just Angel.co. CHARLES: And what about an ICO? SAM: So, an ICO is a more modern one. And it's kind of fraught with some concerns around regulations and transparency today. But I think since Thanksgiving there's been a massive wave of conversation about cryptocurrencies. And an ICO is essentially a way of creating your own cryptocurrency. The way I always explain to people, I love the analogy that people make around, think of it like I want to go build an amusement park. And in that amusement park, everything, rides, food, everything, is going to be denominated and payable in Sam-bucks. CHARLES: Ah, right. SAM: And… [Chuckles] And so, my options… CHARLES: [Laughs] That makes sense. SAM: Yeah. And my options are I can go to a bank and borrow money, I can go to investors and say, “Hey, give me the 10 million dollars it's going to take to build it,” or I can just go to the people in the place where I'm building it and say, “You want this amusement park to exist? Why don't you pre-buy these Sam-bucks?” And each one is going to cost a dollar today. And we create this universe of Sam-bucks and they're essentially valuable once you can use them in the park. And there are certainly exceptions. There are other versions of cryptocurrencies and other uses for them. But that's a conversation for another day. CHARLES: Ah, mm. SAM: I think that's just a good, easy way to understand it. CHARLES: Oh no, I like that. It's like, well not quite like carnival tickets. But yeah, that's something that everyone's familiar with. Same thing as the Xbox Marketplace. Very similar thing. So, the idea is you would buy a bunch of Sam-bucks… you would get them at pennies on the dollar, so to speak, today. SAM: Yeah, right. By the time it opens, maybe a hotdog would cost just one Sam-buck. CHARLES: Right. SAM: Whereas, when it's coming in, we'd have to spend five dollars to get that one Sam-buck. Right, the idea being those people who got in early will be rewarded. And you can see it's like a further extension of a Kickstarter or something else that you're allowing people to pre-buy into a network. CHARLES: Right. Right, okay. I can see that. ERICK: That's very interesting. [Laughs] CHARLES: And so, it's got a range of options too, because if you're really interested in the services you can go ahead and spend them on the services and get a lot of value that way or you can actually trade for someone who does want the services if you don't. SAM: I think that's exactly right. And it's just, the one that I think I would just caveat is there is a huge amount of concern at the moment, and maybe concern is too strong a word, but uncertainty around one, what are the value of these coins, these tokens? And two, how will governments react to something that looks potentially like a security or a currency? And so, that's something that still is being worked through. And even though they haven't figured that out there's still a massive amount of money being raised through these ICOs. CHARLES: [Laughs] So, it does beg the question. Why is a cryptocurrency necessary? Why not just use Xbox Marketplace points? Why not just say, “Here are Sam-bucks.” SAM: [Chuckles] CHARLES: And there's a row in my database. [Laughter] CHARLES: That's your balance of Sam-bucks. SAM: So, I think we're about to get way beyond the [inaudible] [Laughter] SAM: But I think the argument would be that some of these things are better decentralized. So in my example, you're right. That might just make more sense. But I think there are some examples around cryptocurrencies that are supporting a network of decentralized services where a centralized database historically was inconvenient or didn't provide the amount of transparency that people were looking for. CHARLES: Right, right. SAM: And so, that's a topic for a whole other podcast. CHARLES: Yeah, right. No, it makes sense. SAM: [Laughs] CHARLES: I think it's a matter of scale, right? If you're going to be just buying services but if you're going to have secondary markets where you're trading in this currency, I can see that. So, let's… [Chuckles] We'll reel that back in. SAM: [Chuckles] CHARLES: And ask a question that occurred to me. So now, we talked about your day-to-day. What exactly, when you're looking at a company to basically give money to, what are you looking for? What are the things you're like, “Oh man, I want to throw dollars at this company,” versus, “Mm. I'm going to keep them and give them some feedback and send them on their way.” SAM: There's always a set of factors that we evaluate. And I think the waiting is probably different for different types of investors. And then there's I'd say for me as a corporate VC being a part of GE, there's an extra lens which is, how is this relevant to GE? What does it mean for GE to be an investor? But if I think about just the kind of general industry lines it's: team is a really big one. So, who's building this company? Do I believe in their ability to reach this vision that they're laying out for me? Another one would be technology. What have they actually built? Is that hard to build? Do the things they want to build in the future, will those be hard to build? And do they have the skills and the people to do it? Then their technology, maybe an extension of that would be intellectual property. And besides intellectual property, just defensibility of a business in general. So then, you start thinking about, can somebody else just come along and to the same thing? Because if so, then maybe there's not a strong advantage in what the company has done so far. And then lastly, it's also just traction. How far along are they? How much have they proven the ability to execute on the plan that they're laying out? CHARLES: Right. ERICK: So, you're a corporate investor. So, there's other types of investor like an institutional VC? What are the differences between an institutional VC and a corporate VC and the other types of VC? Potentially what they'd be looking for, in terms of what they wanted best. SAM: Yeah. So, I think generally I categorize investors as institutional or corporate. And corporate [inaudible]… ERICK: Yeah. SAM: Corporate or strategic. And then there are people who exist on a spectrum there. But generally, an institutional means this is a group that is raising money from a set of limited partners who are the people who invest in the fund that are pension funds or wealthy individuals. They're large pools of institutional capital and their pure purpose is to earn return. And they may have a certain focus because they believe in this part of the market, or they like this kind of company or the stage of company. But essentially, their job is to return more money to the limited partners of that fund that were put in. That's their role in the world. And then on the corporate side, if we go the most extreme version of corporate VC, this is a group that is a part of a larger corporate. They're investing that company's money. So, in this case for me it's GE. I'm investing GE's money into these startups. And that means that I only have a single backer being GE. And I also maybe have a different lens, because my purpose is one, to earn financial return. I want to go out and I want to find good companies. I want to earn returns just like the other institutional venture capitalists. But I also have the goal of, and the strategic goal may differ by company, but for me it's about how can I help GE advance? How can I help GE understand a market? And how can GE be helpful to this company in achieving their goals? And so, for each company we use that lens as well, as a corporate. CHARLES: What I'm hearing is that you want to invest… I guess the thing is you can experience return that's not just cash. It's not just dollars. You'll experience return in raising the ocean of the business that GE is in, right? So… SAM: You said it much better than I did. [Laughter] CHARLES: Well, it's all… paraphrasing is actually easy. [Laughter] ERICK: Oh, yeah. SAM: An important skill. CHARLES: That makes a lot of sense. So, the question I have then is, you said you were looking for companies that kind of swim in a specific ocean. And each company is farther along. Are you usually finding this company I want to work with, like you are going out and finding them? Or they're coming to you looking for investment? Or is it really just, depends. SAM: So, we call that part of the process sourcing, sourcing investments. And they come from all over. So for us, there are a few different ways. One is we tend to be thesis-driven. Meaning we go out and we say, the world is changing in this way and therefore we're interested in this kind of company. And so, we'll proactively go out and research. We're also, I mentioned, a little later stage. So, I don't tend to do seed investments. I tend to do ‘Series A' and more often ‘Series B' and later. So, companies that have often already raised a seed round or raised a ‘Series A' round. So, I can actually search databases to say, “Okay, in the last two years who has raised a seed round or ‘Series A' round and these other things I'm looking for whether it be location or tied to investors or other things.” So, that's one way of being proactive is saying I want to go out and look for companies in this space that look like this. And that can be either like I mentioned, desktop research like searching the web, searching databases. Or it can be just going to conferences, right? So, on thing we spend a lot of time on in the IoT world is artificial intelligence and machine learning. It's been a big, big topic over the last year that a lot of people have invested in. So, we may go to different conferences that focus on that topic, meet lots of people that are working on it. Some companies, some individuals that are either investing in or advising their companies. And we'll talk to them. What companies are rising out of that space that we should be looking at? What technologies are changing in that space that we should be thinking about? And just trying to get smarter so that we can make the right investments and help the right companies find their way to work with GE and make our products better and help them advance their own enterprise. CHARLES: Are you investing with a mind that eventually GE might acquire this company and integrate it into GE itself? Or is it really just, “Hey, we're just going to take a part of it. We're going to have maybe a seat on the board to be able to steer a little bit. But we're pretty much going to let it be its own thing with its own autonomy and go where it was and just benefit through those secondary and tertiary effects.” SAM: Yeah, acquisitions from our portfolio by GE happened. But they're certainly not the explicit goal or our focus. I know we've had one, maybe two of our portfolio companies acquired by GE, one that I was directly working with called Bit Stew. So, we made the investment in the company. It was with the goal of using their data management platform for a lot of our applications. And at some point in working with GE and GE Digital, they decided, you know, this would make sense to be a part of GE. That wasn't why we made the investment. But it did end up being acquired by GE. And I know the team is doing really well. And it's been at GE for about a year now. So, it does happen. But when I said one or two, that's versus a portfolio of a hundred plus companies. CHARLES: Right. SAM: Since we started investing. And so, that's not what we're looking to do every time. Much more often it's about again, how does the company make GE more competitive and a better company, a better place to work. And then how do we help them advance their goals? Whether it be bringing them developers, or finding them other routes to market, or just being a customer. CHARLES: Right. SAM: So, that's really how we think about strategic value. There's a lot of different ways to create it. CHARLES: Yeah, I'm curious. Because it seems like also in a lot of these companies you're investing in potential competitors. Extensively you're operating if not in the exact same market, maybe very similar markets. There's a little bit of overlap. And so, you're kind of investing in potential competitors, right? So, where's the balance of here we're funding our competitors versus we're going to move into these markets ourselves. SAM: Yeah, and funding of “competitors” can happen. I think that we talk about that more in theory and say, “Oh sure, we'd be willing to fund a company that's out disrupting the space that we're playing in.” And we do that. It's rare that you see startups that are directly head-on competing with much more established companies like GE or other industrials or even other consumer companies. They don't take these companies head-on because that's not a way that startups have been successful in the past, right? We talk much more about disruption and saying, how is this company doing something that may indirectly compete with GE? So, you think about things like, for anybody that's not familiar with GE… actually, a lot of people associate us with our appliances which we actually don't manufacture anymore. That's [inaudible]. [Chuckles] SAM: We sold that business a few years ago. Almost everything we sell is like big, heavy industrial equipment. So, we sell aircraft engines, locomotives. We sell gas turbines, wind turbines. So, here and there a couple of things that do power generation. One trend that's affecting that industry is distributed generation of energy, energy storage. And those are parts of the market that are a less significant part of GE's business than say, heavy-duty gas turbines that sit in a power plant and generate a massive amount of power. And so, if you look at that and say, “Wow, GE Ventures is out funding storage companies. Does that mean they're funding competitors?” Well, it means that we're funding innovation that may disrupt the future of our business, but that's part of being a VC and that's part of the value that GE Ventures brings to GE. CHARLES: Right. SAM: We're out there looking at markets before they're large enough or in scope for GE. CHARLES: Mmhmm, right. And so, yes you're disrupting the space but then you're going to be a part of that disruption and have strong connections to those markets if you need to actually migrate your business completely over to them. That's kind of what I'm hearing. SAM: Yeah, absolutely. Better to disrupt yourself, right? CHARLES: [Chuckles] SAM: And be a part of the ecosystem in the future because I think the future happens with or without you. And it's really key that we get out in front of it and a part of that, a part of that discussion, a part of that process. CHARLES: And so now, you've been saying that this is, GE, this has been pretty explosive? There's a lot more happening through GE Ventures. There's a lot more happening in other companies globally, having these corporate ventures. Where do you think the balance is going to lie to say, “Hey,” I'm just going to throw out some numbers, just for theory here, it's like, “10% of our business is essentially this distributed network of semi-autonomous or mostly autonomous startups. And then we have our core business.” Does that stabilize at 50/50? Does it stabilize at 75% the other way with GE essentially becoming a capital management company? Or is it somewhere in the middle? SAM: So, GE Ventures will never be a meaningful part of GE's revenue, a meaningful part of its business as a percentage. The overall venture industry is full of funds that are on the order of like, bigger funds are on the order of, in the billions. The single-digit billions. And GE itself is a much, much larger company. Well over a hundred billion dollars in enterprise value. So, I think GE Ventures will always be a small part of the company financially. And the impact will be largely felt through how we help the rest of GE navigate the future. ERICK: You said that sometimes you go and look for companies, startups to invest in or sometimes startups come to you or come to a VC looking for funding. Now, I'm a developer or a startup founder. And I'm going to look for funding. What are some of the mistakes or pitfalls that you see that startup founders or people with an idea fall into when looking for funding that you can help them avoid? SAM: Yeah, and we do see companies that come to us. So, I mentioned a lot about how I go out looking for companies based on a thesis or a set of relevant factors or relevant things for GE. But we do have a number of inbound requests. People know some of the bigger VC brands. They know GE the big company. So, we do get inbound interest and we also get referrals from networks of VCs and some are employees and other things. But for the companies that are seeking us out, the ones that are going out looking for funding, there are some things that are really well-known in Silicon Valley and other places, or you could research online and find, but may not be obvious at first. And so, I think the first one is, who are you talking to? What investors are you seeking out? Depending on what stage you're at, what kind of business you're in, you have to understand what the landscape of potential investors are and which ones might be interested in a company like yours. So, I think there are tons of good mentors that can help people navigate that. Maybe less commonly outside of Silicon Valley, in Boston, New York, in the places where you have traditional venture ecosystems. But you see a ton of resources available online whether it be things like Fred Wilson's blog, AVC.com, or Crunchbase, TechCrunch. You can read and understand and from headlines tell what people care about. And I think that's fundamentally a really important first step. You don't want to waste an hour talking to somebody who will never… this is somebody that invests in really late-stage growth equity companies and I'm coming to them for my first investment. That's not going to work. So, I think finding the right people, step one. I think when you're going through the process of pitching and talking about your business, the pitfalls are all about understanding the strengths and weaknesses of your business and where you are today. And so, for every company, that's different. But I think just being open and honest versus glossing over a lot of the risks, these are all really risky companies. If they were easy, then you'd have a lot more competition. And so… [Laughter] SAM: I think that's one thing that I see, too. You have some company that comes in and say, “Look, here are the parts I've figured out and here are the parts I still have to figure out.” And that's a really good conversation to have. There are other companies where they say, “Look, we've figured the whole thing out. We just want you to give us some money.” And I don't think a lot of investors necessarily buy into that. And certainly, there are investors of every stripe. So, I may be speaking too broadly. But I think that's a really important part of the venture investment process, right? You're looking not just for money but also for counsel and for somebody that you're going to work with over the next, sometimes seven years or longer. CHARLES: Yeah. SAM: [Inaudible] going to be on your board and participating. So, it's a really important part. CHARLES: So, you're looking, you're actually looking not necessarily for all the answers but you're looking for the questions that they're asking, too. SAM: Yeah, absolutely. And demonstrating they understand the ins and outs of the business. And that they have the capacity to carry this onto that next stage and hopefully beyond. CHARLES: Mmhmm. So, now you said something that caught my interest there that you work with some people sometimes seven years. You enter into these long relationships. Do you generally ever do any type of, I want to say almost like… mentoring might be too strong of a word, but in the pre-investment, in other words before you actually invest in a company, do you ever work with them to prepare them for investment to say, “Hey, I think there's potential here. Work on A, B, and C and then let's talk.” And you have this image in your mind. You go, you pitch to an investor, and it's either thumbs up or it's like thumbs down and you never talk to them again. Versus, is there some ground in between where there's a conversation that evolves that eventually ends up in an investment being made? SAM: Absolutely. I think one of the parts of this industry is even when I'm not an investor in a company, I may know a company and say, “It's not a fit for me for GE Ventures but I still think that we can provide help.” It's one of the things I love about tech and about venture in general, is that people are often willing to pitch in, even when they don't have a direct financial incentive. And so, I see that a lot whether it's helping a company where we've met them and we later see an opportunity and say, “Oh, you should go and talk to this company or that company.” And then often, we may see a company that's pitching us ahead of where we would typically invest. Maybe they're looking for a ‘Series A' but given the space that they're in or what we're doing at the moment, it may not be the right time for us. And so, we'll continue to track along and keep up and get updates. Some companies do a really good job of actually providing proactive updates and sending out monthly or quarterly reports to investors they've met with before. I think there's a wide range of ways that founders do this. But it is a really good way to keep people interested in the prize. And then when you come back and say, “Hey, now I'm out raising my ‘Series B',” that's not a surprise. I knew that you were hitting these milestones, that you were doing everything you said you were going to do. And you've demonstrated a level of credibility that really adds to the pitch that you made the first time around. ERLICK: You said something, metrics. So, a venture capitalist, after they make an investment, what are some of the expectations that they may hold this startup that they just invested in… what are those expectations that they may hold them accountable for? Or those metrics that they'll be looking at? SAM: Yeah, so I think some of the really high-level ones that are common across businesses, generally growth is a really big one. So, I almost said revenue. But I wanted to caveat… [Laughter] SAM: And say growth could mean different things. It could mean number of developers. It could mean number of downloads if you're an app. It depends on what the business is. But I think growth is a huge one. Growth is a really important, that top line, that's what's going to drive a lot of the value in the business. And then below that, demonstrating that you can hit the milestones around things like margins. So, how profitable is each unit you're selling? Or how profitable is each customer? And lastly, how are you doing managing your spend? So, that's great that you're earning the right amount of money for each customer, but are you doing it by… do you have a massive number of employees and offices and all the things that are too expensive to allow you to use your money wisely as you reach the next stage? And so, those are the big milestones. It's really just growth, margins, and operating cost or burn rate as we call it. CHARLES: Mmhmm. So, that sounds like a lot of work to actually evaluate these companies. Do you do your due diligence once you've already moved in pretty solidly into the process? SAM: Yeah, these processes can move really fast. And depending on the timing, generally it's, you jump in, you learn as much as you can, as fast as you can, and you make a decision so the company can move on. I'll say there's a lot of work that goes into considering and deciding which companies to spend more time on, both for us and for them. We don't want to waste a company's time evaluating, going through more meetings, if it's not a really strong candidate for us. Because they could be spending that time better with other investors who are a better fit. And I'm not going to pretend to like the evaluation part. I have a lot of respect for the amount of not just work but of a person's energy and really, their life goes into these companies. And so, I think the hard part is building the company. And so, it's hard for me to say that evaluating is a hard part. I'm trying to understand as much as I possibly can in a month or two. I'm not going to know as much about the business as the founder does. And I'll be wrong a lot. I may miss something and not understand, whether it's because I don't see the market but it's there or because I have some underlying assumption about the way things should work that they don't meet. And I think that that's something that investors have to come to grips with. You try and get as smart as you can as fast as you can, but you're not always going to get to the right answer. ERLICK: You said that it was growth, spend, and profits were some of the metrics. That is almost all of the essential components of a business plan. I remember one time, one of our previous conversations, you emphasized how important it was for companies, or even at just a simple startup, to put together a basic business plan. Is that something that you can elaborate on a little? SAM: Yeah. So, most companies show up with a pitch deck. So, they have a set of PowerPoint slides and then they have a set of materials behind that where if you go deeper into an area they may have a white paper about their technology and they may have an Excel financial model that explains why they have these expectations about what growth and margins and all those things will look like. So, there are all of those pieces that come together into a business plan. The business plan could be written or it could be that PowerPoint. But very traditionally, it's a PowerPoint or some kind of presentation that is shared in person. There's usually a version that's sent in advance to confirm that the company and the investors should meet. And then once you clear that bar, there's a deeper presentation that often you'll give to either one or a set or a whole team of investors. And you'll go through and explain why it is you think this is a good investment opportunity for them and why you'd like to work together. And then you have a discussion about whether that's a good fit, about some of the underlying assumptions, and come to either a set of next steps for the diligence or a decision that it's not the right fit, it's not the right time to take the relationship further with more diligence and that kind of stuff. ERLICK: Yeah, because I see… well, I know a few people that have startup ideas and they kind of put the business plan on the back burner and put the actual prototype more at the forefront. They say, “Oh, we can worry about the business plan later.” [Laughs] SAM: [Chuckles] Well, I think… there's something to be said to that. There's something to be said for product and growth winning. So if you… Let's start at the early stages. If you have something that's working and that's really obvious, you may not need a… ERLICK: True. SAM: To go raise money. It all comes down to, do you have enough to get enough investors interested to raise the round that you want to raise? Because you want to have enough investors involved, enough demand, that you can be selective about who you want to work with and on what terms, right? So, what valuation and how much of the company am I giving them, and all of those things. So, if you can do all of those things with nothing but an app and one chart that shows a hockey stick of growth, that's awesome. CHARLES: You're hot. [Laughter] SAM: Often it does require much more and a much longer plan. So, even if you say, “Look, it's growing like crazy,” there's usually some set of questions behind that. So, that's great. Your free app is growing like crazy. How are you going…? [Laughter] SAM: To get paid for that? And you'll talk about that. And you'll say, “Here are the things we're planning on doing,” or here are the assumptions that we're making. And the more original, the more unique the business model is, the more discussion and explanation that may require. And that's where the business plan and a pitch deck come in handy, because it's a really good presentation aide or pre-reading to get to that answer faster. ERLICK: So, this evaluation it seems, is a two-way street. The VCs evaluating the company and also the company or the startup evaluating the VC to know whether it's going to be a good relationship. SAM: Oh, absolutely. Yeah, the best companies have choice. They have a number of investors who are interested in funding them. And certainly, that might be different at different stages or at different times, depending on what's going on in the economy and in tech and in other places. But generally, VC is a very competitive industry. I'm trying to sell my money and services as an investor versus other options that you have. And so, while it's maybe not as competitive as only one of us can buy the company like in an M&A situation. There are often more than one investor. There's still a very intense set of competition around, okay, who's going to be involved in the deal? How much money will they be able to invest? So, that's something that really can come in handy for founders. ERLICK: And what was that you just said there? M&A situation? SAM: Oh, sorry. When a company is being bought. So, when a company is being bought, it can look kind of like a fundraising process, but instead of selling a part of a company, you're selling the whole thing. And so, in that case, obviously it's a competitive situation where there's only one winner. And this is a different process. Often, the rounds that we're a part of, we're not… we're buying a minority stake just like any VC. We may be buying 5% of a company, 10% of a company. And often we're being joined by other venture investors. We really actually commonly partner with the institutional firms and they'll take a board seat. We'll invest alongside them and be an observer on the board and provide counsel. And so, it is a very competitive process. And that, while M&A is a winner-take-all, there is one buyer who is ultimately going to own this company going forward, the investing process for a venture is much more collaborative. But it is still competitive, because there can only be so many investors in one company. CHARLES: And you want to choose the right one on both… the right set. Alright. Well, I think we're running up against time. This has been a fascinating conversation into an aspect of our industry that really is providing the fuel that drives so much of this forward. So, I guess I'll close by asking you, already talked about Resin. We had them on the podcast. We love them. Are there any conferences or products that you're investing in that you feel like our audience might want to know about or anything like that? SAM: Well one, you mentioned Resin. CHARLES: Yeah. SAM: I know you guys have been a good friend to them and Elrick and I met at their hackathon. I would recommend to anybody, go try it out. It's a really cool way to play with hardware products. I am not a developer and I required a lot of help from Elrick at the hackathon. [Laughter] SAM: But at the same time, it is something that almost anybody can pull out of a box and start playing with. So, I think that's a great one. The episode you did on them were fantastic. So, I really enjoyed those ones. I'd say in general, I'm always out looking to meet new companies that are going to benefit from working with GE. I spend a lot of my time not just trying to invest but also trying to find partnerships for companies that we're looking at within GE, either selling to us or working with us. And so, if somebody thinks that there's an opportunity to do that, then I encourage them to reach out. Because I think there's a ton of opportunity. It's a really big company that really has a ton of opportunity for other partners. CHARLES: Alright. If they wanted to reach out, how would they get in touch with you? SAM: Yeah, I think maybe the best way to initially make contact, I tend to be pretty active on Twitter. So, my handle is just @SamCates. S-A-M-C-A-T-E-S. And you can also learn more through our website. If you're curious about some of the businesses I mentioned, so just GEVentures.com. And it's about to go through a whole refresh. So, go check it out. CHARLES: Alright. Well, fantastic. We will definitely look for that. And for everybody else, you can get in touch with us on Twitter at @TheFrontside or send us a line at info@frontside.io. Thank you everybody for listening. Thank so, so much Sam, for being on the podcast. SAM: Yeah, of course. It was a blast. I'm a big podcast fan and I've really enjoyed catching up on your episodes. CHARLES: Ah, and thank you Elrick, always. ERICK: It was great. SAM: Elrick, when you finish building your Raspberry Pi Battleships, I want to play. CHARLES: [Laughs] ERICK: Oh, yes. Yes. It's in the works, man. It's in progress. SAM: Alright, I'm waiting. CHARLES: Alright. Well, take it easy, everybody.

Tracy Lee: @ladyleet | ladyleet.com Ben Lesh: @benlesh | medium.com/@benlesh Show Notes: 00:50 - What is This Dot? 03:26 - The RxJS 5.5.4 Release and Characterizing RxJS 05:14 - Observable 07:06 - Operators 09:52 - Learning RxJS 11:10 - Making RxJS Functional Programming Friendly 12:52 - Lettable Operators 15:14 - Pipeline Operators 21:33 - The Concept of Mappable 23:58 - Struggles While Learning RxJS 33:09 - Documentation 36:52 - Surprising Uses of Observables 40:27 - Weird Uses of RxJS 45:25 - Announcements: WHATWG to Include Observables and RxJS 6 Resources: this.media RxJS RX Workshop Ben Lesh: Hot vs Cold Observables learnrxjs.io RxMarbles Jewelbots Transcript: CHARLES: Hello everybody and welcome to The Frontside Podcast, Episode 91. My name is Charles Lowell, a developer here at The Frontside and your podcast host-in-training. Joining me today on the podcast is Elrick Ryan. Hello, Elrick. ELRICK: Hey, what's up? CHARLES: Not much. How are you doing? ELRICK: I'm great. Very excited to have these two folks on the podcast today. I feel like I know them… CHARLES: [Laughs] ELRICK: Very well, from Twitter. CHARLES: I feel like I know them well from Twitter, too. ELRICK: [Laughs] CHARLES: But I also feel like this is a fantastic company that is doing a lot of great stuff. ELRICK: Yup. CHARLES: Also not in Twitter. It should be pointed out. We have with us Tracy Lee and Ben Lesh from This Dot company. TRACY: Hey. CHARLES: So first of all, why don't we start, for those who don't know, what exactly is This Dot? What is it that you all do and what are you hoping to accomplish? TRACY: This Dot was created about a year ago. And it was founded by myself and Taras who work on it full-time. And we have amazing people like Ben, who's also one of our co-founders, and really amazing mentors. A lot of our friends, when they refer to what we actually do, they like to call it celebrity consulting. [Laughter] TRACY: Which I think is hilarious. But it's basically core contributors of different frameworks and libraries who work with us and lend their time to mentor and consult with different companies. So, I think the beautiful part about what we're trying to do is bring together the web. And we sort of do that as well not only through consulting and trying to help people succeed, but also through This Dot Media where it's basically a big playground of JavaScripting all the things. Ben and I do Modern Web podcast together. We do RX Workshop which is RxJS training together. And Ben also has a full-time job at Google. CHARLES: What do they got you doing over there at Google? BEN: Well, I work on a project called Alkali which is an internal platform as a service built on top of Angular. That's my day job. CHARLES: So, you've been actually involved in all the major front-end frameworks, right, at some point? BEN: Yeah, yes. I got my start with Angular 1 or AngularJS now, when I was working as a web developer in Pittsburgh, Pennsylvania at a company called Aesynt which was formerly McKesson Automation. And then I was noticed by Netflix who was starting to do some Angular 1 work and they hired me to come help them. And then they decided to do Ember which is fine. And I worked on a large Ember app there. Then I worked on a couple of large React apps at Netflix. And now I'm at Google building Angular apps. CHARLES: Alright. BEN: Which is Angular 5 now, I believe. CHARLES: So, you've come the full circle. BEN: Yeah. Yeah, definitely. CHARLES: [Chuckles] I have to imagine Angular's changed a lot since you were working on it the first time. BEN: Yeah. It was completely rewritten. TRACY: I feel like Angular's the new Ember. CHARLES: Angular is the new Ember? TRACY: [Laughs] BEN: You think? TRACY: Angular is the new Ember and Vue is the new AngularJS, is basically. [Laughs] CHARLES: Okay. [Laughter] CHARLES: What's the new React then? BEN: Preact would be the React. CHARLES: Preact? Okay, or is Glimmer… BEN: [Laughs] I'm just… CHARLES: Is Glimmer the new React? BEN: Oh, sure. [Laughs] CHARLES: It's important to keep these things straight in your head. BEN: Yeah, yeah. CHARLES: Saves on confusion. TRACY: Which came first? [Chuckles] BEN: Too late. I'm already confused. CHARLES: So now, before the show you were saying that you had just, literally just released RxJS, was it 5.5.4? BEN: That's right. That's right. The patch release, yeah. CHARLES: Okay. Am I also correct in understanding that RxJS has kind of come to very front and center position in Angular? Like they've built large portions of framework around it? BEN: Yeah, it's the only dependency for Angular. It is being used in a lot of official space for Angular. For example, Angular Material's Data Table uses observables which are coming from RxJS. They've got reactive forms. The router makes use of Observable. So, the integration started kind of small which HTTPClient being written around Observable. And it's grown from there as people seem to be grabbing on and enjoying more the React programming side of things. So, it's definitely the one framework that's really embraced reactive programming outside of say, Cycle.js or something like that. CHARLES: Mmhmm. So, just to give a general background, how would you characterize RxJS? BEN: It's a library built around Observable. And Observable is a push-based primitive that gives you sets of events, really. CHARLES: Mmhmm. BEN: So, that's like Lodash for events would be a good way to put it. You can take anything that you can get pushed at you, which is pretty much value type you can imagine, and wrap it in an observable and have it pushed out of the observable. And from there, you have a set of things that you can combine. And you can concatenate them, you can filter them, you can transform them, you can combine them with other sets, and so on. So, you've got this ability to query and manipulate in a declarative way, events. CHARLES: Now, Observable is also… So, when Jay was on the podcast we were talking about Redux observable. But there was outside of the context of RxJS, it was just observables were this standalone entity. But I understand that they actually came from the RxJS project. That was the progenitor of observables even though there's talk of maybe making them part of the JavaScript spec. BEN: Yeah, that's right. That's right. So, RxJS as it stands is a reference implementation for what could land in JavaScript or what could even land in the DOM as far as an observable type. Observable itself is very primitive but RxJS has a lot of operators and optimizations and things written around Observable. That's the entire purpose of the library. CHARLES: Mmhmm. So, what kind of value-adds does it provide on top of Observable? If Observable was the primitive, what are the combinators, so to speak? BEN: Oh, right. So, similar to what Lodash would add on top of say, an iterable or arrays, you would have the same sorts of things and more inside of RxJS. So, you've got zip which you would maybe have seen in Lodash or different means of combines. Of course, map and ‘merge map' which is like a flattening sort of operation. You can concatenate them together. But you also have these time-based things. You can do debouncing or throttling of events as they're coming over in observable and you create a new observable of that. So, the value-add is the ability to compose these primitive actions. You can take on an observable and make a new observable. We call it operators. And you can use those operators to build pretty much anything you can imagine as far as an app would go. CHARLES: So, do you find that most of the time all of the operators are contained right there inside RxJS? Or if you're going to be doing reactive programming, one of your tasks is going to be defining your own operators? BEN: No, pretty much everything you'd need will be defined within RxJS. There's 60 operators or so. CHARLES: Whoa, that's a lot. BEN: It's unlikely that someone's going to come up with one. And in fact, I would say the majority of those, probably 75% of those, you can create from the other 25%. So, some of the much more primitive operators could be used… TRACY: Which is sort of what Ben did in this last release, RxJS 5…. I don't know remember when you introduced the lettable operators but you… BEN: Yeah, 5.5. TRACY: Implemented [inaudible] operators. BEN: Yeah, so a good portion of them I started implementing in terms of other operators. CHARLES: Right. So, what was that? I didn't quite catch that, Tracy. You said that, what was the operator that was introduced? TRACY: So, in one of the latest releases of RxJS, one of the more significant releases where pipeable operators were introduced, what Ben did was he went ahead and implemented a lot of operators that were currently in the library in terms of other operators, which was able to give way to reduce the size of the library from, I think it was what, 30KB bundled, gzipped, and minified, to about 30KB, which was about 60 to 70% of the operators. Right, Ben? BEN: Yeah. So, the size reduction was in part that there's a lot of factors that went into the size reduction. It would be kind of hard to pin it down to a specific operator. But I know that some of the operators like the individual operators themselves, by reimplementing reduce which is the same as doing as scan and then take last, implementing it in terms of that is going to reduce the size of it probably 90% of that one particular file. So, there's a variety of things like that that have already started and that we're going to continue to do. We didn't do it with every operator that we could have. Some operators are very, very common and consequently we want them to be as optimized as possible. For example, map. You can implement map in terms of ‘merge map' but it would be very slow to do so. It might be smaller but it would be slower. We don't want that. So, there are certain areas we're always going to try to keep fairly a hot path to optimize them as much as possible. But in other spots like reduce which is less common and isn't usually considered to be a performance bottleneck, we can cut some corners. Or ‘to array' or other things like that. CHARLES: Mmhmm. TRACY: And I think another really interesting thing is a lot of people when learning RxJS, they… it's funny because we just gave an RX Workshop course this past weekend and the people that were there just were like, “Oh, we've heard of RxJS. We think it's a cool new thing. We have no plans to implement it in real life but let's just play around with it and let me learn it.” I think as people are starting to learn RxJS, one of the things that gets them really overwhelmed is this whole idea that they're having to learn a completely new language on top of JavaScript or what operators to use. And one of our friends, Brian Troncone who is on the Learning Team, the RxJS Learning Team, he pulled up the top 15 operators that were most commonly searched on his site. And some of them were ‘switch map', ‘merge map', ‘fork join', merge, et cetera. So, you can sort of tell that even though the library has quite a few… it's funny because Ben, I think the last RX Workshop you were using pairs and you had never used it before. BEN: Yeah. TRACY: So, it's always amusing for me how many people can be on the core team but have never implemented RxJS… CHARLES: [Laughs] TRACY: A certain way. BEN: Right. Right, right, right. CHARLES: You had said one of the recent releases was about making it more friendly for functional programming. Is that a subject that we can explore? Because using observables is already pretty FP-like. BEN: What it was before is we had dot chaining. So, you would do ‘dot map' and then call a method and then you get an observable back. And then you'd say ‘dot merge' and then you'd call a method on that, and so on and so forth. Now what you have is kind of a Ramda JS style pipe function that just takes a comma-separated list of other functions that are going to act upon the observable. So, it reads pretty much the same with a little more ceremony around it I guess. But the upside is that you can develop your operators as just higher-order functions. CHARLES: Right. And you don't have to do any monkey-patching of prototypes. BEN: Exactly, exactly. CHARLES: Because actually, okay, I see. This is actually pretty exciting, I think. Because we actually ran into this problem when we were using Redux Observable where we wanted to use some operators that were used by some library but we had to basically make a pull request upstream, or fork the upstream library to include the operators so that we could use them in our application. It was really weird. BEN: Yeah. CHARLES: The reason was because it was extending the observable prototype. BEN: Yeah. And there's so many… and that's one way to add that, is you extend the observable prototype and then you override lift so you return the same type of observable everywhere. And there are so many things that lettable operators solved for us. For example… CHARLES: So, lettable operators. So, that's the word that Tracy used and you just used it. What are lettable operators? BEN: Well, I've been trying to say pipeable and get that going instead of lettable. But basically there's an operator on RxJS that's been there forever called let. And let is an operator and what you do is you give it a function. And the function gives you the source observable and you're expected to return a new observable. And the idea is that you can then write a function elsewhere that you can then compose in as though it were an operator, anywhere you want, along with your other dot-chained operators. And the realization I had a few months ago was, “Well, why don't we just make all operators like this?” And then we can use functional programming to compose them with like a reduce or whatever. And that's exactly what the lettable operators are. And that's why I started calling them lettable operators. And I kind of regret it now, because so many people are saying it and it confuses new people. Because what in the world does lettable even mean? CHARLES: Right. [Laughs] BEN: So, they are pipeable operators or functional operators. But the point is that you have a higher-order function that returns a function of a specific shape. And that function shape is, it's a function that receives an observable and returns an observable, and that's it. So, basically it's a function that transforms an observable into a new observable. That's all an operator. That's all an operator's ever been. It's just this is in a different flavor. CHARLES: Now, I'm curious. Why does it do an observable into an observable and not a stream item into an observable? Because when you're actually chaining these things together, like with a map or with a ‘flat map' or all these things, you're actually getting an individual item and then returning an observable. Well, I guess in this case of a map you're getting an item and returning an item. But like… BEN: Right, but that's not what the entire operation is. So, you've got an operation you're performing whenever you say, if you're to just even dot-chain it, you'd say ‘observable dot map'. And when you say ‘dot map', it returns a new observable. And then you say ‘dot filter' and it returns another new observable. CHARLES: Oh, gotcha, gotcha, gotcha. Okay, yeah, yeah, yeah. Yeah, yeah. BEN: So, this function just embodies that step. CHARLES: I see, I see. And isn't there some special… I feel like there's some proposal for some special JavaScript syntax to make this type of chaining? BEN: Yeah, yeah, the pipeline operator. CHARLES: Okay. BEN: I don't know. I think that's still at stage one. I don't know that it's got a lot of headway. My sources and friends that are in the TC39 seem to think that it doesn't have a lot of headway. But I really think it's important. Because if you look at… the problem is we're using a language where the most common use case is you have to build it, get the size as small as possible because you need to send it over the wire to the browser. And understandably, browsers don't want to implement every possible method they could on say, Array, right? CHARLES: Mmhmm, right. BEN: There's a proposal in for ‘flat map'. They could add zip to Array. They could add all sorts of interesting things to Array just by itself. And that's why Lodash exists, right? CHARLES: Right. BEN: Is because not everything is on Array. And then so, the onus is then put on the community to come up with these solutions and the community has to build libraries that have these constraints in size. And what stinks about that is then you have say, an older version of Lodash where you'd be like, “Okay, well it has 36 different functions in it and I'm only using 3 of them. And I have to ship them all to the browser.” CHARLES: Mmhmm. BEN: And that's not what you want. So, then we have these other solutions around tree-shaking and this and that. And the real thing is what you want is you want to be able to compose things left to right and you want to be able to have these functions that you can use on a particular type in an ad hoc way. And there's been two proposals to try to address this. One was the ‘function bind' operator, CHARLES: Mmhmm. BEN: Which is colon colon. And what that did is it said, “You can use this function as a method, as though it were a method on an object. And we'll make sure that the ‘this' inside that function comes from the instance that's on the left-hand side of colon colon.” CHARLES: Right. BEN: That had a bunch of other problems. Like there's some real debate I guess on how they would tie that down to a specific type. So, that kind of fell dead in the water even though it had made some traction. And then the pipeline operator is different. And then what it says is, “Okay, whatever is on the…” And what it looks like is a pipe and a greater than right next to each other. And whatever's on the left-hand side of that operand gets passed as the first argument to the function on the right-hand side of that operand. CHARLES: Mmhmm. BEN: And so, what that means is for the pipeable operators, instead of having to use a pipe method on observable, you can just say, “instance of observable, pipeline operator and an operator, and then pipeline operator, and then the Rx operator, and then pipeline operator and the Rx operator, and so on.” And it would just be built-in. And the reason I think that JavaScript really needs it is that means that libraries like Lodash can be written in terms of simple functions and shipped piece-meal to the browser exactly as you need them. And people would just use the pipeline operator to use them, instead of having to wrap something in a big object so you can dot-chain things together or come up with your own functional pipe thing like RxJS had to. CHARLES: Right. Because it seems it happens again and again, right? Lodash, RxJS, jQuery. You just see this pattern of chaining, which is, you know… BEN: Yeah, yeah. People want chaining. People want left to right composition. CHARLES: Mmhmm. BEN: And it's problematic in a world where you want to shake off as much unused garbage as possible. And the only way to get dot chaining is by augmenting a prototype. There's all sorts of weird problems that can come with that. And so, the functional programming approach is one method. But then people look at it and they say, “Ooh, yuck. I've got to wrap things in a function named pipe. Wouldn't it be nicer if there was just some syntax to do this?” And yeah, it would be nicer. But I have less control over that. CHARLES: Right. But the other alternative is to have right to left function composition. BEN: Right, yeah. CHARLES: But there's not any special syntax for that, either. BEN: Not very readable. CHARLES: Yeah. BEN: So, you just wrap everything. And the innermost call is the first one and then you wrap it in another function and you wrap that in another function, and so on. Yeah, that's not [inaudible]. But I will say that the pipe function itself is pretty simple. It's basically a function that takes a rest of arguments that are all functions. CHARLES: Mmhmm. BEN: And so, you have this array of functions and you just reduce over it and call them. Well, you return a function. So, it's a higher function. You return a function that takes an argument then you reduce over the functions that came in as arguments and you call each one of them with whatever result was from the previous. CHARLES: Right. Like Tracy mentioned in the pre-show, I'm an aspiring student of functional programming. So, would this be kind of like a monoid here where you're mashing all these functions together? Is your empty value? I'm just going to throw it out there. I don't know if it's true or not, but that's my conjecture. BEN: Yes. Technically, it's a monoid because it wouldn't work unless it was a monoid. Because monoids, I believe the category theory I think for monoid is that monoids can be concatenated because they definitely have an end. CHARLES: Right. BEN: So, you would not be able to reduce over all those functions and build something with that, like that, unless it was a monoid. So yeah, the fact that there's reduction involved is a cue that it's a monoid. CHARLES: Woohoo! Alright. [Laughter] CHARLES: Have you found yourself wanting to apply some of these more “rigorous” formalisms that you find out there in the development of RxJS or is that just really a secondary concern? BEN: It's a secondary concern. It's not something that I like. It's something I think about from time to time, when really, debating any kind of heavy issue, sometimes it's helpful. But when it comes to teaching anybody anything, honestly the Haskell-isms and category theory names, all they do is just confuse people. And if you tell somebody something is a functor, they're like, “What?” And if you just say it's mappable, they're like, “Oh, okay. I can map that.” CHARLES: [Laughs] Right, right. BEN: And then the purists would be like, “But they're not the same thing.” And I would be like, “But the world doesn't care. I'm sorry.” CHARLES: Yeah, yeah. I'm kind of experiencing this debate myself. I'm not quite sure which side I fall on, because on the one hand it is arbitrary. Functor is a weird name. But I wish the concept of mappable existed. It does, but I feel like it would be handy if people… because there's literally five things that are super handy, right? Like mappable, if we could have a name for monoid. But it's like, really, you just need to think in terms of these five constructs for 99% of the stuff that you do. And so, I always wonder, where does that line lie? And how… mappable, is that really more accessible than functor? Or is that only because I was exposed to the concept of mapping for 10 years before I ever heard the F word. BEN: Yes, and yes. I mean, that's… CHARLES: [Laughs] BEN: Things that are more accessible are usually more accessible because of some pre-given knowledge, right? What works in JavaScript probably isn't going to work in Haskell or Scala or something, right? CHARLES: Mmhmm. BEN: If someone's a Java developer, certain idioms might not make sense to them that come from the JavaScript world. CHARLES: Right. But if I was learning like a student, I would think mappable, I'd be thinking like, I would literally be thinking like Google Maps or something like that. I don't know. BEN: Right, right. I mean, look at C#. C#, a mapping function is always going to be called select, right, because that's C#. That's their idiom for the same thing. CHARLES: Select? BEN: Yeah. CHARLES: Really? BEN: Yeah, select. So, they'll… CHARLES: Which in Ruby is like find. BEN: Yeah. there's select and then, what's the other one, ‘select many' or something like that. [Chuckles] BEN: So, that's C#. CHARLES: Oh, like it's select from SQL. Okay. BEN: Yeah, I think that's kind of where it came from because people had link and then they had link to SQL and then they're like, well I want to do this with regular code, with just using some more… less nuanced expressions. So, I want to be able to do method calls and chain those together. And so, you end up with select functions. And I think that that exists even in Rx.NET, although I haven't used Rx.NET. CHARLES: Hmm, okay. ELRICK: So, I know you do a lot of training with Rx. What are some of the concepts that people struggle with initially? TRACY: I think when we're teaching RX Workshop, a lot of the people sort of… I'll even see senior level people struggle with explaining it, is the difference between observables and observers and then wrapping their head around the idea that, “Hey, observables are just functions in JavaScript.” So, they're always thinking observables are going to do something for you. Actually, it's not just in Angular but also in React, but whenever someone's having issues with their Rx applications, it's usually something that they're like nesting observables or they're not subscribing to something or they've sort of hot-messed themselves into a tangle. And I'm sure you've debugged a bunch of this stuff before. The first thing I always ask people is, “Have you subscribed?” Or maybe they're using an Angular… they're using pipes async but they're also calling ‘dot subscribe' on their observable. BEN: Yeah. So, like in Angular they'll do both. Yeah. There's that. I think that, yeah, that relates to the problem of people not understanding that observables are really just functions. I keep saying that over and over again and people really don't seem to take it to heart for whatever reason. [Chuckles] BEN: But you get an observable and when you're chaining all those operators together, you're making another observable or whatever, observables don't do anything until you subscribe to them. They do nothing. CHARLES: Shouldn't they be called like subscribable? BEN: Yes. [Chuckles] BEN: They probably should. But we do hand them an observer. So, you are observing something. But the point being is that they don't do anything at all until you subscribe to them. And in that regard, they're like functions, where functions don't do anything unless you call them. So, what ends up happening with an observable is you subscribe to it. You give it an observer, three callbacks which are then coerced into an observer. And it takes that observer and it hands it to the body of this observable definition and literally has an observer inside of there. And then you basically execute that function synchronously and do things, whatever those things are, to set up some sort of observation. Maybe you spin up a WebSocket and tie into some events on it and call next on the observer to get values out of your observable. The point being that if you subscribe to an observable twice, it's the same thing as calling a function twice. And for some reason, people have a hard time with that. They think, if I subscribe to the observable twice, I've only called the function once. CHARLES: I experienced this confusion. And I remember the first time that that… like, I was playing with observables and the first time I actually discovered that, that it was actually calling my… now what do you call the function that you pass to the constructor that actually does, that calls next or that gets passed the observer? TRACY: [Inaudible] BEN: I like to call it an initialization function or something. But the official name from the TC39 proposal is subscriber function. CHARLES: Subscriber function. So, like… BEN: Yeah. CHARLES: I definitely remember it was one of those [makes explosion sound] mind-blowing moments when I realized when I call my subscribe method, the entire observable got run from the very beginning. But my intuition was that this is an object. It's got some shared state, like it's this quasar that I'm now observing and I'm seeing the flashes of light coming off of it. But it's still the same object. You think of it as having yeah, not as a function. Okay. No one ever described it to me as just a function. But I think I can see it now. ELRICK: Yeah, me neither. CHARLES: But yeah, you think of it in the same way that most people think of objects, as like, “I have this object. I have a reference to it.” Let observable equal new observable. It's a single thing. It's a single identity. And so, that's the thing that I'm observing. It's not that I'm invoking this observable to observe things. And I think that's, yeah, that's a subtle nuance there. I wish I had taken y'all's course, I guess is what I'm saying. ELRICK: Yeah. BEN: Yeah. Well, I've done a few talks on it. CHARLES: [Laughs] BEN: I always try to tell people, “It's just a function. It's just a function.” I think what happens to a lot of people too is there's the fact that it's an object. But I think what it is, is people's familiarity with promises does this. Because promises are always multicast. They are always “hot”. And the reason for this is because they're eager. So, by the time you have a promise, whatever is producing value to the promise has already started. And that means that they're inherently a multicast. CHARLES: Right. BEN: So, people are used to that behavior of, I can ‘then' off of this promise and it always means one thing. And it's like, yeah, because the one thing has nothing to do with the promise. It wasn't [Chuckles] CHARLES: Right. BEN: This promise is just an interface for you to view something that happened in the past, where an observable is more low-level than that and more simple than that. It just states, “I'm a function that you call. I'm going to be able to do anything a function can do. And by the way, you're giving me an observer and I'm going to do some stuff with that too and notify you via this observer that you handed me.” Because of that you could take an observable and close over something that had already started. Say you had a WebSocket that was already running. You could create a new observable and just like any function, close over that, externally create a WebSocket. And then everyone that subscribes to that observable is tying an observer to that same WebSocket. Then you're multicast. Then you're “hot”. ELRICK: [Inaudible] CHARLES: Right. So, I was going to say that's the distinction that Jay was talking about. He was talking about we're going to just talk about… he said at the very beginning, “We're just going to talk about hot observable.” ELRICK: Yup. CHARLES: But even a hot observable is still theoretically evaluating every single time you subscribe. You're getting a new observable. You're evaluating that observable afresh each time. It just so happens that in the lexical scope of that observable subscriber function, there is this WebSocket? BEN: Yeah. So, it's the same thing. Imagine you wrote a function that when you called it created a new WebSocket and then… say, you wrote a new function that you gave an observer object to, right? An observer object has next, error, and complete. And in that function, when you called it, it created a new WebSocket and then it tied the ‘on message' and ‘on close' and whatever to your observer's next method and your observer's error message and so on. When you call that function, you would expect a new WebSocket to be created every single time. Now, let's just say alternately you create a WebSocket and then you write a new function that that function closes over that WebSocket. So, you reference the WebSocket that you externally created inside of your function. When you call that function, it's not going to create a new WebSocket every time. It's just closing over it, right? So, even though they both are basically doing the same thing, now the latter one of those two things is basically a hot observable and the former is a cold observable. Because one is multicast which is, “I'm sharing this one WebSocket with everybody,” and the other one is unicast which is, “I am going to create a new WebSocket for each person that calls me.” And that's the [inaudible] people have a hard time with. CHARLES: Right. But really, it's just a matter of scope. BEN: Yeah. The thing people have a hard time with, with observables, is not realizing that they're actually just functions. CHARLES: Yeah. I just think that maybe… see, when I hear things like multicast and unicast, that makes me think of shared state, whereas when you say it's just a matter of scope, well then I'm thinking more in terms of it being just a function. It just happens that this WebSocket was already [scoped]. BEN: Well, shared state is a matter of scope, right? CHARLES: Yes, it is. It is. Oh, sorry. Shared state associated with some object identity, right? BEN: Right. CHARLES: But again, again, it's just preconceptions, really. It's just me thinking that I've had to manage lists of listeners and have multicast observers and single-cast observers and having to manage those lists and call notify on all of them. And that's really not what's happening at all. BEN: Yeah. Well, I guess the real point is observables can have shared state or they could not have shared state. I think the most common version and the most composable version of them, they do not have any shared state. It's just one of those things where just like a function can have shared state or it could be pure, right? There's nothing wrong with either one of those two uses of a function. And there's nothing wrong with either one of those two uses of Observable. So, honest to god, that is the biggest stumbling block I think that I see people have. That and if I had to characterize it I would say fear and loathing over the number of operators. People are like… CHARLES: [Chuckles] BEN: And they really think because everyone's used to dealing with these frameworks where there's an idiomatic way to do everything, they think there's going to be an RxJS idiomatic way to do things. And that's just patently false. That's like saying there's an idiomatic way to use functions. There's not. Use it however it works. The end. It's not… CHARLES: Mmhmm, mmhmm. BEN: You don't have to use every operator in a specific way. You can use it however works for you and it's fine. ELRICK: I see that you guys are doing some fantastic work with your documentation. Was that part of RxJS 2.0 docs? TRACY: I was trying to inspire people to take on the docs initiative because I think when I was starting to learn RxJS I would get really frustrated with the docs. BEN: Yeah. TRACY: I think the docs are greatly documented but at the same time if you're not a senior developer who understands Rx already, then it's not really helpful. Because it provides more of a reference point that the guys can go back and look at, or girls. So anyways, after many attempts of trying to get somebody to lead the project I just decided to lead the project myself. [Laughter] TRACY: And try to get… the community is interesting because I think because the docs can be sometimes confusing… Brian Troncone created LearnRxJS.io. There's these other visualization projects like RxMarbles, RxViz, et cetera. And we just needed to stick everybody together. So, it's been a project that I think has been going on for the past two months or so. We have… it's just an Angular app so it's probably one of the most easiest projects to contribute to. I remember the first time I tried to contribute to the Ember docs. It literally took me an hour to sit there with a learning team, Ember Learning Team member and… actually, maybe it was two hours, just to figure out how the heck… like all the things I had to download to get my environment set up so that I could actually even contribute to the darn documentation. But with the Rx, the current RxJS docs right now is just an Angular app. You can pull it down. It's really easy. We even have people who are just working on accessibility, which is super cool, right? So, it's a very friendly place for beginners. BEN: I'm super pleased with all the people that have been working on that. Brian and everybody, especially on the accessibility front. Jen Luker [inaudible] came in and voluntarily… she's like the stopgap for all accessibility to make sure everything is accessible before we release. So, that's pretty exciting. TRACY: Yeah. ELRICK: Mmhmm. TRACY: So funny because when me and Jen started talking, she was talking about something and then I was like, “Oh my god, I'm so excited about the docs.” She's like, “I'm so excited, too! But I don't really know why I'm excited. But you're excited, so I'm excited. Why are you excited?” [Laughter] TRACY: I was like, “I don't know. But I'm excited, too!” [Chuckles] TRACY: And then all of a sudden we have accessibility. [Laughs] ELRICK: Mmhmm. Yeah, I saw some amazing screenshots. Has the new docs, have they been pushed up to the URL yet? TRACY: Nah, they are about to. We were… we want to do one more accessibility run-through before we publish it. And then we're going to document. We want to document the top 15 most viewed operators. But we should probably see that in the next two weeks or so, that the new docs will be… I mean, it'll say “Beta, beta, beta” all over everything. But actually also, some of our friends, [Dmitri] from [Valas] Software, he is working on the translation portion to make it really easy for people to translate the docs. CHARLES: Ah. TRACY: So, a lot of that came from the inspiration from the Vue.js docs. we're taking the versioning examples that Ember has done with their docs as inspiration to make sure that our versioning is really great. So, it's great that we can lend upon all the other amazing ideas in the industry. ELRICK: Oh, yeah. CHARLES: Yeah, it's fantastic. I can't wait to see them. ELRICK: Yeah, me neither. The screenshots look amazing. I was like, “Wow. These are some fabulous documentation that's going to be coming out.” I can't wait. TRACY: Yeah. Thank you. CHARLES: Setting the bar. ELRICK: Really high. [Laughter] CHARLES: Actually, I'm curious. Because observables are so low-level, is there some use of them that… what's the use of them that you found most surprising? Or, “Whoa, this was a crazy hack.” BEN: The weirdest use of observables, there's been quite a few odd ones. One of the ones that I did one time that is maybe in RxJS's wheelhouse, it was just that RxJS already existed. So, I didn't want to pull in another transducer library, was using RxJS as a transducer. Basically… in Netflix we had a situation where we had these huge, huge arrays of very large objects. And if you try to take something like that and then map it and then filter it and then map it and then filter it, we're using Array map and filter, what ends up happening is you create all sorts of intermediary arrays in-memory. And then garbage collection has to come through and clean that up. And that locks your thread. And over time, we were experiencing slowness with this app. And it would just build up until eventually it ground to a halt. And I used RxJS because it was an available tool there to wrap these arrays in an observable and then perform operations on them step-by-step, the same map, filter, and so on. But when you do that, it doesn't create intermediary arrays because it passes each value along step to step instead of producing an entire array and then doing another step and producing an entire array, and so on. So… CHARLES: So, will you just… BEN: It saved garbage collection and it increased the performance of the app. But that's just in an extreme case. I would never do that with just regular arrays. If anything, it was because it was huge, huge arrays of very large objects. CHARLES: So, you would create an observable our of the array and then just feed each element into the observable one at a time? BEN: Well, no. If you say ‘observable from' and you give it an array, that's basically what it does. CHARLES: Okay. BEN: It loops over the array and nexts those values out of the array synchronously. CHARLES: I see, I see. BEN: So, it's like having a for loop and then inside of that for loop saying, “Apply the map. Apply the filter,” whatever, to each value as they're going through. But when you look at it, if you had array map, filter, reduce, it's literally just taking the first step and saying ‘observable from' and wrapping that array and then the rest of it's still the same. CHARLES: Right. Yeah. No, that's really cool. BEN: That was a weirder use of it. I've heard tell of other things where people used observables to do audio synchronization, which is pretty interesting. Because you have to be very precise with audio synchronization. So, hooking into some of the Web Audio APIs and that sort of thing. That's pretty interesting. The WebSocket multiplexing is something I did at Netflix that's a little bit avant-garde for observable use because you essentially have an observable that is your WebSocket. And then you create another observable that closes over that observable and sends messages over the WebSocket for what you're subscribed to and not subscribed to. And it enables you to very easily retry connections and these sorts of things. I did a whole talk on that. That one's pretty weird. CHARLES: Yeah. Man, I [inaudible] to see that. BEN: But in the general use case, you click a button, you make an AJAX request, and then you get that back and maybe you make another AJAX request. Or like drag and drop and these sorts of things where you're coordinating multiple events together, is the general use case. The non-weird use case for RxJS. Tracy does weird stuff with RxJS though. [Laughter] CHARLES: Yeah, what's some weird uses of RxJS? TRACY: I think my favorite thing to do right now is to figure out how many different IoT-related things I can make work with RxJS. So, how many random things can I connect to an application using that? BEN: Tracy's projects are the best. They're so good. [Laughter] TRACY: Well, Ben and I created an application where you can take pictures of things using the Google Image API and it'll spit back a set of puns for you. So, you take a picture of a banana, it'll give you banana puns. Or you can talk to it using the speech recognition API. My latest thing is I really want to figure out how to… I haven't figured out if Bluetooth Low Energy is actually enabled on Google Home Minis. But I want to get my Google Home Mini to say ‘booty'. [Inaudible] [Laughter] CHARLES: RxJS to the rescue. [Laughter] BEN: Oh, there was, you remember Ng-Cruise. We did Ng-Cruise and on there, Alex Castillo brought… TRACY: Oh, that was so cool. BEN: All sorts of interesting… you could read your brain waves. Or there was another one that was, what is it, the Microsoft, that band put around your wrist that would sense what direction your arm was in and whether or not your hand was flexed. And people… TRACY: Yeah, so you could flip through things. BEN: Yeah. And people were using reactive programming with that to do things like grab a ball on the screen. Or you could concentrate on an image and see if it went blurry or not. ELRICK: Well, for like, Minority Report. BEN: Oh, yeah, yeah. Literally, watching a machine read your mind with observables. That was pretty cool. That's got to be the weirdest. TRACY: Yeah, or we had somebody play the piano while they were wearing one of the brainwave… it's called the OpenBCI project is what it is. And what you can do is you can actually get the instructions to 3D print out your own headset and then buy the technology that allows you to read brain waves. And so with that, it's like… I mean, it was really awesome to watch her play the piano and just see how her brain waves were going super crazy. But there's also these really cool… I don't know if you guys have heard of Jewelbots, but they're these programmable friendship bracelets that are just little Arduino devices that light up. I have two of them. I haven't even opened them. CHARLES: [Laughs] TRACY: I've been waiting to play with them with you. I don't know what we're going to do, but I just want to send you lights. Flashing lights. [Laughter] TRACY: Morse code ask you questions about RxJS while you're working. [Laughter] CHARLES: Yeah. Critical bug. Toot-toot-toot-too-too-too-too-toot-toot. [Laughter] CHARLES: RxJS Justice League. TRACY: That would actually be really fun. [Laughter] TRACY: That would be really fun. I actually really want to do that. But… CHARLES: I'm sure the next time we talk, you will have. TRACY: [Laughs] Yes. Yes, yes, yes, I know. I know. we'll do it soon. We just need to find some time while we're not going crazy with conferences and stuff like that. CHARLES: So, before we head out, is there any upcoming events, talks, releases, anything that we ought to be, we or the listeners, ought to be aware of? TRACY: Yeah, so one of the things is that Ben and I this weekend actually just recorded the latest version of RX Workshop. So, if you want to learn all about the latest, latest, newest new, you can go ahead and take that course. We go through a lot of different things like multiplex WebSockets, building an application. Everywhere from the fundamentals to the more real world implementations of RxJS. BEN: Yeah. Even in the fundamentals area, we've had friends of ours that are definitely seasoned Rx veterans come to the workshop. And most of them ask the most questions while talking about the fundamentals. Because I tend to dig into, either deep into the internals or into the why's and how's thing. Why and how things work. Even when it comes to how to subscribe to an observable. Deep detailed information about what happens if you don't provide an error handler and certain cases and how that's going to change in upcoming versions, and why that's changing in upcoming versions, and what the TC39's thoughts are on that, and so on and so forth. So, I try to get into some deeper stuff and we have a lot of fun. And we tend to be a little goofier at the workshops from time to time than we were in this podcast. Tracy and I get silly when we're together. TRACY: It's very true. [Laughter] TRACY: But I think also, soon I think there are people that are going to be championing an Observable proposal on what [inaudible]. So, aside from the TC39 Observable proposal that's currently still at stage one, I don't know Ben if you want to talk a little bit about that. BEN: Oh, yeah. So, I've been involved in conversations with folks from Netflix and Google as well, Chrome team and TC39 members, about getting the WHATWG, the ‘what wig', they're a standards body similar to W3C, to include observables as part of the DOM. The post has not been made yet. But the post is going to be made soon as long as everybody's okay with it. And what it boils down to is the idea of using observables as part of event targets. An event target is the API we're all familiar with for ‘add event listener', ‘remove event listener'. So, pretty much anywhere you'd see those methods, there might also someday be an on method that would return an observable of events. So, it's really, really interesting thing because it would bring at least the primitives of reactive programming to the browser. And at the very least it would provide maybe a nicer API for people to subscribe to events coming from different DOM elements. Because ‘add event listener' and ‘remove event listener' are a little unergonomic at times, right? CHARLES: Yeah. They're the worst. BEN: Yeah. CHARLES: That's a very polite way of putting it. BEN: [Chuckles] So, that's one thing that's coming down the pipe. Other things, RxJS 6 is in the works. We recently tied off 5.5 in a stable branch. And master is now our alpha that we're working on. So, there's going to be a lot of refactoring and changes there, trying to make the library smaller and smaller. And trying to eliminate some of the footprints that maybe people had in previous versions. So, moving things around so people aren't importing stuff that were meant to be implementation details, reducing the size of the library, trying to eliminate some bloat, that sort of thing. I'm pretty excited about that. But that's going to be in alpha ongoing for a while. And then hopefully we'll be able to move into beta mid first quarter next year. And then when that'll be out of beta, who knows? It all depends on how well people like the beta and the alpha, right? CHARLES: Alright. Well, so if folks do want to follow up with y'all either in regards to the course or to upcoming releases or any of the other great stuff that's coming along, how would they get in touch with y'all? TRACY: You can find me on Twitter @ladyleet. But Ben is @BenLesh. RX Workshop is RXWorkshop.com. I think in January we're going to be doing state of JavaScript under This Dot Media again. So, that's where all the core contributors of different frameworks and libraries come together. So, we'll definitely be giving a state of RxJS at that time. And next year also Contributor Days will be happening. So, if you go to ContributorDays.com you can see the previous RxJS Contributor Days and figure out how to get involved. So, we're always open and happy and willing to teach everybody. And again, if you want to get involved it doesn't matter whether you have little experience or lots of experience. We are always willing to show you how you can play. BEN: Yeah. You can always find us on Twitter. And don't forget that if you don't find Tracy or I on Twitter, you can always message Jay Phelps on Twitter. That's important. @_JayPhelps. Really. TRACY: Yeah. [Laughter] BEN: You'll find us. CHARLES: [Chuckles] Look for Jay in the show notes. [Laughter] CHARLES: Alright. Well, thank you so much for all the stuff that y'all do, code and otherwise. And thank you so much Ben, thank you so much Tracy, for coming on the show. BEN: Thank you. CHARLES: Bye Elrick and bye everybody. If you want to reach out to us, you can always get in touch with us at @TheFrontside or send us an email at contact@frontside.io. Alright everybody, we'll see you next week.

How do we ensure a high level of quality and maximize the refactorability of our code? Frontsiders, Wil and Charles, talk about their battle tested techniques for testing web applications, not only in React JS, but in any JavaScript framework. Links Acceptance Testing Integration Testing jest Cypress.io Assertion Ember CLI Mirage mirage-server react-trigger-change Transcript CHARLES: Hello everybody and welcome to The Frontside Podcast, Episode 90. My name is Charles Lowell, a developer here at The Frontside and your podcast host-in-training. And with me today is Mr. Wil Wilsman. Wil, who just got back from Nodevember just walked straight into the office and is ready to podcast with us on a very, very, very interesting subject, I think, today. We're going to be talking about acceptance testing in JavaScript applications, especially some of the techniques that we've developed here around testing React applications based on the lessons that we learned from the Ember community. But really, more than just React applications. Really, testing any JavaScript application from the inside out, making acceptance tests for that. So, I think we're going to talk about some of the challenges that you encounter and some of the really novel solutions that are out there that we had nothing to do with. And I guess we really didn't have, just more of cobbling together of various techniques for a powerful witch's brew for acceptance testing. Anyway, so Wil, just to round out the problem space or explore the problem space, what are some of the challenges that you encounter with an acceptance test? Actually, let me back it up even further. What is an acceptance test in a JavaScript application compared to what people normally encounter? WIL: Acceptance testing or end-to-end testing is just a problem that every JavaScript app should face. Not everyone does, but they definitely should. And basically, it's how the user interacts with your app through the browser. And every part of that we want to test, from the browser triggering browser events, interacting with the app, not calling functions or clicking buttons, and we're pretending we're a user. CHARLES: Yeah. You know, I know that when we showed up in the React space, that was not really the way that most people tested their applications. WIL: No, not at all. they're all about unit testing. Make sure every small piece of your code works, and to some degree integration testing, making sure your components work with other components. But, nothing is out there really for those big acceptance tests that you want the user to click a button and expect them be brought to a page or these fields to be filled out, et cetera. CHARLES: Mmhmm. Yeah. And there certainly was a very high level of maturity around unit testing, like you said. There are tools like Enzyme and… WIL: Jest. CHARLES: Yeah, Jest. But I was actually shocked to find out that Jest didn't even run in the browser. WIL: Yeah, it's all virtual. CHARLES: It's all virtual. It's completely and totally simulated and stubbed. And that presents some problems. WIL: Yeah. The main problem is cross-browser testing. Some people might consider that to be separate from their acceptance testing but you should be able to just run your acceptance tests in multiple browsers and be able to also test cross-browser support. CHARLES: Mmhmm. Yeah. And so, if you're using something like Jest, you're never actually running the code inside Safari. You're never actually running it inside Internet Explorer. You're actually running it in NodeJS. WIL: And you know, your user is not going to run it in Node. [Laughter] WIL: They're going to use a browser. CHARLES: I don't know about your users. WIL: [Chuckles] CHARLES: [Laughs] You know, we like to stick to the pretty advanced. It's like, go to getNodeJSBrowser.com. WIL: [Laughs] CHARLES: Enough of this Firefox BS. But not seriously, it was certainly a problem. We were looking around, because we never like to build anything ourselves if we can avoid it. But it really just seemed like there was not an off-the-shelf solution for writing these big style acceptance tests in JavaScript. There are some services out there. There's a couple now. What was the… WIL: I think the main one here is Cypress. CHARLES: Cypress, yeah. So, there's Cypress now. I've watched the instructional videos but never actually tried to integrate it into my application. WIL: Yeah. I think at its core it takes the same approach that we've been doing with how we're interacting with our tests. CHARLES: Mmhmm. Okay. The main difference is, is it that it's a service? Like you have to edit your tests through… WIL: Yeah. CHARLES: Their web browser, their web interface, and use their assertion library? WIL: Yeah. I'm not sure about the editing part. But yeah, it's their assertion library. I'm pretty sure it's their test runner and it's their testing environment. Really, the only control through that is through their UI, or through settings, basically. And you're stuck with those. You can't use other… I don't think you can use Mocha with Cypress. CHARLES: Right. WIL: Although it's very much like Mocha… CHARLES: Right. WIL: It's not. CHARLES: Right, right. And I also noticed, we'll touch on this later, the assertions, most of the side effects that were happening were happening right there inline inside your assertions. And that might be an opaque statement, but we will actually get into that later. WIL: Yeah. And I think one of the things about their side effects so to speak is everything leading up to a side effect is a promise with Cypress. CHARLES: Mmhmm. WIL: So, when you select a button and click it, Cypress is going to wait for that button to actually exist before it clicks it. CHARLES: Right, right, which is actually pretty cool. So, that's actually a perfect into into one of the primary challenges with doing acceptance testing in general in a JavaScript application. This is a problem when you're doing it in Ember. It's a problem in React. It's really a problem anywhere. And that is, how do you know when the effects of a user's interaction have been realized? Right? WIL: Yeah. And in Ember you take advantage of the run loop. Once that action happens, you wait for the run loop to complete and then your tests run. CHARLES: Right. So, the idea is that I've clicked some button or I've typed some key or I've moved the mouse. And then I listen for the run loop and when it's “settled” then I can now run my assertions because I know that the side effects that I was looking for have now been realized. WIL: Yeah, hopefully, if you're... CHARLES: Hopefully. WIL: Writing your app right. [Chuckles] CHARLES: Right, right. But that actually presents some problems in itself because it requires visibility into the internals of the framework. WIL: Yeah, so Ember is built with testing in mind. CHARLES: Mmhmm. WIL: And other libraries like React just being a view library might not be built with testing in mind. So, we don't have those hooks to wait for this loop to complete, wait for all of these things to be rendered before you continue. CHARLES: Exactly. And so, this is, I think it's actually kind of both a blessing and a curse. Because there are such strong conventions in Ember, they were able to build this wonderful acceptance testing regimen from the get go. WIL: Yeah. CHARLES: But like you said, that doesn't exist at all in the React ecosystem. And so, what do you do? There's no run loop. You're cobbling together a bunch of different components. And maybe you're using Redux, maybe you're using MobX, maybe you're using… you're certainly using React. And all of these things have their own asynchronies built-in. And there's not one unifying abstraction that's keeping track of all the asynchrony in the system. And so that presents a challenge. So, the question is then, if you're trying to not actually check and observe the state of a system until the right moment, how do you know when that right moment is? WIL: Yeah. And in early testing of a side React project I had, I would basically wait for a state to be complete before I continued my ‘before each'. And in the testing we're doing now, it's essentially what we're doing except the state is what the browser sees, or what the user would see in the browser. CHARLES: So you were actually querying the... WIL: Yeah. So, I was using Redux. So, in my app I was saying, when the Redux app is done loading... CHARLES: Mmhmm. WIL: The instance is set to true or false, then continue the test. CHARLES: Mmhmm. And so, what that means, what we're doing, is doing the same thing except observing at the DOM... WIL: Yeah, exactly. CHARLES: Level. And what it means is we actually… I would love to set this up and have a big reveal but I guess we'll just have a big reveal, is that essentially what we do is polling. WIL: Yeah. CHARLES: Right? So, when we run an assertion, let's say you click a button and you want the button to become disabled, there's an inherent asynchrony there. But what we will do is we'll actually run the assertion to see if it's disabled not one time. We'll run it a thousand times. WIL: Yeah, as many times as needed until it passes. CHARLES: Right, exactly. As many times as needed until it passes. And I think that is, at least to most programmer instincts, an odious idea. WIL: Yeah. CHARLES: [Laughs] WIL: It's like, “Oh wait, you're just looping over every single assertion how many times?” CHARLES: Yeah, exactly. And it feels, yeah, it feels weird as an idea. But when you actually see the code that it produces, it just sweeps away so much complexity. WIL: Yeah. And… CHARLES: Because you don't worry about asynchrony at all. WIL: Yeah. And it's pretty genius. If I'm a user and I click a button, it's loading when I see that it's loading. So, our tests are going to wait until that button says it's loading. And then the test passes. CHARLES: Right. And so, what we do is we essentially, we use Mocha but you could do it with QUnit or anything else, is that when you run your assertion, you declare, you have an ‘it' block or I guess, what would it be in QUnit? WIL: A test? CHARLES: A test? WIL: I think it's just a test. CHARLES: You have your test block. And so that function that actually runs the assertion and checks the state will actually run, yeah it could run three times. It could run a thousand times. It's just sitting there waiting. And it will time out. And it will only fail if that assertion has failed a thousand times or it has failed through, I think two seconds is our default. WIL: Yeah, yeah. I think we default to the runner's default timeout. CHARLES: To the runner's default timeout, yeah. WIL: Yeah. Or you can set that yourself with how we have it set up. And the other thing that comes from that is if your tests are only failing when they time out, how do you know what's actually failing? And our solution to that was we catch the error every time it fails and right before the timeout actually happens we throw the real error. CHARLES: Yeah. Exactly. But the net effect is that you're able to write your assertions completely and totally oblivious of asynchrony. You don't, we don't have to worry about asynchrony pretty much at all. I mean, we do, and we'll get into that. So, I made a global statement and then immediately contradicted it. WIL: [Chuckles] CHARLES: But hey, you got to be controversial. But for the most part, asynchrony just disappears because asynchrony is baked into the fabric. So, rather than thinking about it as a one-off concern or a onesie-twosie, it's just every single assertion is just assumed to be asynchronous. And so, that actually means you don't have to deal with promises. You don't have to deal with run loops. You don't have to deal with anything. You just write your assertion and when it passes, it passes. And there are some really unique benefits for this. And there are some challenges. So, I think one of the first benefits is that it's actually way faster. WIL: Right. CHARLES: Which is counterintuitive. WIL: It's incredibly fast. CHARLES: It's very fast. WIL: Yeah, for all the loops that's happening you might think every loop is going to slow it down slightly. But it really doesn't. Our tests, each test, even though it asserts five or six times, it takes milliseconds. CHARLES: Mmhmm. WIL: The test itself might only loop twice. CHARLES: Right. Exactly. Whereas if you're waiting for a run loop to settle, you might have some… you click a button, it disables, it also fires off an Ajax request and does all this stuff. But if all my assertion wants to know is “Is this button disabled?” then I only need to assert until that has happened. I don't need to wait until all the side effects have settled... WIL: Mmhmm. CHARLES: And then do the assertion. I just know, “Hey, my assertion, the thing that I was waiting for - that happened. Let's move on.” Yeah. And so, it's so, so fast. And that was actually, I didn't predict that. But I was definitely pleasantly surprised. WIL: Yeah, that was a very nice surprise. And all of our tests ran so much quicker than they would have in a run loop environment with Ember or something. CHARLES: Right. Yeah, yeah. That was, we actually had just come off a project where we were having that thing, that exact problem, which was that yeah, our animations were slow. Or, the animations were fine. They were perfect. [Laughs] But there were slowing down the tests. WIL: Yes. I think in that project, it was like, 30-minute tests... CHARLES: Mmhmm WIL: For the whole suite to run. CHARLES: Yeah. To which I'll add a public service announcement. I think this is a conjecture but I do believe that animations are best applied not to individual components but by the thing that uses a component. So, I shouldn't have an animation that's like, implicit to a dialog. It should be the thing that's showing the dialog that gets to decide the animation to use. Anyway, just throwing that out there. WIL: [Laughs] CHARLES: Because animations are about context. And so, the context should provide the animation, not the individual atom. Anyway, moving on. WIL: Some other podcast. CHARLES: Yeah. [Laughter] CHARLES: That's another podcast right there. But there's also, this does present some challenges or requires code to be structured in a way that facilitates this. So, there are some challenges with this approach, some things you need to be aware of if you're using this kind of system. We've kind of settled on a name for what we call these types of assertions and these types of systems. WIL: Yeah. We call them convergent assertions, because you're converging on something to happen. It's going over and over until it happens. CHARLES: Right. WIL: And yeah, a lot of these challenges that we've come across are things that you might not think of, like there are a few instances of false positives... CHARLES: Mmhmm. WIL: That happen with these convergent assertions. CHARLES: Right. So, what would be an example there? WIL: So, the most common example that I'm seeing so far is when you're asserting that something didn't happen. CHARLES: Mm, right. WIL: That would immediately pass. But if it takes your app... CHARLES: [Laughs] WIL: A few seconds for it to actually happen, then you could still have an actual failure but your test passed immediately. CHARLES: Right, right. So, what's the countermeasure then? WIL: We invert our assertions. So, we make sure they fail for a certain amount of time. CHARLES: Right. So, the normal case where you just want to say, “I want to make sure that my state converges to this particular state.” WIL: Alright. I said fail at first. I meant, pass. We have to make sure it passes for a certain period of time. CHARLES: Right, exactly. WIL: So yeah, the normal way is it fails until it passes, and then it passes. When you invert one of these convergent assertions, you're just making sure it passes repeatedly and if it fails at any point, you throw a failure. CHARLES: Right, okay. And so, that's like, if I want to check that the button is not disabled, I need to check again and again and again and again. WIL: Until you're comfortable with saying, “Alright. It's probably not going to be disabled.” CHARLES: Yeah, exactly. And so there, it's kind of weird because it is dependent on a timeout. WIL: Mmhmm. CHARLES: You could go for two seconds and then at the very end it becomes disabled. So, you just kind of have to take that on faith. But... WIL: Yeah. CHARLES: In practice, I don't think that's been much of a problem. WIL: No. CHARLES: It's more indicative of, if your button disables after... WIL: A few seconds. CHARLES: A few seconds, what's up with your... WIL: Yeah, what's up with your app? CHARLES: Yeah, exactly. WIL: If you're waiting for an Ajax request or something, an example, then you should be using something like Mirage Server. CHARLES: Right. Which is, man, we got to get into that, too. There are a couple of other things that I wanted to talk about too, with these convergent assertions. And that is, typically when you look through the READMEs for most testing frameworks, you see the simple case of the entire test, the setup, the teardown, and the actual assertions, are in the actual test. WIL: Yeah. The ‘it' block in Mocha or the test block in QUnit. You click a button, and then make sure it's disabled, and it moves onto the next test. CHARLES: Mmhmm. WIL: Then you click a different button or the same button and you assert something else in the next test. CHARLES: Mmhmm. Right. WIL: And yeah, you can't do that with convergent assertions because they're looping. So, if you click a button in a loop it's going to keep clicking that button over and over and over again. CHARLES: Yeah. [Laughs] Right, right. So, it means that you need to be very conscientious about separating the parts of your tests that actually do the things that actually act the part of the user from the part of your test that's about observation. WIL: Yeah. So, our solution to that is we move out all of our things that have side effects like clicking a button or filling in a form, all that stuff happens in ‘before each's. And all of our actual assertions happen in these convergent ‘it' blocks that loop over and over again. So, our ‘before each' runs and clicks the button and then we have 10 or so tests that will loop and wait for various states to... CHARLES: Yeah. WIL: Be true. CHARLES: Right. That means that yeah, all these assertions do is they read state. And you just have to, you do have to be conscientious. You're not allowed to have any side effects inside your tests, your actual assertion blocks. WIL: Mmhmm. CHARLES: But that's actually, it's a good use case for the whole Act-Arrange-Assert, which has been around way, way, way before these techniques. But here, we're doing Act and Arrange in our ‘before each'... WIL: Mmhmm. CHARLES: And then we're doing Assert later. And I think it actually leads for more readable... WIL: Yeah, definitely. CHARLES: Things. WIL: And it also opens the door to something that we can't really take advantage of yet but if you have 10 assertions with one ‘before each' side effect, you could run all of those assertions in parallel. CHARLES: That's right. WIL: And your tests would be 10 times more faster. CHARLES: Mmhmm. Yeah, exactly. Or you could run them in parallel or you could just run them one after the other but you wouldn't have to run that ‘before each' 10 times. WIL: Yeah. But something with that, that I found, is if we move all of our side effects to ‘before' blocks instead of ‘before each' blocks, sometimes a test three tests down that's waiting for something to happen... CHARLES: Yeah. WIL: That thing might have already happened earlier... CHARLES: Yeah. WIL: And it already went away. A loading state is the best example of that. CHARLES: Mmhmm. WIL: You show the loading state, the loading state goes away. So, if you move that button click into a ‘before' and that loading state test is three tests in, that loading state is already going to be gone. CHARLES: Yeah, so I think the long story short is we've kind of come to the conclusion that we would have to write our own runner. WIL: Yeah. CHARLES: Essentially to take advantage of this. But that said, we've done some sketching about what we would gain by writing our own runner. And the speed, we're talking about exponential speedups. WIL: Yeah. CHARLES: Maybe taking an entire acceptance test suite and having it run in five or six seconds. WIL: Yeah. We're talking about these tests that are already extremely fast. CHARLES: Mmhmm. WIL: Each test takes a few milliseconds or tens of milliseconds to complete. But then if you can run all of those at the same time, all of your tests for that entire ‘describe' block just ran tens of milliseconds. CHARLES: Right. Yeah. So, it's really exciting and pretty tantalizing. And we would love to invest the time in that. I've always wanted to write our own test runner. But never had, [chuckles] never really had a reason. WIL: Yeah. CHARLES: Certainly not just for the sheer joy of it. Although I'm sure there is joy in writing it. But that, yeah, we'll have to wait on that. But I am actually really excited about the idea of being able to maybe bring this back to the Ember community. WIL: Yeah. CHARLES: Because acceptance tests getting out of control in terms of the speed is I think a problem with Ember applications. And I think this would do a lot to address that. WIL: Yeah. CHARLES: I think, how long, if we were just using a stock Ember acceptance testing setup for this, I think we have about 250 tests in this React app... WIL: Yeah. CHARLES: How long does it take to run? WIL: Right now I think our tests take something like 20 seconds. And that's also somewhat due to they have to print the tests on the screen on Travis so that takes a little time. In an Ember setup, that could maybe take a few minutes. I mean, that's not that big of a deal, a few minutes. CHARLES: Right. WIL: But compared to 20 seconds. CHARLES: Right. you're still talking about an order of magnitude... WIL: Yeah, exactly. CHARLES: Difference. And using this, I think you could get a 30-minute test suite... WIL: Yeah. CHARLES: Down to the order of 3 minutes. WIL: Now when we're talking about those times, we're talking about the tests themselves. Of course, the CI would have to download stuff and set up the [inaudible]. CHARLES: Mmhmm, right, right. WIL: And that of course all adds to the time. CHARLES: Yeah, mmhmm, yeah. Earlier you mentioned, we talked about Ember CLI Mirage. This is actually something that is having now been using it for what, 2 years or something like that, it's just… it's impossible... WIL: To go back, yeah. CHARLES: To go back. It is. It's like [chuckles] you come outside the Ember community and you're like, “How is anybody ever dealing without this?” WIL: Yeah. CHARLES: [Laughs] WIL: A lot of the mocks are usually mocking the function that makes the request and it returns it in that function. That's what's out there currently, minus the Mirage stuff. CHARLES: Mmhmm. WIL: But once you use Mirage, you're mocking the requests themselves. CHARLES: Yeah. And you've got such great support for the whole factories. I love factories. It's something that is very prevalent in the Ruby community, and maybe not so much elsewhere. But the ability to very, very quickly crank out high-fidelity production data... WIL: Yeah. And you don't have to have files upon files of fixtures. CHARLES: Yeah, exactly. And you can change, if something about your schema changes, you can change the factory and now your test data is up and running. So, Ember has this tool called Mirage which is just, like I said, it's so fantastic. Oh yeah, it's also go support for running your application. Not just in your tests, but you can actually run your application with Mirage on and... WIL: Oh right, yeah. CHARLES: And you've got now the most incredible rapid prototyping tool. WIL: Yeah. You don't need to connect to a server to see fake data. CHARLES: Right, right. And we were even talking about this yesterday to a potential client. they're trying to, they've got to present something to investors. And how wonderful is it to just be like, “You know what? We just don't want to invest in all, we don't want to move the inertia, invest the money to generate the force to move the inertia of a backend.” Especially in this particular use case, the backend was going to be really, really heavy. WIL: Yeah. And there were some questions about the backend that we couldn't address quite yet but we wanted to start working on something that we could show, something demo-able. CHARLES: Right, exactly. And so, Mirage is just so wonderful for that. But again, Mirage is, it's an Ember-specific project. WIL: Right. CHARLES: So, the question was, “How are we going to use that?” WIL: And you actually took this on yourself. CHARLES: Mmhmm. WIL: I just saw this pop-up one day and boom, you converted Ember Mirage to vanilla JavaScript. [Chuckles] CHARLES: So, I did extract it. But the lion's share of the credit goes to the developers of Mirage themselves. Sam Selikoff and the Mirage community, they built Mirage not using much of Ember. There were some utilities that they were using, but mainly things like string helpers to convert between camel-case and dash-case, and using a Broccoli build, or using an Ember CLI build. WIL: Yeah. That was one of the challenges that we came across using Mirage outside of Ember, was how do we autoload this Mirage folder with all this Mirage config and Mirage factories and models, et cetera. CHARLES: Right. The internals were all just straight up JavaScript classes, for the most part. And so, extracting it, it was a lot of work. But 90% of the work was already done. It only took three or four days to do it. WIL: Amazing. CHARLES: Yeah. So, it was actually a really pleasant experience. I was able to swap out all of the Ember string helpers for Lodash. So now, it's good to go. It shares a Git history with Ember CLI Mirage, so it's basically a fork. WIL: Mmhmm. CHARLES: Like, a very heavily patched Ember CLI Mirage. But I keep it up-to-date so that it doesn't... WIL: Good. [Inaudible] CHARLES: Yeah, so I think the last time I merged in from master was about a month ago, something like that. Because it's got all the features that we need but it's not a big deal to rebase or just to merge it on over in. Because yeah, it's a really straightforward set of patches. WIL: Was there any talk with the creators of Ember Mirage about getting this upstream? CHARLES: So, I've talked a little bit with Sam about it. And from what I can tell, his feeling on it is like, “Hey, my goal right now is to focus on this being the best testing and data stubbing platform for Ember. Anything that happens out there, outside of that scope, that's great. And I certainly won't get in the way of it. But I'm pretty maxed out in terms of the open source credits that I have to spend.” WIL: [Chuckles] CHARLES: And there hasn't been much motion there. I'm happy where it is right now. I would like to see it merged into upstream. I think it would be great to have basically this Mirage Server and then have Ember CLI bindings for it. WIL: Yeah, yeah. I was going to say, either another Ember CLI specific package for Mirage or maybe to make it a non-breaking change or something. CHARLES: Yeah. WIL: Just like an Ember-specific entry point. CHARLES: Right, exactly. And I think that's definitely doable, if someone wants to take it on. I will say, we have been using this extracted plain vanilla JavaScript Mirage Server now for what, almost six months? WIL: Yeah. CHARLES: And it really hasn't... WIL: Yeah, I don't think I ran into one issue with that. CHARLES: Yeah. It's solid. It's really, really good. So, kudos to the Mirage team for doing that. And if anybody is interested in using Mirage in their projects, it's definitely there and we'll put it in the show notes. WIL: Yeah. We call it Mirage Server. CHARLES: Mirage Server, yeah. So, I don't know. Maybe it's time to reopen that conversation. But it has become a very integral and critical piece of the way that we test our JavaScript applications now. So, what are the foundations of it? We've got, we're using Mirage. We're using these convergent assertions. We're using Mocha, although that's really... WIL: Yeah, we have jQuery and Chai jQuery just to help us out with interacting with the browser as a user would. And I think one of the big challenges with that actually, I just remembered, was triggering changes in React. CHARLES: Yeah. WIL: I think this is pretty specific to React. You might run into problems with the view. I don't know how to mess with view. But in React, at least I think 15 or React 16, one of them, they changed the descriptor of the value property on an element so that they can appropriately interact with it, make changes, watch for changes, et cetera. So, when you set this value property using jQuery or just straight up ‘.value()', that change event isn't triggered in React. Your on-change handlers are never called. CHARLES: Wait, they actually update the JavaScript property descriptor of the DOM element. WIL: Yes. CHARLES: Boo. WIL: Yeah. So, there's a nice little helper out there called React Trigger Change. I dug through it and I've stripped some of it down to just be for more modern browsers, more modern React. But there's a lot of good code in there. And basically, it just caches that descriptor, updates the value, triggers the change, and then adds the descriptor back. And that ends up triggering that React element. CHARLES: Okay. [Laughs] WIL: [Chuckles] Yeah, so that calls your handlers and that's how we get around that. CHARLES: Right. There's a few fun little hacks there. But I think it is good to tie that into a larger point, is that the amount of touch that you have with the framework is actually very low. WIL: Yeah. CHARLES: So, the amount of affordances that we've had to make just for React, there's that, that you just mentioned. And is there… there's not much else. We had to write a test harness to mount the app. But that's like... WIL: Yeah, yeah. Our describe application helper is pretty React-specific. CHARLES: Right. WIL: So, you'd have to render it and set up a Mirage server, et cetera. CHARLES: Right. But that's application-specific setup. WIL: Yeah, that's one file. CHARLES: Right. WIL: So, your goal for acceptance tests is you want to be able to have a refactor and your acceptance tests still pass. CHARLES: Right. WIL: So, what if that refactor involves switching libraries? CHARLES: Right. WIL: If you're writing Ember acceptance tests, you're going to have to rewrite all your acceptance tests. CHARLES: Right. WIL: That's a huge downside. CHARLES: Right. WIL: So, with this method of interacting with the actual library very little, we have that one file that sets up our app and then we have that one trigger change helper, we remove those, we can use whatever framework we want underneath this. And our tests would still work. CHARLES: Yeah, exactly. And I think that we actually could theoretically, and honestly I have enough confidence in this style that we're developing the tests now, we could refactor this application to Ember and not have to rewrite our tests. WIL: Yeah, exactly. CHARLES: In fact, the tests would be an aide to do that. WIL: Yeah, and the tests would be faster than Ember testing with that run loop problem. CHARLES: Yeah, exactly. that's really something to think about or to think on, is like, “Wow. You're really at this point completely, not completely, but very loosely coupled to the actual internal library code.” Which is one of the goals of a nice, big acceptance test, is to be able to make major changes, break big bones, and be able to set them and have your acceptance test suite be the bulwark that holds it all together. WIL: Yeah. CHARLES: So, I actually don't know what a bulwark is. WIL: [Laughs] CHARLES: I just know that it's a really strong thing. [Laughter] CHARLES: Maybe we could put that in the show notes. [Laughter] WIL: A link to what that is. CHARLES: [Laughs] So, alright. Well, I'm trying to think if there is anything else that we wanted to mention. Any challenges? Any next things? WIL: So, one of our next steps is something we mentioned that Cypress does, is they wait for elements to exist before they interact with them. And we're actually not doing that in our app currently. And we don't have helpers out there for it yet. CHARLES: Right. WIL: But that's very much the next step. When we go to click an element in our ‘before each', we have these describes that are nested. Say, you have nested describes and you get down three levels into a ‘before each' when you're clicking a button. That button might not exist yet. CHARLES: Right. WIL: And especially since we're using jQuery, if you trigger a change on an empty jQuery element, it's not going to throw an error. It's just not going to tell you that it triggered anything. CHARLES: Right. WIL: So, we get those skips where that button's not getting clicked and we should really be waiting for that button to exist. CHARLES: Right. So, what we've done right now is we're converging on our assertions at the backend of a test. But at the frontend of a test we need to also be converging at some state before we can actually interact with the application. WIL: Right, yeah. CHARLES: So yeah, so that part is missing. And that actually brings up, we are very slowly but nevertheless doing, we're collecting these convergent assertions and convergent helpers in a repository on our GitHub account. We're going to be adding these things so that you can either use them out of the box or use them to make your own testing library. WIL: Yeah. And one of the other next steps that goes along with waiting for the element to exist is when you need to chain convergences. Like, wait for this element to exist and then click it and then wait for this thing to happen before actually running a test. And that presents the problem of our convergences are waiting for that timeout and those timeouts will accumulate. So if you have three chained convergences, that's now a six thousand millisecond timeout as opposed to a two thousand millisecond timeout. CHARLES: Right. WIL: So, one of the next steps is getting that tracking under control so if you chain three convergences together, they're smart about it and they still fail under the two thousand millisecond timeout. CHARLES: Right, right. So yeah, so we're going to be collecting all this stuff that we're learning into some publicly available code. We have a repository set up. I don't know if I want to announce it just yet, because it's really early days. WIL: Yeah. CHARLES: But that definitely is the plan. And that way, whether you're using Mocha or whether you're using QUnit or whether you're using Chai or jQuery, you've got these underlying primitives that help you converge on a state, whether that state is to interact with some piece of the DOM or to just assert some observation is made about that state. We'll be continuing on that. But by all means, get in touch if this is something that is of interest to you. Let's make something happen, because it's something that we're pretty excited about. And honestly, it's pretty comfortable living inside the four walls of this test suite. WIL: Yeah. CHARLES: It feels pretty good. WIL: It does, yeah. They're very fast. And some places in the test might need a little reworking, but for the most part all of our tests are very well-written, very well-readable. And you can just open up a test and know exactly what's going on. CHARLES: Yeah. Alright. Well, I think that about does it for Episode 90. Wow, Episode 90. WIL: Man, coming up on that 100. CHARLES: Yeah. We're going to have to have a birthday cake or something. WIL: Do we celebrate Episode 100 or Episode 104? CHARLES: What's 104? WIL: 104 would be 2 years. CHARLES: Oh really? WIL: Well, I mean 2 years' worth of podcasts. CHARLES: Oh, right. 2 years' worth of podcasts. Yeah. WIL: Yeah, like if you go every week. CHARLES: Maybe we should celebrate a hundred hours or something like that. WIL: Oh yeah. CHARLES: We can add up the thing or celebrate… I don't know, be like, “You've literally wasted 2 years of your life.” WIL: [Laughs] CHARLES: “2 weeks of your life listening to the podcast.” Anyway, so that's it for Episode 90. and thank you so much, Wil. WIL: Thanks for having me. CHARLES: It's always a pleasure to talk about these topics with you. And as always, if you need to get in touch with us, please reach out to us on Twitter. We are @TheFrontside. Or you can send an email to contact@frontside.io.

Kaylie Kwon: kaylieEB | kayliekwon@gmail.com Show Notes: 02:14 - Kaylie's Journey Into Software Development 09:25 - Implementing a Design System and Attacking Higher-level Workflows 15:43 - EDS Collaboration and Public Availability 19:07 - Getting Involved with The Yarn Project 20:57 - Selective Resolution 23:37 - The Warmth of the Yarn Community 27:11 - Handling Issue Communication and Tracking Resources: Eventbrite britecharts Eventbrite Spectrum Transcript: CHARLES: Hello everybody and welcome to The Frontside Podcast. My name is Charles Lowell, I'm a software developer here at the Frontside and your podcast host-in-training. With me today is Elrick Ryan. Hello Elrick. ELRICK: Hey, what's up Charles? CHARLES: Not much. Are you enjoying your morning so far? ELRICK: Yeah, my morning is going well. Everything is good. CHARLES: Lots of cups of coffee? ELRICK: No cups of coffee yet. I've been drinking a lot of green tea. CHARLES: I've actually heard that's really good for you. ELRICK: That's what I heard too. That's why I started drinking it. CHARLES: Did you continue because it tastes good or do you just live on the idea of how good it is for you? ELRICK: A little bit of both. It doesn't taste that great but it's not horrible. It's almost like an acquired taste and then when you add in, "This is good for me," then it tastes great. CHARLES: Okay. We got a nice [inaudible] there. ELRICK: Yes. CHARLES: Anyway, I guess we should, at some point get on to the main content of our podcast. We have a very interesting guest with us today who has her fingers in all kinds of pies that we were talking about just at the pre-show, just before we were recording so we're really, really happy to have Kaylie Kwon. Thank you for coming on the show and welcome. KAYLIE: Hi. Thank you for having me. CHARLES: It's going to be great. Now, you are a software engineer at Eventbrite, that's correct? KAYLIE: Yep. CHARLES: What kind of things do you do over there? KAYLIE: I used to work on part of the feature team that worked on their reserved seating product but not too long ago, I moved to our frontend platform team, which is a team that helps other frontend engineers move faster through things like working on infrastructure or Eventbrite design system and dev tools. CHARLES: So like getting into the tools that unlock the exponential productivity of the developer team? KAYLIE: Uhm-mm. CHARLES: We're going to dig into all of that because you just listed a bunch of really interesting stuff. I'm really excited to talk about the design systems, in particular but lots of different stuff. But before we do, I understand that you have a fairly unique way of entering in to the position that you're in now. Your journey didn't follow the traditional arc. Would you be willing to elaborate on that or tell us that story? KAYLIE: Yeah, totally. I graduated with a degree in art history, not related to computer science at all. Then right after, I moved to New York and worked for a small startup. It was marketing/business development role and I wasn't really happy with it. I was working on some design-y stuff on the side with HTML and CSS but I just felt like my brain just needed to be stimulated a little more. I applied to this all women's bootcamp program called Hackbright in San Francisco. It was a three-month intensive program and luckily coming out of that, I had at least some initial knowledge to get my foot in the door and Eventbrite was one of the hiring partners. They brought me in for interview. I actually had no idea that it was going to be a frontend engineering role because my bootcamp was totally in Python and it just had more with a backend focus. Ben, who spoke at React Boston Conference with me, was actually one of the interviewers and he gave me this Clojure problem and I solved it but in Python just using recursion. He was like, "You got it. Just convert it to JavaScript," and I was like, "No, it just can't be done. No JavaScript." But there's the reason he would chose to hire me and they onboarded me as an internal bootcamp within Eventbrite. The first three months I was there, I learned about React, Redux, JavaScript, ES6, all of that good stuff. Then they moved me to a feature team, where I continued, I guess working on the product and then I became involved more with open source projects. I really expressed interest in when you're a new engineer and coming onboard, there's all these assumed knowledge that isn't documented anywhere or something will be really obscure and hard to use but people will just assume that that's the status quo. This idea around developer experience and helping other developers move faster, it just kind of become a natural interest of mine. I was talking to the platform team, Ben in particular, expressed my interest in working in these areas. Just about like a month ago, we did a big re-org and I landed on the platform team. Currently, we have a lot of projects in flight. One of them being moving our dependency system from our old codebase, which first was written in RequireJS to webpack. We're building out our Eventbrite design system, which is basically a shared UI component library that other teams could leverage. Our platform team will just come in and make sure that the API is usable across different teams and maintain a consistent brand in terms of look and feel. We're also working on other tooling stuff like making sure we use Docker for our dev environment and making sure that the frontend containers don't break, making sure that everyone is on the same version of ESLint, Node and things like that. CHARLES: How do you make sure that the frontend containers don't break? KAYLIE: It's actually hard. I think one thing that we're trying to test right now is using Yarn Offline Mirror and having better caching. When you build a container, it'll look into the cache directory, which is just bunch of committed tarballs. That way, they don't have to fetch to the network each time because once the lockfile or package.JSON changes at our current state, it'll rebuild the entire container and it could take a very long time. We just have a lot of packages that we've added over time. Other things, we're experimenting along with our platform team on the backend side about having remote images. Instead of devs building their containers locally on their computer, having them remotely built by a CI system and then just pulling the container and the images down. CHARLES: Really, really you're like all over the place. That sounds so much fun. KAYLIE: Yeah, it's a lot of context switching. Sorry, if I was jumping too many topics at once. CHARLES: No, absolutely not. I think it's actually fascinating and it's actually capture the kind of scope because when you are doing development, if you yourself are not running up and down the stack, the tools that you use are. The better the tools, the better you're able to focus on the one little sliver of the stack that you're working on. If I don't have to worry about where are my Docker images are coming from or if there's a temporary network flip that I can't install my dependencies, if I don't have to worry about those things, then that makes me more effective so it's important to just kind of lay out all of the stuff that goes into making a quality developer experience. KAYLIE: Yeah, the dream is like frontend people wouldn't have to worry about any of the backend stuff and they just have this isolated environment, where they could just work on what they do best, which is JavaScript and writing features in React. Everything else just works and they could see that replicated through their dev environment as well as QA and Prod and hopefully, make every tooling that they use, like testing and linting all easily, intuitive and accessible. CHARLES: How is it that you're working up and down the stack, making sure that your CI systems, your Docker images but then also at the same time, working on a design system, which you've got collaboration, I assume with some pretty hard core devops teams but also then you're interfacing with designers. You kind of flesh out your design system. Are those the same people on the platform team? Or there are different groups within it? KAYLIE: We have designers and researchers actually, as part of our engineering unit. We work pretty closely with them to define guidelines on what the design system should look like because coming into making this designs system, one thing we really wanted to make sure that is that both sides of engineering and design have input, rather than the previous old version called Style Guide, which was more engineering-driven that not. One team would need a model so they would build a model and a different team would build a slightly different model and we would end up with five different models. They want to be maintained over time and there wasn't really a focus on accessibility or consistency of brand so the design system project was to eliminate all of those pain points. CHARLES: For people who may not be familiar with design system but it's something that certainly is cramping up more and more, what's the general strategy you take? Clearly, you talked about the kind of the pains that it solves, like I'm experiencing this pain where I've got five dialogues, I've got three ways of laying out forms, I've got these problems. What's the strategy that I go about as my organization for trying to implement this? Like now, I want to do a design system, where do I start? KAYLIE: One big thing, I think that helped us was developing Eventbrite design system as just the standalone product. You could run Eventbrite design system as a standalone with all of the documentation and components with each of their different props that could be configured and it has its own set of tests. All of the components are API-driven so there's nothing specific about business logic that it assumes. For example, our button component just assumes that I'll be receiving a type of submission and some [inaudible] body. It doesn't make any assumptions that it's going to be anything Eventbrite specific. It's high-level configurable that the end user wants it to be. Another thing that helped is we have a planned approach session right before we start working on a new component. A developer who would be building the component would meet with one of the frontend platform members and will be discussing the API of the component, the CSS that would go in it and the designer would do the final QA. The development takes longer than if you were to write just a component for your app but we're trying to build it out for a long term use. We have versioning for Eventbrite design system as a library so whenever you make updates, we added to the changelog and then it gets released and we bumped in Core, which is where all of our apps live and people have to get the new version. Basically, it's an orchestrated effort and we have a process built around scheduled releases and bumping it in Core. CHARLES: I get the wanting to have a uniform buttons, uniform labels, dialogs and things like that. How do you attack the problem of higher-level workflows like a form submission and validation process that might have a lot of different pathways? How do you guide that using a design system? KAYLIE: Those are actually really good questions because we went through issues like that. Validation form field or components that have logic around more complex inputs and validation is really hard because different teams use it in a slightly different way so we ended up having two versions of complex inputs. Now, we're in the process of deprecating the V1 and then having people move over to V2. We also use the Atomic Design Principles for Eventbrite design system. It means, things like atoms would be composed of buttons and we have molecules, which are slightly more complex. Then one level above that would be organisms, which are things like forms. Obviously, as you move up to like the higher level, it becomes more difficult for us to decide does this belong in EDS or does it belong with the app. There's lots of refactoring that sometimes ends up happening as a result of something built one way and us realizing later down the road that it wasn't actually very extensible and we just built it for this particular use case. Sometimes, it's the opposite where this was too generic. We can make it a little bit more specific and add more logic to it. It really depends on the component but that's been our strategy, just taking the components as needed. We're still in the earlier stages. We've released EDS a year ago and now, we're almost close to finishing it. We're just missing a handful of components. They'll be more [inaudible] that's available as opposed to adding a brand new component. CHARLES: Did you find guidance in existing design systems? One thing is why not just use one of the existing ones that's out there. Is it a matter of branding? Is it a matter of, "There are certain interactions which are unique to our product that we need to design system to capture them?" I guess my question is, if this is something I want to take on myself, what tools would I be able to reach for and what other design systems would I be able to look to versus what I have to contribute myself? What am I going to be looking for to share with the community and what am I going to look for that's develop that's uniquely mine? KAYLIE: I think consistency was a big issue. Sort of the genesis of idea came actually even before I joined the company but I think what they were going for was that we already had an existing component library called Style Guide and it wasn't really working out for us. To build like a common UI library was a natural assumption but making it better, making it more reusable. We looked at companies like Airbnb and Microsoft and I learned a lot from what they were doing. You definitely wanted some components that are specific to Eventbrite such as a ticket card or a media card that we use for our browse pages which would be needed for Eventbrite specific pages. I think mostly, the designers wanted more control because relying on third library means we don't always get what we want. We're actually thinking about making EDS open source at some point in the future, where it could take themes so other companies or individuals can leverage it but use their own theming scheme. If Spotify came and wanted to use EDS but using their colors and brand logo, then they could do so, just by layering a different configuration style to it. CHARLES: That's such an interesting idea. Is this something that you all have explored, maybe collaborating with another company? I'm trying to think what would be the benefit of making it publicly available, unless you would be getting lots of contributions back to improve EDS itself. Is that the idea? KAYLIE: Yeah. We're definitely set on making it public at some point in the future but making it open source is a different conversation because like you said, there's pros and cons that comes with open sourcing a package. We recently open source britecharts, which is a D3 library. It's been getting a lot of contributions. It's a good recruiting tool but it's also a lot of maintenance work outside of developers normal work hours. Also, we have to start worrying about like when we make a breaking change to a component, we're not just breaking it for our own product but for other developers. We currently have external developers. We have Eventbrite plugin system called Spectrum so developers are already building on top of that and they've been asking for something like this, where they could leverage and match the look and feel with the rest of the product. The downside is lots of maintenance hours and worrying about all the people that would be breaking the component for by just solving your use case. I feel like I didn't fully answer one of your questions, which was you have a different dynamic of people working on both really backend ops things, as well as a really frontend design system work. We definitely got very smart people on the team. Some people definitely have expertise in one area versus on others. Currently, we have five developers and some of us lean more towards the backend of the frontend work and I am one of those people working on things like Yarn and Docker and build system work but it's funny because I was thinking if I had a portfolio then, there wouldn't be any visual components to it, even though I'm a frontend developer. It would just be like terminal screens. We try to divide the work but everyone tries to, I guess develop, at least some shared knowledge around why we make the decisions that we do. CHARLES: It's interesting how they all intersect. I feel like the trend of the last 10 years has been to dev of all the things. I think the first thing was this artificial divide between developers and testing and that came together with the test driven development and test obsessed. Then there was this divide between developer and an ops and that divide went away now with the advent of the devops movement. Now, there's this divide between developers and design and I feel like that kind of wall is collapsing right now. You have developers participating a lot more in design and designers spending a lot more in development. We're seeing that but it's just funny how the devs starts integrating with all the things. KAYLIE: Yeah, that's a really good insight. CHARLES: You said you kind of naturally gravitate towards more of the backend doing the working with Docker, working with Yarn. How did you get actually involved with the Yarn Project? KAYLIE: Eventbrite converted over from NPM to Yarn maybe a year ago and the benefits that we got from converting over was awesome because we were manually editing NPM shrinkwrap, which is a nightmare and the installation speed of the container was really slow just because we were on NPM and it didn't really have any advance caching mechanisms at that point. Yarn just sped up a lot of things for us. I really like the user interface outside of just installing. You actually get a lot of freebie commands like 'yarn why,' that tells you why you have a particular dependency or you could do 'yarn check.' It was just a lot more helpful. I've been wanting to contribute to open source for a while so I did a little bit of work before then but the community was really encouraging when I first try to solve and pick up some first contribution bugs off of their backlog. At the time, they were pushing for 1.0 release so there's a lot of excitement about what Yarn will be and all the new features will be adding. I kept trying to pick places where I felt like I could be of help and then, Christoph, the manager of the Yarn Team and I believe [inaudible] team as well, reached out and asked if I wanted to build the selective resolution feature for them. I was like, "Yeah. I'll give it a go." Then I did it. CHARLES: What is selective resolution? KAYLIE: Good question -- CHARLES: Because I use Yarn every day and I've never heard this term. KAYLIE: It's something that became available with 1.0 release, much like workspaces and most of the time, you're not going to need it but sometimes, you're using a library. That library will have a nested dependency that for some reason, has a bug or you can't work with that package or maybe you just want to dedupe the package so that all of your dependencies end up using one version of that particular nested package. Selective resolution is like a way for you to override other libraries dependencies -- CHARLES: Oh, that's really cool. I like that because I've had that happens to me where I want some version of a library that has got a bug fix and yet, some other library that's depending is requiring this library and they're getting the old version and I'm like, "Nah!" ELRICK: That was happening to me last night. I wish I knew about this. CHARLES: Yes, seriously. KAYLIE: Yeah. Before this -- CHARLES: I'm glad we had you on justifying about this. KAYLIE: Awesome. Before this, you would have to file an issue with the original maintainer and maybe, it'll get fix. Maybe it won't or maybe you're stuck on the old version and you can't do anything about it. We had a really similar issue with the PhantomJS package, where we wanted to use a next patch version with a bug fix but then, something that was requiring it wasn't letting us use it. It works. I verified it. It seems like Facebook started using it as well so it was a pretty rewarding to work on that. CHARLES: That's exciting. I also plan on using it the next time I encounter this. ELRICK: To get this feature, is this a flag that you have to pass? How does it do the selective resolution? KAYLIE: As long as you're using Yarn 1.0 or above, you define resolutions field inside of your package.json, like how you would define that dependencies, you also have extra [inaudible] resolutions. On the left hand side, you put in a glob pattern that you want to match and if you want to match all packages, you just enter the name of the package. Then on the right side, you enter whatever version or path that you wanted to match. It could be a file or a GitHub link or it could be a version or a range or whatever you want. ELRICK: Nice. CHARLES: That's fantastic. Now, you mentioned that as you were coming to work on this, you were looking for features to work on but the community actually did a very good job of drawing you in and getting that contribution from you, which is actually pretty amazing when you think about it. I think a lot of open source projects, either flourished or failed by their ability to attract contributors. What was it that was particularly inviting? KAYLIE: It was a combination of things. I think one thing is they tried to point you to the actual code like if you want to submit a PR, then this is where you would get started. I actually started doing that myself on some of the issues. Everyone loves PRs more than issues so I think giving people filing the issues, some, I guess empowerment to try to fix it on their own, I think is great. They also have a Discord Channel for devs to talk about any questions that they might have, how to set up their initial dev environment, to test things on Yarn. Also, they were really nice. They were really thankful when I posted a PR or commented on the fix. They also use a lot of emojis, which helps. I think I personally found it really rewarding because it made me a better developer. Before contributing to Yarn, I didn't know that much about Node. For me, it was just fun to learn more about like, "This is how something else works," and also, the codebase uses a lot of different linting configurations, which I hadn't really used before so that was a nice learning curve there as well. ELRICK: For your initial time going to Yarn when you didn't know much about it, was the champion from the project that worked with you to get you over the hump or places that you were stuck or did you just have to kind of figured it out on your own or did you ask questions on the Discord Channel or in Slack Channel or anything? How did that process go initially? KAYLIE: I definitely pace myself. I just picked up easier bugs on a repos if possible and BYK and Mel who maintain the project would give guidance, especially through PR comments and they also answer any questions on issues. If I ask questions like what's the right approach here, because sometimes you get a bug and it's not just a bug. It actually has to do with philosophy of like, what should Yarn do in this case. There be like really minor edge cases like maybe NPM does that in a particular way, should Yarn respect that or should it try to be better? Those discussions are, I think really helpful and interesting and understanding what's going on. CHARLES: It's fantastic when the conversation just builds and you're learning stuff and then you actually feel like you came out with the best solution that was available to you at the end of the process. ELRICK: You gain a lot of context about the project during those conversations. CHARLES: Right. KAYLIE: Yeah, what was good about those selective resolution feature was it was completely community-driven, even from the RFC standpoint which submitted by someone in the community and then it was implemented by me, who doesn't work in Facebook. I think that's awesome. CHARLES: There's been a lot of thought that was put into the feature even before the implementation. That's always so critical to getting people's and giving them some specifications, some blueprint of what they're actually going to build. One of the things also that I wanted to touch on too is you mentioned before the show that they have a very unique take on the way they handle communication with the community, not just in terms of pull request but also in terms of issues. KAYLIE: Yarn issue count is around 800 -- CHARLES: Boy, that's a lot. I can feel daunting when you look at that, right? KAYLIE: Yeah, but it's actively [inaudible] project, a lot of people are passionate about it and there are bots that other projects use to just automatically close issues when they're not active. But something that BYK told me was when issues are closed, people take it somewhat personally and we just want to make sure that there's like a human touch to it and we, at least get to the issues without just automatically closing them. I really respect that. I think even when people are really frustrated, the maintainers never really lose their shit. They're always very graceful and when someone is acting outside of the standards of Code of Conduct, there's a gentle reminder. So far, all of the interactions that I've seen to the issues have been really constructive, rather than being like, "Sorry, we're not going to work on this." It feels like it is a community project and people care about how it's being used. It's actually not easy given all the different operating systems that Yarn has to accommodate. It's like a pretty low level tool so I give the guys a lot of kudos for handling that. CHARLES: Eight hundred issues means that with a human touch, that means 800 people have to actually respond to those issues and usually those 800 people are not actually 800 people. They're like 10 people or some number significantly below 800. How do you attack them to try and make sure that they're responded to in a timely fashion? KAYLIE: I think issue tracking is the hardest part of open source. I think it's in the order of issue tracking and then reviewing PRs and then submitting PRs because writing code is writing code but understanding other people's code is more difficult and understanding other people's issues and what the bug is, actually is even more difficult. You don't, obviously have their exact dev environment so sometimes, it's hard to repro. I think we do a lot of things that a lot of other projects leverage, which is we label the issues, we define priority if it's actually impacting a lot of people or if it's a critical bug like you can install a package versus maybe a warning message that could be tweet. Then we kept a board, like a GitHub board to track issues when we were heading for the 1.0 release. I'm not sure if we're still doing that but that helped to a degree. Other people from the community, not just the maintainers, jump in to try and help out an issue, which is probably one of the best things. Then when we merge pull requests, we close the issues that have been referenced. CHARLES: Yeah, it's really wonderful when that thing happens. I'm so curious like how to [inaudible] because I know that it's so disheartening when you're working on a project and you file an issue and maybe, it doesn't even go answered for one day, two days or two weeks or you submit a pull request and no one's even commenting on it. It can be really disheartening and kind of make you question the viability of the product itself, especially if you see a lot of activity elsewhere. On the one hand, I also understand that the maintainers are human and they probably have a lot of obligations. I know that I've got a lot of projects where I let the issues languish and I even have one that I'm using where I can't get any response and it's just so frustrating. KAYLIE: Yeah, even Yarn is definitely not perfect. There are definitely issues that sort of go buried. You could add us at YarnPackage/Core. That pings all of us and the core team. You could call out specific people but that's a tough one. CHARLES: This isn't with Yarn, by the way. It's a totally separate project. It's fantastic when there's that basic acknowledgement. It makes such a difference in people's perception of the entire enterprise. Looks like we're actually approaching time. If we want to give a special shout out to any talks you're going to be giving, any events that you'll be at or -- KAYLIE: Actually, Ben on my frontend platform team is speaking at Nodevember about Next React. I think that's where the shout out. And then Christoph will be speaking at AgentConf in Austria, I think in January about the future of dev tools. I think both things are [inaudible]. CHARLES: All right. Fantastic. ELRICK: Any slick future features coming out in Yarn that we should know about? KAYLIE: I'll keep you updated. CHARLES: She's sworn to secrecy. KAYLIE: Yeah. I'm not sure what I'm supposed to [inaudible] yet. CHARLES: I could hear the pause of conspiracy in your voice. Thank you so much, Kaylie for coming on the show. This has been a really wonderful conversation that's just gone all over the map and those are my favorite kind. Thank you very much. KAYLIE: Thank you so much for having me. This has been a lot of fun, guys. CHARLES: Well, if people want to get in touch with you, how would they go about doing that? KAYLIE: I'm a rarity, where I don't have Twitter. You can email me, I guess. CHARLES: File an issue on Yarn. KAYLIE: Yeah. I'm KaylieEB on GitHub and KaylieKwon@Gmail. CHARLES: As always, you can get in touch with us. We're at @TheFrontside on Twitter or just drop us a line on the email. You use that on occasion too at Contact@Frontside.io. Thank you, Kaylie. Thank you, Elrick and thank you for listening.

Dan Gebhardt: @dgeb | Cerebris Show Notes: 01:33 - The JSON API Spec and Pain Points it Solves 08:40 - Tradeoffs Between GraphQL and JSON API 19:33 - Orbit.js 26:30 - Orbit and Redux 32:22 - Using Orbit 37:24 - What's coming in Orbit? Resources: orbitjs.com (Guide Site) ember-orbit Transcript: CHARLES: Hello everybody and welcome to The Frontside Podcast, Episode 87. My name is Charles Lowell, a developer here at the Frontside and your podcast host-in-training. Joining me today in hosting the podcast is Elrick Ryan. Hello, Elrick. ELRICK: Hey, what's up, Charles? CHARLES: How are you doing today? ELRICK: I'm doing great. CHARLES: Are you pretty excited? ELRICK: I'm very excited for this podcast because this is a topic that I've heard a lot about but don't know much about and it just seems so awesome that I'm just very stoked to hear all the details today. CHARLES: Yeah, me too, especially because of who's going to be giving us those details, he's one of the kindest, smartest, most humble and wonderful people that I've had the pleasure of meeting, Mr Dan Gebhardt. Hello, Dan. DAN: Hey, Charles. Hey, Elrick. Thanks for having me on. I really enjoyed listening to this podcast. It's nice to be part of one. CHARLES: It's good to have you finally on the show. We talked over chat and we talked over email and we meet every once in a while and conferences and it's great to get to share more widely some of the great conversations that always arise in all of those contexts. For those who don't know you, you are a founder at Cerebris and that is your company, which is involved very heavily in a lot of open source projects that people are probably familiar with. One of them that we're going to be talking about today is JSON API. I bet most people didn't know that you are one of the biggest driving factors behind both the specification and several of the implementations out there. DAN: Yeah, that's been a pretty core focus of my open source work for the last few years. Actually JSON API Spec, which is perhaps a somewhat confusing name for those who aren't familiar with it. It was started by Yehuda Katz in almost three and a half years ago, I think now and it hit 1.0 a couple years ago and has stabilized since then and we've seen a lot of interesting implementations on top of it. There are some exciting stuff that's actually coming soon to this Spec that I'd like to share with you guys today. CHARLES: To give us a little bit of context, why? What pain am I experiencing that JSON API is going to solve or it's going to address or give me tools to deal with? DAN: One of its prime motivators is the elimination of bikeshedding. There's a lot of trivial decisions that are made with every implementation of an API and JSON API makes a lot of those decisions for you about how to structure your document, how to include relationships and lengths and metadata in a resource, how to represents relationships from hasOne/hasMany. Even polymorphic relationships have a type of that data. JSON API has opinions about all these things at the document structure level and it also has opinions about our protocol usage, how to use HTTP together with this media type to make requests and for servers to return responses, how to create a resource, how to add resources to relationships and things like that. CHARLES: Also, it's not just this is a serialization format. It's very much like also delving into the individual interactions and how they should be structured, more about the conversation between client and server. DAN: Yeah, in that way, it is somewhat unusual as a media type that covers both. CHARLES: Can you dig into that a little bit because I'm very curious? Something made my ears prick up was when you said, it tells you how to, for example add relationships to a resource. What would that look like? DAN: A lot of the influences behind JSON API are hypermedia-related. It's influenced by RESTful principles and includes a lot of hypermedia aspects. One aspect is how a resource represents relationships in terms of the data in document, the type and the ID that specify a linkage to that another resource in the same document but it can also include links to discover those relationships. There's a self-link for a relationship and a related link for relationship and the self-link will return the data for that relationship in the type/ID pairs. The related link will return the related resources. The Spec doesn't have strong requirements or any requirements about URL usage but instead, it describes where to find resources through these hypermedia links. If you want to say, add records to a relationship, you'd follow the self-link for that relationship when that was returned with a resource. Then you would send a post to that endpoint and you would include the relationship data in the terms of type and ID pairs. It gets down to that specifications so that removes the ambiguity of how to interact with these resources and mutate them and retrieve them. CHARLES: I see, so is there an idea then that you are going to explicitly model the relationships as individual resources? Or is that the recommendation or the requirement? DAN: The link for a relationship would point to an endpoint, which would then model the relationships that are represented that endpoint, so to say just to speak a little more concretely because certainly, it makes some simple concept sound a lot more esoteric than they really need to be. Let's just talk about an example. Let's say, we're talking about articles and comments and maybe an author. Let's say, you've fetched a collection of articles from an article's endpoint and within the article resource, you would have a relationships number, which would include comments and then comments could have links, which one of the links would be a self-link and a related link. The related link could be followed to then retrieve all the comments for that particular article. You could also, if you wanted to add a comment for that article, post to the self-link for that relationships. You post to that whatever endpoint that is specified. Maybe it's 'articles/1/comments.' It could be anything that you want. Now, the Spec does have some recommendations to make everything fit nicely in terms of the URL design patterns and such but those are not, by any means required but having those recommendations just eliminates more bikeshedding opportunities. We find it that people who are gravitate towards the Spec really appreciate having a lot of these trivial decisions made for them so even if we don't want to come down and be hard line about requiring those particular answers, we can at least provide some guidance for how things can work together nicely. There's a whole recommendation section on the site for things like URL design patterns. CHARLES: Right, so things that aren't prescribed but these are best practices that are recognized. DAN: Yeah, exactly. CHARLES: A question then that comes to mind, it sounds like JSON API solves a lot of these bikesheds or just kind of comes in and takes one side or the other for modeling both the resources and the relationships between those resources so there's the... I don't want to call it a schema but the boundaries around which resource are very clear and where they live and how they connect together. I was hoping we could maybe contrast that with some another approach, which is also become very popular and that's the GraphQL approach where you're essential assembling views at runtime for the client. It's very easy to marshal the data that you need to present to your view because you've got only one endpoint, as opposed to having to coordinate between them. I can understand the appeal of that and I was wondering if you have any insight into what the tradeoffs are between the systems and what are some of the capabilities that one can do that the other can't. CHARLES: Yeah, sure. I'm glad that you brought that up because I feel like GraphQL has become a real juggernaut, at least because of its marketing. It's very effective in being marketed for its use to developers and it's capabilities versus REST, as if a RESTful system can't possibly achieve the same outcome or the same efficiency. I'm glad to compare and contrast the two. To be honest, one of our short term goals is to better tell the story on the JSON API site, which was always a kind of a more technical spec-y site and a marketing site. That hasn't really helped its uptake as much as it could as some of the GraphQL sites are very sleek and polished. Anyway, let's get down to it. GraphQL allows you to basically define the data that you want for a particular view and that can bring together multiple related resources. It defines a way to specify exactly which fields you want in that graph of resources. We'll just stick with the articles, comments and authors example. You can specify that you want a collection of articles and perhaps the comments-related to that and the authors and you could have it all assembled in a single response. JSON API also allows you to do just that. It allows you to make requests for multiple related resources to constrain the fields that are returned for each resource and to include all of these related resources in a single document. The main difference in the representation is that JSON API requires that resources only be represented once in a single document. GraphQL may have repetition of resources throughout the document that's returned. For an instance, your articles that may nest authors and those authors like Charles Lowell, may have written three of those articles and that representation of that author is going to be repeated in that JSON API compound document, which is a term for document, which has a primary dataset combined with related resources. That single author would only be returned once as related resource and the linkage between the primary data and the related resources would be established to type/ID pairs. Instead of having the author represented three times, the same type/ID pairs would just be providing that linkage to the same author and that author resource would only be represented once. This happens to be ideal for client-side applications that number one, basically want to minimize the size of a payload that sent. Number two, don't want to after-handle repetition of data by doing extra processing of pushing the same record multiple times into a memory store that is keeping that data. I think that GraphQL is well-suited to applications that request data and display that data pretty much as returned. There is no intermediate holding onto that data in, say a memory store for later access. Basically, it lines up well with a component library link React, which wants to display that data that's returned from the server. If it wants to display that collection again, it will simply request that collection again and pretty much throw away the data once it has been rendered. CHARLES: I can see that. Dan, you and I might be some of the only folks who remember. I don't know if you ever did any Microsoft Access Program. DAN: Yes, I did, believe it or not. CHARLES: Doesn't it feel a little bit like the Access pattern all over again, where you have your components requesting data from basically, constructing a query and requesting it -- DAN: Yeah. CHARLES: And then throwing it up on the screen. DAN: You're going deep there but I do remember that. Definitely, there is that same paradigm. CHARLES: It's really powerful. DAN: It is and it's pretty accessible too because it's a direct representation of what you've requested and there's no intermediate processing. I guess the question is, whether that intermediate processing provides some value. Actually, holding onto that data provided some value because as far as I'm concerned, GraphQL is great for that rendering of DOM data, where the data has no meaning except outside of the rendering. But if you want to actually have models that have some intelligence about that data, then you want to use a store to keep those models in and you want to be able to reuse those models for other purposes. CHARLES: What might be an example? What's a concrete use case that we can ground this discussion? DAN: I would say that the big one is offline. You simply can't have just DOM data that's useful in any way in an offline application or an optimistic application, where you are doing some things client-side and only say undoing them if a request fails. But if your data is DOM and only structured for a particular view, then all you can do with that is redisplay that view. But if you understand the schema of your data and that data is available in a store, then regardless of whether you have a network connection, you can actually display that data in different ways. If that same article shows up in a collection in a list, you could also display that article on its own in a different format with more fields. If you want to, say allow editing of that data, you could allow for an editor when your app is offline. Allow changes to be made to that data and then redisplay it because you understand the fields that are in that data. CHARLES: Right and then at some point later, then spool those changes back to the server. DAN: That's right. CHARLES: It almost sounds like, ironically if a system like JSON API where you have very concrete boundaries around each of the underlying resources in your data model, it allows you to essentially do rich-querying on the client and not just the server. DAN: Yes, that's absolutely true. CHARLES: Because I feel like what you just described to me it's like, now we have some sort of store over which we can map all kinds of different queries to our own liking and there's no dependency on the server. DAN: Yeah and if you just want more web app to be pretty much a view representation of what's on the server and without additional intelligence, then GraphQL really lines up well with your needs because any extra processing you're doing is just not valuable to you. But I think a lot of the really interesting things being done in client-side applications are where your client application is pretty loaded with a lot of intelligence and you're out there autonomous and able to make sense of data. In that case, then thinking about the data only as it pertains to views is not nearly as powerful. CHARLES: Right, so you could do something like that with GraphQL but then you would have to, essentially structure your queries such that they drew the boundaries around the individual resources anyway, rather than composing them on the server. You'd have to query them discreetly into a store and then run your local operations. Then I guess at that point, it's like what are you doing? DAN: Yeah, you're still doing the extra processing of handling the repetition of any nodes that repeat and such. That's just extra processing you have to do but I agree that you certainly could structure your GraphQL queries to return data that is then loaded, say to a store that really has awareness of the data types but I don't think that is -- CHARLES: But then you're defeating the purpose, right? DAN: Yeah, it's not its selling point and it's not its strong suit. CHARLES: You've done a lot of work on the JSON API Spec. JSON API allows you to fetch discrete resources and their relationships but still, keeping one representation of each resource in the payload so it's optimized for wanting to do client-side processing and have intelligence based on these entities, which are in a store. You actually maintain a fairly mature, at this point, framework called Orbit, which helps you do some of these things. Now, what Orbit does today and I understand that you've got a lot of new features that are really exciting, that are coming down the pike. Before we get into those, what is Orbit and what do you use it for and how does it use JSON API? DAN: Orbit is a data access and synchronization library, which sounds sufficiently vague because it has a lot of low-level primitives for structuring client-side. Also, actually isomorphic can be run on the server and nodes so it's not even only used for client-side purposes but that was its original purpose. The abstraction that it includes are allowed for synchronization of data changes across multiple sources of data. Source of data might be represented by, say a JSON API server, an in-memory store, an IndexedDB database in your browser, a local storage, all of these sources of data can support an Orbit interface, which provides their access to their data and also broadcasts changes to that data. In order to coordinate the changes across multiple sources, say to back up all of your data that's in memory to IndexedDB source, you can observe the changes on one source and then sync those changes up with another. For instance, you want to structure an offline application which you have been in-memory store, which uses client-generated IDs, which then syncs up with a backend JSON API source and every change that gets made to the memory store needs to be backed up, you could configure multiple coordination strategies between the sources to make sure that the data flows so that every change that is made to the store is immediately backed up to IndexedDB. If it can't be backed up, then it fails. You can add some error handling and then when you're online, you can then also sync those changes up with a backend so you're basically pushing those changes that are local to a remote store and you're not slowing down your offline app, which you're communicating with optimistically and then only handling, say synchronization failures when there is a problem. In order to handle those problems, Orbit sources are very deterministic about their tracking of changes and they provide get-like rollback capabilities so you can look at the history of changes to a particular source and reset the history to any point there and basically handle conflicts and merges in a very get-like way. Often I use cases, the primary driver of Orbit's whole architecture, I realized that it needed to be able to give you the tools to handle any conflicts that happen when changes get sync up. Also, give you the different tools to model all the different places of data is kept in order to support the offline mode. That's a kind of a broad overview of Orbit. There's a new guide site, OrbitJS.com for those who want to dig a little deeper into it. The data is structured in the JSON API format internally to the store and the standard operations are very much influenced by the standard JSON API protocols that are allowed in the base Spec over creating records and removing records and all that crud for both records and relationships. That's where JSON API comes into Orbit. CHARLES: Right, I see. The primary use case for Orbit is offline. Is that fair to say? DAN: Yeah, that was the primary driver, although it's just not the primary -- CHARLES: It seems like you could use this in a lot of places where I might use Redux or something like that, like on the server to model... I don't know, a chat app. DAN: Yeah, definitely. CHARLES: I have a bunch of different information streams coming together and how am I going to merge them and make sense of them. DAN: Yeah, in fact, that it's primitive level. Orbit has essentially an async redux-like model for queuing up changes and applying those changes. The change sets are all immutable. There's actually a lot of immutability use here throughout the library. In order to ensure that the changes that are applied are tracked deterministically, we just can't have those changes mutating on us. There is definitely some overlap with Redux concepts in terms of the general tasks or action concepts in Redux but instead of Redux's synchronous approach, everything in Orbit is async. CHARLES: What does that mean? Redux is synchronous in the sense that there's a natural order to all actions. For those of us familiar with Redux, are you saying it would be like a store where actions can be dispatched at any time or is it more like, I've got multiple stores happening and I need to resolve them somehow so each one is synchronous? How can I make sense of that? DAN: In Redux, the actual application of an action is performed synchronously. CHARLES: Right. You can have asynchronous processes but there is a natural order to the actions that those asynchronous processes yield and then those are applied synchronously to the Redux Store. DAN: Yeah. To compare and contrast Orbit and Redux, I guess you'd first have to say there's a primary difference of -- CHARLES: I think there are a lot of people are familiar with Redux. I think it's not so much to compare and contrast it but just to use it is as an analogy of like, "Here's how it's the same here. Here's how it's different," because that's compare and contrast. DAN: There you go. CHARLES: But not in terms of evaluating them but it's like, "Maybe I should be using this instead." DAN: Right, they are sort of on different levels, although there are some primitives in Orbit to it and it's shipped across multiple libraries. There are some primitives, I think that could be useful outside of the main Orbit data application. Anyway, the way that Redux state changes are applied, the function is synchronous is all I was getting at and on Orbit, every state change that applied to a source is asynchronous so the result is never applied immediately. You'll always get a promise back and you'll never have that application happen immediately. That's one clear distinction. Another is that a redux has a big singleton global state for the entire application. Orbit very much has a model of state per source so there can be any number of sources in a particular application and the source might be an in-memory source or might represent a browser storage in XTP or might represented a socket that streaming data in. All of these have state at different, temporarily distinct state that even if they all converge to a common state, the Orbit models separately so that there's a set state per source. I'm just contrasting the global apps state that exists in Redux with the per source state in Orbit. CHARLES: It sounds like there's nothing that would be fundamentally incompatible of using Orbit really in conjunction with Redux, where Redux is kind of a materialized view of all of your different data sources presented as what you're going to render off of, right? DAN: Yeah. You could use it in a similar way to Redux Saga, where Orbit fills the role of Saga, where it's doing the asynchronous actions that results flow back into the Redux state. CHARLES: I'm just imagining having one big global atom, which is your Redux store and now I'm saying, prescribing this is an optimal architecture but I'm saying, one way it could work is it picks and chooses and assembles off of the different sources as new data becomes available. As the states change for those sources, it can be integrated into a snapshot state, which is suitable for rendering or provides one view for rendering. DAN: Yeah. You're basically talking about the in-memory source, perhaps merged with other applications state, which is not so resource-specific and that is possible to model. CHARLES: What I think I might be hear you saying is you could also just use another source which is the merge itself. DAN: Yeah. I'm not sure how much we want to continue this thought exercise because the architecture becomes almost not something I'd recommend. But I would actually like to explore how Orbit and Redux could be used together optimally. I played around a bit with Redux but I have not written a full-fledged application with it, other than a [inaudible] location. I definitely defer to you for Redux best practices and such and how people are using it in real world applications but I'd be really interested to talk that over again soon. CHARLES: Well, I just certainly don't count myself a Redux expert, although we have developed some applications with it. We'll put that on the back burner or something to explore it later. I will say this, I find Redux to be both wonderful and terrible, kind of in the same way the Java is both wonderful and terrible. We'll leave it at that. DAN: Okay. ELRICK: That was going to be my question. This is why I was very excited to hear about today was Orbit because I've heard so much about it. In terms of the implementation of Orbit into an application, what would that look like from a high-level? Has anyone used Orbit in the production app? Have you built any apps using Orbit? DAN: Yeah, definitely. There are people using Orbit with React, with Vue, with Angular and with Ember and there's an integration library called ember-orbit which makes Orbit usage really easy in Ember. In a lot of ways, working with ember-orbit feels a lot like working with Ember Data but it allows a lot more flexibility. I suppose one of its strengths and weaknesses is there's a lot of configuration that's possible because there's a lot of possibilities. The internals are exposed of how data get synchronized so you can define your strategies and sync up different sources. In terms of how it's actually used in an application, you'd start by modeling your data in terms of the resources that are in the application. You'd have a schema that defines your articles, your comments and your authors, just to keep that example going. Then that schema would be shared among all the sources in your application. You would have one source, say that might be the in-memory source and another source that is the representing a browser storage so you could, say swap out either local storage source or an IndexedDB source and use either one to provide that backup roll. You would declare those sources, you connect them to each other with strategies so that, say when memory storage changes, you would then sync that change to the browser storage source. Then you'd have back up and you'd be able to then, refresh your page and view the same data you were looking at before. Now then, if you probably want to wire up a remote source so that you're communicating with the servers so you bring in JSON API source and you would then set up a new strategy for working with that. You have to decide like, "When my memory storage changes, do I want this change to happen optimistically or pessimistically?" By that I mean, "Do I only want it to appear successful if it's been confirmed by the server." Depending upon whether you want to be optimistic or pessimistic, you setup your strategies a little differently. If you handle this change pessimistically, you wanted to block success on the successful completion of pushing of that change to your remote server. You have the set of tragedies that define the behavior of your application and then doing your crud operation is probably pretty much directly with your memory source. Then if you wanted to, say do an edit in a form, you might fork the store, now the store keeps its data in an immutable data structures. That forking that store is very cheap so you don't have a bunch of data that's copied. You're just keeping a pointer to that and getting a new pointer to that same immutable data structures. Every time they get changed, there been new references. There's an immutability under the hood but you're pretty well insulated from the annoyances of working with that immutable data structures. At that form, you make your changes, you then merge your changes back, you'd get a condensed change set of operations that then can flow through your strategies. It flow through to your backup source. It could flow through to the back to the server. I think it would feel pretty familiar for users of Ember Data because there are a lot of the API influences came from that library. But obviously, people are using just plain Orbit with other libraries, with other frameworks and finding it useful there but it definitely involves a little more configuration up front to do all that wiring that might be more implicit in library like Ember Data. CHARLES: I understand that before we go, there is some pretty exciting new things coming in Orbit. Do you feel like you're ready to mention a couple of those things or they've been kind of mixed in with the conversation? DAN: Let's see. I have the guides up, which I mentioned, which is pretty new in the last couple of months. In the last year, we did a rewrite and Orbit is now completely in TypeScript and there are no external dependencies. For a while there, I was using RxJS and observables internally and immutable JS so there's now an internal immutable library. It's lighter-weight with fewer dependencies now. I'm excited about that and finally feel like I can recommend people digging in with the guides that are up. I'm hoping to get up the API docs soon. I will say I'm excited. I just got back from a retreat in Greece. Séb Grosjean who owns the company, BookingSync does this amazing thing with the Ember community, where for the group that's working on Ember Data, he invites them every year to come to his family's place in Greece. He grew up working with his family on his rental properties, which was the inspiration for his company, BookingSync and said, "This is a fantastic opportunity that for us to get together and collaborate in a really nice place," and I had a really productive time this last week. This is the very first time I had gone. It was just fantastic and I worked with the Ember Data team. Igor Terzic and I spiked out some interesting collaboration between Orbit and Ember Data so I'm really looking for it to see where those go and hopefully, we'll see a little bit more Orbit, either directly or just through influence appearing in Ember Data. I'm looking forward in working more closely with the Ember Data team. We'll see what comes of that. CHARLES: Yeah, I, for one am very excited to see it. I'm resolved now. I'm just looking at these guides. These look fantastic and I'm resolved to give Orbit, at least a try here, either in some of our applications or maybe trying to spin up some new ones and have it the basis for some of ideas I've been playing with. DAN: That would be awesome and there's a [inaudible] channel, which I hang out into if you have any questions, if anyone out there does. CHARLES: Before we go, if anyone is interested in JSON API, is interested in Orbit, is interested in Cerebris, we mentioned a lot of things that in one way or another, map back to you. How do we get in touch to find out more about these different entities/projects? DAN: I'm at @DGeb on Twitter. My company site is Cerebris.com. Also check out, OrbitJS.com for the new guides. Reach out to me. I'm on the Ember Core Team so I'm also hanging out in the Ember community Slack, depending upon what you want to talk with me about. I'm in all these different places so I love to hear from you all. CHARLES: All right. Fantastic. We'll make sure that we put those in the show notes and I guess that's about it. Do you have anything else you want to leave folks with, any talks, papers or big news coming around soon? DAN: Something that we didn't really get a chance to talk about today, which I'm really excited about is JSON API operations, which is an extension to the base Spec, which I'll be proposing very soon. There's a future to the JSON API once it hit 1.0 a couple of years ago. It didn't just stop. We're looking at different ways to extend the base Spec and use it for different and interesting purposes. JSON API operations, I think one of the most interesting ones. The idea is basically to allow for multiple requests that are specified in the base Spec to be requested in a batch and perform transactionally on the server so the Spec will define how would each request gets wrapped. Each operation very much confirms with the base Spec concept of a request. For implementations, there's a lot of opportunity to reuse existing code for how to handle each particular operation but to provide a whole new set of capabilities by allowing you to batch them together and process and transactional it because it just unlocks a ton of different things you can do, all based on the same base concepts from JSON API. I'm really excited to have something to announce soon about that. CHARLES: That sounds like it might solve a lot of problems that are always associated with those things. It always comes up. What's our batch API look like? I don't think I've been on a project that didn't have a months-long discussion about that. I ended up like kicking it down the road and I'm just flumping something in place. DAN: Yeah, all those messy edge cases where people figure out how do we create multiple related records altogether in a single request and people do it ad hoc and do it with embedding and such and want to standardize that in the same way, that we've standardized the base operations. CHARLES: Well, that is really exciting, Dan. I wish you the best of luck and we'll be looking for it. DAN: Thanks a lot. Thanks for having me on, guys. CHARLES: It was our pleasure. Thanks. With that, we will say goodbye to everybody. Goodbye, Elrick. Goodbye, Dan. Goodbye everybody listening along at home. As always, you can get in touch with us. We're at @TheFrontside on Twitter or you can see our website at Frontside.io or just drop us a line at Contact@Frontside.io. Always love to hear from you with new podcast topics, anything that you might be interested in so look forward to hearing from you all and see you next week.

Alex Matchneer: @machty | FutureProof Retail Show Notes: Charles and Alex Matchneer have a great discussion that centers around routing in Ember.js: what they want to see in a router, what problems it solves, what's wrong with the routing solutions we currently have, and what the ideal future looks like in respect to routing. Resources: Episode 067: ember-concurrency with Alex Matchneer Cordova ember-rideshare react-router Transcript: CHARLES: Hello everybody and welcome to The Frontside Podcast, Episode #86. My name is Charles Lowell, your developer here at the Frontside and podcast host-in-training. I'm flying solo today. It's been a while but that's okay because I've got a really fantastic guest on. Actually, we debated this at the beginning of the show, whether this was the third or the fourth time he's actually been on but no times are too many so hello, Alex Matchneer. Welcome back to the podcast. ALEX: Thank you. It's great to be back. CHARLES: You're still at the same place that you were the last time. ALEX: Yeah. Still working at FutureProof Retail. I'm still working on bunch of mobile ember-cordova apps and that's definitely occupying on my time. CHARLES: Nice. Because FutureProof Retail has a large hardware component and we were doing a series on IoT, we were originally going to have you on the show to actually talk about that experience of what it's like to be a part of a startup and develop software that's going to be running on a bunch of devices and the unique set of problems that poses. But in the pre-show, we decided to scrap that because there's actually a topic that we're both very interested in and you've been heavily involved in lately and might be a really interesting preview as to what's coming in the Ember community and at large. Today we're actually going to go back to talking about the same subject that we talked about in our first podcast, which is routing: what we want to see in a router, what problems does it solve, what's wrong with the routing solutions that we have today. Talk about what that beautiful, ideal future that we want to live in looks like with respect to routing. You've been thinking about this a lot lately. What have you been thinking? ALEX: I'm an Ember core team emeritus and back when I was on it and I'm a lot more active, I did a lot of work on the router, particularly with how it handles asynchronously loading data when you click on links and go to different sections of your app. I spend a lot of time over the last three or four years figuring out the nice patterns for what you actually want to use if you're building out lots of Ember apps. Then kind of around that time, right after landing some cool stuff and some not cool such us query params, which has been a challenging aspect, I start working at this company FutureProof Retail that is like 90% of the Ember work that I do there is in mobile apps. We use Cordova so we're basically running these apps inside a web view, inside either iOS or Android so that we can stay with the technologies we are most familiar with, such as JavaScript and CSS and HTML and build apps using that. We can use Ember to do that. What I found was that I couldn't really apply a lot of the same patterns, all these nice conventions that Ember router gives you. I couldn't really find a way to map that onto what I need to build in mobile apps and there's a few different reasons. I got really busy with the startup, just trying to build these things and kind of went off the happy path where I really just couldn't find a way to make it look like an Ember app. One of the nice things about the whole points of convention over configuration as this sort of Ember and Rails philosophy is that, one of the benefits is that if you know Ember and know Rails, you can drop into someone else's apps as long they're following these basic conventions and immediately know how to be productive and know how it's structured, know how to make a change to it and have it maintain a convention and not just have everybody who's using some framework build these totally different apps from each other that have no shared conventions and whatnot. Everyone is supposed to be able to learn from each other, grow with each other as long as they stay with these conventions. I couldn't really find out how to stay within Ember conventions and build this mobile apps. For a long time, I just didn't really contribute too much to the Ember router at all. I kind of fell out of touch with how most people are using it because most people are building these desktop-centric apps and here I am working on these mobile apps after three years. CHARLES: What are some of the specific use cases that were just impossible to, or not impossible but presented a challenge? ALEX: The first one is which is I think is actually one of the easier problems to solve but still some challenging is that you want something that's called stack routing or stack navigation in a mobile app, which is if you're actually building a native iOS app or an Android app, they both have different names for how they provide you this. But you're thinking of things in terms of stacks. In Android, you might open another activity, which is a full frame of a page in your app and you can push it and then when you press the back button, which is built in in Android phones, it'll pop that off the stack and take you back to where you were. In iOS, they give you a UI navigation controller and let you push and pop view controllers and that is how they want you to think about these applications. That is contrasting to what Ember makes you think about, which is go and define your static hierarchy of all the different places that you can be in an app. But with stack-based navigation, you don't necessarily know upfront all the different orderings of which frames are going to be pushed onto what and you might have situations where you want to be able to dynamically push, say an 'Add a Credit Card' page to where you are and maybe it depends on some data that's been loaded at some lower level in the stack and you can't model that as nested routes in the way that you might think about it in classic Ember apps. It's a different structure -- CHARLES: Now, when you say lower in the stack, I'm curious, if you're entered in aren't you... Oh, you mean... I see, previously in the stack. Okay, so lower in the stack so you're thinking like your current position is at the top of the stack. ALEX: Right, yeah. CHARLES: I see. Now, let me just clarify this in my own head. Your Ember routing structure is ultimately realizes a static tree but at any moment, you are entered into one path through that tree so you do have something resembling a stack. It's just is it the pathways that the ways that you can actually get nodes onto the top of the stack is you're limited because that can't be dynamic. ALEX: Yeah, but even then, it's hard to describe what the difference is but the kind of stack that you're thinking of in terms of the classic Ember router map is more like you're in these different substates than you are different frames that you've pushed onto your -- CHARLES: There's a finite and fully enumerated set of next states. ALEX: Right. To be very concrete, if you have a post route and then a post show route and then a comments route under that and these are all nested in a row, then if you're in the comments route, you are in a kind of hierarchical stack that might have loaded the post that you're looking at and maybe the post call-to-action above that and the comments for that but you're still in one thing. You've just expressed that one thing in terms of these substates so that every other state that's in the parent state can share the same data loading. That's different from saying, "I'm on this page and now, I want to push another page on it and maybe tap some of the data that has been loaded on previous pages." That's more of a navigation stack in a hierarchical substates stack. CHARLES: Is the difference then, the data dependency? Because if you think of the Ember classic where you got the static tree, at least theoretically all of the data in the leaf nodes depends on the data that's above. It's not just being able to dynamically push stuff onto the new stack but it's also saying, you want to be able to push stuff that might have no dependency on the stuff further up and it doesn't need to be re-rendered if stuff further up the stack changes. ALEX: Correct. CHARLES: But sometimes it might. ALEX: Right so there are a lot of corner cases that come out if you try to model this new way that a lot of corner cases have been thought out of if everything matched nicely to this hierarchical substate classic Ember stack but not for navigation. If you want to do something that's stacked routing-based, I've had a few different approaches. At our company, we maintain a suite of different apps that are sort of retailer or grocery-centric and the first one we did, which is more popular flagship one is Mobile Checkout, which is an app that lets you going to stores, scan items with your phone and checkout and skip the cashier line, which is great if there's huge lines and you just want to buy a little handful of things or maybe in your shopping cart. But that is like any other mobile app is really conducive to this step navigation approach. Then we had to make a few apps after that such as like another app that is [inaudible] do a manual check then ordering app and other of handful things that you can imagine is might be used on a grocery store. I took the opportunity to like, "I don't really like how the routing turn out the main mobile checkout shopper apps so let's try different things." If you approaches, at least have their pros and cons without really feeling you're solving the problem and one is to maintain your own in-memory stack of where you were, every link to you, you might recall where you were and then use that logic in addition to what's in a URL to decide what transitions to make, which to use Liquid Fire for that. But already, there's these weird growing questions like, "Why are you even using the URL? Is it helping you at all?" That was the main issue with the main app that we did. The other approach was to try and not even use any of the 'router.map' stuff at all. I use the router.map to basically just create one wildcard route. You can use normal Ember to use it like '*half' and that basically collects the rest of the URL as a param that you can use to do whatever you want with. I was using that to basically pass to another, which is internally used by Ember to do the stack-based parsing like grab a little bit of the URL and then parse the param for that then grab another. Every time you could see your stack in the URL. That has its benefits but the worst part about it is that it's getting further away from Ember so any add on that you might want to use at Internet of Things in terms of which route you're in and has conventions like that you just can't use. I can't think of a good example at the top my head but it's like the further you get away from those norms, the less the Ember system can help you and on your own building your own framework. This is all to say that I think I have enough experience at this point to bring home some of the things to Ember and I'm excited to get back into contributing to Ember with this one particular thing that I'm focusing on now, which is... I don't even know what to call it. It's like -- CHARLES: What does it do? The route stuff? ALEX: It's route stuff. Actually, let me get into the other... That's what is tricky about stack routing and tricky to sort of, if you already have to go through a mental hurdle with thinking of the Ember router and as a stack of states or substates and you train your brain to think that way, it's really hard to take yourself out of it and realize that what you're trying to build with like a classic mobile navigation is almost looks like the same thing but it's really different. The other challenging problem, which is specific to our particular app is that you wouldn't think of it as a very heavily server-driven app but if you're writing an application that at any point can get a message from the server like, "Hey, your status has changed," and that state is heavily coupled to navigation of where you're allowed to be in your apps for the state of some certain model, then you're going to have a really hard time, I'd say in modeling an Ember. I have a really hard time convincing people of this until they've actually tried to do it themselves, which is why I'm going off and just building things showing people. CHARLES: You don't have to convince me because I think one of the biggest problems is the router is like the one non-reactive piece of Ember, which is unfortunate because it's essentially, what is the equivalent of the Redux store in a Redux application, where it's the state that drives literally the entire application and yet, any type of non-hash change driven updates, you have to manually manage. Every time that we've done it, it's been a problem and depends on what data, at that point you have to be very thoughtful because, at least from the highest level, if there's damage to a piece of the tree higher up, you need to realize those effects of that damage or that change all the way down the tree. ALEX: Exactly. That is a great way of putting it. This is maybe a good time to mention this thing called ember-rideshare. I've had a really hard time describing these problems to people so I figured what I would do is write this blog a few months back, a little article called ember-rideshare. It's just a given name to the kind of app that still really hard to write in Ember. It's a mobile app. It involves stack routing but the other part is really difficult about it is this problem of the router being in a silo. It is reactive but it's only reactive to that URL. Other things changes, they need to, like you said come in and patch up something else about the router in case you add some URL that is no longer able to present some model of whose status changed. That's an article on a blog that I can probably link to in show notes or something. When I talk about ember-rideshare, imagine using Ember to build Uber or Lyft and it's got just the slightest bit of the whole thing. The whole point of the app is to coordinate your client-side request of I want to ride with the server going off and doing a bunch of things and finding a nearby driver, displaying you bunch of driver locations and it'll show up. Then finally, find you a driver. It's a constant communication. Throughout that point, you can sort of imagine modeling all the different screens as routes but the routes that are actually allowed to see at any given time are heavily dependent on what is the current state of the user's current ride. But you shouldn't be able to go to a route that says like 'cancel ride request,' if you haven't requested a ride in a million of these other things. If you're an Ember developer and you think that's an easy problem to solve, you're probably thinking, "I would use before model hook when I'm entering that route to check the state of the model," and if it doesn't make sense for the route of entering, I want to transition elsewhere. That's fine. That's good if you're doing an app if the user is the one deciding where to navigate to. But then when you're on a route like that and then the server tells you that your ride is done, you can't still be on that route so you've got to have some kind of validations that is like, "This is no longer a valid route to be in. Is the user still in this route?" CHARLES: "Where am I going?" ALEX: Yeah. Before model doesn't really help you. It's this one-shot discrete event and you just can't capture all the different things. The ember-rideshare describes some of these problems a little bit more detail but that's the main issue with it. Like you said, what is actually missing about the router? Maybe it's reactive but it's only reactive to the URL, what about all these other things that are happening into your app? I think there's a handful of APIs in Ember that they're great but they're kind of siloed off in a way. If you want to make two different kinds of worlds meet, you've got to write a bunch of your own code yourself or you just have to do mentally going back and forth and being like, "I did this, so I can't use this kind of API." I did a lot of work on the Named Blocks RFC, which previously there is silos between if you're passing blocks to a component versus data, you've got to think about them differently and all the ways that you might forward that data to a different internal component, if you want to build these composable, reasonable internals, you got to be kind of split-brain about it. I feel the same way about how the Ember router works. It's only good at dealing with stuff that has to do with the URL and you're on your own, if you needed to react to data changing. That's what I'm trying to fix. Does that correlate with your experience of working on Ember stuff as well? CHARLES: Absolutely. I think that's a great way to put it. I think we've come to a consensus of the problem statement. I am curious to see a big separate query params. I'm going to throw that wildcard out there or maybe we should save it for later. ALEX: Yeah, I definitely going to come back to it. If I say all this cool stuff and I still don't have a solution to that, then what am I talking about? CHARLES: Right. ALEX: Which to be honest, I haven't thought of every single possible thing. I'm doing the thing where I talk about it on a podcast that everyone can guilt me into really finishing it. I actually really think that I'm going to finish it. I'm very confident in stuff I'm working on. I'm very excited to bring it to people but it is not all 100% fleshed out and I definitely appreciate anyone's help to those interested, understands the nature of the problem and wants to help me work on some of this stuff and like that, in Ember community Slack or wherever. CHARLES: Yeah, I'm really excited to hear it and see in what ways we might be able to contribute. ALEX: Basically, the goal is to find some underlying primitives that can model the current behavior without mistake because obviously, we can introduce something that's going to break into Ember apps. Basically, to recognize that the URL is something that goes through multiple passes of transformation, to eventually become the thing that displays stuff on your screen, from the very foundation of it, and this is the actual mini-course of what Ember router does internally because it involves a few different libraries and maybe this is a re-hash from the podcast that I did with you guys but -- CHARLES: Can I just say that there are some things that the Ember router really does right, that are fantastic? One of those things is it baked in to every single piece of data. It doesn't do the stack but in that tree that it models, every single node in that tree abstracts away the asynchrony of that node. I think that's absolutely huge so you get both the dependency enumerated like these are the things that I need to marshal the data to render myself and it's implicit that it might take some time. I might need to draw on a couple of different things to actually assemble this data so the asynchronous nature is modeled up front and it's implicit and it's there every single time, which turns out to be the right thing. The sampling that I've missed has been an excruciating void in all the other routing solutions that I've tried outside of the Ember community is that they just punt over asynchrony to you. You deal with it, not our problem and it's like, "Actually, that is the problem." Anyway... ALEX: That's a great point because if the router doesn't help you with any of this stuff at all, then it basically means that every one of your pages that you might want to render after the fact, probably has to have some loading logic like if data is loading, show us spinner. Otherwise, here's all the data -- CHARLES: Yeah, if something happen wrong. ALEX: Right and sometimes that is actually what you want to do. Sometimes you want to do these skeletal in UIs that looked like the page that's about to display but the date isn't there yet so everything is, regardless going to be wrapped in these 'if' statements, 'else' statements. I worked in ember-concurrency and some people are using that to basically move more of that loading into controllers, that's fine. If that's what you're actually trying to do and that's what you're opting into, that's a perfectly reasonable solution but most of times, chances are you're entering a route and you don't want to have to teach the entire template tree underneath it that has to handle all these different states. There's these nice ideas that work in some cases and I'd like to make them work in more cases than Ember helps with and a whole loading all the promises and the model hooks and absolutely going into the loading state are really cool primitives that Ember is going to do for you. The other frameworks, they don't try to be opinionated. They won't do any of that for you. Sounds like you ran into that with some of your React stuff? CHARLES: Yeah. I definitely did. There's just not much help when you actually want to model asynchrony. You can do it. It's pretty easy. You just implement the right hooks or model a series of actions, either with a Saga or Epic, if you're using redux-observable. But again, you have to assemble it by hand and you have to generate those abstractions by hand and you just want to have them at hand already and not have to worry about that. But the advantage, though is that generally those ones that you do have at hand or that you generate are fully reactive. If new information comes that's germane to that particular leaf in the tree or that particular note in the tree, there's no difference between the initial state and the update state. Whereas, in Ember, you got your first shot and then that data is now at rest. ALEX: Right. I definitely have been looking at React router, in particularly v4. I think it's all contentious for people to see it at first but being able to put things like in your render function, you can say, "If this data is present, something that's going to be past and be a prop or something," then show a loading spinner or otherwise, start matching these subroutes. That's really cool. That's expense that you can't look at essential map of all the states of your router can be in but that's also a real problem and if you can demonstrates that the state world is not in a separate silo than the routing world. CHARLES: With great power comes a lot of bugs. You do run into a lot of things where you have rogue matching. You have random things that are inside your view tree that are matching against the route and they just render and you have to be very careful because it's almost the difference between blacklisting and whitelisting. I see what you're saying. It could be confusing. ALEX: Yeah. I think it's definitely a tradeoff. I think if I had something like a match, I might have been able to maybe arrive at a stack routing solution a little earlier. I'm not sure about that. It's definitely something that could be handled by React router. I think one of things that React and React routers better at in general is that everything is, more or less a component that is more easily swappable or something else here. You're not going to have as many of those silos and I really do think, it went through a lot of churn and maybe, some people had trouble, maybe a lot of people, I don't know had trouble kind of following all the major versions. But I think React router Version 4 is pretty damn cool. I think there's a fullest realization of that kind of modular mindset. CHARLES: I think the biggest problem I have with it, though is it requires the view tree to model your routing structure. That bothers me. I feel like you could do the exact same thing. You could have a way to express your routes, not necessarily with a separate routing file. I supposed you could do it with JSX or something but actually have it be kind of orthogonal to your view tree. The way you can model this dynamically updating thing that can match against anything and maybe, even express it all in one place. Although once you get a big tree, it could be hard to control that. The part that I've come into most conflict and maybe who knows, maybe I just haven't used it enough, we've only got one application that we're using the router V4 on. But the fact that it's actually in the view tree, it bothers me. It's in the state objects. It's hard to adapt to Redux because that state is opaque. It's the routers controlling it and I would it to be not have to pass through React components but just be like, give me the firehose of the router state. ALEX: Right. I love what you're saying. If I'm going to bring this stuff to Ember, I can't suddenly make it work like matching within the view tree. That's not what I'm working at or proposing here. All the stuff is basically to empower that firehose to respond to more things that can drives views and respond to them in a live way, not like a one-shot async validation, only when you enter. CHARLES: Maybe this is what the problem that you're trying to solve and one of the things it's really nice to be able to match against anything inside the view tree is that Ember's rendering process of a route is very opaque. The process, by which an outlet gets connected, that's not something that you really have much visibility into. Is that a good statement of the problem? ALEX: That's definitely part of it. You definitely have to go to the documents. I think it's telling that -- CHARLES: I've never done it. I don't really know how that works and I've written a lot of Ember code. ALEX: How what works? CHARLES: How the route gets rendered, like the mechanics around, which I understand how the route object actually, you makes the decision to render its template and do all that stuff. I know it as a user but I don't know the mechanics and I wouldn't know how to extend it. ALEX: I'm not sure if the stuff I'd work on but it immediately make some of that stuff more clear. One of the goal or constraints is to really try and break down the silos. Whatever I'm about to propose bringing to Ember, I want it also be something that would be useful, possibly at the component or template or controller level, rather than just being this thing that lives only in the router's weird black box of logic that occasionally calls hooks that everyone knows about. CHARLES: Right. In a sense, I'd say that they both suffer from that same problem. I'm curious to hear about the firehose. ALEX: To actually get into what I think you're building here, we can dance around it all day and then we -- CHARLES: Just save it for the last 30 seconds of the podcast. That way there could be no -- ALEX: We're swapping JS for React router V4. Bye! It's basically this. What's happening today is that you have a URL, it's going to be parsed in a way that you've tied it to via the router map file, which every Ember app has the place to go to see all the different places that you can navigate to an Ember app, which is great. You basically taught Ember how to break your long URL string into these usable bits and that's going to give you an array of these things that internally who cares what they're called but they're called handler infos and they basically say, "The first element of this array is named application. Every Ember app has one. It doesn't have any params." The next one, it starts getting into what your URL actually is. Maybe it corresponds to the '/post' portion of the URL so that's going to be named 'post,' and that doesn't have any extra params either. Then there's this thing that is post show or something like that. That has a dynamic param because that's the part of the URL as like the '/123' and that corresponds to the post ID. It's basically, if you like thinking of things in terms of transformations or observables or mapping and functional transformations, that's taking a URL and turning it into an array of these useful POJOs of information. The goal is to keep transforming that into something eventually has enough data to display and templates and whatnot. In this giant black box of the Ember router, it's going through those transformations and then it's going to go through this long series of using these params and this useful array of POJO information, start hitting hooks on people's routes to load data. Hit before model after model, redirect all these things to give tasteful names to all the tons of validations and checks that you might want to do. You do cool things in your before model hooks, check if the current user is actually an admin to prevent them from going into any '/admin' subroute. That's a really cool place to go and it's also a great convention. If you're new in Ember app, you realize you can't go on this route. It should sort of click in your head and that sounds like they've got one of these redirect hooks to ensure that you're not going anywhere you're not supposed to go. All these things are really still to this day, extremely strong, well-designed, it went through many passes of review before it landed. I think they cater to a certain kinds of user-driven clicking around apps but they are extremely strong to this day. I think the only thing that's missing is the smell. That example I gave like checking if the user is an admin, it's a bit of a smell that is not reactive. It's a hook. If it passes, great. You're in the route. It's not going to keep on checking that. What I want to do is basically, either in addition to or as an alternative to specifying these one-off model hooks or these hooks that you, not only really just fire one time, have essentially what is an async computed property or an async validation that is upfront about things it depends on. Ember is going to be smart enough to constantly reevaluate these things as stuff changes. It can depend on not just URLs or URL parameters but it can also depend on data. If you're thinking about ember-rideshare, which again is the imaginary Ember app that it's essentially Lyft or Uber, if you have a current ride model loaded somewhere, maybe by a parent route or maybe it's some sort of service, you should be able to specify it like an async property or validation that says, "I depend on ride.state," and for all these subroutes, you would want to say that, either upon entry or any point in the future, if the state ever changes to something that I don't know how to handle that go to some default route. That would be already, particularly in my app, which is a subset of a different kind of ember-rideshare app, that would be a huge help because the only other alternative is to build a sibling-central coordinator to the router that isn't the router but has to sort of agree with it and then, every one of these frames that you might push onto the navigation stack, they have to do some little chunk of code and then invoke this logic and be like, "Did the state change? Go where you're supposed to go," and they have to do that logic. It would be, I think a great win for conventions as it has if it's a benefit to make people shout out their states in advance to empower them to shout out also their data constraints in advance so that you get things like automatic redirects and things change, I think that would be huge. I know that would immediately benefit off of it and I think it would fall in the same kind of problem solving that they worked on like Ember-related stuff which people don't realize how big a problem is until they see there's a better way of doing stuff. I think with that being there -- CHARLES: As an example, let's say that you're an admin and then all of a sudden, you got fired and there's an event that comes from a server that's this person is no longer an admin and it wipes out the Ember data store and then redirect you outside of the admin route or something like that. ALEX: Yeah, that's a perfect example. To be pedantic, I think a lot of people do hard refreshes between login/sign-off stuff but if you have it all in your Ember app, that would just happen automatically. You'd still want the ability to have more graceful transitions because one of the tricky things about having stuff driven by data is that you have this giant matrix of like, "If I'm in this state and this event happens, how do I handle it? How do I make it look well-designed to the user?" But you're not going to be able to hit every one of those constraints so to just have some basic logic that's just like, "Oops, something happened," you're not an admin so we move you to the sign-in page. For in those cases, we haven't fully filled in all those leaks. I think it would be a huge win and you can just progressively decorate things according to the common flows that people take through your app. CHARLES: You know, I'm just imagining this. Model promise, for example would be some computed property, then how would you enumerate your dependencies? Just do the mechanism that we have now? Or are you imagining something entirely new? ALEX: I don't have a strong opinion on it because the moment I start saying what that specific syntax is, more people will agree on what's missing and what we need to have, regardless and be like, "I don't like it." I'm leaning toward something inspired by a lot of my learnings from observables, which is actually we talked about last time. The whole thing about observables is that there's almost limitless flexibility as to if you're in observable, it can take that event. It has been another observable based on that thing. If a URL changes and you're listening to that via observable description, inside that, you could kick off another observable of Ajax request based on that URL and it doesn't make you enumerate all these things upfront. I think there is going to be a compromise between that. I think when you get into these kinds of problems, you run into stuff like Relay, which is familiar with -- CHARLES: I haven't used Relay. ALEX: Just the idea of dynamically collecting all of your dependencies upfront before hitting the server and asking for specific chunks of data that you need, it's a very promising idea. There's cases of just dynamicism where the data comes back from the server, then you realized that you need this other piece of data and there's no way you could have collected upfront, unless you statically wrote it upfront. I expect to find that with this approach that there's going to be some stuff where you just have to be more upfront about it. But I had a cool little strike the other day on auto-computed properties and I'll also link to that. It's a different way of running computer properties where you don't have to specify your depending keys upfront but your getter function gets passed a getter function itself. CHARLES: It's past the dependencies? ALEX: Not even that. Imagine writing a computer property and the first [inaudible] is a function that you can call to get a property off of this but also track that you've got that property. If it ever changes, it'll invalidate again. That means if you're implementing a [inaudible] in computer property, you don't have to write first name twice, both in your dependent keys and in the actual getter in your function, which I think is kind of cool. I'm trying to make that pattern work for this data loading thing so that you don't have to have this huge verbose thing. You just lift this stuff in one place. I've sensed that the magic will probably break down in some complicated cases but that's what I'm trying to run with because I think it's pretty cool and succinct and sort of the natural evolution of what people think of as computer properties. The other major constraint and this is also what we're talking about because it's one of the best kept secrets about the router or it's one of these things that everyone's benefiting from without realizing it, is that if a transition occurs in the router, everything in the router is going to be a possibly long asynchronous chain of operations that it collects all the data that it needs for the new routes to display. In that time, if something happens, if some hook comes along and has an exception, it can load data from the servers. If something happens then it just says 'transition.abort,' that's going to stop whatever transition is in place and you're going to stay exactly where you were and if you're not stuck in a partial transition state, that's pretty awesome. That's basically database atomic transaction semantics that people have been benefiting from if they've been using Ember for years at this point. But again, it suffers a problem being locked away in the router. That is a cool concept. You should be able to specify like I intend this change of the state this way and if I gave you something that is logically inconsistent or can't be fulfilled, don't leave me in a weird half-assed state that I need to somehow fix and know how to fix all the different places, where I might be kicking off this transaction. I'm trying desperately to preserve those semantics when data comes into it. One of the hardest things to do is and honestly, can be one of the hardest sells for people who are used to thinking about Ember is there's an issue of if you imagine whatever API we're talking about, it's probably going to live on the route. Some kind of hook that might be called resolve or something else, like what is the value of this context object that every function has? Is it a route? It's tempting to want to do that and maybe, that will end up winning but winning out is the best API to get people to use. The thing to realize is that there is no consistent value of this. This implies that there's a state of the world and you're looking at it and currently, these things have these values. But in the transaction phase, there is no stable 'this object' and you can wind up with some weird surprises. I know because, not actually these days but particularly, when a lot of the stuff landed and people started trying to do weird things and these transaction hooks, there's just like, "Why can't I grab the controller? The property isn't what I expected?" Honestly, all the stuff that is gross about query params because of this fundamental violation. You have something that pretends to be a property that is there today but is still driving this asynchronous thing that could fail. CHARLES: I kind of viewed this as playing an off-note in the jazz thing like you only want to reserve using this, unless you're the Miles Davis of JavaScript, don't use this. ALEX: And by Miles Davis, you just mean like the god of concurrency that's incorrect race-condition-y code. CHARLES: Right, so it's just like you've got the right reason and you can spot the one-in-a-million case, where it's appropriate. You can spot it in an instant. ALEX: Exactly. I'm not that person and I don't know too many people who are and that's not the API you want to land. I'm trying to, maybe wean people off on dependency on this because the way we've gotten around it in the past is to use again, is more discrete, get the value functions called 'get model' and 'get params.' These are all very in-depth stuff if you're pretty experience Ember developer but it's a way of getting a value from one of these parent routes when you're inside a transition and the rest the world can't see it but you can because you call this hook at the right time. It's super gross because it's just a method on a route that anyone can call in any given time, whether you're inside this transaction or not. The branching logic of, "Should I look up the data from the transaction object?" because once valid, I should have get the current value of a loaded route. It's really gross to me and it causes real problems that confuse people and causes them to write issues because they've given an API that makes them feel good about treating these things as stable objects. CHARLES: I'm trying to imagine now, just like a spike in my head. I know you don't want to get too into syntax but essentially, modeling the route tree as a set of observables, where essentially, instead of returning a promise from your model, you're just mapping an observable off of some combination of the URL state or what are the other streams of state you want to merge to realize that route. But what I'm not seeing, which I'm sure you also have the answer is the original problem, which was stack routing. What we've been talking about is making the router fully reactive like this fully reactive tree that's always on. But that problem seems almost orthogonal to the stock routing problem. ALEX: It is. It's been very tempting to combine them. Why it is such a hard problem? Because you've got navigation stack, which almost to this route hierarchy stack that [inaudible] about but they're separate so you can't really apply the same lessons. Then you've got stack routing, which is you want the ability for routes to while they're loading, reference data that is dynamically available to them. I don't have a solid answer but I would say, the one thing that I think is going to help is that you have a few options for what you want to stash how you want to represent a URL or where you want to stash your hierarchy. Actually just track it in-memory and if you refresh the page, it'd be like, "I depend on some data that I expected to be there but it's not. It transition elsewhere," which is not a great developer experience. You could want to be able to make changes and refresh the page and continue where you left off. Otherwise, URLs aren't actually used by mobile app users. But the other place that you could possibly put the navigation where event stack is in a query param because that can be fully dynamic and you can just sort of manage every single page. The most current page you've pop is just some top-level route but you're tracking the state on the side. I think if you solve the problem of being able to depend on things that aren't the URL or go through a more complex transition than what the router gives you by default, I think it would be possible to treat that query param or that thing you're stashing in in-memory as another source of data. The other thing that I want to try and make sure that this new API has is really treated dependency injection where you specify all the things that you need and you don't really care from a route's perspective where they come from. I think if you had that, that would solve a lot of problems with stack routing and where it gets data from. To be very specific, today if you were in that post '1, 2, 3' comments route and you needed to access the post model from within the comments route, you would probably do this model for post. Basically you're naming not just the model that you need. You're naming the route that you know provides it upfront, which I think is that. Actually, the real reason it's kind of the smell is that, if you ever need to change the nesting, maybe you need to introduce another level or you want to nest all that under an admin route. Then suddenly, you're asking for the wrong route name. You're not really sure all the different things you need to update if you ever change the nesting of your router. There's solutions like relative URLs that a lot of people thrown around but I think -- CHARLES: To go back in the observable world and specifically, the redux-observable world, it's like a simple map. You're just mapping down off of a global prop, you've got some tree of state and you're just mapping off... What was that like? A model hook and you're just mapping down off of that? Wherever that state lives, you're mapping to it and now you kind of slicing off your little garden hose off of the firehose. But still one huge -- ALEX: I've tried to apply observables to this problem. I don't think I've never seen the observable analogue of is this idea of dependency and injection. To model something as a stream that transforms over time, that's proven to be very useful but to sort of say, "I am an observable that expects these objects given to me," I'm not really sure what that API would look. CHARLES: I would say, just as a straw man perhaps, you have this dependency that it's a well-known location. It's a well-known name. With dependency [inaudible] in classic, it's like, "I depend on the off service. This thing called 'service:off' or whatever. Imagine that you have some pool of state and there's some key called ‘service.off' there and as long as I'm just basically basing my stream, the first thing I do is map off of this and maybe map off of another key and then combined those into a single stream, then I can be sure that I have those things at all times. If they change, my mapping function or my transformation function is going to get evaluated again. Does that make sense? ALEX: Yes, I think we should [inaudible] C without code or something. CHARLES: And maybe I'm thinking about it wrongheadedly but that would be a simple mechanism. ALEX: Could you run by me one more time --? CHARLES: Yeah. Let's say that we've got some authentication service that you want to depend on like you want to inject on it. You want to inject that dependency so why can't you base your stream off of that key? You have observable map, for example. The list of transformations that you would have to do to peel off multiple keys, I'm sure you could write helpers for it. But basically, probably if you're going to be wanting to inject multiple dependencies will -- ALEX: The problem is this. Basically, if you want to write your resolved observable, if this thing based on observables, remember that there is no this in a route because of the transactional reasons of what we've talked about earlier, what are you getting that from? You need to have something passed into you, to be like 'context.get observable blah.' CHARLES: I would just assume that it's implicit. I was thinking a bit basically, the simplest case would just be an observable that was basically taken off of the entire global state or whatever of the router or what have you. The way the redux-observable works is every single epic is what they call them is just a transformation on the global stream. Usually, the first thing that happens is they map down to the local context so the -- ALEX: Like a path? CHARLES: They have a helper like action of type, blah. You only see a subset of the actions that get maps to the Redux store. I think it's redux independent but at least in theory, every single epic is basically going off of the entire global state but the first in reality, what the first thing that happens is you're like, "I am only interested in this subset of the state," so you do a map off of the global state down to your local scope and then you work from there. In fact if you had the convention around that, you could even make that part implicit. It's like I return an observable that it's only seeing the stream of local states. ALEX: That makes sense if there's sort of canonical state of the world but what you're doing when you're transitioning into a route is trying to feel out another state in an asynchronous manner. Redux is the action causes state to change, now the state is this. But the action for type thing, I think that makes sense if you are subscribing to the world global action on this one store when you're constructing this new tentative, may not actually become the store, you're depending on values. What we need in our API is something that depends on values that are from a tentative store. CHARLES: It's similar so in redux-observable, you're mapping actions to actions and you're not necessarily mapping actions here. You want to get state into the equation. ALEX: Yeah and it's so almost observables. It's just this twist of transaction dependency injection. It sounds really over-engineered but the thing is it exists in Ember today and if it exists in a less siloed way, I would certainly benefit on it. I think everyone else would too. CHARLES: Okay. With that hand wave... ALEX: Oh, I didn't mean for that to come as a hand wave. CHARLES: No, no, no. I'm kidding because I think we actually have a lot more to talk about here and we're running out of time. One of the thing that I want to ask is, talking about redux-observable, talking about redux and stuff, have you given any thought as to what this might look as a library that everybody could use? ALEX: I basically have something that's using Ember CLI only because it's so easy to just use it as a sketch pad and get test passing but everything I'm building so far is just ES6 class syntax that can be transpiled in it to whatever. I'm actually realizing, there's a lot of overlap between some of the primitives that are involved and Glimmer so it may or may not have a pass that uses references for tracking when things change until no one to invalidate and refire these async hooks. But either way, I'm going to make sure it lives in the JS usable world and not just Ember's special object model end. CHARLES: Right. Those interfaces are pretty narrow. The things that implement those interfaces are huge and complex but the way, at least I understand it, isn't the reference interfaces themselves -- ALEX: They're really simple, yeah. CHARLES: -- Really simple. It could almost be copied and pasted and not have much maintenance overhead in there. Here's a question and this is probably getting too far into the weeds. Can you not model a transaction as an observable? Essentially, with a flatMap, you would merge in some observable into the chain that was basically a transaction of all the other observables from which it is composed. ALEX: You know, a transaction as it builds up all the new state over time could be part of the main tree and if there is an active transition, then that's future potential state that the world might become and it could be modeled as a leg of the Redux state. I think you could theoretically do that. Definitely worth a try. I don't think I would benefit too much from doing it now and I think this could be a premature optimization but I think there would be just quite a bit of intermediate object collection to express that. I think theoretically it works but how it's going to physically map to Ember in the near future, it would be harder [inaudible] in a way. There's actually a lot of stuff that is very redux-y that again, a lot of Ember people don't maybe know about because it's internal but the way that Ember [inaudible], I think since Edward brought some of his learnings of Liquid Fire back to core Ember, there's this concept of outlet state, which describes -- I'm not an expert on it -- what's rendered where and then each outlet gets a chunk. Like you said, a little piece of the firehose or garden hose, pulled off the main thing so it can just focus on the one piece of state. Those are simple objects that produce this part of this transformation process. That's kind of redux-y in the way that everything just gets a new POJO and stuff changes but it's not strictly redux, obviously and probably won't become that just because it's already good enough on its own. CHARLES: Yeah. I think it's actually good at this point to be hand wavy because the most important thing is to be non-committal about the syntax, like you said because that's when the bikeshedding begins and now it's not the phase. The phase is to come to some agreement about what is that we would love to see. ALEX: Basically, the thing is this. I think people need to realize that Ember won the bet that the URL is an important thing to build apps around and if you have a state that's representable in URL, that state should go in the URL so you can send links around and not break the web and have an app that works that's built on half-assed routing. The only thing I'm proposing is going to make that go away. It's just that there is already this giant world of stuff that's not expressible in Ember today because it is driven by state. If you make that as easy to express and as upfront to express, I think you can have shared conventions versus what everyone is building these apps that I have to do, which is to make a sort of separate router of state-aware stuff and not have to make those two things agree with each other because it's really hard. CHARLES: Right. At that point, you're writing your own framework. Maybe this is the next big thing because I feel like Ember usually has the best stuff way, way, way, way before. Now, we're finally getting to a point where everybody seems to realize that having a CLI is absolutely critical to the developer's experience and most frameworks aren't taken seriously until they've achieved that. It was the same thing with a router back in the day. I'm wondering what that next thing is. ALEX: I don't know. I don't think this is going to be it. I just think it's a good progression. I think a way forward that progress is still a pretty legit central structure to build apps around and just would be welcomed. CHARLES: When are you going to be done? ALEX: About two or three days. I don't know. I think I'm basically going to be continuing to get feedback like the way that a lot of that original router stuff came back or it's just like constantly hit people with real examples, Ember twiddles, things are just like, "Oh, yeah. That thing. That's a cool pattern. That sucks in my app. I didn't realize that until I saw this example." These things that really teach people why this is necessary because that's going to get people's urge to be like, "Well, you could just do..." Oh, you can't because the thing that's hard to explain. It's going to be a lot of that regardless and I hope that will kick off in the next few weeks. CHARLES: And the focus of that is going to be the ember-rideshare application. ALEX: I think that's a good one. This is one that everyone's familiar with. CHARLES: Have you already kind of implemented in it, like this kind of Frankenstein-ish, like this is the kind of histrionics that you have to go through in order to implement the style of routing or the style of application using today's Ember? Or have you started to begin experimentation with these new concepts and try to build out better ways of doing it? ALEX: I'm not strictly extracting it from one app. It's sort of combined. Like I said, the few different apps that we had were an opportunity to be like, "This sporadic stuff is hard." The main route recognizer approach was an example to try different stack routing pattern. But the thing that sort of working on is drawing from three different apps and slightly different takes on it. Basically, I have something that is close to being testable in one of my main apps that will be a great chance to validate if all the stuff is as nice as I think it is going to be. CHARLES: Okay. If the people want to get in touch with you, to help to contribute to the conversation or just publicly guilt you into moving faster towards it, how would they get in touch with you? ALEX: I'm at @Machty on Twitter and GitHub and also, the Ember community Slack. I think I'm going to try to get people to talk about this on channel called Dev Dx Router where it's focused on development stuff all around the router. This is kind of funny because I'm talking about this thing that I've only had maybe, 12 people take a look at and comment on and begins these conversations. I think maybe some people are going to hear this and be like, "What are you talking about?" but if it gets people -- CHARLES: No, no, no. You know, the best conversations seemed to be organized around you, man. I'm just trying to think of some of the best development conversations that I've had in 2017 and you were definitely, I would say the one who fomented them. It starts with 12 people but then, if enough people take interest and be like, "Wow, yeah. Oh, man. I didn't even know that was a problem. This would be a cool way of doing it." They have a tendency to balloon and some fizzle out and some end up with real results. Anyway, I'm looking forward to it. ALEX: I appreciate it and likewise, you're definitely one of the best people to talk about this stuff with. CHARLES: Well, I hope other people will love listening to our conversation. With that, we'll head on out. Thank you everybody if you've made it this far. As always, you can get in touch with us at @TheFrontside on Twitter or just send an email to Contact@Frontside.io. We will talk to you next week.

Jay Phelps: @_jayphelps | jayphelps.com Show Notes: 01:25 - RxJS 10:09 - Observers 17:49 - Back Pressure 22:11 - Async Iterators and Generators 31:30 - Mapping Resources: The Observer Pattern Hot vs Cold Observables IxJS redux-observable Transcript: CHARLES: Hello everybody and welcome to The Frontside Podcast, Episode #84. My name is Charles Lowell, a developer here at the Frontside and your podcast host-in-training. With me today is Elrick Ryan. Hello, Elrick. ELRICK: Howdy. CHARLES: You and I have actually been on a roll lately, podcasting the hell out of these podcast. ELRICK: Yeah, I know. That is very true. CHARLES: It's been you and me but it's feeling great. It's good to have you on the show again. ELRICK: Yes, wonderful, man. Let's keep it rolling. CHARLES: All right. We will keep it rolling. Today, we are going to be talking about redux-observable and to help us understand and plumb this topic, we have someone who's very qualified to talk about it. Mr Jay Phelps, who in addition to having been the co-creator of redux-observable, also is on the core team of RxJS, which is a fascinating library on which it's based for many years and is currently a senior software engineer at Netflix. Welcome to the podcast, Jay. JAY: Hey! Good morning, everyone. I guess it's not probably morning to the people listening but good morning to you all. Thanks for having me. CHARLES: I'm excited about it. I know that kind of starting with the fundamentals, RxJS is something that was on my radar for a few years and it definitely [inaudible] once we started using redux-observable but the whole concept, I often feel like the world kind of is turned upside down when I'm working with observables, when I'm working with RxJS and I'm curious, how did you come to be a part of that project and what are the things that you use it to solve? Why did the solutions that you generated shake out that way? JAY: Sure. I actually was not introduced to Rx until I started working at Netflix. Netflix does have a fairly solid reputation for their usage of Rx, not just in the JavaScript world but also in the server world. Netflix wrote the original implementation of RxJava and it's used heavily on our backend systems. CHARLES: For some reason, I had this impression and maybe I'm mistaken that Rx originally came out of Microsoft. JAY: Let me continue with the story. It's confusing and I can actually take a step back and clarify that point in particular. Rx itself was originally came from Rx.net, which was indeed created by Microsoft many, many, many, many years ago. I don't know the exact date. I think it was at least 10 years ago. They, at the same time created or about the same time, Matt Podwysocki who was working at Microsoft and still is working at Microsoft, created Rx.net and RxJS. Then many years had passed and originally, it wasn't as popular as it got in the coming years. After several years, some employees from Microsoft ended up coming to Netflix. Jafar Husain is one of those employees. He came from Microsoft to Netflix and he brought that Rx knowledge and that advocacy. Rx is very ingrained in the Netflix culture and is used a lot by various teams for various purposes. Then when I joined Netflix and I got really exposed to it. One of my coworkers at the time, Ben Lesh was asked by several people at Netflix to consider and look into rewriting RxJS. At the time, the version was RxJS 2.0 and while it was great, we had some specific requirements for our website and some of our other applications that we were hoping for a better performance, smaller bundle size and better debugability and -- CHARLES: Also, when I first evaluated it many years ago, it felt very much like a port from another language, in another culture as opposed to something that from the ground up, considered as a JavaScript library. Is that a fair statement? JAY: Yes, somewhat. Definitely, there were more considerations this time around when it was rewritten and originally, it was going to be Version 3 but the rewriting process took quite a while as these things usually do. By the time we got a version out, it was Version 5. We started when RxJS was at Version 2 but it already released Version 3 and Version 4 by the time it released for the new version like a rewrite had been able to get out. When I say a rewrite, I mean like from scratch rewrite. Matt Podwysocki who was the maintainer, almost the sole maintainer of the previous version, also is now on the core team of the new version of RxJS and has been instrumental in pushing back forward as well, he has far more experience with this than either Ben are I so he's been invaluable. Sometimes, we'll think to make those decisions. We'll be like, "Why was this decision made? Was it made because of .net?" and we'll just assume that and we'll want to change it but Matt has the history involved in that. He knows why things were changed the way they were. For example, we changed one of the operators, flatMap to mergeMap. We know somewhat we go at least, I don't want to speak for the entire team but I regret that decision. Depending on the day that I've been talking to Ben, I could convince him to regret that decision as well. But we thought that mergeMap would make more sense and that very few people in practice would have heard of the word flatMap before and had experience with that so -- CHARLES: I have to say both of the terms coming at it were pretty opaque. I think there was a bout of equivalence in opacity. JAY: Yeah, good to know. That's just an example. I don't want to stick too much on that topic. Maybe someday we'll go back to flatMap and the flatMap still exists. If you're a purist, you can use it. Ben was the primary person who was working on this. He wasn't working on it full time but pretty close to full time to get that initial version out. Even though I used it, my involvement with it was fairly low at that point and then my involvement after it was released got increased. I found more time and started to get more involved, particularly there wasn't a lot of code to write. I have some PRs and stuff like that but particularly on the planning and the issue triaging and PRs and stuff like that, which is a pain in the butt. It's just massive. Particularly, around the same time that the rewrite was getting finished that Angular decided, "You know what? We're going to bet on Rx. We're going to depend heavily on it," so you really can't write Angular without writing some Rx these days. You can get away with not knowing Rx very well. You could just call subscribe and then just do a bunch of imperative stuff but for the most part, the paved path is observables in so many fashions. Now, there's this ngrx. I don't know if you have any exposure to the Angular community. I have quite a bit. CHARLES: I haven't recently. Certainly, since that kind of heavy investment in Rx, I haven't been exposed to it. JAY: I think that was your question, right? CHARLES: Yeah. It sounds like there's a Java implementation that gave rise or a .net implementation or Java implementation gave rise to a JavaScript implementation and that's the one that you got involved with but it suggests very strongly that Rx is an idea and it's played out in a bunch of different languages but really, there is a shift or it's an idea about the way you think about your programs. It's clearly been compelling to you so what is that idea and what is that shift from the way we normally think about things? JAY: The idea was realized very early on -- CHARLES: Yeah, both 10 years ago, right. JAY: Yeah, exactly. They dubbed it their reactive extensions, which is what Rx stands for. Pretty much, name a language and it's been ported to that. There's RxSwift, which is super popular. There's even things like RxCpp and stuff which if you look at it, it's awkward. It seems like we got less language in the world for doing this sort of thing. I actually like C++ in a lot of ways but it was awkward stuffing that stuff in there. It's a really popular pattern and the idea is just basically going all in on the observer pattern, saying that like, "Most people are building things in which you want to be pushed information." You want to be pushed events and the data should stream to you. Modeling most problems in the world as a stream, once you get over the initial barrier of getting away from your normal historical way of looking at things and you look at everything as a stream, it comes very natural because you can actually model literally anything as a stream because it could be a stream of one, it could be a stream of nothing, it could be a stream of infinite number of events or it could be a stream of 10. You can model anything as a stream. Once you start thinking about that, it just becomes very natural and particularly on the UI side of things, I think there's been a lot of success in RxJava stuff at Netflix but RxJava is also used in Rx.net for client-side stuff as well, for mobile development. CHARLES: I remember when I first was introduced to it, I think there was a lot of confusion for me around an observer in the context of Rx and an observer in the context of classical MVC. One particular manifestation of the MVC architecture where you have these kind of mutable objects and you're observing their properties. Like key value observation, which factors heavily into certain UI frameworks. Backbone is one that comes to mind or if you're familiar with Java, basically the JavaBeans, like the property model listeners. I kind of had that conception of what an observer was versus Rx has a very, very different take on observable things. Do you think you could maybe show where they're different? JAY: You can get the normal classic observer pattern using Rx, using a subject basically. But there's a subject class, which you're not going to use super often but there are certain cases where it makes sense. Also, it depends on what libraries. It is used more often in the Angular world because you want to get a stream of clicks or something like that but -- CHARLES: So what would be the subject in that case? JAY: The subject in that case would be you're going to pipe, you're going to emit. Every time they click on something, you're going to 'next' something into that subject, like 'next the event' into the subject. Basically, a subject is a really great way to go from some imperative world to the observable world. Without having to write all sorts of custom glue, you can just basically say, "I've got subject. Any time I 'next' into it, just notify anyone who's listening to this." A subject is hot observable and that's the closest to the typical observer pattern because Rx, it's usually like observables and are usually lazy or cold. That's also what people call it. In the normal observer pattern, there is not necessarily any concept of laziness like you listen to something and that producer is already producing usually. CHARLES: Right. You hit on that and I think that was something that was surprising and kind of delightful when I first started using observables is to realize that they were lazy. Let me make sure I understand it. What was cool is like I was going through some of the demos and I had this observable and is part of, forgive my terminology but when you create a new observer, you pass the function that will get called every time something subscribes to it, right? JAY: When you create a new observer it passes a function -- CHARLES: When you create a new observable, you pass a function that gets called when an observer subscribes and the thing that you can pass is the thing that you can call next on, right? JAY: That's exactly right. When it's a lazy observable, that only get called... Actually, you know, continue. You had a point. CHARLES: I was going to say what was a surprising and cool for me was that every single thing that subscribe to that observable got its own history of that observable from the very beginning. It got its own function invocation so the first example I did, I wanted to iterate over an array and just send 10 items to the observer. Then when you subscribe, you're starting from one every single time: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10. It's not like I subscribe and one gets five of the elements, then I subscribe to another one and that one gets the next five or those two get the next five together. It's like they each get their own version of that observable. JAY: That's exactly right and then that's the main difference between -- CHARLES: -- The whole thing, yeah. JAY: Yeah. This is still observer pattern because the observer pattern, at least to my knowledge, I'm not an expert on this or anything but my understanding is that the observer pattern does not necessarily dictate hot versus cold, per se but traditionally I would say, people interpret it as typically the hot type of thing or also the multicast, if you want to be like these cool buzzwords. CHARLES: Right. What you're thinking of it is when you've got this mutable object, you just dive in wherever it is at that moment and you're now observing events from it. JAY: And that's traditionally because the observer pattern was mostly created for that model view controller type of pattern as well. You can think of like a stream of clicks: the users clicking on things where a stream of keyboard events. That is inherently a hot or multicast stream. You can't tell the user, "Start clicking," or, "Start typing. Don't type." You can tell them that but it's futile. The point being is that you can't control the stream. It pushes data to you and it's already pushing, whether you're subscribing or not, the data is flowing so the traditional observer pattern works really well for that. Then the observables, it's usually lazy, Now again, with subjects and with multicasting, you can get the same behavior. You can get that observable that is hot and shares its subscriptions so that when multiple people subscribe to it, they all get the exact same underlying subscription that is already producing data. CHARLES: I see. That makes sense with to call of multicast because fundamentally, there's really only one observer. There's only one stream and you just happened to be entering in at a certain point. JAY: Right. Then the other one is called unicast but not everyone calls it that but that's usually what it's called. CHARLES: There's a lot of terminology. With subjects, is that just a fancy way of saying a hot observable? JAY: Not necessarily but almost always. If you include them together, you're not wrong 99% of the time. But as a quick drive by definition, it's totally fine. If you're teaching someone really all the things and then they want to fundamentally understand it, it's good to have some distinctions. A subject is hot and multicast but just because something is hot and multicast does not mean that it necessarily has a subject backing it. Maybe from the pattern perspective, you could call it 'the subject' or 'a subject' but in the Rx world, it's not necessarily a subject. CHARLES: Then for cold observables, what are good things that they model, like the execution of an algorithm or something? there's an instance of that algorithm, I'm going to add two numbers or I'm going to reduce this tree of numbers into a product or a sum or something like that, where you start from the beginning, there's a clear beginning, there's a clear end --? JAY: Right. The observables are really good at modeling side effects, for example. Things you want to do like make an Ajax call or read a file or something like that. In fact, reading from a file is actually not a great use case for observables just because you want to be able to control the back pressure. We can talk about that next if you want but -- CHARLES: Yeah. Back pressure is a concept. It's crazy to me and when people use it, I'm like, "How do you even do that?" You can say it and sounds good in practice but it sounds really, really complex. JAY: I think I can summarize it for you. The back pressure stuff is basically you have a data producer and you want to be able to control the rate in which it is producing so that you do not overwhelm the consumer. You can consider two servers. You've got Server A and it's sending events to Server B. If Server B can only process 10 events per second but Server A sends 100 events per second, what's going to happen? There's a deficit. There's a huge 90 events per second deficit and that means that the Server B is going to get more and more behind. Eventually, it will just fall over and run out of memory, CPU or whatever happens. Back pressure is basically just being able to control that somehow. There's lots of ways, there's several policies of back pressure. There's buffering, there's dropping and then there's the pull model where it's like, "I'm going to tell you when to give me the next event," or, "I'm going to tell you to give me six more. I can handle six more, give me six more," or, "Here's the next one." Those were -- CHARLES: Then polling would have to be combined with buffering or dropping still, right? JAY: Possibly but it becomes one of those like you use it just as a rarity. Because the problem with buffering is that it becomes usually unbounded and that means you can -- CHARLES: It's blowing up the producer, right? JAY: Well, you risk money out of memory, mostly. It just becomes completely unbounded. You can bound that buffer and then it becomes dropping. Then if it's dropping, that means it's lossy. In a lot of scenarios, it's important not to drop information. CHARLES: They're useful stuff. JAY: Right. At the same time, there are a lot where you can drop it. Like at Netflix, we have this data pipeline thing for these logs and I got a good talk on this actually. This data pipeline is called Mantis job platform where all this data flows through. It uses RxJava almost exclusively and we have a lot of different back pressure mechanisms but the one of the most popular is dropping data. If the consumer is being overwhelmed with the number of logs and events that are matching to it, we just start dropping them because they're logs. We want to keep them but we make a best effort and it's perfectly fine to drop them. In the observable world, it's usually going to be either buffer or drop. You're not going to usually be able to control the producer's rate. In RxJava, there is something called the flowable, which lets you control the rate by basically saying, "Give me N number. I want N number more." It's become more important on the server-side end of things but a normal observables -- CHARLES: That makes sense. JAY: Yeah and there's even a library that's brand new that Matt Podwysocki came out with called IxJS. Instead of RxJS, it's IxJS and it's iterator. It's for async iterator and regular iterators, which is good for that other end of the spectrum for back pressure. If you want to be able to control the producer, you're saying, "Give me the next five." That's what an async iterator is incredibly perfectly suited for. CHARLES: The async iterators are kind of like the logical inverse of the observables? JAY: That's exactly right. It's the dual, if you want to use the computer science term, I assume you knew that already and it was that was just a plant. CHARLES: Actually, I didn't terminology, the dual but it's like through the looking glass world, right? JAY: Yeah, that's correct. CHARLES: One hand you got Alice in our world and she's looking in the looking glass world. I'm not saying which one is which but one is observables and one is the async generators. When you would want to use async generators is when you want to have the consumer driving the entire stream. JAY: That's right. A lot of people, when they discover async iterators, they'll ask me, "Jay, why wouldn't I just always use that? It seems like they're more flexible of the two." The reason why is because you can't always control the producer. The fundamental example that I was giving earlier like mouse events or keyboard events, you cannot control what the user does so an async iterator would be a very poor primitive to model that because what happens if you decide to give me one keyboard event at a time and one per second. You're like, "Give me the next keyboard event." If you did that one per second and someone is typing on their keyboard, what do you do with all the events while you're waiting? You could buffer them but that's a back pressure problem. You have to choose and it's a very poor primitive model to that. CHARLES: Yeah, that would be a terrible user experience. The users want the consequences of their actions to be realized at the soonest possible point. JAY: That's exactly right. The other benefit of observables is a slight increase in performance because when you subscribe to an observable, it sets up the chain and then now only data needs to flow through that chain. Whereas an async iterator, every time you call next, you're going to get a brand new promise. If you used it for something very high volume, you might be able to see how now you've got a lot of excess allocations: CPU cycles being used, garbage collecting for each one of those promises. In Netflix, we got a lot of streams, which are hundreds of thousands of events per second. If you do that, those allocations of those promises can certainly add up. It's just not the most efficient primitives for that as well. CHARLES: I thought that async iterators used essentially continuations under the covers and not promises. JAY: Under the covers, I'm not sure how you would describe what -- CHARLES: I'm not super familiar -- JAY: It's basically an -- CHARLES: You're talking about like yield, right? JAY: You're talking of generators maybe? CHARLES: Uh-uh. JAY: Iterators and generators are although related because a generator is basically a factory for creating iterators. Does that make sense? CHARLES: Yeah. JAY: Every time you call that factory, you get a brand new iterator out of it. But then when you have an async generator, it's basically the exact same thing except for the iterator returns a promise. You're basically saying, "Give me the next thing." You call next on the iterator and the next will return a promise and that promise will resolve with a value. Why is that useful to standardize? Because you can have syntactic sugar like the [inaudible]. I don't know if you've seen that yet but it's like a loop primitive, where you can basically loop over an async iterator without needing to deal with the whole thing and all that stuff. The same way normal async/await is helpful. This await is helpful at the same time. CHARLES: It's way above my pay grade but maybe you're using both. I'm trying to think with async the way that works with promises, right? JAY: That's correct. CHARLES: So you got both the continuation and a promise. That's even more overhead than the promise because you've got to stuff away that whole call stack and the promise and yield to it. But anyhow... ELRICK: This is very interesting. RxJS is a very deep subject. What I'm interested to know is you have RxJS, you have observables and you have redux. Where's the union? How did that come to play that you guys came up and said, "You know what? We need to create redux-observable library?" JAY: Yeah, great question. It came about fairly organically and over many month period of time, actually. We bring Rx people as I mentioned and we were using redux at the time and we're still using redux but my team that I was on was using redux and we were using redux thunk and the thunks were getting incredibly out of hand. It was very hard for us to do things that we were used to doing that were very simple like debouncing. We were like, "This is really hard. It's just to do something so simple that in Rx, it's so simple." We try to stuff Rx into redux thunk and try to make conventions around that. It just didn't work out well and then we initially made something we called thunk-observables -- CHARLES: We really wish you would have stuck with the name even you didn't -- JAY: Yeah, so thunk-observable is probably what you can expect. You dispatch a thunk which returns an observable and that observable will be subscribed to and that was it. Basically, that was the primitive so that let us have our debouncing but in practice, we found that the thunk stuff cause way more confusion and then also did not let us do composition as much as we wanted to. We then extended thunk-observable thing to get receive a stream of actions so that you could compose the different thunk-observable together and then receive new actions to cancel things and stuff like that. We eventually just figured out that having basically the idea of thunk-observables but having of them be like almost static factories that are like process managers, that's the pattern that we ended up on today. That we coined calling them epics. Basically, an epic is a function, which receives as an argument, a stream of all of the actions. It gets an observable of every action that's going to be dispatched and it was a hot observable so that means that if you happen to subscribe to it later, you're not going to get all the actions from before. It doesn't replay them. It's just all actions from the time you subscribe and continuing. Then what that function is expected to return is a new stream of actions and that stream that it returns of actions gets subscribed to by the middleware under the hood and basically, it just calls stored dispatch on anything you emit. You can imagine the simplest epic would be a function that gets streamed of all the actions, it applies a filter operation on all of those and looks for a particular action and then based on that action, it performs some side effect and then when that side effect comes back, it emits a different action. Basically, to notify redux like, "Here's the results. Here's the change. I'm done," or what have you. let you just make arbitrary side effects but isolate them in a way that fits very naturally with your UI layer that you're using and with redux itself with the purity of the reducers. CHARLES: Right. For example doing an Ajax request, I think that's a good example. You get some action in, you're getting every single action that's dispatched to the store, the first thing you want to do is you want to filter that stream to only the actions that are user click the save button so I want to save off this form. To some action, that's my save action so then my job is to now map that eventually. Out the end of that stream, I want to have like the save either succeeded or the save failed. There's two forks on an Ajax request. The fundamental mapping that's happening is either from the user click save, whatever you want to call that action, I want to map that to the user if the save was successful or I want to map that to the save failed or rejected. The hard part then is executing those side effects and then putting them back into the stream. How would you do that then with redux-observable? JAY: If I'm following correctly, the typical almost all epics, not all of them but almost all, they're going to have a filter at the top of the epic, they're going to filter out looking for the action they want and then they'll have some sort of merging strategy operator like mergeMap, aka flatMap or they have something like switchMap. CHARLES: I feel like that's one of the harder concepts for me and I want to actually going to be a little selfish here and try and bounce my understanding off of a bit so you can be like, "No, your mental model is incorrect," but we'll find out how it's incorrect. With the mergeMap, is that a way of saying, "I can take other observable streams and inline them inside this one stream." That's the way I've been thinking about it. I can go out to the rest of the world, I can have some other stream and I can inserted into the processing chain of this other stream. In this case, I do a mergeMap of the actual fetch or the actual post but it's not a straight chain of implication like do a filter then I do a map then I do this. I've got to take this other observable, this thing, this other asynchronous process, which really represents another stream and I want to merge it into this one stream. Then once that's done, I get the result. now, once that happens outside the mergeMap, now the Ajax results whether it's a success or failure, the result of the request, now it becomes the next item in the stream, which then I have to map down to another redux action. JAY: That's right. The mergeMap operator has an alias and it also used to just be called flatMap so you can envision that you are creating an observable of observables, a higher order of observable. With mergeMap, you want to flatten that. You're saying that every time I get an event, I'm going to call this projection function and it's going to return a new observable, an inner observable and I want to merge each one of those into each other so that it becomes one stream. That means you can have concurrency. That means you can have multiple and simultaneous concurrent Ajax calls that can finish it arbitrary times. I think it's important then to contrast that with some of the other merging strategies like switchMaps or concatMap. With switchMap, as the name implies, you switch between observables so at any point in time, you can only have one observable subscribes to at a time. If a new event comes along and you call the projection function, it returns a new observable. If the previous observable has not yet completed, it will get unsubscribed to and anything it was doing gets cancelled. In the example you're talking about, if you've got an epic that goes and fetches your user model every time they click this button, let's say that they can click that button multiple times and you don't want to make 50,000 requests. The quintessential example is actually the auto suggest stuff. As you're typing key strokes, if the request is in processing and they type another keystroke, you don't want to wait for them the previous request to come back and process it. It's not only wasteful because you'll process the JSON and all that stuff. It can introduce bugs because it may come back out of order. It may come back actually after your new one comes back and that can cause all sorts of crazy weird bugs. That's what switchMap is really great for. I call it implicit cancellation. Ben doesn't like that because he's saying that in a switchMap, you are being explicit about it but you're not calling unsubscribe on it yourself, which is why I call it implicit. It happens automatically. There will never a new event pipes through there. Then there's the third one which I don't -- CHARLES: So this switchMap ever make sense outside the context of hot observables? JAY: I would say that it makes more sense usually on hot observables but there are certainly cases like let's say that you've got a web socket observable and every time you get an event, based on that event, it make some other request and that other request may or may not take a really long time. But if you get another thing back from the web socket, you want to cancel the previous one. That's somewhat not a great example as well because sometimes people use multiplexing for the web socket so that it becomes multicast. But the point being is that there are definitely times where you will use merge, switch or concat. That concatMap being the third one where as you might imagine, you are concating the observables, you line them up. If I get a new event, I'm going to call my projection function, create that new observable but I'm not going to subscribe to it. I'm going to keep it but I'm not going to subscribe to it and tell the previous observable I was subscribing to has completed. In a way, you're buffering the observables and because you're offering them, you can get in trouble where you end up buffering infinite observables. Let's say the first one never completes for some reason because of a bug, you may infinitely buffer them. ConcatMap is not used that often. I'd say it's very rarely used. Its use mainly when you don't want lossy behavior. You want, at most wants semantics like you're only doing one per time. CHARLES: The difference between mergeMap and concatMap is what? JAY: It's you're not going to do concurrent. That's probably the best way to explain it. You're going to do that in sequence instead of concurrently. CHARLES: I see. Maybe an example would be like file uploads or something. You probably want to do your file uploads in parallel but let's say, you want to conserve bandwidth or you're working where you've got a browser that only supports 10 connections or five upload connections but when someone selects 30 files, you don't want to just drop those files. You want to be uploading 10 concurrently but as soon as one finishes off of that 10, you want to start up another one. JAY: That's absolutely right. Another example would be like you are going to hit some API and it could be used as one mechanism to kind of throttle yourself. If you want to guarantee that you're only connecting at most once to this particular end point at a time, no matter what the upstream tells you to do. CHARLES: Right. I could see that. That makes sense. JAY: I can tell that you're a little bit struggling to see the use cases, which is totally normal. ConcatMap is not used very often. It's for of those fundamental operations, which primarily why it's included is it's non-trivial to implement yourself. CHARLES: Are there any other mapping operators that we should know about? JAY: Those are the three primary ones. There's a couple of other like boutique variations but they're all variations on the exact same thing. CHARLES: Right. ELRICK: Do we have like mergeMap, switchMap and concatMap that are specific to management of observable and that's specific to redux-observable. There's people out there using something like redux-saga or they're trying to compare and contrast these two libraries. How do these things contrasts? Because I don't think you would use either of these mapping operations inside of a saga. Is that correct? JAY: I just want to make a minor clarification. The Rx stuff and these operators that we've been talking about: mergeMaps, switchMap and concatMap, they are not actually related to redux-observable really in any way. The only thing they're related is that you just might happen to use them. Redux-observable is actually a very tiny library and it defers pretty much everything to just normal idiomatic RxJS code and that's really the biggest pro in comparing it to say redux-saga. Redux-saga came significantly over a year before redux-observable and without a doubt, were influenced. The pattern is very, very, very similar but there are some just fundamental differences and one of those is that difference. The most obvious thing is if you already know RxJS very well, without a doubt you're going to want to use redux-observable. I could tell you that, it's not possible but it's very unlikely you'd choose redux-saga over redux-observable if you're already a really big Rx fan because you basically know how to do it. You just have to think that the items you're modeling, the events you're modeling are actions, which just happens to be a convention really. There are some other differences though between redux-observable and redux-saga and the biggest difference being that redux-saga takes this effects as data approach, which is like Haskell if you're familiar with that. In a lot of ways, it's identical with the IO monad but it basically just means that you're not actually performing the side effects yourself right then and there when you call their operators. There are special utility functions, instead there's like an engine or a middleware underneath that performs the side effects on your behalf. You create a generator and that generator is the pattern that is called a saga is what they call it. That generator yields these objects, these effects objects. One of the effects objects might be to call some particular API to make an Ajax request so you're going to yield that object that basically describes the side effect you want to perform but doesn't actually do it itself. It's just an object. In the middleware will form it for you so why would you put that indirection. The indirection helps when you want to do things like testing. If you want to test it, you don't actually need to perform any of the side effects. You can just call next on the iterator that you get back from the generator, you call next on it, you will get the side effect object, the effect as data that you want to perform and you can just assert based on that. You don't need to do mocking in all of that stuff. You just have to assert that the effects that were yielded were the ones that you expected. Now, I don't like that approach personally because I actually use redux-saga quite a bit, not lately. It was like a year ago. You end up implementing almost all, not all but a lot of your business logic in your actual tests themselves. Your test become less about testing the behavior or the outcome of a particular thing and more about testing how it gets to that outcome and what steps it takes to get there. Some people think that's fine. For me personally, it felt like any time I made a change in my saga, I had to make the exact same change in my test even though the behavior of the saga did not change. The actual observable outside side effects did not change in any way but I may be refactored or renamed something. It felt very redundant and to me, felt brittle because I started to wonder who tests the test then. If the tests are in a lot of ways are reimplementation of the saga, how did I really test that the behavior of what I was trying to accomplish really was accomplished. I'm not an expert on it so certainly, I'm sure there are people who have patterns around this that can mitigate some of my concerns. But for me, I'm used to testing Rx and I'm used to Rx in particular so the pattern for me of using observables just made a lot more sense to me. Also, with either of these libraries, the learning curve is really steep. If you don't know redux-observable, let's say you don't know Rx, learning Rx just to use redux-observable is a pretty huge undertaking. I would usually recommend people to not do that. A lot of times these days is spent me helping people who are frustrated because they dug themselves in a hole by using all these new technologies. They'll pull in TypeScript and [inaudible] with every all of the cool new hotness and I don't blame them because they've been told that this is all the cool stuff. CHARLES: We got some great advice on that that you can have one vanity technology on every project and live it yourself. Indulge in that coolness, in that hotness and do something exciting but make sure you have one vanity technology and one only. Everything else, be very comfortable with and one is just crazy experimental like this is the coolest new thing and we're going to just drop it in. But if the rest of your chassis is solid, then you can support that crazy experimental engine that requires you to stand up on the front of it and feed gas in with your mouth like they did in Mad Max. JAY: And there are exceptions obviously but for the most part, I subscribe to that. I answer a lot of questions on Stack Overflow and redux-observable. I actually get sad and frustrated when I see people that it's very clear that they shouldn't be using this. Their use case is not complex and they haven't learned Rx yet. It's one thing if you're like, "I'm learning this and I know that I'm going to have pain." That's totally acceptable. There's plenty of times where it's like, "It's totally okay that there is no concrete deadline and I can take my time," but a lot of times and I would say, a majority of the time, you're slipping on the deadline and that's not a good thing. The people who are coming to me are just frustrated. They're like, "Why is this so complicated?" You're right, it is really complicated but its complication helps with some of the more complicated use cases. That's the irony behind both redux-saga and redux-observable is that they're both really complex for the trivial use cases like all I want to do is make an Ajax call. That's it. I don't want to cancel it, debounce it, I won't do anything. I just want to make an Ajax call. They are the biggest hammer you could possibly think of. Don't use them for that. CHARLES: The thing is if you're not sure, whether you need redux-observable, then you actually don't need redux-observable. JAY: I would say that usually that is the case, yes. The same with redux-saga. They're in the exact same league. It's basically, "Do you need complex side effect management?" With the redux-saga, the complications, even though generators have been around for a little while, a majority of people still are not familiar with them because they're just not used all that often. They're kind of mystifying. I would say, they're not super hard to learn. They're just alien. Then to add on top of that, the effect is data and you've got multiple curveballs. Same thing with redux-observable, they could never use observables. It's pretty alien. The reactive programming idea and model is pretty alien. I would advise, if you're not using redux, don't even consider redux-saga and redux-observable personally. If you're already are using redux and you think that you might need something like this, experiment with the primitives, try and see how well your team can pick up RxJS just by itself. Just learn some of the regular RxJS tutorials, don't even look at the redux-observable docs because it's not useful in any way when it comes to this. Just try and learn a little bit Rx and see does it click. Because some people is like, "I don't understand why everyone thinks it's complicated. It's easy." But a majority of people, it takes a while before they get that like Neo in The Matrix, I-know-kung-fu moment. CHARLES: All right. We're almost at time but I hope that that moment comes to everybody. I think we've certainly enjoyed a lot of success with it already and I think once you do get your head around the basic use cases and you know how to do an Ajax request, you know how to do just simple saves and gets and what have you, doing the trivial things becomes easy because you know which pathways to travel. But before we head out, is there anything that you've got coming up in the near future, any talks, appearances, meet ups, anything whether you or otherwise that people should be aware of? JAY: I would say on this topic, there's a new beta for RxJS with this new way of using operators. It's being dubbed lettable operators, like the 'let' keyword but it's not related to that. It's basically just a way of finally being able to import operators and to use normal tree shaking that people have asked for forever. Because the problem with observables is that they're prototype-based methods and you can't reliably tree-shake methods or prototypes. We've been trying to experiment with ways to have [inaudible] and the lettable operator stuff is interesting to check out. It's a stopgap measure until JavaScript has something like the pipeline operator, which just actually moved to stage one. It's a brand new operator. If you're familiar with a lot of functional programming stuff like F# and a lot of the functional programming language is actually have the pipeline operator, it'll make it so that you can basically have syntactic sugar to apply a function, basically to pipe the result of a function into the first argument of the next function, etcetera. You can pipe the argument and return values through a series of functions. If you do that without this syntactic sugar, you've got this massive nested function invocation, which is incredibly hard to read and hard to maintain. That's why the pipeline operator is so great. I would encourage people, that's in beta right now. I think it's 5.5 but it's in beta right now. I encourage people to check it out, find bugs and get feedback. Maybe this is completely off base and it's not the right direction that the team should be going but it's based on a lot of collaboration, particularly with the Angular community. They've been, in particular asking for this because they're pretty big and they've got to use the Clojure compiler and all sorts of things for trying to make their bundles incredibly small. For me personally, I don't have any Rx talks coming up. I've been pretty obsessed with web assembly here lately. I'm an armchair compiler nerd. I don't do compilers for a living but I have done them personally for a number of years so I'm obsessive with web assembly. I have number of talks in web assembly coming up but just nothing related to Rx at this point. CHARLES: That's totally okay. I'm actually, also have been obsessed with web assembly. JAY: Have you guys done a podcast on that yet? CHARLES: I don't know. ELRICK: No, not yet. CHARLES: I actually started out to write my own list compiler in web assembly and they got totally derailed on the list compilers. Actually I ended up switching tracks on the whole web assembly thing but I was really, really excited about it. Probably, it was about three months ago or something like that but I'm still excited about it. I just haven't been working on it actively so I'm very curious to hear about those talks. Let's post them on the show notes and who knows? We do a lunch and learn every Friday here and usually, it's one of us getting up there but sometimes, we'll just watch a talk. One of us has been wanting to watch. ELRICK: And you're always welcome to come back to any time and geek out with some web assembly. CHARLES: I'd say, we haven't podcast web assembly, you know? All right, you guys, we've been at this for another hour. Let's go. Everybody listening, strap your headphones on, we're going down for another hour. Changing the subjects: web assembly. It starts right now. ELRICK: It's going down. CHARLES: No, but we will have to have you on for that. Thank you so much for coming on and talking with us about observables, Rx, redux-observable but if folks want to continue the conversation with you, they can get in contact with you how? On Twitter, email? JAY: The best way is going to be Twitter probably. I'm at @_JayPhelps. Thank you guys very much for having me on today. It was a blast. I love talking about this stuff. ELRICK: Thank you. CHARLES: Thank you and for anybody out there, we can also be reached at @TheFrontside on Twitter or just drop us a line and Contact@Frontside.io and have a great week and we'll see you next.

Julie Moronuki: @argumatronic | argumatronic.com Show Notes: 00:57 - Julie's Unique Origin Story Into Programming 03:47 - Good Resources vs Bad Resources for Learning Haskell 11:18 - Areas to Look at Before Taking on Haskell and Functional Programming 15:56 - Terminology 17:50 - The Haskell Pyramid 25:51 - Learning Haskell Vocabulary 28:20 - Monoid and Functor 42:06 - Advice for Someone Who May Not Be Interested in Programming Resources: Haskell Programming From First Principles (Haskell Book) Natural Language Processing (NLP) Learn You a Haskell for Great Good! Programming in Haskell by Graham Hutton Haskell: The Craft of Functional Programming by Simon Thompson Real World Haskell by Bryan O'Sullivan, John Goerzen, and Don Stewart Introduction to Functional Programming Course with Eric Meijer The Joy of Haskell Haskell eXchange 2017 - A Monoid For All Seasons Transcript: CHARLES: Hello everybody and welcome to The Frontside Podcast, Episode 83. My name is Charles Lowell, a developer here at the Frontside and your podcast host-in-training. With me today on the podcast is Elrick also. Hello Elrick. ELRICK: Hello. How you doing? CHARLES: I'm doing well. I'm glad to have you on this one. I'm glad to be doing this podcast in general. We have someone on the podcast today who I've been following for, I guess probably about two years because she published a book that has been very, very helpful to me. It's one that I recommend to a lot of people. It is learning Haskell from first principles. With us on the show is Julie Moronuki, who is co-author of that book. Thank you so much, Julie for coming. JULIE: Yes, hi! Happy to be here. It's nice to finally get to talk to you. CHARLES: Yeah. One of the reasons I wanted to have you on the podcast was because I feel as though you have one of the most unique origin stories because of programming and entering in the tech world. Most of us are curious, we either come from video games or maybe we just start fiddling with the web browser. You enter the maze from the entrance that is like hidden from all, I would say. You went straight to writing a book on Haskell, is that --? JULIE: That is what happened. In 2014 on Twitter, I met my co-author, Chris Allen and he has been trying to figure out better ways to teach people Haskell because the on-ramping, I guess of people to Haskell can be quite difficult. The materials that exist are not always accessible and people felt like they need the advanced math degrees before they can write Haskell. He was trying to figure out better ways to introduce people to it. Since I was this person who's never programmed before -- I have no background -- and then he thought, "This will be a very different experience, trying to teach Haskell to her." Because I have a linguistics background and stuff he thought, "That would be interesting too and maybe, she'd be interested eventually in doing NLP." I said, I'm not -- CHARLES: What's that? Acronym alert. JULIE: Oh, yeah. Sorry. Natural Language Processing. I said, "You know, I've never done any programming and I don't play video games and I never have had any desire to learn computer programming. I don't think I'm going to like this. I don't think this is going to last but sure, I will try," and so I did a little bit. I read a little bit of 'Learn You a Haskell for Great Good.' I've read some other things. CHARLES: This was before you guys had the idea of actually writing a book. JULIE: Yes. He had the idea of turning some of his thoughts about teaching Haskell into a book and as he would explain things to me, like the questions I had about 'Learn You a Haskell,' I'd be like, "We should write this down," and he would say, "It's so hard to write it though. It's easy when I'm explaining it to you and it's so hard to write it." Initially, it started that I was helping him at things that he was teaching me and then as we got further into the book and I started reading a lot of other Haskell stuff on my own and figuring stuff out, I was writing more and more of it. Then we were kind of equal co-authors after not too long. That's how it happened. I really didn't think that I would stick with Haskell or with programming. I'm still sometimes I'm not sure about programming. I'm not sure about this whole making software thing. But Haskell is so interesting to me that I'm still here. CHARLES: That is fantastic and it's a great story. I'm curious, when you were doing the proto-research to learning Haskell, coming from really truly first principles and having no experience of programming, what made a good resource versus a bad resource? What are the things that you gravitated towards and say, "This is really instructive." What was the tone there? JULIE: One of the major problems ahead of most of the Haskell resources that exist is they assume that you've done programming before because nobody learns Haskell as a first language so they all assume that you have done some programming before. They would make references to things that if you were a programmer, you would know what they meant but I didn't. That was one of the hardest things for me. Even 'Learn You a Haskell' does that to some extent. CHARLES: What's an example of that? JULIE: I had learned a little bit about recursion from linguistics because that's a thing in human language so I really understood recursion but most of the Haskell resources explain it to you primarily in terms of, "This will be like your loops in other languages." I'm going to be like, "I don't know what a loop is. This isn't helpful for me." There are a lot of things that I didn't understand so when people talk about Haskell as being a pure functional language, neither pure nor functional necessarily, I didn't have anything to contrast them with so they didn't necessarily make sense to me as things that make Haskell different from other languages. I didn't know what imperative programming was and people would say, "In contrast to imperative programming, functional programming does this," and I'd be like, "Okay, but I don't understand what the imperative programming way is so this contrast isn't making any sense to me and same thing with purity." There were a lot of things I had to learn, in fact about mutable state because I didn't know anything about it. I had some understanding of how computer memory works but still some of the ways that people talk about it were not obvious to me. CHARLES: Do you find that seeking out that contrast actually wasn't helpful? Is it noise since at least at the beginning, it's something you'll never do. It's like saying, "Over in France, they wear these kind of socks." Since I'm going out into the street in front of my house, I don't really care. JULIE: Right. In the beginning, it was a lot of noise and I understand why they do that because they are making the assumption that everybody who is learning Haskell has come from some other programming language, probably an imperative one so I understand why that happens but in the beginning, it was very much noise for me. I noticed a lot of Haskell resources, one of the first things they tell you is that in Haskell you can't do 'x = x + 1'. I was like, "If I'm reading this like it's mathematics, why would I think I could do that." If you come from a different programming language, you might well think that you can do that but in Haskell, we can't so making that contrast, when I didn't have that background was really just confusing for me. Now, because I teach people and most of them do have some background in an imperative language, understanding the contrast is more helpful to me but in the beginning it was just confusing and noise. When we wrote Haskell book, we tried not to make those kinds of references and like, "Let's assume that everybody is just like Julie, doesn't know a different programming language that we can contrast it with and let's try to write a book like that." CHARLES: Right. I think that's a key insight because some people would say there's a lot missing or that difference might stand out. Now, that you pointed out, I can see it but I don't think I noticed it while I was reading it. But one of the things that I like is because I also tried to learn Haskell through 'Learn You a Haskell,' and I didn't find it very helpful. I found it entertaining and it's not a knock against the authors. Some of the sketches were really cute but it was still more explaining... I don't know. It was explaining more of the how, than the why, if that makes any sense where I felt as though in your book, there were a lot more analogies to actual human experiences, using the visceral language saying, "A mono is something you can mash together or squeezed together." That really connected for me. Whereas, explaining it in terms of concatenation and laws and stuff like that. Those things seem cited to the secondary resources to the primary resource. JULIE: Yeah. I think that's kind of helpful for me too. There are different Haskell books that have, I think different things about them that are good. I forget the name of the book but Graham Hutton's book, the way he talks about recursion was really helpful to me. The way he explains recursion and of course, folds but folds are things that he's known for so those parts of that book are helpful for me. But really the best book other than my own of course, for me is Simon Thompson's. I think it's called ‘The Craft of Functional Programming' and I think it does better at explaining things just in terms of Haskell. Real World Haskell, I guess is really good. It was harder for me because I hadn't been a programmer before. I think it's got so many practical exercises that -- CHARLES: Was that the O'Reilly book by Irish gentleman whose name eludes me? JULIE: Yes, Brian O'Sullivan. It makes more sense to me now but there were things in it that are sort of programmer things. Because I'd never made software before, that were really confusing for me. But Simon Thompson's, because his book does have exercises and they were ones that I could understand and do. They were fairly self-contained. My first experience actually in writing a program that does IO was from his book and I was just so thrilled. I was like, "I got it. I did it." That was really helpful book for me but I don't see people recommend that one as often but that was probably the best one for me. CHARLES: Yeah, it's always a balance because the Real World Haskell didn't really worked for me, almost because the examples were too pragmatic or too complex and I picked this up when I was 10 years into my programming career and I struggled to follow the JSON parser example, which is parsing JSON is something that I've actually done several times in multiple languages and I still struggled with it. JULIE: Whereas for me, I don't even know what JSON is. This is not something I've ever dealt with. I know what it is now sort of, but it's still not something that I deal with very much. I was just like, "What is this? I don't even know what to do here." It wasn't quite as helpful for me. I've heard a lot of people have success with that one but I think they don't share quite the same richness of programming experience with Brian O'Sullivan. I think it's a little bit more difficult. ELRICK: These are a lot of amazing resources that I wish I knew about when I try to learn Haskell. I took an online course with, I think it's like Eric Meijer and that class was very intense. Looking back, what would you say are some areas that someone should, either start to look into before they step into the Haskell world, being that you didn't come from a programming background but connecting to dots backwards now? What would you say are some areas that people can slowly ramp up into to get into Haskell and functional programming? JULIE: When I teach people Haskell, the people who have the easiest time are people who have been writing Scala for a while and they've moved over to the FP in Scala side. When I first started Haskell, I heard a lot of people make jokes about how Scala is a gateway drug to Haskell. I think there's actually might be so truth in that because I certainly have a lot of students that were Java programmers, then they got interested in Scala because maybe Scala is better for some things than Java and then they start moving more and more over to the FP in Scala side. Those are probably the students that have, I think the easiest time making the transition to Haskell that I've had anyway. But you know, I think even JavaScript, trying to write in a more functional style and there are some resources for that and really, there's a very good tutorial about monads that uses all the code examples in JavaScript. I think a lot of the concepts that you can start to approach them from other languages. Haskell is still going to be weird in a lot of ways and another thing that works for a lot of people is going to Elm. Elm is similar to Haskell but different. I think that that has worked also for a lot of people getting them into understanding more functional programming concepts but with the much easier... The word easy is so -- ELRICK: It's like a relative term like, "Oh, this is easy." JULIE: It is. CHARLES: Easy to say, right. ELRICK: That's what I thought when I step into learning Haskell and functional programming. I was like, "How bad could it be?" JULIE: Right. Learning Haskell can be very bad. I'm not going to kid around about that. It's a shame because I don't think that it needs to be that bad but the way it's presented oftentimes, for various reasons, I think why Haskell gets presented the way it does but I don't think it needs to have it like that. The designer of Elm, whose name I'm not going to try to pronounce because I don't know how you say his last name, he really made an effort to for example, the error messages in Haskell can be very intimidating. The situation there has improved since I started learning Haskell but they can be quite intimidating and he really made an effort to make very friendly error messages, very helpful error messages. I think that it shows and then it makes a difference for people who are learning. If you start with Elm and then you do want to see what Haskell or PureScript, which is also frontend language, mostly. It compose of JavaScript but it's very Haskell-like, then from Elm, let's see if we can get a little more hardcore Haskell. I think the transition to Haskell or PureScript is easier from there. I think it does help to move in the functional direction from whatever language you're in, if you do FP in Scala or try moving to more functional JavaScript or even Elm. Then Haskell will make it more sense from there or be a little easier to approach. CHARLES: Yeah, and I definitely think that for, at least from my perspective, I've been able to take a lot of those concepts that I've learned from Haskell and then apply them, even inside Vanilla JavaScript. There are things that have become indispensable like mapping and folding and they exist in JavaScript. You can reduce arrays, which is a similar to a fold and then you can map arrays but understanding that map, the key insight for me that I got from learning Haskell is that there's a whole class of values that you can map, not just arrays. The standard JavaScript object is essentially a Functor and will get a little bit to that because for people listening what that word even means and the meta around the fact that they're all these weird words and how do I go about something I want to ask you about. But the trees can be mapped and the objects can be mapped and all of the sudden, it's like this one concept that I use so much for lists, it's available on all these different data structures and it's get me thinking like, "What other data structures can I use this operation? What are the things are Functors that I'm working with?" Really, it's changed my perspective to think about the type of the data structure, in terms of the operations. JULIE: I'm in favor of keeping the terminology that we have but just explaining it much better. That's the approach that I take but it can be very hard, especially it was your first learning Haskell. I don't know if you've seen the Haskell pyramid but to get sort of productive where you can write programs in Haskell is not a very high bar. It feels like it is when you first start but it's not really very high bar but Haskell just keeps growing and growing and getting deeper and deeper so you're always approaching new libraries that you've never seen before and you feel then you've been learning Haskell all over again because they're written in a very different style of Haskell or they have even more terminology, even more kinds of Functors that you've never heard of before or something like that so you're always approaching these things over again. It can be a very intimidating feeling and it makes a lot of people very uncomfortable and I'd say, if you like Haskell and that does make you feel uncomfortable, then you don't actually need to do that because a lot of people write Haskell very happily every day in their jobs even and don't do that. They don't mess with some of the newer, super cool libraries that have all this funky terminology and stuff. Some of them don't mess with them at all. CHARLES: But certainly, there is some concepts that are core. I'm thinking of like applicative and Functor and all these things that I'm learning about and I'm curious to hear about your experience as you climb that pyramid. What is the pyramid entailed? First of all, I'd love to hear more about it because this is actually the first time I heard about the Haskell pyramid. JULIE: Say you understand monads, then you can write really a lot of Haskell programs. Probably at some point, you will need to understand monads transformers but if you just get to the point where you understand monads pretty decently, you can write a lot of software so after that, then learning more is maybe going to improve your Haskell, maybe let you write some things that you couldn't write before but a lot of it above, not that these things are necessarily in an hierarchical progression. We cover monad transformers in a fair bit of detail in Haskell book but if you get anything beyond what's in Haskell book, one of those things that some of them are very interesting, some of them can make you much more productive but some of them are also people do them for fun to explore the space and some people love them and some people hate them. Haskell lets you do a lot of things for fun and exploring mathematics in ways that are interesting and exciting and may influence and in fact, have influenced other languages like [inaudible] in PureScript but not really necessary for basic Haskell programming. A nice thing happened while we're writing Haskell book. I was writing, I think it's chapter six, which is about type classes. I was writing that chapter and at the same time, my co-author had started writing the Monoid chapter. The type classes chapter comes in chapter six and we introduce a lot of the basic type classes: num and eq and some of those in that chapter because I do think it's important. Type classes are very special thing about Haskell so I think it's important to, at least start coming to groups at them early. Some people disagree with me about that and think they can ignore them for much longer. But at any rate, it is where it is and I felt that that was important. Maybe the real motivation for type classes, really until we started writing the Monoid chapter so he started writing that while I was working on type classes chapter and he sent me the beginnings of the Monoid chapter to look at. At first I thought, "We've got addition and multiplication and list concatenation and this just doesn't seem interesting. What is this generalization of a Monoid that I'm supposed to get from these three things? And why bother making it a type class," because additional and multiplication are already in the num type class and then list concatenation is just for list so why make this into a type class and what's that motivation there. With eq, we want a quality -- CHARLES: Is that how you pronounce 'eq?' JULIE: That's how I pronounce it because 'equal' or equality. CHARLES: Okay, so this is a type class for doing what? Making sure to being able to compare two values on the same value. JULIE: Yes and it's a weird one because for most data types, you can have an eq instance and you want probably, in a lot of cases to have that but we don't want because function is a data type in Haskell so you don't want to have an eq instance for functions and that's why equality is not implemented generally for everything. That's why it's a type class so there's no instance for functions because that's not decidable. You can't decide if two functions are equal, generally. Some functions you can but in the general terms, for datatype, you can't. CHARLES: That's actually a pretty profound statement. Proof of which is left as an exercise for the listener. JULIE: We got to the Monoid and I was like, "What is the [inaudible]," or something. It turns out that there are Monoids everywhere. There's all kinds of things that you want to, either concatenate or make a product of. Then having this as a type class and thinking of it in terms of like, "We've got this abstraction. We've got this category. We've got this algebraic structure. Now, we can look for in all these other places," because once you've named the thing, then you can talk about it and think about it in a little bit of the different way. It's like, "Now, we've got this group of addition, multiplication, list concatenation." Now, we've got an abstraction of that and we can think, "Where else can I see this pattern?" and it turns out it's all over the place. For me, that was one of our thought like, "Type classes are actually really cool and powerful and interesting thing." For me, that was when it seemed like, "The terminology is worth it because, now I want to think about finding these algebraic structures and in all these other places." CHARLES: Right and like a Monoid, it could essentially be called, if you're using a Java interface, like 'mashable togetherable' or 'concatenatable' or something like that. But there's a kind of one-to-one correspondence but it is a vocabulary that just needs to be learned. JULIE: I don't know much about category theory or anything but the other cool thing about Monoid for me was that there are almost always two because there's almost always one that's destructive or additive or concatenative and there's almost always one that is conjunctive or a product or multiplicative. It's often across product that would be the zipless Monoid that exist in base and it's a cross product of the two lists. There's almost always two, whereas when you think of Monoids in the very abstract looking category theory, it doesn't matter if it's addition or multiplication. The operation doesn't matter, whether it's addition or multiplication or concatenation or cross product because you generalize the actual operation to the extent where what it's going to produce. It doesn't matter anymore. For me, I still think of Monoids in terms of like set theory or Boolean Algebra, then that's one of the things that I think is difficult with Haskell where people talk about Monoids in terms of category theory but I think that's not very helpful for the actual programmer who has to actually deal with the two different instances like sum and products or concatenation and zipping are going to actually act different in a program. CHARLES: Right, they're going to yield a different set of values. JULIE: Yes. CHARLES: Is there a baseline vocabulary? I kind of think of it like learning a new language, right? JULIE: Yep. CHARLES: When you're learning Haskell, you're not just learning a new language. You're literally learning a new language. I could go and I could learn Japanese but it's going to be a struggle at some point. People say certain languages are hard and certain language are easy. I don't generally subscribe to that. I think that most of it is just going about and living in a place where they speak this language and you'll absorb it and it's the decision to go and live there -- that's kind of the primary one. But let's say, you're a foreigner and you're travelling to this country called Haskell that's got this strange language. Like other human languages, it's just got different names associated with different concepts and some of the concepts might even just be unique to that country. Just like when you're travelling and acquiring a human language, there's a certain level of vocabulary that you need to achieve before you can do things like buy groceries and be able to transact financial exchanges or have a conversation about the weather. What are the kind of the levels of vocabulary that you need to acquire to be operational in Haskell or I would say, even in functional programming because now that I've been exposed to this, I see it in Clojure. I actually see people doing this JavaScript and in Erlang, in Elixir and what have you. JULIE: Yeah, I don't really know how to answer this question. How to buy groceries in Haskell? CHARLES: Let me let scale that down because I had this horrible tendency to spend five minutes asking what I say is going to be single question but it's actually like 30. Let's take down the scope. When you were learning this vocabulary, at what point did you feel like you're really gaining traction? We're you really starting to connect the dots? JULIE: For me, I think when I got through Functor. It was when I felt like -- CHARLES: Functor and what comes before Functor? JULIE: Monoid. I think once you understand Monoid and Functor, then a lot of other concepts in Haskell will start falling into place because this is not obvious to everyone but I think once you really understand Monoid and once you really understand Functor, then applicatives are monoidal Functors and that's not obvious to everyone. Like I said, it's not obvious at first certainly, and monads have characteristics of both Monoid and Functor as well. Then you start saying, "There's all these other Functors. There's profunctors and bifunctors. I think once you really understand Monoid and Functor, a lot of the rest of Haskell starts falling into place and then type classes like alternative. Alternative is another kind of Monoid. We have all these other names that if you can see the general pattern of Monoids and Functors, I think to me anyway, a lot of it then just started falling into place. Applicatives to me seemed, I don't want to say obvious or simple but in traverse, it's same sort of thing so we have these other names for it -- traversable -- and I was like, "Why was it called traverse. I don't understand this word at all." But once I saw the type signature and what actually happens with what the function traverse does, I was like, "Okay, I see what's happening here." For me, those were the two big hills. Once I got through Monoid and Functor and really understood them well, then a lot of other stuff just come and fell into place for me. ELRICK: This is really interesting. How was a Monoid explained to you when you were first starting to learn Haskell? Then now, how do you explain what a Monoid is to someone that's learning Haskell? JULIE: When Monoid was first explained to me, it was the pattern of there's addition and multiplication and list concatenation so it generalize out that pattern and that was really hard for me to understand at first because list concatenation and addition are similar but multiplication is different. I was like, "What do these three things have in common?" What they have in common is that they take two values of a certain type and return another value of that type and that's the type signature of the main function, that's in the Monoid type class. But that doesn't really tell you very much. A lot of functions could do that, in theory at least. How you combine them is really what's interesting about Monoid and also what makes concatenation and addition different from multiplication. Fortunately in college, I had had a fair bit of exposure to Boolean Algebra so figuring out that like, "There's actually two basic genres or varieties of Monoid and they are disjunctive or additive or they are conjunctive or multiplicative," and figuring that out, to me I always think that Monoid should really be, maybe two different type classes, one for the additive Monoid like list concatenation and addition and things where you are adding two things like a set union. Then conjunctive, which would be this intersections or multiplication or cross products. I always think there's maybe should be two different type classes but there's not a good way to do that really in Haskell. Instead, we have this one type class and then we do this ugly business of wrapping them in different type names. CHARLES: Is that why you'll have a constructor for some so it's just a wrapper for an integer? JULIE: Yeah. CHARLES: I don't know if that's so bad. JULIE: I don't like it but -- CHARLES: Yeah. You know what? You do a lot more than I do so I'm going to take your word for it. JULIE: Yeah, that's exactly why. Sum and product are the wrappers for integers because integer doesn't have a Monoid. It has two Monoids over it. CHARLES: I see. There's lots of ways to combine integers. JULIE: Yeah and those are the two basic ones. Then because Monoids also have an identity so with semi-groups, then you get even more semi-groups for integers because you get max and min, because they don't have an identity so there's semi-groups. CHARLES: There's always risk getting down into the weeds with the vocabulary but I think that there's a message here because your answer to the question is really, "When I understood Monoid and I understood Functor," from that point on, the overhead that you had to expend to get other things was lower than the overhead that you had to expend to get those initial two things. For anyone listening, Monoid and Functor are probably opaque terms. You have no idea what the hell they mean. We've been talking about in things like that a little bit but then it's okay because they're a finite set of opaque terms and they're very achievable and once you can achieve those, then you've done 90% of the work and now, you're just combining them into interesting and novel ways. JULIE: Yes. I will say it that a lot of people do tell us about Haskell book that applicative is actually the hardest chapter in the book, not monad but applicative. CHARLES: Really? JULIE: Yeah. A lot of people do tell us that. Because that's the first time that you've taken the concept of Monoid and the concept of Functor and combining them into a new thing so then, once you've done that with applicative, then after that, really it's all downhill. CHARLES: Right. It seems like there's a couple of key insights. As you're climbing that hill, I like that analogy is like one, just understanding that there things like type classes so you've through attacking Monoid and through attacking Functor, you realize, "There is such a thing." By recognizing there is such a thing as a Functor, you recognize that there is the potential for other type classes like it. Then through combining it with Monoid, to get applicative, you can see, "I can actually compose these things into new instances of those things," and then that's either the crest of the hill or the Pandora's box, depending on which way you look at it. I think there's a hopeful message in there that if you can invest the time to learn these opaque terms and making them transparent to you, you can really, really, really lean heavily on that knowledge in going forward. JULIE: Yeah. I'm writing a new book now called 'The Joy of Haskell.' The idea of The Joy of Haskell is meant to be an intermediate book. For people who already know some Haskell but we want to make words like Functor more general, like in Haskell book we really focused on the type class called Functor when it's actually a concept from mathematics or actually originally from linguistics oddly enough but we really focused on the type class in there, rather than trying to explain what a Functors are generally. In the new book, in The Joy of Haskell, we're going to try to take a lot of these terms like Monoids and Functors and catamorphisms and all these other words that Haskell has used all the time and try to explain them generally. Then also give examples like interesting uses from different libraries and stuff like that. It'll service both, hopefully a guide to the vocabulary of the Haskell ecosystem and also some documentation and examples for libraries and things like that that are useful because these things do have uses. They do get used in interesting and exciting or terrifying -- maybe those are related -- ways. That's the goal of the new book is to try to make a guide to all of this vocabulary that Haskell use all the time. We're trying to do that. How do I explain Monoids, you asked. You've got two values of whatever type. It doesn't matter the type and in general, there would be two ways you can think of to combine them, either making a sum or a union of all the values in them or making some product of those values, if they contain multiple values or even if they only contain one. That's how I explain them now. I'm not certain that addition and multiplication are actually the best ways to start with that because addition and multiplication don't act quite like set union and intersection do. I'm actually thinking of them in terms of and this is how I explain monoids to the people now, I start from set theory and that sounds really heavy but it doesn't have to be because I think a lot of things about sets are -- CHARLES: They're very intuitive, especially if you have visuals. JULIE: They're very intuitive, for people to think about. Yes, exactly. I explain Monoid now more in terms of set union and intersection. I'm actually giving a talk in October. It's coming up in just a couple weeks at Haskell eXchange in October 12th and 13th in London and I'm giving a talk there called 'A Monoid For All Seasons' and I'm going to try to explain the theoretic motivation for Monoids and try to explain them in those terms. Semi-group is a little bit different because lacks the identity but I'll try to explain the alternative type class and monad plus this really the same thing as alternative. These things are also just Monoids so we have these different names because it's a different type class alternative but it's really just another kind of Monoid. I'm giving that talk about set theory in Monoids in October, in a couple of weeks. People keep asking me on Twitter, "What's your obsession with Monoids," because my name on Twitter is Monoid Mary so I try to explain why I love them so much. CHARLES: Actually, it's an awesome point, which I've just gotten to experience it is what you see like, "Oh, there are these abstract things," you start searching for them. A lot of times, you'll uncover them and it'd be a real timesaver. There's the thrill of unearthing it in the first place and then when you could say, "Now that I've identified this thing as a Monoid, there's so much less that I have to write." There's like less work that I have to do. It's the same reason that we write frameworks for ourselves in software. It's like, "We love Ruby on Rails because of all the work we don't have to do." Now, you have to expend a lot of energy to work with it, using Rails an example but there's lots of software frameworks. It's like, "If you can find a good persistence framework or you can find a good thing for making a library for handling HTTP requests and responses, why would you write it all by hand in the first place?" I think the thing that's exciting for me as a developer is being able to see, "Monoid is a thing. Functor is a thing and I can now actually use this and I can use it almost as a looking glass to explore the world around me. When I see something in the landscape that just leaps out through that lens is another great one." I've been on a big kick lately but being able to say, "This is going to save me so much time because of the thoughts that I don't have to think and the code that I don't have to write." I think connecting it back to the pragmatic, I certainly have become really obsessed, maybe not about Monoids but having a type class large in your mind. JULIE: I think it's a really powerful thing. Sometimes that jargon is really useful. It's useful in a sense that it like compresses a bunch of information into a single word to remember. It's like teaching my eight-year old multiplication and we were talking about like, "It's like addition," and for us adult, I'll just go ahead say, "It's associative and commutative," but showing him that you can do those things and that addition is like that too and we're talking about that and he was so excited to learn that there's this word 'commutative' that encapsulates this idea for both concepts so he doesn't have to think like, "Addition does this thing. Multiplication does this thing." He doesn't have to remember both of those things, like he just remembers, "Commutative and they're both like this." It kind of compresses that information and what you have to remember and think about. Then it does make it easier to see that pattern in other things, then we can find commutativity in other things because now we have this pattern that we can look for and we got a name for it. We can talk about it and really, there's a lot of stuff like that in Haskell where we find some pattern that we find useful or we want to be able to talk about or easily translate to a bunch of different types, not translate is quite the right word but you know what I mean, I think. Then we give it a name and we make type class for it and then it's, "Now, we find it even more place for us." CHARLES: Right. It's about thinking less, right? JULIE: Yeah. CHARLES: That's a big misconception is that it's not about thinking more, it's about thinking less. JULIE: It really is. I think it's because there's so much kind of upfront work, where you have to learn all this new stuff upfront, then people mistake that for how much work we're always doing but in Haskell it's like, "We did all this work upfront and now we're now we're not going to think about these things anymore." ELRICK: That sounds like a good title for a book, 'Learn Haskell and you will think less," but it's true. When I struggled through that online class, I came out of that just being able to pick up any functional programming language and just hit the ground running. It is definitely a plus and you will think less. JULIE: Yeah, in the long term, I think that you do. Haskell is not a perfect language. There are things that probably can be improved. CHARLES: Now, before we go, I wanted to ask you, having had this very unique on-ramp into programming, which apparently you're still not convinced about. I'm curious what it would take to actually convince you but the real question that I have is there any advice that you have for someone who does not have a stereotypical background in programming who may not think that they would find programming interesting, who might have any number of roadblocks in terms of their own conceptions about the path forward. What advice would you have for them? JULIE: I am a bit joking when I say that I'm still not sure about writing software. I don't feel like I'm good at it and I think this is really the key. There are a bunch of domains in programming that I don't personally care about. I don't want to make web apps and I have nothing but respect and admiration for people who do. To me, it's very, very hard. CHARLES: Mostly because our tools aren't the same. JULIE: Yeah and there's just so many things outside your own program, there are just so many things that you have to think about and deal with because there's the network and there's other people's computers and they might be doing in other people software and what it might be doing. It is insane so for me it's very hard. There's a lot of domains of programming that I don't care about and when I thought about programming, that's the kind of thing I would think about. I certainly knew a lot of people who are web developers or the common programming jobs, I guess. Some of them just weren't that appealing to me and I'm not interested in making games or graphics so those are the kinds of things that I thought about for programming. There are things though that I am interested in doing. I'm very interested in natural language processing and I guess that's related to machine learning. I've recently taken up an interest in things like the raft protocol, the consensus protocol. Those kinds of things interest me a lot and there's a lot of the theory that interests me. I'm reading a dissertation right now about implementing a non-strict lambda calculus, which is what Haskell is. It's a non-strict lambda calculus and this guy's dissertations are theoretically implementing a non-strict lambda calculus. To me, the theoretical side is really interesting but then I am also interested in certain kinds of software. For some reason, I have developed quite an interest in making Twitter bots. I think that the advice I would give -- I'm rambling a little bit -- to people who think they're not interested in programming so why should they learn or whatever, is just find the thing that you are interested in and there's probably a way you can make software for that and maybe that will be the thing that will get you interested. It might not be Haskell, maybe you are interested in making web apps, in which case I would say go for Elm or PureScript, obviously because I like functional programs but Haskell might not be the best first language for you in that case but find the thing that you're interested in and there probably is a way to write software to do that. There's probably something in programming that will interest you. It's such a vast field. CHARLES: All right. I really, really like that answer. ELRICK: Yeah, that's a beautiful advice. Find your domain. CHARLES: Yeah, it's bigger than you think. JULIE: It's much bigger than you think. CHARLES: And there is a place for you. Thank you so much for coming on the show, Julie. I really, really, really enjoyed our conversation. JULIE: Yes, so did I. This is a lot of fun. CHARLES: Thank you. Now, before we go, I understand that you are going to be in London, was it roughly very, very soon, you said you were giving a talk. JULIE: Yes, the 12th and 13th of October. It will be recorded for people who can't get in. It will be recorded, I believe. CHARLES: You will be talking on 'A Monoid For All Seasons.' JULIE: Yes. CHARLES: And then you've also got The Joy of Haskell book, which you're hacking away right now, right? JULIE: Yes. CHARLES: With that, thank you so much for both of you. Thank you all for listening. What's a good place for people to reach out for you? JULIE: If they're on Twitter, I'm very active on Twitter so I'm @argumatronic on Twitter and my blog is also Argumatronic and that has more contact information. CHARLES: Fantastic. We'll link to those in the show notes. For everybody else, thank you for listening. You can get in contact with us at @TheFrontside on Twitter and Contact@Frontside.io over email. We'd love to hear from you. This just in, we're running a special. If you go to our website and enter the promo code 'ELRICK20,' you can get that 20% discount on your next custom developed web application. Go check that out. Take it easy, everybody. Bye-bye. JULIE: Bye-bye.

Chris Chuter: @Chris_Chuter Show Notes: 00:47 - Peeple: What is it? Why? 02:59 - Iterations and User Testing 13:32 - Complexity of Installation 17:26 - Device Integration 22:15 - Setup and Installation 25:35 - Laws and Building Codes 26:39 - Getting Started in this Space 31:29 - Ensuring Quality, Integration Testing, and Deployment Pipelines 33:18 - The Manufacturing Process Resources: If This Then That (IFTTT) Transcript: CHARLES: Hello, everybody and welcome to The Frontside Podcast, Episode 82. My name is Charles Lowell, a developer here at the Frontside and your podcast host-in-training. With me is Elrick Ryan. Hello Elrick. ELRICK: Hey, hello. CHARLES: And today, we are going to be continuing our series on the Internet of Things and we have someone on the podcast today who's going to talk to us about the Internet of Things. His name is Chris Chuter and he is the CEO, inventor and founder of Peeple. Hey, Chris. CHRIS: Hey. How is it going? CHARLES: It's gone well. Thanks for coming on the program. Peeple, what is it? Why don't you give us a quick overview of the product? Obviously it pertains to IoT, what is it and how did you become involved with it? Let's delve into that. CHRIS: Yes, sure. Let me give you the elevator short version first then we can dive deeper. Peeple is caller ID for your front door. The idea is when you get a phone call and you don't answer the phone, what happens? It goes to your voicemail. You know someone called you. But today, if someone comes to your house, you have no idea that they came unless you're there. This is the central problem that we solved with Peeple. It's a little device, a hardware device, an Internet of Things device that fits over the peephole in your door in the inside of your house. When someone knocks or doors open, you get a push notification on your phone. You can open up the phone and you can see a live view of your peephole. In a nutshell, Peeple is a smart peephole. CHARLES: Is it more for the case when you're not home at all or do you find the people use it for what you would traditionally use a peephole. CHRIS: It depends on the person. Now, my personal use case is for keeping track of wandering kids and that's actually inspiration for this invention. I have two boys and when one of my boys was three years old, he managed to open the door, walk out, go on to the street and walk down to the end of the street. Now, I live in Austin and I live right off the edge of a very busy street. Now, my kid didn't die or anything like that. It's not a really sad story but a neighbor brought my kid home and it was one of those moments as a parent where you're like, "Oh my God. I'm a terrible parent." But being an inventor and an engineer, I was like, "I'm going to hook something up that just tells me when my door is opened or closed," and it morphed into this invention. We showed it to people at South by Southwest almost three or four years ago. That's when we realized we were on to something that didn't exist. It was just a little camera on the door. CHARLES: Tell me about those first versions. I'm so curious. It sounds like there's a lot of layers of functionality that you've been through, a lot of iterations so I'm curious about that. What's was that zero iteration look like? CHRIS: Version 0 was made in 24 hours. It was a hackathon for... I can't remember the name of it. There was a hackathon group that recently imploded and we won this hackathon. The hackathon thing was to make something... I'm not sure if this is for Internet of Things but we were all making that kind of stuff. I made this little Raspberry Pi demo with a little mini door and I had talked to my wife and this is how I was able to make this invention, to keep track the kid as I was busy doing other stuff but I talked her into giving me 24 hours to make this one thing. Then me and another guy, David we won this hackathon. We were like, "We've got to turn this into a real thing," because one of the awards of the hackathon was you go to Silicon Valley, you show this off and you do all this cool stuff with it. We were like, "We've got to actually turn this into something that's presentable." That was Version 0. It was just a little Raspberry Pi. CHARLES: Now, what were you doing to detect the state of the door? CHRIS: That's the crazy thing. The first version of the device had more sensors on it than the final version. The first version had everything. It had a doorbell, it had a knock sensor, it had a motion, it had a speaker that played Paul McCartney's 'Someone's Knockin' At The Door,' but it had an accelerometer. I threw everything in there the first thing and half of it worked for the hackathon demo but it was good enough to win. This is something that, I guess I could call wisdom now but the real thing I learned is you start with everything and then you narrow and get it more tuned and highly focused and more precise as a device, like the difference between the iPhone and the Samsung phones. One of them is to throw everything into it and then the iPhone is just really specialize into a few things really well. The next three years, we're pulling stuff out. CHARLES: What are some examples of that calling that you're describing where you're saying, "I'm to take this out? I'm going to take this out. I'm going to take that out." CHRIS: We got rid of things like the doorbell and some of the other sensors, mainly because it was just a wiring issue and as well as we wanted to keep track when the door was opened and closed. It didn't make sense to have the speaker on there at the time so we really focused more on the accelerometer and the knock sensor for the first version of Peeple. CHARLES: That is not the final version. Is it mostly just the accelerometer? What if someone doesn't knock? I assume there's some sort of detection that goes on with the camera. CHRIS: That's the next version. That's something that we've been working on right now, what we're going to be delivering. We have delivered our first, I would say Version 1.0 of Peeple devices to our customers. There's a thousand of these or so in the wild, all around the world and the next version we have added -- and I guess this my first real announcement of this -- a motion detection module. It's not a camera-based. It's more or less magic and it just works through the door. That's the most I'm going to say on it right now because we're probably the first hardware device that it's actually using this technology. ELRICK: That's an excellent pitch. Everyone loves magic. CHRIS: Yes, it's basically magic. It works through the door. ELRICK: As you were going to these iterations, were you doing like user testing to see what users wanted? Or did you internally say, "This doesn't make sense. Let's just take this out." CHRIS: Absolutely. That's the second part of this story. After this hackathon happened, we prepared to go on the road show to go and show it off to Silicon Valley but in the meantime, this hackathon group, I think it was called AngelHack, it imploded. One of their founders made all these disparaging comments about homeless people and what essentially happened is we lost the award. They said, "We're sorry. We can't give you the award," but we had spent about three months fine-tuning, making something pretty and putting a pitch together. I went in and I pitched at a TechCrunch Meetup in Austin and we came in second at that but during that meetup, I met one of the reporters and said, "You really need to talk to these guys in San Francisco called Highway1," so I did. We eventually ended up moving to San Francisco. Now, the reason I mentioned that to answer your question is they understand this idea of user testing, I think better than a lot of people. Even though they were focused on working on hardware and getting an IoT device that works out there, they were drilling it into our heads is, "You have to get this in people's homes now. I don't care how bad it is. I don't care if you have to hire people, to sit at a peephole and just look through it and pretend like there are hardware device. You got to do this and you have to find out what the problems are, what works. I want you to look at your biggest fears of this thing and you quash them and you do that before you put any Silicon down," so we did that as best we could. CHARLES: So you did that with the Version 0 and Version 1 devices? CHRIS: Exactly, just a Version 0, I have all these pictures. We put them in about 12 to 20 homes and we have these long extension cords powering this thing because we didn't have the batteries to figure out. We had these huge lag problems. It would take like 30 seconds to a minute before something would happen. We had all these issues but in the end, people were still like, "It had these issues. You couldn't do this," but the fact that I had a door log, a door diary as what we're calling it now, that's something I never had before. That's where your secret sauce is so we ran with that. CHARLES: Yeah. That's the kind of thing it never even occurs to you. CHRIS: Exactly. In the app, or at least the early versions of the app, is you have these versions like a calendar that are like, "Okay, I got 10 visits yesterday. I got 20 visits today. No one came to visit me today. I'm so sad," but I have a calendar of, I think it was May of last year when I got visited by three or four magazine salesman in one week so you could correlate that with, "Did we have any break ins?" or something like that. CHARLES: Yeah, it would be interesting to be able to share that data with your neighborhood or somehow coordinate that. one of things I'm curious about too is you did this user testing you were talking about, doing the wiring and the installation, it's a conversation that always comes up when you're talking about custom hardware because there's always the drive to be small, there's always the drive to be have a small form factor and then you have challenges of power like how do you power this device. How cumbersome is the installation onto someone's door? CHRIS: Yeah, we had it all. That's a big difference, I think between San Francisco or Silicon Valley and other towns is there's this acceptance and there's this readiness to participate in the tech scene. We did a call out for volunteers and we had no problems finding them. They didn't mind us coming to their house and hooking up these big, bulky things and just being real intrusive. The fact that we found these people and they were the key to this early stage of, "Do you become a product or do you not?" We were only there for four months but by the end of this time that we were there, there was this legitimate tangible feeling of we're not a prototype anymore. We're a product and we didn't have a product. It was just prettier but we could see the light at the end of the tunnel. I don't think that would have happened had we not gone through this very painful experience with all these poor people that we inflicted our device on. CHARLES: This actually is fascinating because obviously, you're back in Austin now and I never heard of programs like that, like sign up to have someone come up and test it at some alpha stage prototype in your home. That sounds crazy and yet, it sounds like they were just going out of the woodwork. CHRIS: In San Francisco, it's not a problem. If I put the call out now, I probably have to really like, "Here's an Amazon gift card." I have to start doing a little bit of bribery. ELRICK: I think I would sign up just to see the cool tech. CHRIS: Yeah and those people exist. I think we don't have the means to really find them. That infrastructure already exists. In Silicon Valley, you just go down to Starbucks. CHARLES: There ought to be some sort of meetup for people who want to experiment with very early stage IoT devices here in Austin. Maybe, we'll have to look at it. If that doesn't exist, I would love being a guinea pig. I actually think there is an untapped willingness here but there's just not -- CHRIS: I think you need a critical mass of hardware people and hardware devices that are ready to be put in doors or put in the houses. There's definitely some in there. I have a lot of friends and there are hardware meetups that we go to but this stuff takes so long and it's so hard as hardware is hard. There's that small window of, "We got this little idea of a water sprinkler. Do you think anyone want to try it out?" or something like that and then the moments gone. Then six months later, there's another one. CHARLES: Yeah. I wonder if there's a way to really decrease that iteration cycle so that you can get feedback more quickly. I guess the problem is when you need a physical device, you just needed a physical device. CHRIS: We're talking about the Maker Movement and the MakerClub. If you're part of those, these people are hard to find. People that go to Maker Faires, that's the people you're looking for. CHARLES: Right. Now, transitioning because ultimately your target customer base is not makers, not people who are willing to put up with wires and cabling and people doing protracted installation. What does the kind of 1.0 product look like? Because what I'm curious is what immediately jumps to mind is this thing sounds like it's going to probably consume a lot of power. How do you get the power to that and what are the challenges and what are the tradeoffs that you have to make to try and get that power consumption down or get the installation complexity down? How complex is it today to install? CHRIS: I guess, I'll toot my own horn a little bit but I think we have one of the easiest IoT devices on the planet to install. You can possibly not even need tools. You can use your fingers but the biggest challenge for any IoT device is getting that home network connection. If there's been a few technologies through the years in which they've tried to fix this problem, basically just like self-pairing or things like that, like how Bluetooth can sometimes be really cumbersome. Now imagine that with Wi-Fi, it's the same thing but now you've got a password you've got to throw in there. That's really the only real hiccup with the installation on our device and we tried a few things. We went through about three different Wi-Fi chips before we settled on what we were using now. The first Wi-Fi chip was a TI one, which offered this nice pairing capability but it just didn't work half the time. Then we switched to a Broadcom chip, which was really solid and stable but turned out to be the most expensive component in the whole device so we had to get rid of that. The Wi-Fi issue was something we had to solve early because it goes also toward your power consumption. We have a camera and a Wi-Fi chip and both of those take up to 140 to 200 milliamps of juice when they're on. We had to be really smart of when this thing was going to be on and that's essentially when we went in parallel with the knock accelerometer. This device stays asleep most of the time and that's how we get the many months of battery life out of it. We put a rechargeable battery inside, it only turns on when it needs to and it's just hanging around waiting for an event for the rest of the time. Those were the things we were solving to get the Version 1. CHARLES: Now, it's waiting for some event but in order to receive the event, doesn't the accelerometer need to be on? Or is there some motion detector that --? CHRIS: That's a solved problem. good news was that accelerometers are extremely low power in the nano or picoamps but that's also another reason why the motion detection was going to be a hard problem because that is not, unless you're using what's called a PIR that is not a low power solution. CHARLES: Acronym alert. What is a PIR? CHRIS: It's an infrared proximity detection. That's how almost all motion detection cameras work. They have one hole for the camera and another hole for the PIR. The problem with these are is they don't work well in sunlight, outdoor-light and things like that in one of our use cases so we were kind of stuck. That's why we've recently come up with this new motion solution that doesn't rely on that technology -- the magic solution. CHARLES: All right. When we're going to find out about the magic solution? CHRIS: As soon as I ship this next version because it is being used in a few products but it's not really stateside yet and I want to save my thunder but it's something that I think is really cool. It really is magic. It's just amazing to me that it works. CHARLES: Well, I'm eager to see it. You were talking about Wi-Fi being one of the biggest challenges. That's a perfect segue. The connection to the network for something that we're always curious is discovering a new and interesting device is always a pleasure and then the next thought that almost funnels immediately after is how can I integrate this with other strange and wonderful devices to make something even more wonderful? A question we ask everybody is have you thought about how this might be a participant in an ecosystem so if there were other devices around the home, how would they even talk to the people? How might it offer information to someone looking to, maybe do some custom integration in their home? CHRIS: That's a lot of questions in one. Essentially, there's two ways of looking at it. You can look at it from your customer's perspective, what kind of customer do I think is going to have this or is going to use this the most. Back when we came up with this, there were a lot of do-it-yourself types and If This Then That protocol was out there but we really wanted to focus on something that was incredibly easy to use and didn't require you to program anything. I was really frustrated with the whole idea of Internet of Things because it almost implied that you had to be a programmer to use it. I didn't like that at that time. I've since come around to it because there's all these great tool kits out there. We initially looked at integrating with HomeKit. We thought they'd be perfect but what a lot of consumers don't realize is early HomeKit -- I don't believe it does that anymore -- made you modify your hardware to put in this special Apple hardware. When you're making a device, it is so hard just to get the hardware down. It's so expensive. To add anything or to put anything else in there, it's a huge friction point. It's really something that small startups just can't afford to do. A big Nest or a company like that have no problem but when you're making a one device, this is a big deal so we weren't able to really leverage something like HomeKit for an API. But we do have our own cloud-based API. We're RESTful API but it's just not documented and put out in a way where we want to have people programming it. But the good news is we did leverage several APIs when we were making things like the app and doing things like the push notifications and things like that. Now, it turns out that a lot of the case we used are now integrating with things like Alexa and other device protocols so we essentially get those for free. This whole ecosystem is forming around us. Just most important is to get your device out there because you have a vision for what the device will be used for. But then your customers tell you what the device is really useful for and that's when the real work starts. CHARLES: Right. I guess, it's true you have your first line of customers and I guess the use case what I was thinking of is me being a developer. I'm thinking what products could be built then using this as a component, so to speak. Have you'd given any thought to that or have anyone had approached you to say, "This is amazing. I'd like to build this meta product that integrates that," or is it kind of early days? CHRIS: Early on, that was the approach of the Internet of Things and it merged away from that in my experience. Early on, it was all about building blocks. You got to understand, these are old Zigbee Z-Wave programmers and that was the whole concept. Then it got turned on its head by, "I really have this problem that I need to solve and I don't want to have to make a bunch of building blocks to do this." For attacking it from the other side, like you're saying, building up into pieces, I really recommend you talk to the Twine guys -- super mechanical -- they're here in Austin as well. A year or so before, we came out with Peeple. They put out this device which was exactly what you're talking about. An Internet of Things type hub where you just add in all the pieces and then you integrate with everything. They can better give you a story of how that lifeline goes. CHARLES: Yeah, because it's always something you think about because you've got all these wonderful things. CHRIS: Yeah, some would say, an Internet of Things. CHARLES: Yup, or at least a floor plan. ELRICK: When someone gets a Peeple device, what is the full installation story and set up? What is the walkthrough for that? CHRIS: We have a little video of that. What you essentially do for Peeple when you're installing it on the peephole in your door, you unscrew the peephole. Now, the way Peeple's work is they need to handle doors that are variable width, depending on where you live. There's no real standard. All of the Peeple's work by having a shaft that you screw onto another side so it's basically two pieces. Now, one of those shafts holds this bracket that we include in the package. You screw that onto your door with the peephole holding it to the door, then you turn on the Peeple device and you connect it to your home Wi-Fi and then you're ready to go. That's it. CHARLES: That's the hardware side of the onboarding and then what about the software? How do I go and look at my door diary? CHRIS: You do this during the installation. You go to My.Peeple.io and there's a little button to add your Peeple device. UI-wise, it's one user interface among all the platforms whether your Android, iPhone or on a browser. You just go to that webpage and associate your account to your Peeple devices. You will have to log in. You can log in with Gmail, Facebook or just a regular email. Then you add your device and any time you go back to that page, it will show you only the videos from your device so you have a list of all the events from your Peeple device on that page or in that app. CHARLES: That is interesting. I'm looking at the videos right now online. Although my problem actually is I've got a glass door. CHRIS: Yes, we got you covered as well. CHARLES: You do? CHRIS: Yes. The reason you have a glass door or a peephole and many people don't realize this is it because it's required by law. If you ever plan to have run out your house as a multi-family unit, you have to have a peephole or a window surface to where people can look out. Once we figured that, that's when we realized we were onto something. The first versions of Peeple came with these little adhesive pads that we called gecko skin and this is where we learned a valuable lesson. No matter how sticky you make your stickers, they're not sticky enough. We included three of these little tabs in every device to put on a glass door, if you had glass so the Peeple device would work the same way for glass door, except that you would use a sticker, instead of unscrewing the peephole. The only problem with the stickers were is they were not sticky enough. If there was condensation or a weather event or something like that, these things would fall off so we made a modification. We found better stickers and I mailed those out to all the people. But this is why hardware is hard. You're going to make these mistakes. In all our testing, we didn't find this but of course, once you have a thousand testers, you find a little more. ELRICK: That's interesting that you brought up the laws about the peephole. Were there any particular building codes or anything of that nature that you guys had to be concerned about when having Peeple installed things on their doors that you had to figure out before shipping them out? CHRIS: Not really. The Texas property code is more geared among making landlords do the right thing. In case you're wondering, I think it's Texas Property Code 94-152 that covers this. There must be an external viewable portion for all multi-family units to the front entryway. Now, this is just the Texas law. We had to look this up in a few other states and it turns out there's one in San Francisco, there's one in Virginia but they're all different. But so far, we haven't had any issues with any property codes or building code issues. CHARLES: This has been an almost four-year odyssey for you that you've been on, right? CHRIS: Right. CHARLES: You've been involved in this scene and working with hardware probably for a long time even before that, it sounds like. For people who are just getting into it, because I feel like there's this wave cresting now, where these types of startups and these types of side projects and hobby projects are just starting to enter the mainstream. Do you have any advice for anybody who would want to get into this space? CHRIS: Well, that's a great question. Of course. Now, contrary to what you just stated, I didn't have much of a hardware background. I'm a software guy. I can personally attest to the pains of becoming a hardware guy. Now, the irony of this is I do have a master's degree in electronics engineering but electronic engineering is so huge. It's such a big field that you can spend your entire career not doing much hardware. But I always had the ability to go back and build some circuits but I would say the number one thing, if you're not a hardware guy is go to some of these meetups or get involved in a community and find yourself one, someone who has experience doing hardware because coming from the software room, you're used to this flexibility of changing a few lines of code and being everything changing. Now, when you get a hardware guy onboard and our hardware guy's name is Craig, when he comes to work -- CHARLES: Or gal. CHRIS: Yeah, or gal, of course. When they look at the same problems you're looking at, they're like, "Hold on a second. Let's step back. Let's test this." There's this quantitative slowing which you need to have as hardware because once you build a PCB, a circuit board, you are now stuck with that board for the next month or so because it takes a while to make another one so get that right before you jump around and do all these changes. My first advice would be is get help. There's no shame in going out there and you might be surprised. There are so many people out there that want to join in. If you have a good idea, there's plenty of people who want to contribute. CHARLES: Would you say that there are communities out there like the software communities where you have meetups? Some of the software meetups are just fantastic, where people are so welcoming and they're just so excited to share the information that they themselves are so excited about. CHRIS: Yes and there's the same thing as on the hardware side. You would definitely go to a few hardware meetups, there are several in Austin. There's at least one every week and it's a great chance for people to tell these kinds of stories. This is a maker type community so they welcome these ideas because that's what fuels their enthusiasm. Every time someone is doing something new, they want to hear it. That's the change now. This decade has happened to where you can go out and buy a few modules and make your little device. Then there's the next big step of turning it into going from prototype to hardware but you can get all those kinks out without having to make your own printed circuit boards, without having to have a huge firmware background. Just knowing a little bit of tech and a Raspberry Pi, you can test out your inventions at this early stage without having to invest all this money and these other things. There's never been a better time to do it. I would leave your listeners with is if you got something swirling around your head, get a Pi, get a little Arduino and do it. There's nothing stopping you. CHARLES: Yeah, it's shocking how affordable they are. CHRIS: I don't even touch on China, by the way but that's the next step. CHARLES: That's the great thought that I want to leave everybody with but I actually have more questions so we won't leave everybody with that. We'll keep on going because I want to talk about China and I want to talk about something that was in there. You've touched on it a couple of times when telling your story how you go from this just do it, get it out there, get it into people's homes, just get the Version 0 out, just buy an Arduino, slap together something terrible, that is at least one millionth of the dream that you have and you've taken your first step on that odyssey. That's a very common story in software. The way that we develop software too is have these agile methodologies and these techniques to reinforce them, testing, continuous integration, continuous deployment. How does that play out? A fascinating subject to me personally is how do you do that in the context of hardware. A question that I love to ask is how do you do things like ensure quality? How do you do integration testing? How do you have a deployment pipeline if you've got these Peeple devices out there on tens of thousands of doors globally? How do you push out a bug fix or a feature update? What's the automation around that look like? CHRIS: The over-the-air updates are your friend. If you're going to make a hardware device, I recommend making a Wi-Fi enabled device because then your firmware is not locked, then you can do over-the-air updates. That has been a lifesaver. We've done maybe a dozen software updates to our device to date, sometimes little changes, sometimes big changes. But what happens is any time the Peeple device wakes up, it says, "Hello, server," and the server says, "I got an update. First, let me give you all these images." Give me the code. The devices are constantly upgradable, just like you'd expect with software. Now, with some of these Bluetooth devices, you can't do that. You've got to go out the door being ready to go with no issues. It's a friction point to tell someone, "Your headphones can't work now. You need to plug it into a computer. You need to download this firmware upgrade. You need to update the firmware doing it by hand." That just isn't going to fly in today's consumer market so I would recommend if you can, make your device a hardware Wi-Fi device, get a Wi-Fi module in there and that opens up the world to you on doing a lot of these updates, to answer the last part of your question. CHARLES: You mentioned China, since you're touching on the manufacturing process or just the market over there or --? CHRIS: Yeah, be ready to fully commit. I've been to China, maybe four times now. I have a 10-year visa. It took a while to find the right partner and you've got to be boots on the ground in the factory for a couple of weeks just getting the whole line up. It's a whole another product when you're at the manufacturing stage. You're making all these little test things, they've got to hook up the boards to certain devices, they've got to put the firmware on it, they've got to do these things. It's a whole another job. That's why when you do these Kickstarter. They say, "We're going to be out in three months," and then six months later, "We're still working on it." I have a lot of empathy for this because I've lived it. You think, "I've got everything done. My hardware works. All I have to do is team up with someone to just make it and with them, we'll ship it." There's a whole another level to just a manufacturing piece and you can't really learned. There's no real textbooks to learn this because every factories are different. Our factory is right north of Shenzhen and we talked to some US manufacturers but they just weren't competitive to be in the discussion so you pretty much have to go overseas and then you have to sit down with them and just a little bit of communication difficulties can bring down a whole manufacturing line so it's very important that you're very hands on and you see your product all the way to package. ELRICK: That's interesting. I know of it but I never really thought about it because I was really not in that position. What are some of the higher level of things that you should look out for when evaluating a manufacturing partner? CHRIS: We talked to about a half a dozen before we decided on our manufacturing partner. The big one for me was cultural fit. I talked to some of the big ones like the one that makes the Apple phones, we talked to them for a while and I just found that I would say, "We would like to do this or we need this," and then the next week, they'd be asking a question, "What about this?" and I'm like, "Oh, you didn't understand what I was really asking," so you would lose weeks just by tiny misunderstandings. I found a manufacturing partner that has a subsidiary here in the US and my main contact grew up in the United States but he also goes to China every other week. Having that kind intermediary made everything so much easier. The communication was never an issue. I was able to get things done almost twice as quick with the other manufacturers I was talking to. In the end, they also came up with a great price so it turned out to be a win-win. I would recommend talking to the bigger manufacturers but spend a lot of time on the smaller ones and really figuring out is the communication up to snuff to really make your product. It's huge. CHARLES: What a story. I'm really glad that we got to have you on the podcast, Chris because you have the story that starts from literally slapping a Raspberry Pi and an accelerometer and speaker and apparently a bunch of other things on your front door and with an extension cord and walking a continuous path to where you're flying back and forth between China and Austin to inspect and ensure your assembly line and making a real product. It demonstrates that it can be done by the fact that you have done it so I think it serves as an inspirational case for a lot of people out there who might think that this is something that they might want to do. Or think that they're capable of. Thank you so much for coming and talking about Peeple. Everybody, you can go ahead and check it out. It's Peeple.io, right? CHRIS: That's correct. CHARLES: All right. Also, is there anything else that you'd like to announce other than the magic, which you're going to keep a lid on? CHRIS: Yes, I know I'd appropriately teased everyone about that but you can go to our website. If you go to Shop.Peeple.io, we're taking preorders for this next magical version, the Peeple Version 1.1, I guess I'll call it. I would like to add just before we go is if you're going to endeavor to do something like this, make sure you have a very understanding family because they couldn't have done it without a wife and kids that understood my craziness and allowed me to have just a complete mess of our house for, I guess, for three years now. CHARLES: Thanks again and thanks everybody for listening to this episode. You can get in touch with us on Twitter. We're at @TheFrontside and you can always find us on the web at Frontside.io and there's a contact form and we'd love to hear from you, for any reason whatsoever. Thanks, everybody and we'll talk to you next week.

John Boyd: LinkedIn Show Notes: 01:27 - Knocki 03:20 - The Device 06:19 - Complexity 08:44 - Software Distribution 14:01 - Allocating Memory 18:27 - Finding Hardware Hacking Libraries 22:01 - Updating and Diffing 24:06 - Migrations 26:51 - Decentralization of IoT 35:39 - Managing the Knocki Ecosystem 40:17 - Communication Standardization Resources: Malloc Transcript: CHARLES: Hello, everybody and welcome to The Frontside Podcast, Episode #81. My name is Charles Lowell. I'm a developer and your podcast host-in-training here at the Frontside. With me today is Elrick Ryan. Hello, Elrick. ELRICK: Hey, how are you doing Charles. Welcome back. CHARLES: Yeah, thank you. It's good to be back. Today we're going to be continuing the ongoing series that we've been doing intermittently on the Internet of Things. It's a really fascinating, almost to a person fascinated with here at the Frontside. Today, we have with us to talk about this, someone who's very, very knowledgeable on the subject, John Boyd, who I got an opportunity to talk with, I guess it was about a month ago and I wish that we had the podcast recording equipment there in the room because it was just a very, very well-versed engineer, exactly the person you want to be the CTO of your company, which is very lucky for Knocki, the company that he works for, because he is in fact the CTO there. Welcome to the show, John. Thanks for coming. JOHN: Yeah, thank you very much, Charles and I'm excited to be here. I'm excited to join the conversation this week. CHARLES: Yeah, why don't you start by what it is that you do at Knocki? Most of our audience comes from software and design and product management backgrounds. You've got a very strong hardware background. How does that play in to what you do at Knock? JOHN: Yes, certainly. As you previously mentioned, I'm CTO at a startup called Knocki, which you can mount onto any surface and turn that surface into a user interface. We're recently funded on Kickstarter so we're in the process of actually trying to develop this hardware but the central concept is any surface that you mount this on will now listen for touches and vibrations so you can say, mount it on a desk and tap three times on your desk and control your smart home around you. If you have smart speakers or TV, you can tap three times out of four times and control those devices with a really natural interioractive interface made out of anything in your home already. CHARLES: Tabletops, mirrors, I assume you've tested this on a lot of different services. JOHN: Yes, I'm sure we'll talk about that more a little bit later but the goal is to be able to turn any surface into user interface. That means if you really wild and you want to use it on the window, I recommend it. But we're thinking desks, walls, doors. It has a lot of applications for disabled and handicapped individuals. Think of a child or someone in a wheelchair that can't quite reach a light switch, if they have a Knocki mounted on the wall, they can still knock on the wall to control the lights. We feel like it adds a new level of user interface to people's lives that can be helpful. CHARLES: Definitely. Seeing the product and hearing you talk about it, I definitely got that impression. Now, the device that you actually brought into the office because you did come in and talk to us, like I said it was about a month ago but it was extremely tiny. In our explorations into the Internet of Things, we do things like control our lights from within the office. At least, we're trying to control our lights within the office. For us, we're using the standard kit. We've got Raspberry Pis that we're using, that are have access to a plug and they've got a full Linux install, just a really powerful processor and by comparison to the things that you were talking about, that's energy hog by comparison. We think of it as being very lightweight but if you're talking about making some small device, it's actually really, really wasteful of resources, so to speak. What is that transition that spectrum which you moved from these one-off hobbyist things where you're using high-powered equipment to these really custom devices? How do you make that transition? And what is the difference between the two? JOHN: Our devices are about the size of a hockey puck, which is much smaller if you can think of a Raspberry Pi. Pretty difficult to fit that inside of a hockey puck, especially when you want to start adding some sensors to detect knocks and taps on a surface. I don't hate or dislike the Raspberry Pi or BeagleBone Black or any of those really quick SBCs that can get you started with IoT. But they have -- CHARLES: Acronym alert. What is an SBC? JOHN: SBC, single board computer. It's any of those credit cards size computers. CHARLES: Okay, great. So nothing against the SBCs like BeagleBone Black or Raspberry Pi. JOHN: Exactly. It's a great way to prototype ideas and get in a proof of concept out there and there are some cases where actually, they're great choices for a full-fledged product. A lot of cases in IoT, people are more concerned with things that you carry around with you so they have to be battery powered and you need to be a little bit more conscious about energy economy. You need to be very cost-effective with your components and it doesn't make sense to buy an expensive Raspberry Pi for each unit. CHARLES: Did you actually start with a Raspberry Pi, when you were developing this product or something that's like an SBC? JOHN: I actually went straight to a microcontroller dev kit. I started with Texas Instruments' CC3200 LaunchPad. It's a little bit lower level than SBC like the Raspberry Pi. It doesn't run Linux. The firmware I started off writing as a proof of concept was still embedded C bare metal software. CHARLES: How much complexity does that add? There's just a lot of nice things about having an operating system and being able to have your compilers, I guess you have a compiler tool chain, but having being able to install big programs like interpreters so that you can run Ruby and JavaScript on there. There's just nice things like scheduling. If you've got a bunch of processes running on this device, you don't have to worry about them, saying who's going to get what processor time. I assume that you're having to deal with all of that if you're writing the firmware by hand using C, right? JOHN: That's 100% true. There's definitely some great advantages to using a little more powerful system that can run a full Linux stack or full OS. As you mentioned, the design complexity is reduced a lot because you can import other people's code and you have a full operating system to handle most of the drivers in the system. You're right. There's a lot more complexity. We have to write all of that ourselves in C. But that's the fun part about it to me. I love getting down there and writing drivers that can communicate with accelerometers and set them up. As far as scheduling goes, for getting concurrent software running on your embedded system. There are RTOS's -- real-time operating system that can provide basic scheduling. For the brave, you don't even have to use that. You can use a lot of the embedded timers inside the microcontroller itself. But to answer your question, it is a lot more complex but one of the tradeoffs to get a device that small, beautiful and also has a battery life that can last many months or a year. CHARLES: Yeah, it almost sounds like the complexity but you're not going to save yourself any time prototyping it in tools that have all those things because you're essentially going to have to be rewriting your system from scratch, because those things are just a nonstarter if you want low profile devices. JOHN: Yeah, there's definitely a lot of rework would have to be done but those SBC systems are still very useful for prototyping the cloud side. Internet of Things is hardware and internet when it comes to building out your cloud interface. CHARLES: Yeah, that's definitely true. You're running a bunch of software on this device. The software that you've written, how do you actually distribute the software because we're very used to in our world, software distribution is not a problem. That's what made the web so popular. While we were willing to deal with really crappy tools on the web for a really, really, really long time, the distribution model was just so nice. You're also having to deal without that too when you're operating in the device space. But the challenges are still there. If you've got a bug on one of these things, how do you even detect it and how do you get a fix out there? JOHN: Obviously, any software is prone to bugs. Nobody writes a perfect code the first time. If you do, I'd love to hear about it. Obviously, one of the big concepts in IoT is security and to have a secure product, we need to be able to patch bugs as they arrived. A big really important feature in any good IoT product is the ability to remotely upgrade the firmware or send the patches as part of the maintainability that prevents big software bugs from turning your IoT product network into a botnet. A lot of our time is actually spent trying to make sure that our remote update capabilities are reliable, always functioning and globally distributed. You'd think this is an easy problem to solve but when you're working on a microcontroller that's not running an operating system, running bare metal code, things get a little bit more complicated when you want to make sure that any device anywhere in the world can install the next version of firmware reliably. CHARLES: Right. At any time there's a software update, it's always, it bugs me and then do I want to do this and it's always optional. There's none of that, right? It's just what a new version of the firmware goes out, boom! It goes out there. JOHN: You can design it in different ways. There are some great products out there. Apply the firmware update through the user's phone so you may open up your products application and it says, "There's an update available. Go update." That's definitely one way to do it but that's the problem if the user is not home and maybe they've set this device up in a guest house and they won't be home for six more months, then you have a device that could be vulnerable for six months, which is a long time in the world of software. CHARLES: Yeah, that's true. JOHN: To get around that, obviously our preferred solution is to have the device checked into our cloud servers to see if the device itself has updates available and then go through the download and update process that way, just to make sure even if the user is not home or never opens their mobile app, it will still get those critical security updates. CHARLES: Sounds hard. You're running a risk of bricking someone's device if that update doesn't go very well or it loses internet in the middle or power. JOHN: Very true, especially when you go towards a bare metal microcontroller with limited memory and limited processing capabilities, unreliable internet connection, a lot of work has to be done on the device side to make sure if something goes wrong during the firmware download process or installing the image correctly that it has a backup image. If you're downloading a new firmware upgrade and the download gets corrupted halfway through, make sure you have an old image that you can boot into. That's one part of it. The other part is detecting that it went bad if it gets past downloads in your image and then it reboot itself and tries to boot into it, how do you know that that image actually isn't behaving the way you want it to and then go ahead and revert back into that original stable version. CHARLES: I assume there's some key so that you can verify, not only that the image is not corrupt but it's a certified Knocki image that's coming down the wire? JOHN: Exactly. We signature verification, again something that I think anybody on the internet should be using when you download new software but make sure that the new firmware update was actually written by Knocki and you're not installing someone else's code. Another important factor is just please use HTTPS secure SSL connections to your server, then that reduces the possibility of someone taking over and giving you their own firmware image. But there are a lot of low power devices out there that are being used to make IoT products. These low power devices are important for many reasons but they have restrictions and sometimes, their security capabilities are limited. Maybe doing encryption on the device and actually are doing certificate verification. That's a costly operation. CHARLES: It sounds like there's a lot of cycles that that consumes. JOHN: Definitely. Most people try to make sure they have the resources to solve these problems but at the same time, there are a lot of developers out there that are cutting corners and that's where you get these big news stories about IoT products getting taken over. CHARLES: Along that vein, it's your reality but it constantly blows my mind that things that you're living without when you're programming for these devices like Knocki is do you have to write your own network stack? When you're doing these downloads, that's kind of like got it all. You've got the encryption piece that you've got to do to make sure that you're connecting over SSL so you've got to do the whole handshake and you've got to do the key exchange and the certificate verification and then the packets come in asynchronously so your message is arriving asynchronously in bits so the header is being assembled, now I've got the HTTP headers, now I can go ahead and get the body. There's a lot that happens for us when we're making a simple Ruby request. We're basically like resource.get. Boom! And it just comes to us fully assembled in memory. How much do you have to hand roll all of that? Are there libraries for doing it? How do you put that process together of just even downloading the image? JOHN: Fortunately, there are tons of open source freely available libraries for embedded C software that can help us solve these problems -- CHARLES: Is this like a genre of software like if I want to go look for these libraries, how I look for them? JOHN: In my example, all of our firmware is written in C or C++. Since we're working on a microcontroller with limited resources, it's important to look for libraries that don't use dynamic memory allocation. That's why it's a really big [inaudible]. Some software relies really heavily on that but -- CHARLES: When you say dynamic memory allocation, you're talking about like Malloc? JOHN: Exactly. CHARLES: You are basically are allocating memory on the heap. When you're doing for this, you basically want to do everything on the stack. Now, is that just because the instruction set of the processor doesn't support it or is it because it's just there be dragons like here there be dragons? JOHN: That particular scenario is actually just due to resource limitations. There's just not a lot of memory on our device. We do use Malloc in some cases but we have to be very careful about when we use it and make sure that it's always going to have the memory required or if it doesn't have the memory required, there's some fail safes involved. If you just use someone else's open source library and they're allocating memory left and right, they could end up causing issues on your embedded system. CHARLES: Right. Now, just a little bit of background for people who might not be fully familiar with Malloc, it's just when you're executing a program, you have this heap memory, which is where you store random stuff and then you have your stuff on like the call stack. Your variables that are on the call stack are in one place and then your just generic data structures that could be accessed from anywhere are in this thing called the heap. Our dynamic languages that we use like Ruby and JavaScript, the heap is hot stuff. Like everything gets allocated on the heap, that's why they consume these huge amounts of memory and then the things that are on the stack, really are just pointers that are referencing these big bags of data that are on the heap. But it sounds like you've got the exact opposite situation where you don't want to have big bags of memory that are just floating around in a heap and you want to do everything inside that stack. JOHN: Exactly. I couldn't have said it better. CHARLES: Anyway, you're looking for libraries that don't do that because it sounds like any time you want to allocate memory on the heap, that's going to be shared for the whole program, that space is very limited so you want to be very, very, very strict. You want to control that process. You don't want any other library that's doing it for you. Is that fair? JOHN: That's correct. That's also one specific example, dynamic memory allocation of the things that you want to make sure your other software libraries aren't going to be abusing. But in general, you need to make sure that any code that you're putting is compatible with your system. It doesn't have some special hardware requirement that your embedded system doesn't have. CHARLES: Right. For people who want to get into hardware hacking, is there some golden seal of approval like the people say like, "This library is great for embedded devices." Like I said, a lot of times when you're coming into it, you don't know what to look for so what you're really looking for is some expert or authority on the subject who can say, "This is good. This is not good." It is like, "Don't even look at this library because you're going to find something else because this is not embedded-friendly." JOHN: That's a good point. I wish there was a golden seal of approval or I wish I knew one, at least. Normally, most of our code that we uses are hosted on GitHub. Usually, we try to find software that was optimized from embedded systems and the author of that code will usually mention -- CHARLES: That [inaudible] me. JOHN: Exactly. This was designed for microcontrollers. ELRICK: I was going to ask if there's a golden standard when you're building these type of devices. Is there a checklist of things that if someone's going to build something similar that these are good things on your checklist that you should attempt to check off, if you're building this sort of device or want to build something similar. CHARLES: Now, you mean things like update and whatnot? ELRICK: Like updating or like how you were mentioning avoiding dynamic memory allocation. Anything, you can just shoot from the hip, like these are things that you should watch out for a lot of your battery power, you should look out for this or anything. JOHN: Yes. I definitely think the number one consideration that the biggest check box and [inaudible] before it goes out the door is going to be your security suite. Make sure your internet connections are encrypted: SSL, TLSL, that good stuff. Then as we hit on earlier, making sure that you always have a way of updating the device but don't use back doors. A lot of people think to update your device, you should put a back door access and you can go in and download updates that way. That's not the answer. ELRICK: That's like the back door that they were looking for in Apple like, "Do you guys have a backdoor to get into your device?" No. JOHN: No. That can be a controversial conversation. CHARLES: Yeah, or they're like, "Come on, really. It's okay. You can show us the backdoor." No, there is no backdoor. "I know you have to say that. Blink-blink." ELRICK: That's an interesting problem that you guys are solving on how to update these devices. You guys are essentially hand rolling or developing custom software to do that. JOHN: Again fortunately, we're using a Texas Instruments SSC system on a chip. They provide some core functionality, some core drivers that really help us out. For example, they provide a special bootloader that can really assist with a lot of the firmware download back up framework image checking, that sort of functionality. We don't have to write it all by scratch but we do have to write the logic to make sure that the device does check for updates and it doesn't forget to check in and talk to us. ELRICK: On the cloud side, do you guys have to write any custom software to do diffing, to make sure like -- Oh, do you diffing? Or do you just update everything all complete, like once you're updating, you're going to get a brand new update or do you diff and say, "You only need this." JOHN: Since we're working on the system that we're using, it just requires a fully-compiled image that gets installed by the bootloader. We can't really send just a patch to one part of the firmware, if that's what you're asking. CHARLES: But I assume there must be some state that's on the Knocki itself. Just even the credentials for the local Wi-Fi network, what devices it's connected to, part of the system is updating and part of it is not, I assume but how do you make sure that that state is compatible with the new firmware? JOHN: Yeah, that's another great point to keep in mind. The way we keep most of, we call it nonvolatile memory, every time the device reboot, it's going to forget about everything that was stored in RAM so we need to have somewhere in nonvolatile to store these things. We have a file system on the device that we can create files with different device configurations, algorithm, settings, Wi-Fi credentials, that sort of stuff. CHARLES: That file system, is that anything that we would even be familiar with like ZFS or is it just a custom file system that you've written or that you found on GitHub. JOHN: No, fortunately this is just a standard FAT file system. We do have some creature comforts there but that's not necessarily the norm. CHARLES: You heard it here. Is that FAT16? JOHN: No, it's FAT32. CHARLES: FAT32, described as creature comfort. JOHN: Yeah, we have a different perspective of creature comfort. CHARLES: There's a couple of things because immediately, what this brings to mind is for people who are familiar with Ruby on Rails, they have this concept of migrations, where you're migrating the schema of your database and as you have to transform the data from one format to the other, you're running these migrations. One of the things that's nice about that is if, let's say I have some system that is at Version 1, but let's say, I have one of the devices that hasn't taken an update, it starts at Version 1 and it needs to go to like Version 100. But you could have 10 format changes in between there. Is there a way to handle that case where you're basically incrementally applying a bunch of transforms? JOHN: Yes. That's another great point. We take this on a case-by-case basis. Fortunately, being a small relatively simple system, there's not a whole lot of state data to keep track of. But to handle that situation, we've written are own OTA server-side software that manages the devices sending updates -- CHARLES: Acronym alert, OTA? JOHN: I'm sorry, yeah. Another acronym, OTA -- over the air updates. That's our slang for remotely sending firmware updates. CHARLES: Sorry to interrupt. It's just we have to unpack acronyms. JOHN: No, I'm sorry. I use a lot of jargons here. CHARLES: You know what? The thing is, so do I and I just never even notice it. JOHN: To handle that scenario, the way we handle it, our cloud knows what devices are out there and what firmware updates we've sent out to it. Furthermore, when the device checks in with the cloud and ask, "Do I have an update available?" It also tells the cloud, "By the way, I'm running Version 1.0." The cloud knows, if it's on Version 1.0, there's going to be some incremental changes that need to be made before we get to that last update and we can apply those changes incrementally. CHARLES: I see. I feel like we've touched on so many of these concepts that are universal to development but only projected into the hardware space. We've talked about dynamic allocation of memory and data migrations and it sounds like what you're describing in a way with OTAs is continuous delivery, where you have some way of automatically pushing out an update and all the stuff that's involved in that. It's just really cool to hear to view through such a vastly different lens than what we're used to. ELRICK: We've been talking a lot about communication between devices and back to the cloud in things of that nature. Does that play into the conversation around decentralization of IoT infrastructure and what does decentralization of IoT even mean? JOHN: Decentralization as a new methodology or ideology that a lot of people are adopting, I shouldn't say new. It's been around forever but the idea is from a high level, looking at the internet, most of the internet is access through some central, server is hosted on you name it -- XYZ cloud hosting provider. The way you do your URL DNS resolution that goes through centralized DNS servers that say, "You want to look at Netflix?" Netflix is stored over here on this AWS server farm. Decentralization, the idea is we don't necessarily need to talk to this DNS server and talk to AWS just to get content from specific providers. If you look at IoT for example, a lot of times in our case, we want to tap three times on the table and then later on, it will do the cloud, send the message and then turn on your Philips Hue light bulb in the living room. It would be great if the message could just go directly from Knocki to the Philips Hue light bulb, rather than going to our cloud, on some centralized hosting provider, then to Philip Hue's cloud, on their provider then out to the Philips Hue light bulb. Those are some of the really popular technologies that's a lot of people are talking about that really take advantage of the concept of decentralization. But it does -- CHARLES: Let me understand because why these would be necessary. When I get why it's compelling, if I want to have my Knocki talking directly to my Philips Hue light bulb without getting your servers involved, without getting Hue's servers involved, it seems like it's going to be a lot faster and just a lot more robust. There's just less links in the chain but it presents its own problems, like on both ends of the conversation between the Knocki and the Philips Hue, how do they agree that this is sanctioned by a user? That's just leaps out. That's a hard problem to solve. ELRICK: That you use some sort of like public-private key type of encryption to say, "It's me. Am I allowed to do this?" CHARLES: How do you decentralize that? JOHN: Well, I'd like to preface this by saying I'm not an expert on that particular subject but the goal is, if you're familiar with the bit torrent protocol and how it keeps track of a lot of different peers on a network using distributed hash tables, the idea is if you know at least one other person on the network, that person can say, "There's some other people that you may be interested in talking to, that may actually want your message. I'm just a bystander on the network and I don't really need your message but this guy is interested in it." In our application, that would be our server. We have to ask our server, who's out there that wants to hear what I have to say. The server is going to say, "Knocki 123, this Phillips Hue is over here at this address, this unique resource identifier, he's going to be very interested when you have two taps or three taps on the desk so just go ahead and talk directly to him. You don't need to talk to me." There's a lot more of that goes into that about making sure that the network can heal itself if somebody goes offline. But as I said I'm not really an expert in that subject. CHARLES: Right, but it really is compelling. Would you then, maybe have some device that was just kind of your coordinator in your home or multiple devices that would act as these bit torrent trackers? JOHN: Yeah, I think -- CHARLES: Or would the devices themselves actually be able to do that, like the Knocki could actually participate in the conversation about what other devices there were in the home. JOHN: Exactly, yeah. I think in a true peer-to-peer network, any peer can talk to another peer and eventually learn where the other node that they want to talk to is. You don't have to talk to any one particular person but you can ask anybody and they can tell you how to talk to the person you're looking for. The really big advantage to decentralization in my opinion is security. A lot of times if everything is controlled through one central point, that's one central point of failure. If someone DDOS's your cloud service, then now your entire network of devices is offline, just because one location got attacked. If it's a decentralized network, there's no one central point of failure and it's very, very difficult for someone to attack your network. CHARLES: Right, that's true but the tradeoff then is complexity that your decentralize network has to agree, somehow come to some consensus. It's very easy to generate with consensus when you have one process or one point that's driving everything. JOHN: Exactly. Another big tradeoff is ownership of the data and enterprise today are really big revenue your point for a company is being able to have ownership of data and extract meaningful insights. But if your device doesn't talk to your central server every time I want to do something, how does your server know everything that your device is doing and you lose a lot of that data. There's a tradeoff there in how you're going to get the data you need to run your business but also let your device run autonomously on decentralized network. ELRICK: Do you think that this is going to be helpful or harmful to IoT? What's your views on decentralization? JOHN: I think it could be very powerful. Right now, I'm not aware of any products that are really using a decentralized architecture for IoT and the main reason for that is companies and developers are a little slow to adopt it because they want to have that ownership of every data packet that goes to the network. They own it. They can see it. But I think in the future, people will start to realize that they can still get the data that they need to run this business. They can still have visibility and control over the network the way they need to run their business without controlling every single packet. When that happens, I think it's going to be a revolution for the internet as a whole but it's really going to revolutionize IoT and devices will get lower power. They'll get faster and they'll get more secure. CHARLES: When you say being able to get the data that they need, is it just being able to asynchronously spool off the data later? I guess I'm trying to understand how they get the data if it's never talking to some central servers? Or is it just you will get the data at the time you want it or there will be some delay? I assume you can also have your server being part of... I don't know. I'm just curious how you see that playing out. JOHN: I think, every developer is going to have to tackle that on a case-by-case scenario but take for example a big brand smart thermostat company. They have a device that's going to control your AC heating and air and the house and it also collects a lot of the data from when you're home and when you're using it to be smart and adjust the temperature at certain times of day even when you're not home. Again, I don't work for any company that does that and I don't know how they're doing their devices under the hood but traditionally, they tap to a centralized server and they send a lot of this information whenever it's happening, always to the server. Every time the user adjust the temperature, it sends an update to the server and says, "The user just updated." In a decentralized network, these devices can just talk to themselves and say maybe periodically or every day and it'll just send one update and say, "The user adjusted it." You can still talk to a central server but it doesn't have to rely on the central server. CHARLES: Right. It's just what we call, an out of band process. JOHN: Exactly, not mission critical. CHARLES: Okay, I got it. Talking about the decentralization and interacting with other devices, how do you manage the ecosystem right now with Knocki? It's a general purpose interface to rise. It serves really the role of a keyboard or a mouse or some way of controlling other devices and other systems. I assume that in order to do that, you have to understand the capabilities of those systems or maybe you don't. How do you integrate these two devices? Let's go with the thermostat and the Knocki or maybe one that you're more familiar with that you've done. Do you all have to write the integration? Can a third party write the integration? Or is there some way to automatically discover and map the existing inputs of the device. I feel like we've got all these new devices are coming out day to day then and now, there's more and more permutations in which to confine these devices into a coherent system and I'm just curious to hear about that integration story from your perspective. JOHN: Certainly. If we want to configure our Knocki to tap three times and turn on our Philips Hue light bulbs -- I keep using Philips Hue just because that's what I've been actively working on lately. We currently rewrite the integrations in our own backend so the user pulls up the mobile app and says, "Knocki on my desk, every time I tap three times, listen to this Philips Hue," and then we have an integration where in the mobile app, they can essentially set a lot of the parameters that a Philips Hue light would use based on API that Philips Hue would provide us. That's the way most integrations are going to happen with third party products. They expose an API and we can write a little module and the user can configure that API. CHARLES: I see and as far as making affordances for third party people, if they want to change the behavior or add like intelligence, obviously they can configure it from the app but if I want to say add behaviors or something like that. JOHN: When you say add behaviors, you mean add new -- CHARLES: I mean like, rather than turning the lights on and off, say I want to strobe the lights or flash the lights, maybe I'm someone who's running a theater or something and during intermission, I want to knock three times to flash the overhead lights. I don't know if that's something that your integration with Hue could do but if I want to be able to add that. JOHN: Okay, I see your question. We try to enable as much of the products functionality as possible through our own integration on our mobile app but say, you're a hacker and you've come up with your own smart light that turns on any sort of party mode and flashes different colors whenever you want and your Philips Hue or any other smart light just can't quite do what you wanted to do. In the future, our goal is to have an open API that people can access and they can hopefully control their own homemade IoT devices. CHARLES: Now, what about for existing ones. You can definitely flash the lights with the Philip Hue but you're going have to have some custom software to do it, right? Do you see what I mean? You have to send a series of messages to it in sequence. JOHN: In that scenario, we currently don't support that and don't have a plan to support that. In our research, that's a really small use case of people that would be interested in that. Also, it's difficult now if we wanted to do some sort repeated command, you knock three times and then every 30 seconds, it's going to send a command in your light bulbs. We have to be careful about having processes that run away and you have a bunch of CPU power forever in the cloud. We may include features like that in the future. I think the most likely path for that sort of stuff is we'll have an open API that people can direct Knocki's inputs to their own server and then their own server can flash their Philips Hue lights as much as they want. ELRICK: Is there any standardization between the communication and what these API supposed to look, like the communication between devices? anyone can have an API, expecting one thing and someone that's writing software to communicate with that, wouldn't have to go look it up. Do you know of any standardization? JOHN: Yes. I know there have been a couple of companies out there trying to put a standard on the market and I think a standard would be a great idea. ELRICK: Yeah, I think so too. JOHN: It would be wonderful if we could just write generic control structures or information flow structures and anybody can hook their stuff up to it. As far as I know, I haven't seen any that really fit the bill. CHARLES: It feel like there's something that programming systems like software developers have been chasing for a long time is to have some distributed set of peers that they can look each other up. You can discover the capabilities of a thing without ever having to even know about in the first place. But I haven't really known that worked really well and hit that sweet spot. I'm thinking of DCOM and Java. There's like Java distributed beans or something like that. You have this idea of these objects in the cloud, which seems kind of analogous to what we're talking about now, except we're talking about actual devices, rather the software devices but who knows. Maybe it'll pan out where we'll have some standard for discovery and integration. JOHN: It's interesting that there hasn't been one already. You look at IoT and it's really ripe for standardization because a lot of the communication between devices takes the same format. You're generally just passing a small message saying, on or off or, "I read this temperature at 75 degrees. Who knows, maybe someone will solve it. CHARLES: Yeah, maybe so. Maybe the folks at Kasita. They're active integrators. They were on the podcast two episodes ago and one of their challenges was getting all these 30 things to talk together well. Maybe we can follow up with them and if they could have a standard, what they would like it to look like? JOHN: If they get on that, I would love to hear what they were working on. CHARLES: I think, maybe they mostly, have a wish list. It is like, "I wish it did this. I wish it did this. I wish it did this." ELRICK: Maybe we need to have like a 10-way podcast. It's like IoT companies and we can hash it out like the TC39 of IoT on the Frontside Podcast. CHARLES: Right, and then everybody punches each other. All right, well thank you so much, John for coming and talking with us. It's always fascinating. You can find Knocki at @Knocki on Twitter and Knocki.com. It's a great product and like I said, always a fascinating conversation so thank you so much for coming on the show. JOHN: Yeah, thank you very much for having me. It was a great conversation. CHARLES: With that, we'll say goodbye. Thank you, Elrick. ELRICK: Thank you. CHARLES: We are, as always, the Frontside at @TheFrontside on Twitter, Frontside.io on the web or just drop us a line over email, Contact@Frontside.io. Thanks everybody.

Mike North: @michaellnorth | mike.works Show Notes: 00:51 - Transitioning from CTO to Independent Trainer 03:37 - Customizing Content and Developing Curriculum 06:37 - Bringing a Developer Into the JavaScript Ecosystem 12:47 - Training Developers with Non-Traditional Backgrounds 16:56 - Keeping Up with “Fifth Gear” 19:27 - Developing Frontend Masters Courses 22:40 - “Progressive Web Apps” 34:37 - Web Security Resources: LinkedIn's REACH Program IndexedDB Transcript: CHARLES: Hello, everybody and welcome to The Frontside Podcast, Episode 79. My name is Charles Lowell, a developer at the Frontside and your podcast host-in-training. With me today is Elrick, also at the Frontside. Hello, Elrick. ELRICK: Hey, what's going on? CHARLES: Today, we are going to be talking with Mike North, who is doing all kinds of interesting stuff as per the usual so we'll jump right in. Hey, Mike. MIKE: How is it going? I'm glad to be here. CHARLES: Last time that I saw you, I think it was about a year ago at the Wicked Good Ember Conf and we were standing on the beach, drinking scotch and talking about Fastboot but you were doing something completely and totally different then than you are now so I was wondering, we were talking the conversation before we started rolling, that your role nowadays is independent consultant and personal dev trainer. I was wondering if you talk a little bit about that move from the CTO role that you're playing at your old company to kind of moving into that independent trainer, like why and how. MIKE: Yeah, I do remember talking about Fastboot at Wicked Good Ember. It feels like things have moved quite a bit since then. I have always loved teaching developers. When I've been a team lead, it's the favorite part of my job just because I get profound satisfaction out of helping people get over these hurdles that most of the time took me a much longer time with blog posts and podcasts and incomplete examples and libraries that were out of date and Stack Overflow with half answers. I've decided to dedicate myself to trying to make it easier for people in an increasingly complex web development world to wrap their head around everything. While I was a tech lead or a CTO, I always had to split my focus between helping developers grow and something else. Oftentimes, that something else was where the deadlines were and the time pressure was. It felt a little bit like I was driving a car that only had first and fifth gear where you're like on the bleeding edge of open source and what was the latest commit to master and [inaudible]. Then like, "Oh, let's be extremely patient with this person. They've never seen promises before because they came from another programming language. Let's help them digest this at their own pace." It's this slow and patient process of building up from the fundamentals and then the bleeding edge is like, "Let's use Babel Stage 0." It was very hard for those two aspects to exist at the same time in myself so I decided I'm just going for the training side. That's really all I do these days. CHARLES: It was so, but now would you qualify that as the first gear or the fifth gear? MIKE: That's the first gear. It gets you off the ground. It takes you from stop and gets you moving and then you have to develop your own expertise beyond that. But I like to think I'm developing a really, really excellent first gear. Today for example, I'm converting a bunch of Python developers at LinkedIn who are basically the ops team. I'm teaching them Ember and JavaScript at the same time through a series of about 20 exercises over three days. That process is many weeks long without assistance so this is like, "Let's get rolling much more efficiently and quickly," than via DIY approach. CHARLES: Now, do you find you have to custom-tailor for the environment or the developers moving from like someone coming from, say C# would have a different experience than someone coming from Python? MIKE: Absolutely. When I have my material, I have sections that I can drop. If you are a C# developer, I do not have to explain conceptually what 'async' and 'await' mean. You've been working with that for a while. I probably throw up a little example in C# and then the equivalent in JavaScript to sort of create a bridge from your existing expertise into the JavaScript world. Another one -- this is very true -- is teaching Ruby developers how to use Elixir. You don't have to say, "This is a router. We have controllers. There are actions and controllers." There are so many parallels that really it's more useful to help, rather than teach things from scratch to create connections back to the expertise they already have so they're not starting from zero and they can say like, "In the Ruby world, I would think of doing XYZ." Now, I have a map in between that and this new thing. CHARLES: Obviously, there's a lot, a lot, a lot of languages and environments that you could transition to, probably more than matches your own personal experience, in doing that frontline development. What kind of research do you have to do to develop a curriculum for, say someone coming from Clojure or someone coming from Scala or something like that? Maybe that never happens. MIKE: I have a pretty, pretty broad background. My entry into programming was a subset of C and then I graduated to C++ and Java and Ruby and I used to do ASP stuff. I've written iOS apps. I feel like I have enough of a foothold into various areas like I know one JVM language. That is usually enough. If you're running a lot of Clojure, I can at least speak Java to you because odds are, you're working with that and you're seeing that and you know it. Oftentimes, I have what I need. There are situations where I can borrow something in a very cursory level. Not to rip on Scala but I have not found it valuable to make connections to that particular language for clarity and [inaudible] but I have used Haskell before and I'm not a Haskell developer but it is a pure functional language. When trying to help people understand how is this different, then the JavaScript got them running where the Ruby ends up running. It's useful to use something like that. It's a very small language, very simple and you can wrap your head around the basics. ELRICK: What are some of the particular challenges that you face when bringing in a developer outside of the JavaScript ecosystem into JavaScript since JavaScript is kind of the Wild West that you can do everything in JavaScript? What are some of the challenges you face in bringing in a new developer from Python or C or whatever that may be? MIKE: You put it very well. It is definitely the Wild West. You can do anything if you have enough [inaudible] yourself and enough power to get serious stuff done. Really, it's like the explosion in number of choices and tools, the explosion of complexity. I learned JavaScript when it was something that you sprinkle on top of your Rails app for a little interactivity, a little animation on a screen or something like that. I was lucky to learn it at that point in time when that was the norm because I've been able to gradually accumulate for more than ten years now. The tooling like using Grunt, using Golf, using Brunch and then stepping up to other more sophisticated build tools. I learned those one by one in the context of real projects. Now, it's like the mountain is so high, people don't know where to start so that's a big challenge for developers. To throw them into a meaningful project like if you asked a mean JavaScript developer, not angry but the average JavaScript developer, they're like maybe -- CHARLES: I should dare to say that the average JavaScript developer is mean. MIKE: A little bit and probably maybe [inaudible] with me as well, depending on [inaudible]. But they're going to spin up some project with webpack and Babel and all of these tools. If that's your first exposure to the language and to working with the language, you're operating in an environment that you don't understand. Research shows that is the less effective option there to slowly building things up over time. I spend a lot of time going back to the basics and making sure we're not working with promises until we've explicitly focused on them, chained a couple together, managed errors and then now, we can work with Fetch. We're not going to jump into that and throw ourselves into this deep end of the pool. We want to incrementally build up skills. It takes a little bit longer but when you have that understanding as you're learning, you get a lot more out of it because anything that you can't get a grip on to as you learn it, it sort of just evaporates into thin air and don't retain that, even if you kind of fill in those holes later. CHARLES: Yeah, it could be so hard too. Actually, this has been an experience that I've been having, I would say almost for the past two years, as the tools advance, not only you are starting from a place of not understanding but the tools themselves do not teach you. I've had two moments where I got really mad. One actually was on an Ember project and one was a project using webpack but it was the same fundamental problem where in one I was actually working with someone who was very new to JavaScript and an error happens and the stack trace was some just big bundled garbage that gave no insight at all. MIKE: In vendor.js. CHARLES: Yes, in vendor.js or in bundle JS. It was like, "How is anyone supposed to learn?" The most fundamental thing about working with Ruby or working with Node or working with anything is you get a stack trace. MIKE: Debugging is really hard. I think it just takes a little time reaching out to people who are experiencing the Stockholm Syndrome like most of the time, JavaScript developer. We all are working with Ember CLI and webpack. I'm not ripping on these tools but we're used to that complexity in our lives. When we see that stack trace, we're like, "Oh, well. I probably need a source map. I'll make sure that that's there. It's natural that I'm debugging a file that the browser is not really seeing like it mapped back to my source code debugging." This is natural to us. But if you put that in front of a developer who hasn't been living under those circumstances, the number of times they raised their hand is like, "What the hell is this?" It is just amazing and it really helps. I've reset my expectations to what a normal programming experience should be and JavaScript does not provide that today. That is really challenging to keep someone in the midst of all that. CHARLES: I feel like it's hard and do you think we'll ever achieve that? Or is it just going to be a constant hamster wheel of progress versus the tooling to educate what progress has been made or to communicate what progress has been made? MIKE: I think the tooling is fine but it's just that we have a gap in terms of learning experience. We just need really -- I'm not voluntary here because I've got a ridiculous backlog -- a couple long tail horses working with vanilla JavaScript, rendering some stuff on the screen, maybe a course of React but no JSX yet, just create component. A couple of things to fill in a gap between where maybe code school leaves off and where you are expected to be by the time you start interviewing for a spot as frontend developer on a team but there's a huge chasm right now. There's the intro guides and then there's professional life and trying to bridge the gap between those is ridiculously a challenge right now due to the huge ramp up of complexity from like, "Let's do some stuff in the console," to, "Running transpile JSX code with async [inaudible]. We've got regenerator in there to polyfill generator functions." There's so much in your average JavaScript at these days. CHARLES: Your work that you're doing at LinkedIn, part of it is trying to bring and train developers who come from more nontraditional backgrounds, including a lot of things like boot camps. What is your experience of their experience coming in? Are boot camps doing the right thing? Are they teaching the right things? Are they trying to kind of parachute them on top of that mountain? Or do you find that they're just at the base camp, so to speak? Because it sounds like your approach is like you've got to really start from fundamentals so that you can understand the layers of complexity if you're going to, someday stand on them. MIKE: I think a lot of the boot camps are doing an excellent job. These days, the employees we have at LinkedIn who come from boot camps, I would bet on them against your average MIT grad every time, just because their education is so practical. It's amazing that in the world of computer science, the stuff that you're taught in school is a little bit farther removed than one would expect, compared to the stuff that we do every day in our jobs -- building real apps. I do not need to know in my day-to-day work at LinkedIn how an operating system works or how to build a device driver. This is a little bit too fundamental. It's the wrong abstraction for practical everyday work for most people. Where in these boot camps, they focus completely on the practical. In fact, I've been fortunate enough to get involved with the REACH program here at LinkedIn, where we hire explicitly people from nontraditional backgrounds like boot camps. They're not all from boot camps but many of them are. We just hired 30 of them in March. The pilot program, I think we've hired two or three in our New York office and it just went really well. It started like, "Let's double down and double down again and double that again." This time, we're doing 30 and I expect there will be a new round next year where we poll even more. The idea is we take these REACH candidates and pair them with a mentor engineer for six months. At the end of that six months, we had to make a decision as to like this person at the level we expect of an entry level software engineering hire. From what I've seen, we're doing really well at preparing these folks and they're unbelievably valuable to the teams that they've been placed in. ELRICK: That's amazing. That's very interesting. Is there a standardized curriculum thing that each mentor will follow to get this person after they entered his REACH program and then ramp them up or is it like each person just goes and looks at what the person knows and then ramps them up accordingly. MIKE: I'd say, it's a mix of both. We have a set of technical trainings for them or we'll have a testing expert from within the company and teach a little testing seminar to them. There's that standardized curriculum there. But the nature of being taught by boot camp or teaching yourself is that you're going to have holes in your knowledge and it's not often predictable where those holes will be. That's why we make sure we do this mentorship very explicitly and over a long period of time so that if it turns out that you never learned about how to work with tree-data structures. That was not part of the go-no/go decision that brought you on but we should probably, at least get you there. At least to the point where if you're traversing a down tree and you're like parent and child, what is this, what do you mean by leaf-level node. This is stuff that is actually meaningful for web developer in some cases. CHARLES: In the context of the work that you're doing with the REACH program but also touching on something that we talked about at the beginning about the first gear and the fifth gear, part of generating a curriculum is still being in contact with what's up in the fifth gear right because ultimately, what you're trying to do is you're working with people who are in first gear or looking to get a smooth transition in the first gear but at the same time, you want to set them up and you want to be in contact for what's in fifth gear now is going to be first gear in five years. How do you feed that in? MIKE: I'm fortunate to have a great team that I work with here. This group that I roll up to in LinkedIn, they're experts and you probably know of like Chris Epstein and Tom Dale and Steph Petter. A 15-minute coffee break with one of these people is enough to keep [inaudible]. Sometimes, it's a little bit like drinking from a fire hose because it's like I spend an hour with a student trying to help them understand like, "This is why a Promise is useful. Here is the callback equivalent," and then now, "Let's dive in to Glimmer. Why this track annotation is the right way to go for automatic updating." It sends me for little bit of a loop sometimes but it is definitely keeping me up to date. The other factor, of course is when you've been doing this for a while. History sort of repeats itself so a lot of the patterns that we're seeing today, I've seen somewhere else. I was working with code splitting when I was writing Dojo JavaScript code years and years ago. I was defining my module layers in a very explicit way. I had to do that. I didn't have done a webpack that would figure out, put these splits are. But I have that experience to look back to and for that reason, it is not often that an entirely new concept comes along. Oftentimes, they're like amazing refinements on things that how to smell like stuff that we've used before in the software engineering world. CHARLES: Yeah or here's something that has never been used, is very prevalent in these other context which we're going to apply here. MIKE: Exactly. CHARLES: And like, "Oh, my goodness. It's a perfect solution." In addition to the work that you're doing with LinkedIn and developing those training curricula and stuff, you're also doing some work for Frontend Masters in an area that's very exciting, I think to me. I'm sure it's exciting to you because you decided to throw a whole lot of time into developing a course for it. That's in the development of progressive web apps, which for me has been like this thing that I'm so curious about but I'm like a kitten playing with a little yarn ball. I want to dive in but I'm just going to tap it with my paws right now. MIKE: Yeah, it's a really interesting area and I think that even if you're not using progressive web technologies today, it's one of these things that sort of reinvigorates your energy for JavaScript's future and what may be possible soon. Steve and I have put together this amazing progressive web app course, which has I think like 18 short examples of iteratively building up a grocery shopping app. If you've used InstaCard or something like that, we start out with app already built and it's like a single-page app as doing everything that you would expect. After a few of the exercises, it works offline. After a few more, you can add stuff to the card and background sync, push it to the API when you come back online. We get deep, deep, deep into service workers. That's one of the areas that my work at LinkedIn and my teaching with Frontend Masters overlaps really well because I've been heavily involved in creating our service worker for LinkedIn.com. I may be able to take some of what we've learned here and disseminate it a little bit so that, hopefully fewer people have to learn the hard way. It's best to keep things simple at first and add on functionality. I'm about to cross like the [inaudible]. This is my favorite just because the example turned out to fit so well and in particular, on Frontend Masters, I think Steve and I have had contrasting teaching styles but they complement each other so well because I'm like the 'melt people's brains' instructor. I love to throw people exercises that are like 120% of what they can do and it's going to hurt, just like when you're lifting weights at the gym, like you're going to beg for mercy but we're going to make you strong. Then Steve, just listening to him, even with I am in the classroom and he is teaching me Electron. He's so energizing and he's really funny too but not in an overtly cracking jokes kind of way. He's just so fun when he teaches. I think it is a really good combination just because things lined up just by luck and through hard work and just the right way out of a couple of important areas. CHARLES: Now, just for people who might not be familiar with the term progressive web apps, what does it encompass? Do people actually call them PWAs? MIKE: No. I'm going to start, though. I like that. That carries very well over a video chat or something. Nobody knows how to spell that: P-U-A? P-W-U-A? It is a rejection of the old idea that in order to take advantage of some web technology, it has to be supported in all of the browsers that we need to support. The idea here is to hold as a core tenet of our design practices, the idea of progressive enhancement, meaning we serve up a basic experience and where we can take on these superhero features, like the ability to work offline, the ability to receive push notifications, we go ahead and do so. If your browser doesn't support this, that's unfortunate. No big deal. You still get a good experience. But if you're using a very recent version of Chrome or Safari or you have a new Android device, these browsers can take advantage of sophisticated metadata or sped up a background process that can serve up data to your app and your app doesn't even know that there's something between it and the API. That is the idea of progressive web apps -- apps that become superheroes where possible and they still work and provide a great basic experience for antiquated browsers like IE8 and Safari. CHARLES: The idea theoretically, you could work without any JavaScript or whatsoever. What's the ground floor there? MIKE: That is ideal. I think server-side rendering, which is what you're talking about there, even if JavaScript is not working, just HTML and CSS will provide a basic experience. That's great but that's not a modern browser technology thing. If you have JavaScript turned off in today's Chrome, like Chrome 60, versus IE9, both of them working with them without JavaScript. What we're really talking about here is app-like characteristics, where we are pushing web technology to the point where you will swear that this came from an App Store. It's on your home screen. It's running in the full screen. You're getting push notifications. It works offline and you can store a large amount of structured data locally on the device. All of the stuff sounds like the list of reasons to reach for native mobile technology because the mobile web is not good enough. But in fact, it has a feature set of this family of progressive web technologies. It's really like a web app that is so good and so modern that it feels and looks just like a native mobile app. CHARLES: That sounds so hard to do right. MIKE: Well, it is now, just because what we have to work with can be thought of it like a basket of ingredients, rather than a solution that we drop in. But over time, as more people start working with these ingredients, I think we're going to see a lot of consensus around the best patterns to use and boilerplate code will fall away as we can identify that the set is in fact commonly needed and not a beautiful and unique snowflake. CHARLES: Because it seems like the thing that I always struggle with is not wanting to put the critical eggs in the basket of a superhero feature or have you being able to provide an alternative if the superhero feature doesn't exist. Some features, if you just don't have it, that's fine. You can turn it on if the capabilities available but certain features are very critical to the functioning of your application. I'm casting about for an example and I'm not finding one immediately but -- MIKE: Offline is a great one. That fits pretty neatly. If you're using an older browser or if you're using Safari, which by the way, I should stop ripping on Safari. For the listeners out there, we saw a commit lend in webkit, where service for APIs are beginning to be stubbed out. No longer do we have to look at length. Service worker, enthusiasm and Safari has got it in the five-year plan. There was motion last week. We haven't seen motion in ages so thank you Safari Team. Thank you. Keep up the good work. CHARLES: Is there a discipline of Safari-ologists who monitor the movement of Safari to bring this news? MIKE: Of course, we monitor it because right now, Chrome and Firefox, they are pretty much hopeful in terms of supporting this modern stuff. Opera supports this modern stuff. Samsung's fork of Chromes support this modern stuff. Especially when we think about the mobile web, you got to worry about Android and you got to worry about iOS Safari and right now, like we've talked about these progressive web apps, you don't get that superhero experience on an iPhone or an iPad. Once we crossed that threshold, this is going to have a breakaway level of adoption because there are no more excuses. Essentially, for a mobile web experience, you can send push notifications to the user. That is huge. That is probably at the top of the list for why some people use native apps, instead of mobile web. The more we can do that, the more we can make it so that a great LinkedIn experience can be delivered to your phone without having to install a binary. I just have to update Facebook the other day and it was over 100 megabytes. Why do we need to do that? You should be able to make it work with less. I'm sure that there's some great stuff in there. Apparently, Snapchat filters are popular but I don't need this. Can we code split that away or something because I don't want to have to download that? I can't even download it on the cell network because it's over 100 megabytes. It's really exciting to see the web start to compete with this heavy mobile experience because now I think is ready. CHARLES: Now, when you talk about push notifications, you're talking about being able to send things to my lock screen. MIKE: To your lock screen while the browser is not on the foreground, while the app is not open. Essentially, you're installing a lightweight process that runs in the background. It receives events that originate from your server and the user can tap on them and then your little lightweight worker process in the background decides what to do when that tap happens, like open up the app, take them to this URL or something like that. That is a game changer. That's huge. Or background sync like the user added some items to their cart and then they lock their phone and now, their plane has landed. That's why they were offline and they get back on the internet and without them having to touch their phone, now we can push that data to the server and everything's in sync, rather than like, "Please revisit your app. We need to run some JavaScript code to flush IndexedDB or API." It still feels like a hack at that point. This is a fluid experience. ELRICK: Wow. This is exciting for me as I don't have any more space on my cellphone, thanks to all the apps that I have to install to do various things on the web. MIKE: You're not alone. CHARLES: Yeah, it's crazy and just the amount of code sharing that you can have, I guess that doesn't happen much these days on the web where you've got these popular libraries out on CDNs so that the chances are that you've got jQuery 1.2.1 on your cache, you've got 16 versions of jQuery so most of your web applications don't have to do that. I guess we kind of do the equivalent of statically linking everything. MIKE: There is a benefit near that where we have imperative code managing our cache, instead of just relying on the HTTP cache or app cache, if you have a vendor.js file that is not changing over six months, there is no reason you should be re-downloading that every time you deploy your app or letting the browser evict that, just because memory pressure is high from Google image search results or something like that. We really don't have much control over it. But with a service worker, we can say, "Hold on to this," or maybe like prefetch the next version of the app so that we're going to show you the old version now but the next time you refresh, here's the new version available instantly. It's downloaded in the background and it's like click to update your version, like it's already here waiting for you. That's huge. That's amazing. CHARLES: That is amazing. Although the complexity skeptic in me is thinking, "Oh, my goodness. Now, we've got all this state that we're storing on the server. We have to have data migrations." We need some sort of migration mechanism for our clients-side state and perhaps some transaction and rollback in case you're not able to successfully migrate your data. It sounds like a lot of fun but I'm just imagining we really are getting started here. Has there been any work on that aspect? MIKE: If you've ever worked with IndexedDB, it does have a concept of migrations. Basically, the data you store on a device has a version and when you read in what's called a file but it's a database, when you read that in, the first thing you do is you basically bring it up to date incrementally. You'll bring it in, you're looking for version nine like your code wants version nine. What you see is version two because your user hasn't been at your site for six months and you're going to take it from two to three to four to five to six. Each of those, essentially constitutes a migration. We just have to apply the same principles of forward-compatible changes. The escape hatch here is remember it's progressive enhancement so if we had to destroy everything, fall back to a basic experience and start from scratch, like discard all of our data, it's really being held there as an optimization. Some people use this immutable caching strategy or basically, like rolling out a new service worker version constitutes for the most part. Any data that wasn't created by a user you're going to discard that and you're going to fetch it new. You don't have to worry about like, "Crap. This six month-old thing is still plaguing half our users and we can't get rid of it," like you can have [inaudible]. But you should really check out this course. It is simpler than you think and what we demonstrate is not a trivial like hello service worker. It is taking in a classic single page app, making it completely offline, having it exist on the home screen and I think the service worker ends up being no more than 100 lines of code. It's not too bad. ELRICK: I'm definitely going to check that out because my progressive web app journey is still on just service workers. MIKE: That's very [inaudible], though. ELRICK: Yeah. I'm definitely checking it out. Sounds like a really fantastic course. MIKE: I've been focusing a lot on this area and another one is security. The reason I picked these two is because developers are not really going to learn about these on the critical path to [inaudible] plus they learn about them the wrong way. As the JavaScript world is becoming radically more complex with each passing year, I've tried to target some of my efforts towards areas where they are not getting as much attention as I'd like to see, just because we have to focus somewhere. Obviously, getting the app out and figuring out how to make the build tools work for us. Without that, we can't do anything at all. One of the courses that's coming in September for Frontend Masters is a one-day web security workshop or we'll do with like cross-site scripting, how to work with certificates because if you start playing with HTTP/2 -- the next generation of HTTP -- you will need to generate some certificates for development at least today you need to. I've seen some amazingly smart developers get this dangerously wrong to the point where they compromise their own machine and anything that's on that machine, just by trying to set up dev environment. Typically, I'm an optimist but when it comes to this PWA stuff and security, I am paranoid. I feel like, we as a community need to get together and have the discipline to brush up in these areas so that as we introduce all of this new stuff, we don't end up opening a bunch of holes. Nowhere near the same rigor as put into frontend compared to backend and now, the line is blurred. Right now, we're server-side rendering so our code is running on the backend somewhere so injecting something can really mess things up in a bigger way. ELRICK: Yeah, I think that's a fundamental characteristic of someone does going to be involved in security paranoia. You have to be paranoid about everything. MIKE: Yep. I don't trust anything. CHARLES: It's important to make those things easy because I'm definitely fall more into the hippie camp like, "Everything is going to be fine. Let's trust everybody," which is I know is totally unrealistic. But then you get into these secure technologies and you learned enough of it just to get the task that you're going to do and then you forget. SSL is a great example. Over the course of my career, I've learned how SSL certs have worked probably, at least 10 times. ELRICK: Right, [inaudible] you had to set it up in production. CHARLES: Yes, exactly and then I promptly forget about it, never worry about it again and then the next time I'm like, "How did that work? What's this trust chain? What?" ELRICK: Exactly. I read a study from Carnegie Mellon a couple of years ago that showed developers observe security best practices dramatically less than the general public and the general public is not good. Do you know what I'm talking about when I say a certificate warning and a browser, there's big scary red screen saying like something is wrong here? Before the Chrome team put some effort into improving that, 70% of people would click through those and proceed anyway. After their improvements, over a third of people still clicked through and that number when you just look at Canary versions of browsers, that number is actually considerably higher close to 50% of our developers. We're trained by every broken certificate system that exists on the internet like the legitimate ones or maybe some things just expired. They're training people to just click straight through these things and as a result, it is terrifyingly easy to mess with people. We have to remember as developers, our machines, those have the private deploy keys and those have the SSH keys to commit code to GitHub, we have to treat that like it's a private data. It's really, really important that we make it easy and that we make sure that that easy path is also very safe. CHARLES: Absolutely. All right. Well, thank you so much Mike for coming by and talking with us. We touched on a lot of subjects but I feel like I certainly learned a lot. MIKE: Yeah, thanks. It's been so much fun talking with you this morning. CHARLES: Anybody who wants to go and check out those courses, they're on Frontend Masters. Now correct me if I'm wrong, you've obviously got the one on progressive web apps or PWAs. If it doesn't work offline, it's faux-PWA. MIKE: Yes, I like that. That's going to become a t-shirt sometime soon. CHARLES: The fundamentals of progressive web app development, which is now released if I understand correctly. MIKE: Members have access to everything, you can watch the raw video now. The edited course will be available later this year. CHARLES: Okay, and that's with Steve Kenny. I am very much looking forward to looking at that and learning more about it. Then you've also got ones coming up in September on TypeScript web security in Visual Studio Code. MIKE: Yep and members can watch that as a live-streamed event. Frontend Masters even ask people to watch the comment stream so you'll have a proxy question asker or hand raiser in the room. It's really a great experience to be part of a live thing. CHARLES: Oh, man. That sounds awesome. Then if you are obviously doing your independent consulting and if people want to get in contact, how would they do that? MIKE: You can find me on Twitter, @MichaelLNorth or you can visit my website, Mike.Works and I have all of the courses I teach and outlines and I can just open up a little chat bubble on the lower right, ask me any questions that you have. I am really passionate about teaching people. If you like that's useful for your team, please reach out and I'd love to talk. CHARLES: Fantastic. Thanks, Mike and thanks everybody for listening to us. If you want to get in touch with us, you can always do that. We're on Twitter at @TheFrontside and email, Contact@Frontside.io. Thanks, Mike. Thanks, Elrick and I will see you all later. MIKE: Thank you so much. ELRICK: Bye.

Jason Jaynes: @jasoncjaynes Jeff Wilson: @ProfDumpster Show Notes: 00:53 - “Professor Dumpster” and Founding Kasita 05:33 - The Startup Industry 07:45 - Building the Kasita Team and Creating the Design 12:25 - Integrating Devices 16:33 - Challenges of Building These Ecosystems 24:36 - Controlling the Ecosystem: Will there be third-party developers and applications? 30:16 - Device Cohesion and User Experience 33:23 - Privacy Resources: Data for the People: How to Make Our Post-Privacy Economy Work for You by Andreas Weigend Kasita is hiring! Transcript: CHARLES: Hello, everybody and welcome to The Frontside Podcast, Episode 78. My name is Charles Lowell, a developer here at The Frontside and your podcast host-in-training. With me today are Jeff and Jason from Kasita. Now, Kasita is one of the most exciting products that I think we've gotten to work on here at Frontside in the last five years. We're going to be just talking about it because, I think it touches on a lot of the aspects of what makes software development and startups and just the emerging economy exciting. I'm really thankful that we get to have you all on the podcast. Welcome Jeff and welcome Jason. JEFF: Thanks for having us. JASON: Excited to be here. Thanks, Charles. CHARLES: Now Jeff, you are the founder of Kasita, the CEO and I believe your official title over there is 'Professor Dumpster.' Maybe you could actually unpack for us a little bit of what does that title mean? How did Kasita come about and what is it today? JEFF: A couple of years ago, I did a radical, social experiment around housing. I went and sold everything I own for a dollar an item out of a 3000-square foot house and moved into a 33-square foot used trash dumpster for a year. The idea of that project was to live in 1% the size of an average American home and try to use 1% the energy and water of the average American home. The project took a little bit of a twist, you might say and about part way through it when the dumpster started getting tricked out, I started thinking about the whole nature of housing and how we need to do something different and how that grand future probably would not be a gated community of dumpsters. CHARLES: Now, I assume you cleaned out the dumpster before you actually went to live in it. JEFF: Yeah, it was a fixer-upper. We give it a bit of a scrub and did some testing to make sure there wasn't anything nasty left in there. That went for about a year and a couple of months after that, I actually first set down with Jason because he was the only person that I knew in the entire startup scene, in the entire world. He said, "Wilson, you had some crazy ass ideas like this dumpster thing you told me about. This one might actually work, this Kasita thing." Here we are today, we're working together. CHARLES: Wow. This was something you just did on a lark. You didn't have the idea of starting this business but it was actually through the process of actually living in this dumpster for a year that the idea emerged or was there a master plan going in? JEFF: I don't know, Jason do you remember any kind of master plan when I first told you about the dumpster? JASON: No. When we first met to talk about the dumpster, it was an early morning, I believe in 2010 or 2011 and you're incubating the idea. At that point in time, there was nothing on your mind or you aren't looking towards the future of housing at all. You were just trying to figure out how you were going to move into a dumpster and people thought you would be crazy. Of course, I've validate it and I thought people would think you would be crazy. CHARLES: That is a pretty radical idea, the future of housing being 1% of what it is now. How do you see that playing out? How is that possible? How do you shift people's mindset away from that? JEFF: One of the bigger things we're trying to do with Kasita, there needs to be a massive shift in the wider way that we live in our homes. As everything else is moving towards on demand and as a service and as everything's being sort of productized, those are some of the core ideas behind Kasita. We think about Kasita a lot more like an iPhone or a Tesla than we would think about it as a single family home or an apartment block or even a micro-unit. That's why Jason and I are standing together here today is I represent a lot of ways, a kind of vision and origin story of Kasita but in a lot of ways, Jason represents the future of the software and integrated IoT that's going into these things. CHARLES: There is definitely a lot going into these things. I remember when Jason first started telling me about it because it is like an iPhone or a Tesla but, I think especially the Tesla is a great analogy because you have not just like a normal software or even really a hardware project, you've got architectural concerns. You've got manufacturing concerns. You've got, I assumed geopolitical concerns in terms of the politics around zoning and housing and real estate, all rolled up into a big startup. When I think startup, I think let's get a web application up and running and we're providing some service. This is cross-cutting at least five industries, it feels like if not more. I'm curious, what's been the experience in terms of wrangling that aspect because I think it is very unique in a startup today but it got me wondering is this going to be the normal in five years? JEFF: We've seen a movement recently in the venture community. Even a few years ago when we first started raising money was highly-regulated industries are hard, hardware is hard, "Thank you very much. We're going to go looking for our next two Stanford computer science dropouts to shove into a wee work and not have to deal with all of this kind of stuff." I think I've seen a shift to where people from the individual level up to the folks funding these things, see the massive opportunity in highly-regulated complex problems like housing and you're right. Jason and I are looking out over our shop floor here where we've got guys out there that are plumbers or traditional electricians all the way upstairs here to folks that have been mayor pro tem of large cities with PhDs. Bridging all of those individuals into a startup culture and then looking at the complexity of the landscape from a regulatory standpoint, autonomous cars are a breeze relative to the kind of complexity we're dealing with. CHARLES: Did you know this complexity walking in or was it a classic overoptimism? JEFF: No, it wasn't classic overoptimism. I'm always asked, "Are you a designer? Are you an architect? Are you a real estate developer? Are you a technology guy?" and I think if I would have been any of those besides a guy living in a dumpster, I wouldn't ever been crazy enough to try this. It's one of our core precepts as well. Jason had never worked with IoT stuff before. Our head of manufacturing used to build LEDs for Philips. Our quality guy inspected Cadillacs. Our manufacturing engineer built Boeing jets. The ideas that we're not pulling a lot of people from these traditional industries, we're pulling smart people that are passionate about our mission and to solve this, what is really a Rubik's Cube of a problem. JASON: Yeah, I think the other thing to add to that that Jeff is not getting himself enough credit is that from very early on, Jeff always looked at Kasita as a product that was going to incorporate multiple disciplines. He was very careful in how he orchestrate it and built the team to make sure that he was bringing the right expertise and the right areas together and then forcing those different disciplines to figure out how to meld and work together to build the Kasita. But the Kasita was from the beginning just about building a micro-urban home. It was about building a product of which part of that was a home, where people live obviously, but there's a whole lot more to it that we're working towards. I think even go back and Jeff, it might be relevant for you to talk a little bit about the approach that you took to just create an initial design for Kasita, which I think is revolutionary in itself. JEFF: A big part of our DNA was product from conception. When I was living in the dumpster, I recruited a couple of the top architects in the country really to help me turn that dumpster into a home. The way you're trained in architecture school, I think a lot of folks come in there with Buckminster Fuller kind of dreams and you're told pretty quick that you better bring things up to code and you better make things that sell or you're not going to eat when you get out of here. The idea was that we would start off with a product designer and not design a home. The kind of struggles in the dumpster taught me that we needed to go at a different approach so I went and recruited an industrial designer. One of the requirements for that person that he or she had never designed a home. This person had lived under a staircase and never designed a home so I said, "You're perfect." CHARLES: I like that and I'm curious, Jason from your perspective, what was it like to have gone through this? It sounds like what you're doing is asking people to bring their expertise but not their set of expectations like the industrial designer. What was it like for you coming primarily from the software development world to step into this pan-technological realm and what was that experience like and what were the things that stretched you and you found surprising? JASON: I think early on, I realized that it was going to be a bit more challenging maybe than I thought. Really, what it required was me to think outside of my discipline. Obviously, not only from the perspective of what we were doing on the IoT frontend, how we were melding software and hardware together but then going all the way over to the physical building structure and thinking about on a weekly, daily, hourly basis on how we are interacting with the other disciplines. An early example was, and this is one that I remember that's quite funny is one area that we wanted to make sure that we had covered in our research and understanding from IoT perspective was smart locks and how we were going to provide a smart locks for the data. We went out and did a lot of investigation, brought a number of leading smart lock solutions into the lab and tested them and narrow our list down. Then I recall vividly walking over to the architects to excitedly tell them we had selected our smart lock that we were going to use. They very quickly inform me that that lock wouldn't work because we needed a mortise lock and not a standard door lock. I realized that you can't work in a vacuum and just solve your problems. You have to be working together to make sure the solutions and the products you're selecting at work in accord with the overall design. That's continued to manifest itself. Every day, I'm down on the manufacturing floor, working directly with the electricians and others to make sure that our equipment is placed properly, where are we going to place our equipment, how are we routing around plumbing and pipes and other things that exist there and how are we locating things properly. It's an ongoing experience, which has definitely taken me out of my traditional software role but it's done so in a very exciting way and I've enjoyed it. It's just realizing that you have to actively be communicating across the organization with all groups and really, you can't take anything for granted. CHARLES: The number of different disciplines and technologies is really staggering, even if you limit it to just considering the set of devices that you're integrating. I was actually hoping we could talk a little bit about that. Now inside each Kasita, at least the ones that you're building right now, how many different devices do you have? How do you take all these different devices and turn them into a product or integrate them into something that itself is one product? JASON: If you were just to look at the technology bill of materials, what the products are that we're incorporating into our current Kasita design, there is around 50 different products and product parts that we're bringing together to build out the technology solution. If you narrow that down to what the end user is actually seeing and looking at, there are about seven noticeable products that the end user would see or they would recognize everything from a Sonos connecting amplifier to an Amazon Dot to a Nest Thermostat. Obviously, getting to that list of bill of materials and deciding on that 'subassembly of technology pieces,' took us quite some time in a number of iterations and a lot of outside engagement and talking to experts and trying to decide what were the best devices to bring in. But the other side of the equation was something that we kind of decided very early on in the process and kind of thinking the world of first principles was that, we wanted to make sure that Kasita was the primary interface to the user. We didn't want somebody else sitting between us and the end user. We wanted to be able to work with other products but we still felt at the end of the day that the end user, when they were living inside of a Kasita, when they were controlling the Kasita, when they were changing the state of the Kasita, they needed to go through our interface. With that as an initial first principle, you can begin to imagine that all the other parts of the system architecture and the way that we design things, the way that we select products and built things, it begin to derive themselves. Everything from that, immediately we needed an app and lo and behold. We were able, fortunately to work with you guys, the Frontside, to help us get our initial app concept up and going. It went from there and I can talk more about it. CHARLES: I think I really like that as a first principle. I really just want to inject a vigorous sense of agreement because I think it's so important, especially when this is the place where you're living. You want to imbue that inhabitant with a sense of ownership and control. I don't know if you would be able to do that if there were a bunch of different touch points and it didn't feel integrated under one product. In other words, this is my home, this is my Kasita. Is that the idea behind making sure that there was really only one interface? JEFF: We prefer to say 'Mi Kasita.' CHARLES: I love it. JASON: Absolutely, that's the idea. I think from a consumer perspective, if you've ever personally gone out and ventured through the halls of Home Depot or Best Buy and purchased some smart products off the shelf and brought them into your house and try to get them up and running, you very quickly learn that. It's not only challenging to get these devices connected in a way that you can control them but there's also this notion of there's an app for that. Every physical device you ended up putting in your how, has its own app for control and that becomes very overwhelming in a very short amount of time for the user. We did not want that to be the case with the Kasita. We wanted them to walk in the door from day one and immediately feel at home and feel like they have complete control of the Kasita, in much the same way when you go purchase an iPhone or you purchase a new Garmin watch or you purchase a new Android device, you're up and running with that ecosystem and you're interacting with that interface. We wanted people to be interacting with the Kasita interface to control their home because that's part of the product. CHARLES: I like that. It must present some unique challenges because I think you said it best. Every single device that you have comes with its own ecosystem and that ecosystem has its own APIs, its own web interfaces, its own applications and though there are walls around those ecosystems, what are some of the challenges you encounter in trying to punch holes through those walls so that you can hand information and control from one ecosystem to the other while providing a seamless experience to the user? JEFF: When you're talking about that, Jason one of the things that is often left out of this equation is at this specific point in space-time, it's very difficult to do that. But then to have any sort of semblance of planning for the future and future-proofing the system as developers usually call it, one of the reasons why you don't see a lot of Nest thermostats in multifamily development is because a developer knows that they're not going to ever have to replace a normal light switch. If it's a Lutron switch or if it is a Nest thermostat at some point, it's going to have to be replaced. Not only the physical replacement of the stuff but from a software side, making sure that we can continue to communicate with these devices in the future, I think is a big problem to solve. JASON: That's absolutely right. I think very early on, we recognize and realize that we were going to have to build software and a component that acted, if you will as a gateway for sitting between the end user and the end devices and facilitated the control of the end devices. Obviously, being able to accomplish that, one of the challenges is and I think, Charles you've seen this in your world because I know you've got experience with IoT is this whole proliferation of standards and protocols like if we're going to talk to the lightbulb or we're talking via Z-Wave or ZigBee, or do we have to go through a Philips Hue hub because that's the only way to actually communicate with it. Is there a separate way via Thread or Bluetooth you communicate with this device? In a very quick fashion, you get to this point where you can imagine that you've got a physical hardware controller that has four different radios in talking to four different device types. One for talking to Z-Wave, one for talking to ZigBee and it becomes overwhelming. We did a lot of research across the protocols that were available, mapping them across the devices. Early on, we were excited about the potential of Z-Wave but more recently, where we've shifted our attention quite honestly is looking for devices and device manufacturers who see the opportunity and Wi-Fi enabling their hardware devices and then providing either direct control of those devices in an IP-centric way over a local area network or even through the cloud. What that affords us back to Jeff's future-proofing concept is if you have Wi-Fi up and running and the device can get on the Wi-Fi network and there's a way to communicate with it, then it makes it a lot easier for us to sit between the user and that device and send commands and control that device. The other side of that, which I think continues to be a challenge and will be a challenged for the foreseeable future is a lot of the device manufacturers to the point that you brought up are still forcing you to go through the cloud to communicate with their devices. They don't allow for a local area network communication directly with the device and there's good reasons for doing that. But what that means is if you lose internet connectivity, you no longer have control of that device. CHARLES: Obviously, you've got probably pretty strict criteria about what it takes for a device to be integrated with Kasita. Is that a nonstarter right there? JASON: It's actually not. A nonstarter with be the device communicates via protocol that we can't interface with or the device works over a Wi-Fi network but has no API for controlling cloud or local. The third piece of that equation and fundamentally is the final nonstarter and really probably should be the first one and it's one that we take into consideration every time is that there should be a physical override for the user if internet connectivity is lost. What I mean by that is if we select a smart switch and the smart switch goes offline and there's no more connectivity, the user has still be able to walk to the wall and press the power button and the light should come on. There always has to be an ability for the user to fall back to the same old fashioned physical control in the absence of Internet connectivity or local area network connectivity. But the primary things are ability to fall back to physical control, ability to communicate over Wi-Fi or standard IP-based protocol, then the third one would be some form of API access, either remotely via the cloud or locally via the local area network. CHARLES: Wow, that's actually a great list. It's got me wondering, obviously you've encountered devices that have fallen on both sides of that divide. Do you feel like that's just a blip and we're going to be trending more towards devices that are happily and easily integrated or are we still seeing some moving and jostling as people maybe try and corner little parts of the market and make their device deliberately make it not easy so that you'll try and force people into that ecosystem? JASON: The latter, however we have two guerillas in the market right now that I think are helping drive the other direction in the way of Amazon and Google with Google Home and Amazon Echo. What they're doing is they're saying, "If we sit in the center and one of the interfaces for voice control for the user to control their home, then we're only going to work with devices that we can communicate with and that we can control through the cloud," and quite frankly, what that does is it puts the burden back on the device manufacturer. You could actually say three if you threw Apple in there. I don't want to leave Apple out with HomeKit. But my point is that the device manufacturer now has to find a way that the end device can either communicate via standard TCP/IP network-based connectivity that we all know and love from a developer community perspective or they have to insert a hub into the equation that can handle that form of communication and then communicate over its own proprietary wireless connection, which is in the case of Philips Hue, it's exactly what they do. JEFF: I would draw analogies here to some people get really tired of this, particularly the real estate people of me talking about the iPhone but that kind of leap into and integrated piece of hardware and software. There were certain things happening in 2007 that didn't make the iPhone or something like it, something that might happen but something that had to happen. This kind of cold death to the universe that we could see with all of these walled-off ecosystems, go in their directions and iterating into a space to a nobody owns anything and nothing talks, I think Kasita is a solution to that to where we're looking like combine all this stuff under one roof and build a single user experience, much like not having to pull your Palm Pilot out of one pocket, you're Rio MP3 player out of another and you're your Razor or whatever it was out of the other like integrating into a single experience, rather than a sort of convenience, which is what a lot of the IoT spaces right now in these walled-off ecosystems. CHARLES: That actually makes a lot of sense and clarifies it in my mind quite a bit. It clarifies one thing but then, immediately raises new questions. When the iPhone first came out, you had a set of basic integrations between your MP3 player and your web browsing and your calling and calendaring, so and so forth. Then, I don't know what was it like, a year and a half later, they actually came out with an SDK so that you could actually develop apps -- third-party developers could actually develop. Sell and distribute in apps -- to the iPhone. We're all really happy with the way that worked out. I guess my question is does this analogy carry forward then also for Kasita? Is there a future where you have third-party developers who are actually selling integrations or apps that would run on this integrated IoT product that is Kasita or am I stretching the analogy too far? JASON: I think the analogy is good with the exception that we're not looking to control the entire IoT ecosystem in a way that Apple maybe had look to control the mobile phone ecosystem with providing all of that in one box and the iPhone. We want to work with numerous hardware providers and even from that perspective, numerous folks that want to provide interfaces into our system. As we develop an architected Kasita technology system, we've taken an API-first approach and that's allowed us to build our user application layer right on top of that API but in the future, we see the opportunity to work with third-party developers to extend that, up on that and build their own interfaces to the end user. Then on the other side of the equation, if you think about what's actually controlling the devices, we're architecting that system in a way that a hardware manufacturer could take an SDK and add Kasita support for their product directly in and make it plug and play when it gets to the Kasita. We definitely see the opportunity, Charles to reach out and allow everybody to be part of this. We consider it quite frankly, a necessary thing. But we don't also want to pretend that we would look to control the whole ecosystem because we just don't have that level of scale, if you will. JEFF: And you know -- CHARLES: Not yet. JEFF: Yeah, and we try to keep our ego in the dumpster, so to speak as well. CHARLES: What would a third-party app even look like in the context of Kasita? Have you thought of like what are some things that you might be able to do? JEFF: If you don't want to call it directly an app, I think the first stage -- Jason and I haven't talked about this -- maybe more like an Alexa Skill to where you can have the Kasita do certain sets of tasks around a particular experience, which we're already building into the system the idea of moods but I don't know in terms of apps. JASON: Yeah, it's actually a really good idea. Even though we haven't talked about it, it always scares me a little bit when my boss is coming up with ideas on the fly that we have to implement but -- JEFF: But actually we will have our first -- we're going to call it a skill app, a Kasita skill app. We'll be releasing that say, October 1st. CHARLES: You heard it here first, folks. JASON: To take Jeff's idea a little further, I think that is an interesting concept when you think about the Kasita as being an end product and you provide interfaces whether it's the ability for people to write skills that tie into the Amazon Echo or an IFTTT-type capability. The Kasita, as a whole can be controlled -- all the lighting, the sound, all the different temperature, etcetera -- so now you're asking end users to write skills, to control the entire state of the building or of the home and not just doing it on a one-off basis writing skill to turn this light on and off or set the thermostat to this level. You basically box all of that together and make it much easier for people to get from Point A to Point B through our system. JEFF: Could you say that we're turning the entire Kasita into a board for people to play with, like treat the Kasita as your breadboard? JASON: I think there is some opportunity for that to the degree that will allow the user to have that much flexibility on the hardware side. I think it is still up for question but I think there's a lot of opportunity there, Charles and not only inside of the Kasita but then you can begin to see other applications as Kasita begin to multiply and people use them from many purposes. Let's take a sample of somebody owns 10 Kasitas and they use them as Airbnb properties and they allow users that live in Kasitas to come in for a short period of time into their Kasita and bring their Kasita profile with them. Immediately, they can make the Airbnb Kasita feel exactly like their Kasita feels when they're at home. Those are some interesting opportunities and ways that we see this technology potentially evolving. CHARLES: So it will have the same moods, the same behaviors. Any customizations or third-party extensions would also be in effect provided they were software-based? JASON: Yep. CHARLES: That would actually be quite amazing. I guess the other question I have in terms of hackability of Kasita is we're very interested in the IoT space and very interested in these products and we have some side projects here at Frontside also like I do a bunch of hobby stuff at home, where I try to integrate a bunch of these things. But one of the things that I really like about what you all are doing is that it's very much 'omakase' in the sense of there's an option of 10 smart locks, there's an option of this thermostat, there's an option of a million different devices but what we've done or what you've done is selected ones that we know are going to work well together. We've built the software, the control systems, both computer control systems and human control systems to get them to work together as a cohesive product. I would love to do is say, "I would just like to buy that product for my house," even though my lame tinkerings with smart switches, smart locks and audio controls and lighting, which are fun and gratifying the first few times but they don't really play nice together, give you that super sweet feeling. JEFF: This goes to the overall philosophy of Kasita. We want a turnkey, one-click housing solution. Not only for finding you a place to rent so that you're not fishing around on Craigslist for roommate or having to pay some outrageous fee in New York. You don't have to go mattress shopping. At some point, you should just have to show up with your iPhone and your toothbrush. When you start thinking about the technology inside, it's almost like folks don't really care what kind of Foxconn chip is in their iPhone or even if it was Foxconn that put it there, they just want it to work and they want it to be seamless and turnkey. It sets up a whole philosophy around, not only our smart kid in the Kasitas but it shouldn't even be a smart kid anymore. At some point, it should just be an experience so ultimately, what sort of UX inside of the Kasita are all of these things bringing you. I shouldn't have to really look at a blue glowing dot that lights up every time I walk by it to be at a comfortable temperature in my house. I shouldn't need a black tube over on my desktop that I yell commands at. I just talk or it should anticipate those actions. That's a future that I look forward to in Kasita to where we move away from having to tinker with devices and even knowing what those devices are to a true-like depth of experience. CHARLES: I like that a lot. Now, one thing that we haven't covered. We touched on it a little bit at the very beginning of the show when we talked about people feeling in control and feeling like they're truly the owner of the space is the issue of privacy. Obviously, there's a lot of a user's behavior that's going to be passing through software channels as their intentions move through the devices in the Kasita. Of course, all of these devices, they have their own ecosystems, their own vendors so how do you ensure that people's data is going to be protected, especially as it moves through potentially a bunch of different public clouds. JEFF: Yes, we gave a lot of thoughts to this. Actually, Jason put me on to this book called 'Data for the People' by Andreas Weigend. We took some inspiration on that, from that and set out on what we call it the four cornerstones of this future of the connected home. Those are agency, transparency, security and then the actual benefit that you get from this home. I gave a talk at South by called, 'The final frontier of AI is in your living room.' If that isn't black mirror, creepy enough to attract enough people, I don't know what is. In that talk, I won't take them out of order. First, we need to make sure that we're focused on transparency. Do people know what's actually being collected on them? I've been toting around my iPhone for 10 years. I'm pretty sure they know everywhere that I have been since then. I'm not really all that sure. Second, agency. Can I actually do something about it? Are we allowing people the ability to switch off, switch on, control where that data goes? Then third, security. Are we providing another level of security above what you would get out of the box? I'll let Jason talk about that in a minute. Then, the last is benefit. Am I getting ads? Am I getting a slightly better news feed focused on ads or am I getting my rent subsidized? Am I getting a better user experience, better sleep within the connected home? Those are the ways that we think about that in a bigger level. CHARLES: Is the idea that there's no benefit than it's exploitative? You want to make sure that there's benefit? JASON: Yeah, I think that onus is if you taking individual data and using it, then the onus is on you as a data collector to try to provide benefit back to the end user. If you can't do that, then I think the question should be why are you collecting the data in the first place? our goal is really looking at it from the perspective of if we know when users are turning lights on and off and what they're setting the temperature in their house to and when they're going to sleep at night, when they're waking up because we know when they turn everything off and turn it back on -- JEFF: Or where this things on the floor are from the vacuum robot. JASON: Yeah, exactly. If we have insight into that information, how are we taking that information and combining it in a valuable way that benefits the end user? I think that's the first question that we have to ask when we start looking at the data that we're collecting. But at the same time as Jeff said, that data collection really has to be based on this notion of agency, transparency and privacy or security. An agency is simply I have control over whether my data is collected. Transparency, from the perspective of I understand how my data is being used and where it's being sent and then of course, security, I know that my data is being securely transmitted and stored. When you think about security, we spend a lot of time thinking about not only the data at rest -- once it's been collected is it properly being stored and encrypted and protected -- but then how is that data being transmitted and are we putting the proper fail safes in place to make sure that somebody else can easily gain access and take control of my home and of the things that are important to me by finding back doors into the system and ways to breach them? Those are the cornerstones that we think about and we put first and foremost in our mind as we build out our architecture, build out our system and as we begin to take that data and to turn it back in useful and interesting ways for the end user. CHARLES: I think that's really important. I think it's a great comfort to hear that you all have a framework for thinking about this so that it's going to be integrated into every aspect of it. I think it's just so important, especially when it's something as critical as the space in which you're living. It's good to hear that it's not just an afterthought but that it's something that's been integrated from the start. Well, Jeff, Jason thank you guys so much for coming by and talking with us. I really think that Kasita is an exciting product and I think that it was an exciting project, certainly for us to get to work on, even though we were only seeing a very small sliver of it. We still got to perceive the whole enchilada that you guys were working on and see that just what a unique startup that really is, not just you're moving outside of software, integrating a bunch of different devices, integrating that with a unique home that's going to be designed, architected, manufactured and then thinking, then even rolling it up a degree further about how is this going to be integrated into the urban spaces in which we live. I hope that we see more startups that really engaged all those different disciplines. I think that with the technological changes that are happening, that's more and more a possibility. The price on software, the price on materials, the price on these smart devices is all coming down so it really enables people to take on scopes that might have been just completely impossible, even with someone who's overly optimistic. I hope that people look to it as an inspiration and it really was a great project for us to work on. I also understand that if someone does want to jump into this space and get involved, you all are hiring. JEFF: That's right. We are hiring for a broad range of positions. We're expecting to be doing a lot, more hiring soon. You can go to Kasita.com/Work and at the bottom of the page, you can also see that we have an open house here in Austin every Thursday morning from 9:30 to 11:30. The folks can come in and check out the crib. CHARLES: All right. Fantastic. I certainly really enjoyed getting the tour the space, what was that? Back in March? When you revealed the baby units? JEFF: Yeah, it was March at South by. CHARLES: Yeah, it's really something to see. If you are in Austin or you live here, take the time, go see it. It's really cool. With that, I guess we'll wrap it up. Thank you everybody for listening and as always, you can get in touch with us at @Frontside on Twitter or Frontside.io or send us an email at Contact@Frontside.io. Thank you all and see you next week.

We discuss what the deal is with Canadian whiskey and then talk about why we like The Economist. Your pals, @cowboyd (https://twitter.com/cowboyd) & @cote (https://twitter.com/cote).

In this episode, we talk about IoT: what's coming, why we're intrigued, and how we've already started it incorporating it in our office. In the next episodes to come, we will be having guests on the show to take a deeper dive into this technology. If you have any suggestions or know people we should reach out to, please get in touch! Transcript: CHARLES: Hello, everybody and welcome to The Frontside Podcast, Episode #77. My name is Charles Lowell, a developer here at The Frontside and your podcast host-in-training. Today, I have with me two other developers here at The Frontside. This is going to be a Frontside-only podcast and we're going to be introducing a topic that hopefully we're going to be podcasting a lot about in the coming weeks and months just because it's something that's kind of grabbed the interest of the office and seems like it's something that needs to be talked about. Hello Joe and hello Elrick. JOE: Hello, Charles. ELRICK: Hey, what's going on? CHARLES: Everything, really. Today we're going to be talking about the Internet of Things and we'll be talking a little bit about how we came to be interested in this topic and why we think this topic is important. Let's talk about why this topic is important. I think that this is a very important topic because IoT is only becoming more and more prevalent. It's emerging from the status of being this niche or boutique or very esoteric technology that's only worked on by a very small group of people to becoming very, very open and available and accessible so that anybody can buy a Raspberry Pi or an ODROID or Arduino and slap some Linux on there and connect it over the internet to a bunch of different things and the space of creative possibilities is just exploding. For me, it's very similar to where we were in the early 80s. You know, I see these IoT devices as being the hobbyist's computers, the Z80, Apple IIe, the Commodore 64 and that the people who are hacking on those things 30 years ago are going to be the people who are now leading the tech space today. I think another big and relevant analogy is web technologies. There was this inflection point where web technologies became very open, accessible, available and the people who were in it ended up being able to ride that wave for 10 years to where we are now. In both of those examples, we had the hardware and the PC revolution where the computation was distributed across a bunch of these different devices. Then over that time, we saw a migration over to the cloud and these web technologies where everything was centralized. Now, I actually think that there's a pendulum swinging back where we're actually going to see more and more computation distributed amongst physical devices, except this time, it's not going to be manifest as a PC. It's going to be manifest as these networks of devices that are just all around us. I really do think that we are on one of those watershed moments where these distributed networks of tiny devices are going to be the big next platform that when you invest in it now, this is something that's going to yield dividends for the next 20 years. I think it's an important topic but I don't think we had a well-crafted thought about it but we just kind of stumbled into the space. I was thinking we could start a little bit by talking about how we got into this and how it captured our imagination. If you rewind the clock to the stone age of 2015, I think it was the end of 2015 and it was Christmas break, that's often a time when people go and they hack on individual projects and Brandon, his project that for whatever reason, he decided to take on was he was really into Hue Bulbs at the time. We had Hue Bulbs around the office and we wired up some demos to control them from a website. He decided he wanted to take those Hue Bulbs and make them so they were accessible from our Slack. He built a server in Elixir because he also wanted to learn Elixir because if you're having fun in hacking around, it might as well pick up as many new things as you can. He built an API in Elixir that talk directly to the Hue Bulbs and the Slack integration that talk to the Elixir API and we actually are able to control all of our lights purely from Slack. We could turn them all on, we could turn them all off. That was great but then as we began to use it, we were wishing that we had control over our lights from our phones. We wish we had control over them through the website. I think, Elrick, isn't that was your first contact with the Frontside, wasn't it? ELRICK: Yes. That was my first contact with the Frontside. I was working on the lights app. I initially started working on just the user interface and bringing some different animations and working on the actual experience and the user story on that side about controlling the lights and what particular things you needed to do in trying to craft a UI around that. That's what I initially started. CHARLES: That was really fun. ELRICK: Yeah, that was really fun. That just started progressing more and more. As you said as we started to think about how could we access these lights from different places, using different devices and then that's how we stumbled into the Internet of Things. CHARLES: And it turns out, there's actually a lot of tech in the form of platforms out there that have been developed to help with this, although I would say that the water are still pretty murky as to kind of the best set of patterns to follow. ELRICK: Yes. JOE: That's hard to find information, especially with regard to design patterns. Since we've been working on this light thing, there's been so many times I've Googled and looking for prior art and found none or next to none. It's very much the Wild West. ELRICK: Yeah, because it's like going from a point where you're controlling one piece of data per se, like you have one sensor that does one thing. Now, it's starting to grow until you can have one sensor that can do multiple things and send it across different types of data and then how do you structure that data, how you capture it, how do you hold that state somewhere and it's one to one source of truth. It's just going to be the Wild West of how do you manage this, how do you structure it. It is definitely growing and changing constantly. CHARLES: I think one thing that is difficult is it feels very much like they're aligned in terms of silos. For example, the Hue has the Hue Bridge, which is capable of talking to the light bulbs and then they also have an API which is under development by which you can connect publicly to servers hosted by Philips to talk to the hues inside your office but if you want to integrate your Hue API like we did with Slack or with your iPhone or maybe some other device that you're trying to control, it becomes a little bit more difficult. You have all these vendors like Nest, MyQ and there's a whole bunch of lines like doorbells and smart this and that and everything and they're very good at talking. They have an ecosystem, this large vertical ecosystem, assigned with each one but actually getting cross cutting communication is a problem that I think is something that we've had to deal with and it's very, very difficult where we want to start having these devices talking to each other. ELRICK: Yeah, that area right there is ripe for innovation. I don't know the names off the top of my head but I know that there are people trying to make a smart hub per se. You can think of it like Jarvis from Iron Man. You buy that thing, you put it down in your house, you tell it all the devices you have and that takes care of all the communication between everything. There's definitely an area there that someone can step in and say, "You know what? I figured it out and here's your Jarvis Box." JOE: We're starting to see stuff like that with Alexa and Google has something similar. That's a little scary to me. I think that the one thing that needs to be made clear is when you're talking about these silos, it's a very good point because we think they're decentralized. We think these things are decentralized but in a way, they're not yet. We don't have peer-to-peer communication necessarily like Hue. They're going to public API but you're going through their ecosystem. You're passing through their lens, so to speak. We think Slack has distributed teams but there's a centralized server where those messages passed through so how do we break from that into full decentralization? CHARLES: Right, I know that's – ELRICK: The Jarvis Box. You could probably have a server at your house that keeps all your data there and then it spits out what it needs to spit out to the IoT server somewhere if they're doing some collection. When you leave your house, to say, "I need that information to come back to my cell phone now." Maybe in the future, you'll be able to control that, either from your house or just send out the pieces of data that you need and the centralized stuff, you can just keep at your house. CHARLES: The whole question of ownership is one that I feel is something that we have not addressed head on. Everybody is just rushing forward with how do I implement this, how do I get it done and it definitely is worth taking a step back and understanding who owns the things that I'm working with and that I'm inviting into my home. I think that smartphones provide a great example of how it can work really well for the consumer. I think certainly, in their inception I think this is mostly true if you have an iPhone. Most Android devices, you actually own that piece of hardware and the things that you install on it are very much controlled by you. I think that Apple especially, gets a big shout out for making sure and putting in those safeguards so that anyone who's participating in the ecosystem has to first acknowledge that the data is going to be owned by the user. I think that's maybe a little bit less true than it was back in 2009 or whatever but I think that there's definitely a lot of thought that went into that upfront, that I worry isn't going into with Alexa. Is Amazon protecting? Is there an understanding that if you're participating in that ecosystem that ultimately, the thing is owned by me? I feel the same way about a lot of these AI and robots where it may participate in the conversation but who is it really serving? Is it serving you or is it a proxy to serve somebody else like a Google or an Apple or an Amazon? JOE: I may just be a pessimist but I think it's safe to say that it's almost always the latter when money is involve. ELRICK: They had some situations arise where the powers that maybe we're trying to get the actual recordings and different things as Alexa is always on. Let me turn mine off because she's going to say, "Oh, did you ask me for something?" I have one sitting right here in front of me. They have been in situations where people had said, "Because that's constantly recording and that recording is going somewhere," and then if situations have arisen, they said, "We want that recording," and then Amazon is like, "No. We're not going to give you that recording because that is private information." They're trying to find a way to get around that and what laws and things are going to come out of this area that we're in right now, it's still unforeseen. But I think that companies that are in this space, know that the future of their company rests on them protecting that data and user data because if you don't, then people will sidestep and go elsewhere. CHARLES: Right. In so far, they hold that as a value. In so far, people are conscious of those concerns. If that's something that people are willing to pay money for, then you've got a market driving force pushing you in that direction. But if people don't care, they don't think and they're just like, "Whatever. It's cool," that's not going to be something that a business is going to roll into their product because ultimately, if people care, then it'll affect their bottom line. If they don't but it won't and they're going to act in their own best interest. ELRICK: True. CHARLES: I do worry that there needs to be a social awareness of what kind of powers these devices actually will end up having over our lives and hopefully, those will guide it but you're absolutely right. ELRICK: True. I view all of this IoT stuff and data is not too far off of what people do on Instagram per se like you have your pictures, you can either post crazy pictures or you can post casual pictures. How you use the power that these IoT devices are giving you is essentially falls into your hands like what am I going to send across this thing. I think that hopefully, the power falls into the user's hands and they empower people with these devices and not make them feel like a prisoner in their own home or car because this IoT things are popping up in vehicles now. If you step into your car, you start talking and your car is listening. If they go from it like the same way we approach our applications and such and say, we're going to empower the user, I think if these IoT companies take that approach and learn from the mistakes that were made in software by not empowering users, then after a couple years they're like, "Oh, my goodness. We need to empower the user." When Steve Jobs was preaching about this in the 80s and everybody thought he was crazy. Don't fall into our mistakes. Empower the users and I think that this technology in this space would just keep flourishing if they do that. CHARLES: Absolutely but it is going to take a generation of engineers to make sure they're always pushing in that direction, a generation of users who don't just wait for companies to hand power to them but demand it. ELRICK: Demand it, yes. CHARLES: Yeah, demand it and a generation of business owners who are going to listen and think about the long game and realize that that's the path to long term health and viability. ELRICK: Yep, even outside of the whole privacy thing where it's like there's too much data being sent out. People are building just cool stuff with IoT that doesn't really send that much data outside of normally that we do. Even on our phone, people use GPS all the time and that is sending data about all your locations, where you are, what restaurant you're at, what bus stop you're at, what bus you're on, what plane you're on and people are building a lot of cool things, just even using that. I saw the other day that someone had a bicycle, it has GPS and lights and gyroscopes and all kinds of stuff in that bicycle. When you're riding, the lights will go off and say, "It's time for you to take a right." It will blink in a certain sequence or take a left. It register your speed and it all comes back to your phone so it's not too outside of the norm of what we do on a regular day. There's people building things just in that sweet spot per se with these IoT devices that are building some pretty cool stuff. JOE: It's a very good point because Slack doesn't have to be centralized. It can be peer-to-peer. Hue doesn't have to be centralized outside of having a bridge on your local network. We don't really need to be phoning home for all of this stuff and if we move towards like a true decentralization, we don't need trust at that point. A company has our best interests at heart if we think about it as your trust ideal to remove the need for involving third party in the first place. CHARLES: Yeah, so what would that look like? I'm going to fast forward a little bit because we were a little bit further along on our journey and we've been experimenting with Amazon IoT services and we've been maintaining our own APIs to control our Hues directly. While they're still going through the bridge, it's not incorporating any other ecosystem but we are still routing all of this stuff through this low level Amazon infrastructure. There's a class of problems that that solves which it does help to have those primitives to be able to access your IoT devices through a firewall, to have them and be able to, at least have a known way to update themselves and distribute software to them. There's these fundamental infrastructural problems but at the same time, Amazon doesn't have any access to that data that's moving through their land, so to speak. What they're essentially doing is leasing you a railroad but they don't have new visibility into what's contained inside the cars. JOE: Do you know that? CHARLES: I actually don't know that because of course, it's through the Terms of Service. ELRICK: Who reads EULAs? They're too long. JOE: I think it's more often than not, people are going to use convenience over privacy. CHARLES: That's true so it is in keeping with what I understand of other Amazon services, which do have those guarantees. I don't know in particular for the Amazon IoT. But let's talk about that a little bit. Let's talk about a little bit about our setup and why we went to using Amazon IoT services and what it provides for us. ELRICK: We decided to use the Amazon IoT platform as a means to allow us to one control the bulbs from anywhere, to get access to them and then also to be able to distribute that change to anything we want. Coming through IoT or coming through their platform, when a change happens, you don't necessarily just have to send it to our one set of bulbs. You can send it to anything you want. You can send it to a phone, to another application somewhere, to a database. It gives you the ability and the flexibility to distribute that change or that state change anywhere. CHARLES: Which is I guess getting at the heart of it is actually managing this distributed state beast of a problem and really, the AWS IoT just helps you get your foot in the door. There are still a lot of cans of worms that are involved once you get there but for the first point that you have said, I want to unpack that a little bit because it's a problem very familiar to us but might not be to the listeners, you've got the set of devices and they come up, they connect to your Wi-Fi and that's fantastic and they can talk to other things on your Wi-Fi, on your local network and can discover services there. But what if you want to control them from outside like I want to send a message from Slack and have it affect the lights in our office. You've got to move through some public cloud to do that because Slack servers are not on our local area network. What you can do then is have essentially one thing that the IoT services provides is your device comes online and it immediately calls home to a generic location and opens up, what is in practice a web socket. You can program in whatever language you want but that's probably the analogy that's most familiar to everyone. It basically connects a web socket that then you can send messages to it in real time so any time I want to connect to that, I can do it and I don't need Hue's API. I don't need Slack's API. I can just talk to one API which is the low level Amazon -- AWS IoT API -- and I can send real time messages to my devices. That's a huge problem solved right there. But it's hard to maintain that infrastructure yourself. We could write our own AWS IoT but then we'd probably host it on AWS anyway. JOE: The real world is not a JSON Blob. That becomes a problem. In college, I took a course where we programmed robots for the majority of it and what you quickly find out is that you can't count on revolutions of a wheel or what have you. The world is imperfect. Keeping a state is one thing but keeping state reflected back and keeping state up to date is where the challenge has been for us. CHARLES: That is right because you've got this highly distributed systems. That's kind of a second class of problems that it attempts to solve for you. You got these highly distributed set of devices but even if the connections are 99.9% reliable, sometimes they're highly latent. You can't control the latency on the connection and sometimes, it fails altogether, which can affect one, how do I even read state from these things. Is the button pressed? Is the button not pressed? Is the light on? Is it off? Is the wheel spinning like you said? Or is it off? These are things that you need to know and then you need to react to those changes like, "We're spinning at 90 RPM. I want to bump it up to 10. How do I get my system to converge on that desired state based on my current state?" It's hard because you don't know all of the demons of distributed state management are in full like they have ripped off their masks and they're roaming about. ELRICK: Yep. I saw them introduced something the other day but I haven't had time to dive too deep into it. It was something called Greengrass that it will continue to gather and allow you to utilize your devices locally and it will keep all that data and then it will do the diffing, let's say when you connect back online until what your old state was and what the new state is and then go about updating everything. JOE: That could be very useful. ELRICK: Yeah. It just got implemented probably three weeks ago or something like that. It's inside of the IoT platform. I just clicked in and they said, "We have a new feature now called Greengrass," but I haven't got time to dive too deep into it but like you were saying, state management is something that's extremely difficult, especially across a distributed systems. They know it's a problem and it seem to be addressing that problem and trying to make it simpler for people and give you these tools to say, "Here are some stuff that you can leverage," and a lot of that is great. CHARLES: I think that's an excellent point and I think that it's also worth mentioning too that there's two sets of state that you have to manage. There's the runtime state, which controls the flow of data as your system operates. Then there's the static state of just what is the code that's going to run on this device. Let's say, my robot or my button that's got V1 of the software, that all it does when I push it, it rings a bell. That's V1. I want to add this awesome feature to this button that when I push it, it rings a bell and it also pops open a Topo Chico from the refrigerator or something like that. The question is how do I get that software from my laptop with that Topo Chico enhancement all the way to my button, which is what essentially amounts to being across the internet inside this private network. In the current state or when you're first starting out hacking, let's say this is based on a Raspberry Pi, I just burn a new Raspberry Pi image with my new software with V2. I walked over and I stick it into the Raspberry Pi and that doesn't really cut it. That does a great job but now, I want to turn this into a business and I want to have 20,000 of these things installed or let's think big like every home in America gets one. Every home in the planet, I want two billion of these type of devices. What happens when I come out with V3? ELRICK: Then you can either go the route of hiring -- CHARLES: Hiring a favor. ELRICK: -- Technical folks to go out, to update all your Topo Chico poppers or have your users struggle to do it or what we did, implement Resin. Let Resin update your Topo Chico poppers around the world. CHARLES: Right. There are a lot of problems in terms of static state management, runtime state management, peer-to-peer communication and problems of resiliency and robustness. I'm hoping that we can discuss these over the coming weeks and months because each one is a topic in of itself. ELRICK: And offline management too. CHARLES: And offline management too, there you go. There's another one. There's a lot to explore, a lot that's unknown and there might be people who have answers to all of these and there might be papers on them but they're buried in weird corners of the internet. I'm hoping that we can fill the podcast with a couple of guests to come in and talk about these different things. ELRICK: Yeah, that would be fantastic. CHARLES: Yeah, I'm really looking forward to it. ELRICK: I started playing around with Watson IoT. It is an IoT service that allows you to leverage the natural language processing and computing from Watson. It's pretty awesome. CHARLES: Wow, that is really cool. ELRICK: That's another space of IoT that we can explore and hopefully, we can explore over the next few podcasts. CHARLES: Yeah, awesome you all. Well, I think that's about it for this episode. Thank you, Joe. JOE: Thank you, Charles. CHARLES: Thank you, Elrick. ELRICK: Thank you, Charles. It was fantastic. CHARLES: And I look forward to hacking on the lights with you guys. That is always one of my favorite things to hack on. I don't get to do it enough but I think we're going to try and have a big throw down on state management on Friday, right? ELRICK: Oh, yeah. CHARLES: It is going to be exciting. It's going to be super nerdy and we'll let you all know what the outcome of that is. See you all next week. As always, please don't hesitate to get in touch with us. You can get us on Twitter at @Frontside or send an email to Contact@Frontside.io. We always love to hear from our listeners. Take care!