The Frontside Podcast

In this episode, Austin Parker, Principal Developer Advocate at Lightstep talks about the OpenTelemetry Framework, which is an observability framework for cloud-native software and a collection of tools, APIs, and SDKs. You use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) for analysis in order to understand your software's performance and behavior.

In this episode, Puneet Lath, Director of Research and Development at Expensify, talks about the unique way Expensify is using open source with their products by not just open-sourcing software tools but also open-sourcing the front end of the product itself. They are rebuilding their products on React Native to be fully cross-platform and doing so in an open-source manner. All code is public, and anyone can see it and contribute to it.

In this episode, Steve Pereira—the founder of Visible—talks about how his unique approach to mapping helps customers get products out of the door fast and efficiently. Maps flow to build alignment, clarity, and confidence, and Steve focuses on two primary areas to drive exceptional business outcomes: Flow and value. Value stream thinking and methodologies, augmented by continuous improvement and performance.

Data is at the center of everything we do. Yet, how can we trust it in a world where more "organic" food is consumed than produced? In this episode, Jason Kelley—the Global General Manager for Blockchain Services at IBM—talks about how data trust and transparency are applied to COVID-19 vaccines. 

"Java has done an excellent job at ruining types for everyone for quite a while—explains Marcel after describing the tech pub scene in London—but it's important to know there's more than one kind of type system." Along with Shane, they outline what's exciting about ReasonML and their experience with new languages and tools around types.SPECIAL GUEST:Marcel Cutts | @marcelcutts: Founder of Asgard Tech, ReasonML London OrganizerShane Wilson | @wilsonashane: Founder of Peyk Inc.SHOW NOTES00:38 - ReasonTwitter: @reasonmlDiscord03:25 - BuckleScriptOCaml06:01 - Reason + BuckleScriptTypeScript16:07 - Reason: Interoperation & AdoptionClojureScriptToolchainSyntax30:00 - Operating at the Compiler Level vs the Run-Time Levelppx (pre-processor extension)34:29 - Last thoughts on, and why use Reason?Flow44:43 - repkgsknitLerna 

Monorepos are the new muse of library maintainers, but what happens when your project grows past 100 packages in the same repo? What about thousands? Rush.js was created for those cases, and Pete—who started the project while working at Microsoft—is here to tell us about it.SPECIAL GUEST:Pete Gonzalez | @octogonzDuring the day, Pete works at HBO in Seattle on their streaming media apps. Prior to that, he was at Microsoft for 9 years, and before that, he worked at various consulting companies. A long time ago he was a cofounder of Ratloop, a small company that makes video games.SHOW NOTES01:24 - Rush.js: What is it and what is it for?Rush on GitHub04:47 - Problems with Managing Large CodebasesRush Stack: provides reusable tech for running large scale monorepos for the web07:22 - How does Rush provide a solution for build orchestration?13:34 - Rush Stack Opinion: How to Lint, Bundle, etc.16:53 - Using Rush Stack: Getting StartedThe pmpm Package ManagerYarn Plug'n'Play24:27 - Getting Technical About VersionsPhantom DependenciesDoppelgangersPure Dependencies32:47 - Thoughts on Monorepos36:30 - Getting Started (Cont'd) + Efficient TypeScript Compellation43:28 - Does Rush have a size limit? Is it for bigger or smaller projects? Both?44:34 - Using pieces of Rush in non-Rush projects? 

In this episode, Charles and Taras discuss "big ideas" and all the things they hope to accomplish at The Frontside over the next decade. Please join us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at contact@frontside.io. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer. This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. Transcript: CHARLES: Hello and welcome to The Frontside Podcast, a place where we talk about user interfaces and everything that you need to know to build them right. Today, we're going to talk about big ideas and the future of Frontside. TARAS: Yeah, starting with is Frontside is good idea? CHARLES: No, we're going to just talk about how do you know that an idea is good? We've touched on it a couple of times before. Like how do you, how do you go about validating a big idea, how do you discover a big idea? What do you do? TARAS: And then, even when you have big ideas like big tests, what does that mean for the world? How do you make a big idea an idea that a lot of people like and agree with and actually use on day to day. CHARLES: Yeah. It turns out that it's not easy. There's a lot of work involved with that. A lot of it crystallized around the conversations we're having about what exactly is big test and recognizing that big test isn't really a code base. It's not a toolkit. Even though it does has aspects of those things, it really is an idea. It's an approach. It's a way of going about your business, right? TARAS: Yeah. Especially when you put big test functionality in place, when you start doing big testing and then you put together things like using Mocha and Karma, big tests in that kind of test suite is really just like interactors and the idea of big testing. There's nothing else. All the interactors do is just give you an easy way to create composable like [inaudible] objects, so you don't have to write -- you have the components but you don't have to write selectors for each element in the component, especially if gets composed. But that's like a very small functionality that does a very specific thing. But big test itself, it takes a lot of work to actually -- we had this firsthand experience on the project we're working on right now. We are essentially introducing like Ember's acceptance testing but for React in a react project and having to explain to people what is it about this that actually makes it a really good idea and having people in the React world see that this is actually a really good idea. It's kind of incredible. When you actually try to sell something to somebody and convince somebody that this is a good idea is when you realize like how inadequate your understanding of the idea really is. You really have to start to break it down and understand what is it about this that is a really big idea. CHARLES: Yeah, I completely agree 100% because to be clear, we've actually been doing this now for two years almost. So, this is not the first React project where we've put these ideas in place. But I think in prior examples, we just kind of moved in and it's like we're going to do this because this is what we do. And we have firsthand knowledge of this working because we've operated in this community where this is just taken on faith that this is the way you go about your business. You have a very robust acceptance test suite. And because of that, you can experience incredible things. When you and I were talking before the show, we were kind of commenting on inside the Ember community, you can do impossible things because of the testing framework. You can upgrade from Ember 1 to Ember 3 which is a completely and totally separate framework, basically. You're completely and totally changing the underlying architecture of your application. You can do it in a deterministic way and that's actually incredible. TARAS: And what's interesting too is that the React core team kind of hinted a book at this also in their blog post about fiber or moving to fiber because one of the things that they talked about there is that knowing how the system is supposed to behave on the outside allowed them to change the internals of the free app framework, specifically about test suite for the React framework, but it allowed them to change the internals of the framework because they were testing kind of on the outside. The system kind of is a black box and that allowed them to change the internals and the test suite essentially stayed the same. So this idea of acceptance testing your thing is really fundamental to how Ember community operates. But other communities have this as a big idea as well. It's just applied in different areas. CHARLES: Right. And so applied to your actual application, this is something that's accepted in one place, but it's not an accepted practice in other places. But you can make your argument one of two ways and say, "I have experienced this and it's awesome. And you can do architectural upgrades if you follow the patterns laid out by this idea." And that works if you have a very, very high, high trust relationship, but you don't always have that, nor should you. Not everybody is going to be trusting you out of the box. So, you're going to have to lay out the arguments and really be able to illustrate conceptually practically how this is a good idea just so that people will actually give it a try. TARAS: And it takes a lot. One of the reasons why it was easier for us to introduce big testing to the current project we're working on is because we were able to write, like we've done the implementation for this in a previous project. So when we were convincing people this is a good idea, we're like, "Look, your test suite can be really good. It's going to be really fast. And look, we've done it before." So the actual process of convincing people that a big idea is a really good idea is actually kind of a complicated process that requires a lot of work and partially requires experimentation. You have to actually put an implementation in place and show that you're going to have to build up on your successes to be able to get to a point where you can convince people that this is actually a really good idea. People who have not heard about this idea, and especially people who might have counter, like they have people of authority in their community that have counter views. For example, quite often when it comes to big tests, when you bring up big tests, people will reference a blog post by a Google engineer that talks about how functional testing or acceptance testing is terrible. And for a lot of people, a Google engineer means a lot and the person makes really good points. But that's not the complete idea. The complete idea is not just about having an acceptance test suite. It's a certain kind of acceptance test suite. It's an acceptance test suite that mocks out at boundaries so you don't make API requests to the server, you make an API request to a [inaudible] server using something like Mirage or whatever that might be. So, the big idea, it has like new ones that makes it functional, but getting people who are completely unaware, who don't necessarily look up to you as an authority to believe you like, "Yes, that actually sounds like a really good idea." It is not a trivial task. CHARLES: No, it's not. Because the first time you try and explain it, you're arguing based on your own assumptions. So, you're coming to it safe in the knowledge that this is a really, really good idea based on your firsthand knowledge. But that means you're assuming a lot of things. You're assuming a lot of context that you have that someone else doesn't and they're going to be asking questions. Why this way, why this way, why this way? And so, you have to generate a framework for thinking about the entire problem in order to explain the value of the idea. And that's something that you don't get when it's just something that's accepted as a practice. TARAS: I think simulation is actually a really good example of that. If you haven't had experience with Mirage, if you don't know what having a configurable server in your tests does for you, you will probably not realize that similar ideas apply to, for example, Bluetooth that when you're writing tests for your Bluetooth devices, or you're writing tests for an application that interacts with Bluetooth devices, you actually want to have a simulation there for Bluetooth so that you can configure it in a kind of similar way to the way you would configure a Mirage for a specific test. You want to be able to say, "This Bluetooth device is going to exist," or, "I have these kinds of Bluetooth devices around me, they have the following attributes. They might disconnect after a little while." There's all kinds of scenarios that you want to be able to set up so that you can see how your application is going to respond. But if you haven't seen a simulation with something like Mirage, you're going to be going like, "I don't know what the hell why would this be helpful." CHARLES: It seems like lot of work. One of the things that we've been working on, as I said, is trying to come up with a framework for thinking about why this is a good idea because we can't just assume it. It's not common knowledge. For example, one of the things that we've been developing over the last year and more recently in the last few months is trying to understand what makes a test valuable. At its essence, what are the measures that you can hold up to a test and say this test has X value. Obviously, something like that is very, very difficult to quantify. But if you can show that and you say, "This test has these quantities and this test has these quantities," then we can actually measure them. Then it's going to allow people to accept it a lot more readily and try it a lot more readily. So, the ideas that we're playing with right now is that you kind of have to evaluate a test on one on speed, tests that are fast, have an intrinsic value, or rather test that are slow. The upside that you gained from the test is very quickly bled away or offset if the test is slow. So, I can have a very comprehensive test that tests a lot of high value stuff. But if it takes three days to run, it's going to be basically worthless. Another axis on what you can evaluate is in terms of coverage. I'm not talking about coverage of lines of code. I'm talking about use cases and units of assemblage. So, there's the module. Those modules are then stitched together into components. Those components are stitched together into applications. Those applications are downloaded onto browsers. And I would consider it a different unit of assemblage. Your application running on Firefox is a different assemblage than your application running on Chrome, is a different assemblage than your application running. You have this access, which is the coverage of your unit of assemblage. That's another way that you can evaluate your tests. So if I have a test that runs only on Node in a simulated dom, there's a cap, there's an absolute cap on the value of that test and it cannot rise above a certain point. And the other thing, another access that we've identified is isolate ability, an ability to run a test. So if I have a test suite comprised of 1500 tests, but if one fails in the middle, I have to restart the test from the beginning. That's going to decrease the value. Maybe it's related to speed, being able to run the tests without having to install a bunch of different dependencies. So that's another access. And so trying to really understand the variables there, that's something you have to be very systematic about thinking about the tests so that you can actually take your idea and explain it to someone who's not coming at it from first principles. Imagine you have to explain an if statement to somebody who's never programmed with an if statement before. TARAS: That is going to be very difficult. CHARLES: It's going to be difficult, right? TARAS: Yeah. In general, it's very challenging to go from an experience that somebody had, like overriding somebody's experience conveying your own personal experience is very difficult. And getting someone to experience something is very difficult. And so that's what I think a lot of the work that we've been doing over the last little while has been breaking down these problems into a way of understanding them so that we can actually explain why these things are important. Like what we've had to do this recently with a Bluetooth work that we've been doing. We have a partner that's implementing a Bluetooth abstraction for mobile devices and trying to convey to them the value of being able to simulate devices. That's something that we saw with Mirage. We knew that being able to simulate devices in tests in different environments is extremely valuable. We know this firsthand from our experience. But trying to justify to them why we think this is important and why they should rejig all of their thinking about how they're going to architect this middle layer between the native APIs and the application APIs, why they should rejig this layer to make it so that it will allow for simulation, like convincing very technical, very knowledgeable and experienced people that these ideas are important, it requires kind of fundamental understanding of the value that they provide. And I think this touches on what really I think Frontside is going to be doing going forward is creating conditions for this kind of thought to be cultivated, to be able to create business environment where our clients gain benefit from this kind of very deep insight, insight that transcends the source of those ideas. Because there are certain ideas that are really fundamental and they're beautiful ideas. For example, in React world, a functional component is a really beautiful idea. It's a really simple concept. And kind of going along along with what you're saying about this kind of fundamental ideas, they will drive you to the next point you couldn't even imagine. I think the work that the React core team has done -- they've set out this functional component as a primitive, but it wasn't possible for a long time to really make the functional component a reality until they've gone through the process of actually understanding what connects all the dots so that you could eventually get to a point where like, "Oh, actually this [inaudible] API is a way for us to enable the fundamental concept of having a component." Not only is it simple, but it actually works. You can write a performance application using this fundamental concept. So, the work that we're going to be doing is first of all, making it possible to have conversations at this level. So, people that we're going to be working with Frontside, clients who we work with, companies that work with us, who partner with us, it's going to be all to support creating this environment where we can have this kind of ideas. Then being able to extract these great ideas from the frameworks and actually making them available across frameworks. You don't have to be locked into a specific community to be able to benefit from that. One of the first steps we're working on now is now that we have an understanding about big tests, we believe this is a good idea, we have ways to justify why it's a good idea. We have clients who have benefited from this good idea. Now we're in position to fill in the gaps that are missing from big tests, from being something that could be used in any framework. We want it to be really easy for people to be able to say, "I really love acceptance testing, but I have to work in a React project or I have to work in a Vue project, I have to work in an Angular projects." It's like, "It's no problem. I have big tests. Big tests is going to give me what I want." The big tests, the idea, but it's going to come with all the little pieces that you need to be able to assemble the functional test suite that is going to give you the benefits that Ember's acceptance test suite provides for Ember projects. CHARLES: I mean, why should you have to compromise on these things. If it is a good idea and it is a fundamental idea, whether it's how you manage concurrent processes, whether it's how you manage testing, whether it's how you manage routing, why should you have to compromise? Why should you have to say, "You know what? I love..." I don't know, what's a comparable system? "I love air conditioning." Why should I have to go into a car that doesn't have air conditioning? Because every single car has a different air conditioning system, or every single house has a different air conditioning system. I'm showing the fact that I live in Texas here by using this example. But we've developed air conditioning systems that are modular so that they can be snapped on to pretty much any house and you don't have to build a custom thing for the circulation and refrigeration of air every single time or have it just be like it's part of the house. Or we have this wood that ships with ducks. It's like, no, we want to separate out that system. TARAS: You don't have to commit to living in a specific area to get the benefit of being comfortable. You don't have to give up the comfort for specific areas. The good idea of air conditioning is not restricted to just one area. You can actually experience it wherever you go and have it be available wherever you would go. You have to be able to, like if you're moving to a new house, you can install air conditioner and now you're going to have like cool air. That's the kind of the theme I think of what we're going to be doing. But there's so much work to do because we're kind of, I would say, probably 30 or 40% into having -- if we were to look at like big tests as a big idea is used and is well known in our industry, we're probably maybe 5% on that. If we take into consideration what it takes for us to be able to convey what it takes to convince people is a good idea. I think we're, if you add to it, we'll probably add another 15%. So the next step is actually creating some tooling around it that would make it really easy for people to consume. Because right now, what's really unfortunate is that you have big tests that is available. If we set it up in the React project, doing it with create-react-app right now is kind of difficult. So we're going to make that easier. But then companies still need to make a choice between the ergonomics [inaudible] Cypress or the speed and the comfort and the control that you have with big tests. We have to basically eliminate the need to make that choice by making the tooling that you get in Cypress and making that tooling available for big test projects. So that is going to kind of bring us up to maybe 60%. And then the remaining part is taking the software that enables the big idea and then making the software work across all different frameworks so that it's really easy to install. On Angular, it's just going to be add a plugin and it gives you big testing. You don't have to rely on their unit testing or you don't have to rely on the end to end testing as using Selenium. You just have big tests and it works just as well as it works and React and just as well as it works in Ember. So you can get the benefits of that. Going through the whole process and then bringing into the world this way, we have a lot of work to do because every part of the architecture of building modern frontend applications requires this level of effort. If you look at routing, routing is extremely inconsistent across frameworks. There are different opinions for every framework. There are different opinions of how to implement it. And that's problematic both on an individual level and a specific project level because it's so hard to know what you should use. And in many cases, it's not complete. What constitutes a routing system on React looks different than what would be in Ember, and it looks different than what it is in Angular. And each one of them has their own trade-offs. So if you find yourself in a situation where you've been using React and now you have to use Angular, you have a steep learning curve. But this also has an interesting effect of like, "Well, we have a world now where micro frontends are a thing." There are really good reasons why companies might want to use multiple frameworks for single platform because different frameworks have different benefits and different teams based on their location might prefer a different framework. There's lots of different reasons why companies choose and we want developers to be able to choose specific framework. But how do you do that when you need to have basically micro frontend architecture, you need to provide an SDK that provides a consistent user experience like -- you need to provide an SDK that provides a consistent developer experience, but then that developer experience because it's developer experience, needs to enable consistent user experience regardless of what framework the team decides to use. So now, routing needs to be more robust. Routing without having a strong concurrency primitives is going to be very limited and it's not going to be as portable across frameworks as one that actually has very robust concurrency primitives. So now we'll start looking at concurrency. We've got concurrency primitives that are different for each framework. We've got Ember concurrency, we have a Redux-sagas. We have observables. React is introducing suspense and their hooks. And now each one of those things is different. So now, we need a concurrency primitive that is framework agnostic that is externalized, that we can consume to build stuff. Now that's a piece of the routing system. There's other things like state machines. State machines are an important part. If you look at the authentication system, the core of an authentication system is a state machine. How do you express a state machine in a way that is framework agnostic, in a way that works in every framework. That's an area that we need to explore. So, there's a lot of work for us to do to take these big ideas and externalize them from the framework so that they can be consumed by frameworks. And the applications that use this frameworks, that is a lot of work. And that's essentially what Frontside is setting out to do is hold these good ideas and extract the pieces that are actually core to these ideas. And then make them available independently and then see what that will provide. In a world where we have all of these pieces in place and we have kind of the granular little pieces that we need to be able to have a really strong concurrency primitive, we have a great routing system that's using this concurrency primitives. We have a state machine mechanism that can work with any framework, has very comfortable APIs. We have all these pieces. What kind of collaboration will that allow? What is the world that that will create? I think for people who are in the Ember community, we saw firsthand what it means for us to stand the shoulders of giants. We have wonderful, brilliant people creating really awesome tools. Now, if that same level of collaboration was happening in a way that was framework agnostic, but we were all standing on shoulders of giants and we were helping each other create something like this, what kind of world would that create? That's what Frontside is setting out to find out. CHARLES: Yeah, and it's going to be fantastic. I should say it's going to be really, really fun. It's going to be challenging. Like you said, there is a lot of work to do. TARAS: A lot of conversations to have. I mean, I think it's difficult to have these conversations because we all have good reasons for thinking certain things and a lot of times it requires understanding each other to find the place. Core teams do this. It's getting everybody to align on where we want to go, provide leadership like doing all of that work and doing it in a way that is actually possible. Because to create an environment with this kind of research that is happening, it requires a lot of money. Charles and I cannot do this. A small group of people probably cannot do this either because we need real use cases. We need real projects to make sure that these ideas are not dream implementations or just like vaporware. We need the pressure of real projects, real clients' relationships to create software that is going to make a huge difference for their developers. So we need those relationships with clients so that we can actually create the conditions for this kind of thing to emerge. That needs to also happen for us to make this kind of a situation possible. So, there is a lot of work to do but it's a really worthwhile cause. CHARLES: And I think that it's one of the reasons that we start with testing as the most important use case because it's something that you don't want to be just saying, "Hey, you want to come help us fund research to make everything better?" There is real pressure, there's financial pressure and there's a duty to make sure that if a project has a deadline, that deadline is met. You want to come in at cost and on time. That's the goal of every project. And you want to try and strive to do your best to achieve that. And testing is an area where the research almost always pays off. I mean, where it has clearly. You can invest a lot of money and get a lot of reward in return because you're losing so much money by not testing or by focusing your investment in testing on lower value targets. TARAS: For people that are familiar with acceptance testing, I think that just goes like a no brainer. Acceptance testing just enables you that. So by putting big test as a first kind of objective, but putting acceptance testing as a first kind of milestone, it sets us up to benefit in the way that Ember community has benefited where the acceptance test now became a mechanism that made it possible for the framework to change. And so, we're kind of setting up the same conditions here. When you start working with us or with any of the companies that work with us, the first benefit you get is you get the acceptance testing. Now, developers are more productive because they know that they have a certainty that if something broke, they will know. If they broke something, they will know. But what that also gives us is a way of introducing change in a systematic way. So, if we find that we have a new routing mechanism, we can start to refactor our React router setup to use this new routing mechanism and we can do it safely because we have our acceptance testing in place. So in many ways, it's kind of like the first step. And it's actually a great first step for business and is a great first step for tech technology. That's kind of a really awesome overlap. CHARLES: In the podcast with Dylan Collins, we talked about this is like step one is make experimentation cheap because experimentation can be wildly expensive and you can throw away a lot of money if you're just experimenting without constraints. And there is nothing like a great application/acceptance test suite that puts those constraints in a completely and totally unambiguous way. So I think that's actually a great summary of our next decade. TARAS: Yeah, I think so too. I think it's a good to show you the perfect next step. And I think this is going to show us the perfect next future though we can't really imagine it right now. But I do know one thing, this future is going to include a lot of people. Fundamentally, what we're setting out to do is a big idea. And so, big ideas are brought to the world by a lot of people. And I think one of the things that's important is that from day one, we include everyone in the process. For anyone who's interested in these big ideas, we want to make it possible for you to participate in. So that's the reason why Frontside is going to have an RFC. Like on our Frontside website, we're actually going to start writing and publishing RFCs. The first RFC that we're going to have is going to be for big tests, which is to lay out why big test is a big good idea for frontend for our industry. And as we start talking to people, there's actually lots of applications beyond frontend. So, we're going to focus on writing these RFCs and sharing them with the world so that we can include as many people in it as possible so everyone can participate. And ultimately, hopefully, that will have an influence on our industry and actually allow us to leverage these big ideas in a way that is not tied to any particular source of these brilliant ideas. CHARLES: Yeah, hat tip to the Ember community for the whole RFC thing, which I guess they didn't invent either. TARAS: Yeah, they borrowed it also, sorry. But I mean, it's been adopted everywhere. And I think it is a really interesting way of making people part of your process and making it our process as our meaning, like everyone who works with us. Whether you work for Frontside, that there's a client of Frontside or you work with the company that is partner with Frontside, whatever that might look like, we want to start to create places where we can be having these conversations and together surfacing these great ideas and making them part of a tool sets. CHARLES: I think that's a perfect way to wrap up. Like I said, you pretty much described our next decade. Maybe if we get more people, it's going to be significantly less and we'll move onto the next thing even more rapidly. I am fully pumped. I'm ready to see this. There's a lot of work to do and I feel great about getting to it. TARAS: I'm very excited about all the work that we're going to be doing, so let's get to it. CHARLES: Yup. All right. We'll see everybody next time. Thank you for listening. If you or someone you know has something to say about building user interfaces that simply must be heard, please get in touch with us. We can be found on Twitter at @TheFrontside or over just plain old email at contact@frontside.io. Thanks. See you next time.

In this episode, Charles and Taras discuss "transparent development" and why it's not only beneficial to development teams, but to their clients as well. Please join us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at contact@frontside.io. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer. This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. Transcript: CHARLES: Hello and welcome to The Frontside Podcast, the place where we talk about user interfaces and everything that you need to know to build them right. It's been a long summer and we're back. I'm actually back living in Austin, Texas again. I think there wasn't too much margin in terms of time to record anything or do much else besides kind of hang on for survival. We've been really, really busy over the last couple of months, especially on the professional side. Frontside has been doing some pretty extraordinary things, some pretty interesting things. So, we've got a lot to chew on, a lot to talk about. TARAS: There's so much stuff to talk about and it's hard to know where to start. CHARLES: So, we'll be a little bit rambly, a little bit focused. We'll call it fambly. But I think one of the key points that is crystallized in our minds, I would say over this summer, is something that binds the way that we work together. Every once in a while, you do some work, you do some work, you do some work, and then all of a sudden you realize that a theme, there's something thematic to that work and it bubbles up to the surface and you kind of organically perceive an abstraction over the way that you work. I think we've hit that. I hit one of those points at least because one of the things that's very important for us is -- and if you know us, this is things that we talk about, things that we work on -- we will go into a project and set up the deployment on the very first day. Make sure that there is an entire pipeline, making sure that there is a test suite, making sure that there are preview applications. And this is kind of the mode that we've been working in, I mean, for years and years and years. And where you say like if what it takes is spending the first month of a project setting up your entire delivery and showcasing pipeline then that's the most important work, inverting the order and saying that has to really come before any code can come before it. And I don't know that we've ever had like a kind of unifying theme for all of those practices. I mean, we've talked about it in terms of saving money, in terms of ensuring quality, in terms of making sure that something is good for five or 10 years, like this is the way to do it. And I think those are definitely the outcomes that we're looking for. But I think we've kind of identified what the actual mode is for all of that. Is that fair to say? TARAS: Yeah, I think one of the things I've always thought about for a long time is the context within which decisions are made because it's not always easy. And it's sometimes really difficult to really give it a name, like getting to a point where you have really clear understanding of what is it that is guiding all of your actions. What is it that's making you do things? Like why do we put a month of work before we even start doing any work? Why do we put this in our contract? Why do we have a conversation with every client and say, "Look, before we start doing anything, we're going to put CI in place." Why are we blocking our business on doing this piece? It's actually kind of crazy that from a business perspective, it's a little bit crazy that you be like, "Oh, so you're willing to lose a client because the client doesn't want you to set up a CI process?" Or in a case of many clients, it's like you're not willing to accept -- the client is going to say, "We want to use Jenkins." And what we've done in the past, in almost every engagement, we're like, "Actually, no. We're not going to use Jenkins because we know that it's going to take so long for you to put Jenkins in place. By the time that we finish the project, you're probably still not going to have it in place. That means that we're not going to be able to rely on our CI process and we're not going to be able to rely on testing until you're finished." We're not going to have any of those things while we're doing development. But why are we doing all this stuff? It was actually not really apparent until very recently because they didn't really had a name to describe what is it about this tooling and all of these things that makes why is it so important to us. I think that's what kind of crystallized. And the way that I know that it's crystallized because now that we're talking to our clients about it, our clients are taking on to picking up the language. We don't have to convince people that this is a value. It just comes out of their mouth. Like it actually comes out of their mouth as a solution to completely unrelated problems, but they recognize how this particular thing is actually a solution in that particular circumstance as well even though it's not something Frontside sold in that particular situation. Do you want to announce what it actually is? CHARLES: Sure. Drum roll please [makes drum roll sound]. Not to get too hokey, but it's something that we're calling Transparent Development. What it means is having radical transparency throughout the entire development process, from the planning to the design, to the actual coding and to the releases. Everything about your process. The measure by which you can evaluate it is how transparent is this process to not just developers but other stakeholders, designers or people who are very developer adjacent, engineering managers all the way up to C level executives. How transparent is your development? And one of the ways that we sell this, because I think as we talk about how we arrived at this concept, we can see how this practice actually is a mode of thinking that guides you towards each one of these individual practice. It guides you towards continuous integration. It guides you towards testing. It guides you towards the continuous deployment. It guides you towards the continuous release in preview. I think the most important thing is that it's guided us, by capturing this concept, it's guided us to adopt new practices, which we did not have before. That's where the proof is in the pudding is if you can take an idea and it shows you things that you hadn't previously thought before. I think there's a fantastic example. I actually saw it at Clojure/conj in 2016, there was a talk on juggling. And one of the things that they talked about was up until I think it was the early 80's or maybe it was the early 60's, the state of juggling was you knew a bunch of tricks and you practice the tricks and you get these hard tricks. And that was what juggling was, is you practice these things. It was very satisfying and it had been like that for several millennia. But these guys in the Physics department were juggling enthusiasts and I don't know how the conversation came about, you'd have to watch the talk. It's really a great talk. But what they do is they make a writing system, a nomenclature system for systematizing juggling tricks, so they can describe any juggling trick with this abstract notation. And the surprising outcome, or not so surprising outcome, is that by then, once you have it in the notation, you can manipulate the notation to reveal new tricks that nobody had thought of before. But you're like, "Ah, by capturing the timing and the height and the hand and we can actually understand the fundamental concept, and so now we can recombine it in ways that weren't seen before." That actually opened up, I think an order of magnitude of new tricks that people just had not conceived of before because they did not know that they existed. And so, I think that's really, as an abstract concept, is a great yardstick by which to measure any idea. Yes, this idea very neatly explains the phenomenon with which I'm already familiar, but does the idea guide me towards things with which I have no concept of their existence? But because the idea predicts their existence, I know it must be there and I know where to look for it. And aha, there it is. It's like shining a light. And so I think that that's kind of the proof in the pudding. So that's a little bit of a tangent, but I think that's why we're so excited about this. And I think it's why we think it's a good idea. TARAS: Yes. So what's also been interesting for me is how universal it is. Because the question of is this transparent enough? That question could be actually asked in many cases. What's been interesting for me is that asking that question in different contexts that I didn't expect actually yielded better outcome. At the end of the day, I think that a test for any idea is like, is it something that can help you more frequently than not? Like is it actually leading you? Does applying this pattern, does it increase the chances of success? And that's one of the things that we've seen, thinking about just practices that we're putting into place and quite asking are they transparent enough? Is this transparent enough? It's actually been really effective. Do you want to talk about some of the things that we've put in place in regards to transparency? Like what it actually looks like? CHARLES: Yeah. I think this originally started when we were setting up a CI pipeline for a native application which is not something that we've typically done in the past. Over the last, I would say, 10 years, most of our work has been on the web. And so, when we're asked to essentially take responsibility for how a native application is going to be delivered, one of the first things that we asked kind of out of habit and out of just the way that we operate is how are we going to deliver this? How are we going to test it? How are we going to integrate it? All the things that we've just talked about is something that we have to do naturally. But because this is not very -- like continuous integration and build is very prevalent on the web. I think that testing still has a lot of progress on the web, but it's far more prevalent than it is in other communities, certainly the native community. So when we started spending a month setting up continuous integration an integration test suite, spending time working on simulators so that we could simulate Bluetooth, having an automated process with which we could ship to the App Store, all of these things kind of existed as one-offs in the native development community. There are a lot of native developers who do these things. But because it's not as prevalent and because it was new to us, it caused a lot of self reflection both on why is it that we feel compelled to do this. And also we had to express this, we had to really justify this work to existing native development teams and existing stakeholders who were responsible for the outcomes of these native development teams. So, there was this period of self reflection, like we had to write down and be transparent about why we were doing this. TARAS: Yeah. We had to describe that in SoWs. We actually had really long write ups about like what it is that we're setting up. And for a while, it was, people I think would read these SoWs and I think they would get the what's of what we're actually going to be putting into place. But it wasn't until we actually put it into place and we've seen a few like really big wins with the setup -- one of the first ones was the setting up preview apps where preview apps in -- the web are pretty straightforward because we've got Netlify that just kind of gives it to you easily. CHARLES: Netlify and Heroku. It's very common. TARAS: Yeah, you activate it, it's there. But on the mobile side, it's quite a different story because you can't just spin up a mobile device that is available through the web. It's something kind of very special. And so we did find a service called Appetize that does this. And so we hooked up the CI pipeline to show preview apps in pull requests. So for every pull request, you could see specifically what was introduced in our pull request without having to pull down source code, like compile it. You could just click a link and you see a MVC stream of a mobile device and that application running on a mobile device. So the setup took a little bit of time. But once we actually put it in place and we showed it to our clients, one of the things that we noticed is that it became a topic of conversation. Like, "Oh, preview apps are amazing." "This is so cool." "Preview apps are really great." And I think in some ways, it actually surprised us because we knew that they were great, but I think it was one of the first times that we encountered a situation where we would show something to a client and they just loved it. And it wasn't an app feature. It was a CI feature. It was part of a development process. CHARLES: Right. So, the question is then why was this so revelatory? Why was it so inspiring to them? And I think that the reason is that even if we have an agile process and we're on two week iterations, one week iterations, whatever, there's still a macroscopic waterfall process going on because essentially, your business people, your design people, maybe some of your engineering people are involved at the very front of the conversation. And there's a lot of talking and everybody's on the same page. And then we start introducing coding cycles. And like I said, even if we're working on two week iterations and we're "agile", the only feedback that you actually have, whether something is working, is if the coder says it's done. "I'm Done with this feature. I'm on to the next feature for the next two weeks." And after that two weeks, it's like, "I'm done with this feature. I'm on to the next feature." From the initial design, you have the expectation about what's going on in the non-technical stakeholders minds. They have this expectation. And then they hope that through the process of this agile iterative development cycles, they will get the outcome that satisfies that hope. But they're not able to actually perceive and put their hands on it. It's only the engineers and maybe some really tech savvy engineering managers who can actually perceive it. And so they're getting information secondhand. "Hey, We've got authentication working and we can see this screen and that screen." And, "Hey, it works on iOS now." "I have some fix ups that I need to do on Android." So, maybe they're consuming all of their information through standups or something like that, which is better than nothing. That is a level of transparency. But the problem is then you get to actually releasing the app or whether it's on the web, whether it's on native, but this is really a problem on native. You get to where you actually release the app and then everybody gets to perceive the way the app as it actually is. So you have this expectation and this hope that was set maybe months prior and it just comes absolutely careening into reality in an instant, the very first moment that you open the app when it's been released. And if it doesn't meet that expectation, that's when you get disappointment. When expectations are out of sync and grossly out of sync with reality, even a little bit out of sync with reality, you get disappointment. As fundamental and explanation of just the phenomenon of disappointment, but it's an explanation of why disappointment happens so often on development projects. Is this kind of the expectations and hopes of what a system can be in the minds of the stakeholders? It's kind of this probability cloud that collapses to a single point in an instant. TARAS: And that's when things really hit the proverbial fan. Now, you have the opposite. So everything that was not transparent about your development process. So everything that was hidden in the opaqueness of your development process, all of those problems, either on a product side, maybe something didn't quite get implemented the way it's supposed to. Like you actually found out two weeks or three weeks before you're supposed to release that that feature wasn't actually quite implemented right. It went through testing, but it was tested against the Jira stories that were maybe not quite written correctly. So the product people are going like, "What the hell is this? It's actually not what I signed up for. This is not what I was asking for." So, there's that part. And then on the development side, you've got all of the little problems that you didn't really account for because you haven't been shipping to production from day one. You actually have like application not really quite working right. You didn't account as supposed to integrate with some system that is using Chorus or something you didn't account for. Like you have a third party dependency you didn't really fully understand. But because it wasn't until you actually turned it on that you actually started to talk to the thing properly, and you realize there's some mismatch that is not quite working. But now you've got everything that was not transparent about the development process, everything that was hiding in the opaque corners of your development process is now your problem for the next three weeks because you've got to fix all of these problems before you release. And that's what I think where a lot of organizations kind of find themselves in is this position where they've been operating for six months and like, "Everything is going great!" And then three months or three weeks before, you're like, "Actually, this is not really what we were supposed to do. Why did this happen?" That time is really tough. CHARLES: Yeah. That's what we call crunch time. And it's actually something that is lot of times we think of it as inevitable, but in fact it is actually an artifact of an opaque process. TARAS: Yeah. CHARLES: That's the time when we have to go, everybody's like, "We're ordering pizza and Dr. Pepper and nobody's leaving for a month." TARAS: Yeah. I think there are people that do that practice like functional testing as part of development process or acceptance testing, I think they could relate to this in some cases where if you had to set up a test suite on an application that was written without a test suite, first thing you deal with are all the problems that you didn't even know were there. And it's not until you actually start testing, like doing functional testing, not integration or unit testing where you're testing everything in isolation, but when you're perceiving the entire system as one big system and you're testing each one of those things as the user would, it's not until that point you start to notice all the weird problems that you have. Like your views are re-rendering more than you expected. You have things that are being rendered you didn't even notice because it's hard to see, because it happens so quickly. But in test, it happens at a different pace. And so, there's all these problems that you start to observe the moment that you start doing acceptance testing, but you don't see them otherwise. And so, it's the process of making something transparent that actually highlights all these problems. But for the most part, if you don't know that there are transparent options available, you actually never realize that you are having these problems until you are in crunch time. CHARLES: Right. And what's interesting is viewed through that lens, your test suite is a tool for perception. And to provide that transparency, not necessarily something that ensures quality, but ensuring the quality is a side effect of being able to perceive bugs as they happen or perceive integration issues at the soonest possible juncture. To shine the light, so to speak, rather than to act as a filter. It's a subtle distinction, but I think it's an important one. TARAS: About functional testing and acceptance testing. I think one of the things that I know personally from experience working with comprehensive acceptance test suites is that there is certainty that you get by expressing the behavior of the application in your tests. And I think what that certainty does is it replaces hope as opposed to having hope baked into your system where you think like you're hoping. I think for many people, they don't even perceive it as hope. They perceive it as reality. They see it as, "My application works this way." But really what's happening is there's a lot of trust that's built into that where you have to say like, "Yeah, I believe the system should work because I wrote it and I'm good. And it should not be broken." But unless you have a mechanism that actually verifies this and actually insures this is the case, you are operating in the area of dreams and hopes and wishes, and not necessarily reality. And I think that's one of the things that's different. A lot of the processes around highlighting or shining light on the opaque areas of the development process. And it's actually not even just development process. It's actually the business process of running a development organization. Shining light in those areas is then what gives you the opportunity to replace hope with real validatable truth about your circumstances. CHARLES: And making it so that anyone can answer that question and discover that truth and discover that reality for themselves. So, generating the artifacts, putting them out there, and then letting anybody be the primary perceiver of what that artifact means in the context of the business, not just developers. And so, that kind of really explains preview apps quite neatly, doesn't it? Here we've done some work. We are proposing a change. What are the artifacts that explain the ramifications of this change? So we run the test suite. That's one of the artifacts that explains and radiates the information so that people can be their own primary source. And look at it in a developer centric, although you can tell, any old person can tell if the test suite's failing, it's not a change that we should go with. But the preview app is something we take this hypothetical change, we build it, we put it out there and now, everyone can perceive it. And so, it calibrates the perception of reality and it eliminates hope. Which is like if your development process is based on hope, you are signing yourself up for disaster. I like what you said that it implies a huge amount of trust in the development team. And you know what? If you have a cracked development team, that trust is earned and people will continually invest based on that trust. But the fundamentals are still fragile because they still can open up a crack between the expectation and the reality. And the problem is when that happens, the trust is destroyed. And it's during that crunch time, if it does happen that you lose credibility and it's not because you became a worse developer. It's not because your team is like lower performing, it's just that there was this divergence allowed to open. But then the problem is that really lowers the trust and that means that unfortunately that's going to have a negative knock on effect. And reasonably so. Because if you're an engineering manager or a product manager, you're something like this and you're losing trust in your development team and their ability to deliver what you talked about, then you're going to want to micromanage them more. The natural inclination is to try and be very defensive and interventionist and you might actually introduce a set of practices that inhibit the development cycle even further and lower the team's abilities to perform right when they need to do it the most, then you end up destroying more trust. TARAS: Yeah, it's a spiraling effect I think because it's in the process of trying to make things better. And then you start to introduce practices. Like maybe you're going to have meetings every day reviewing outstanding stories to try to get everybody on the same page, but now you're micromanaging development team. The development team starts to resent that and now you've got this like people hating their job. It starts to get messier and dirtier and more complicated. And the root cause of that is that from day one, there was a lot of just [inaudible] about getting into it and just starting to write some code but what you didn't actually do is you didn't put in place the fundamentals of making sure that you can all observe a reality that is honest. And I think that kind of fundamental principle, it's interesting how when you actually start to kind of take this idea and when you start to think about it in different use cases, it actually tells you a lot about what's going on and you can actually use it to design new solutions. One of the things that Frontside does, I don't know if those who've kind of worked with us before might know this or might not, but we don't do blended rates anymore. Because we don't actually, one of the challenges with blended rates is that they hide the new ones that gives you the power to choose how to solve a problem. CHARLES: Yeah. There's a whole blog post that needs to be written on why blended rates are absolute poison for a consultancy. But this is the principle of why. TARAS: Yeah. I think it's poison for transparent consultancy because if you want to get the benefits of transparency, you have to be transparent about your people. Because alternatively what happens is that you start off relying on your company's reputation and then there is a kind of inherent lie in the way that the price points are set up because everybody knows that there is going to be a few senior people, there's going to be a few intermediate people, a few junior people. But these exact ratios of those or who is doing what, how much people are available, all of those things are kind of hidden inside of the consulting company so that they can manage their resources internally. And so what that does is it simplifies your communication with the client. But actually what it also does is it disempowers you to have certain difficult conversations when you need the most. And you could say, "Look, for this kind of work, we don't need to have more senior people working on this." We can have someone who is junior who is at like $100 an hour, $75 an hour as opposed to being $200 or $250 an hour. We can have that person working on this and we can actually very clearly define how certain work gets solved. It requires more work. But then what it does is it creates a really strong bond of honesty and transparency between you and your clients. And it gives you a way, like now the client starts to think about you as a resource that allows them to fulfill on their obligations in a very actionable way. They can now think about how they can use you as a resource to solve their problems. They don't need a filter that will process that and try to make it work within the organization. You essentially kind of become one unit. And I think that sense of unity is the fundamental piece that keeps consulting companies and clients glued together. It's the sense of like, "We can rely on this team to give us exactly what we want when we need it, and sometimes give us what we need that we don't know we need." But that bond is there. And that bond is strong because there is no lie in that relationship. You're very transparent about who are the people that's working on it. What are they actually going to be doing? How much is this costing us? CHARLES: It's worth calling out explicitly whether on the flip side of it is, is if you have a blended rate, which is the way that Frontside operated for, gosh, pretty much forever, is that people will naturally calibrate towards your most senior people. If I'm going to be paying $200 an hour across the board, or $150 an hour across the board, or $300 across the board, whatever the price point is, they're going to want to extract the most value for that one price point. And so, it means that they're going to expect the most senior people and become resentful if what I'm paying is $300 for a task. If I've got five senior people, it's a better deal for me. For the same price to get five senior people than two senior people to a medium level people and one junior person. And so, it has two terrible effects. One is that they don't appreciate the senior people to be like, "Hey actually, these are people with extraordinary experience, extraordinary knowledge, extraordinary capability that will kick start your part." So they are under appreciated and then they're extremely resentful of the junior people. It's like, "I'm paying the same rate for this very senior person as I am for this junior person? Get this person off my project." But if you say, "You know what, we're going to charge a fifth of the cost for this junior person and we're going to utilize them," then you're providing real value and they're appreciating it. They're like, "Oh, thank you for saving me so much money. We've got this task that does not require your most senior person. That would be a misallocation of funds. I'd be wasting money on them. But if you can charge me less and give me this junior person and they're going to do just as competent a job, but it's going to cost me a fifth of the money, then that's great. Thank you." So, it flips the conversation from 'get this god-damn junior person off my project' to 'thank you so much for bringing this person on'. It's so critical. But that's what that transparency can provide. It can totally turn a feeling of resentment into gratitude. TARAS: What's interesting is from business perspective, you make the same amount of money. In some cases, you actually make more money. I think in that way, it's a consulting company. But that's not the important part because the amount of value that's generated from having more granular visibility into what's happening is so much greater. It's kind of like with testing where any of those things where when you start to put, when you start to shine light on these kind of opaque areas and then you start to kind of flush out the gremlins that are hiding there, what you then start to do, what you kind of discover is this opportunity to have relationships with clients that are honest. So you could say, for example, like one of the things that we've done recently is we actually have like 10-tier price point model, which allows us to to be really flexible about the kind of people that we introduce. So, there's a lot of details that go into the actual contracting negotiation. But what it does is it allows us to be very honest about the costs and work together with our clients, like actually really find a solution that's going to work really well for them. And then this is kind of a starting point when we start thinking about transparency in this kind of diverse way, you actually start to realize that there are additional benefits that you might have never been experienced before. One of the things that we found recently is that one of the initiatives that we kind of launched with one of our clients is we wanted to bring together, there's a general problem that exists in large projects, which is that if you have a really big company and you have like, let's say 20 or 30 interconnected services, your data domain, like the older data, kinds of data you work with is spread over a whole bunch of microservices spread over potentially a bunch of different development teams spread over a bunch of different locations. What usually has happened in the past is each one of those problems or the domain, the data domain has been kind of siloed into a specific application. We worked with a bank in the past and that being had for every, they had 80 countries. In each country they had 12 different industries, like insurance and mortgage and different kinds of areas of services they offered. And then for each of the country, for each of the service, they had a different application that provided that functionality. Then the next step is, let's not do that anymore because we now have something like 100, 150 apps, let's bring it all together under a single umbrella and let's create a single shared domain that we can then use. And so, a GraphQL becomes a great solution for that. But the problem is that making that change is crazy complicated because the people on the business side who understand how all the pieces fit together. On the other side, you have the developers who know where the data can come from and how to make all that real. And on the other side is there's like frontend implementers who actually build in the UIs that are consuming all these services. On a project that we're working on right now is we're building a federated GraphQL gateway layer that is kind of connecting all these things, bringing all these things together. But the problem is that without very specific tooling to enable that kind of coming together of the frontend, the backend, the business people having coming together, creating a single point of conversation and having a single point of reference for all the different data that we have to work with and different data that is available to the gateway, without having something like that, without having that transparency in the actual data model, it is really difficult to make progress because you don't have shared terminology, you don't have shared understanding of the scope of the problem. There's a lot of dots in context that needs to be connected. And for anyone who has worked with enterprise, you know how big these problems get. And so what we've done on a project that we're working on now is we actually aimed to bring transparency to this process. What we actually did is put in place, start to build an application that brings together all of the federated services into a visualization that different parties can be involved in. And so I think one of the kind of common patterns that we see with transparency in general is that we are including people in the process, in the development process that were previously not included. So in the past, they would be involved in the process either very early on or very late in the process, but they wouldn't be involved along the way. And so what this kind of transparency practice actually does is it allows us to kind of democratize and flatten the process of creating foundations for pieces that touch many different parts of the organization. And so this tool that we created allows everyone to be involved in the process of creating the data model that spans the entire organization and then have a single point of reference that everybody can go to and have a process for contributing to it. They don't have to be a developer. There's developers who consume it. There are business people that consume it. There are data modeling people that consume it. Like there's different people parties involved. But the end result is that everyone is on the same page about what it is that they're creating. And we're seeing the same kind of response as we saw with preview apps where people who previously didn't really have an opinion on development practices or how something gets built, all of a sudden they're participating in the conversation and actually making really valuable suggestions that developers couldn't really have exposure to previously because developers often don't have the context necessary to understand why something gets implemented in a particular way. CHARLES: Something beautiful to behold, really. And like I said, it's wonderful when a simple concept reveals things that had lay hidden before. TARAS: Yeah. It's a very interesting lens to look at things through. How transparent is this and how can we make it more transparent? I think asking that question and answering that question is what has been kind of giving us a lot of -- it had been very helpful in understanding our challenges in the work that we do on a daily basis and also in understanding how we could actually make it better. CHARLES: I apply this concept in action on my pull requests. I've really been focusing on trying to make sure that if you look at my pull request, before you actually look at the code, you can pretty much understand what I've done before you even look at the diff. The hallmark of a good pull request is basically if by reading the conversation, you understand what the implementation is going to be. There's not really any surprises there. It's actually hard to achieve that. Same thing with git history. Spending a lot of time trying to think like how can I provide the most transparent git history? That doesn't necessarily mean exactly the log of what happened moment to moment, day to day, but making sure that your history presents a clear story of how the application has evolved. And sometimes that involves a lot of rebasing and merging and branch management. I think another area that has been new for us, which this has revealed those things that I just described are areas where we're kind of re-evaluating already accepted principles against a new measure, but introducing an RFC process to actually a client project where we're making architectural decisions with our developers, the client's developers, external consultants. You've got a lot of different parties, all of whom need to be on the same page about the architectural decisions that you've made. Why are we doing this this way? Why are we doing modals this way? Why are we using this style system? Why are we using routing in this way? Why are we doing testing like this? These are decisions that are usually made in an ad hoc basis to satisfy an immediate need. It's like, "Hey, we need to do state management. Let's bring in Redux or let's bring in MobX or let's bring in whatever." And you want to hire experts to help you make that best ad hoc decision? Well, not really. I mean, you want to lean on their experience to make the best decision. But having a way of recording and saying this is the rationale for a decision that we are about to make to fulfill a need. And then having a record of that and putting it down in the book so that anybody who can come later. First of all, when the discussion is happening, everybody can understand the process that's going on in the development team's head. And then afterwards and it's particularly important is someone asks a question, "Why is this thing this way?" You can point directly to an RFC. And this is something that we picked up from the Ember community, but this is something that open source projects really by their very nature have to operate in a very highly transparent manner. And so, it's no surprise that that process came from the internet and an open source project. But it's been remarkably effective, I would say, in achieving consensus and making sure that people are satisfied with decisions, especially if they come on afterwards, after they've been made. TARAS: We actually have this particular benefit that could experience that particular benefit today where one of the other things that this RFC process and transparency with the architecture, how that kind of benefits the development organization is that a lot of times when you are knee deep in doing some implementation, that is not a time you want to be having architectural conversations. In the same way like in a big football team, they huddle up before they go on a field. You can't be talking strategy and architecture and plans while you're on the football field. You have to be ready to play. And this is one of the things that the RFC process does is it allows us to say, "Look, right now we have a process for managing architecture so that with the RFC process you can go review our accepted RFCs. You can make proposals there." And that is a different process than the process that we're involved in on a daily basis, which is writing application, using architecture where we have in place. And so that in itself can be really helpful because well intentioned people can bring up these conversations because they really are trying to solve a problem, but that might not be the best time. And so having that kind of process in place and being transparent about how architecture decisions are made allows everyone to participate and it also allows you to prioritize conversations. CHARLES: Yeah. And that wasn't a practice that we had adopted previous to this, but it's something that seemed obvious that we should be doing. It's like, how can we make our architecture more transparent? Well, let's do this practice. So, I keep harping on this. But I think it's the hallmark of a good idea if it leads you to new practices and new tools. And we're actually thinking about adopting the RFC process for all of our internal developments, for maintaining our open source libraries. TARAS: There is something that we've been working on that we're really excited about. So, there's a lot of stuff happening at Frontside. But one of the things that we've been doing is working on something we call the transparent node publishing process, which is something that I think we originally drew inspiration from the way NativeScript has their repo set up. But one thing that's really cool about how they have things set up is that every pull request automatically is available, like everything is available. Very quickly, a pull request is available for you to play with and you can actually put it into your application as a published version in npm and actually see exactly if that pull request is going to work for you. You don't have to jump through hoops. You don't have to clone the repo, build it locally, link it. You don't have to do any of that stuff because if you see a pull request that has something that you want but then is not available in master, there's an instruction on the pull request that tells you, "Here's how you can install this particular version from npm." And so you essentially you're publishing. Every pull request automatically gets published to npm and you can just download and install that specific version for that particular pull request in your project. That in itself I think is one of those things I suspect that is going to be talked about. It actually can alleviate a lot of problems that we have on a development processes because like the availability of the work of people who are participating in the project, there is kind of a built in barrier that we are essentially breaking down with this transparent node publishing process. And so, that's something that we're very close to to having it all on our repos and we're going to try it out and then hopefully share it with everyone on the internet. CHARLES: I didn't know that the NativeScript did this. I thought that the idea that came from it is like how can we apply these transparency principles to the way we maintain npm packages. The entire release process should be completely transparent, so that when I make a pull request, it's available immediately in a comment. And then furthermore, even when a pull request is merged, there's no separate step of let's get someone to publish it. It's just now it's on master. Now it is available as a production release. You close the latency and you close the gap and people perceive things as they are. There is nothing like, "Oh that emerged. When do I get this?" This is something that I can't stand about using public packages is you have some issue, you find out that someone also has had this issue, they've submitted a pull request for it and then it's impossible to find if there's a version and what version actually supports this? And it's even more complex between projects that actually do backporting of fixes to other versions. So I might be on version two of a project. Version three is the most recent one, but I can't upgrade to version three because I might be dependent on some version two APIs. But I really need this fix. Well, has it been backported? I don't know. Maybe upgrading is what I have to do, but maybe downgrading. Or if I'm on the same major release version, maybe there's been 10 pull requests, but there's been no release to npm. And it can be shockingly difficult to find out if something is even publicly available. And the transparency principle comes in to, "Hey, if I see it on GitHub, if I see it there, then there's something there that I can touch and I can perceive for myself to see if my issue has been resolved or if the things work as I expect." TARAS: I'm really excited about this. I'm really excited about this kind of clarification, this crystallization of transparency. And I'm also seeing our clients starting to apply it to solving problems within their organization as well is very inspiring. CHARLES: Yeah, it is. It is really exciting. And honestly, I feel like we've got one of those little triangular sticks that people use to find water. I feel like we have a divination stick. And I'm really excited to see what new tools and practices it actually predicts and leads us to. TARAS: Me too. I'm really excited to hear if anyone likes this idea. Send us a tweet and let us know what you're seeing for yourself about this because I think it's a really interesting topic and I'm sure there's going to be a lot that people can do with this idea in general. CHARLES: Thank you for listening. If you or someone you know has something to say about building user interfaces that simply must be heard, please get in touch with us. We can be found on Twitter at @TheFrontside, or over just plain old email at contact@frontside.io. Thanks and see you next time.

Rich Harris talks about Svelte and Reactivity. Rich Harris: Graphics Editor on The New York Times investigations team. Resources: Svelte Please join us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at contact@frontside.io. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer. This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. Transcript: CHARLES: Hello and welcome to The Frontside Podcast, a place where we talk about user interfaces and everything that you need to know to build them right. TARAS: It's actually a really nice, Rich and I'm really, really happy to have a chance to actually chat with you about this because Svelte is a really fun piece technology. In many ways, it's interesting to see our technology evolve and our industry evolve through innovation, real innovation. I think Svelte 3 has really been kind of that next thought provoking technology that kind of makes you think about different ways that we can approach problems in our space. So, really excited to chat with you about this stuff. RICH: Well, thank you. Excited to be here. TARAS: I think quite a lot of people know, Rich, about your history, like how you got into what you're doing now. But I'm not sure if Charles is aware, so if you could kind of give us a little bit of a lowdown on where you kind of come from in terms of your technical background and such. RICH: Sure. I'll give you the 30-second life history. I started out as a reporter at a financial news organization. I had a Philosophy Degree and didn't know what else to do with it. So, I went into journalism. This was around the time of the great recession. And within a few weeks of me joining this company, I watched half of my colleagues get laid off and it's like, "Shit, I need to make myself more employable." And so gradually, sort of took on more and more technical responsibilities until I was writing JavaScript as part of my day job. Then from there, all these opportunities kind of opened up. And the big thing that I had in mind was building interactive pieces of journalism, data-driven, personalized, all of that sort of thing, which were being built at places like the New York Times, and The Guardian, and the BBC. That was the reason that I really wanted to get into JavaScript. And that's guided my career path ever since. CHARLES: It's interesting that this D3 and all that did come out of journalism. RICH: It's not a coincidence because when you're working under extreme time pressure and you're not building things with a view to maintain them over a long period of time, you just need to build something and get it shipped immediately. But it needs to be built in a way that is going to work across a whole range of devices. We've got native apps, we've got [inaudible], we've got our own website. And in order to do all that, you need to have tools that really guide you into the pit of success. And D3 is a perfect example of that. And a lot of people have come into JavaScript through D3. CHARLES: And so, are you still working for the same company? RICH: No. That's ancient history at this point. CHARLES: Because I'm wondering, are you actually getting to use these tools that you've been building to actually do the types of visualizations and stuff that we've been talking about? RICH: Very much so. I moved to The Guardian some years ago. And then from there, moved to Guardian US, which has an office in New York. And it was there that I started working on Svelte. I then moved to the New York Times and I'm still working on Svelte. I've used it a number of times to build things at the New York Times and the people have built things with it too. And so, yeah, it's very much informed by the demands of building high performance interactive applications on a very tight deadline. CHARLES: Okay, cool. So I've probably used, I mean, I'm an avid reader of both Guardian and the New York Times, so I've probably used a bunch of these visualizations. I had no idea what was driving them. I just assumed it was all D3. RICH: There is a lot of D3. Mike Bostock, the creator of D3, he was a linchpin at the graphics department for many years. Unfortunately we didn't overlap. He left the Times before I joined the Times, but his presence is still very much felt in the department. And a lot of people who are entering the industry, they're still becoming database practitioners by learning from D3 examples. It's been a hugely influential thing in our industry. TARAS: How long is a typical project? How long would it take to put together a visualization for an article that we typically see? RICH: It varies wildly. The graphics desk is about 50 strong and they will turn around things within a day. Like when the Notre Dame burnt down a couple of months ago, my colleagues turned around this interactive scroll driven webGL 3D reconstruction of how the fire spreads through the cathedral in less than 24 hours, which was absolutely mind blowing. But at the same time, there are projects that will take months. I work on the investigations team at the Times. And so, I'm working with people who are investigating stories for the best part of the year or sometimes more. And I'm building graphics for those. And so that, it's two very different timescales, but you need to be able to accommodate all of those different possibilities. CHARLES: So, what does the software development practice look like? I mean, because it sounds like some of this stuff, are you just throwing it together? I guess what I mean by that is, I guess the projects that we typically work on, three months is kind of a minimum that you would expect. So, you go into it, we need to make sure we've got good collaboration practices around source control and continuous integration and testing and all this stuff. But I mean, you're talking about compressing that entire process into a matter of hours. So what, do you just throw right out the window? What do you say? "We're just doing a live version of this." RICH: Our collaboration processes consist of sitting near each other. And when the time calls for it, getting in the same room as each other and just hammering stuff out on the laptop together. There's no time for messing around with continuous integration and writing tests. No one writes tests in the news graphics, it's just not a thing. CHARLES: Right. But then for those projects that stretch into like three months, I imagine there are some. Do you run into like quality concerns or things like that where you do have to take into account some of those practices? I'm just so curious because it sounds like there's actually, the difference between two hours and two months is, that's several orders of magnitude and complexity of what you're developing. RICH: It is. Although I haven't worked on a news project yet that has involved tests. And I know that's a shocking admission to a lot of people who have a development background, but it's just not part of the culture. And I guess the main difference between the codebase for a two-hour project and a two-month project is that the two-month project will strive to have some reasonable components. And that's, I think, the main thing that I've been able to get out of working on the kinds of projects that I do is instead of just throwing code at the page until it works, we actually have a bit of time to extract out common functionality and make components that can be used in subsequent interactives. So, things like scroll driven storytelling, that's much easier for me now than it was when I first built a scroll driven storytelling component like a couple of years ago. CHARLES: Yeah. That was actually literally my next question is how do you bridge that, given that you've got kind of this frothy experimentation, but you are being, sounds like, very deliberate about extracting those tools and extracting those common components? And how do you find the time to even do that? RICH: Well, this is where the component driven mindset comes in really handy, I think. I think that five or 10 years ago when people thought in terms of libraries and scripts, there wasn't like that good unit of reusability that wasn't the sort of all encompassing, like a component is just the right level of atomicity or whatever the word is. It makes sense to have things that are reusable but also very easy to tweak and manipulate and adapt to your current situation. And so, I think that the advent of component oriented development is actually quite big for those of us working in this space. And it hasn't really caught on yet to a huge degree because like I say, a lot of people are still coming with this kind of D3 script based mindset because the news industry, for some interesting and historical reasons, is slightly out of step with mainstream mode development in some ways. We don't use things like Babel a lot, for example. CHARLES: That makes sense, right? I mean, the online print is not like it's a React application or it's not like the application is all encompassing, so you really need to have a light footprint, I would imagine, because it really is a script. What you're doing is scripting in the truest sense of the word where you essentially have a whole bunch of content and then you just need to kind of -- RICH: Yeah. And the light footprint that you mentioned is key because like most new sites, we have analytics on the page and we have ads and we have comments and all of these things that involve JavaScript. And by the time our code loads, all of this other stuff is already fighting for the main thread. And so, we need to get in there as fast as we can and do our work with a minimum fuss. We don't have the capacity to be loading big frameworks and messing about on the page. So that again is one of these sort of downward pressures that kind of enforces a certain type of tool to come out of the news business. TARAS: A lot of the tooling that's available, especially on like the really fatter, bigger frameworks, the tools that you get with those frameworks, they benefit over long term. So if you have like a long running project, the weight of the abstractions, you've experienced that benefit over time and it adds up significantly. But if you're working to ship something in a day, you want something that is just like a chisel. It does exactly what you want it to do. You want to apply it in exactly the right place and you want to get it done exactly, like you want the outcome to be precise. RICH: That's true. And I think a lot of people who have built large React apps, for example, or large Ember apps, they sort of look at Svelte and think, "Well, maybe this isn't going to be applicable to my situation," because it has this bias towards being able to very quickly produce something. And I'm not convinced that that's true. I think that if you make something easier to get started with, then you're just making it easier. If you build something that is simple for beginners to use, then you're also building something simple for experts to use. And so, I don't necessarily see it as a tradeoff, I don't think we're trading long-term maintainability for short term production. But it is certainly a suspicion that I've encountered from people. TARAS: This is something that we've also encountered recently. It's been kind of a brewing discussion inside a front side about the fact that it seems to be that certain problems are actually better to rewrite than they are to maintain or refactor towards an end goal. And we found this, especially as the tools that we create have gotten more precise and more refined and simplified and lighter, it is actually easier to rewrite those things five times than it is to refactor it one time to a particular place that we want it to be. And it's interesting, like I find this to be very recent, this idea is blossoming in my mind very recently. I didn't observe this in the past. CHARLES: Do you mean in the sense that like if a tool is focused enough and a tool is simple enough, then refactoring is tantamount to a rewrite if you're talking about 200 or 300 lines of code? Is that what you mean? TARAS: Yeah. If you're sitting down to make a change or you have something in mind, it is actually easy to say, "Let's just start from scratch and then we're going to get exactly the same place in the same amount of time." But this kind of mantra of not rewriting makes me think about that, makes me question whether that's actually something that is always the right answer. RICH: I definitely question that conventional wisdom at all levels, as well. I started a bundler called Rollup as well as Svelte more recently. And Rollup was the second JavaScript bundler that I wrote, because the first one that I wrote wasn't quite capable of doing the things that I wanted. And it was easier to just start from scratch than to try and shift the existing user base of its predecessor over to this new way of doing things. Svelte 3 is a more or less complete rewrite. Svelte has had multiple, more or less, complete rewrite. Some of them weren't breaking changes. But Svelte itself was a rewrite of an earlier project that I'd started in 2013. And so in my career, I've benefited massively from learning from having built something. But then when the time comes and you realize that you can't change it in the ways that you need to change it, just rewrite it. And I think that at the other end of the spectrum, the recent debate about micro frontend has largely missed this point. People think that the benefit of the micro frontend is that people don't need to talk to each other, which is absolute nonsense. I think the benefit of this way of thinking about building applications is that it optimizes for this fact of life that we all agree is inevitable, which is that at some point, you're going to have to rewrite your code. And we spend so much energy trying to optimize for the stability of a code base over the long term. And in the process, lock ourselves into architectural and technical decisions that don't necessarily make sense three or four years down the line. And I think as an industry, would be a lot better placed if we all started thinking about how to optimize for rewrites. CHARLES: So for those of us who aren't familiar, what is the debate surrounding micro frontends? This is actually something I've heard a lot about, but I've actually never heard what micro frontends actually are. RICH: Yeah. I mean, to be clear, I don't really have a dog in this fight because I'm not building products, but the nub of it is that typically if you're building a website that maybe has like an admin page, maybe it has a a settings page, maybe it has product pages, whatever. Traditionally, these would all be parts of a single monolithic application. The micro frontend approach is to say, "Well, this team is going to own the settings page. This team is going to own the product page." And they can use whatever technologies they want to bring that about. And the detractors sort of attack a straw man version of this, "You're going to have different styles in every page. You're going to have to load Vue on one page. You're going to have to load React on the other page. It's going to be a terrible user experience," when actually its proponents aren't suggesting that at all. They're suggesting that people from these different teams coordinate a lot more that are free to deviate from some kind of grand master architectural plan when it's not suitable for a given task. And darn right. I think it means that you have a lot more agility as an engineering organization than you would if you're building this monolithic app where someone can't say, "Oh, we should use this new tool for this thing. We should use microstates when the rest of the organization is using Google docs." It's not possible. And so, you get locked into the decisions of a previous generation. CHARLES: Right. No, it makes sense. It's funny because my first reaction is like, "Oh my goodness, that's a potential for disaster." The klaxon's going to go off in your head, but then you think, really then the work is how do you actually manage it so it doesn't become a disaster. And if you can figure that out, then yeah, there is a lot of potential. RICH: Yeah. People always try and solve social problems with technology. You solve social problems with social solutions. CHARLES: Right. And you have to imagine it too, it depends on the application, right? I think Amazon, the Amazon website is developed that way where they have different teams that are responsible even down to little content boxes that are up on the toolbar. And the site doesn't really, it shows, right? Like it shows like this is kind of like slapped together, but that's not what they need. They don't need it to not look like there's slight variation with the different ways that things behave. They need to be showing for their business to work. They need to be showing the right thing at the right time. And that's the overriding concern. So having it look very beautiful and very coherent isn't necessarily a thing. Same thing in Spotify, used as another example of this. I didn't know if it was called micro frontends, but I know that they've got a similar type thing, but they are clearly the experience and having it look coherent is more important. And so, they make it work somehow. And then like you're saying, it probably involves groups of people talking to other groups of people about the priorities. So yeah, it doesn't sound to me like just like you're going to adopt micro frontends guarantees one particular set of outcomes. It really is context dependent on what you make of it. RICH: Totally. TARAS: I'm curious though, so with Svelte, essentially for your reactivity engine, you have to compile to get that reactive behavior. RICH: Yeah. TARAS: How does that play with other tools like when you actually integrate it together? I've never worked with Svelte on a large project, so I can't imagine what it looks like at scale. I was wondering if you've seen those kind of use cases and what that ends up, if there's any kind of side effects from that. RICH: As you say, the reactivity within a component is only in the local state within that component or to state that is patched in as a prop from a parent component. But we also have this concept called a store. And a store is just a project that represents a specific value and you import it from svelte/store. And there are three types of store that you get out of the box. A writable, a readable and a derived. And a writeable is just, var count = writable (0) and then you can update that and you can set it using methods on that store. Inside your marker, you can reference or in fact inside the script block in the component, you can reference the value of that store just by prefacing it with a dollar sign. And the compiler sees that and says, "Okay, we need to subscribe to this store as value and then assign it and apply the reactivity." And that is the primary way of having state that exists outside the component hierarchy. Now, I mentioned the writable, readable, and derived are the built in stores that you get, but you can actually implement your own stores. You just need to implement this very simple contract. And so,, it's entirely possible to use that API to wrap any state management solution you have. So you can wrap redux, you can wrap microstates, you can wrap state, you can wrap whatever it is, whatever your preferred state management solution is, you can adapt it to use with Svelte. And it's very sort of idiomatic and streamlined. Like it takes care of unsubscriptions when the component is unmounted. All of that stuff is just done for you. CHARLES: Digging a little bit deeper into the question of integration, how difficult would it be to take wholesale components that were implemented in Svelte and kind of integrate them with some other component framework like React? RICH: If the component is a leaf node, then it's fairly straightforward. There is a project called react-svelte which is, I say project, it's like 20 lines of code and I don't think it's [inaudible] they did for Svelte 3, which I should probably do. But that allows you to use a Svelte component in the context of React application, just using the component API the same way that you would [inaudible] or whatever. You can do that inside a React component. Or you could compile the Svelte component to a web component. And this is one of the great benefits of being a compiler is that you can target different things. You can generate a regular JavaScript class and you've got an interactive application. Or you can target a server side rendering component which will just generate some html for some given state which can then later be hydrated on the client. Or you can target a web component which you can use like any other element in the context of any framework at all. And because it's a compiler, because it's discarding all of the bits of the framework that you're not using, it's not like you're bundling an entire framework to go along with your component. And I should mention while I'm talking about being able to target different outputs, we can also, as a NativeScript project, you can target iOS and Android that same way. Where it gets a little bit more complicated is if it's not a leaf node. If you want to have a React app that contains a Svelte component that has React [inaudible], then things start to get a little bit more unwieldy, I think. It's probably technically possible, but I don't know that I would recommend it. But the point is that it is definitely possible to incrementally adopt Svelte inside an existing application, should that be what you need to do. CHARLES: You said there's a NativeScript project, but it sounds to me like you shouldn't necessarily need NativeScript, right? If you're a compiler, you can actually target Android and you could target iOS directly instead of having NativeScript as an intermediary, right? RICH: Yes. If, if we had the time to do the work, then yes. I think the big thing there would be getting styles to work because Svelte components have styles. And a regular style tag just to CSS and you can't just throw CSS in a native app. CHARLES: Right. Sometimes, I feel like it'd be a lot cooler if you could. [Laughter] RICH: NativeScript really is doing a lot of heavy lifting. Basically what it's doing is it's providing a fake dom. And so, what the NativeScript does is it targets that dom instead of the real dom and then NativeScript turns that into the native instructions. CHARLES: Okay. And you can do that because you're a compiler. TARAS: Compilers has been on our radar for some time, but I'm curious like what is your process for figuring out what it should compile to? Like how do you arrive at the final compile output? Manually, have you written that code and then, "I'm going to now change this to be dynamically generated." Or like how do you figure out what the output should be? RICH: That's pretty much it. Certainly, when the project started, it was a case of, I'm going to think like a compiler, I'm going to hand convert this declarative component code into some framework plus JavaScript. And then once that's done, sort of work backwards and figure out how a compiler would generate that code. And then the process, you do learn certain things about what the points of reusability are, which things should be abstracted out into a shared internal helper library and what things should be generated in line. The whole process is designed to produce output that is easy for a human to understand and reason about. It's not like what you would imagine compile [inaudible] to be like, it's not completely inscrutable. It's designed to be, even to that level of being well formatted, it's designed to be something that someone can look at and understand what the compiler was thinking at that moment. And there's definitely ways that we could change and improve it. There are some places where there's more duplication than we need to have. There are some places where we should be using classes instead of closures for performance and memory benefits. But these are all things that once you've got that base, having gone through that process, that you can begin to iterate on. CHARLES: It's always curious to me about when is the proper time to move to a compiler, because when you're doing everything at runtime, there's more flexibility there. But at what point do you decide, "You know what? I know that these pathways are so well worn that I'm going to lay down pavement. And I'm going to write a compiler." What was the decision process in your mind about, "Okay, now it's time." Because I think that that's maybe not a thought that occurs to most of us. It's like, "I had to write a compiler for this." Is this something that people should do more often? RICH: The [inaudible] of 'this should be a compiler' is one that is worth sort of having at the back of your head. I think there are a lot of opportunities not just in DUI framework space but in general, like is there some way that we can take this work that is currently happening at runtime and shift it into a step that only happens once. That obviously benefits users. And very often we find that benefits developers as well. I don't think there was a point at which I said, "Oh, this stuff that's happening at runtime should be happening at compile time." It was more, I mean, the actual origin has felt that it was a brain worm that someone else infected me with. Judgment is a very well known figure in the JavaScript world. He had been working on this exact idea but hadn't taken it to the point where he was ready to open source it. But he had shared like his findings and the general idea and I was just immediately smitten with this concept of getting rid of the framework runtime. At the time, the big conversation happening in the JavaScript community was about the fact that we're shipping too much JavaScript and it's affecting startup performance time. And so the initial thought was, "Well, maybe we can solve that problem by just not having the runtime." And so, that was the starting point with Svelte. Over time, I've come to realize that that is maybe not the main benefit. That is just one of the benefits that you get from this approach. You also get much faster update performance because you don't have to do this fairly expensive virtual dom different process. Lately, I've come to think that the biggest win from it is that you can write a lot less code. If you're a compiler, then you're not kind of hemmed in by the constraints of the language, so you can almost invent your own language. And if you can do that, then you can do the same things that you have been doing with an API in the language itself. And that's the basis of our system of reactivity, for example. We can build these apps that are smaller and by extension, less bug prone and more maintainable. I just wanted to quickly address the point you made about flexibility. This is a theoretical downside of being a compiler. We're throwing away the constraints about the code needing to be something that runs in the browser, but we're adding a constraint, which is that the code needs to be statically analyzable. And in theory, that results in a loss of flexibility. In practice, we haven't found that to affect the things that we can build. And I think that a lot of times when people have this conversation, they're focusing on the sort of academic concepts of flexibility. But what matters is what can you build? How easy is it to build a certain thing? And so if empirically you find that you're not restricted in the things that you can build and you can build the same things much faster, then that academic notion of flexibility doesn't, to my mind, have any real value. CHARLES: Hearing you talk reminded me of kind of a quote that I heard that always stuck with me back from early in my career. I came into programming through Perl. Perl was my first language and Perl is a very weird language. But among other things, you can actually just change the way that Perl parses code. You can write Perl that makes Perl not throw, if that makes any sense. And when asked about this feature, the guy, Larry Wall, who came up with Perl, he's like, "You program Perl, but really what you're doing is you're programming Perl with a set of semantics that you've negotiated with the compiler." And that was kind of a funny way of saying like, "You get to extend the compiler yourself." Here's like the default set of things that you can do with our compiler, but if you want to tweak it or add or modify, you can do that. And so, you can utilize the same functionality that makes it powerful in the first place. You can kind of inject that whole mode of operation into the entire workflow. Does that make sense? That's like a long way of saying, have you thought about, and is it possible to kind of extend the Svelte compiler as part of a customization or as part of the Svelte programming experience? RICH: We have a very rudimentary version of that, which is pre-processing. There's an API that comes with Svelte called preprocess. And the idea there is that you can pass in some code and it will do some very basic, like it will extract your styles, it will extract your script and it will extract your markup. And then it will give you the opportunity to replace those things with something else. So for example, you could write some futuristic JavaScript and then compile it with Babel before it gets passed to the Svelte compiler, which uses acorn and therefore needs to be able to have managed other scripts so that it can construct an abstract syntax tree. A more extreme version of that, people can use [inaudible] to write their markup instead of html. You can use Sass and Less and things like that. Generally, I don't recommend that people do because it adds these moving parts and it makes like a lot of bug reports of people just trying to figure out how to get these different moving parts to operate together. I don't know, it means that your editor plugins can't understand what's inside your style tag all of a sudden and stuff like that. So, it definitely adds some complexity, but it is possible. At the other end, at a slightly more extreme level, we have talked about making the cogeneration part plugable so that for example, the default renderer and the SSR renderer are just two examples of something that plugs into the compiler that says, "Here is the component, here's the abstract syntax tree, here's some metadata about which values are in scope," all of this stuff and then go away and generate some code from this. We haven't done that so far, partly because there hasn't been a great demand for it, but also because it's really complicated. As soon as you turn something into a plugin platform, you just magnify the number of connection points and the number of ways that things could go wrong by an order of magnitude. And so, we've been a little bit wary of doing that, but it is something that we've talked about primarily in the context of being able to do new and interesting things like target webGL directly or target the command line. There are renders for React that let you build command line apps using React components. And like we've talked about, maybe we should be able to do that. Native is another example. The NativeScript integration as you say, it could be replaced with the compiler doing that work directly, but for that to work presently, that would mean that all of that logic would need to sit in core. And it would be nice if that could be just another extension to the compiler. We're talking about a lot of engineering effort and there's higher priority items on our to do list at the moment. So, it's filed under one day. CHARLES: Right. What are those high priority items? RICH: The biggest thing I think at the moment is TypeScript integration. Surprisingly, this is probably like the number one feature request I think is that people want to be able to write Typescript inside the Svelte components and they want to be able to get TypeScript when they import the Svelte component into something else. They want to be able to get completion [inaudible] and type checking and all the rest of it. A couple of years ago, that would've been more or less than thinkable but now it's like table stakes is that you have to have first-class TypeScript support. CHARLES: Yeah, TypeScript is as popular as Babel these days, right? RICH: Yeah, I think so. I don't need to be sold on the benefits. I've been using TypeScript a lot myself. Svelte is written in TypeScript, but actually being able to write it inside your components is something that would involve as hacking around in the TypeScript compiler API in a way that, I don't know if anyone actually or any of us on the team actually knows how to do. So, we just need to spend some time and do that. But obviously when you've got an open source project, you need to deal with the bugs that arise and stuff first. So, it's difficult to find time to do a big project like that. CHARLES: So, devil's advocate here is if the compiler was open for extension, couldn't a TypeScript support be just another plugin? RICH: It could, but then you could end up with a situation where there's multiple competing TypeScript plugins and no one's sure which ones are used and they all have slightly different characteristics. I always think it's better if these things that are common feature requests that a lot of people would benefit from, if they're built into the project themselves. I go really light in the batteries included way of developing and I think this is something that we've sort of drifted away from in the frontend world over the last few years, we've drifted away from batteries included towards do it yourself. CHARLES: Assemble the entire thing. Step one, open the box and pour the thousand Lego pieces onto the floor. RICH: Yeah, but it's worse than that because at least, with a Lego set, you get the Lego pieces. It's like if you had the Lego manual showing you how to build something, but you were then responsible for going out and getting the Lego pieces, that's frontend development and I don't like it. CHARLES: Right. Yeah. I don't like that either. But still, there's a lot of people advocating directly. You really ought to be doing everything completely and totally yourself. RICH: Yes. CHARLES: And a lot of software development shops still operate that way. RICH: Yeah. I find that the people advocating for that position the most loudly, they tend to be the maintainers of the projects in question. The whole small modules philosophy, they exist for the benefit primarily of library authors and framework authors, not for the benefit of developers, much less users. And the fact that the people who are building libraries and frameworks tend to have the loudest megaphones means that that mindset, that philosophy is taken as a best practice for the industry as a whole. And I think it's a mistake to think that way. TARAS: There is also, I think, a degree of a sliding scale where you start off with like as the more experience you get, because there is more experience you get closer, you get to that kind of wanting granular control and then they kind of slides down towards granular control and then slice back up to, once you've got a lot of experience, you're like, "Okay, I don't want this control anymore." And then you kind of cast that and you get into like, "I'm now responsible for tools that my team uses," and now you're back to wanting that control because you want things to be able to click together. It's kind of like a way that your interest in that might change over time depending on your experience level and your position in the organization. So yeah, there's definitely different motivating factors. Like one of the things that we've been thinking a lot about is designing tools that are composable and granular at individual module level, but combined together into a system for consumption by regular people. So like finding those primitives that will just click together when you know how to click them together. But when you're consuming them, just feel like a holistic whole, but at the same time not being monolithic. That's a lot of things to figure out and it's a lot of things to manage over time, but that's solely the kind of things we've been thinking about a lot. RICH: I think that's what distinguishes the good projects that are going to have a long lifespan from the projects that are maybe interesting but don't have a long shelf life is whether they're designed in such a way that permits that kind of cohesion and innovation tradeoff, if you think of it as a trade off. Anyone can build the fastest thing or the smallest thing or the whatever it is thing. But building these things in a way that feels like it was designed holistically but is also flexible enough to be used with everything else that you use, that's the real design challenge. CHARLES: It's hard to know where to draw that line. Maybe one good example of this and, these are actually two projects that I'm not particularly a fan of, but I think they do a good job of operating this way. So, I guess in that sense, it means I can even be more honest about it. I don't particularly care for Redux or like observables, but we ended up using, in one of our last React projects, we had to choose between using Redux-Saga and Redux-Observable. The Redux-Observable worked very well for us. And I think one of the reasons is because they both had to kind of exist. They had to kind of co-exist is their own projects. Like Redux exists as its own entity and Observables exist as their own kind of whole ecosystem. And so, they put a lot of thought in like what is the natural way in which these two primitives compose together? As opposed to the Saga, which I don't want to disparage the project because I think it actually is a really good project. There's a lot of really good ideas there but because it's more like just bolted on to Redux and it doesn't exist outside of the ecosystem of Redux and the ideas can't flourish outside and figure out how it interfaces with other things. Like the true primitive is still unrevealed there. And so, whereas I feel like with Redux you actually have to really, really true primitives. Now, they're not necessarily my favorite primitives, but they are very refined and very like these do exactly what they are meant to do. And so when you find how they connect together, that experience is also really good. And the primitive that arises there I think ends up being better. Is that an example of what you guys are talking about? RICH: Maybe. [Laughs] TARAS: No, I think so. I mean, it's distilling to the essence, the core of what you're trying to do and then be able to combine it together. I mean, that's been kind of the thing that we've been working on at the Frontside. But also within this context, it makes me think of how does a compiler fit into that? How does that work with the compiler? It's just like when you add the compiler element, it just makes it like my mind just goes poof! [Laughter] CHARLES: Yeah, exactly. That's why I keep coming back to like, how do you, and maybe I haven't, you just have to kind of go through the experience, but it feels like maybe there's this cycle of like you build up the framework and then once it's well understood, you throw the framework away in favor of like just wiring it straight in there with the compiler and then you iterate on that process. Is that fair to say? RICH: Kind of, yeah. At the moment, I'm working on this project, so I referred a moment ago to being able to target webGL directly. At the moment, the approach that I'm taking to building webGL apps is to have webGL components inside Svelte in this project called SvelteGL. And we've used it a couple of times at the Times. It's not really production ready yet, but I think it has some promise. But it's also slightly inefficient, like it needs to have all of the shade of code available for whichever path you're going to take, whatever characteristics your materials have, you need to have all of the shade of code. And if we're smart about it, then the compiler could know ahead of time which bits of shade of code it needed to include. At the moment, it just doesn't have a way of figuring that out. And so that would be an example of paving those cow paths. Like if you do try and do everything within the compiler universe, it does restrict your freedom of movement. It's true. And to qualify my earlier statements about how the small modules philosophy is to the benefit of authors over developers, it has actually enabled this huge flourishing of innovation, particularly in the React world. We've got this plethora of different state management solutions and CSS and JS solutions. And while I, as a developer, probably don't want to deal with that, I just want there to be a single correct answer. It's definitely been to the advantage of the ecosystem as a whole to have all of this experimentation. Then in the wild, there are projects like Svelte they can then take advantage of. We can say, "Oh well, having observed all of this, this is the right way to solve this problem." And so, we can kind of bake in that and take advantage of the research that other people have done. And I think we have made contributions of our own but there is a lot of stuff in Svelte like the fact that data generally flows one way instead of having [inaudible] everywhere. Things like that are the results of having seen everyone make mistakes in the past and learning from them. So, there are tradeoffs all around. TARAS: One thing on topic of data flow here and there, one thing that I've been kind of struggling to compute is the impact of that as opposed to something where you have like one directional data flow because it seems like conceptually it's really simple. You set a property like in two way balance system, like you just propagate through stuff but we don't really have a way, you don't have any way of assessing what is the true impact of that computation. Like what is the cost of that propagation where I think it's almost easier to see the cost of that computation if you have like one directional data flow because you know that essentially everything between the moment that you invoke transition to computing the next state, that is the cost of your computation where you don't have that way of computing the result in a two way balance system. Something like Ember Run Loop or mobx or zones, Vues, reactive system. All these systems make it really difficult to understand what is the real cost of setting state. And that's something that I personally find difficult because this clarity that you have about the one directional data flow and what it takes to compute the next state, it's almost like because that cost is tangible where you're thinking about like mutation of objects and tracking their change like that cost is almost immeasurable. It just seems like a blob of changes that they have to propagate. I don't know. That's just something that I've been thinking a lot because especially with the work that we'll be doing with microstates because as you're figuring out what the next state is, you know exactly what operations are performed in a process where that might not be the case with the system that tracks changes like where you'd have with zones or with Ember Run Loop, or Vue. RICH: I would agree with that. The times that I found it to be beneficial to deviate from the top-down ideology is when you have things like form elements and you want to bind to the values of those form elements. You want to use them in some other computation. And when you do all that by having props going in and then events going out and then you intercept the event and then you set the prop, you're basically articulating what the compiler can articulate for you more effectively anyway. And so conceptually, we have two way bindings within Svelte, but mechanically everything is top down, if that makes sense. CHARLES: Is it because you can analyze the tree of top down and basically understanding when you can cheat. This might be really over-simplistic, but if you're kind of with the event, you're collecting the water and then you have to put it way up on top of the thing and it flows down. But if you can see the entire apparatus, you can say, "Actually, I've got this water and it's going to end up here, so I'm just going to cheat and put it over right there." Is that the type of thing that you're talking about where you're effectively getting a two way binding, but you're skipping the ceremony. RICH: It's kind of writing the exact same code that you would write if you were doing it using events. But if you're writing it yourself, then maybe you would do something in a slightly inefficient way perhaps. For example, with some kinds of bindings, you have to be careful to avoid an infinite loop. If you have an event that triggers a state change, the state change could trigger the event again and you get this infinite loop. A compiler can guard against that. It can say this is a binding that could have that problem, so we're going to just keep track of whether the state changes as a result of the binding. And so, the compiler can sort of solve all of these really hairy problems that you had faced as a developer while also giving you the benefit in terms of being able to write much less code and write code that expresses the relationship between these two things in a more semantic and declarative way without the danger. TARAS: This is one of the reasons why I was so excited to talk to you about this stuff, Rich, because this stuff is really interesting. I mentioned that we might, so we have a little bit more time. So I just want to mention, because I think that you might find this interesting, the [inaudible], the stuff that we were talking about that I mentioned to you before. So, I want to let Charles talk about it briefly because it's interesting, because it essentially comes down to managing asynchrony as it ties to life cycle of objects. Life cycle of objects and components are something we deal with on a regular basis. So, it's been an interesting exercise and experimenting with that. Charles, do you want to give kind of a low down? CHARLES: Sure. It's definitely something that I'm very excited about. So, Taras gets to hear like an earful pretty much every day. But the idea behind structure concurrency, I don't know if you're familiar with it. It's something that I read a fantastic -- so people have been using this for a while in the Ember community. So Alex Matchneer, who's a friend and often time guest on the podcast created a library called ember-concurrency where he brought these ideas of structure concurrency to the ember world. But it's actually very prevalent. There's C libraries and Python libraries. There's not a generic one for JavaScript yet, but the idea is just really taking the same concepts of scope that you have with variables and with components, whether they be ember components, Svelte components, React components or whatever there is, you have a tree of components or you have a of parents and children and modeling every single asynchronous process as a tree rather than what we have now, which is kind of parallel linear stacks. You call some tick happens in the event loop and you drill down and you either edit an exception or you go straight back up. The next tick of the event loop comes, you drill down to some stack and then you go back up. A promise resolves, you do that stack. And so with structure concurrency, essentially every stack can have multiple children. And so, you can fork off multiple children. But if you have an error in any of these children, it's going to propagate up the entire tree. And so, it's essentially the same idea as components except to apply to concurrent processes. And you can do some just really, really amazing things because you don't ever have to worry about some process going rogue and you don't have to worry about coordinating all these different event loops. And one of the things that I'm discovering is that I don't need like event loops. I don't really use promises anymore. Like actually, I was watching, I think it was why I was watching your talk when you're talking about Svelte 3, when you're like -- or maybe did you write a blog post about we've got to stop saying that virtual doms are fast? RICH: Yes, I did. CHARLES: So I think it was that one. I was reading that one and it jived with me because it's just like, why can't we just go and do the work? We've got the event, we can just do the work. And one of the things that I'm discovering is with using the construction concurrency with generators, I'm experiencing a very similar phenomenon where these stack traces, like if there's an error, the stack traces like three lines long because you're basically doing the work and you're executing all these stacks and you're pausing them with a generator. And then when an event happens, you just resume right where you left off. There's no like, we've got this event, let's push it into this event queue that's waiting behind these three event loops. And then we're draining these queues one at a time. It's like, nope, the event happens. You can just resume right where you were. You're in the middle of a function call, in the middle of like [inaudible] block. You just go without any ceremony, without any fuss. You just go straight to where you were, and the stack and the context and all the variables and everything is there preserved exactly where you left it. So, it's really like you're just taking the book right off the shelf and going right to your bookmark and continuing along. Rather than when you've got things like the run loop in ember or the zones in angular where you have all these mechanics to reconstruct the context of where you were to make sure that you don't have some event listener. An event listeners created inside of a context and making sure that that context is either reconstructed or the event listener doesn't fire. All these problems just cease to exist when you take this approach. And so, if it's pertinent to this conversation, that was a surprising result for me was that if you're using essentially code routines to manage your concurrency, you don't need event loops, you don't need buffers, you don't need any of this other stuff. You just use the JavaScript call stack. And that's enough. RICH: I'm not going to pretend to have fully understood everything you just said but it does sound interesting. It does have something not that dissimilar to ember's run loop because if you have two state changes right next to each other, X+=1, Y+=1, you want to have a single update resulting from those. So instead of instruments in the code such that your components are updated immediately after X+=1, it waits until the end of the event loop and then it will flush all of the pending changes simultaneously. So, what you're describing sounds quite wonderful and I hope to understand that better. You have also reminded me that Alex Matchneer implemented this idea in Svelte, it's called svelte-concurrency. And when he sent it to me, I was out in the woods somewhere and I couldn't take a look at it and it went on my mental to do list and you just brought it to the top of that to do list. So yeah, we have some common ground here, I think. CHARLES: All right. TARAS: This is a really, really fascinating conversation. Thank you, Rich, so much for joining us. CHARLES: Thank you for listening. If you or someone you know has something to say about building user interfaces that simply must be heard, please get in touch with us. We can be found on Twitter at @thefrontside or over just plain old email at contact@frontside.io. Thanks and see you next time.

Philippe De Ryck joins the show to talk all things security: the importance and why you should be taking active steps, how to do it in your codebase effectively, and what can happen during a breach. Philippe De Ryck: Pragmatic Web Security Resources: OWASP Top 10 OWASP Top 10 Proactive Controls Please join us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at contact@frontside.io. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer. This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. Transcript: CHARLES: Hello and welcome to The Frontside Podcast, a place where we talk about user interfaces and everything that you need to know to build them right. My name is Charles Lowell, a developer here at The Frontside. Joining me, also hosting today is Taras Mankovsky. Hello, Taras. TARAS: Hello, hello. CHARLES: And as always, we're going to be talking about web platforms, UI platforms, and the practices that go into them. And here to talk with us today about a pillar of the platform that I certainly don't know that much about, and so, I'm actually really happy to have this guest on to talk about it is Philippe De Ryck who owns his own company called Pragmatic Web Security. I understand you do trainings and are just generally involved in the small space. So, welcome, Philippe. PHILIPPE: Hi. Nice to meet you. CHARLES: Wow! I almost even don't even know where to start with this subject because I'm kind of like the hippie developer mindset where it's like, "LaÖlaÖlaÖlaÖlaÖ we're in this open land and nothing's ever bad going to happen and we're just going to put code out there," and nobody would ever take advantage of any holes or anything like that. And I think that a lot of developers share that mentality and that's how we end up with major, major security breaches. And so, like I said, this is something that I'm actually very eager to learn but I almost even don't know where to start. I need training, man. [Laughter] PHILIPPE: Well, that's good to hear. No, you're totally right about that. If you're not into security, it seems like this fast space for a lot is happening and you don't really know how or why and what really matters to you and should I even be worried about this. And let me start by addressing the very first thing. Yes, you should be worried because maybe you're not building something that somebody cares about but you always have something that somebody wants, even the simplest of attacks always targets a valuable resource. Just to give you a very simple idea today, cryptocurrency is all the hype and you have a taker that's just aiming to misuse your users' computers to mine crypto coins because it essentially saves them a bunch on electricity cost. So, there's always something to grab. Usually, it's data or services or worse. But even in the most minimal cases, you have hardware, you have devices, you have network capacity that somebody might want to abuse. So yes, security, I would say, always matters. CHARLES: What's the best way to get started? You said understanding that everything we do, we're holding onto resources that might be valuable, that someone might want to seize but I'm just getting started with my application. I don't know anything about security. Where do I get started on just understanding the space? And then before I even look at tools that I wantÖ PHILIPPE: You want the honest answer for that? [Laughter] PHILIPPE: The honest answer is probably hire someone who has security knowledge. I don't mean this in a bad way. I've come a very long way in my career doing what I do now. And if I look at that, if you are aiming as a developer with no knowledge about security to build a secure application, it's going to be very hard. There's a lot of things you need to know, intrinsic knowledge. These are not things you can simply read a small book in a week, you know all of these security things that you'll know what to do. So, if you have no previous experience at all, I suggest to find some help. CHARLES: Right. It's like saying, "Hey, you've never written a data layer before but you want to go out and you want to write a massively distributed system where you have all these notes talking to each other. You're not going to read the O'Reilly book 'How to Build Distributed Systems' in a week and go out and do the same thing." It's the same thing with security. You need to understand the entire context. And there's no substitute for experience. PHILIPPE: Sorry, I actually like that comparison because in a sense, you're right, it's like these other very complex topics you don't expect to learn that in a week or a month and right a functioning data layer. But the difference is if you fail at writing that data layer, your application is probably not going to work. While if you fail at securing the application or seeing potential vulnerabilities, it's still going to work just a bit more than you anticipated. It's going to result leaking all your data to an attacker or opening all doors so that they can gain access to your server, stuff like that. So, I would say that the consequences of not getting it right are, at least in the beginning, very invisible. It's only after things happened that it's like, "Oh, crap!" And you should pay attention to that. CHARLES: Yeah. And then you have these back doors and these leaks that are set in stone and may be very hard to change. PHILIPPE: Yeah, absolutely. And honestly, the worst part of the breach is for a company, it might be reputation damage. But what you really should be worried about is all that personal information that's being leaked to, most cases, publicly leaked to anyone but in other cases, it's sold on [inaudible] markets and actually abused by people looking for someone else's identity or credit card information or stuff like that. And the companies usually get away with some bad press, maybe a small dip in their stock price but eventually, they'll bounce back. But it's the users that suffer from these breaches for a very, very long time. TARAS: What do you see the kind of hot zones around concerns that companies have around security? Because I imagine it's hard to be concerned about everything, so they're probably thinking about specific things like this thing worries us. Like what kind of things do you see companies and teams need to worry about? PHILIPPE: That's an interesting question. You have all different kinds of companies and all different levels of security awareness and what they're worrying about. I would say if you have the companies that are not very good at or don't have very much security knowledge, they're probably not to worry about things because otherwise, they would have started investing in improving their practices. If you look at the companies that are at least very aware of the landscape, I'm not saying that anybody here is perfect, but some of the companies are actually doing quite a good job. One of the most interesting challenges today is dealing with dependencies. So, all of your packages, you depend on npm, Maven, Python packages, Ruby gems, and so on. All of them form a huge attack factor in most applications today. That's definitely a problem that a lot of companies struggle with and it's very hard to find good solutions. TARAS: GitHub recently, I saw their vulnerability alert service that I've been getting a lot of notifications from on some of the open source libraries that we use. They have a lot of dependencies. And a lot of projects have the same dependencies. So, the moment that one notification goes on, it like lights up on all of the GitHub repos that I have. So, I have been going through and like updating dependencies and all those libraries. PHILIPPE: Yeah, that's a very good example. Absolutely. A lot of the projects we build today usually start out by installing a bunch of dependencies. Even before you've written the first line of code, you already have a massive code base that you are relying upon. And a single vulnerability in a code base might be enough, it's not always the case, but it might be enough to compromise your application. And that leaves you, as a developer, in a very hard place because you haven't written any lines of code yet. You have built a vulnerable application and that starting point can be very terrifying. So, there's a lot of reports on this. And actually, if you want some numbers, 78% of the vulnerabilities discovered in existing applications like the ones you mentioned. If GitHub alerts you like, "Hey, there's a problem in one of your dependencies," it's often even an indirect dependency, meaning that you include a framework, if you include React or Express or whatever you're building, that one of your dependencies of one of those projects actually has a vulnerability. If you look at the trees of these packages, they get quite big that it's not dozens but it's thousands of packages that we're talking about. CHARLES: Yeah that's the other thing is how do you know how to interpret these security vulnerabilities because some of them, we get a lot of security vulnerabilities for node packages but we only use them for our development tools to build frontend. So, if we're building a React application and there's some security vulnerability in some node packages that we're using in our build tool, then that doesn't actually get deployed to the frontend. So, maybe it's not a concern but if we were actually using it to build a server, then it would be absolutely critical. And so, how do you evaluate because the same security vulnerability is not a vulnerability in one context, but might be in another or maybe I'm thinking about it wrong. You see what I mean? PHILIPPE: Yeah, sure. I totally get what you mean. Actually, I have observed the same things. I also get these security alerts on my projects, and sometimes it's devDependency, so it seems like we don't need to care about that. You're right in the sense that you have to assess the criticality of such a report. So, they will have a rating, a severity rating saying like, "This is a minor issue," or, "This is a major issue," that should be a first indication. And then a second thing to look at is, of course, how are these things used in practice. It's not because it's a devDependency that it's not exploitable because it all depends on what is the vulnerability. If there's an intentional malicious backdoor in the library and you're building that on your build server, it might give an attacker access to your build server. So, that might not be something you actually want to do. So in that case, it does matter. Of course, if it's only stuff you run locally, you can say like, "OK, this is less important." But usually, updating or fixing these vulnerabilities also requires less effort because there's no building and deploying to production servers either. So, it's a matter of staying up-to-date with these. And one of the things that people struggle with is handling this in a lot of different applications. You mentioned you had a lot of GitHub repos and the vulnerability starts popping up in all of them and you have to fix and update all of them. You can imagine that major companies struggle with that, as well, especially if you have quite a few different technologies. Managing all of that is insanely hard. CHARLES: Right, because you just usually look at it and you're like, "Oh, I've got to download this." And maybe, "I haven't used it this repo for a while. I've got to clone it up, I've got to update the dependency. I've got to make sure I run all my tests locally, then run all the tests in CI and make sure I didn't break anything by upgrading. I might have fixed closed security hole but broken my functionality." And so, make sure that that is all intact and then push it out to production. Even on the small, it's like I'm looking, "OK, maybe this is going to take me 30 to 45 minutes." But if you have four or five of those things, you're looking at half your day or maybe even the whole day being gone and that's if you have the processes in place to do those automated verification. If you have a very high confidence in your deployment pipeline which I don't think a lot of places have. So, it sounds like these are complementary, like you really need in order to keep a secure application, you have to keep it up-to-date because I think what I'm hearing is you should just evaluate all the threats. You should fix it if you can. The first part of my question is, am I kidding myself when I say, "Oh, I can ignore this one because it's just local or it's just a devDependency." PHILIPPE: The answer to that question is briefly, I would say they are less critical. CHARLES: That's cool. PHILIPPE: In general, the rule is update if you can. And actually some of the tools out there that monitor vulnerabilities, they will automatically create a pull request in your repo saying to upgrade to this version and then you can automatically run your tests if you have them, and you can very quickly see whether some conflicts are generated by updating that dependency - yes or no. And in most cases, if it's a minor version bump, it's going to work as expected and you can easily push out the new version without that vulnerability. So, I would say fix if you can. If it goes quickly, then definitely fix them. But I would focus on non-devDependencies first instead of devDependencies. CHARLES: Yeah. PHILIPPE: Second thing I wanted to add is you paint a very grim picture saying you have to spend a lot of time updating these issues and I can totally understand that happening the very first time you look into this. There's going to be some stuff in there, I can guarantee that. But if you do this regularly, the effort becomes less and less because once you have up-to-date libraries, the problem is bad but it's not like we have 50 new vulnerabilities every day, fortunately. CHARLES: Right. PHILIPPE: So, once you have done that, it's going to be a bit less intensive than you might anticipate at first glance. Of course, if you're using these projects, if you're reusing the same library, then you'll have to update them everywhere. That's the downside, of course. CHARLES: It's probably a little bit dangerous to be assessing the criticality of the security threats yourself if you're not an expert, and kind of in the same way, it's dangerous to be assessing an architecture if you don't have an expertise in our architecture, I guess is the thing, because you might not understand the threat. PHILIPPE: Yeah, that's, again, absolutely true. It again depends very much on how it's deployed and what it's used for. That's going to be one important aspect. Another thing that might be very useful is, how deep is the dependency that creates the vulnerability or has the vulnerability? Because for example, if you have your tree of dependencies, if you dependency is like five or six levels deep, the chances of malicious data are reaching that specific vulnerability, and that specific library is going to be fairly small. Because usually, libraries have a lot of features and you only use part of them in your application. So, the other one is address of the features is just sitting there and if it's never used and it's also not exploitable. So, that might play a role as well. I saw a presentation about a month or two months ago from how Uber manages these things and they struggled with a lot of those things as well. And they eventually decided that they really care about vulnerabilities going three levels deep. And something that goes deeper is considered to be less relevant or less urgent to update because chances of exploitability are going to be very small. CHARLES: That's actually really interesting. TARAS: One thing that got me thinking about something that is actually happening right now. A friend of mine has a WordPress site that was hacked. But what's interesting about WordPress, I think the fact that WordPress site was hacked is not really a surprise but I think what's interesting about that is that the frequency and the sophistication of these attacks has increased. The tooling has improved also in the WordPress ecosystem. But at the same time, I think there is actually more people that are aware of the kind of exploits that could be done. There are a lot of people going after WordPress sites, but it kind of makes me think that there's probably going to be a time when the vectors of attack for web applications are going to become pretty well known as well. Because of the architecture, there are a fewer of them. But as the awareness of the actual architecture becomes more common, I think the angles of attack are going to become more interesting. Like one of the things that I was reading about a couple days ago is that there are some researchers that found a way to attract users based on a combination of JavaScript APIs that are available in the browser. So, they are actually able to fingerprint users based on the kind of things that they're using, the application for [inaudible] extensions they have installed. I think people are going to get more creative. And that's kind of scary because we've seen this happen already in WordPress and people are greedy. So, there are going to be ways. I think there's going to be more people looking at how to get into and how to exploit these vulnerabilities. PHILIPPE: Yeah. That's actually a couple of very good examples that illustrate the underlying issue. So, this browser-based tracking of users, it's called browser fingerprinting and it's been going on for a while. Back when I did my PhD, I had colleagues working on those things and you have other people at universities doing research on this. And yes, you can use things like JavaScript APIs in the browser to identify a particular user with a very high probability. It's not perfect but it's usually enough to identify a user for ad tracking or those purposes. By the way, these things also have a legitimate purpose. So, they are also used to keep track of a particular user to prevent things like session hijacking or detect problem logins or stuff like that, so they can also have a legitimate use case next to tracking. But they very clearly show how security will always be a cat and mouse game. Tracking used to be easy. You just set a cookie in a browser and the cookie was there next time and you knew who the user was. And then, users became a bit more savvy. You had browser extensions trying to block listings because let's be honest, they're kind of shady. So, users probably don't want that. And then the attacker started moving towards other things and getting more advanced. And you see that in other areas of security, as well. So, I consider that a good thing because as we make things harder for attackers, they will have to get more creative and it will become more difficult to exploit or to take advantage of applications. That's the good side. The bad side or the dark side of that equation is that unfortunately, the vulnerabilities are not going away. It's not because we now have these somewhat more advanced attacks using advanced features or even CView-based vulnerabilities that the old things like SQL injection and [inaudible] have disappeared in applications. That's also not true and that means that it makes it just a bit more harder for everyone on the defensive side to build more secure applications. You're going to have to know about the old stuff and you have to learn about the new stuff. CHARLES: Again, we come back to that idea. It's all a bit overwhelming. Aside from the solution of like, "Hey, let's hire Phillippe. Let's hire some other security expert." We were actually in your training, and obviously, I don't want to divulge all the secrets or whatever. If we were to attend your training, what do you see is the most important thing for people to know? PHILIPPE: There's no secrets there. [Chuckles] What I teach is web security. I kind of like to think I teach that in a very structured and methodical way. But in the end, there's no secrets and I don't mind talking about this here on the podcast because I honestly believe that everyone should know as much as they can about security. What do I teach? I can talk about specifics but I can also talk about generic things. One of the general takeaways is that one of the best things in my opinion that a developer can do is realize when they don't know something and actually admit that they don't know something, instead of just doing something. Maybe having like a brief thought like, "Hmm, is this secure? Well, it's probably good. I'm going to deploy it anyway. We'll see what happens." That is not the right way of doing things. If you do something and you recognize like, "Hey, this might be security sensitive. We're dealing with customer information here. We're dealing with healthcare information. We might want to look at what plays a role here," and then you can go ask someone who does. You probably have a colleague with a bit more security knowledge, so you can ask him like, "Hey Jim, or whatever your name is, do you think that this is OK or should we do something special here?" Very much like you are doing, asking me questions right here. That's one important takeaway that I hope everyone leaves with after a training class because not knowing something and realizing that you don't know it allows you to find someone who actually does. That still leaves us with that point which you wanted to sidestep. CHARLES: [Chuckles] PHILIPPE: A second thing is to realize that security is not a target. It's not something you're going to hit. It's not a holy goal that after working really hard for two years, you're going to hit this security milestone and you're done. It's always going to be a cat and mouse game. It's always going to be a moving target but that's OK. That's how things are. And that's the same with all other things in the world essentially. It's an evolving topic and you'll need to be ready to evolve with that as well. TARAS: One of the challenges that I see into quite often in teams is that at the individual level, people really try to do their best, maybe the best of their abilities. But it's often, when it comes to being part of a group, it's often like they do best within the kind of cultural environment that exists. I'm curious if you've seen good or kind of environments or cultures for engineering teams that are conducive to good security. Are there kind of systems or processes the companies put in place that you've seen to be very effective in preventing problems? Have you encountered anything like this? PHILIPPE: Ideally, you have developers that are very well educated about security but honestly, it's going to be insanely hard to find these people because a developer not only has to be educated about security, they also need to know about UI design and JavaScript frameworks and other frameworks and all of these things. And it's virtually impossible to find someone up-to-date on all of these things. So, what most companies do today that seems to work quite well, even though it's very hard to judge whether it's working or not, is they work with security champions. So, you typically have a dev team and within a dev team, you would have a security champion, one or two or five, depends on how large your teams are, of course, that is knowledgeable about security. So, that developer has some knowledge. He's not an expert but he knows about common attacks and common dangers and how to potentially address them in the application. So, having that person embedded in the team allows the team to be security aware because when you have a team meeting like, "Hey, how are we going to solve this particular problem?" That person will be able to inject security knowledge like, "Hey, that seems like a good idea but if we're using SQL in the backend, we need to ensure that we don't suffer from SQL injection." Or if you're using a NoSQL database, it's going to be NoSQL injection and so on. And that already elevates the level of security in the team. And then, of course, security champions themselves are not going to be security experts. They're mainly developers just with a security focus. So, they should be able to escalate problems up to people with more knowledge, like a security team which can be a small security team within their organization that people can easily reach out to, to ask like, "Hey, we're doing something here and I know that this is security relevant and I'm not entirely sure what's happening here. So, can we get a review of this part of your application?" Or, "Can you guys sit on the meeting to see what's going on and what's happening there?" And I think that structure also makes sense. It's still going to be hard to build secure applications because there's still a lot of things to address, but at least, your teams get some awareness. And then of course, you can help your security champions to become better and they will get better over time. You can augment them with the security architects. You can train your security champions separately with more in-depth knowledge and so on. And that veteran or that setup seems to work quite well in many large organizations today. CHARLES: Yeah. I like that. It gets me to thinking, so having the having the security champions, having people who have this as part of, not their specialization, but at least part of their focus, being in the room, being part of the conversation because we try and do that and provide that service when it comes to UI but we also have a bunch of processes that kind of automate the awareness of quality. So, the classic one is your CI pipeline, your deployment pipeline. So, you're automating your advancement to production. You're automating your QA. It's still no substitute for having someone who's thinking about how to have that quality outcome but you still have some way of verifying that the outcome is quality. Are there tools out there that you can do to kind of keep your project on the security Rails. I'm thinking something that we we've done recently is having preview apps, so that we get a tight feedback loop of being able to deploy a preview version of your application that's on a branch but it's talking to a real backend. There's a lot of more software and services that are supporting this and it's kind of become an integral part of our workflow. So, testing automated deployment preview apps, there's this kind of suite of tools to make sure that the feedback loops are tight and that the quality is verified even though you have people, you also have people guiding that quality. It's just making sure that the standards are met. Is there a similar set of tools and processes in the security space so that we've got these champions out there, they're being part of the conversations. They're making suggestions but they can't be everywhere at once. And is there a way to make sure that the kind of the ways that they're guiding the application, just verifying that the application is going in that direction? Or an alarm bell has sounded. We mentioned one which is the automated pull request with the, "Hey, you got this dependency and there was a pull request." Are there more things like that, I guess, is what I'm saying. PHILIPPE: Yes, there are. But I would dare to say not enough. So yes, you have some security tools you can integrate in your pipeline that do some automated scanning and they tried to find certain issues and alert you of those issues. So, these things do exist but they have their limitations. A tool can scan an application. Some of the findings are going to be easy and fairly trivial, but it's good to have the check in place nonetheless. But some of the more advanced issues are very likely to be undetectable by those automated tools because they require a large amount of skill and expertise to actually craft and exploit to abuse that particular feature in an application. So, we do have some limitations but I like discretion because I do believe that we need to leverage these mechanisms to ensure that we can improve the security quality of our applications. A very simple thing you can do is you can run an automated dependency check when you build the application and you can use that to decide to halt deployment when it's a severe vulnerability or go ahead anyway when you consider this to be acceptable because if you automate all of those things, things can go wrong as well. We can talk about that in a second. So yeah, these things can be done. But what I strongly encourage people to do to ensure that they can kind of improve the code quality is to flag certain known bad code patterns. So if you're building an Angular or a React application, if you're using functions that output go directly into the template, that's going to be very dangerous. So, we know these functions in Angular, they're called bypassSecurityTrustHtml, bypass security should be kind of a trigger and this kind of security irrelevant. And in React, that property is called Dangerously Set innerHTML, also indicating like a 'developer watch out what you're doing'. So, what you could do is you could set up code scanning tools that actually flag these things whenever they appear in application because sometimes people make mistakes. You hire an intern and they don't really know the impact of using that property and they use it anyway which would cause cross-site scripting vulnerability. If you're code scanning to flag these things ensures that it doesn't get pushed to production unless it's a benign case which is actually approved to be in there, then you can definitely stop some of these attacks coming on for sure or some of these vulnerabilities happening. TARAS: I think the hardest thing to understand is when someone doesn't understand what they're doing that what they will create is so cryptic that I think any tool that tries to figure out what it is that person is doing I think will have a really hard time. The person making the thing doesn't understand what they're doing, then the system is not going to understand what they're doing which makes me think that one of the things that we think about a lot at Frontside is this idea of trying to understand the system from the outside as kind of looking at a system as a black box and wonder what kind of tools are available specifically for inspecting the application from the outside, like as if somehow understanding what the application is doing based on what's actually going on inside of the runtime and then notifying someone that there could be something off in the application, but through exercising the [inaudible] things like, for example, memory leaks is not something you can catch unless you have a test suite that has like a thousand tests and then you will see over time that your application is actually leaking memory. But if you run individual tests, you'll never see that. I wonder if there's anything like that for security where at runtime, there's actually a way to understand that there might be some kind of a pattern that's incorrect in the application. PHILIPPE: If only, if only. It depends on who you ask. There is such a concept that's called Dynamic Application Security Testing. Essentially, what you do there is you run the application, you feed it all kinds of inputs, and you monitor when something bad happens. And that means that you have detected vulnerability. So, these things do exist. But unfortunately, their efficiency is not always that good. It very much depends on what kind of security problems you're trying to detect. And they can, for example, detect some instances of things like cross-site scripting or SQL injection or things like that. But there will always be limitations. I've seen tools like that being run as an application where you actually know there's a vulnerability because it has been exploited. There is a manual written exploits and the tool still doesn't find any vulnerabilities which is not surprising, because these things are really hard to make an abstraction of to be able to find that in an automated way with a tool. If you would have such a tool that would be, I think, that [inaudible] would be a lot better. I think there's a lot of funders working on that. But at the moment, those tools are not going to be our savior to build more secure applications. CHARLES: Yes. I mean, it's kind of like linting, right? Or you can make tests. We've been through this kind of all the features or the aspects that we want our application to have, whether it be accessibility. There's certainly a very comprehensive suite of lint level checks that you can run to make sure that your application is accessible. You can run a suite of three thousand things and if it triggers any of these things, then yes, your application won't be accessible but it's not a substitute for thinking through the accessibility architecture. The same thing goes with code linting. You're not going to solve bugs with a linter that makes sure that it's formatted and that you're declaring your variables right and that you're not shadowing things. But you can definitely eliminate a whole classes of things that might be put in there just for maybe even you know what you're doing and you're just forgetful. PHILIPPE: Yes, these rules exist, as well. They're not extensive but there are linting rules for Angular used for security, for example. But the problem in linting is that they are very useful to find potential instances of security relevant features or security relevant functionality. But the linting rule alone cannot decide whether something is OK or not. Just to give you a very simple example, if you use the bypassSecurityTrustHtml function, if you give that function a static snippet of HTML, that's going to be fine unless you write your own attack essentially. But if you feed that function user inputs, you're going to be in a lot of trouble. And making that distinction with a linter is going to be difficult unless you have a static string in the arguments. But if once you start having that from variables to dynamically decide to have a different code path, then that's going to be very, very difficult to decide automatically. So, yes, you can use that to find the places in the application where you should be looking for, in this example, a cross-site scripting in Angular but the linting alone is not going to give you an answer whether this is OK or not. That still requires knowledge of how Angular handles this things, what happens, and how you can do things safely. TARAS: Sounds like we keep going back to nothing beats having knowledgeable developers. PHILIPPE: Yes. Unfortunately, that is true. However, with that said, I want to highlight that frameworks like Angular, well mainly Angular, make things a lot better for developers because yes, you still need knowledgeable developers but the ways to introduce a cross-site scripting vulnerability in an Angular application are actually very, very limited. It's not going to be one, but there's going to be maybe three or four things you need to be aware of, and then you should be set. While if you would have done the same for PHP, it's going to be 50,000 things you need to be aware of that are potentially dangerous. So, yes, frameworks and libraries and all of these abstractions make it a lot better and I really like that. That's why I always refer to abstract things away in a library so that you actually have the ability to look for this dangerous code patterns using linting rules in your code base and that you can, at least, inspect the go to see whether it's OK or not, even though you might not be able to make an automatic decision. You, at least, know where to look and what to approve or how to change in the code base. TARAS: I think that's one of the things that oftentimes is not taken into account that the frameworks are different. And I think of big differences in how much -- like right now, the most popular framework, I think, React. But it's such a thin layer, it's such a small part of the framework that you can hardly call it a framework. But it is something that companies rely on. But then when you consider how much of that code that you need to write, to make React into a complete framework for your company, the amount of code that your team has to write versus the amount of code that your team has to write when you use something like Angular or Ember, there's definitely a lot less parts of the framework that you need to write or a lot less parts of the framework you need to choose from what's available in the ecosystem. Like in Angler and Ember, and I'm not sure what the story is with the view, but the pieces, they come from kind of a trusted source and they've been kind of battle tested against a lot of applications. But I don't think that enters into consideration when companies are choosing between Angular or whatever that might be because they're thinking like what is going to be easiest for us. What is going be [inaudible] for developers? They're not thinking about how much of the framework are we going to need to put together to make this work. CHARLES: I can say it sounds, Taras, like almost what you're saying is by using the frameworks that have been battle tested, you actually get to avail yourself of code that actually has security champions kind of baked into it, right? Is that what you were saying? You keep coming back to 'you need developers who are knowledgeable about security', and if you're using kind of a larger framework that covers more use cases, you're going to get that. Do you think that that is generally true, Philippe? PHILIPPE: Yeah. I think it is and that's why I mentioned that I liked Angular before because Angular actually does offer a full framework. And because they do that, they made a lot of choices for developers and some of these choices have a very, very big and positive impact on security. On the other hand, if you make those decisions, you become an opinionated framework and some people don't like that. They actually want the freedom to follow their own paths and then a less full featured framework like React might be an easier way to go. CHARLES: But I think what happens is folks don't enter into that decision with their eyes open to the fact that they then now need to be their own security champion because they just don't even see it. We said the most dangerous thing is the things that you don't know. PHILIPPE: Yeah, absolutely. And I totally agree. That's something that's at least a couple of years and probably still today, many companies moving into this space struggle like, "Which framework do we choose and why do we choose one or the other and which one will still be there in three years because we don't want to switch to another thing in three years," which is risky to our developers. I like that you said that Angular has this security champion knowledge built in because in Angular 2 and every version behind it, but the new version of Angular essentially, they spent a lot of time on security and they learned from their mistakes in the first version because there were some and they took that and they built a more robust framework with security built in by design or by out-of-the-box. Angular offers, for example, very strong protection against cross-site scripting. It's just there, it's always on and unless you actively sidestep it, it's going to protect you. And that's one of the things I really like about Angular and how they did that. CHARLES: Yeah, that's one of the things that I really like too because I remember there was a blog post back, this is probably, I don't know, almost 10 years ago now, maybe seven or eight years, where someone was comparing why they were more interested in using, their servers were implemented in Ruby and why it was better to use Rails than just Sinatra which is just a very, very, very lightweight HTTP framework. And one of the things that he was pointing to was this new vulnerability was discovered and if you were using Rails, the middle way where the middle square stack is managed by the framework, you just upgrade a minor version of Rails. And now, by default, there's this middleware that prevents this entire class of attack. PHILIPPE: Was that a cross-site request forgery? CHARLES: I think it might have been. PHILIPPE: I think Rails was one of the first to offer built in automatically on support for that. So yeah, that was a very good early example of how that can work really well. CHARLES: And the advantage from the developers' standpoint, because the contrast that, if you'd been writing your application in Sinatra which is this is very, very low level based right on top of rack and you're managing the middleware stack yourself and there are no opinions, then not only do you have to like fix this security vulnerability, you have to understand it. You have to get to do a lot of research to really come up with what's going on, how is this going to affect my application and then I can deploy a fix. And that's like a huge amount of time, whereas you have the freedom to not even understand the attack. I mean, it's always better to understand but you can defer that understanding invariably knowing that you're kind of invulnerable to it. And I think for people who enjoy kind of pretending, not pretending, but that the security world doesn't exist and say, "Hey, I want to focus and specialize on these other areas and attain deep knowledge there." It's very reassuring to know that if a defense for a novel attack comes out, I can avail myself of it just by bumping a version number. PHILIPPE: Yeah, absolutely. If you have everything in place to actually upgrade to that version that fixes those, that's a preferable solution. Towards the future, I believe it's going to be crucial to ensure that we can actually keep things up-to-date because everything that's being built today is going to require continuous updates for the lifetime of the application. I definitely hope that the frameworks get better and more secure and start following these patterns of naming the potentially insecure functions with something that indicates that they are insecure. I think that's definitely a good way forward. CHARLES: Yeah. Can I ask one more question? Because this is something that is always something that I wonder about whenever you talk about any aspect of a system. And part of it is folks will not appreciate good architecture until they've experienced some sort of pain associated with not having that architecture in place. Their project fails because they couldn't implement a set of features without it taking months and years and they just ran out of runway, ran out of deadline. Those types of people who've been on those projects appreciate having a nimble system internally, good tooling. Folks who have experienced good tooling understand how much time they could save, and so, have a very low tolerance for bad tooling. A tool takes too long or is misbehaved or is not well put together, they just can't stand because they know how much time they're losing with security. Is there a way to get people to care about it without having some sort of breach, without having gotten smacked in the face? When you do your trainings, is it generally, "Hey, someone has experienced a breach here, and so they want to bring you in." Or is there some way to get people raise awareness of the problems they don't have to experience that pain but can just experience only the benefit? PHILIPPE: That's, again, a very good question and that's also a very good illustration of why security is so hard. Because if you get everything right, nothing happens. [Laughter] PHILIPPE: Or it might be if nothing happens, that nobody cares enough to actually try something against your application. So, there's no positive confirmation if you've done a good job. You can keep putting things off but eventually, there's going to be vulnerability and it's a matter of how you respond to it. We recently had a cross-site scripting in Google's homepage, one of the most visited pages on the web. And somebody figured out that there were some weird browser thing that could be abused and that resulted in a vulnerability on, let's say, such a simple page. So, even there, things can go wrong. So, what would be a good way to draw with some awareness about this is I would recommend following some simple new resources or some Twitter feeds. I have some security relevant articles there but plenty of other people in the industry have as well. And when you read such an article about security incidents, just think about whether this could happen to you or not. And that should probably scare the shit out of you. Simple examples like the Equifax breach, one of the biggest, most impactful breaches of the past few years happened because of an Apache library that was not updated. I think the Apache library, they had a known vulnerability in there. We knew about it. We had a patch, yet it took too long to install that patch and the attackers abused that vulnerability. This is something that probably can happen to each and every one of us because the attacks started, I think, 72 hours after the vulnerability and the patch had been published. So, ask yourself, "Would I have updated my servers in three days after I got that vulnerability report on GitHub?" Yes or no. And if the answer is no, then the same thing can happen to you. Other cases: Magecart is a very big problem, people injecting credit card skimming malware in the JavaScript library. Are you including third party JavaScript libraries? If yes, then chances are that this can happen to you. And there's nothing preventing someone from exploiting that. It's probably just because you got lucky that nobody tried to do that. And the same thing you see now with all these attacks npm packages where people actively try to get you to install a malicious package as one of your dependencies. And again, everybody can fall victim to these things. So, if you read the articles with that mindset, I probably guess that your security awareness will grow rapidly and you will start caring about that very fast. CHARLES: Yeah. TARAS: Lots to think about. CHARLES: Yeah, there's lots to think about because the next thing that occurs to me is how do you even know if you've been targeted. Because a good attacker is not even going to let you know. PHILIPPE: Yeah. CHARLES: It's just better to siphon off your blood, like you said, than to kill the -- you want to be a vampire bat and come to the same cow every night and just take a little bit of blood rather than the lion that kills the cow and then the cow's gone. PHILIPPE: I would say constant monitoring is going to be crucial and you need that data for all kinds of different purposes. You need to monitor everything that happens, first of all, for a post-mortem analysis. If something happens, you want to be able to see how bad it was. This user apparently got a full admin access and if you have decent monitoring, you will be able to retrace his steps to see what they did or did not get. So, that is one very good use case. A second use case is you can use that data to detect attacks. Usually when the attacks are noisy, it's an automated scanning tool but it might be an attacker trying to do things. Again, that may be something very useful for you to act on to see if there is a problem to prevent that user from connecting, or so on. And then, another very good use case of these things is actually inspecting the logs manually as an ops engineer or whatever, who is responsible for doing that, because that might again yield new insights. I've been talking to someone who said that they discovered an abuse of one of their APIs just by looking at the logs manually and detecting a strange pattern and looking and digging deeper into it. And the automated monitoring tools that they had installed that trigger on certain events like a mass amount of requests to the authentication and stuff like that, they did not catch this particular abuse. So, I would say monitoring there is absolutely crucial, for sure. TARAS: So, the takeaway is higher attentive knowledgeable developers who will learn about security. PHILIPPE: I would say the takeaway is security knowledge is essential for every developer. So, I encourage every developer to at least have a little bit of interest in security. I'm not saying that everyone should be a security expert. We should at least know about that the most common vulnerabilities in web applications, what they mean, what they might result in, and what to be on the lookout for. So yes, I think that's one of the crucial things to start with. And then within an organization, you should have someone to fall back on in case that there are security relevant things that you actually can talk to someone who does see a bigger picture or maybe the full security picture to decide whether these things are a problem or not. I think we're closing or nearing the end here, but one of the things we haven't talked about is how to actually get started in security. What if you are interested in security after hearing this podcast and you want to get started? I want to give you just a few pointers so that you actually know where to look. One of the first things to look at is OWASP. And OWASP is the Open Web Application Security Project. It's essentially a nonprofit that has the mission to improve the security posture or knowledge of developers, and they have a lot of resources on various different topics. They have a lot of tools available and things like that. What you want to start with as a developer is the OWASP Top 10, which is a list of the 10 most common vulnerabilities that exist in applications, just to open your eyes like these things exist in applications today and are definitely a problem. And then, there's a complementary Top 10 called the Proactive Controls and that's about how you, as a developer, can actually prevent these things. So, what should we know about implementing security, which guidelines should we follow. And these two documents are a very good place to start. And then there is a huge community that's actually mostly very eager to help people figure out the right way of doing things and solving these problems we have in our ecosystems. TARAS: Awesome. That's great. Thank you very much. CHARLES: Yeah. I'll take that in. That is really, really helpful. Well, thank you very much, Philippe, for coming on and talking about security. I actually feel a lot better rather than usually I'm thinking about securities kind of stresses me out. [Laughs] PHILIPPE: You can bury the problem but it's going to return in the future anyway, so you might as well get onboard and start learning. It's not that scary if you actually -- it's a lot of fun. So, you should learn about security. CHARLES: Well, I am looking forward to diving in. If anyone wants to get in touch with you, how would they do that on Twitter or Email? PHILIPPE: Yeah, sure. I'm on Twitter. I'm always happy to chat about security. You can reach me by Email as well. I'm very easy to reach. And I'll be happy to help out people with questions. Sure. CHARLES: All right. Thank you so much, Philippe. Thank you for listening. If you or someone you know has something to say about building user interfaces that simply must be heard, please get in touch with us. We can be found on Twitter at @TheFrontside or over just plain old Email at contact@frontside.io. Thanks and see you next time.

In this internal Frontside Podcast episode, Charles, Taras, and Jeffrey analyze the NativeScript Mobile Platform. Please join us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at contact@frontside.io. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer. This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. Transcript: CHARLES: Hello and welcome to The Frontside Podcast, a place where we talk about user interfaces and everything that you need to know to build them right. My name is Charles, a developer here at Frontside. With me today are Taras and Jeffrey. TARAS: Hello everyone. CHARLES: Today, we're going to be talking about NativeScript, in particular, and evaluating technologies and frameworks, kind of at the meta level. So, I'm kind of excited about it because we've been pretty heavily involved with NativeScript for the past three months or so. And so, we've gotten to look at it both from beginners' eyes being kind of totally fresh to the platform, but then actually having to start to pump up against some of the edge cases which is what always ends up happening when you actually use a framework for real. Let's get started. TARAS: All right. I think there's a lot of things that we could talk about because when we would start looking at NativeScript, the length that we were looking at NativeScript through this is that this platform that our client is going to be using for doing development of large applications. So, what does NativeScript need to have to be able to support potentially hundreds of developers building apps? We started looking at it and one things that made us consider NativeScript early on was it kind of provides a platform that allows you to encode in JavaScript and run it on mobile. And we saw this kind of emergence of Angular and Vue.js running on top of NativeScript. So, those things together is kind of exciting. CHARLES: There was also an implementation in progress of React and there were a couple of spikes of Ember also running on top of NativeScript. So, my first impression was initially very favorable. The onboarding experience is actually pretty nice because it was JavaScript and the application was interpreted, there's the ability to completely and totally dynamically change the application at runtime. So, they have essentially an application called the NativeScript Playground which lets you flash a QR code at it and then it will go in to the URL associated with that QR code and it will download all of the assets for a NativeScript application running at that URL. So, all the JavaScript, all the templates, all the whatever, it'll pull it down, it will actually start running like within that app. So, the Playground app then becomes your actual app that you want to use. There's no App Store, no TestFlight, no Google Play. There's no gatekeeping to delivering your application into a running app. And I thought that was really, really cool and really, really compelling. TARAS: We should clarify that this is specifically for preview purposes because if you're going to be shipping the application to production, you still need to go through all those things before... CHARLES: Yes. TARAS: But the onboarding process, you could just install the preview app and then you can point a QR code and it will open that app, whether it's in Angular or in Vue, that app will open up in the preview app and you have a native app that you could play around with. CHARLES: Right. JEFFREY: And that's key both for the engineers who are playing around with this and building this and also really key for the non-engineers who are part of the team to be able to really easily spin up and see what the engineers on the team are working on. CHARLES: That's exactly why we thought, "Hey, we want to be able to use this mechanism for preview apps." In the same way on the server side, you have preview apps associated with a pull request. When we saw this, what we immediately wanted to do was have a bot post a comment onto a pull request with a QR code, so that anybody could just, boom, test out this app on their phone. TARAS: We ultimately ended up setting that up but not quite that way because the original idea of being able to have something like danger bot post the QR code to the comments, you can kind of point out with your phone and open the preview app, that didn't actually pan out. Charles tried to implement that. What happened there? CHARLES: What it actually turned out was that the preview functionality was dependent on a central server, a central NativeScript server. So rather than kind of statically bundling the assets and just saying 'these assets are this URL and just pull them in and bootstrap your NativeScript application that way', it required a lot of extra stuff. So, it required you to be running a Webpack Dev Server that was building your assets and then basically registering and doing some port forwarding with that dev server to a central NativeScript service that was provided by the company that underpins NativeScript. And that connection needed to be hot and live the whole time for that to work. So, while it was really cool that you could get the QR codes up and running, unfortunately that functionality could not be decoupled from the hot update and the central service. Those central services were kind of hard coded into the tools. TARAS: Yeah. So we eventually ended up implementing the preview apps that we wanted but we ended up using Appetize.io to essentially -- the process there is you build the app, you upload the app to Appetize and then danger bot embeds a link to a URL where you can open that app and it will essentially stream like it's running somewhere in a simulator for iOS, an emulator for Android and it will stream a video of that and you can interact with it, kind of like a VNC setup. CHARLES: Yeah. TARAS: And that actually accomplished the goal. It's just we weren't able to do the way that we thought we were hoping to do it straight off with the preview app mechanism. CHARLES: It accomplished the goal. And Appetize is an incredible service that lets you preview the apps on pretty much any type of Android device, any type of iOS device, right there inside of a pull request. But what it didn't allow us to do was pop up your actual device, your actual phone and scan a QR code off of the pull request and pull down the assets. That would have been amazing. But it doesn't always work out that way. And I don't know if that would work long term anyhow because you can't pull down native libraries over the wire and funk them in. That's a big, big no-no. So, the process does have limitations. But nevertheless, that part was really cool. TARAS: Yeah. That was kind of the entry point, the onboarding. And then I think one of the things that was kind of, I remember at the time when we were talking about the NativeScript architecture because we were starting to understand more about how it works. The idea itself is really kind of amazing actually because you have this V8 where you can run your JavaScript code and then they're kind of wired together on iOS and Android. They're wired to the native implementation. So when you're interacting with it, I think the thing that's really great about NativeScript is that the runtime environment for JavaScript essentially gives you API access. In JavaScript, you could say, "I want to create a Java view," and there will be a Java view that's rendered in the actual native device. You're using the same -- the APIs that you find on the Android docs or iOS docs, all of those APIs are available to you as JavaScript. So, you [crosstalk] as JavaScript. And it's seamless, right? CHARLES: Yeah, and it makes it very, very handy. The language is different but the APIs are exactly the same. There is an attempt to make cross-platform components and cross-platform classes that serve the needs on both platforms and then delegate to the platform on which you happen to be running. But those are not mandatory, and the low level APIs are always available to you. An example of this is in iOS, kind of the core foundational object is NSObject. All the controllers, the views, the things, all of them are descended from this object. I can go from object and I can go in from JavaScript and I can just say {let object = new NSObject} and boom! I've got a reference to the actual object and I can pass it around to any other iOS API. That is really, really powerful that there's nothing off limits. There's nothing at an arm's distance. There's really not much you can't do because all of those things are available to you. There's nothing that's off limits. That means that they can build cross-platform components on top of those APIs. Whereas a sort of system like React Native which does have cross-platform components, that's kind of where the base layer is but you can't crack open the hatch and go down the next level and start mucking around, unless you want to actually start meddling with the React Native source code or recompiling Swift in Java code. TARAS: For me, I think this architecture is probably my favorite part of NativeScript. JEFFREY: Mine too. CHARLES: Yeah, me too. TARAS: I really like this part. I kind of hope that everything else is as clever as that was. CHARLES: Because among other things, it allowed us to write a Bluetooth. We were able to implement Bluetooth using nothing but JavaScript. We didn't actually have to go down and do any Swift and do any asynchronous message passing between the iOS libraries and the JavaScript libraries. It's like, "No." We've just got a very simple cross-platform interface that instantiates an implementation for Android and an implementation for iOS, but both of them are like JavaScript. And so, it really is you're doing native development but it's JavaScript all the way down. TARAS: Yeah. And when you're writing plugins, your plugin is actually JavaScript plugin that is assuming iOS APIs and Android APIs. CHARLES: Yeah. And if you have to have a native plugin like a CocoaPod or an Android Package, you just install it and you can instantiate it from JavaScript. There's no fuss, no muss, no ceremony. It's just like, "Hey, I want to use the..." what was the one we like to use? The Material-UI floating button which is a CocoaPod. You download it, you link it into your application, and then you just instantiate it from JavaScript. TARAS: That was really cool. The challenging part was that a lot of that kind of awesomeness, like everything around it wasn't quite as polished. And so, one of the big things is that like around tooling, because one of the things about having grown up in a way like in the Ember community, in a sense, we have a certain expectation of what the level of polish from tooling that we would expect. And it's kind of supported in the way like when you look at how React or React Native tooling is, even Angular tooling, it's very polished. You kind of expect to see what you need to see when you're looking at a CLI input and you don't see anything else. That level of polish. I think part of the changes that they're going through, maybe that's part of the reason but that same level of polish isn't available around the tooling. CHARLES: There are these fantastic qualities about the platform and it is amazing. We were using Angular and a lot of people are using Vue and things like that and that actually is pretty incredible. And there is nice tooling, there is command line stuff, but we started to run into issues where, for example, it was very clear that we were pretty much, as far as I could tell, one of the very, very few people running a NativeScript project on CircleCI or in a CI environment at all. It had capability for testing, both for acceptance testing and for unit testing, but it required changes to the core framework and the core tools in order to get those tests to work in a CI environment. JEFFREY: Before we kind of get into the testing story there, some of the issues were around determinism of reliably reproducing your whole NativeScript environment and stack every time because that's such a key feature of doing it. And on a CI server, it's like, "Hey, we need this to load in the same exact packages every time." And so, we ran into challenges there. TARAS: I think we spent almost two days. There's example projects in different combinations. One thing that was off was that there's a pattern that is applied in a lot of the plugins in NativeScript ecosystem is installing things. So, you run npm install and npm install will generate some files. And so, when we're trying to move it over to a CI, there were files, like there's hooks, like TypeScript hooks that were excluded that you can ignore, but they were necessary to compile the TypeScript. And so, what was happening is when we're running these at CI, the application, we would build the app but the app would crash the moment that you start it. And the reason for that was that the JavaScript files that were transpiled from TypeScript to JavaScript, those JavaScript files were actually never included because they were never transpiled in CI because the hooks directory, like we weren't preserving it between our tasks and so... CHARLES: Right. We weren't caching. This was an artifact of the install. And so, we were caching the install, so essentially the yarn.lock was not changing. But the directory was not getting generated unless the cache key changed. TARAS: And we spent spent quite a lot of time... CHARLES: Two or three days out. TARAS: Yeah. CHARLES: What that said is, "Oh, nobody's really running this in CI." Nobody's actually building an app from scratch every time. TARAS: There are people in NativeScript team that actually does a great job of documenting. They did have example projects that exist but sometimes that example project doesn't fit like a perfect combination of what you're looking for. There was an example project that was showing how to run on CI but it didn't use TypeScript. And so, that's where we lost a lot of time. CHARLES: Right. JEFFREY: So, let's talk about testing since that's kind of the core, the most important part of why you even want continuous integration capabilities to begin with. What did we run into there? What did it look like? TARAS: Well, I think it's safe to say that we were really on a bleeding edge of testing capabilities in NativeScript ecosystem with Angular, at least. But I think it was still an interesting project. We were using the latest builds. And I have to say I think this is one of those things that's going to be kind of consistent through this, is like the people in NativeScript team are amazing. They're so easy to work with. They're so accommodating. When we ask for stuff, they're on it. But it was a lot of things we're trying to figure out like how do we run unit tests, what can we do. Ideally, we wanted to run, first and foremost, we started with how do we run functional testing. So we spent quite a lot of time trying to get Appium set up. I spent a good two to three weeks on that and it was not productively spent time. CHARLES: I think ultimately, we had to pull back from it. And there were a number of reasons. Part of that is there are multiple paradigms for how you can build your NativeScript application. So as we speak, there's a move towards using Webpack to build all of your JavaScript in your style sheet assets because it's very much like a React Native application. You've got style sheets, you've got JavaScript assets, that some of them might be in TypeScript, some of them you might be using Babel, and you need to actually transpile them down to include them in a way that your underlying JavaScript runtime is going to be able to understand. But that wasn't always so. They have their own build system and packaging system, they kind of used the TypeScript compiler ad-hoc, if you were using TypeScript, which we were. And so, this was kind of this orthogonal complexity, I guess, where you have your unit testing and it has to play nice with this one package or Webpack. There were multiple ways to package your app. And so, we ran into problems where, like TypeScript kept coming up as a problem and the way in which we were bundling our assets. So, in order to get TypeScript to work, we kind of had to get Webpack running. But the problem is it felt like three quarters of the tooling wasn't Webpack compatible yet. And so, it meant that other pieces of the build were breaking because of this. And so, we had to be on the bleeding edge of several different aspects of the runtime. And the problem is when you're on the bleeding edge, that can break other stuff. TARAS: But there's complexity in running on native platforms that I think a lot of this complexity is kind of leaking to development experience because one of the challenges is your tests need to run on the native device in the application. So, you have to build the app. You have to push the app into the actual device. So, there's like all the setup of installing the at the app on the device. CHARLES: You have to launch the simulator. TARAS: Yeah, right. CHARLES: To make sure the device is connected. TARAS: And you run your tests in there. So, that created kind of this situation where we say let's just kind of set Appium aside and just use unit testing which is a very small fraction of the kind of testing that we actually want to do. It will test very little. But let's just do that because getting functional testing to work was really kind of not going anywhere. So once we start doing unit testing, one of the challenges is that it takes like 30 seconds to start your tests. And then, if you for whatever reason, made a mistake, the moment you cancel the build, it leaves, like it doesn't clean up of itself well. So, it leaves processes running in the background. And so now, you spend another like 10 to 15 minutes Googling around for a cookie, "How do you find these processes and stop them?" So, we eventually settled on having a script that does that, but this is the kind of things you have to end up doing because there's a bunch of things that are wired together, but they're not wired together in a way that is seamless. And so, you end up kind of just debugging a lot of stuff where you just want to run some tests but you end up doing all these other stuff. CHARLES: Right. TARAS: And you spend a couple of minutes just doing something that you'd expect to happen in like 20 seconds. CHARLES: Right. There is a feeling that every aspect of the system is coupled to every other aspect of the system in kind of varying ways of interconnectedness. And that's not what you want for a very, very complex system. You want it to be extremely modular. So, I think we should keep the command line tool. There's probably a separate discussion, I think, about that. But you have to close the book on the Appium and the unit testing. I think the other problem was that you have to run these things on simulators. On macOS, that's not a problem because the simulators ship with X code. And so, you don't actually require an external service. Whereas in CI on Android, it's very unlikely that you're going to have Android emulators on hand because they require a separate virtual machine. Android emulation is actually quite heavy. If you're running through Android Studio or something locally, you essentially need VirtualBox or some equivalent to run your Android simulator because you actually need that simulated hardware. If I understand correctly, that was actually not something that had been really accounted for. It was that you might want to be running simulators not on the same machine as what you were developing on or what the actual that you were building on. TARAS: Yeah, a lot of the tooling seems to be designed around this idea that you're going to be building and running everything on your machine. And so, you can spin up a virtual machine easily. But in CircleCI, for example, they don't support running a virtual machine inside of a Docker container because for that, you need a feature of a virtualization that is not supported in many CI platforms. You have to run a parallel server if you want to have like Appium running, for example. You need to have a separate server running like an Azure or a Google Cloud somewhere that is able to run virtualized servers that have a host machine that's being guest systems that are running the actual Android emulators of different versions. And so, when I started doing research in this, there are companies that are doing this really well but it's not unusual to be using hardware from Amazon that costs thousands and thousands of dollars per month. I think for anyone who's getting into mobile development, I would say the hidden gem of Android world is Genymotion. Those that do a lot of Android development, they know about it. But Genymotion has both like a desktop environment and it has SaaS offering that they're in the process of releasing. And so, what it allows you to do is when you run it locally or on your local machine, it allows you to create a virtual machine that is running in VirtualBox and then it allows you to run kind of optimized environment for running Android. And when you do that, it's really fast. It's very smooth. It makes running Android devices locally as easy as it is to run iOS devices on macOS. CHARLES: I remember starting out and trying to actually just get any Android emulator running on my Mac and I couldn't even do it. JEFFREY: It was such a huge time saver. CHARLES: Yeah. TARAS: And to have this Saas offering is really great because you could basically create your virtual machines on demand and then you install into a virtual machine from your CI server and then you run your tests there. That's kind of the key that I found to be able to run tests and automate it against emulated devices for Android. Genymotion is really great. CHARLES: Yeah. Again that's the kind of thing that you need when you're in CI. And so, one of the things, I think, one of our discoveries is that there just isn't -- when we started working on this and we haven't seen a culture of running these tools in the cloud and accounting for the fact that you might have not all of the tools running on the same machine. From, I would say, the beginning, I remember the kind of the diagnostics command didn't work but we were running it on a CI server. So, there's a diagnostics command that you run to see do you have this, do you have that, do you have that. It would work and give meaningful results when I wanted to debug my CI server because when we were initially getting set up, something wasn't building right, there was some dependency missing. And I just wanted a diagnosis but it was trying to install all those tools for me. And I was like, "No, no, no. I don't want you to do anything. I don't want to install them. I'm going to be doing all of that as part of the setup of the CI environment. It's going to be installed, it's going to be cached. I don't want you to just try and like massage my system into a suitable state for NativeScript development. I just want you to diagnose what is wrong. Tell me, am I missing this compiler? Maybe I've got the wrong version of Android SDK. Tell me what's going on." And I couldn't get that to work. That was very frustrating. I think it was because the kind of bulk of the assumptions was that it was going to be individual developers working on their own laptops or their own desktop computers to build, to test, to distribute these applications. I think that's becoming less and less the case. I mean, at this point, that's not a way that we're willing to operate. TARAS: And we eventually figured out how to do all this stuff, right? CHARLES: Yeah, we have. JEFFREY: We have. TARAS: We have the entire process working but it took a lot longer than one would imagine. It took all the time that we had allocated to it which we thought was very generous amount of time but it took like almost a month to get everything set up. The great part of this is that we do have now everything working. And so, there's a repo where people could take a look if they want to get all stuff working on CI, but it took quite a bit of work in figuring out. CHARLES: Yeah. Actually, I think worth probably a Screencast to show some of those capabilities because it is really exciting. I mean, when you actually think about the pipeline in its entirety. But we never were able to get functional testing working. TARAS: And then the challenge here is that because we were essentially looking at NativeScript, going back to this question like, "What do we need to be able to have like hundreds of developers potentially running on this platform?" And so there's a lot of considerations and this tool is just one of them. I think the other one that is a big one is like what are the capabilities of the view layer because that's where most of developers were spending most of their time. We got stuck a little bit about that because I spent a lot of time working in the view layer. The thing that was really great and the thing that I really liked about it is the fact that you have a collection of components that you can use in Angular. You render it as component and then that component is going to look correctly on iOS and is going to look correctly on Android. From a single code base, it's building appropriate components for iOS and Android. What I think is really confusing in that case, though, is because the Android and iOS components don't have parity in a sense. They don't behave exactly the same. And there is also a kind of a reputation in the NativeScript documentation that Android tends to be slower, much slower than iOS. And so, when you start to run into performance problems and you start to run into those pretty fast because it is not really clear what is necessary to not optimize NativeScript, when you start to run into performance problems, it's not really clear like where is it coming from. Right now, the profiling that they have for the UI is very limited. They're kind of in the process of migrating over to chrome.debugger, but profiling in chrome.debugger is not implemented. You can do performance optimization using Android tooling but that's only going to tell you performance of the Java side, or the iOS side is not going to tell you the performance of the code that's running inside of JavaScript. It's not really clear what is causing the problem. If you don't know what's happening, you kind of write it off as like, "I think it's just Android being slow." In reality, when you actually start to dig deeper, you realize there's things about the Android implementation of the components that are different or the views that are different than iOS. And it's the differences that add up to weird performance problems. That's probably the thing that gave me the most hesitation because one of the things that made me think like if we want to be able to give this to a team of like 50 people, we need to have our own view layer because we cannot rely on components. An example of this would be, they have a list ticker on iOS, it doesn't omit change events when you scroll. If the list is moving, it change events and not omit it. But on Android, every time that a different item shows up on a screen, it changes the selection. And so now, you've got this view that's a meeting on Android as a meeting change events. I made an issue around this and the response was that while there's a workaround that you can have for this, but that's hard. Work around is not a solution. CHARLES: Right. When you have a leaky abstraction like that. TARAS: Part of the problem is because people use leak abstraction. And so, what's happened in Native -- we actually got on the call with NativeScript core team and they're excellent in really being very helpful, understanding what the problems are, and providing pass on making things better. But what's happened as a result of having this leaky abstraction is that people are relying on the leak. And so now, the leak is the API. And so, we can't change that. JEFFREY: Right. CHARLES: And the answer that you really need there is, "We can't change that without breaking stuff. Here's our migration path for deprecating this and introducing a new API." And that gets more into the process stuff and it seems like the process for making changes to the underlying API, I think, could use a little love in the sense that it's kind of opaque as to where the platform is going. There's not a concept of like an [RSC], there's no roadmap about what to expect. What is this API going to look like in the future? Is this stable? If I were writing a software and someone said, "Hey, there's this leaky abstraction," I think my reaction would be, "We've got to fix this." And we also have to acknowledge that there are users who may depend on this. And so, we have to be very deliberate about it. TARAS: The challenge with this too is that NativeScript kind of outgrew its hands because I think originally, it wasn't meant to be hosting Angular and hosting Vue. Vue didn't exist. Angular didn't exist when NativeScript started. So I think what's happened is that these views that were available, I wouldn't call them components because they don't act like components, but they're exposed in Angular like components but the API feel like Vue objects. So these Vue objects that you consume, that you render in Angular, for example, or in Vue.js, they are the same APIs that NativeScript had before Angular and Vue.js. CHARLES: Right. You know what? It feels like there's a MVC framework, like a Circa 2010, 2012 MVC framework that has now become the foundational layer for Vue frameworks that have had significant advances in the way we conceive of model in Vue and how data is generated and passed around and how views are rendered off of the data and how reactivity is changed. But there's still, the underlying platform has not evolved. And in fact, this was originally user-facing APIs and now these APIs have become foundational for other user-facing APIs but haven't had the iteration and evolution to make them robust. TARAS: And flexible enough. As a result, you have the situation where not only is it really super easy to deoptimize the views simply because the requirements of keeping performance expectations are not obvious. One of the things that I found is that the list which is, lists are like 50% of most applications. Before I go into the problem with list, the nice thing about lists in NativeScript is that because they're interacting directly with native APIs, you have really fast list when they're optimized. They're really easy to work with. But they easily get deoptimized by the fact that the expectation to keep the list fast, you have to use this API in NativeScript called array observable and observable. And this is not to be confused with like... CHARLES: [Inaudible] observables? TARAS: Yeah. CHARLES: It's not to be confused, but in fact, every conversation involves a lot of confusion. Because we were using observables, right? TARAS: And we were actually using observables. So, we're using observable [inaudible] and we're using this array observables and object observables. And so, it's necessary for NativeScript to, essentially what it expects for list to be fast, is it expects that it's going to receive an array observable which is an object that wraps an array because it needs to know when an order or length of data rate changes. So what happens when you pass an array observable, a NativeScript array observable into a list? It will listen for change events on that object. But if you want to change the value of each of the items, like if you want to change a property on the object and have your view remain optimized, the array observable has to have an observable object which allows NativeScript ListView to listen for changes, property changes on the object. You pass this array observable which contains observables that ListView listens for changes on to make sure that it knows how to correctly apply this change to the list. If you don't have this magic, like if you haven't figured out this recipe for ListView performance success, you're going to have a really hard time because it's really not clear at what point and how this thing got deoptimized, why has it just gotten slower. CHARLES: There's a lot of iteration that needs to happen there and it's not clear what the plan, what the priority, or even how you will even begin to go about this. Because I think that the internal working is that it seems basically to be controlled by one company. I don't recall seeing any contribution from anybody except for Progress which is Progress Incorporated is the company that's kind of the controlling interest, the original company that developed it. TARAS: The way this showed itself very practically is that to make changes too -- so they have a ListView which comes with NativeScript public and there's RadListView which is the component that has a lot of stuff on it. Like if you want to pull to refresh or if you want to do like laser loading a data or if you want to do a filtering, you want to do -- so most people use RadListView. But RadListView, you can install, so there's no limitation when you build to install it, and your node modules has the source code for that. But the source code, the original TypeScript code, untranspiled code is not publicly available. They have a process for doing this and it's very nice that everybody's very kind and very accommodating. You send an email, they'll give you access to this repo and then you'll have the ability to contribute. NativeScript core team is very helpful and they're open to contributions. There are changes that need to be done to the Angular implementation to make it faster without having to put the requirements of the observable thing, and so they can give you a path to make that stuff happen but it's not open source in the sense that it's not a traditional open source that we would kind of expect. So, there's all kinds of hoops that you need to jump through and the source code is very difficult to read because it's transpiled from TypeScript to JavaScript. CHARLES: And there was a certain level of opacity in terms of process. For example, I filed an issue which was actually a blocker. For us, it was actually causing our Android build not to work. I didn't hear anything about it. And then, all of a sudden like four days later, a fix came through referencing another repository on which this thing depended with. There was not a lot of context service. So it was obviously referencing a bunch of context that probably happened between two people in a face-to-face conversation. But I couldn't really tell what was going on, why it was an issue, because there was no comment. It was just a pull request that was referencing this issue. I never got a notification. I actually had to go and be like, "Hey, I really would like for this issue to be solved. I wonder if I..." I was actually going to post a, "Hey, is there any progress on this?" Or, "Is there any way that I can help? What can I do to get this looked at?" And I saw that there was another pull request that had referenced my issue. And it was merged and I looked down, but then there was no indication of when this would be available for public release, how I might be able to work around it. And so, the strange loop that didn't get connected was, "Hey, you've got a user who files an issue. You actually use this as the impetus to fix the issue and make a release." But then that whole process was completely invisible to me. TARAS: You know what? It sounds like you wanted for it to work [inaudible] but you got a pulling mechanism. CHARLES: Yeah, exactly. Well, I wanted someone to say like, "Hey, here's what's going on, and we're looking right into it." Or, "We're going to look into it in like two months," or, "We can't address this now. But here's a workaround for it." Or, "I don't have a workaround." That's just kind of the expectation that you have when you're playing with open source. In many ways, it does not feel like an open source project. TARAS: Let's just do a quick note about Saas. Jeffrey, what did you find about the styling of NativeScript views? JEFFREY: All the components that come kind of shipped as part of the NativeScript core set of components all have styles attached to them. They have CSS attached to them. And as part of the standard data script workflow, with your build toy, you have SaaS available which is very nice. But actually on a recent project, we're not using Saas at all. We're simply using post-CSS and we were able to kick out some CSS variables that turned out to be really nice for theming. So as kind of a future friendly experiment, we were trying to have a light theme and a dark theme since that is very recently now a core part of Android and very likely will be part of iOS this year, where there's kind of a light theme and a dark theme for everything. We were trying to do that. The simplest way to do that with standard web tools is with CSS variables. You can have the flexibility, you have the theming with those. It's so nice. You just, "Hey, my primary color is this color in one scenario and it's this color in another." And we just didn't really have the flexibility to do that with SaaS by itself. And so, that's kind of a limitation of the tooling right now that I hope in the future, we'll have some more sophisticated CSS tools. And really, NativeScript's move toward Webpack and having that as a primary part of the workflow really opens up that possibility that I hope somebody runs with in the near future. TARAS: Yes, let's bring it all back together. CHARLES: Can we pause for a moment? Because I actually do think it's important that we at least touch on the command line. I can give a little bit of a kind rant in here but I think that's actually something really important that we have to talk specifically about that. The other thing that I wanted to touch on very briefly as we kind of draw to the close is the command line tooling, in particular in NativeScript. I think that this is probably one of the weakest points of the platform. And again, I don't want to disparage anybody working on NativeScript. It's an extraordinarily complex problem. This is a command line tool that needs to manage launching simulators, installing things into simulators, pushing code to those simulators. It needs to handle hot updates to things that it's running on, devices and simulators. So, it needs to be building JavaScript assets either with Babel or with TypeScript. It needs to be building those SaaS assets that you were just talking about, image assets. But it needs to be doing all of this for two platforms, so it needs to be managing everything that I just described. It needs to be managing on iOS. Everything that I've just described needs to be managed on Android, as well. It needs to work for a single developer's desktop. It also needs to work with all of those components that I just described distributed out in the Cloud. So, we're talking about an extraordinarily complex piece of software. And I think that unfortunately, the NativeScript CLI does not inspire confidence because it can do all of those tasks. But Taras, you also mentioned often if you stop the process midway, it will leave a thousand things open and they're just spewing output to your console. The console output, unfortunately, means there's a big noise to signal ratio because it puts out all of the content for Webpack. Every little thing that it's doing with any of the devices, it's logging to the console. So, it doesn't give you a sense of control. So, what you really are looking for in terms of a command line is, "Hey, I've got this incredible sprawl of complexity and I want to feel like I'm on top of it." And unfortunately, by leaving these things open and having so much console output and having the console output not be formatted well, there's all kinds of colors. Every single tool that you're using whether it's Webpack or whether it's Karma or whether it's just console outputs that you are happing inside of your NativeScript application, the brand of those tools comes through. Webpack is a great example. Its console output feels very Webpack. So when you've got Webpack content randomly interleaved with your console content from your Mocha content, from Karma, all of these competing brands, it doesn't feel like a cohesive developer experience. And so, I really, really hope that -- so, to the point being where I felt like I could not live with that command line tool without rewriting it myself. If we want to use this platform long term, we'd have to either have an alternative command line tool or really, really, really help the NativeScript team completely and totally rewrite the command line experience. TARAS: I would love to work on fixing a lot of these parts about NativeScript if there was a way to actually do it in terms of like, if they wanted to pay us to help them kind of bring some of these things to a state that would match. For example, what's available in Ember or available in React CLI, I would love to do that. CHARLES: React Native, yeah. TARAS: Yeah, let's do that work. But who knows what's in store? A lot of awesome platform like the idea around NativeScript architecture is fascinating and it's really, really powerful and really wonderful people doing some, trying to tackle really challenging problems, but it's all glued together in a way that doesn't instill confidence. And it just makes everything feel wobbly, just makes it feel like you never know, is it a problem? Where's the problem from? What is causing this? CHARLES: Yeah. And if I fix this thing, is it going to break something else? TARAS: Yeah, we've seen it happen actually with one of the solutions that was introduced to a bug that you were referring to earlier. CHARLES: Yeah. So that was our three months experience working with NativeScript. TARAS: We are considering other things now, very seriously looking at Flutter as an alternative for the same client, same scenario. Flutter is looking pretty exciting. There's a lot of things that are really good there. So in three months, we'll do another report and talk about Flutter and what we found. So, that's it. CHARLES: And I will say I'm actually not like super excited about dart but I'm in dart spot. JEFFREY: That's a whole other conversation for yet another episode. CHARLES: I think that, to continue the conversation maybe next week, next time we have kind of an internal podcast, is I would like to really talk about platform evaluation because really you need three months, at least, to get a good idea of this. Is this going to work for the next five years? And most of the time, we give it a week or give it a two week. Or someone comes on who's really excited about this one particular technology and you go off on that tangent. I think there's an interesting meta discussion about how do you select technologies. And we don't have time for that now, obviously. But it's definitely something that I want to have in the future. TARAS: Sounds good. I think that will be a good conversation for sure. CHARLES: I guess that is kind of the executive summary on NativeScript from our perspective. With us being three months in, I think, like you said, there's a lot there. Thank you for listening. If you or someone you know has something to say about building user interfaces that simply must be heard, please get in touch with us. We can be found on Twitter at @TheFrontside or over just plain old email at contact@frontside.io. Thanks and see you next time.

Luke Melia, Aaron Chambers, and Mattia Gheda john Taras and Charles to discuss all things deployment! Luke Melia: Luke has been working with Ember since it was under early development as Sproutcore 2.0. Ember.js powers a SaaS company he co-founded, Yapp, and they funded their business for a couple of years doing Ember consulting under the Yapp Labs moniker. They're full-time on product now, and his engineering team at Yapp (currently 3 people) maintains around 6 Ember apps. Luke helps to maintain a bunch of popular addons, including ember-cli-deploy, ember-modal-dialog, ember-wormhole, ember-tether, and more. He started the Ember NYC meetup in 2012 and continues to co-organize it today. Aaron Chambers: Aaron Chambers: Aaron is the co-author of EmberCLI Deploy and is currently an Engineer at Phorest Salon Software, helping them move their desktop product to the web platform. He's been using Ember for 5 years and maintains a number of plugins in the EmberCLI Deploy ecosystem. Aaron loves trying to work out how we can ship JS apps faster, more reliably and with more confidence. Mattia Gheda: Mattia is a Software Engineer, Ember hacker, Ruby lover and Elixir aficionado. Currently he works as Director of Development for Precision Nutrition where Ember, Ruby and Elixir power several applications. He loves meetups, organizes Ember.js Toronto and co-organizes Elixir Toronto. Resource: Immutable Web Apps Please join us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at contact@frontside.io. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer. This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. Transcript: CHARLES: Hello and welcome to The Frontside Podcast, a place where we talk about user interfaces and everything that you need to know to build them right. My name is Charles Lowell, a developer here at the Frontside. With me also co-hosting today is Taras Mankovsky. Hey, Taras. TARAS: Hello, everyone. CHARLES: Today, we have three special guests that we're going to be talking to. We have Aaron Chambers, Luke Melia, and Mattia Gheda who originally met collaborating on fantastic open source library that we, at the Frontside, have used many, many times that saved us countless hours, saved our clients hundreds of thousands of dollars, if not more. ember-cli-deploy. We're gonna be talking not about that library in particular but around the operations that happen around UI. So, welcome you all. LUKE: Thanks, it's great to be here. CHARLES: Like I said, I actually am really excited to have you all on because when we talk about the platform that you develop your UI on, something that often gets short shrift in communities outside of the Ember community is how do I actually deliver that application into users' hands. Because obviously, we don't want it to be working just on our laptop. We want it to be delivered to our users and there are myriad ways that that can happen and it's only gotten more complex since the last time we talked which must have been like three or four years ago. I kind of just have to ask, I think that what you all were talking about then was cutting edge is still cutting edge now but there must have been some pretty incredible developments like in the last three or four years. What have been kind of the new insights that you all have? LUKE: I think that what we realized as we got started with ember-cli-deploy and a project kind of came together as a combination of a few different open source efforts, something that Aaron was working on, something that our collaborator Mike was working on. We decided to come together under one umbrella, joined forces. And what we realized pretty soon is that deployment needs vary a ton between companies. And so, we are coming from this background in Ember community where we had this attitude where nobody is a special snowflake. We all kind of have the same needs for 90% of what we do. And that's true. I really believe in a lot of that Ember [ethos]. But when it comes to deployment, you know what? A lot of companies are special snowflakes or it's at least is much more fragmented than kind of our needs around on the JavaScript side. And so, what we decided to do was to try to evolve ember-cli-deploy into a platform essentially, an ecosystem that could let people mix and match plug-ins to do in their organization without locking them into an opinion that might simply be a non-starter in their org. CHARLES: It's hard enough to have opinions just around the way that your JavaScript code is structured but when it comes to rolling out your app, it really does encompass the entire scope of your application. So, it has to take account of your server. It has to take account of your user base. It has to take account of all the different processes that might be running all over, distributed around the Internet. Maybe somewhere on AWS, maybe somewhere on Legacy servers but it has to consider that in its entirety. So, it's having opinions that span that scope is particularly difficult. LUKE: Yes. And so, you mentioned a bunch of technical details which are absolutely forcing factors for a lot of words in how they do their deployments but what we found in talking to people that there are also people in political aspects to deployment in many cases. Engineers kind of own the JavaScript code that's running within their app, more or less. But when it comes to pushing the app into the world and a lot of companies, that means they're interacting with sysadmins, ops folks, people who have very strong opinions about what is an allowable and supportable way to get those deployments done and to have that stuff exist in production. And so, we needed to come up with an architecture that was going to support all these kind of varied use cases. And so, we came up with this system of essentially a deployment pipeline and plugins that can work at various stages of that pipeline. And that ecosystem has now grown quite a bit. It's actually, I don't know Aaron if you and Mattia would agree but I think it's probably the best decision that we made in this project because that ecosystem has grown and evolved without us needing to do a ton of work in maintenance. And it's been really great. I think Mattia, you pulled some of the current numbers there. MATTIA: Yeah. I pulled some numbers just yesterday and we have currently 150 different plugins attached to themselves, to different parts of the pipeline. So some of them are about how to build the assets, some of them are about how to compress them, some of them are about shipping. And they allow people to ship it with different ways like we are seeing [inaudible] with just simply Amazon APIs or Azure APIs and some of them even are about just how to visualize data about your deployments or how to give feedback to the user about what was deployed and represent the information. And then this is kind of a bit more detail, probably specific to us but we also added this idea of plugin packs. So, in order to help people define their deployment story, we created this ideal of plugin packs. Plugin packs are simply group of plugins. So, plugins grouped together. As a user, if you want to implement what we call a deploy strategy, you can simply install a plugin pack and that will give you all the plugins that allow you to deploy in a specific way. And that's kind of like an optimization that we added just to make it easier for people to share deployment strategies, share ways of deploying applications. CHARLES: Right. It's almost like an application within the framework. MATTIA: Yeah, exactly. But to stay on the community side, I think that the interesting part about what Luke was saying which was a great success for us is that all we maintain as the core team for this project is the core infrastructure, so the pipeline and a thousand of plugins. Everything else is community-based. And often, even in my day-to-day work, I end up using plugins that I didn't write and that I don't even maintain. But because the underpinning of it were designed especially by Luke and I are flexible enough. It just like has been very, very stable and very, very reliable for many years. So, I will say definitely the idea of like in the spirit of what Ember is, I guess, creating a shared ecosystem where people can add what they want and extend what was provided has been the one single biggest win of this project. CHARLES: One of the things that I'm curious about is we've talked about how you're allowing for and kind of embracing the fragmentation that happens in people's kind of the topology of their infrastructure. What do you see is the common threads that really bind every single good deployment strategy together? MATTIA: My biggest thing here, and we actually have some shared notes about this, but my biggest thing about this is the idea that building and deploying an application for me is divided into three parts. There's building part where you have to decide how to compile your JavaScript application and how to produce some sort of artifact. There is the shipping part where it's about deciding where you're going to put the artifact. And then there is the serving part which is how you show it and deliver it to your users. I think that these three are the underpinnings of any deploy strategy. What we did with this project is just acknowledging that and give each one of these a place. And so, the entirety of what we do in what Luke defined as a pipeline is simply give you a way to customize how you build, customize how you ship, and then customize how you serve. So yeah, I think that that's kind of the root. And the question that everybody that wants to deploy a modern JavaScript application have to ask themselves is how do I want to build it, how do I want to ship it, and how will I serve it to my users. And these things are completely independent one from the order in the sense that you can have something build it, something ship it and something serve it, and that's what we end up doing in most of our deployments, I find. CHARLES: It's good to think about those things as soon as you possibly can and make sure that you have all three of those bases covered really before you start adding a whole ton of features. TARAS: Sprint 0, right? LUKE: In Agile, we call Sprint 0 the phase the thing you do right in the beginning. You've got a skeleton of an app and then you get the deployment infrastructure going, you have the test infrastructure going, so that there is no task within your actual feature development where you have to do those things. And I think that can be a valuable concept to embrace. I would just add to Mattia's three points that for deployments, to me, some very simple qualities of a good deployments are repeatability. You need to be able to reliably and consistently run your deployment process. Sounds simple but there's plenty of operations that have run up way too long on manual deployments. So, we don't want to see those rollback capabilities if you have a deployment that you realize was a mistake right after it gets into production. I'm sure none of us have ever experienced that. CHARLES: That never happens to me. LUKE: You want to have a method to roll that code back. That's something that can be remarkably complex to do. And so, having some guardrails and some support mechanisms to do that like ember-cli-deploy provides can be really useful. But whatever your approach is, I think that's a necessary quality. And then I think we start to step into kind of more advanced capabilities that a good deployment architecture can provide when we start to think about things like personalization, A/B testing, feature flags, these kinds of things. And that requires more sophistication, but you cannot build that on a deployment foundation that's not solid. AARON: I think for me, one of the things I've been really thinking about a lot lately, it's a bit of a mindset shift, I think, to get to where the things Mattia was talking about separating those different parts of deployment. And so, I really start to realize the traditional mindset around deployments like I build some stuff and I ship it to the server and then the users get it. But if we can actually stop and actually split our understanding of deployment into two separate phases. One is the building and the physical shipping of the files; and the other one's actually making them available to people. You open up this whole world of our features that you wouldn't normally have. So to be able to actually physically put stuff in production but not yet have it active, as in users don't see it yet but you can preview those versions in production against production databases. And then at some point after the fact, decide, "Okay, I'm now going to route all my users to this new thing," And to be able to do that really easily is massively, massively powerful. And so, to me, the thing I've been thinking about lately is it is a small mindset shift away from packaging everything up and pushing and overwriting what's currently there to being something, again like Luke said, immutable deployments where everything we build and ship sits next to all the other versions and we just decide which one we want to use to look at it any time which leads into then, I guess, A/B testing, feature flags and things. So I guess deployment really is not so much about the physical shipping, that's one part of it. To me, deployment now is shipping of stuff, as in physical deployment and then the releasing it or enabling it or activating it to users. CHARLES: Or routing it. Sounds like what you're describing is an extraordinarily lightweight process. AARON: It is, yeah. CHARLES: To actually route traffic to those files. AARON: It is. It's incredibly lightweight. That's the amazing thing about it. When you think about it, you're building a few JavaScript files and CSS files and images and putting them on a CDN, and then you just need a tiny web server that basically decides which version of the app you want to serve to people. There's not much to it at all, really. CHARLES: I mean that's absolutely fascinating, though the capability that you have when you have the ability to have these versions, the same versions or different versions of your application sitting along next to each other and being able to route traffic. But it also seems to me like it introduces a little bit of complexity around version matching because only certain versions are going to be compatible with certain versions of your API. You have different versions of your API talking to the -- so the simplicity of having kind of mutable deployments, so to speak, is that everything is in sync and you don't have to worry about those version mismatches. Is that a problem or this could just be me worrying about nothing? But that's kind of the thing that just immediately jumps out to me is like are there any strategies to manage that complexity? LUKE: To me, what you're describing, I kind of think of as a feature not a bug. And what I mean by that is that it is very simple to have a mental model of, "Oh, I have a version of my JavaScript code that works with this version of my API." And as long as I kind of deploy those changes together, I'm good to go. The reality is that that's impossible. The JavaScript apps that we write today, people are using anywhere from two seconds at a time to two days at a time. It's not uncommon these days to have some of these dashboard apps. People literally live-in for their job eight hours a day, nine hours a day, keep the browser tab open and come in the next morning and continue. And so, obviously there are some mechanisms we could use to force them to reload that kind of thing. But at some point in most apps, you're going to have a slightly older version of your JavaScript app talking to a slightly newer version of your API for either the span of a minute or perhaps longer depending on their strategies. So to me, the process of thinking about that and at least being aware of that as an engineer thinking about how your code is going to get from your laptop into the world, I think it's an important step that we not paper over that complexity and that we kind of embrace it and say, "Hey, this is part of life." And so, we need to think about just like we need to think about how your database migrations get into production. That's not something that you can paper over and just have a process that it's going to take care of for you. It requires thought. And I think that this, in the same token, how different versions of your JavaScript app are going to interact with your API requires thought. An exact parallel also had different versions of a native mobile app that go into the app store. How did those interact with different versions of your API? So, I think you're right. There's complexity there. There's ways that we can try to mitigate... CHARLES: Keep repeating ourselves if we think that that [inaudible] actually doesn't exist even in the simple case? MATTIA: Yeah. I think that that's to reiterate what Luke is saying. That's exactly the point. You can pretend it's not there but it is and you have no way to avoid it. Once you ship something to a browser, you have no control over it anymore. And so, you have to assume that somebody is going to be using it. LUKE: Aaron, I think you too, I don't know if you can share it. But you recently told us some stories about kind of what you encountered in your work about this and of how long people were using versions and stuff. AARON: Yeah. Something that we hadn't sort of put a lot of thought into. But the last place I worked at, we had quite a long lived app and we're using feature flags and we're using launch [inaudible] something and it gives you a list of flags and when they were last requested. And there were also flags that we removed from the code and it was just a matter of waiting until all the users had the most current version of the app and weren't requesting the flag anymore. But this one flag just kept getting requested for months and we just could not work out why. It really sort of opened my eyes up to this exact problem that these long lived apps set in the browser and if you have someone that just doesn't reload the browser or restart the machine or anything, your app can live a lot longer than maybe you actually realize it is. So we're shipping bug fixes, we're shipping new features, and we're all patting ourselves in the back. We fixed this bug but have we really? If your users haven't reloaded the app and gotten the latest version, then you haven't actually fixed the bug for some number of people. And it's really hard to tell them as you think about this and put things in place, really hard to tell what versions are out in the world, how many people are using this buggy version still. CHARLES: Yeah, that's an excellent point. I haven't even thought about that. I mean, what is the countermeasure? AARON: We hadn't made it until we came across [crosstalk]. CHARLES: It's nothing quite like getting smacked in the face of the problem to make you aware of it. AARON: That's right. CHARLES: So, what's the strategy to deal with that? AARON: I guess for me, my learnings from that would be from very early on thinking about how we're going to encourage people to reload, to start with, and maybe even have the ability to force a reload and what that means but then that has gotchas as well. You don't want to just reload something when a user is in the middle of writing a big essay or something like that. But definitely thinking about it from the start is one of the things you've got to think about from the start. But I guess something that I'd like to implement and I've kind of thought through but not really explored yet, but the ability to see what versions are out there in the world and there are things I've been thinking about in terms of this little server that serves different versions. Maybe we can start having that kind of tracking what versions are out there and who's using what and being able to see because it would be great to be out to see a live chart or a dashboard or something that sort of shows what versions are out there, which ones we need to be aware of that are still there and even what users are using and what versions they can maybe even move them on, if we need to. But there's definitely a bunch of things that aren't immediately obvious. And I don't know how many people actually think about this early on, but it's critical to actually think about it early on. MATTIA: Yeah. I was going to share what we do which is very similar to what Aaron said just maybe for the listeners to have some context. The first thing you can do is basically what Gmail does, which is every time a web app sends a request, an [inaudible], it will send the version of the app with the request and the backend can check. And the backend can check if you are sending a request from the same version that is the most recent deployed version. And if it's not, this sends back a header and the same way that Gmail does, it will display a pop up that is like, "Hey, we have a new version if you want reload." And on top of that [inaudible] is that we have a dead man's switch. So if we accidentally deploy a broken version, a part of this process, the frontend application tracks in the headers. And if a special header is sent, it force reloads, which is not nice for the user but it's better and sometimes is critical to do so. CHARLES: Right. I remember that was that case where Gmail released something. They were doing something with broken service workers and the app got completely and totally borked. I remember that my Twitter blew up, I don't know, about a year ago I think, and one of the problems I don't think they had was they did not have that capability. MATTIA: I mean, you learned this the hard way sadly. But I think these two things are definitely crucial. And the third one, [inaudible]. I thought, Luke, you had the ability that Aaron was talking about like tracking versions in the world. And I think that's more useful for stats so that you know how often your users update. And then you can make the design decisions based on that and based on how much you want to support in the past. LUKE: Yeah. We haven't implemented that but it reminds me whenever there's a new iOS version, we do a bunch of mobile work in the app and we're always looking at that adoption curve that's published. A few different analytic services publish it and say, "OK. How fast is iOS 12 adoption? How fast are people leaving behind the old versions?" And that helps to inform how much time you're spending doing bug fixes on old version versus just telling people, "Hey, this is fixed in the new OS. Go get it." But if you are able to see that for your own JavaScript apps, I think that would be pretty hot. CHARLES: Yeah. Crazy thought here but it almost makes me wonder if there's something to learn from the Erlang community because this is kind of a similar problem. They solved 20 years ago where you have these very, very long running processes. Some of them there's some telephone servers in Sweden. They've been running for over a decade without the process ever coming down. And yet they're even upgrading the version of Erlang that the VM is running. And they have the capability to even upgrade a function like a recursive function as it's running. And there's just a lot of -- I don't know what the specific lessons are but I wonder if that's an area for study because if there's any community that has locked in on hot upgrade, I feel like it's that one. LUKE: That's a terrific analogy. I bet we could learn a ton. Just hearing that kind of makes me think about how kind of coursed our mental model is about updates to our JavaScript apps. We talked to Aaron, we're talking about kind of this idea of a mutable apps and you have different versions side by side. But the idea of being able to kind of hot upgrade a version with running code in a browser, now that's an ambitious idea. That's something I'd say, "Wow! That kind of thing would be a game changer." CHARLES: Yeah. AARON: Makes me feel like we got a whole bunch of work to do. CHARLES: We welcome them. I'm always happy to give people plenty more work to do. No, but how they manage even being able to do migrations on the memory that's running. I don't know if it's something that's going to be achievable but it sounds kind of like that's the direction that we're heading. LUKE: It does, as these apps get more complex and they continue to live longer. The idea, the work arounds that Mattia mentioned about kind of showing a message and having a dead man's switch, these are all certainly useful. And today, I would say even like the best practice but they were not what you would want to do if you could magically design any system. If you're taking a magical approach, the app will just be upgraded seamlessly as a user was using it. And they would be none the wiser, the bugs would be fixed. End of story. There's no interruption to their workflow. At least for me personally, I don't really think about that as a possibility but I love the Erlang story and analogy and to say maybe that is a possibility, what would it take? I would obviously take a collaboration across your JavaScript framework, perhaps even JavaScript language features and browser runtime features as well as your backend and deployment mechanism. But I think it's a great avenue for some creative thinking. CHARLES: I'm curious because when we're talking about this, I'm imagining the perfect evergreen app but there's also feels like there's maybe even a tension that arises because one of the core principles of good UI is you don't yank the rug out from underneath the user. They need to, at some point and we've all been there when the application does something of its own agency, that feels bad. It feels like, "Nope, this is my workspace. I need to be in control of it." The only way that something should move from one place to the other without me being involved is if it's part of some repeatable process that I kicked off. But obviously, things like upgrading the color of a button or fixing a layout bug, those are things that I'm just going to want to have happen automatically. I'm not going to worry about it. But there is this kind of a gradation of features and at what point do you say, "You know what? Upgrading needs to be something that the user explicitly requires," versus, "This is something that we're just going to push. We're going to make that decision for them." LUKE: Yeah, it's a great question. One of the things that I'm curious what you all think is when you think about the mental model that our users have of working in a browser app, do you think that there is a mental model of, "Oh, when I refresh, I might get a new version." Do people even think about that? Or are they just like your example about a button color changing as kind of a minor thing. I don't even know if I could endure stack. We've all been I think in situations where you do a minor redesign and all of a sudden, all hell breaks loose and users are in revolt. Take the slack icon. So, I think it's a fascinating question. CHARLES: I don't know. What's the answer? Do you always ask for an upgrade just observing? I don't have any data other than observing people around me who use web applications who don't understand how they actually work under the covers. I don't think it's the expectation that this code, this application is living and changing underneath their feet. I think the general perception is that the analogy to the desktop application where you've got the bundled binary and that's the one you're running is that's the perception. AARON: I'd say the difference there is that, and with all these new ways of deploying, we're shipping small things faster in multiple, multiple times a day or even an hour. So it's not the sort of thing you really want time to use. There has been an update, need to upgrade as well. And that's the difference between the desktop mentality. And if that's the mentality they have, it sits quite a bit of a shift, I guess. MATTIA: It makes me think that one of the tools that the users -- if you take a look at the general public, there's probably one tool that everybody can relate to which is Facebook. So, I think if there is a way to say what do people generally expect. There is a business user which I think we are often most familiar with but the general public, probably what they're most familiar with is what happens in Facebook. And I don't use Facebook almost. I haven't used it for a couple of years but I wonder how much of what people experience in Facebook actually impacts the expectations around how applications should behave. LUKE: I think that's a really good question. I do think your underlying point of you have to know your own users I think is an important one also. Obviously, some folks are going to be more technical than others or some audiences will be more technical than others. But I would even question, Charles, your suggestion that people think of it kind of as like a binary that it stays the same until you refresh. I think people have an idea that web apps improve over time or sometimes they get bugs but hopefully that they improve and change over time, and that there is a tradeoff there that means sometimes there's something new to learn but at the same time, you get new features. But I don't know that people necessarily associate that with and it happens when I hit reload or it only happens when I open a new browser, like I don't know that it's that clear for people in their head. CHARLES: Right. I can see that. But the question is if the evolution is too stark, I think people tend to get annoyed. If they're in the middle of a workflow or in the middle of a use case and something changes, then it gives it a feeling of instability and non-determinism which I think can be unsettling. LUKE: Definitely. We all value, as engineers, we value getting into that flow state so much of like, "Oh man, I'm being productive. I don't have any distractions." And you kind of owe that to your users also to be able to let them get into that state with your app and not be throwing up, "Hey, there's new stuff. Reload." "I'm in the middle of something. Sorry." CHARLES: Yeah. I definitely do the same thing. Sometimes, I let iOS be bugging me to upgrade for a month until I finally start to feel guilty about security and actually do the upgrade. LUKE: Right. CHARLES: Although once they started doing it at night, it actually made it a lot better. LUKE: That's an interesting idea, too. I think there's a natural tension between the lower integration risk that we have as the engineering teams of shipping very frequently. Aaron mentioned shipping a dozen times a day. We certainly have been there and done that as well. I would say on average, we ship a few times a day. But the reason that we do that is because we know the faster we get code into production, the faster we can trust it. So it passes all your tests, it passes your [inaudible], but you don't really know if you're being honest. You don't really know until there's thousands of people using it in production. And yet, this conversation makes me think about there is a tension between how frequently you do that versus your users' kind of comfort level and expectations. CHARLES: And maybe there is a thing where you can kind of analyze on a per user basis how often they're active in the application and try to push updates on times that are customized to them. LUKE: Like when a user has been idle for 30 minutes or something like that. CHARLES: Yep. Or even like track trends over months and see when they're most likely to be idle and schedule it for them. LUKE: Good point. CHARLES: Something like that. TARAS: I have an idea. We should introduce screen savers into web apps. And so when the user stops using the app, just turn on screen saver and do the upgrade. LUKE: I can see the VC patch. It's after dark but for the web. CHARLES: Enter install flying toaster. AARON: It does that to open up the idea as well of that automated checking that things are okay well after the fact because it's all right to sit there maybe activate something and sit there even for an hour and make sure there's no bug request coming in. But if no one's actually received your app, then of course it's not going to come in. And it's very easy to kind of move onto the next thing and forget about that. I guess it's not something that I've ever put a lot of time in and I haven't worked anywhere that's had really great automated checking to see is everything still okay. And I guess it's an interesting thing to start thinking about as well an important thing. CHARLES: Like actually bundling in diagnostics in with your application to get really fine grained information about kind of status and availability inside the actual app. AARON: I'm not exactly sure, really. I guess I maybe wasn't thinking about inside the app but I don't know what it looks like exactly. But there is that element of shipping fast and getting stuff out there. But are we really making sure it all works later on when everyone's actually using it. LUKE: Yeah. This by the way, I just wanted to say when we were talking earlier what are the essential qualities around deploying an app. And this reminds me that one thing that we didn't mention but is very simple is your app should have a version. And it should be unique and traceable back to what was the GitHub, git commit that was the origin of that code. It's just a very simple idea but if you're going to be analyzing errors in production when you have multiple different versions of your JavaScript app running, you're going to need to know what version caused this error and then how do I trace it back and make sure that the code that I'm trying to debug is actually the code that was running when this error happened. CHARLES: Do you only use just [inaudible] or do you assign like a build number or using SemVer? What's a good strategy? LUKE: In our case, we use git tags. And so, our CI deployment process for our Ember apps basically looks like this is we work on a PR, we'll merge it to master. If it builds, the master gets deployed into our QA environment automatically using ember-cli-deploy from our CI server. And then once we're happy with how things are on QA, we do git tag or actually I use an ember addon called ember release that does that tagging for me and I'll tag it either in patch minor or major, roughly [inaudible] although it doesn't matter that much in the case of apps. When there is a new tag that builds green on CI, that gets deployed automatically into production by ember-cli-deploy. And so, that's kind of a basic flow. That tagging, just to be clear, the SemVer tag is just going to be number.number.number. You can get more sophisticated than that and I think both Aaron and Mattia have a system where even in the PR stage, there's automatic deployments happening. So maybe one of you want to mention that. AARON: We're slightly different. Every time we push the pull request, that gets deployed to production. We're able to preview up pull requests in production before we even merge into master which we find super useful to send out links to stakeholders and maybe people that have raised bugs just to get them to verify things are fixed. And then at the point that it's all Google merged, the pull request to master which will automatically do another deploy which is the thing we'll ultimately activate, we activate it manually after the fact. We just do a little bit of sanity checking but we could automatically activate that on merge to master as well. But yeah, the being able to preview a pull request in production is super powerful for us. CHARLES: That is definitely a nice capability. It's hard. It's one of those certain workflows or patterns or tools that you remember life before them and then after them and it's very hard to go back to life before. I would definitely say kind of the whole concept of preview applications is one of those. AARON: Absolutely. It's a daunting concept if you're not there, previewing something that's essentially a work in progress and production. And there are some things you want to be careful with, obviously. But for the most part, it's a super valuable thing. As you say, it's a world where once you're there, it's very hard to step back and not be there again. CHARLES: So I had a question about, we talked about I think it was 162 plugins around the ember-cli-deploy community. What for you all has been the most surprising and delightful plugin to arrive that you never imagined? MATTIA: That is a good one. I'm pulling up the list. What I can tell you, for me, it's not about a specific plugin. The surprising part was the sheer amount of different strategies that people use for the shipping part. At least, I found that the build part is similar for most people, like most people want to do the things that you're supposed to do. So, you want to build your application and then you want a minify it in some way. And there's a bunch of options there from gzip to more recent technologies. But the way people deliver it to servers and the difference in the solutions, that I think for me has been the biggest thing, where people that ship [inaudible] are people that ship directly to Fastly, people that use FTP files, people that use old FTP, people that use our sync, people that do it over SSH. We have people that ship stuff directly to a database because some databases actually have great support for large files. So we even use it as a storage. We have people that do it in MySequel, people that do it in [inaudible]. CHARLES: It's actually storing the build artifacts inside of a database? MATTIA: Yes, I've seen them in that. It's kind of interesting like the solutions that people ended up using. And so for me, I think that that's been the most fascinating part. Because as we were saying at the beginning, I'm just seeing now, we even have one for ZooKeeper. I don't have an idea what this does but it's probably related to some sort of registration around the seven-day index. That, to me, I think has been the biggest surprise. Everybody ends up working in a different environment. And so, that flexibility that users need has been by far the most surprising one. AARON: I think that's also been one of the challenging things, one of the enlightening things for me. I think in the Ember ecosystem, addons and even just Ember itself, it's all about convention of configuration and doing a lot of the stuff that you do for you. I think people expect them to see a lot of point to do the same thing. But the key thing here is it just really automates all the things you would do manually and you need to understand exactly what you want your deployment strategy to be before use them to see a lot of [inaudible] could do for you. You need to decide do I want to install my assets on a CDN and do I want to install my index in Redis or in console or in S3 bucket. You need to know all these things and have decided on all these things and then ember-cli-deploy will make that really easy for you. And this is one of the educational things, I think we still haven't even nailed because there are always people that want to know why this doesn't work. But deployments are complex thing and as what you were saying Mattia, there are so many different variables and variations on doing this that there's no sensible configuration ember-cli-deploy could really provide out of the box, I guess. And so, that's why we ended up with a pipeline that gives you the tools to be flexible enough to support your strategy. LUKE: I think the closest that we come to the convention is that if any app is using ember-cli-deploy, you can run ember deployment targets or ember deploy prod and ember deploy QA and you can expect that that's going to work. What you don't know is how has it happened to be configured in this project. Charles, your question about kind of the most surprising thing that's come out of the ecosystem. For me, I would say -- Mattia mentioned plugin packs earlier which are groups of plugins that kind work together well. And so, we've seen some plugin packs like you might expect, like an AWS pack for deploying to AWS. But the more interesting ones to me that we've seen a lot of, companies open source their plugin packs. So what you naturally fall into as a company that's adopting ember-cli-deploy that has multiple ember apps is that you are going to develop your own plugin pack for internal use because generally speaking, companies follow the same deployment pattern for each of their apps. There's usually not any reason to vary that. So then the new thing that happen on top of that is people said, "Why don't I make this open source so other people can kind of see how we do it?" And that's been a really delightful part of the process to kind of get a peek into how other organizations are orchestrating their deployments. And if people are curious about kind of looking at that themselves, you can go on NPM and look for keyword ember-cli-deploy-plugin-pack and pull up all of those. And you can kind of poke around and see what different companies have open source there. CHARLES: I actually love all three of those answers because it really is for me when you have a constellation of people around a particular problem, it's the surprising solutions that emerge that are some of the most exciting that would have lain hidden otherwise. It would have been kind of buried beneath the source of company A or company B or company C but actually having it all out in the open so that you can inspect it and say, "Wow, where has this solution been all my life?" Something that you have never imagined yourself. LUKE: It's so funny that you mentioned that because that actually is the origin story way back, I'm talking like 2013 probably when we were very early Ember adopters and we were trying to figure out how do we deploy this thing. We're deploying it with our Rails app, like literally deploying the Ruby code and the JavaScript code together which took forever which is a disaster. And I heard through the grapevine, just exactly what you're saying Charles, where the good ideas are kind of hidden inside of a company. I heard through the grapevine that Square had this approach that they were using where they would deploy their JavaScript assets and then deploy their index HTML file, the contents of that file, into a database, it's Redis in their case, and serve then it out of there. And it empowered all of these interesting situations of like having multiple versions, being able to preview the release, et cetera. And so we then set out to copy that idea because there was nothing open source. So we had to create it ourselves which we did in Ruby which we made it inaccessible to many JavaScript shops in the first place. Then the evolution of that kind of over time and of Mattia and Aaron and Mike and the rest of the community kind of talking together has now moved this into the open source sphere where these ideas are more accessible and we've created an ecosystem encouraging these ideas to stay out in the open. It's so true that there's just gems of ideas that have been created by really brilliant engineers inside of companies that could be benefiting so many people, they just haven't seen the light of day yet. CHARLES: Yeah, that leads me to my next question. I would say most of the ideas that we've been talking about today really, except for the build part, how Ember specific are they? Obviously, the ember-cli is a great resource and has a lot of great opinions for actually building the JavaScript assets themselves. But the second two phases of the pipeline really can vary freely, if I understand correctly. And so, have you all ever thought about trying to maybe kind of abstract these processes and these plugins so that these same ideas don't remain not just inside of a company but inside a community that spans a set of companies so that it's available for a wider audience? How integrated is it with Ember and what kind of effort would that be? LUKE: That's a great question. Aaron or Mattia, one of you guys wanna talk a little bit about the history here? MATTIA: Yeah, sure. We've been thinking about this as well. In the past, a guy on the ember-cli-deploy team, Pepin, has actually started this effort and it kind of prototyped this very idea of separating the ember-cli-deploy part from Ember CLI itself and make it a bit more generic. And he started a project called Deploy JS which I can give you a link for the show notes later. I don't think that the project is currently maintained but definitely the start of the effort is there. And the funny thing is that it was surprisingly easy. I think that we didn't get there mostly because we just all use Ember at work. As you know, open source is mostly motivated by the needs that an individual or a group of people have. But if any of the listeners were very interested in this, I think they should definitely get in touch and we will be happy to talk to them and see what can be done here. AARON: And also if you look, there's actually a plugin called ember-cli-deploy-create-react-app and there's also ember-cli-build-view. So it can and is being used to deploy non-Ember apps which I think is super interesting because the only real Ember part of it is just using the CLI to discover the addons and plugins. And from then on, it's really out of the hands of Ember. But it sort of leads into a little bit, Luke mentioned this concept of immutable web apps. And I've been thinking a lot about this lately because a deployment strategy that ember-cli-deploy use as an example a lot and it's kind of become [inaudible] ember-cli-deploy in ways, the lightning approach which is this whole idea of splitting or putting your assets in CDN and your index HTML separately maybe in Redis and serving that. I've been trying to work out how I can talk about this to the wider JavaScript community in a non-Ember way and knowing full well that the concept of lightning deployment means nothing to anyone outside of Ember. Just by chance, I was just talking to some people and this terminology of immutable deployments kind of rose. I started searching around and I come across a website called ImmutableWebApps.org which was just scarily the same as what we've been talking about for the last three or four years with ember-cli-deploy. And a way to boil down at a framework-agnostic level, what are the key points that you need to consider when building a JavaScript web app to make it immutable. And it was just really amazing seeing it. This website was put up 3 weeks before I did my Google search coincidentally. And it's basically word for word what we have been talking about the last three or four years, So, someone else in the other side of the world's been coming up with the same ideas in their company like you were talking about earlier and we've reached out to him. I guess that, to me, is sort of the way forward that I want to sort of pursue to try and get these ideas out in a framework-agnostic way to the rest of community and say, "Hey, we thought about deployment in this way. Have you thought about building your app in this way to give you these sorts of capabilities?" CHARLES: I think the wider community could definitely benefit from that because most of the blog posts and talks I've seen that concern themselves with deployment of single page applications, it's still much more of the tutorial phase. Like, "This is how I achieve getting this deployment strategy." Not, "This is how I repeatedly encode it as a program," and leverage it that way. And so, definitely getting that message out to a wider audience, I think it's a -- what's the word? It's an underserved market. LUKE: Yeah. I really like this idea also. I think about this ImmutableWebApps.org, if you look at it. It's sort of a manifesto with conceptual description of what are the qualities that your app has to have to qualify as an immutable web app which I think is kind of a funny idea but one that people can start to get their heads around and compare that description to their own apps and say how do I hit this or fall short. And it reminds me a lot of kind of the idea of Twelve-Factor App which is an idea that I think came out of Heroku originally. And it's an idea of a backend app that is portable to be able to easily move across different hosts and easily be scalable to different instances of [inaudible] if your app obeys all of these things then it's going to do well under those circumstances, that will satisfy those needs. So, I think it's a great way of thinking and probably maybe even a better entree into the conversation with the wider community than a library, this is certainly a library called ember-cli-deploy. CHARLES: This is a fantastic discussion. It's definitely reminded me of some of the best practices that I haven't thought about in a long time and definitely opened my eyes to some new ones and some new developments. So often, we can be focused on how our apps work internally, like how the JavaScript code works that we can just kind of -- what's the saying -- lose sight of the forest through the trees or I can't remember. It's like too busy looking down the end of your nose to see past your face. I've probably mangled both of those adages, but maybe 60% of two mangled adages is at least equal to one. This is something that we need to be thinking about more, that everybody needs to be thinking about more. This is actually a very exciting, very useful problem space and I'm just really grateful that you guys came on to talk to us about it. So, thank you, Mattia. Thank you, Luke. Thank you, Aaron. MATTIA: Thank you so much. It was a great time. AARON: It was a pleasure. Thanks for having us. LUKE: Thanks so much. CHARLES: Thank you for listening. If you or someone you know has something to say about building user interfaces that simply must be heard, please get in touch with us. We can be found on Twitter at @TheFrontside or over just plain old email at contact@frontside.io. Thanks and see you next time.

Joe joins the panelists to talk about pull request etiquette. Joe Leblanc: Joe first learned to code on a Zenith computer his dad brought home from work. It had this built in blue LCD monitor and ran on 5 1/4" floppy disks. He used spreadsheets for work and Joe was interested. They spent about an hour going over macros together and he took off from there. Long after the Zenith died, the open-source content management system Joomla! landed in the center of his attention. Joe found himself writing a book about Joomla programming, authoring video tutorials about Joomla for lynda.com, giving Joomla talks, and helping organize Joomla conferences. Since his time in the Joomla community, he's picked up Node, Rails, React, and other frameworks. He's currently coding at True Link Financial and working on a few hobby-projects as well. Please join us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at contact@frontside.io. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer. This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. Transcript: CHARLES: Hello and welcome to The Frontside Podcast, the place where we talk about user interfaces and everything that you need to know to build them right. My name is Charles Lowell, a developer here at The Frontside. And with me also is Taras. Hello, Taras. TARAS: Hello, hello. CHARLES: Today, we're going to be continuing our conversation about platforms, as always, but in particular the pillar of your platform that has to do with how you collaborate on code. It's an important one. And so, we're spending some time on it. And with us to talk about this today is Joe LeBlanc who is a senior software engineer at True Link Financial in our very own beloved Austin, Texas. Hey, Joe. JOE: Hi. CHARLES: Thanks for coming on the show. We're going to be talking about collaboration and I was thinking we could kick off the discussion talking about pull requests because that's typically one of the ways that we collaborate on code. That's really where the rubber tends to hit the road. You have particular interest in the dynamics of a pull request. What experience kind of led you to that interest? JOE: My background has been doing a lot of work as a freelancer. And when you work as a freelancer or do agency work, a lot of times you are really the only software developer that is around or you're working with maybe one or two other people. And I didn't get a lot of opportunity for pull request reviews. Mine was in that space. And then when I moved to a full time job at a place where I was one among maybe half a dozen or a dozen engineers, that's when I really began to get interested in how I could be better at giving pull request reviews and also submitting pull requests that people want to review. CHARLES: What made you notice the need for this? JOE: What would happen is someone would submit a pull request and there would suddenly be just dozens and dozens of comments coming in that were just kind of difficult to keep track of and often were maybe talking about things that didn't necessarily need to be reviewed as a part of that pull request or addressed, maybe things that could be caught by a linter. CHARLES: Right. JOE: And other tools that are a little bit easier to receive feedback from. Like it's easier to have a tool tell you your spacing is off than have me tell that to you. CHARLES: Right. And really that seems like the spacing is off, that's kind of like if you need to deliver that feedback, that's kind of like not what you want to lead with. If you don't have linting in place like after all other issues are sorted out. JOE: Yeah. CHARLES: But it sounds like what you're describing is people who have kind of swarmed over a pull request each with their own kind of pet peeve issue. JOE: Yeah. And then you're just left with this long list of comments to go back and address and you're pushing up more and more commits. And by the end of it, you could have more than 100 comments on this PR. And you thought that you were going to get this done in a day or two, and then suddenly it's the end of the sprint and it's like, "Oh fine, then I get to merge this." CHARLES: And typically, it's actually, in my opinion, an anti-pattern when you have like 10 mergers that happen on the second last day of the sprint or the last day of the sprint. There's always issues with that, right? Ideally, you are merging code throughout the course of the sprint. It kind of like defeats the purpose almost. I guess you're doing your integration in shorter periods but even so, tons of stuff is bound to break when everybody's pushing and everyone's rushing. JOE: Yup. TARAS: So, what would happen in that situation, in your experience, when you have this pull request that a lot of people are commenting on and some of the comments could be addressed by linters. Would you guys go back and actually implement linting? Did that happen? Would you put linting in place so that you'd have consistent formatting on the projects to reduce that kind of feedback? What would be the resolution or a way to improve on a process so you could actually get better feedback? JOE: Yes, we did install a linter and that's something that you can either run locally or you can also run in your continuous integration environment. And it's really good to run it locally first because then you can just catch it before you even submit the PR. And that was something that definitely helped cut down on these sorts of comments and even debates over style. You just have this one thing that is maintaining that style or rather telling you when you're wrong for you, and then you don't have to have all these comments coming in as you submit your PR. And another thing that we would do is we would use, even beyond just things like whitespace and trailing newlines and those sorts of issues, sometimes you would have some of these issues that would come in that wouldn't necessarily be a trailing newline or whitespace issue or something like that but it would be purely someone's sense of style. And in those situations, we came up with a bit of a standard where if you wanted to make a comment that was strictly style that wasn't something that you necessarily wanted to block the pull request over, we would use emojis that had specific [inaudible] thing. So in our case, we would use either a beer emoji or the cocktail emoji to kind of say, "Here, take this with a drink." CHARLES: [Laughs] JOE: This is not something that's super serious. It's a suggestion, "Here, let's discuss this over beers." [Laughter] TARAS: It's a really nice approach to friendlify the actual pull request because people can get really -- I think another way people would say this is like when you say that it's not 'I believe it whole strongly'. How does that go, Charles? CHARLES: It's strong opinions, weakly held. JOE: Yes. CHARLES: Right. You're kind of making an assertion about the way things ought to be or often missing from that is how important it is to you. You've made this thing like it should be like this. That's a very black and white statement, very firmly cut between right and wrong. But how important it is to you if it goes the other way is often missing from the conversation. TARAS: So Joe, what would happen once you made this change? Did you see a shift in the kind of feedback you were getting? What did it look like after this? JOE: After we started implementing the ideas of putting on emojis to signify things that were not super serious, it became a lot easier to just picture all those comments and define the things that really needed to address. And it was a lot faster. I was just able to address these very specific problems and do that first. And then if I had time later, come back and address those style issues. CHARLES: Right. Was the kind of the good feedback there the whole time but it was just obscured by the noise of these kind of exterior issues or side issues? Or did you find that because people's thoughts were more given over to the heart of the matter, that you actually got more comments directly relating to kind of the meat of the implementation? JOE: I think it was really more that the good comments really were buried in there. A lot of times when you see critical comments that seem to be petty, that can really trigger your emotions. And so, I feel like a lot of the times the emotional state that we would get into would prevent us from seeing the good comments that people were leaving, and we would get a little too angry about things that really weren't that big of a deal. CHARLES: Would the tension kind of noticeably rise? JOE: Yeah. I actually remember several instances at my job where -- this is when I was working in office and in the same room as all my co-workers. And one of my co-workers would know that I was reading his review by the number of sighs that were coming out of me. [Laughter] JOE: Yes, I definitely had this reaction that people could tell. Implementing things like these emojis definitely helped. They didn't always fix everything, though. One thing that would happen too is even after filtering out kind of the noise comments and getting into the meat of the matter, we would still wind up in situations where I would want to use one design pattern and another engineer would come along and say, "No, you should do something different," and we would have these arguments. CHARLES: Right. And those are harder to resolve too because those are strong opinions strongly held. JOE: Yes. CHARLES: But it's still a step forward, right? You've eliminated the passionate arguments about the strong opinions weakly held. And so you're able to now grapple with 'how do you resolve these arguments' of these stronger opinions that are more strongly held. JOE: Yeah. Part of the way that we handled this was we came up with a way of being able to have a debate about something on a pull request, yet also keep in mind that we're working under deadlines and that ultimately, it might not be that big of a deal. TARAS: I don't know if you experienced this but there's this kind of a feeling where you know you have a deadline that you want to reach and this feedback is kind of not really helpful to that end goal. But at the same time, there feels like a trade off for you, like trading off quality by accepting something that people don't agree being fully ready. JOE: Yeah. TARAS: But then there is, I guess the tests seem to be like you have some tests and the tests are passing. So, thumbs up. [Laughs] CHARLES: Yeah. Well, here's the thing. This thought just occurred to me when I heard that is when we are reviewing code, probably the most important code to review is actually not the implementation but it's the tests. Because why do people get so care mad about the code being right? It's because they know how expensive it is to get it wrong. And we all have our different opinions of what's right and wrong and what's going to cost us more. But at the end of the day, we're all trying to save us, save ourselves and our team, pain in the future. The tool that we have to do that is the code review. We're trying to make sure the code is "good". But usually what ends up happening, and I know I'm certainly guilty of this, is I'm focusing on the implementation and I look in and say, "Oh yes, there are tests." I don't really look at the test and say, "Is this a good test?" Because a good test is going to lower that cost that we're so afraid of in the future, and that drives us to want to make sure the code is as close to what we conceive of as perfect as it's likely to get and still hit our deadline. It sounds to me like maybe what we really ought to be doing is reviewing the quality of the test because if you have an excellent test, then the shape of the code almost is unimportant or changing the shape of the code, more importantly, is very cheap. So if you do find that the implementation is suboptimal, the cost of rejiggering it into a better configuration is going to be very low. JOE: I've discovered so many times where you are running a test that was written previously either by somebody else or by yourself, and you realize that the test is testing the wrong thing entirely. CHARLES: Yeah, coverage is spotty. It covers 10% of your cases. JOE: It can also might have a stub or a mock in there somewhere that is hiding something that you weren't counting on. Or it could just be you pull up this screen and test then it saves but you're not really testing what it's saving. CHARLES: Right. Can I actually proffer a suggestion? At least I'm going to try and hold myself to going forward, is that when I review a pull request or some proposed change to a code base, I'm going to review the test first. JOE: Ooh, that's a great idea. CHARLES: I'm going to try and start with the test and avert my eyes from the implementation, no matter how curious I am how this person accomplished the solution. I'm going to hide from the solution and focus on the verification. And if I have faith, first and foremost, in the verification, then my faith in the implementation is secondary. TARAS: You know, Charles, I really like this. I think it'd be really interesting to try this out and see what kind of impact it creates. It makes me think that there is actually something really useful in here in terms of providing feedback because the challenge is that when you're giving feedback to someone on a pull request, ideally that feedback is going to be useful and it's going to give the person something to think about in regards to the work that they did. And I think a good way of pushing to work back onto the person in a way that allows them space to internalize what they need to do is identifying cases that might be missing because a lot of times, we test happy paths. We don't test the things that we didn't think about. And that's where the devil is in details. It's in the things that we don't test. Those are the kind of problems that we're trying to prevent with having good code. But really the best way to kind of flush those things out is to make sure we have the right test coverage for it. So yeah, I think if you add to that just when you're looking at a pull request is to go, is a test coverage missing the following cases and actually surfacing not. I'm actually thinking like a danger task. A danger is a tool for adding information to pull requests so that it's kind of like an automated mechanism to comment on pull requests. And so I'm thinking like having something that does what [JAS] does. [JAS] has this thing where they will check based on the last git commit. It will figure out what are the tests that have been added since the last commit and then it will actually show you. So when you run tests, it only runs the tests that you kind of impacted which is something that could be used to surface tests that were written in this commit. I mean, we also have that information in the actual commit itself but being able to see 'these are the tests that were written or added' and then you could use that and figure out what else is missing from this. This could be like a way to kind of add onto this process, something a little bit more automated so you could actually highlight this information very easily in the commit itself. CHARLES: It's funny, my mind was wandering off towards GitHub. When do you use danger and when do you use GitHub actions? GitHub actions have been kind of like brewing and fermenting in my mind. TARAS: One nice thing is that I mean, danger is a little awkward in that it creates one single commit at a topic. It creates a comment first time you commit and then it keeps pushing information into a comment which shows up at the top of your comment thread. CHARLES: Right. TARAS: But that's not how you read comments. You read like top to bottom. You don't read top to bottom to top. And so, I think the nice thing about the actions is that because it's using checks like GitHub checks that actually is part of a different area, it's part of the validation area as opposed to part of the actual comment conversation. This seems like a more appropriate place to put their information rather than the first comment. CHARLES: Yeah. So, do we decide on kind of what the ultimate resolution is for when you have these high order conflicts? JOE: Yeah. So that's one thing that we came up with that I'm really proud of. And so what we do and at least at that job that I was at, what we did in those situations was we said, "OK, if you've gone back and forth on this two or three times and there's still not an agreement as to what should be done or there is no compromise, what you should do is let the author of the pull request go ahead and merge that code," because you want to merge that code, get it out to production, make sure it's serving our users and our clients. And then what you want to do is take that conflict and record it somewhere to specify this interpersonal conflict not a code conflict or a merge conflict or anything like that. So you take this interpersonal conflict, the topic that was being discussed and you put that -- we were just putting it in a Google doc and leaving it there. And then every couple of weeks or maybe every couple of sprints, we would go through that Google doc and just talk about topics that had come up. And then really more often than not, what we would find is that we didn't care about the topic anymore. CHARLES: [Laughs] It's amazing how time has a way of cooling passion over things that really don't matter. JOE: Yeah. And for the things that did matter, we usually have like a really good discussion. Sometimes, we even came up with something different entirely based on just having more people in the conversation and thinking about this problem. TARAS: I like this because there's a bit of process around it. It's kind of like a retrospective on pull request passions. JOE: Yes. TARAS: It sounds like a healthy thing to do for a team, especially. And I think just allocating some breathing room to go through these kind of things could be really important cumulatively over time, especially. CHARLES: Yeah, definitely. It sounds almost like the pull request is then ongoing. You have this collaboration that happens kind of at the front of the change but then is rippling outwards and onwards and hopefully then has an impact on future changes, like if you're disagreeing. So, would you qualify that the types of issues that end up living in this Google document as architectural issues or just, 'hey this is the way we need to talk to each other and this was off and we need to fix this' or is it both? JOE: It was mainly more architectural decisions that was coming up in this, sometimes like code style issues as well. Not so much the interpersonal issues, like the interpersonal issues would come during retros. CHARLES: OK, because it sounds to me like it's not. Then you have the added benefit that your architecture does get to improve because clearly if there's some disagreement about this, then there's some sort of tension there. There's someone perceiving that some problem is not being resolved by a particular implementation. And so it's good to just at least surface this issue again and again. What was your experience with kind of revisiting the architectural issues? Was it mostly 'well, this wasn't really an issue' or is it 'wow, I'm really glad we took note of this because now we have these three ways of thinking about things' and it turns out that given three or four months more experience, this is something that we should be doing. JOE: Yeah, it is definitely a mix of both but I'm kind of leaning more towards the latter. Like we're just glad that we bring things up. Occasionally, there will be one or two topics that will come up that we still would resolve and we would have to agree to disagree on something. Or in some cases, I think we would allow one of the lead engineers to just make the final decision on something. But that was pretty rare. CHARLES: So, apart from appealing to either time, the passage of time or just kind of taking a brain dump of all the different architectural options or deferring to a more senior engineer and just kind of asking them to step in and make the call, are there any processes that you can put in place say to kind of mechanically come up with a decision? Like any set of values that you can use as a ruler to kind of line up a set of things to kind of attach weight to one particular solution? For example, one of the things that I always think of is when I have internal conflict about a decision, there's kind of a part of me that feels like this might be the right way and maybe another way is the right way. And so, a tool that I will use is what's internally consistent with the rest of the system. So, even if I've had an insight that something really ought to be framed,one solution should be framed in a certain way. If there's another solution that's more internally consistent with the way things are existing, then I'll use that as a discriminator to say, "OK, that's one thing that I can use to measure a solution against, that I'm not emotionally tied to." It feels more objective. And if you have those tools, you can kind of pile them up and see which pile ends up being higher for each solution. Do you see what I mean? JOE: I definitely lean more towards keeping things consistent with what's already there. Personally, I probably do that to a fault. Sometimes I need to branch out a little more and get comfortable with possibly breaking something. But you always need to weigh whether it's something that is acceptable to break. Like a lot of the systems I've worked with involve money and it's really important that people either have money when they need it or are charged the correct amount when you charge them with things. CHARLES: They get really mad when you charge them too much. [Laughs] JOE: Isn't it amazing? [Laughs] CHARLES: Yeah. JOE: It could be that you are OK with breaking this other end of it where you might charge them the correct amount but the fulfillment of that product or the service or whatever you're selling, you might have a little more wiggle room in that part of the code base. Where if you break something, somebody calls in and says it's broken, and then you're just able to fix it. CHARLES: Is there anything else? If folks are struggling with the way their pull requests work or their code reviews work and they're experiencing friction and tension with their teammates, is there anything else they can do? JOE: What I would suggest is definitely look at the pull request template feature that is in GitHub and make use of that. Because what you can do with that is come up with several sections of the pull request template and say, "These are the questions that we want to ask before we submit a PR." And if you have those questions and those sections right when you go on that template and people read through that as they're making their pull request, they can often catch problems before they get to another reviewer. I can't tell you how many times I have submitted a pull request, started to write the description and [inaudible] these questions and realized, "Oh, this is wrong," or it's broken or it's totally the wrong thing. CHARLES: Right. It's funny and I feel like you want to be asking those questions not when you're submitting the pull request but before you're even writing the code. JOE: Yes. CHARLES: But a lot of times, we don't do that until afterwards and you're like, "Oh man!" Just this process of forcing myself to think about the problem in this way or think about it holistically totally recasts my implementation. JOE: Yeah, things like, "How can this break after we merge it?" [Laughter] CHARLES: Yeah, that's actually a great question, that of one you're talking on pull request. JOE: Oh yes, definitely. CHARLES: Like imagining the [inaudible] scenarios. JOE: Yep. CHARLES: Man, it's almost like you want to read the -- what are the other questions that you have? JOE: So the ones that we have are, first of all, what is this pull request? How does it fulfill the requirements for the ticket? And then how can this break after we merge it? Are there any post-merge tasks that need to be run? If you need to get on a server and run a task or do something after it has gone to production, that's a place to document that there. And then the final question is, how is this tested? CHARLES: Right. That's a good one. And like I said, I think that's the one where I'm going to be starting when I'm both thinking about how my changes will be reviewed and how I review changes. JOE: Maybe I should consider moving that to the top position. CHARLES: [Laughs] I mean, it is like. I'm just thinking about it and it does seem like we go straight to the implementation because that's what's fascinating to us. And so we might have way too much bias for the importance of the implementation over the importance of the test because I have to say I am so guilty of when I'm reviewing. Just looking for the fact that it has a test and not looking for what the test does. JOE: Yeah. CHARLES: And only referring back to the test and trying to understand how the test accomplishes its task. If I don't understand the implementation, I'm like, "Oh, let me look at the test and see how this code is used to the test." That might be problematic but I know that's definitely a little personal goal that's coming out of this podcast for me. TARAS: Excellent. CHARLES: All right. Joe, do you have anything else that's coming up? Any talks? Any meet-ups that you're going to? Any announcements? Anything that you want to plug? JOE: I have a website at JLLeBlanc.com and that's where I am. CHARLES: All right. Well, fantastic. Thank you so much for coming on. JOE: All right. Thank you. CHARLES: Thank you for listening. If you or someone you know has something to say about building user interfaces that simply must be heard, please get in touch with us. We can be found on Twitter at @TheFrontside or over just plain old email at Contact@Frontside.io. Thanks and see you next time.

Sam joins the panelists to talk about frontend and backend team collaboration. Resources: Worse is better The Tao of Microservices by Richard Rodger Sam Joseph is a CoFounder of AgileVentures, a charity that helps groups of volunteers gather online to develop open source solutions for other charities all around the world. Sam's been mucking about with computers since the early 80s and followed the traditional education system through to a PhD in Neural Nets. Next he went all industry, researching mobile agents at Toshiba in Japan, going freelance and then swung back to academia to research peer to peer system and collaborative systems. He now spends the majority of his time trying to make AgileVentures a sustainable charity enterprise, with occasional moonlighting as a contract programmer. Check out his blog at nonprofits.agileventures.org. Please join us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at contact@frontside.io. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer. This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. TRANSCRIPT: CHARLES: Welcome to The Frontside Podcast, the place where we talk about user interfaces and everything that you need to know to build it right. My name is Charles Lowell, a developer here at the Frontside. With today also is TARAS: Mankovski. TARAS:: Hello, hello. CHARLES: Hey, TARAS:. Today we're going to be continuing our theme when we think about UI platforms and web platforms, continuing the theme of collaboration and with us to talk about this is Sam Joseph. Welcome, Sam. SAM: Hi, thanks for having me. CHARLES: We've already talked a great deal about how the way in which your team collaborates and the communication that happens between your team and between the different pieces of software, your system, form one of the pillars of the platform that you can't just take lightly. You need to actually be intentful about that. I was thinking we could kind of start today's discussion, kind of talking about some of those collaborations. One that we've probably all encountered, which is usually teams will be split into people who are focused on frontend, people who are focused on backend systems, kind of the services that make sure that all of the nodes that are running on our laptops and our desktops and stuff are running smoothly and error-free and obviously, those two groups of people can sometimes arrive with different sets of priorities and how do we resolve those priorities to make sure that that communication flows freely. TARAS:: What's interesting about these frontend and the backend teams is that our users are not seeing that separation. They only see one thing. They only touch one thing. They actually see as one group but there's tends of be this kind of split between the frontend and the backend. It's kind of interesting that how the user get into this. SAM: Yeah. Obviously in some teams, there's a very clear cut distinction between people at the backend and working with the components that are serving JSON over the API and there are some people who are very, very focused on the frontend and drilling CSS and a number of bits and pieces or even just staying explicitly on the design or UX design and there's a mythical full stack developer who is up and down the platform. It doesn't run exactly in parallel but there is this key thing which is almost how much sympathy or empathy can you have for another person who is not you, trying to use something that you set up. If there was a direct parallel, you'd say, "Obviously, all people who will be working on the frontend are more of that sort of person and perhaps, the people on the backend are not so much that sort of person," but actually, I think you can have people who are doing backend stuff and they're designing API is very, very thoughtfully or the kind of people that consumes those APIs and sometimes, you can have people who are very, very focused on the design and the aesthetics when not necessarily so plugged into how will someone else use this, how will it fits into their lifestyle, which might be very different from my own, so that's maybe another axis, if you know what I mean apart from this sort of pure technical [inaudible]. Does that makes any sense? TARAS:: Yeah. What's interesting is that everyone is trying to do a great job. Everyone is setting out to do something really good. What people's way of expressing good might be different, so if someone could be really focused on the quality of their code like they want to do their version of doing a really, really good job is doing the best code they could write. Sometimes that doesn't necessarily equate to the best user experience. I think everyone that I've met -- engineers who are writing code, almost everyone that I know is trying to do a really good job. If everyone is doing a good job, it doesn't necessarily always equate to the best user experience. CHARLES: I think we had a reference that everyone wants to make sure that their system is of the highest quality possible but quality in of itself is not an absolute value. It's relative. In other words, a good definition of quality is how well a system is fit to a purpose. If it fits very well to that purpose, then we say it's of high quality but if it fits very poorly to a particular purpose, then we say, "This is a system of low quality," but what constitutes something of high quality is relative to the purpose. The question is, what is the purpose of writing the frontend system? What is the purpose of writing the backend system? So it seems to me you're going to have a lot of dissonance if the purpose is divergent but if they both share the same purpose, then that is kind of standard quality on both sides. Does that make sense? SAM: That makes sense and what even more makes it tricky is that actually, purposes can evolve over time and so the thing with the frontend that needed to do with the business today is different tomorrow and the day after. The trick then is sort of the frontend and the backend, as people try to do a great their job, there's this question of how far ahead are we looking so you can talk about important things like sort of asking close modification about [inaudible] extension and there's a [inaudible] of different coding heuristics as best practice that say, you should kind of go in this direction. Sometimes, that will be the hill they want to die on. It's like, "This code needs to be this way because of this thing," and the idea is that it's sort of future-proofing them but I think the messy reality is that sometimes, the thing that you put in place to future-proof, the whole system gets canned. In two weeks, the business changes and what have you, so the extra effort that you are pushing in on one day to protect yourself against changes in the future actually gets lost and perhaps, if only you'd be able to deliver a shortcut pack, this one feature that that will got the next line of funding in, either one advocates cutting corners but... you know what I mean? Like --? CHARLES: Yeah, absolutely. I just wanted to chime in and vehemently agree with you that the purpose can change radically, especially if you're in an evolved environment. What constitutes the good code or the good quality solution can vary just as radically because it needs to fit that purpose. TARAS:: When we talk about frontend and the backend, it seems there's this relationship, which is kind of positional like there's the frontend and then there's the backend. One is in sequence from front to back. It's the direction with the frontend. It's closer maybe to the user where the backend is not. But I think in practice, it's probably more like the left and the right hand where when you have to do two things together, you actually need to do two things to coordinate together. What you describe about and this is I think where taking shortcuts sometimes can actually be a good thing is that if you say, "I'm going to spend the next few weeks on a backend and doing this specific thing," and then you find out that 10 days into the iteration that it doesn't fit, the better approach to synchronize would be to stop the action and realign. But if you persevere, then you might overstep and you actually be out of sync. It makes me think that the art of creating cohesive organizations from development perspective is to create context where the two kind of work in synergy together. I think that's where a good tooling that allows its synergy to exist. I think that's a lot of the work of actually creating systems where the user experience is kind of cohesive and integrated. CHARLES: Actually, there was something that you just mentioned in there but it's actually a little nugget that I'm curious to explore and that is if you're 10 days in to like a 14-day piece of work and you realize that it isn't right and it doesn't fit with the whole, right action is to throw it away. I feel like software organizations and this might be a little bit off topic but it's something that really is interesting to me, so please indulge me. I feel like we see the product being the kind of the code output, even in Agile environments and there is an aversion to throwing away code that has been written. There's the, I would say, incentive is to go ahead and persevere because developer time is so expensive. These are days out of people lives, so they've already invested 10 days. Why not just have four more days just so you have it. When in fact, you actually have more if you just throw that work in the trash because it's an obstruction that's not needed. It's a piece of weight. It's actually something that you now are going to own and it's actually going to be cheaper in the long run and it can be more beneficial to your organization to not own it. Do you all see that happen kind of play out where people become attached to software that they've invested and will make the decision to hold on to it? Say, we'll spend two more days to complete it, even though it's the wrong thing, like identifying which things need to be thrown away? I feel like we don't actually do that very aggressively -- to say, "You know what? We need to not complete this work." SAM: I think that is a really tricky one because I think whatever people might say, when you spend time working on something, you become emotionally attached to it. I would say strongly that how logical or rational or whatever sort of a person you are, my experience is being that people working on things want to see them used. I think in a situation with version control where you can keep things on a branch, they can be useful explorations, things don't have to be thrown away in their entirety. Our charity is doing work for the National Health Service in the UK where there's a lot of employees in Europe and I've noticed the user interface that we're supplying them and I was suggesting some sort of minor tweaks and one of the developers has sort of run with the entire, big, advanced search feature that partially solves the problem but brings in a lot of other bits and pieces. He does some good work there. It's a great work but I think he kind of ran further than I was expecting down that line and I think we have now found a much simpler solution that rather than bringing an entire cabinet, it's a little shading off the side of the existing one. But I think that doesn't have to be a loss and I was reassuring him that the work was valuable and that there is interesting learnings there and potentially, we might use it in the future version of the project and now of course, the way that stuff is moving on, just how it's going to branch, it doesn't mean you can then sort of magically lose some of the work. In that case of doing this 10 days out of 14 and so on, the real question is can you get any value from switching somebody that fast? If they spent 10 days in and they realized they don't need that thing, it's like can you switch them quickly onto something else where they'll do four days or might they as well, just finish up there and leave that on a branch. That's a pull request that gets closed and it's there to go back to. That's kind of depends team by team about how quickly you can repurpose people missed rigs, if you know what I mean. CHARLES: I absolutely agree but the key thing is leaving it on a branch and not integrating it into production, like not actually deploying it because I feel like when we see a feature, it's like we've got to get this thing done and it's done and we're just going to get it in versus now that we've learned how to do this, let's put that on a branch and let's rewrite it and let's take a different approach, rather than just being married to the idea that we're going to get it right the first time or that now is the time for this feature because we understand the complexity of what it actually takes. It is a tricky question. I'm wondering if our processes don't include enough implicit experimentation. We talked a little bit about this on a prior podcast that if they don't include some incentive to not deploy features just because you have them. TARAS:: I think from a business perspective, there's not a lot of model to evaluate a certain thing. We don't have really any effective way of evaluating learning. You can measure code by the numbers of lines of code that somebody wrote and how much of code was shipped but you can't really evaluate how much was learned and how much was persisted. I think a lot of this work around supporting effective collaboration is in building a cumulative systems of knowledge like why is a good Git history useful is because it has a built in mechanism for understanding history. It gives you a way to return back to time in your project and understand the context of that specific change and I think this is something that really good teams do this really well. They will respect this because when it's necessary, it is valuable to have this. When you don't have it and you are a year into the project and something happened and then, you are using the only thing you have, which is your troubleshooting skills or you are trying to figure something, as supposed to going back and relying on that history, on that knowledge that's built into the system, what I'm trying to get to is there is an element that is inherent to our development process, which is we don't have a way to quantify it. We have no way to really evaluate it. I think this is the problem that makes it difficult to throw away work because you can measure the amount of time they'll spend but you can't measure the amount of learning that was acquired. CHARLES: Right. Thatís true. SAM: I think if you have a positive team environment, if you have your team that is stable in there -- your contacts are coming in, the money is there, everybody is there and you've got the team, you don't necessarily be able to measure the learning because the output of the team will be good. They're kind of doing that in the background but I think individual comments are not going to want to pay for learning experience. It's certainly not at the rate of software developer's cost. Although, the throwing-aways, if you're familiar with the [inaudible] book, which I think is a [inaudible] is the author, you know, build one, to throw away, you will anyway. We have a frontend mob that we run weekly doing frontend stuff and CSS and so on and we've done like it's a mockup for the client and there's this question about in my [inaudible] and so Iím saying to other developers there, "Let's not get too attached to this. We may need to throw it away and it will be a great learning to sort of restart on that." The [inaudible] to look quite nicely and the client might get attached to it. They want to ship it and I'm like, "Well, actually there's a lot of other stuff that needs to happen," and so, it's a minefield. It really, really is. TARAS:: I think some of the business relationships that we have to this kind of client consulting company relationship, that relationship might not always create the fit to purpose team for specific challenge. You can have a company that has a lot of employees but their team and their team dynamics might not be fit to purpose to the problem they're trying to solve. If you're building a platform over a long time, you want to be creating that space where that learning gets accumulated over time. It doesn't matter what necessarily the relationship is between the companies that are participating in this process. I think what matter is what are you producing as a result. If you're working together with people from different companies and they're working together and they're building this kind of an environment where you can collaboratively build things together and then people learning from the output and that knowledge is being carried over and people are being able to stack their knowledge continuously over time, if you're creating that environment and you're building a large platform for example, then you are creating a build to purpose kind of technology team to fit what you're looking to accomplish and that'll include consulting companies. I think those team, they're kind of secondary but I think it's just [inaudible] to that. It's like building a quality development organization to fit the purpose of building whatever it is that you're trying to build. If you're building something small, you might have a small team that'll able to build that effectively. If you have something big, you could have a big team that is building that effectively but building that team in a way that is appropriate to what you're trying to accomplish, I think that's the real challenge that company struggles to do. CHARLES: Yeah. In the context of these real teams, I guess what can you put in place given that you have backend teams, frontend teams and each one of these needs to be specialized. This is kind of the power of the way that people work is that we're allowed to specialize and there's power in specialization. You can have someone who can be super focused on making sure that you have these high throughput backend systems that are resilient and fault tolerant and all these wonderful things, so that they don't necessarily have to have the entire context of the system that they're working on inside their head at a given time and the same thing someone working on CSS or working on a frontend system architecture, which has grown to be a very complex problem domain in its own right. But these teams can be working in a context with a purpose is whip-lashing around and changing quite drastically and so, how do you then keep that in sync? Because we've identified purpose as being actually something that's quite a dynamic value. How do you keep these teams keyed in and focused and so, that they're kind of locked in on that similar purpose of they're going to be adapting the systems for work that they're responsible for and specialties that they're responsible for to match that? SAM: It's a good question and I have my own bias view there -- CHARLES: I would love to hear it. SAM: I can't bear working on things where I don't understand in great detail how that end user has experienced or what is the effect overall that it's trying to be achieved. You know, I've been working in boards between the industry and the charity for like 30 years and I know people are commenting like, "You were the person who wants to understand everything." I think there were some people who are maybe quite satisfied who get ticket off the general board or what have you and work on that thing and if the API specs have filled in, so that's it. Go home and they weren't losing a sleep over it. But I think if youíre going to be dealing with these changes and sympathetic that the changes is coming down the path, I think you need to have some degree of empathy with the people who are driving the changes. Do you know what I mean? CHARLES: Oh, absolutely yeah. SAM: You know, as we've mentioned already, I think people get attached to what they build and I don't think you can do anything about that. They will get attached to what they build. CHARLES: Yeah, we've all experienced it. It's so true. It's impossible to [inaudible]. SAM: Yeah. We can try and target it but it's sort of part of our nature. I think there are sort of the tools of the design sprint, of the design jam, of these sorts of things where if you can build to some level is obviously that it can't be shipped but can actually tease out the needs of the user, the designs sprint, the Google team, they have an example with kind of like this robot for hotels where it's like the whole thing is they can have robots that can deliver toothpaste to your door if you run out of toothpaste. In this design, obviously, that would be a huge undertaking to make that as a sort of our production system but they used like a remote control of a robot to simulate all of the key touch points when the client of that hotel, will actually interacts with that robot. I think the same is true when you're building these systems if you can, again going to touch back on we don't fit a code in that way that I think an ever better thing is not be building the code in the first place and to build an almost a non-code system and then, maybe some of the backend are not going to be interested in the results of what people have learned through this kind of user experience trial before they [inaudible] forever but I think it's seeing other people try to use the end system that puts you in place where you can be empathetic. I argue for the bias point of view that I think people on the backend of Netflix, trying to optimizing the streams in that or the other, they need to spend some time watching Netflix or at least watching the comedy of Netflix in order to empathize of how their work relates to the end experience and then, when those end experience needs changed, it's important for them to make change on the backend. Do you know what I mean? CHARLES: Right, exactly. They have to watch through that kind of glass window where they can actually help the person. They can't give them any information. The only levers of control they have is through their own work, making the thing changes that they need to make on the backend, so that one of those users is going to have a good user experience. This idea reminds me of something which I believe is the case at Heroku where they rotate everybody in the company through pager duty, which I think is kind of a brilliant idea where the entire team is responsible for providing technical support to the end users. When a problem arises, you get to understand what it's like to be someone who's trying to use the system. You get to exposed to the satisfaction, whether an issue was resolved or when it's working correctly or your pager is quiet for your entire shift but you can also get exposed to the frustration that you're engendering in the users if something doesn't go quite according to plan. SAM: Yeah. I'm not [inaudible]. TARAS:: There is movement in this direction because there is actually a term that has been floating around. It's like a product engineer. It's someone who thinks about the product. These kind of people tend to have the highest value in Silicon Valley as someone who thinks about the product as the outcomes supposed to their code as the outcome. Even in the Agile space, there seems to be movement in that direction because I think one of the challenges, like to do that, you have to understand how you fit into the bigger picture. I could see it being really difficult. Imagine if you're working at a bank or something and being pager duty at a bank would be impossible, simply because their organization is either too big or too sensitive. The way that their company is dealing with that is they have these group of people where you have a product manager working closely with the frontend engineer and the backend engineer so there's this exchange that is happening where people get to understand the consequences of their actions a little bit more. They understand how things fit together. There is definitely a movement happening in that direction that I think just the more, the better because one of the big differences now is the things that we make are very palatable to the users and the quality of the user experience that users are now expecting is much higher than they were in the past. I think the world is changing. CHARLES: Yeah. We mentioned this when we were talking before the show but going to the University of Michigan as I did, I was in school with a bunch of mechanical engineers. Their goal was to go work for auto companies: Chevrolet and Ford and everything like that but their mindset was very much, I think the product engineering mindset. All of them loved cars and wanted to be part of building the coolest, most comfortable, most responsive cars. For whatever definition of a good car -- there are a lot of them -- they wanted to design good cars but they were into cars and they were into the experience of driving a car, so what's the equivalent then of that product engineer in software? I think that every conversation that I had with any of the mechanical engineers who were going into the auto industry, I'm sure there are some of them that were out there but it wasn't about carburetors or whatever. They really wanted to just get into the building of cars. In some aspects, I'm sure they all integrated in some way or another but it was a group that was very focused on that outcome. SAM: In the UK, I'm hearing this term, 'service designer' a lot that's coming up. Are saying that, Charles, you get the feeling that even getting into software, I'm not so excited about using their own? Like the car, I want this amazing car and then I can drive the car and I'll experience the car --? CHARLES: Right and other people will experience the car that I've helped design. SAM: Right and in some way, in software, it's almost like the software itself there's this kind of mathematical beauty of its own and it's like always been more important than the other software heads around me like the software that I wrote. I mean, the user? At the conference and all the software guys love it, that's the key objective, isn't it? Yeah. I work with a lot of learning developers who are rightly focused on wanting to improve their skills and wanting to level up in particular tech stacks that are the ones that will lead to jobs and the future and financial stability and so on but I kind of wish I could inculcate in them this desire that the user experience was the higher goal than which bit of code does this or the other. Maybe, I'm being unfair to them when I say that. Coding is hard. There's a lot strange concepts to grasp than sort of grappling with those concepts -- how does this work or why does this work or should I use this function or should I use this method or what have you. I don't know. I think [inaudible] bit a wall when I'm sort of saying like, "Let's think about it through the user perspective, what does the user needs here." It feels like they want the safer answer of what's the correct solution here, how should this be refactored because you need a skinny controller or a fat model or whatever happens to be. I don't know if that's -- CHARLES: Right, like what's going to make my life easier, in a sense of if I'm going to be maintaining this code and it's important. It's very important. You want to be happy in your job, you want to be happy in your work and so, you want to have the skinny control or the fat model because it's going to lower the risk of pain in your future, supposedly. I definitely understand but that can't be the only incentive. I think it's what I'm hearing. The thought just occurred to me, I actually don't know that much about game development. I know the game industry is certainly has got its problems and I don't know much about the development culture inside the game industry. I do wonder what it's like because it does seem to be somewhat analogous to something like the car industry, where if you're a developer in the game industry, you're probably extremely focused on the user experience. You just have to be. It's so incredibly saturated and competitive for just eyeballs and thumb on controllers. Like I said, I don't really know anything about the game industry when it comes to development but I'm wondering if there's an analogy there. SAM: Yeah. It sounds strong to me and as much of my cousin who works in the game industry and I sort of taught game programming and work through a lot of it. The game industry has got these sort of user testing experience baked in and it kind of like repeat, repeat, repeat, repeat. There's this sort of constant cycle of doing that and in the somewhat wider software industry, in a certain extent, pay lip service to that. CHARLES: The game industry, they live and die by that. The code lives and dies by how it actually plays with real users. SAM: Yes and it feels like in the general world, there's a lot more user interfaces that they just sort of struggle on whatever market dynamics people need to use, these existing vested interest, these banks, these supermarkets and what have you. I guess the market is too big, almost. It's not covered enough but do we really want our industry to be so cutthroat? I don't know. CHARLES: Yeah. We've kind of seen an example of a couple of industries: the car industry, the game industry which is kind of adjacent to where most software development happens but they have this concept of exhaustive user testing, kind of the pager duty if you will, where you get to experience that. So how do we, on our team and when I say our team, I mean anyone who happens to be listening, what's the equivalent of pager duty for the applications that we write? How can we plug ourselves into that cycle of user-ship so we can actually experience it in a real and repeatable way. SAM: In the work that we're doing with the NHS, they have a similar program. There's a lot of people who are working purely death by desk jobs but they do have a framework for us to go and observe in the hospitals and emergency rooms and so on. I guess the 'how can we achieve that outside of those clients who have those framework in place,' it seems like maybe we need a version of the cycle where a different person gets to be the product owner and tries to represent what the users are experience each week. I think almost though, it seems like we need more of that thing you mentioned, Charles which is like kind of being behind the one-way silvered mirror as some sort of framework that connects the loop between some of the individual developers are doing and their experience of how the users are seeing things. I think that's going to be difficult to introduce. Just to sort of back up a little bit with them, I see this kind of idea of Agile, which says, "Let's be using the software which is the thing that's kind of changing and evolving unless you already deployed and you are in the maintenance mode from the beginning and you're getting that thing out there so you have your one-week or two-week or three-week cycles where you keep on having touch points," and I think that's better than touching once every two years. As malleable as modern software potentially is, it's too sticky. It's like you were saying Charles before with that deploying or whatever, these are all gyros and so, you kind of need a really good mechanism for mocking out interfaces and having users experience and there's a couple of [inaudible], vision and there's something else that's sophisticated systems on the UX space where you can put together sort of a simulation of your interface relatively cheaply and you can run these things and then have sort of video capture of the users interacting with the system. I think the difficult part that we have in the software industry is we are in this thing, where there's not enough people wanting to be software developers, partly with I guess the cars and the games, is there's so many people want to be game designers that the industry can kind of set the terms but we're in this inverse situation here with software developers where software developers are so much in demand, they can kind of say, "Don't put too much pressure on me. I'll kind of like, I'll go off to different company." You know, we kind of have to like herd the cats, as they say into allowing these high powered software developers to go in the direction as they want to go in and so, it may be difficult to impose something like that, where you're getting software developers to really experience the end user's pain. I guess what one has to do is somehow, create a narrative to sell the excitement of the end user experience and the beautiful end user experience to software developers such that they're fully out of themselves and say, "Yes, I want to see how the users are using my software." Do you know what I mean? CHARLES: Yeah. TARAS:: Yeah. Because a lot of company had this process where there'll be a product manager, there'll be a designer, they'll design through mock ups and they'll just hand it over to developers to build. There could be some backend, some frontend. I think if you're doing that, there is no emotional engagement between the creators of the actual implementation and the people who are going to be using that. One way to do that is to try to add more of this actual 'however you do it.' You know, introduce more of the personal experience of the users to go along with the actual mock ups so that people understand like, there's actually be a person on the other end that are going to experience this and create some kind of emotional engagement between these two end. How you do that, I think that's a big question but I think if the organization is simply just throwing designs over to development, that's where the part of the problem is and trying to -- CHARLES: Well, actually, that's a fantastic point because ultimately, we've talked about emotional engagement and attachment to the software being a liability but the reason it exists and the reason it's intrinsic to the way we do things is that's how things get built. We get emotional attachment to it. It's the impetus. It's the driving force. It's literally the emotive force causes us to go forth and do a thing. It's why we're going to do it as a good job. Like you said TARAS:, if you just kind of throwing your tickets over the wall, there is no emotional engagement and so people will look for it wherever they can find it. In the absence of an emotional engagement, they will create their own which is good quality software that's 'clean' code because people need to find that meaning and purpose in their work. Maybe the answer is trying to really help them connect that emotional experience back to that purpose of the user experience, so that there is no vacuum to fill with kind of synthetic purpose, if you will. SAM: That's a great point. The charity in AgileVentures that I run, when we get things right, that's the thing that happens in that. We go for transparency at open source. We have these regular cycles where the charity client is using the software in a Hangout with us, with the developers who work on these things and the developers whether they're in a Hangout Live, whether they're watching the video a week later, they see the charity end user struggle with the feature that volunteer developers have been working on and they make that attachments. It's more than just 'I want to learn and level up in this thing.' It's like 'I want to make this feature work for this end user' and we're very lucky to have these charities who allow us to do that level of transparency. The difficulty often comes that I would see in our paid projects where I would love to be recording the key stakeholders using the system but for whatever political reason, you can't always get that. I think that process of actually connecting the developer with the person who using it and doing that reliably, so that they can have that empathy and then get that emotional connection. It's just tricky in the real world. CHARLES: It's very hard. One thing that we've deployed on past projects, which I think has worked fairly well is kind of putting a moratorium on product owners writing stories or writing tickets and actually having the developers collaborate with the product owners to write the tickets. In other words, instead of kind of catching a ticket that's thrown over the wall, really making sure that they understand what is the context under which this thing is being developed and then almost as in a code review sense, having the product owner saying, "Actually, the reason we're doing it is this," and having a review process where the developer is actually creating the story in support from the product owner. Because ultimately if they have that context, then coming up with the implementation is going to be much easier. It's really is about facilitating that upfront learning than being able to do the actual work. That's something but a lot of organizations are resistant to that because the product owners really want to say like, "No, I want it to go this way and I want to just hand this to a developer and I want them to do it." It's kind of wrestling in control and saying, "You've got veto power over this. You're the editor but we really want to make sure that you're on same page with the developer." In cases where we have been able to kind of have product owners make that shift where the developers are owning the story, oh sorry, or the primary authors of the story and more of the code reviewers of the story, if you will, implementation just seems to go so much more smoothly. The questions, the key points kind of come out of the front of the process and by the time you start actually working on the thing, the manager or the product owners has the confidence that the developer understands what is involved in making it work. SAM: Getting that done, what one will say something as tricky to do, I think in the ideal world, you have more of the developers involved in the design sprints or the design jams. The logistics of it are you can't really afford to have all those developers in all of those soft meetings where they are coding away. That sounds a great medium there. I think there's various organizations that do sort of their kick offs where their stories are kind of if not code designed, then there's cooperative voting on the complexities of the stories and making sure that they have folks understand the stories that they're working on and there might have some very different organization that are going to have different constraints on how much time that the organization feels, the different people can be allowed to spent on different sorts of activities. It's sort of tricky, isn't it? CHARLES: It is definitely tricky because any time that you allocate for people, that's the most expensive resource that you have, so you want to be smart about it. SAM: Which comes back to that issue then again of repurposing people. If you've got that feature that you go over 10 days, can you say to that person, "Can I switch you over onto this other thing for four days?" Maybe logically, that would be a better outcome for the sprint but maybe emotionally, that person is so attached to that. They are not fighting that front. The biggest trouble for me, I think over the last 30 years is I assumed that the logical stuff was tantamount. These things like the skinny controller and the fat model, we'd agreed on that that was correct and so, that's why I should pushed for, that's why I should fight for because it's the truth or whatever and actually, maybe I'll sling back around in another 30 years, you can choose your battles because the level of emotional pressure on people, then the whole thing just explodes and nothing gets done. You know what I mean? TARAS:: Sam, I have experienced something very similar and I think about this all the time. It's just how many perfect things are perfectly acceptable to people in using a different lens when I think about technology because the more skilled you are, I think quite often, people become more pedantic about how they approach things but in practice, the more things that you see that are not written but you realize how really imperfect they are and how in many ways, they fit the world very well, people use it. It's being used by many people in its imperfect state, so there is something like this hole between perfect implementation and the role of this perfect implementation in the world that I think that duality is really interesting. CHARLES: Yeah. There's a fantastic paper written by, I think it was Richard Garfield back in the late-90s or early 2000s called 'Less is More,' where he kind of talks about this exact tension and he calls it the MIT school of software and the Berkeley school of software and I think Garfield came from the MIT school but basically, the whole thing was saying the Berkeley School is actually right and the name of the paper is 'Worst is Better.' It's a really interesting essay but just talking about working things that are in people's hands will beat the best design every single time because those are the systems that get used and improved. There's a lot more to it than that. He talks about this exact fundamental tension and obviously, no system exists on either one of those perfect poles -- the MIT school or the Berkeley school. I think what he was trying to point out is that exact fundamental tension that we're talking about, the quest for the 'correct solution' and the quest for a solution. I'm actually have to go and re-read it. I remember it being an intriguing paper. SAM: I guess the thing that makes me think of is sort of related to philosophy value. I read recently about this, attachment to outcomes. I'm going to segue back but we've got this discussion in one of our teams about, this is sort of microservices versus monoliths and in reading this, I doubt microservices, which is actually pretty radical in some of its suggestions that they're talking about it in Greater Than Code Slack where it's sort of saying that actually, if you keep your microservices small enough, then a lot of the things like code quality become actually kind of irrelevant it sort of almost argues that a lot of the things that we like about the Agile and these things are our ceremonies that are only necessary because we trying to feed these monoliths. I'm not saying it is true. Iím just saying it's a pretty radical position that itís taking and it certainly, captures the minds of some people in our organization. Some things that I've been trying to make some simple changes just to smooth off some of the edges of the platform that we're working with and I had this respect of like, "No, we can't just make those simultaneous. We need to move to microservices." They're great and it will be perfect and they will allow expansion and so on and my immediate reaction is sort of, "I'm glad we don't have to build microservices because [inaudible]. It's all about attachment to outcomes, so am I attached trying to automate part of my week that I think will have some positive goals in the future? No one's got a crystal ball. That's only a guess on my part. If I've got some folks who are excited about doing this thing with microservices, maybe I should empower them in that and I should started a mobile microservices and we kind of playing with these set of framework and so on. It's interesting stuff as I'm working my way through the book. At the same time, while we've been doing that, I've actually delegated it to someone else to sort out this sort of thing and I've actually kind of addressed the problem what we would need the microservices for through a different mechanism but still, the microservices thing rolls on and I kind of think, "Well, is that all a complete waste?" But then actually maybe, in two years down the line, it will turn out and that will be a beautiful thing that will enable things. The further that I go on, the more I say, "I just don't know," and actually, if I can detach myself from caring too much about the outcomes one way or the other, I think it's both long and enjoy myself but it's a tricky thing when you got to pay the rent and pay the bills and so on. I don't see any resolutions to that. I love people being able to use things and get stuff done. I'm so excited about that and I guess, I will keep pushing all of the developers that I'm with towards trying to have a better empathy or understanding of their end users because I think software is more fun when you're connected to the end people who are using that. CHARLES: Absolutely. Microservices reminds me kind of the experience that we've had with the microstates library. Because we've kind of identified this one very small slice of a problem, a 'refactor,' it's basically a ground up rewrite. If you've got a library that is a couple of hundred lines of code but it presents a uniform API, then you can rewrite it internally. You can refactor it by doing a ground up rewrite and the cardinal rule or the cardinal sin is you're never supposed to do a rewrite. SAM: Yes and that's the microservices that they're saying. It's like you should rewrite everything all the time, basically. CHARLES: Exactly. You should always be rewriting it. SAM: Yeah. Should be able to throw it away because it's a microservice and it can be rewritten in a week and if you're throwing one away, it's only a week worth of work and you can keep on moving away. It is an amazing idea. CHARLES: I have to read that book. Anyhow, we're going to have you on again to explore -- SAM: Well, let's do another one on microservices. It was a great fun. Definitely, it's time to wrap up but I really appreciate you having me on the show and being able to discuss all these topics. CHARLES: Fantastic and I can't wait to talk about microservices. This might be the impetus that I need to finally actually go learn about microservices in depth. SAM: I'll recommend Richard Rodger's book, 'The Tao of Microservices.' CHARLES: All right. Fantastic. Anything we should mention, any upcoming engagements or podcast that you're going to be on? SAM: Iím always on the lookout for charities, developers, people who want to help, you can find us at AgileVentures.org. We're busy trying to help great causes. We're trying to help people learn about software development and getting involved in Agile and kind of like experience the real software in action, software development where you ideally interact with the end charity users and see how they're benefitting from the product. We love any support and help. You can get involved in that, if you want to give a little bit to open source and open development. We go for transparency. We have more mob programming sessions and scrum and meetings online/in-house every day, so just come and check out AgileVentures.org and maybe, see you [inaudible]. CHARLES: Yeah and if they wanted to say, reach out to you over email or Twitter, how would they get in touch? SAM: It's Sam@AgileVentures.org and I am at @tansakuu on Twitter. Hit me on Twitter or just at Sam@AgileVentures.org. CHARLES: All right. Well, fantastic. Thank you, Sam. Also, if you need any help with your frontend platform, you know where to get in touch with us. We're at @TheFrontside on Twitter or Info@Frontside.io. Thank you, TARAS:. Thank you, Sam and we will see everybody next time. Thank you for listening. If you or someone you know has something to say about building user interfaces that simply must be heard, please get in touch with us. We can be found on Twitter at @TheFrontside or over just plain old email at Contact@Frontside.io. Thanks and see you next time.

Jacob joins the panelists to talk about team collaboration based on his RubyConf 2017 talk, Code Reviews: Honesty, Kindness, Inspiration: Pick Three. Jacob Stoebel is a software developer living in Berea, KY. He spends his days writing web applications in Ruby, JavaScript, and Python, working with data, and leveling up as a software engineer. He works and studies at Berea College. You can find out more about Jacob at jstoebel.com. Please join us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at contact@frontside.io. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer. This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. TRANSCRIPT: CHARLES: Hello and welcome to The Frontside Podcast, the place where we talk about user interfaces and everything that you need to know to build it right. My name is Charles Lowell, a developer at the Frontside. With me today from Frontside also, is Taras Mankovski. TARAS: Hello, hello. CHARLES: Hello, Taras and today, we're going to be talking like we do every time about a piece of the platform that you used to develop user interfaces frontside at your company or organization or wherever it is that you build software. Today we're going to be talking about a piece of the platform that's very, very critical that often gets short shrift or is excluded entirely from what people think of when they think about their tech stack and that's how we as teams collaborate to build and maintain and produce the quality software that we can. With us today is Jacob Stoebel. Welcome, Jacob. JACOB: Hello. CHARLES: Now, what is it that you do, in your day-to-day? JACOB: I'm a full-stack developer for a little company called ePublishing and I mostly work in Rails and React. CHARLES: Rails and React and so, when we were searching for people to talk about how we collaborate these teams, Mandy suggested you because of a talk that you've given at a RubyConf, specifically about code reviews, which I think are actually a huge piece of the collaboration process because it's a major forum where team members get to interact with each other and it's the gateway for making sure code quality is maintained but more than that, I think it's a learning -- a place where we learn. I learned so much both as a reviewer and as someone who is submitting my work and so, it's actually a very important part of the software development process. You have a lot of great examples of how to not do code reviews. JACOB: Yeah, I think I may have been a little bit too indulgent in that talk. I had a lot of fun. I did some research from other people, mainly from anecdotes. I had research from talking to people about really, all the anti-patterns that come out of code reviews. It seems like every few weeks, I'll see a tweet that says something along the lines about how code reviews are broken. I don't really know about that and I have to say, I think I'm kind of lucky at my job that I think they're done in a way that really leaves me feeling pretty positive and that's certainly a good thing but I think what it comes down to -- I'm going to sort of talk about where these ideas come from in a minute -- is that we often have code reviews that for one -- and you can tell me how this is for you too -- often the code review is happening at a point so late in the process, where the feedback that you get may not be actionable. Have you experienced that? JACOB: Before I answer that question, just to kind of echo the sentiment and maybe I'm being presumptuous, I feel like the code reviews that we do are actually very positive, so I haven't got to experience firsthand. Although I have seen conversations on GitHub where it looks kind of like a Celebrity Chef, where you have someone doing the code reviews like Gordon Ramsay up there just screaming and someone has put this plate of food in front of them and kind of picking it apart. That one is extreme but this is actually something that I struggle with, what you were talking about, what is the appropriate point at which to get feedback. I agree that you want to get feedback as soon as possible and sometimes, when you've invested weeks and weeks into something or you're like at Mile 100 and they're like, "You know, at Mile 2, you were supposed to turn right," and now, you're off in the forest and you've been tracking 98 miles in the wrong direction. JACOB: Yeah and the work is due, right? This needs to get ships tomorrow. CHARLES: Right, so you've got massive pressure. This is something that I struggle with myself is when is an appropriate time to really try and be public about what it is that you're doing. JACOB: Yeah. I think that is a really good question and I think what you're getting at and I would agree is that, the sooner, the better and when you can tighten the intervals between feedback is probably better. I'll just take a step back and I'm going to take a longer route to go and get to my point. I am a career changer and before I was in this career, I was a high school theater teacher, so it was really different and I won't give answer why I changed other than this is the greatest [inaudible]. But one thing that I really struggled with is I was working with teenagers and I really wanted to see them grow and improve but at the same time, these were kids and they have fragile egos and I don't want to tear them down, so I came across this really interesting framework for feedback. It is called the Liz Lerman's Critical Response Process and I give credit to her in the talk. This comes out of the dance world. Liz Lerman is a pretty accomplished dancer and choreographer and what she found is that in the dance world and I think it's not too dissimilar from our industry is that the feedback received was often given in a way that was leaving people feel really torn down and mostly, not feeling inspired to go back to their work and make it better. It really felt like feedback is about giving you a grade. It's like, "I'm going to tell you how good of a job you do," and there's certainly a time and place for that but the inspiring question -- I guess the rhetorical question -- that she made is, "Shouldn't the big point about feedback be that it makes you so excited about the work you're doing that you just can't wait to go back to your keyboard and keep working on it," and she found or in the case, back to the dance studio, it really sort of structures this framework for how people can give feedback to a creator and that could be a creator of anything. You mentioned cooking. This could be about food as well that sort of set some guardrails for how we can give feedback that is useful, it's inspiring and it's kind. I'm going to really distinguish between kind and nice. Nice would be, here I'm going to say things that are only pleasant about your work but what I mean by kind is feedback that is really taking care of your team and making them feel like they are respected and cared for as human beings so they don't go home every Friday afternoon and cry on their couch and just drag and coming back on a Monday. That's kind of the basic idea of why we need, maybe a kind of a framework for getting feedback. CHARLES: I agree totally. It's almost like you need to conceive of your feedback is not a gate to quality but a gift to embolden somebody. It's like they've just been doing battle with this code, with this problem, they've been grappling with it and it needs someone to wipe their brow and maybe give them a stiff drink, so that they can get back into the ring and be invigorated. JACOB: Yeah. TARAS: What I'm hearing in this conversation so far is it's kind of like a tone or way of communicating to the person receiving the feedback but sometimes, no matter what your tone is, depending on how your team is set up, depending on the context of your actual code review, it can still kind of land in the wrong place? Have you experienced that where the team conditions impact your ability to actually provide feedback? JACOB: I think I know what you mean. I think what you're talking about is this sort of the organizational structures that are set up are sort of lend themselves to certain modes of feedback and discourage others. I will give this example that I've heard from numerous people and I think this is what you're getting at is feedback that's given at the end. Just like I said, feedback that is given the day before, it has to be shared. Or feedback that it's almost like if we work in an environment where sort of like the hair on fire environment where it's like everything was due yesterday, that's not an environment that's going to be conducive to slowing down, taking a step back and saying like, "Let's point out what we think this work is doing? What questions we have about it? Who has opinions about the direction that's being taken on it?" And really zoom out a little bit more. CHARLES: Do you have any concrete examples of how that early stage feedback has taken place at your work? Is it just over Slack? Maybe, the whole people need to step back from the idea that working on one-week iterations or two-week iterations and at the end of the iteration, everything is due and everything will get merged at that point. How do you kind of break up that structure to say, "No, we're going to try and have checkpoints and milestones?" What are deliverables that you can decompose the work into that fit inside the framework, that are inside the big deliverables, which is sensibly, the feature that you're working on? JACOB: I think first of all, it's the way this thing goes about. I don't think this framework works over Slack. I think it has to be immediate, I think it has to be either in personal or on Skype or Hangouts or something. One of the points of this type of feedback is really about checking in with each other as a team and I think what's great about Slack is that it lets you leave a message and walk away and the unfortunate thing about Slack is it's not meant for sort of attaching how people feel about things or what people's reactions were to them. I think you all have to be in the same space, hopefully with your camera on, if possible and really, just sort of checking in with each other as a team. I can point to times with my team that I think that's really worked out. I think you made a good point when it comes to really getting started with a project because there are times where I made the mistake of not trying to get feedback from my team early on when I was going to start a project and as you can imagine, did that really cost me? It's really getting feedback from my team and my supervisor about the direction this is going to want to go and I'll say from experience, I work on legacy codebases and as you can probably imagine, it's easy to paint yourself into a corner and what the thing about legacy code is that you don't know what pitfalls you're working into. You can get started and you can sort of find into the process, there is some reality about this code that you didn't know and it is really getting in the way of your work that had you known about it, it would have saved you a lot of trouble because you could have planned your way around it. Then the fortunate thing is drawing on the wisdom from people who they know about it because they struggled with it already if they've been around longer than I have. I think that's a really good point. This is a good thing to do. As you're getting started and you can be sharing this is my just initial idea of how I'm going to go about this. What my tech stack is or my understanding of the problem space, all of the above and to really sort of check your assumptions and see if they really check out. CHARLES: I want to circle back a little bit to something you'd mentioned before and that is you want to be kind in a code review and I would say, you probably want to be kind anytime you're giving feedback or interacting with your teammates but what's the process that you can go through if you find yourself struggling? How can I deliver this message that I want to deliver and have it come across this kind? What's a process or checklist that I can go so that I don't open my mouth up and say something that I can't take back or that is going to land wrong? JACOB: I'm glad you asked that and I think that this is a great way to introduce the guidelines for the critical response process. There's basically four steps to it and the way it's structured is you have the person who has created the thing. It could be the person or the team that has created the feature. You have other people who are responders to it. They could be people within your team or they could be others in the organization who are invested in your success. They're not here to tear you down or give you a grade. They're here because they want to see the project do well and then, there's going to be someone who is a facilitator. The way it works and I think this is directly addressing your question is there are guardrails that are going to help you not steer into territory that is going to leave someone feeling torn down. The first step to this process is all the responders are going to say statements of meaning about the work and what I mean by that is you're going to make things that stand out to you that are not attached with an opinion. You could say, for example, "This project is using React hooks. That's something I noticed." You're not going to say, "I think that's a good idea or a bad idea," but you're going to say, "This is what I'm noticing," and that is something that can get jotted down because that's going to be fodder for discussion later on like, "Let's talk about this new feature in React," and let's talk about it later. We can talk about if we think that's a good thing or not or what the implications are for that. The next step after that is that the person who has made the work of the team, they get a chance to ask questions of everybody else about what they thought. You've probably noticed that in a lot of pull requests, someone puts their work out there and then, the next thing that happens is everybody starts commenting on the work and saying what they think about it. This flips it. At first, you as the creator, start asking questions or you can say, "This thing over here, I think it's maybe a little wonky or it's a little hacky but I couldn't think of a better way to do it. What do you think? Do you think it's right? Do you think it's a bad idea?" Or this bit over here, "I'm pulling in the third party library. I think maybe, it's worth it but what do you think? Is it not worth it?" CHARLES: I like that because ultimately, the people who have created the thing are the most familiar with the problem space. They've spent a lot of time thinking about it and so, they can actually direct the conversation to the parts of the implementation that really are the most iffy and the most unknown because everybody is going to feel this need to comment but it's the classic case of bikeshedding really, the true experts or the people who've just been spent all this time implementing and so, people will comment up to their greatest point of familiarity with the problem, which could actually be not that great. Can you say like, "Can you really focus your mental energy on this?" I like that a lot. JACOB: Yeah and really, it sets a good tone. The assumption behind all of this is that the creator, like you said knows the work best and really ought to be in the driver seat when it comes to feedback, so it really sets the tone there and say like, "The creator knows what's most important. Let's give them the first opportunity to frame that," and I know plenty of times like if I'm asked to review a PR and I'm not given any kind of prompt or direction, I will probably scroll to the file. Maybe I'll find the file in that PR that I'm actually familiar with or I'll look for some pattern that's something that's like, "I understand what's going on here. I can give feedback," and if it's all that other thing over there, I'm completely confused, I'm going to ignore it. But the creator could illuminate me a little bit, then I would understand a little bit more and then, I'm in a position to comment on that thing that I otherwise wouldn't have understood. That's the second step. The third step is now the responders get a chance to ask their questions but the important thing about that is that there are neutral questions. The assumption here is that the responders need to better understand the context from which this code was written in order to be able to give their opinion. Here's an example coming from the Rails world. I could say, "Tell me your thought process for using Factory Bot," in this example. If anyone doesn't know that is somewhat of a contentious issue in the Rails community, rather than just coming out guns blazing and say like, "I think this was a bad idea." It's like, "Tell me your process because I want to understand the context you came from and then we can together evaluate if coming from that context makes sense or if it doesn't," so neutral question. They have to be neutral questions. By the way and we've probably all heard this before, this is not a neutral question, "What were you thinking when you decided to do blah-blah-blah-blah-blah." Everyone knows what that means. CHARLES: Right. I was going to say, you can be very, very passive-aggressive with questions. JACOB: Yeah, exactly and this is a place where having a facilitator can really help -- facilitator who is not one of the creators. Someone can just say, "Let's back up. Can you rephrase that without the opinion embedded? Can you just say that again?" and with that again, those are some of the guardrails that keep us on track. After the responders have been able to ask their questions how hopefully everyone has a better understanding of the context from which the code was written, then it's time for opinions with consent of the creators. The way it works is the responders can say, "I have an opinion about using React hooks in this codebase. Would you like to hear it?" The responders can say, "Yes, please. Let me know," or they can say, "No, thank you," because the responders, having the best knowledge of the context, might know that that feedback is not useful. Maybe, they have a really good reason for using React hooks or maybe, they know what's coming in the future or maybe, they know if there's no time to fix it now or it's not worth fixing now. They know the tradeoffs best and so again, those are guardrails and it puts the creator in the place of saying, "That's actually not useful feedback right now. Let's just not use it." They can say, "Yes, please tell me," or they can say, "No, thank you. Let's not talk about that." CHARLES: In the context of a pull request, the process you're describing could be played out in any number of media but in the context of a pull request, the creator is the person actually submitting the code, how do you handle the issue of who pushes the merge button if there's still some opinions that haven't been voiced? For example, a creator says, "No, I don't think that's helpful feedback," is the assumption of then the creator can go ahead and just push the merge button? Or is it basically saying, "I don't want to hear your opinion," relatively rare? JACOB: That is a great question. I think I tried to get at this in the talk. Before one thing, I am probably, like most people don't have the option to just say to my manager, "No, I don't want to hear your opinion." I get that. There is a contrast between the pure version of the feedback process and then reality. They have to balance. But teams can sort of work out the way they give feedback. I have example of an anecdote that someone shared with me once, when I was doing research for this talk. There was this sort of agreement with the manager that their manager wouldn't give feedback on Friday afternoon. They just wouldn't. Everyone preferred that. Everyone sort of wanted, say on Friday afternoons, I'm really going to be just focusing on winding down for the week and I don't want to get dumped on a bunch of feedback. The point being, even though you can't just blanket-ignore feedback, you can work out circumstances with your team for the best way that feedback can be given and circumstances under which, it can be politely declined. CHARLES: I see. TARAS: I'm curious about a different part of this because a lot of this is how to give feedback but I'm really curious about the why part. I think many of us take it for granted that good code reviews are very valuable because a lot of teams that I've encountered that are taking on really big challenges but didn't have a code review process and so, one of things I'm kind of curious is for people or for teams that don't have it in place, what kind of symptoms can they observe in their daily operations that would suggest that maybe code review is something that they need to put in place. CHARLES: Taras, you're talking about kind of the places where we've seen where the culture is just push everything to a branch, there's a 1000 commits there, open up a pull request and no description, no name. It's like, "Here's this thing. I'm taking comments for the next three hours and if everything goes well, let just merge." Are you talking about those kind of situations where really a big culture of pull request or just feedback around change is very, very nascent.? TARAS: Yeah, a lot of times, it's the 'ship it' culture, like this is getting in the way of shipping it. CHARLES: So you're saying like how you sell the entire idea? TARAS: Yeah and if someone is listening who is noticing that, a lot of people would know that we're not really doing code reviews but what are the symptoms that they could be observing that could say like really, this is we need to change, like this can't continue. JACOB: Yeah. One thing that occurs to me as if there's all high level of surprise when you read it, when pull requests are read, it's like, "Oh, you went in that direction." I think that could be an indication that maybe, we could have checked in at the halfway point or even sooner because there seems to be differences of perspective on where this is going or where it should end up that's why [inaudible]. TARAS: At what point do you think people would see this? Is this something that would happen kind of way down the road? Like actually, "How did this end up in the codebase?" CHARLES: That's surprising except deferred even further, right? JACOB: Yeah, who wrote this last few. In having that kind of feedback process, let's sort of take a look at where this project has come in the last few months and see if we can sort of learn from what went well and what didn't. CHARLES: This is related to the concept of surprise, if you see a proliferation of many different patterns to accomplish the same thing, that means the communication is not there. People are not learning from each other and not kind of creating their own code culture together. If that's missing and that manifests itself in all kinds of ways, in bugs, in weird development set up that takes me two hours to get this local environment set up and things like that, if you're seeing those things, chances are you need some way to come together in a code review culture is really, really good for that. JACOB: Yeah and I think in the ideal code review culture, everyone that's sort of brought on board is saying, "I am going to share responsibility in this patch, doing what it's supposed to do and not breaking anything or not burning down the world." You mentioned the 'ship it' culture, I could imagine toxic cultures where the person who shipped it, if it broke everything, it's on them to fix it and they have to get woken up or whatever and the idea about feedback is now we're sort of forming a community around what was done, so it's like the person that push the merge button isn't the only person involved. It's like if we have a culture where we say everyone that participated in the PR is collectively sharing the consequences and that will happen eventually and say like, everyone who's on this thread, it's on all of us if something goes wrong to fix it. I think that is certainly something that one would hope you'd see. TARAS: I'm curious, where do you see this kind of observations usually come from because sometimes, I can imagine there, being a developer, coming on the team and they're like, "Why wouldn't I do code reviews?" and they're like, "Well, we have always not done code reviews," but then there could be someone like a product manager or somebody who's like, "Why wouldn't I do code reviews? We should start code reviews." Have you seen ways of introducing these ideas to teams that have worked out well? JACOB: Yeah and I should probably say that, I'm not a manager, I'm certainly not an expert in how this works so I actually don't have a really great example, personally. I can churn example from working in a previous team, where everyone secretly wanted to be more collaborative but didn't know how to do it because if I'm the first person that puts myself out there and no one knows else knows how to collaborate with me, how to reciprocate, then what was the point? For the top 5% of teams that are just really have great energy together, they don't need a framework to do this sort of thing. They're just doing it naturally. I think for everybody else, we need some kind of guidelines to do this because for better or worse, this is the industry that we're in right now. It isn't built around us. We don't know how to function this way and it's no one's fault. It's just sort of that's the way we've all sort of learn to work in this industry and I suspect we're not the only industry like this but we need some kind of guidelines to do it. TARAS: Maybe it's a difficult question to answer. It varies probably from team-to-team. JACOB: Yeah. CHARLES: Yeah and maybe, we can kind of shift the question just a little bit because I have one that sits alongside not quite the same question but I think related and can bridge it and maybe we can find the answer there. We've talked about a little bit and there's definitely more to unpack there, how to give feedback that's kind and honest and I think to... What was the third? JACOB: Inspiring. CHARLES: Inspiring, that's right. What about from the flip side? This is something that actually comes up with us as consultants but I think it's something that other people will encounter in their jobs too, is what do you do when you are the creator and you're trying to present or share and ultimately solicit feedback from a stakeholder, the CEO of your company, one of your clients, one of your customers and you see them engaging in kind of a mode of feedback that's less than constructive. They're nitpicking on things that aren't really important and maybe, this could be completely and totally inadvertent. Their intention could be that they're trying to help you out but it's really not being helping at all and it's kind of like either tearing you down or just not being productive and it makes you feel... What's the word I'm looking for? Just brings an air of contention to the conversation that's not really helpful to producing the best result for everybody involved. How do you, as the creator actually engage with those people in a positive way and kind of help establish the guardrails and be that agent of change when those guardrails don't exist in their minds yet? JACOB: Yeah, how do you do it, right? It's probably rare that there's going to be an environment where someone's going to say, "We're going to do this framework for feedback," and everyone are onboard from Day 1. I think one of the things that you're probably getting at is the frustration that happens when people start giving feedback. You have this perception that they're giving you feedback that's unuseful and they're only giving it because that's the only thing they can think of or -- CHARLES: Exactly. They got to say something, they need to contribute their two cents to the conversation and some people are trying to be helpful and just aren't. Some people are just trying to appear smart. JACOB: Yeah and it's like, "Oh, boy. They're just really off." CHARLES: But you can derail the main narrative that you're trying to establish and get off in the weeds, kind of skirmishing with these people and after that happens, you're like, "Wait, no. I don't want to end up over there. I was trying to tell a story." JACOB: When I gave this talk once, one idea that someone threw out was when you make a PR, mark up your own code first with everything that you want to draw people's attention to. I think you were getting at this as like, "One frustrating thing is when people getting feedback seem to have less invested than you do." They sort of just flying by and dumping on you and they probably, actually couldn't care less and that can be frustrating. When you're engaging with people that maybe don't know how to get deeply involved in it yet or maybe don't have the time to, you can sort of gently nudge them to sort of like, "I want to talk about this part of it." It's almost like you're giving the answers to the test, which was like, "If you want to be part of the smart conversation, comment over here," and I think from the responders perspective, just speaking personally, I would appreciate that. It gives me an opportunity to feel like I'm actually being useful. We can all probably point to times where all the code review we have to do feels like homework that isn't the best use for our time. It's like, maybe the responder can make us feel like our opinions matter and they will be put to good use when you tell them, "If you comment here, it will probably be put to good use." I think the sort of the way to get started is the responder can just say, "I would really like feedback over here," and I can't speak for everybody but I would suspect that more people than you think would be more than happy to be gently guided in what feedback they should get. CHARLES: Right. I'm thinking how you would do this, for example in the context of a demo that you're giving to stakeholders because there's maybe the inkling of the idea to do that but it's often presented as an apology like, "This screen doesn't work," or, "Your handling isn't quite right. Sorry we're going to fix that." Look at the stuff that's over here and maybe, the way to frame that is a really hard problem based on the legacy architecture that we have for displaying errors and I'm not quite sure how to do it in a robust way and I could really use some feedback there. It's an area of exploration or something that we really need to focus on. You put that out there as I'm demoing this main functionality right now. JACOB: Yes. You sort of say like, "Here's something to watch out for and please, let us know what you think," and then after you could say like, "As a team, we thought up these three possible solutions but we didn't want to move on them because we wanted to get your feedback first, so please impart on us what is the right way to do it." You know, flattery goes a long way. TARAS: I have this imagery coming up in my mind as I'm hearing you guys talk about this that I keep seeing this kind of difference between a line and a triangle, where a line is kind of a tug and pull between, "Did I do this right? No, I didn't do this right." I think those are kind of a bad code review because it's very personal. It's not really where you kind of want to go and then the other way is like a triangle where the end goal that you're trying to get to is somewhere that is beyond both places where you let two people: the recipient of the code review and the giver of the feedback, the end goal that you want to get to is actually someplace else. When we deliver the code and we say, "My code is done," then it invites this kind of linear feedback where it's like, "No, you didn't do it right," but in the other way, "This is where I got to so far. Tell me where you think we could take it next. If you don't think we need to change anything, then we're done. We can merge this." JACOB: Yes. The very intelligent Jessica Kerr said recently on another podcast, there's no such thing as done but you can always make it better. I think you're getting at that point. That's a great question, by the way to ask of your responders -- where should we go next? Where do you see this going next? What will make it better? What would make it more robust? What would make us happier six months down the line when we're looking back on this? But I think of it as the firing range. That's the analogy I gave where it was like, "You put a pull request out and you assert that it is done and if no one can find fault with it, then it's done," and I just don't think that that makes sense. I don't think, really most people actually want to work that way. It's maybe not necessarily even healthy but I think everyone can find, at least I hope, a process that invites them to come into the process and share the way they see things. It can hopefully be enriching and just make everyone feel better along the way. TARAS: This sounds to me like the inspiration part of this conversation because one thing I really like about working at Frontside, I'm an to experienced developer but I know that Frontside is dedicated to doing something much greater than I can do as an individual. When I ask, "What do you think about this?" then I know the feedback is going to be because there something always that is just beyond the reach that could be a little bit better than what we have right now and it's not until I can actually get the feedback from Charles, from Jeffrey and hear, "What do you guys think? Is this it?" and they're like, "What about this?" and I know that the next step is going to be a little bit or maybe even a lot better than what I have right now. I think that's the part that inspires me. I know I have actually gone a little bit further beyond my personal ability to get us to that vision of like this being much better than we could do as individuals. JACOB: Yes and as opposed, "Oh, I screw it up and now, I have to fix it." The framework of find all the errors that I made, even though there may be errors. Let's be honest, there could be things that would be really bad, that would be a security problem but for a growth mindset, to think about it, it's like, "This is a way to make it better." CHARLES: One of the things that you can do to help that is try and find inside every change, try and see the potential that hasn't yet been realized but is enabled by this change, like what exciting paths does this change unlock in your mind, right? And then you can share that. That's a great way to 'inspiration is infectious,' so if you can find inspiration to change, then you might be able to share that with the creators. If something is very personally exciting to you when you see something, maybe spend a little time searching for what you find to be exciting about it. JACOB: Yeah, nice. That's great. CHARLES: Yeah, we might have touched on that. I just bring that up because so often, I'll see the changes that people on my team are submitting and sometimes, it can feel like a cloud burst of like, "We could do this or we could do this or we could do this," and it's great to get excited. JACOB: Yeah. I can share -- recently, there was an example. My coworker opened a pull request and it unlocked something for me where I said like, "You know what? This is making me think about this other thing that I've always hated doing with our legacy codebase. We should do that thing you're doing more because boy, would it make life easier?" and I wonder if it would be worth the time in refactoring X, Y, Z because then I would never ask you A, B, C again and I would be so much happier. TARAS: It makes me think that we need to revisit our pull request template like what would that like? What would be the sections on a pull request template that would facilitate this kind of way? CHARLES: I don't know. I know we're almost a time but before we even get into that, this is a question I have because we talk about pull requests templates, how do you match the amount of process with the scope of the change? Because it's probably a little bit heavy weight if I want to fix a typo and say, "Now we're going to have the creators lay out their case that they're going to ask questions of the reviewers and then the reviewers are going to ask questions and once everybody's been able to write down things that they observe the questions that they have, completely and totally divorced from opinion, now we can talk about opinion that is welcomed." If you're capitalizing one letter, that's probably a little bit heavy weight but on the flip side, it's probably absolutely warranted if it's a major feature that's going to be affecting a huge portions of your revenue stream and so, this is one of the problems I have with pull requests templates is they are one-size, fits-all. Sometimes, a pull request template feels like it's supporting you and sometimes, it feels like the epitome of busy work and I have to spend all this time deleting the sections for the pull requests template. I wish there were ways you could choose different pull requests templates. Maybe, there are. JACOB: So do I. CHARLES: That's a little bit of a quibble that I have is the amount of ritual and ceremony that you have to go through is fixed with a pull request template but it seems like you want to match the amount of process to the scope of the change. JACOB: Yeah, you do and the flipside is also joy. You don't want to give someone too little homework when you really needed the same work. Maybe there's a way to say, when a pull request is opened, the person who opened it or the manager or maybe someone else, has the option to sort of tag the pull request as saying, "This is going to be a bigger conversation. We cannot merge it until we have a face-to-face conversation first with these various stakeholders." Maybe, one of the questions, the first form that everyone gets for every PR gets is what level of PR is this? Is this a quick change? And what people that you just need some quick feedback on? Or is this the long form that you need where there needs to be a sit down with these people? TARAS: Actually, I was thinking what's the adjustment that could be made for pull request is to say, "Here's what a complete pull request looks like. You can opt into whatever section you want," so you decide based on your pull request, what the actual sections of this template are appropriate but then someone can, on the flip side say, "Look, I would really like to learn about the motivations of this pull request. Can you add motivation section?" and understand that from your pull requests. JACOB: Absolutely. You filled out Section A, please answer Section B and C. Yeah, cool. TARAS: Yeah. Because I think what part of the challenge is that we have people look to us, especially people who have a lot of experience, or developers look to us to know what is the right thing to do sometimes or often and I think it's helpful to be a little bit more gentle and saying, "You can decide what is the right amount of detail to set the right conditions for this code review but I will give you some directions for what are things you can consider in including. JACOB: Absolutely. CHARLES: All right. We're a little bit over time, so we should probably go ahead and wrap it up. You are absolutely right, Jacob. This is a topic that keeps on giving. We didn't really move much beyond the pull request and feedback and stuff but we moved a lot around within that topic because it's a big one. Jacob, is there anything that we should mention? Any upcoming talks, meetups? JACOB: No, I have a one-year old and it's all about work and family in this part of the life, so I have nothing to speak of at this point. You can come find me on Twitter as JStoebel. CHARLES: Okay, awesome. Well, thank you so much, Jacob for coming and talking to us. This is a very rich topic. We only really scratched the surface. That's it for our episode today and we'll see you next time. Thank you for listening. If you or someone you know has something to say about building user interfaces that simply must be heard, please get in touch with us. We can be found on Twitter at @TheFrontside or over just plain old email at Contact@Frontside.io Thanks and see you next time.

Here it is, folks! The first episode of our newly rebranded "Frontside Platform Podcast". In this episode, we talk about why platform? What is going to come out of these conversations over time? Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer. We also want to get people thinking categorically, rather than in the moment: Planning strategically Recognizing and knowing obstacles - matching Don't want the framework, but still have the problems Virtue is a weakness in different contexts Laziness is a virtue but also a weakness Not a question of can Shared problem Please join us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at contact@frontside.io. This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC.

We're rebooting The Frontside podcast with a focus on JavaScript platforms. We'll zero-in on what it takes to build, maintain and grow web applications using popular JavaScript frameworks and tools. Join us for deep architectural conversations, interviews with fascinating speakers and stories from the the trenches of building large platforms.

Special Guest: Kris Van Houten: @krivaten | krivaten.com In this episode, we are joined by Kris Van Houten to chat about Functional CSS and Styled Components: pros and cons, the problems that they are trying to solve, and how to choose between one or the other. This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. Transcript: DAVID: Hello, everyone. Welcome to Episode 116 of The Frontside Podcast. I'm David Keathley, a software developer here at Frontside and I'll be your host for today's episode. Joining me as a co-host is Jeffrey Cherewaty. JEFFREY:: Hey, there. DAVID: And we've got an amazing guest with us today, Kris Van Houten. Kris is the author of a very interesting functional CSS library called Elassus, a good friend of mine and today, we're going to be talking about how that compares with another new pattern I've been seeing everywhere recently: styled-components. Hi Kris. KRIS: Hey, how's it going? DAVID: Doing great. Let's just go ahead and jump into things. Kris, you want to give us a little introduction to your functional CSS library? KRIS: Yeah, for sure. I guess, first of all about me. I've been primarily a frontend developer for about seven or eight years now. I don't know, at some point you start losing count but over the years, I worked largely within the Ember framework but I've also worked within React and Vue. Like most of developers over the years, you tend to require an affinity towards certain areas of software development and for me, those areas tend to lean towards accessibility but also, learning how to write CSS with what I call like the future in mind. To try to keep the story as short as possible but a couple of years ago, I was working on a project and started to realize that our CSS is one of those things that just continues to grow over time along with the rest of our application but I also noticed that we tend to have a lot of areas of repetition within our CSS, like how many times are we setting text align to center, how many times are we setting the text color to the primary color or to gray or something of that nature and maybe we could create a utility library that just does all the stuff for us, similar to if you've ever worked with Bootstrap that they had utility classes that allow you to do some things like set the text alignment or the text color or font size and things of that nature. I started thinking around it but eventually, I changed jobs and that idea kind of just went to the side but then, I'd say about a year and a half or two years ago, I came across a blog post called '15kb of CSS is all you'll ever need' and so, immediately I was intrigued. It was actually a blog post on Medium that was talking about the benefits of using something like Tachyons or base CSS, which are two different functional CSS libraries. To explain what I mean by functional CSS, it's just a whole library, a whole arsenal of these utility classes that just do one or two things that allow you to basically take any design and implement it by composing all these classes together throughout your HTML. I was looking at Tachyons, looking at base CSS and thinking, there's a couple of things I like to customize about this but at the time, when I was looking at it, I couldn't really figure out how to customize those values very easily and so, I did every developer does and just decided to make my own and that's where Elassus came into play, where it's a CSS library that is entirely made up of functions that generate your CSS based off of the value of variables in Sass. Everything is customizable, all the way down to how the class names look and the syntax that you use. If you hate what I'm doing, you can customize it very easily. A lot of people might wonder why would you want to use functional CSS, what are the benefits that you get out of it and really, one of the first two things that comes to mind for me that really attracted me to it was that your CSS files start small and they stay small. For the most part, depending on the configuration and the particular library that you're using, your entire CSS payload can be anywhere from 10 to 20 kilobytes, minified and gzipped, which is in stark contrast to some other projects I've worked on, where just one of the many CSS files you're downloading are 745 kilobytes. Dramatic improvements there, so that's why it was instantly appealing to me. But one of the other nice things you kind of get for free by default is a consistent design pattern, right out of the box. Because if you're working on a large team that has maybe multiple designers and many developers, one designer might implement something with the spacing system that's maybe based on five pixels: your padding, your margins, your widths, might be five, 10, 15, 30 pixels, 45 pixels but then maybe, another designer is implementing something based off of more material design, which is base-4 pixels, so it's like four pixels, eight pixels, 16 and so on. Over time, little differences like this can really show up as you navigate between pages of your application and with functional CSS, you're given a limited number of options to choose from. As a result, it actually makes your application feel like it's one cohesive consistent piece of software between pages and between the features that you're navigating through. Those are really two areas that have really attracted me to functional CSS and I guess, is a tl;dr -- if that was a tl;dr in fact -- of what functional CSS is and why I like it. DAVID: One of the things we've got here was the problem that functional CSS and styled-components are both trying to solve. KRIS: Like I said earlier, a lot of the applications that many of us might be working on, they tend to grow at time by either adding features or rewriting and improving existing feature. In JavaScript, over the years, we have made tremendous improvements in our tooling with things like code splitting and tree shaking to really limit and minimize the amount of payload that we're sending to our users at the end of the day. But really, it's only up until recently we hadn't really been talking about how we do that for CSS and typically, as developers, we had deadlines, we're working with them in sprints, we had a real strict time constraints on what we can work on and for how long and so, we tend to focus on making sure the features look right, they work right, and that our test passed. In CSS cleanup is just one of those things that often doesn't get cleaned up as often as you would like. Sometimes, that's also because we assume that a particular set of styles is probably getting used by another team, working on a completely different part of the application. As a result of just not being able to work on this problem or this issue, we likely have many, many kilobytes of orphan styles being shipped to our users at the end of day and our style sheets tend to grow over time. If there's one thing that's kind of become a mantra within software development over the last few years, it's that every kilobyte matters. One of the main reasons I wanted to come and talk to guys today was about some things that I've been tinkering with on the side, just to figure out how we solve this problem and so, their approach of using functional CSS is one of those things but using something like styled-components, which is mostly used in React library of the solution of writing your CSS in your JavaScript and that's another approach to how to tackle this problem. DAVID: That's an interesting thing you said there. I remember, whenever I started off in software development, one of the big no-nos for styling is don't put your styles in your JavaScript. It's interesting to see that that has kind of changed. KRIS: Yeah, indeed. I remember listening to a podcast about two, maybe three years ago and they had a panel on and they were talking about there's a new thing called CSS and JS and I was like screaming in my car like, "This is the most horrible thing in the world you could possibly do. Why would someone want to do this?" and obviously over the last couple few years, the technology has definitely matured and they definitely hardened it, to where when I was working on updating my personal blog, working on moving over to Gatsby, I didn't want to have to pull an entire functional CSS library because I kind of thought that would be overkill for just a couple small pages and so I was like, "Let me try this whole styled-component thing. That way if I'm going to hate it, I can hate it from the experience point of view." I actually found that the developer experience of working with styled-components was really nice and so, I kind of became converted over to that way of thinking about, "I'm not a hater. It's actually a really nice way to write your CSS for your blog or your applications." JEFFREY:: I count myself still as a skeptic about this whole idea, so I'm excited that we're going to talk about it now and then, you have the opportunity to sell me on it, which should be fun. Some of the work I've been doing on lately, we've been doubling down on the idea of CSS modules, that at least fix some of the encapsulation and problems around CSS where everything is sending up in a global scope. How do these ideas of functional CSS kind of take that even farther? What advantages are there over using, just simply using say, CSS modules where we're fixing the global namespace. KRIS: I'd say with functional CSS, again the things that kind of mine for me are you keep the file size small, you have your design patterns that you can get for free right out of the box. Your styles are pretty low specificity, meaning that if you ever have a need to where you get a really customized something because you just can't do a functional CSS, which I found to be very, very few and far between as far as the number and instances I've come across, it's very easy to overwrite if you need to go back into, I guess old fashioned way of actually writing some CSS yourself. But then you also have the benefit of not having to come up with clever class names to describe the different states or different use cases of your component that you're working on or the future that you're trying to implement and so, I'd say those are definitely some of the pros of why I like using functional CSS at the end of the day. I will say that there are some cons -- I'll be completely honest -- and say that one of the areas where functional CSS tends to have a weak spot is in the area of browser specific issues. In my experience, I tend to have a lot of issues with iOS Safari and it's like the bane in my system sometimes. Sometimes, worse than i11 but if you have to really target something that's exclusively for a specific browser, that's really not doable from what I've seen in functional CSS. That's where we should have to go back in and handwrite some CSS for that kind of situation. But also, one of the cons is that sometimes, depending on the element or the component you're trying to style, I think the example is if you have a nav bar that can toggle between being a sidebar but also a top nav bar, that's a completely different set of styles and to achieve those few layouts at the end the day, you have to have a completely different collection of class names that you have to toggle between based off of the state of that component or that feature that you're working on. While that's doable, especially with something like a frontend framework like Vue, React or Ember, it's just a little pain point that sometimes you have to work around but I say, those are the pros and admittedly some of the cons of using something like functional CSS to tackle your styling needs. JEFFREY:: My favorite thing that you mentioned there was how hard naming is. I think everyone can agree, very strong of it, just naming things is the hardest thing. What has been your experience in learning these? They're almost new domain languages for like we want to padding of this amount. Have you found that pretty easy to get up to speed and get fluent in or have you run into some blocks in kind of learning these new languages that some of these CSS tools have presented. KRIS: That's a really good question. When I first started digging into Tachyons -- that was the first functional CSS library I started working with -- there is definitely, I'd say a small learning curve of maybe, I was trying to implement it just to see how it worked and it took me, maybe a couple of hours, if that, to kind of get a feel for how they write their classes and how the classes are put together to do certain things. But once you kind of get through that little initial pain point, it's pretty easy. In one of the things that when I was working on my own implementation of it, I wanted to make sure that no matter what you're doing, all the class names are consistent, that the patterns that you use to your class naming is consistent and I guess, because I wrote it, then it wasn't so hard for me but I also understand that that's not the case for everybody. When I wrote Elassus, one of the things I did is I actually was able to create a compiled JSON version of all the CSS and actually, use that to create like a little React search tool. If you're looking for padding or margin, you can just enter that in a search bar and it'll tell you all the options in the classes that are associated with that and that's one thing I did to kind of help with that barrier but it definitely can be a bit of a learning curve but like I said, it's maybe an hour or two of your time to really get familiar with it. JEFFREY:: That is not bad. KRIS: No, not at all. I was actually really surprised at how fast I got up and running with it. DAVID: That actually ties into something that I was kind of curious about myself. Whenever I went through the coding bootcamp that I did a couple years ago, after we finished going over raw CSS and how that works and the basics of that, they introduced us to Bootstrap as our first real CSS library to play with. That was fairly achievable for someone brand new to frontend work in general. What I was curious about is would you see these functional CSS libraries as equally accessible or maybe, more advanced than something like Bootstrap or Foundation. KRIS: I never thought about that before. I think I kind of see it like as the next step. I've worked on several products over the years where Bootstrap was what they started with. As we went on to build out the feature set of the product, we started to find ourselves battling with the various styles that we get at from Bootstrap for free. I think one of the things about working on a website, where actually using Bootstrap is that sometimes you can just kind of tell that it's a Bootstrap based website. I don't know if I'm making any sense to you guys -- DAVID: Oh, yeah. Definitely. KRIS: That was another reason why I wanted to have something... That had basically forced me to decide how my navigation bar looked. It forced me to think more about how my sidebar is looked or what my spacing scale -- it's the term I like to use -- based on what you're using for your pure margins, your paddings or what's your heights, things like that. It make me think about what those values should be and that's how you kind of break out of the box of having everything look like it was built in the same framework. I would say it's more of a next step if you're talking about your experience as a developer. Yeah, learn CSS. I think it's still incredibly valuable to know CSS and for me, like you David, I kind of got started and learned about this thing called Bootstrap and you can quote me. Back in the day, I said, "This is like the jQuery for CSS. It's amazing," but quickly, I kind of ran into some of those issues that I mentioned earlier and kind of force me think about, "Maybe, I don't want to use Bootstrap. Maybe, I want to handle my own thing," and so, I say functional CSS, as far as like again your education level, it's more or less that step of if you think about how do you want to solve certain problems in our design. DAVID: Okay. We've gone over functional CSS with some and styled-components, how would you choose between the two? KRIS: This is actually really hard because I actually like both approaches. I'm not one of those guys that's like, "That one sucks. I don't want to use it." I actually think both have their place and so, I know I kind of sound like the typical 'this versus that' blog post in technology right now but I will say that there's a few questions you should ask yourself and ask your team if you're considering changing up your approach to CSS. One of those is what is the preferred developer experience of your team. That's something we've been talking about a lot at the company. What we mean by that is how enjoyable is it to work in a particular codebase? Do your developers mind writing their CSS, instead of a JavaScript file? Or would they prefer to have something where they can actually read the HTML and build a C, just by looking at the elements in the classes that are being applied there? What this layout looks like or what's being applied to this layout? You know, questions like that can definitely help determine what direction you should go. Again like I said, one thing I love is that both approaches remove the developer's burden from having to come up with clever class names for all the different use cases or states of your components. I think you're going to get a win-win either way there. However, if you're using React, we already have a ton going on in our JSX and so, if adding a series of class names to determine your design tips is overwhelming for you, then maybe give styled-component a try because it can definitely simplify your JSX output a little bit. If the idea of having JS to handle your CSS seems a little fishy, I think one of the cons of using styled-components is that we're asking JavaScript to do yet another thing for us. If you're one of those people who has a problem with that, then give functional CSS a try. There's definitely more than one flavor out there and I've tried out almost all of them and like them. They all have their nice tidbits inside of them. I'll say, if you prefer to have easy theming, if you're working on a project that has to have a lot of theming involved, then I might be more toward styled-components. If design consistency and design patterns is a priority, then maybe functional CSS is an option for you. With both of those scenarios, easy theming in patterns is possible in both. It's just one is harder in one than the other. With styled-components, there are implementations of it in Vue and Ember but they don't seem to be very widely used yet, so I'd say use at your own risk or contribute to open source to make them better. I'd say for right now, if you don't mind being locked into using React, then styled-components is pretty cool. It also allows you to transfer your styled-components from one project to another without having to worry about. [inaudible] all the CSS because that's a big deal but I would ask myself those kind of questions to determine what solution I would choose. JEFFREY:: I am interested in the performance implications of these tools. You touch earlier on by using tools like this, you can definitely shrink your CSS payload by a lot, so what is the actual output of using these tools? What will it look like? Do you end up with still an external CSS file or is it a scenario where you're kind of getting some CSS injected into your markup? KRIS: Yes with one and no on the other. With functional CSS, you typically have a CSS file that is compiled and maybe, you're using Sass to do the operation for you or post-CSS, so you get an output CSS file that you help you fetch on the frontend of your application. Like I said, [inaudible] if these files are between 10 to 20 kilobytes, which is remarkably small. With styled-components, it's a totally different approach. It's a totally different implementation as well, unless you have a normalize or perhaps a reset CSS that you're fetching on the frontend, all the styles are actually -- I guess, I should start with earlier. What happens is you define your styles in your JavaScripts and at runtime, the component will read the styles that you set and then create a unique class for that instance into the head of your document. It's basically a single class use for that component and so, all in all, it gets compiled at runtime. There's no additional CSS files being injected. It's actually being created dynamically in your actual HTML document. One of the things I was first concerned about was what are the performance implications of having JavaScript do this for us and over the last couple of years, as the technology has been maturing, performance has been an issue but I think it was with V4 -- version 4 styled-components that just released, I want to say, in September or October, they have tremendous performance improvements. It's rapidly fast now, so it's not really something you need to be concerned about. I'd say again, it's one of those things: Can you get over the mental hurdle of having JavaScript do one more thing for you? And can you get used to the developer's ergonomic practice of writing CSS inside a JavaScript file? I know it kind of takes a little bit of getting over that because it just seems so strange but you also get a lot of perks by doing that with having easy access to theming, you get to use some of the power of JavaScript to help build out your CSS, which can be nice as well even if it kind of feels weird. JEFFREY:: Yeah, I agree with the ergonomics do feel weird but you'll get some benefits out of doing that. In the styled-components where it'd bring on the fly, I'm happy to kind of figured out the performance like runtime implications with that but it seems to me that now you no longer have a cache CSS and that you're having to get that all the time on the fly. Maybe for some systems, that is okay but that's definitely something that I would miss having -- the idea that CSS actually can be a little heavier because it will always be cached. KRIS: That is totally a valid point. I think one thing I do kind of like about the styled-component approach is that when you delete a component, all the styles go with it, so you no longer have orphaned styles just hanging out there that are getting shipped down to your user. I agree that the caching is definitely a concern but I just kind of start to weigh the pros and cons and again, you have to decide for your product, for your team what works best for you. Also, one of the pros that I would bring up is that you don't have any orphan CSS hanging out in your application anymore, which is definitely one of the key contributors to growing bloat in your files over time. JEFFREY:: That is very nice. KRIS: Yeah. DAVID: That ties into a question that I was kind of wondering. As your projects get larger and larger and more complex, your style sheets tend to grow along with that and one of the issues that you might have as time goes along is visual regressions like styling bugs. How does using functional CSS or styled-components sort of deal with that or prevent that? KRIS: With functional CSS, like I said, you have those patterns kind of built in out of the box. As long as you don't change the value of what a particular class renders out of, maybe someone goes in and without thinking of it, they change the second value of your spacing scale to a lot of pixels just because they felt they needed to. You're not going to have visual regression that happen over time because the whole premise of functional CSS, those values don't change. Those are basically meant to be seen as immutable values, immutable classes. You shouldn't have style regressions that pop up over time as people continue to work on your application. With styled-components you still have that risk unless you are setting all those same values and your theme properties that you can use within styled-components. Unless you're setting all your different spacing values, you have a tendency to potentially run the same issue. But actually, now that I think about it, maybe not because with styled-components, every component is unique. Every components styles are completely unique, so again, unless you're changing those core values of your themes, you should not have visual regression that happen in a totally separate part of your application when you're working on say, the home page. I would say both handle that pretty well, now that I'm thinking of it. DAVID: That's cool. KRIS: This reminds me, I'm working on a particular piece of software at a particular feature of my work and we don't use functional CSS or styled-components yet. I'm trying to get them to move over so again, this is all stuff I do on my free time but I'm working on editing a component and I had to make some pretty drastic changes to it and I was like, "How is this going to break on some other page that I'm not aware of yet," and so, this requires a lot more going back and checking all the various uses of this particular component. We used it in 10 or 11 different places on different pages and so, we have to go back to each and every single one of those use cases to make sure I didn't break something. Whereas if I was using one of these two approaches, it wouldn't be something that I would have to be concerned about. This is what I'm trying to push through on my own job as well. DAVID: Yeah, it really eat up your time. KRIS: Yeah. JEFFREY:: I want to talk for a few minutes about media queries because, I think that the situation is kind of handled. It's definitely something I've run into: CSS variables and with post-CSS tooling. I'm interested in kind of how does the media query story is handled with the setup. KRIS: With functional CSS, you tend to have, basically say, I have a class that sets a margin on the top of 16 pixels, for example. Let's say that that class name is MT-3... I don't know, meaning for like the third position in your spacing scale, 'MT' stands for margin top. If you wanted to only use that style for perhaps, like your medium break point, then different libraries have a different way to approach this, basically, you're 'MT-3--medium' or '--desktop.' It's the additional characters that you append to the class name to target that specific media query. Also, depending on the framework that you're using, you can also target pseudo states, whether or not this element is focused or on hover, maybe you want the background code to change. There's variations of those classes that allow you to target those specific states, which is really nice in functional CSS. Again, just by looking at the classes that you're adding to your DOM, you don't have to scroll through your style sidebar to figure out like what styles are associated with this class name and why it's being overwritten by these 10 other things that we have going on. Again, you can just look at the HTML and see what's happening and what should happen if you hover over an element. With styled-components, you can still use media queries and all that stuff right out of the box. In order to keep your media queries consistent, that way not everybody is having a handroll, at least by every single time that you need to use them. You can actually store your media queries in a separate JS files and it's kind of import them as you need them, which again, kind of feels like a weird developer ergonomic for your CSS but definitely, it comes in handy and definitely keeps your styling consistent the more you break out the various pieces into small pieces or small components like that. That's how to handle things like media queries and things like that. JEFFREY:: It's interesting how these approaches are kind of pushing the web towards, I'm feeling a lot more like native development, where you build your style and your visual look in conjunction with your component code and those things are not really ever separate. I think, there's some interesting influences coming in there from some made up things. KRIS: One thing I think about a lot, especially when I'm thinking about functional CSS is I'm reminded of this image I saw kind of posted on Twitter somewhere in the interwebs. It was a picture of the Super Mario Brothers start screen and the entire Super Mario Brothers video game was something like 23 kilobytes. This screen that you're looking at, which was like the start screen was like 345 kilobytes or something. My numbers are probably off but it was something drastic like that and this makes me think about, if you were to go back through, there's a couple of documentaries about how the loops that developers had to jump through in order to make old video games for Nintendo Entertainment System back in the late-80s -- I'm probably dating myself right now -- but the things they had to go through and to think about in order to make those games work the way they did is insane and I kind of feel like over the last... I don't know, maybe 10 or 15 years, because we've gotten such fast computers and such fast mobile devices and so on and so forth, we've kind of gotten crazy. I look at video games now and they're 100 gigabytes sometimes and I kind of feel like they didn't need to be 100 gigabyte video game and I think -- DAVID: But it's so much cooler because it is. KRIS: I'm not saying it's not cool but I'm think like if they really took the time to pay attention to some of these details, we probably could automate this a little better. Maybe, I'm just being biased but I think about we're kind of starting to do the same thing and talk about the same kind of problems in web development. Again, just because we can have a 745 kilobytes of CSS file and thought the world falling over, it doesn't mean we should, so maybe, let's take into consideration that not everybody is using a cinema display on a super-fast MacBook to view your application, what about the person who's using an outdated Android device that hasn't been updated in three or four years? How's your product going to work on their device? I'm really glad that people actually started to take that kind of thing into consideration and it even reminds me a little bit of another area that I think a lot about is the area of accessibility: how do people with various impairments or disabilities use your software and how can they use your software but also, outside of that, how do people, who maybe don't have the money to have a fancy mobile device like you have, use your product along those same veins. It kind of dips into the same vein of accessibility when we're talking about how do we optimize our products to make them usable by everyone despite their income level or their ability to have a fancy, nice desktop or a latest iPhone. JEFFREY:: Awesome. DAVID: Yeah. Definitely, you think about who are the majority of users on the internet these days and really, it's an emerging market with underpowered devices and the internet is becoming much more and more accessible for people to get on and browse, so you really got to keep those people in mind. KRIS: Definitely, especially if you're creating software that you want people across around the world to be using, a lot of people don't have access to crazy high speed internet like we do here in the States, so we should probably be taking more time to consider what it looks like for those people to use our product. At my company, what we've been talking about is like we should be testing and basically, starting mobile first with all of our development to see how does this work on our mobile device. I'm a remote developer so we're trying to figure out how to get me mobile devices that I can actually test on our VPN but it's something that we're trying to move towards at my company as well. DAVID: Okay, that about wraps up Episode 116, our final episode of 2018. Man, the year has gone by fast. We are the Frontside. We're the frontend specialists who make your largescale application projects run smoothly by helping your team assemble the right tools and implement the right automated processes. We can ensure your projects can move forward on time and without regressions while preserving quality and long term maintainability. If that's something you're interested in, please get in touch with us at @TheFrontside on Twitter or Contact@Frontside.io via email. Do send us any questions you might have, any topics you'd like to hear about in the future. We look forward to hearing from you. Thanks again to the wonderful Mandy Moore for producing today's podcast. You can find her on Twitter at @RubyRep and that's it.

In this internal episode, Charles and Wil talk about testing issues and BigTest solutions. Pieces of the testing story are discussed, such as the start and launch application, component setup and teardown, interacting with the application and component, convergent assertions, and network. Then they talk about testing issues: the fact that cross browser and device-simulated browsers are not good enough, maintainability and when and when not to DRY (RYE), slowness and why (acceptance) testing is slow, portability and why tests are coupled to the framework, and reliability. Finally, they talk about BigTest solutions: @bigtest/cli to start / launch (Karma recommended for now) @bigtest/react, @bigtest/vue, etc for setup & teardown @bigtest/interactor for interactions @bigtest/convergence for assertions @bigtest/network in the future (Mirage recommended for now) Resources: Justin Searls – Please don't mock me This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. Transcript: CHARLES: Hello, everybody and welcome to The Frontside Podcast, Episode 115. My name is Charles Lowell, this episode's host and a developer here at the Frontside. With me today to talk some shop is Mr Wil Wilsman. WIL: Hello. CHARLES: Hello, Wil. WIL: How's it going? CHARLES: It's going good. I'm actually pretty excited to get to jump into this topic because we're going to be talking about some of the big things that are happening at Frontside and some of the things that we've been developing in almost for the last year. WIL: Yeah. It's been about a year now. CHARLES: It's been about a year and we've talked about it in various podcast but we're going to be talking about it again because there's just been so much progress that we've made, I think in a lot of clarity in kind of what we're going for here when we talk about BigTest and testing big and how we want to roll out the BigTest framework. We just have a lot more experience using it on a number of different projects, so we get to talk about that today. Before we get started, I just wanted to talk a little bit about what BigTest is, both in terms of the framework and also the philosophy. Wil, you're the one who works the most on BigTest. When you think about philosophically, what does BigTest mean to you? WIL: It's the size of your test, not a physical size like size and storage but how much your task actually does. The test itself can be very small as our test are but it tests the whole application from the user interacting with it down to the network requests. That's the definition of the philosophy of a BigTest to me. It's to tests your application from the biggest point of view. CHARLES: Actually, achieving that can be surprisingly difficult, especially in a frontend JavaScript application and there are a lot of solutions out there for testing and we've talked about them. One of the questions that arises is when we talk about BigTest, what exactly are we talking about? Are we talking about a product that you can download and install? Are we talking about the philosophy that you just outlined? Or are we talking about the individual pieces of software that make that philosophy real? I think the answer is we're kind of talking about all three but we want to take this episode to talk about where we're going with the product. What we've identified is the subcomponent pieces of that product. In other words, in order to get started testing big, what are the things that you need to think about? What are the things that you need to do? And then what are the component pieces? Because one of the things that I think is very important to us is that you be able to arrive at wherever you are in your project, whatever framework you are using, whatever current testing solution and be able to begin using BigTest. That means, you might be using some of it or you might be using a lot of it but we want to meet you exactly where you are, so that you can then, get onboarded and start testing big. WIL: Yeah. Definitely an important distinction that we get confusion about is what is BigTests and people just assume like this whole test suite is BigTest but we used the parts of it ourselves like we use Mocha, which is not part of BigTest. We use Chai, which is not part of BigTest. We use Mirage which is kind of part of BigTest but definitely it originate in BigTest and Karma and things like that. BigTest isn't your testing suite. It's not one thing to go-to to grab, to start writing tests. It is a small pieces that you can use in conjunction with other small pieces, just to make it really easy and flexible to test your application. CHARLES: Exactly. Because it turns out that there's a lot going on in the application. Maybe we should talk about what some of those pieces are that you might want to start using BigTest with or that you might need to test big, I guess I should say. What's a good place to start? Let's start with talking about some of the issues that you want to do when your testing big. Then we can talk about what pieces of the testing story that fit in to solve those issues. One of them is you need to test that your application works, like actually works. That means you need to be able to test on a multiplicity of browsers, for example. We're limiting to the domain of web applications. There are actually a shockingly large number of browsers. It's not just Chrome. It's not just Safari. There's Mobile Chrome, Mobile Safari, which are subtly different. There's Edge and I'm sure the Mobile Edge is slightly different too, so you want to be able to test cross browser, right? WIL: Yeah, absolutely and things like Nightmare and JS DOM and things that simulated browsers, we don't necessarily think those are the best tools for writing BigTest because we want to ensure that those browser quirks are caught and tested as well. CHARLES: This is not theoretical like sometimes you'll have a syntax, like the parser is slightly different and you have something that throws a syntax error in Safari or in the Internet Explorer and your whole app is completely busted. If you just take in the time, just even trying to load the app in that browser, you would have caught that. That's what I've been on many times. WIL: Yeah and what I just saw came up yesterday, which comes up frequently is not closing your CSS Selector and Chrome doesn't really care like web to browsers don't care too much but that will fail in Edge and depending on what you're missing, the failing is part of that too but mostly, Firefox and Chrome don't care about that kind of thing. CHARLES: Right. It seems like the majority of testing solutions are kind of focused around Headless Chrome or some variation of Electron. That entire class of really dumb errors has already been caught. Like I said, to actually catch it, it takes less than a millisecond of CPU time just to load it onto the browser and see that thing doesn't work. Unfortunately, they can be catastrophic errors but the problem is how do you actually do. We want to test like cross browser. This is something that we want to do. For me, I just can't imagine shipping an application without having some form of cross browser testing, some capability of being able to say, "I want to test it," like, "We want to work on these eight browsers and so we're going to test it on these eight browsers," but how do you actually go about doing that? WIL: Right now, we are working on the BigTest CLI which will help us launch browsers but that's not complete yet. It has some bugs on. For the meantime we've been using Karma, which is great. Basically, you just have this service that's able to find the browser binary on the system and just launch them pointing to local hosts with your app loaded up and your normal development server take care of loading the test up and running the test. Karma and the BigTest CLI is just there to capture output and launch those separate browsers. CHARLES: Yeah. I remember when I was first using working with Karma and I think Testim is another tool that's in this space. There's Testim, Karma and BigTest actually is we're developing a launcher because launching is something that you're going to need but it's such a weird problem. I feel like with the browser launchers, there's three levels of inversion of control because you're starting a server that then starts another process, which then calls back to your server, which then loads the app resources, which then loads the tests and then runs the test. There's a lot of sleight of hand that has to happen and – WIL: Including injecting the adapter that you use, like the Mocha adapter, the Jasmine adapter that ends up reporting back to the CLI. That's something that Karma and Testim and BigTest will handle for you. CHARLES: Right, so you're fanning out the test suite to a suite of browsers then collecting the results but basically, you need some sort of agent living inside the browser that's going to act on behalf of the test suite, to collect the results. I remember when I first came into contact with Karma and Testim, I was like, "This is so unnecessarily complex," but then, having used it for a while and I think there are some complexity that can be removed but if you want to do cross browser testing, that kind of level of ping-ponging is there's a certain amount of it that just necessary. It's something that's actually quite complex that you need to have in your stack, in your toolbox, if you want to truly test big. WIL: Yeah and all the solutions is mechanisms for detecting when the browser has launched and restarting the browser based on its health check, etcetera and things like that that you wouldn't think of actually loading up a browser but you need to think of when you're doing automated testing. CHARLES: What is it that sets apart, for example the launcher solution? We kind of call this class of solutions launchers, so Testim, Karma, the BigTest CLI. What is it that sets BigTest CLI apart from say, Karma and Testim? WIL: We're trying to be as minimal config as possible and just really easy to get started and going. Karma has a lot of plugins that you need to make sure you have installed and loaded in the options set for those plugins. Testim has some stuff bundled but it still requires this big config bulk at the beginning that you need to passing or that's all what you were doing. We're trying to avoid that with BigTest CLI and one of the ways that we're able to avoid that is by just letting your Bundler handle bunding the test. In Karma, you need Karma webpack or something. Testim has some stuff that it needs and really, we just want like in-testing mode. When you're in the testing environment, just change your index to point your tests, instead of your application and your Bundler will do all the work and we just serve that file and collect the results. CHARLES: Right, so it doesn't matter if you're using Parcel or you're using webpack or you're using Ember CI. WIL: Yeah, Rollup even. CHARLES: Or even just like low level Broccoli or Gulp or whatever. There's a preponderance of bundling solutions and that was always something that was just a huge pain in the butt with Karma. I know it's like just getting to the point where my tests are loaded and you look with Testim, most of my experience come with Testim comes through how it's used in Ember CLI like the histrionics that undertaken just to bundle all your tests assets and your application assets and your vendor assets and just kind of bootstrap that thing. It's a lot of work. WIL: Another thing with BigTest CLI that doesn't include in Karma and Testim does is a concept of a watcher because all these Bundlers, you have HMR -- hot module reloading, Rollup and things like that. It come with plenty of plugins. Parcel is always set out of the box, so if you're using your Bundler, your existing Bundler to bundle your test, you get that watch feature for free, so it's another complexity that the BigTest CLI kind of eliminates. CHARLES: What it means is we've hidden most of that complexity. Just let the Bundler handle it, right? The Bundler is the part of your project that bundles. WIL: Yeah. CHARLES: You should have your launcher actually doing that for you but we still do need to have some way to do that set up and tear down. When we have that testing endpoint, we have some way to say, "We're starting a test, not the application. We're ending the test, tear it down," so how do you abstract that away? WIL: That's kind of something that we can't really avoid. It is just like some sort of dependency on the framework itself, your application framework. It's like you need to mount a React app. You need to mount an Ember app, etcetera and there's different ways to mount those things. This is one of the things that can't really be decoupled as much as everything else can but BigTest has BigTest React and BigTest Vue and we want to eventually gets like BigTest Ember but really, the main export of all these packages is just a simple mount helper that will mount and clean up your application for you and your testing hooks, whether you're using before each from Mocha or before from something else like Jasmine. You know, no matter what you're doing, you just have a hook that mounts your application and then, cleans it up on the next mount. CHARLES: It's worth pointing out here that this is kind of a core concern of testing and testing big is being able to mount your application and tear it down with regularity and having hooks into that process. Whether you're using BigTest or whether you're not, can you still use BigTest React and BigTest Vue, even if you weren't using anything else? WIL: Yeah, absolutely. Like I said, they just export simple mount helpers. I don't even think they have any other inner BigTest dependencies. They just have pure dependencies on their frameworks. CHARLES: Right and so, you could use it, even if you wanted to roll everything else by hand or you wanted to get started somehow and you needed to do set up and tear down, again this is something that's key to being able to test big, so you should be able to use it independently, whether you use the CLI or not, whether you're using any of the other tools or not. All of the tools can be used independently. WIL: Then another feature of the BigTest React and BigTest Vue is the tear down that happens before set up, rather than happening after your test runs, having a separate tear down. This allows it. Whether your test passes or fails, you can look at it and play with it and inspect it and debug it much easier than if you had tear down. You have to disable at tear down or throw a pause in there to keep other or something. CHARLES: Yeah, I love that. When something goes wrong, you can just let the test case run and the last test that it runs, it just leaves at set up. It does the tear down right before the set up. WIL: Exactly, yeah. At the very end of the whole test run, there's an app there waiting for you to play with. CHARLES: If you focus in on a single test, we most commonly use Mocha, so you say a '.only' to run that single focus test, then you have the state of the application at that test case set up and ready to go. You can just play with it, you can inspect it, you can actually just use it as a starting off point and interact with the app normally as you would. WIL: I want to say, Cypress does this too. They do their tear down before they're set up as well. That's how you're able to play with Cypress test. CHARLES: Yeah, I like that trick. Now, we talked about launching, setup and tear down but we haven't actually talked about much of what actually happens in the test cases themselves. We talked about how to start and launch your test suite, how to do that across a bunch of different browsers, how inside of that, you have a separate concern as applications set up and tear down and how you want to lean on how you're actual app is actually bundled because that fits in with the philosophy of testing big. You don't want to use an external Bundler for your test suite. You want to use your real Bundler, how the asset is actually going to look. But when it comes down to actually writing the tests, you need to be able to interact with at the highest level as you possibly can. When I say highest level, we want to verify that the users, when they take certain actions, we'll see certain outcomes and so, we want those outcomes and we already talked about this to be reflected in a real DOM, in a real browser. But at the same time, the real interactions, we want those to be as high fidelity as possible, so you want to be sending events to the browser. You want real mount events, real key events, real interactions. WIL: Yeah, interacting with application. That's another core philosophy that we kind of talked about earlier that defines a BigTest. It's the user interacting with your application. We're not calling methods and expecting other callbacks or arguments to be passed or clicking on a button and expecting a message to pop up that says, "Form submitted successfully." These are user-facing things were starting on and acting on. CHARLES: Yeah and then, it can be really tricky because these things don't happen synchronously. They're happening inside of your browser's event loop. I click that button and then it goes off and there's some loading state and then, I might get an error message that pops up this thing that animates out and then, goes away. The state of the browser is in constant flux. It's constantly changing and so, it can be very difficult to put your finger and say, I want to be in this state if you are limiting yourself to only reading from the DOM. Some frameworks, Ember for example, you have kind of a white box where you can actually inspect the state of the Ember run loop and use that to do some synchronization but it can be very, very hard to coordinate these interactions. WIL: Yeah. You know, to talk about getting to the solution as a BigTest interactor, which is basically modern page components or page objects. If you ever heard of page objects, it's just a way to encapsulate interacting with big pieces of your pages. It's not a new concept. It's been around for a while but BigTest interactor has kind of a new twist on it where they're immutable, composable interactions that are also convergent, which we'll get into later, which basically means if your buttons not there, it won't click the button until it is there. They're really powerful and they're making really easy and fun to write these tests. CHARLES: Yeah, they're super powerful. I remember we talked about convergences last time when we talked about BigTest but interactors, I think are definitely a new development. I think we should spend a little bit of time there talking about, not just the power but also the ergonomics of interactors because they are like page components or page objects, except they're scope to the component. Not only do they have all this wonderful stuff where it'll make sure that the component exists before it starts to interact with it and things like that but their composable. If I have a button, then there are certain operations that are valid for that button. I can click it. I can hover over it. I can do all these things. They're the operations that make it unique to the button. Now, those might actually map to real events. WIL: Similarly, their assertions about that button as well, like as primary is secondary. If this button is repeated throughout your application, you might want to make sure that your form has a primary and secondary button. CHARLES: Exactly. It really encapsulates all the knowledge of how you can interact with both in terms of taking action and reading state from that button. It almost feels like an accessibility API. It would be easy to write a screen reader if you had these interactors for every single component on the page. WIL: That's kind of what it is. It's just like you're defining an API around how your user would interact with your application and what your user would expect in the application. That's the point of page objects and interactors as you're defining this user API, essentially. CHARLES: Yeah and so, really the step that interactor take is that they take the classic page object and it make them composable, so I can have, you kind of touched on this before, a modal dialogue interactor, which is composed out of two button interactors. One for the primary action, one for the secondary action and maybe, it's aware of its own title text, so you can assert on the title text but I didn't actually have to write the individual button interactors for that modal dialog interactor. Then I might have a second modal dialog interactor or a form that's on a modal dialog just composed of the modal dialog interactors and the individual form components, which appear on that particular modal dialog. WIL: It's essentially how we've been building applications lately with components but this is for page objects in your test if you want to mirror that. You don't have to have one-to-one mappings of an interactor to a component but if you do, it's really powerful. CHARLES: Yeah. I found that when we have one-to-one interactors, that's when it just feels the best. WIL: Yeah and on top of this, if you have a component library and your component library exports the interactor that it uses for the component test, like we said, this BigTest technology, they're sprinkled also. We don't have to use interactors in big acceptance tests. We can use them for smaller component tests too, so if we ship these component interactors with the component library, your application that's consuming this component library now can test those components for free, without having to write their own interactors. It can just compose the interactors exported by the library. CHARLES: Man, I almost want you to repeat that word for word again, just so it can sink in. It's so awesome. Because when you actually go to write your tests, you're not starting from ground zero like, "How do I do this?" They're like, "I'm writing some tests for this thing and I'm using these components and so, I've already got the prepackaged interactions for those components." It's like you start writing your tests. If your tests are a 10-story building, it's like you're starting on Floor 7 and you only have to walk up to Floor 10, instead of slogging up all 10 stories. WIL: One really helpful interactor that we work within the open source stuff we've been working on is a date-picker interactor because date-pickers can be really complex. Just having that common interactor and have a date-picker on multiple forms where we can just use that one interactor, we don't have to tell every single test how to interact with that date-picker. We just say pick date and pass the date. CHARLES: Yeah, it's so awesome. That is actually a great example. It doesn't feel scary to write a test for a page that has a date-picker on it or two. If you're doing like a date range or something like that, you're like, "Oh, my God. I don't write the selectors to test this." You just import your date-picker interactor, you set the date, it actually worries about all the low level events and there you go. It feels like you're operating at a much higher level. WIL: Yeah. The interactor API essentially, you're telling me the test what the user would be doing and what the user would be seeing. CHARLES: Yeah. It's worth pointing out again. We've identified starting and launching. We've identified set up and tear down but interaction is a core concern of BigTesting, no matter what tool you're using. One of the things that we found as interactors are something that you can sprinkle on literally any test suite if you're testing an interface and it makes it better. We've used it inside big acceptance tests. We use it inside Jest, doing just little component tests. There are people in the BigTest community who have used it to basically, write component tests against a JS DOM and while theoretically, philosophically, you want to make those tests as big as you possibly can, you can use that piece in your test suite. If you are using a simulated DOM and if you're running a node in a browser, these interactors will still work and you're going to get high fidelity test cases that are resilient to this asynchrony and are composable and if they do have a full-fledged test suite, you can reuse these interactors. They are a really awesome power up that you can bring into your test suite. WIL: And they are not tied to the framework at all. We use them in React for our stuff but we've also written some in Ember. Robert's written some in the Vue and ported some test and one of the beautiful things we've seen from this is that one interactor goes everywhere. You just write the interactor once and you can use it in Ember, in React, in Vue, in those test suites. If the rest of your test suite is framework agnostic, you have this test suite that you can jump frameworks in your test suites until it works and can test your application with high fidelity. CHARLES: Yeah, it's fantastic. I remember when we first tried using interactors inside an Ember test suite because Ember comes with like a big kitchen sink in testing set up but interactors just slotted right in and there's absolutely no issue. WIL: Yeah and there is actually a speed boost even because in most of the Ember test offers a hook into the Ember run loop and interactors are not. There is actually a good speed boos just using interactors. CHARLES: Yeah. This is a good point. It's a good segue because typically, we think of acceptance tests as being really slow and one of the reasons, even the people [inaudible] acceptance tests or testing big as they think like it's going to take a long time. We found that actually we've been able to maintain a happy medium of testing big but also, having those test be really, really fast. When you say you said a speed boost from using interactors with Ember, where is that speed boost actually come from? WIL: I mentioned the Ember test offers a hook into the Ember run loop and interactors aren't and the reason of this is because interactors are converging and they wait for things in the DOM to exist before interacting with them. Instead of waiting for the framework to settle, it just waits for the thing to appear and then interacts with it immediately. If you're starting something about a button toward the top of the page, you don't really care that another button at the bottom of the page has rendered yet, unless of course you have assertion about that but if they're converging, you don't need to hook into the wrong loop to wait for the entire page to load, to interact with just one piece of it. CHARLES: Right. You're just waiting and you say, "I'm expecting something to happen and the moment I detect it, no matter what else is going on, the page could be taking 30 seconds to load but if that button appears and I can interact with it, I can take my action then or I can make my assertion then." It's about kind of removing gates -- artificial gates. WIL: Yeah. Another common thing that's helped with is animations as most test that are hooked into the run loop, you kind of have to wait for some of these animations to finish before you can even interact with the element and that means if a model has a half second animation where it flies in and you have 30 tests around this modal, those tests are extremely slow now because you have to wait for that modal to come in, whereas -- CHARLES: -- Straight up flaky. WIL: Yeah, straight up flaky. Whereas in the actual DOM, that modal is inserted pretty immediately and can be interacted with pretty immediately. With interactors, they don't need to wait for the animation to finish. They can just immediately interact with that modal but of course, if you need to wait for the animation to finish, there are options for that as well. CHARLES: Yeah. If there's some fade in that needs to happen, you can kind of assert on any state and as long as it's achieved at some point, the interactor will recognize it and recognize it at the soonest possible time that it possibly could. I remember getting bitten on one project where the modal animations in particular were so brutal. Not only were they flaky, they just were slow because there was all these manual time outs. It wasn't even a paper cut. It was kind of like a knife cut, like there's someone sitting there and kind of slashing you with a pocket knife. It just was a constant source of pain in your side. WIL: Yeah and that's how you end up with things like waits and sleeps in your test suite. When you need to wait for the animation to happen or something, you just see a sleep for four seconds with a comment because we have to wait for the components to load in. That's kind of a code now. CHARLES: Yeah, that's just asking for trouble both in terms of slowness and in terms of it's going to get flaky again. That has been kind of one the most freeing things about working with interactors and working with convergent assertions on which they're based is you just don't ever have to worry about asynchrony. Really, really truly, most of the time, you're writing your tests, like it's all synchronous and that kind of makes sense because from the user's perspective, their consciousness is synchronous and they don't care about the internal run loop. It's just they were making observations in serial and at some point, they're going to observe something, so the interactor sits at that point and really observes the application the way that your user would. WIL: Yeah. We've mentioned a few times now the convergent assertions, which interactors are based on. A little caveat there if you're using interactors and you're making non-convergent assertions, they might fail or be flaky. That's because interactors wait for the thing to be there to interact with, so as soon as the buttons there, it clicks it but it doesn't wait for after that event has fired and your application has reacted to that event, that's your application is concerned. We need something there like our convergent assertions that can converge on that state and wait for that state to be true before it considers itself passing or in times out. CHARLES: Maybe we should dig a little bit into convergent assertions. I think the last time we had a public conversation on the podcast about this, this is kind of where we were, like we hadn't built the interactors, we hadn't built these other component pieces of the testing story. We were really focused on the convergent assertion. We've talked a little bit about this but I think it's worth rehashing a little bit because it's a unique way of approaching the system but it's also kind of horrifying when you see how it works under the covers. I think when we tell people about the fact that it's basically polling underneath the covers. The timeout is configurable but it's basically polling every 10 milliseconds to observe a state. I remember the first time being confronted with this idea and I was horrified and like my programmer hackles on the back of my neck, like raised up and I was like, "Wait a minute. This is going to be slow. It's going to be computationally intensive." WIL: Yeah. That was my exact thought too because this is going to be slow. If acceptance tests are slow and we're doing an acceptance test every 10 milliseconds, it's going to be really slow and that's actually not the case completely. It's actually the opposite. They're extremely fast. CHARLES: It is shockingly fast. You've got to try it to believe how fast it is, how fast you can run acceptance tests. WIL: Yeah, talking like 100 tests in just tens of seconds. CHARLES: Right. You basically gated by how fast your framework can render. Your tests are not part of the slowness. Your test -- WIL: And also, memory leaks can be costly too. We experience that recently where we had memory leaks that were slowing down our test but we fixed those up in test and put our backup. CHARLES: Yeah, because basically, running the assertion or running the convergence is very fast. It's just a very light ping. I kind of think of it is as it is light as the brush of a photon or something that was bouncing off of a surface, so that you can observe it. It's extremely light and most of the time, it's just waiting so the test and the convergence really just gets out of the way. Just because they can run a thousand times or a hundred times in a second, it's doesn't gun it up. But the thing is it means that your tests run as fast as your application will run. You get back to the point... Was it in React where the kind of the key insight is that JavaScript is not the bottleneck? Well, your tests are not the bottleneck. WIL: Yeah. CHARLES: I guess this is what it is. I don't know if there's anything else that you want to say about convergences. WIL: No. We pretty much summed it up there and that's what interactors are based on. That's how they're able to wait for things in a DOM. It basically polls the DOM until it exists and then it moves on and actually does the interaction. CHARLES: Once again, this is actually a very low level thing on which BigTest is based but this is once again, something that you can use independently. You can write your own convergent assertions. You can write your own convergences that honestly have nothing to do with testing your assertions. It's a free standing library that you can use in your test suite or elsewhere should you choose. WIL: That doesn't need to be a DOM for BigTest convergence there. I use BigTest convergence in BigTest CLI to converge on the browser being launched. Instead of waiting for the browser to report that, I can just kind of poll and see how that process is doing and the convergence waits for that process to start before moving on. CHARLES: Right. I guess the best way I've thought about it it's a way to synchronize on observations and not on callbacks. It's a synchronization mechanism and 99% of the synchronization mechanisms that we're used to, they've involved some sort of callback, a promise, an event-listener, things like that or even a generator where control is handed back explicitly to a piece of code when something happens. Whereas, this is a fundamentally different synchronization primitive, where you are writing synchronous code that's based on observations, so what I observe this, do this. When I observe this, do this. It's extremely robust. WIL: Yeah, very. CHARLES: It is a core piece. A fundamental thing that on which interactors are based on, which the CLI is based, I don't know if it's core to writing tests but -- WIL: It definitely helps. CHARLES: It doesn't helps. We couldn't have BigTest interactor without that. WIL: No, definitely not. CHARLES: Because that's what makes it fast, that's what makes it not flaky at all and having those things, I think it makes it easy to maintain because you can work at the interactor level or the level of user interaction and you don't have to worry about synchronization, so the flow of your tests are very natural. WIL: Yeah. We don't have to explicitly wait for request to be done for making an assertion about your app. That'll just come with convergences, just waiting for test date in application to true. CHARLES: Let's talk about one more piece of the testing issue because when you're testing big, when you're testing in the browser, there's always the issue of what are you going to do about your API. You got to have your API running. It's just always an issue and this is kind of interesting because this sits at the crossroads of testing big and also, getting the most utility out of your test because in an ideal world, if you're testing really big, you're going to be using a real API. You're not going to poke holes in reality. WIL: Yeah. One of the things that we avoid in BigTest is poking holes. We're not shallow mounting the components and testing the methods and the results. We're fully mounting these things and fully interacting with them through the full DOM API. CHARLES: Yeah, exactly, using real browsers. It just occurred to me the irony of us talking about reality being things that are still running inside of a computer processor. I think we've inherited this term from that talk that Justin Searls at AssertJS in 2017. It's a really, really excellent talk. I think he gave it at RubyConf. It's the 'Don't mock me.' WIL: Yeah, it's one of my favorite talks. CHARLES: Yeah, it's a great talk. In it, he talks about the value of a test is a balance of how many holes you poke in reality and sometimes, you encounter a test where all it is like holes in reality. Whether you're mocking this, you're mocking that, you're mocking the DOM, you're mocking the browser, you're mocking your network layer, you're mocking this external API and the more holes you poke, the less useful it's going to be. Network is one of those where it can be very difficult to not poke holes in that reality because it's a huge part of your application. Your frontend application is how it's going to interact with the server but at the same time, servers are gigantic pieces of software themselves, each with their own dependencies, each with their own set up and tear down -- WIL: Have their own concerns. CHARLES: Yeah, exactly. They might be in a different language. They've got runtime, things like they might need external C libraries and crazy stuff like that. They're their own beast. To get a true big end-to-end test, you going to have to stand up your server but the problem that presents is you want your tests to be also isolatable. If you're a developer, I can go to a repo, I can do an install of my dependencies and I can run the tests without having to do any external dependencies other than the repository and the language in which I'm working. This is one where we kind of have tried to walk the line of not wanting to poke holes in reality but also, have the test be containable to the actual application. In order to do that, you need something that presents a high fidelity version of the network. You can kind of try and have your cake and eat it too. You want to have something that acts like a server and really acts like a server but it's actually not a server. WIL: And still poke as few holes as possible and the application and how that's all set up, we don't want to be intercepting methods and responding with fake data. That's not a good way to mock that network. CHARLES: Right. We want to be calling actual fetches, calling actual XMLHttpRequest. Ideally, if you've got service workers, making actual service worker requests. WIL: Basically, as far as the application is concerned, it's talking to a real server on us. CHARLES: Yeah and that's kind of the litmus test for is it a hole in reality or is it just a really great illusion? WIL: Yeah and that's a good name for Mirage, right? It's a really great illusion. CHARLES: Yeah. It is a simulation of reality, so we use Mirage, which is something from the Ember testing world but something that we have extracted and made available as BigTest Mirage. WIL: Yeah. The main difference just being is that we've taken away the Ember dependencies and the run loop stuff. It's just plain JavaScript Mirage. It works exactly the same as you use it in Ember minus the auto imports and the file... Oh, man. I can't think of that word. Aside from automatically importing your files for your server config, you have to do that manually because Ember is what provides that but other than that, it's a form of Mirage. You define models and serializers and factories and all the good stuff. CHARLES: Right and then you can use those factories and you can use those models to really give a high fidelity server. If you are building something in whatever framework, you can use BigTest Mirage to simulate that network layer. Again, we've used it in a number of different scenarios but having that in place means that you're going to be able to have those high fidelity tests where your application is actually making XMLHttpRequest but it's all isolatable, so that it can be run in repo. This isn't really related to testing but it has a fantastic capability where you can prepopulate, you can use the factories to prepopulate your server with data, so that you can use the application without the actual server being implemented. WIL: Yeah. That's extremely powerful. That's we were talking about earlier and getting at is the scenarios which are setting up specific, essentially fixtures but you're generating these fixtures. Factories are essentially high level fixtures, network of fixtures. CHARLES: Yeah, higher order of fixtures. WIL: Yeah, so the scenarios are just setting up these fixtures for a scenario of your applications, like the backend is down or the list only responds with two items as opposed to 5000 items, something like that. You want to be able to, not only test these things but be able to develop against it and Mirage makes that really easy because you can just start your app with Mirage-enabled point to that scenario and you're there. You have that exhausted scenario to develop in. CHARLES: If you've never used Mirage, it is really hard to understand just how incredibly powerful it can be. We've used it now on at least four projects, where we did develop the entire first version of the product without any backend whatsoever. It's an incredible product development tool, even apart from testing, that then informs the shape of what the API was going to be. I know we've talked about this on the podcast before but it's really an incredible technology and it is available to you no matter what framework you're using. I think it's one of the best kept secrets in JavaScript development. WIL: Yeah. That's definitely great. That said, though it does have some fallacies. It's great but it can be a little slow sometimes, so we are eventually working on a BigTest network like another piece of the BigTest pie that you'll be able to sprinkle into your application but in the meantime, praise Mirage. CHARLES: Yeah. We are going to be offering an alternative or maybe collaborating for another version of Mirage but hopefully, we can make Mirage faster, we will be able to make this thing faster, so that it can use service workers and be used in a bunch of different scenarios. Just to recap, we've talked about a lot of different components but over the past year, a couple of years, these are the things that we've identified as being really key components as big part of your acceptance testing and really your testing stack. How you're going to start and launch these things? How are you going to set them up and tear them down? How are you going to interact with the application from a user, both in terms of making assertions and how are you going to take action on behalf of the user and still have it be maintainable, have it be resistant to flakiness, have it be performant? BigTest is the answer to that for those particular areas of the testing story and so, some were using we're using existing components like we use Karma, we use Mirage to date. Those, we did not develop but where we see kind of key pieces of that puzzle missing is where we started writing the BigTest solutions so things like the interactor. Eventually, we are going to make BigTest into a product that's you're going to be able to use kind of out of the box, just like you might install Cypress, where it's a very quick set up and we make all of the decisions about the components for you. But in the meantime, we're really trying to take our time, identify those pieces of the puzzle and build the software component that fits that piece of the puzzle at the absolute best so when they're polished, use them in a more comprehensive product. Things like convergence, things like interactor, things like BigTest React, BigTest Vue and very soon, BigTest Ember. These are things that you can use today, to make your tests just that much bigger and that much better, especially interactor. It's been an incredible journey this past year as we kind of develop these individual pieces and there's just going to be more goodness to come. WIL: Absolutely. Right now, I'm working on some validation type API for interactor that I'm hoping to land soon. That'll open up the possibilities of maybe hiding away those convergent assertions a bit more in your tests and just handling this automatically. It'll be pretty good. CHARLES: It's really exciting. Writing test has got more and more easy and more and more fun over the last year for us. I think we're already starting in a pretty good place. If you have any questions about BigTest, how would folks get in touch with us? WIL: We have a BigTest Gitter channel. You can find a link to that on the BigTest website: BigTestJS.io. Just ask us questions on Gitter and we'll try to answer them. CHARLES: And as always, you can ask us directly. You can send email to Contact@Frontside.io or reach out to us on Twitter at @TheFrontside or you can actually reach out to the BigTestJS Twitter account directly and just call us on Twitter at @BigTestJS. Thank you very much, Wil. WIL: Thank you, Charles.

Guest: Dillon Kearns: @dillontkearns | GitHub | Incremental Elm In this episode, Dillon Kearns joins the show to talk about techniques for experimentation with Elm, making those experiments safe, the concept of mob programming, why you would want to experiment with Elm in the first place, and how you too can begin to experiment with Elm. Resources: Grant Maki's talk on experimenting in your team "Types Without Borders" by Dillon Kearns @ Elm Conf 2018 Dillon's Elm GraphQL library How Elm Code Tends Towards Simplicity by Dillon Kearns The CSS as ByteCode Talk by Richard Feldman This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. Transcript: CHARLES: Hello, everybody and welcome to The Frontside Podcast, Episode 114. My name is Charles Lowell. I'm a developer here at the Frontside. With me today as co-host on the show is David. Hello, David. DAVID: Hey, guys. CHARLES: David is also a developer here at Frontside and we are going to be talking about something that we've been talking, I guess a lot about recently and we're talking about Elm. I think we first started talking about this several years ago and then it kind of simmer down a little bit but recently, it's been top of tongue. With us to talk about Elm today is Dillon Kearns. Welcome Dillon. DILLON: Thank you so much for having me. CHARLES: I understand that you are a full time Elm consultant. You have a background as a Lean and Agile coach but have recently transitioned to doing Elm consulting full time. Now, what exactly does that mean in 2018 to be an Elm consultant. DILLON: Actually a lot of my motivation for getting into Elm consulting in the first place is I kind of realize that Elm to me is just an extension of the things that I was passionate about with Agile and software craftsmanship. I'm trying to help teams have a better experience with their code, make it more maintainable, make it easier to change, make it easier to drive things based on customer feedback and I really believe that Elm helps people do that. I used a lot of the background and experiences that I've had with Agile and Lean coaching and a lot of those same skills, in order to help organizations adopt Elm. One thing I've seen a lot of teams struggling with is trying out a lot of different frameworks. I've encountered teams that have spent months, very painfully trying out different frontend frameworks and having trouble coming to consensus about that. One of the things that I think really helps address that is having an experimental and iterative approach, that you can really use the scientific method to focus on learning, rather than getting it right the first time. I think that there's really a need to help teams through that process of introducing a new frontend framework like Elm, so that that's why I've gone into full time Elm consulting. CHARLES: That's an interesting process. It sounds like you really need to be constantly sending out spikes, doing research on whether it's Elm or some other technology to help you kind of bridge the chasm to the next generation. How do you actually do that as an organization? My guess, this kind of a question independent of Elm but maybe we can talk about how you see that play out in the context of Elm. DILLON: Right and actually, for any listeners interested in that question, I would really highly recommend Grant Maki's ElmConf talk from this year. He spoke about exactly that topic and it was at ElmConf that it's relevant whether your team is considering Elm or looking at other frameworks. I think that the key is you need to get good at experimenting in a way that's low risk and in a way that you can be constantly learning and seeing how these different technologies fit in your codebase and fit for your team. There's a quote that I really like from Woody Zuill. Have you guys heard of mob programming before? CHARLES: I heard of mob programming from a paper by Richard Garfield a long, long time ago, almost 20... I don't know if it's the same concept. DILLON: Yes. It gained a lot of momentum these days. Mob programming is essentially pair programming but with more people involved. I've really enjoyed that process actually. I think it's actually a great way to experiment with different technologies because you get all of the minds together and it's a very good way to kind of transfer knowledge and explore things together but Woody Zuill talks about mob programming and he likes to ask the question, "Why did we begin doing mob programming for the team at Hunter Industries that originally started mob programming?" People would give answers like, "Because it cuts out code review from the process because you have lots of eyeballs on it in real time," or, "Because it reduces bugs," or, "Because it gives you better quality code. It gives all the best ideas into the product in real time," and all those things are valid points that are really good benefits of mob programming. But he says those things may be true but actually, they're not why we tried mob programming. The reason we tried mob programming was because the team wanted to try it. That's a really important point. The team needs to be experimenting with things that they're passionate about and they need to be exploring things on their own terms. But with that said, another lesson from that story of kind of his team at Hunter Industries discovering mob programming is that the team didn't discover mob programming in a vacuum. Really, the team discovered mob programming because the team became really excellent at experimenting and evaluating those experiments and then, they like to talk about this phrase that Kent Beck coined, 'turn up the good.' When something is working well, we often focus on the negative things and trying to eliminate those things but what happens if we take the things that are working well and 'turn that dial up to 11.' CHARLES: Yeah, I love that. I remember in the kind of the original layout of extreme programming, talking about how I really just wanted to turn up all the things that were working for 11 or to 11, so testing, refactoring, incremental releases and things like that. DILLON: Exactly. CHARLES: I actually had one question that's maybe a little bit of a diversion. This is actually the first time I've heard of mob programming. It's definitely not the same sort of mob programming I learned about in Richard Garfield's paper. I think it was more referring to massively distributed open source in the form which is really kind of commonplace now that happens on GitHub. I think it's maybe, an obsolete definition of mob programming but how many people would be in a mob like two, three, four, five, six, seven, 10? DILLON: That's a great question. Really, the answer is of course, it depends. That's a consultant's favorite answer but it really does. My rule of thumb is I find it usually three people is a very nice size for a mob. I find that mobs tend towards around three or four people but that being said, it's important to note that mob programming is all about this idea that what is the true cost of programming. I think that often we look at programming as the act of writing code, initially and that's a very limited way of looking at coding. Because of course, 90% of our effort is spent maintaining code, making decisions around code, reproducing bugs, fixing bugs, communicating with customers about bugs -- bugs are extremely expensive -- the farther out they get, until eventually they get to the point of a customer discovering them, bugs are in extremely expensive part of software. If we can minimize bugs, that's very valuable. When you look at programming on this bigger scale and look at the bigger picture of programming, then you realize that you may be able to get one person to write the code faster but then, that person needs to code review it. That person needs to go and ask somebody question down the line when they don't have context because they weren't involved in the decision making. For example, maybe there's a UX person who doesn't have context on certain choices that were made, so there's a lot of churn, so you can kind of eliminate that churn by getting all the relevant people involved right away and that's the idea. In my experience with mob programming, it works really well to keep kind of a core of around three people. Sometimes, somebody goes up to have a conversation with somebody, take a break or answer somebody's question, maybe somebody from another team has a question that type of thing and so, the team can keep coding as a pair or whatever. But ultimately, the idea is that you get faster because you're building up this shared context and you're not spending as much time down the line answering questions, doing code reviews and things like that. CHARLES: Right. I see. DAVID: That kind of matches with my experience. Mob programming on previous teams, the way we had it set up is there was a regular mob programming chat session that the whole team was invited to but it was optional. You can just show up if you wanted to and really, that sort of made it so that there was a set of people who regularly attend -- three to five people in a session -- and they were the core group, essentially. DILLON: Right. That's another great point. Invitation is a powerful technique. If you're kind of mandating the people try an experiment or work in a certain way, ultimately it's much more powerful to let the team experiment on their own and follow their passion and they'll discover great things. It's about experimenting, rather than choosing specific experiments. While we're on this topic of kind of the real cost of coding, I think this is a good point to talk about this quality in Elm because, I think that this is one of the things that really motivates me to use Elm myself and introduce it to others is that, I think that Elm really get something about programming where there's a sort of superficial ease of certain techniques that Elm kind of goes beyond and says, "Actually, let's optimize for a different set of things that we think make code more maintainable and more delightful to work with in the long run." CHARLES: I wanted to also transition between, we were on a little diversions on mob programming but do you use mob programming as explicit technique for introducing Elm when a team is considering adopting it? DILLON: That's a fantastic question. I absolutely do. Of course, I honor the ways of working in a particular organization or team. I think that's important to do but I do strongly encourage using mobbing as a technique for knowledge sharing and when I'm on-site with a client, I find it extremely powerful as a technique for knowledge sharing and also, let's say you do an experiment, somebody is off in a corner and they're trying out Vue.js or they're trying out Elm or they're trying a particular coding technique. Then they come back to their team and they say, "Hey everybody, I tried this great thing," and now they have to spend this time convincing everybody and saying like, "Wait a minute. You didn't try this, you didn't try it that way. It wouldn't actually work in our context because of this." I think that it's very powerful to have everybody kind of involved in that process so that you can evaluate it together as a team. CHARLES: Because the thing is like, when you experience win or you experience fail, it's a very visceral feeling and that's the thing that sells you or turns you away. You can argue until you're blue in the face but words have a very limited capacity to convince, especially when compared with like physical and emotional feeling. It sounds like you can get everybody to have that shared experience, whether for the good or for the bad, you're going to arrive at a decision, orders of magnitude more quickly. They have to rely in conviction of that decision spread around the team. DILLON: Exactly. I think that hits the nail on the head and you say that we have this sort of skepticism of arguments from theoretical conversations, rather than 'show me the money,' but it's actually, try solving a real problem in this and that's exactly as it should be. I think that's one of the big antidote from this problem that I've seen in a lot of environments, where there's this analysis paralysis, especially with the state of the JavaScript ecosystem these days. I think that one of the keys to improving that situation is to get good at trying things, rather than theorizing about things. We have a tendency to want to theorize and when we do that, then we say, "Can it solve this problem? Can it solve this problem? Can it solve that problem?" You can talk about that until the cows come home but it doesn't get you anywhere and it doesn't really convince anybody of anything. The key is to find very small experiments and what I really recommend and what I'm dead focused on when I'm initially working with a client is getting something into production. Now, that doesn't mean that you need to have a road map for turning your entire application into Elm. In fact that's the whole point, is that you're not trying to do that. The point is you're trying to get as realistic of an experience as possible for what problems might occur if we do this? Will the team enjoy working with this language? Will it work well with our built pipeline? Will there be any unforeseen issues? You don't know until you actually try it, so you've got to try it and you've got to try it in tiny, tiny steps and low risk experiments. CHARLES: Right but you've got to try it for real. You don't want to try it with a TodoMVC. DILLON: Exactly. It needs to be meaningful, to really have a good understanding of what it's going to be like. CHARLES: I would say that I tend to agree but I've definitely encountered the counterargument and I also think this counterargument makes sense or perhaps where the pushback lies is if I'm constantly experimenting, then what I'm doing is I'm internally fragmenting my ecosystem and there is power in similarity. Any time you introduce something different, any time you introduce one fragment, you're introducing complexity -- a mental complexity -- like maybe I have to maintain my Elm app and I also have to have my Legacy... Or not Legacy, I've got my other JavaScript tool kit that does it in one way. Maybe I've got a couple of more because I've run these other experiments. I'm not saying that there is one way but there is power in uniformity. There is power in diversity. Where do you find the balance? DILLON: Those are all excellent points. To me, I think really the key is it's about the scientific method, you could say. The thing with the scientific method is that we often forget the last part. We get really good at hypothesizing about things. Sometimes people leave it at that, which we kind of just discussed. Sometimes, people go past the hypothesizing stage and they actually run the experiment and that's great. But then, the majority of people, if they get to that point, will forget to do the last step which is to evaluate the results. I think the key here is you need to be experimenting and this is what it means for it to be a low risk experiment. It means that you're not setting yourself off in a direction where you can't turn back. You want to set it up in such a way that you can turn away from it with minimal cost. One of the things that is really helpful for that is if you build a tiny, independent, little widget in your application, try building that in Elm. Some people will do that with a little sort of login badge in the corner of their application. One of the teams where I've introduced Elm at a Fortune 10 company, actually where we introduced Elm, we started out with just a tiny little table in one page and if we wanted to back that out, it would have been trivially easy but we decided that we wanted to go in further and invest more. CHARLES: That makes a lot of sense. Effectively, you need to have a Plan B. Don't sync all of the available time that you have to invest in an experiment. Make sure that you have a Plan B and if you need to do this widget or this table in Angular or React or Ember or whatever, you are thinking about that -- how would that work. DILLON: Exactly and the thing with experiments is the purpose of an experiment is not to build something. It's to learn. I really like this kind of ethos of lean startup, which I think is really getting much more into the mainstream in the software industry, which is a wonderful thing. The idea of lean startup, the kind of core concept is this idea of validated learning. Basically, in an environment where there's uncertainty, which is pretty much most of the things you're doing in software, the main goal is you're not shipping a product like you would be if you're trying to manufacture cars as quickly as possible. The main thing that you're producing is what they call 'validated learning' and so, you want to minimize the amount of time it takes to validate or invalidate your assumptions about something and then, you want to make it as cheap as possible to move on from that. CHARLES: I like that. So if you're going to organize your development process around this principle or maybe not organize it but integrate it into development process, how do you know that you're conducting a healthy number of experiments, versus I may be conducting too many experiments? Is there a metric that you can look at? We need to have this many experiments running at all times or this is just too many or something else. DILLON: That's a really interesting question. I think I would tend to think about that more again, as looking at the way the experiments are run, rather than 'are there too many experiments?' That's just not a problem that I've seen there being too many experiments. The pain that we tend to really see in environments where experiments are hurting teams is the way the experiments are being done. It's hard to backtrack from those experiments and as you were saying before, you kind of put yourself down this path where you can't walk it back and you create this sort of rift in the way the code is being written, which makes it more difficult to work in that codebase. The thing with experiments is they can have really big payoff. Now, you want to make sure that you're not just going in and picking up every shiny object that you see. One thing that can keep you honest with that is if you're kind of coming up with a hypothesis before you start. If you're saying, "This is the value to our business and to our team if we attempt this thing and this is what will prove that it seems to have that value and this is what will tell us, 'Actually, it doesn't have that value and we should drop it and cut our losses.'" CHARLES: That's a great heuristic. As you're saying and imagining how that might have saved my bacon in the past because I've definitely made the mistake of playing with too many shiny objects and picking things because I didn't fully evaluate what I thought the value. I was explicit with myself about what is the value that is going to bring to this project or this business. I have a theory about it but I am not thinking what is my hypothesis and how am I going to validate or invalidate? I'm thinking, I've got a short term pain that I'm experiencing and I'm grasping for this thing, which I think will solve it and I'm not properly evaluating how it's going to affect me long term. DILLON: Right and that could be a great team practice to play around with is often, teams will kind of come up with action items out of retrospectives. One thing that I think can be really beneficial for teams is to kind of flip that notion of doing action items which again, it's really just doing the middle part of the experiment where you're conducting the test but you cut out the hypothesis part and the evaluation part. Try to bring that into your team's retrospective and try to have explicit hypotheses in the retrospectives and then, in the next retrospective, evaluate the results. CHARLES: All right. I will definitely keep that in mind but this feels like a fresh take on kind of how you manage software development that I haven't encountered too much, being more scientific about it. It sounds like science-oriented development. DILLON: Right. DAVID: I like that. DILLON: There are a lot of buzzwords these days in software development, in general and it's really becoming a problem, I think in the Agile community but really, what it boils down to is these basic elements and basing decisions on feedback is one of those fundamental unit. You can call the scientific method, you can call it lean startup and validated learning, you can call it agile, you can call it whatever you want but ultimately, you need to be basing things on feedback. I think of it almost like our nerves. There's actually a disorder that some people have, which can be fatal, which is that their nerves don't tell them when they're feeling pain. I think this is a great analogy for software because that can happen to companies too. They don't feel the pain of certain decisions not landing well. Because they're not getting feedback from users, they're not getting feedback from metrics and recording, they're not getting feedback from doing that final evaluation step of their experiments, so when you fall on the ground, a small cut could be extremely harmful because you don't know the damage it's doing to you. CHARLES: I think that is a good analogy. One of the things that I'm curious about is we've been discussing a lot of techniques for experimentation and how you can integrate that into your process and how you can make your experiments safe, so let's talk a little bit about -- first of all, two things -- why would I want to experiment with Elm in the first place? Because ultimately, that's why we're here and why we're having this conversation. What's compelling about it that would make me want to experiment? And then how can I begin to experiment with Elm? DILLON: I actually just published a blog post yesterday. It's called 'How Elm Code Tends Towards Simplicity.' To your question of why would a team consider Elm, I kind of talk a little bit in this blog post about a case study at a Fortune 10 company where I introduced Elm to a few of the teams there. One of the teams there, we had actually seen an Angular project that they had worked on and often, in an enterprise environment, you have projects moving from one team to another. I actually had my hands on this Angular project. It kind of moved over to another team and we were experiencing some major pain trying to make changes in this codebase. Even making the simplest change, we were finding that there were a lot of bugs that would be introduced because there's some global variable. There's some implicit state. Sometimes, it was even reaching in and tweaking the DOM and really, the topic of conversation at our team lunches was how afraid we were to touch this codebase. Fast forward a few months and this team was asking my advice on picking a new frontend framework and I introduced them to Elm. They took a run with it and it was pretty remarkable to see this same team that had really struggled with AngularJS and they didn't really have a strong sense of what were the best practices. They weren't getting any guidance from the framework itself and the tooling around it and they actually loved the experience of working with Elm because they were saying, "This is amazing. Maybe it takes a little time to figure out how to solve a particular problem on Elm but once we do, we know that we've done it in a solid way." One of the things that I think is most powerful about Elm is that it keeps you from shooting yourself in the foot. I think that's a really good headline kind of summary of what I love about Elm. For example, tweaking the DOM. Now, it might seem like a pretty obvious thing that we just won't tweak the DOM and that's fair enough. That might not be a problem for a lot of teams. People wouldn't even reach for that technique because they're disciplined about it. But at a certain point, you start taking on enough things and then go from kind of those basic things that are going to make your code more unreliable and unsafe like tweaking the DOM and you start getting into the realm of best practices. There's so much discussion these days in the JavaScript community about best practices, which is great. It's great to discuss that but my concern is that there's a new best practice each week and the team has to agree on it, you have to find techniques for enforcing it, people have to make sure that these best practices are being followed in code reviews. Then when you look at a given piece of code, you have to trust that those best practices are being followed, so it requires a lot of work to make sure, in your reducers, in redux that you are not mutating anything. With Elm, data is just immutable. That's just how it is. There are a lot of these kind of things that are baked into the language and the expressivity of the type system allows you to bake in your own constraints. One of the things that I find really compelling about Elm is its design really prevents you from shooting yourself in the foot and it gives you tools for making sure that you take it even a step further and it helps you enforce these best practices at a compiler level. CHARLES: Now what's interesting here is it's almost like the opposite tension of experimentation is a work, right? like here, we have an example of uniformity being the more powerful track but then inside the actual macroscopic process, you want a lot of experimentation and diversity. But at the microscopic level, inside your application, it sounds like you want less experimentation and you derive a lot of strength from that but -- DILLON: That's a great point. CHARLES: -- Experiments that are possible, yeah. DILLON: I think that there is a lot of pain these days in the JavaScript community. We hear people talking often about JavaScript fatigue and it's a real thing. It takes a lot of work to stay on top of the latest best practices and frameworks and that can be a lot of fun. I love learning about the latest new frameworks and tooling but ultimately as you're saying, we don't want that experimentation so much about the fundamentals. We want some dependable, solid fundamentals and then we want the experimentation to happen within there. I think that's exactly what we see in the Elm ecosystem. We have a single kind of data store or way of managing state in Elm. It's called the Elm Architecture. In fact, it's what Redux is based on and it worked extremely well and you don't have to experiment with different data stores in Elm because that's just what Elm code looks like. Now, if you want to experiment in Elm, then there is a lot of innovation happening. One of my favorite things about Elm is that the compiler and its expressiveness has sparked a lot of creativity. One of my favorite things about Elm is the library called Elm UI. Actually, a client that I'm working with right now, it's a really interesting case study. They are kind of a very small startup. They just kind of branched off of a larger startup. They're building some tooling for this ecosystem. They were engineers at a company called Procore that does cloud document management for construction companies. They wanted to get a product-ready for a big conference for their potential clients. The reason they brought me in to help them was because they wanted to reach this ambitious target of being able to do a demo of this brand new product at this conference and they wanted to iterate very quickly. One of the things that really drew them into Elm in the first place is this library Elm UI. Elm UI essentially, Richard Feldman gives a talk on it, where he uses the analogy of it being treating CSS and HTML as bytecode for your views. I think that's a really apt way to put it. If you break down this idea of CSS -- Cascading Style Sheets, it removes the cascading part of CSS and it removes the sheets part of CSS. What you're left with is a way of expressing style and it's a way of expressing style that is able to part ways with all of the baggage of the entire history of backwards compatible decisions that CSS has ever made. If you want to vertically aligned something, then you just say, "Align vertically," you know, center vertically. If you want to center something horizontally, you say center X. It creates a high level language for expressing views. My experience with Elm UI, this may not be the right choice for every team but I love it. I use it on all of the projects that I maintain personally. I love using it because it gives you that same sense of invincibility refactoring that you get with Elm, which is remarkable that you could have that feeling with managing views. CHARLES: It's definitely something that feels like a dark art and it can't be called science. It's an art. It's a science for some people but it's historically been a dark art and something fiddly to work with. In terms of being able to make the experiment with Elm, when we talked a little bit about why you might want to experiment with it in the first place, what the business case is, I guess my next question is or a question that immediately comes to mind is supposing that we have decided to experiment with this, how do you mitigate that experiment? We talked about lowering the cost, having a way to turn away from it, having a way to make it inexpensive. For example, one of the things that I think of when evaluating a new technology is how well can I use it with old technologies. I have a lot about best practices in my tool bag. We all do. We got our all favorite libraries and pathways that are just familiar to us. One of the things that I've noticed is when adopting a new technology, one of the things that makes it easy to experiment with is how well it works with the existing technologies. I know that, we talked about Elm UI, kind of rethinking style in CSS and your views and Elm itself as a completely different language within JavaScript, that can be both liberating but it can also be limiting in the sense that I can't reach back for my existing tool if no tool exists in this new space. The kind of experience that I've had where this is really worked is systems like JRuby or Clojure, where there's a very clear pathway to be able to use Java libraries from those environments, so you always have kind of an escape hatch. What's that like in Elm? DILLON: This is a really interesting conversation because it highlights, in some ways some of the most defining features of Elm. In terms of how do you kind of pull Elm into an existing application, there are a lot of different techniques for that. It's pretty straightforward to create a little Elm app. We usually don't call them components for reasons that we can get into if we want to but that's a whole can of worms. But if you've got a little Elm application that you want to use to render a widget on your page, then it's as simple as just calling Elm.yourmodule.init and rendering it onto the page there. That's quite straightforward and if you want to interface with your existing code there are several ways to do that. There's something called port in Elm, which is how you kind of communicate by sending these messages and data back and forth between your Elm app and JavaScript. Now, this is one of the decisions, I think that defines Elm as the language and the reason this is important is because Elm decided not to make the choice that a lot of other compile to JS languages do. For example, if you look at ReasonML or PureScript or a more extreme example, TypeScript. TypeScript is a superset of JavaScript, so it's trying to allow you to gradually introduce this to get some incremental improvements for your JavaScript code, so it's extremely easy to experiment with it, which we've talked about the importance of experimentation. Now, the challenge with this technique, the tradeoff here is it's great, that it then becomes very easy to transition into it and that's an excellent strategy for the goals of TypeScript. Elm has a different set of goals, so the things that elm is focused on giving you is a truly type-safe experience. When you're working with Elm, if your Elm code says that this data is a float, then it is a float. Either, it is a float or that code is not being run and so, that's very different than the experience in TypeScript where you have these escape hatches. This is an inevitable choice for any compiled to JS language. Are you going to have escape hatches or not? Elm is really the only language out there, I think that chooses not to have escape hatches and that is actually the thing that that I love about the language because that's the only way you can truly have guarantees, rather than, "Yeah, I'm pretty sure that these type guarantees hold." DAVID: Yeah, wishes and dreams. DILLON: Yeah. CHARLES: What does it mean to have no escape hatches? because you talked about ports. Does it mean like it's impossible to use an external JavaScript library? DILLON: That's an excellent question. You absolutely can use JavaScript libraries. It means that it's being explicit and upfront about the fact that there's uncertainty in these areas. That's what it comes down to. Take for example dealing with JSON. In a JavaScript application, what we get when we're dealing with JSON is you make a request up to the server, you have some callback that passes in the data you get back and then you start pulling bits and pieces off of it and you say 'response.users subzero.firstname' and you hope that none of those things are null, none of those types are different than what you expected. In a way, it's kind of letting you pretend that you have certainty there when in fact, you don't and with Elm, the approach is quite different. You have to explicitly say, "I expect my response to have this shape. I expect it to be a list of things, which have a first name and last name which are strings," and then Elm says, "Okay, great. I'm going to check your assumptions," and if you're right, then here you go and you're in a well typed-space where you know exactly what the types are and if you're wrong, then that's just another type of data, so it's just a case statement where you say, "If my assumptions were correct, then do something and if my assumptions were incorrect, then you decide what to do from there." CHARLES: Right. For me, it sounds like there is some way because ultimately, I'm going to be getting unstructured but I'm going to be getting JSON back from the server and maybe, I have some library that's going to be doing that for me and enhancing it and adding value to that JSON in some way. But then at some point, I can present it to Elm but what you just saying is I need to be complete in making sure that I handle each case. I need to do or handle the case. Explicit about saying if the assumptions that Elm wants to make, turn out to not be true, Elm is going to make me handle the case where those assumptions were not true. DILLON: Exactly. I think that TypeScript of any type is the perfect illustration of the difference. TypeScript of any type is sort of allowing you to say, "Don't type check this. Trust me here," and Elm's approach is more kind of just be explicit about what you want me to do if your assumptions are incorrect. It doesn't let you kind of come in and say, "No, I know I'm going to be right here." CHARLES: Right but there is a way to pass data structures back and forth. DILLON: There absolutely is and actually, there's a technique that's starting to gain some traction now, which I'm really excited about, which is rather than using this sort of JavaScript interrupt technique we talked about, which is again, it's very much like communicating with a server where you're kind of sending messages and getting data back -- getting these messages with data back. But there's an alternative to that which is using web components. Actually, there's quite good support for assuming that you don't need to be compatible with Internet Explorer. Basically in a nutshell, if you can wrap a sort of declarative web component around anything, it could be a Google Maps API, it could be a syntax highlighting JavaScript library, something that you don't have an Elm library for but you want to use this JavaScript library, it's actually quite a nice experience. You just render that custom element using your Elm code just as you would any other HTML in Elm. CHARLES: Yeah I like that, so the HTML becomes the canvas or composition with other JavaScript and the semantics are very well-defined and that interface is actually pretty thin. DILLON: Exactly and the key again is that you wanted to find a declarative interface, rather than an imperative one where you're kind of just doing a series of statements where you say, "Do this and then set this value and then call this and then set this call back." Instead, you're saying, "Render this Google Maps custom element," which is centered around these coordinates and has this zoom level on. You declaratively give it the bit of information that it needs to render a particular view. CHARLES: Okay. Then I guess the final question that I have around this area is about being able to integrate existing tools and functions inside of an Elm application. Because it sounds like you could theoretically develop large parts of your application, is there a way that you can actually have other areas of your application that are not currently invested in Elm still benefit from it, in the sense for kind of need of JavaScript APIs that Elm can make available. DILLON: Right, so you're kind of talking about the reverse of that Elm reaching out to JavaScript. You're asking about, can JavaScript reach out to Elm and benefit from some of its ecosystem? CHARLES: Exactly. I say that is that another potential vector for experimentation. DILLON: It's a really interesting thought. I haven't given it too much thought, to be honest but I actually have heard it come up before and my gut feeling is that it's probably more fruitful to explore the inverse, reaching out to JavaScript from Elm and the reason is kind of the main appeal of Elm is that when you're operating within Elm, you have this sense that if it compiles, it works. Because again, this central decision to not allow escape hatches is what allows you to have that sort of robustness, so you have this feeling of bullet proof refactorings and adding new features seamlessly where you change your data modeling to say, "Here's this other case that can be represented," and then suddenly, the Elm compiler says, "Tell me what to do here, tell me what to do here and tell me what to do there," and you do it and your app is working. That's the real appeal of Elm, I think and you don't really get much of that by just calling out to an Elm library from within JavaScript. That's my gut feeling on it. CHARLES: Okay, that's fair enough. On the subject of interrupt and using tools like JSON, you actually maintain a GraphQL library for Elm. You probably have a lot of experience on this. Maybe we can talk about that as a concrete case that highlights the examples. DILLON: Yeah. I think to me this is one of the things that really highlights the power of Elm, to give you a really amazing refactoring and kind of feature creating experience. A lot of Elm libraries are prefaced by the author name, so it's still DillonKearns/ElmGraphQL. I spoke about it this year at ElmConf. In a nutshell, what it does is it actually generates code based on your GraphQL schema. For anyone who doesn't know, GraphQL is just kind of a language for expressing the shape of your API and what types of data can return. What DillonKearns on GraphQL does is it looks at your GraphQL schema and it generates an API that allows you to query that API. using this library, you can actually guarantee that you're making a valid query to your server. Again, you get this bulletproof experience of refactoring in Elm where you can do something like make a change in your API and recompile your Elm code and see whether you've made a backwards incompatible change. All of this effort of doing sort of this JSON decoding I was talking about earlier where you kind of have to explicitly say, "These are my assumptions about the shape of the JSON that I'm getting back." When you're using this library, you no longer need to make any assumptions because you're able to rely on this sort of schema of your API and so you know when you're requesting this data, you don't have to run it, see if it works and then tweak it and run it again -- this sort of cycle of checking your assumptions at runtime. It moves those assumptions that you're making from runtime to compile time and it can tell you when you compile your application, it can say, "Actually, this data you're requesting, it doesn't exist," or, "It's actually called this," or, "This is actually the type of the data." CHARLES: Right. I love that. How do you do that? Because it seems like you've got a little bit of a chicken and the egg problem because the schema is defined outside of Elm, so you have to be able to parse and understand the schema in order to generate the Elm types to be able to compile Elm code against them. Maybe I'm not -- DILLON: That's exactly right. That's exactly what it does. Now, the nice thing is that GraphQL is really designed for these types of use cases. It supports them in a first class way. If you have a GraphQL API, that means you have built into it whether you know about it or not, a way to introspect the schema. All of the queries for kind of interrogating that GraphQL server and asking what types of data does this return, what are all your queries that I can run, it's built into it by the framework, so that comes for free. Getting up and running with this package I built is as simple as running a little npm CLI, pointing it to either your URL for your server or the JSON form of your schema, if you prefer and then, it generates the code for you. CHARLES: Wow, that sounds fantastic. This is the exact kind of thing that feels like it would be cool if I could just start using this library to manage the GraphQL of my application but I'm consuming that GraphQL from other JavaScript but it's the Elm code that's managing it. Do you see what I mean? DILLON: I hadn't considered that. I guess you could. You're right. Maybe I'm so smitten with Elm that it's hard to see an in-between but I guess, you could get some benefits from that approach. CHARLES: Right and as an experiment of course. DILLON: There you go. There you go. CHARLES: All right. With that, I think we'll wrap it up. Thank you so much, Dillon for coming on and talking with us on the podcast. DILLON: My pleasure. I really enjoyed the conversation. CHARLES: I actually got so many great tidbits from so many different areas of software development in Elm but also, just in kind of other things that I'm interested in trying. It was a really great conversation. DILLON: I had a lot of fun and I love discussing these things. For any listeners who are interested in this stuff, feel free to reach out to me on the Elm Slack or on Twitter. I'm at @DillonTKearns. I'm also offering a free intro Elm talk for any companies that are kind of entertaining the idea of doing an experiment with Elm. If you go to IncrementalElm.com/Intro, you can find out about some of the talks that I'm offering. CHARLES: All right. Well, thank you very much and we, as always are the Frontside. We build software that you can stake your future on and you can get in touch with us at @TheFrontside on Twitter or Contact@Frontside.io on email. Please send us any questions you might have, any topics that you'd like to hear about and we look forward to hearing from you and we will see you next week.

Guest: Philip Poots: GitHub | ClubCollect Previous Episode: 056: Ember vs. Elm: The Showdown with Philip Poots In this episode, Philip Poots joins the show again to talk about the beauty of simplicity, the simplicity and similarities between Elm and Ruby programming languages, whether Elixir is a distant cousin of the two, the complexity of Ember and JavaScript ecosystems (Ember helps, but is fighting a losing battle), static vs. dynamic, the ease of Rails (productivity), and the promise of Ember (productivity, convention). The panel also talks about the definition of "quality", making code long-term maintainable, and determining what is good vs. what is bad for your codebase. Resources: Michel Martens mote Learn the Elm Programming Language and Build Error-Free Apps with Richard Feldman Worse is Better: Richard P. Gabriel Gary Bernhardt's Destroy All Software Screencasts Zen and the Art of Motorcycle Maintenance: An Inquiry into Values The Calm Company It Doesn't Have to Be Crazy at Work This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. Transcript: CHARLES:: Hello, everybody and welcome to The Frontside Podcast, Episode 113. My name is Charles Lowell. I'm a developer here at the Frontside and with me today are Taras Mankovski and David Keathley. Hello? DAVID:: Hey, guys. TARAS: Hello, hello. CHARLES:: And we're going to be talking with a serial guest on our serial podcast, Mr Philip Poots, who is the VP of engineering at ClubCollect. Welcome, Philip. PHILIP: Hey, guys. Thanks for having me on. CHARLES:: Yeah. I'm actually excited to have you on. We've had you on a couple of times before. We've been trying to get you on the podcast, I think for about a year, to talk about I think what has kind of a unique story in programming these days. The prevailing narrative is that folks start off with some language that's dynamically typed and object oriented and then at some point, they discover functional programming and then at some point, they discover static programming and they march off into a promised land of Nirvana and no bugs ever, ever happening again. It seems like it's pretty much a straight line from that point to the next point and passing through those way stations. When I talk to you, I guess... Gosh, I think you were the first person that really introduced me to Elm back at Wicked Good Ember in 2016 and it seemed like you were kind of following that arc but actually, that was a bit deceptive because then the next time I talked to you, you were saying, "No, man. I'm really into Ruby and kind of diving in and trying to get into Ruby again," and I was kind of like, "Record scratch." You're kind of jumping around the points. You're not following the preordained story arc. What is going on here? I just kind of wanted to have a conversation about that and find out what the deal was and then, what's kind have guided your journey. PHILIP: There was one event and that was ElmConf Europe, which was a fantastic conference. Really, one of the best conferences I've been to, just because I guess with the nature of early language, small conference environment. There's just a lot of things happening. There's a lot of people. Evan was there, Richard Feldman was there, the leading lights of the Elm community were there and it was fantastic. But I guess, one thing that people have always said to me is the whole way track is the best track of the conference and it's not something I really appreciated before and during the breaks, I ended up talking to a guy called Michel Martens. He is the finder of a Redis sourcing company and I guess, this was just a revelation to me. He was interested in Elm. He was friends with the guys that organized the conference and we got talking and he was like, "I do this in Ruby. I do this in Ruby. I did this in Ruby," and I was like, "What?" and he was like, "Yeah, yeah, yeah." He's a really, really humble guy but as soon as I got home, I checked him out. His GitHub is 'soveran' and it turns out he's written... I don't know, how many gems in Ruby, all with really well-chosen names, very short, very clear, very detailed. The best thing about his libraries is you can print them out on paper. What I mean by that is they were tiny. They were so small and I guess, I just never seen that before. I go into Ruby on Rails -- that was my first exposure to programming, that was my first exposure to everything -- unlike with Rails, often when you hit problems, you'd start to dive a bit deeper and ultimately, you dive so deep that you sunk essentially and you just accepted, "Okay, I'm not going to bend the framework that way this time. Let's figure out how everyone else goes with the framework and do that." Then with Ember when I moved into frontend, that was a similar thing. There were so many layers of complexity that I never felt like had a real handle on it. I kind of just thought this was the way things were. I thought it's always going to be complex. That's just the nature of the problem. That's just the problem they're trying to solve. It's a complex problem and therefore, that complexity is necessary. But it was Elm that taught me, I think that choosing the right primitives and thinking very carefully about the problem can actually give you something that's quite simple but incredibly powerful. Not only something quite simple but something so simple that it can fit inside your head, like this concept of a program fitting inside your head and Rails, I don't know how many heads I need to fit Rails in or Ember for that matter and believe me, I tried it but with Elm, there was that simplicity. When I came across this Ruby, a language I was very familiar with but this Ruby that I had never seen before, a clear example was a templating library and he calls it 'mote' and it's including comments. It's under a hundred lines of code and it does everything you would need to. Sure, there were one or two edge cases that it doesn't cover but it's like, "Let's use the trade off." It almost feels like [inaudible] because he was always a big believer in "You ain't going to need it. Let's go for that 80% win with 20% effort," and this was like that taken to the extreme. CHARLES:: I'm just curious, just to kind of put a fine point on it, it sounds like there might be more in common, like a deeper camaraderie between this style of Ruby and the style encouraged by Elm, even though that on the surface, one is a dynamically typed object oriented language and the other is a statically typed functional language and yet, there's a deeper philosophical alignment that seems like it's invisible to 99% of the discussion that happens around these languages. PHILIP: Yeah, I think so. I think the categories we and this is something Richard Feldman talks. He's a member of the Elm community. He does a lot of talks and has a course also in Frontend Masters, which I highly recommend. But he often talks about the frame of the conversation is wrong because you have good statically typed languages and you have bad statically typed languages. You have good dynamic languages and you have bad dynamic languages. For all interpretations of good and bad, right? I don't want to start any wars here. I think one of the things that Elm and Ruby have in common is the creator. Matz designed Ruby because he wanted programming to be a joy, you know? And Evan created Elm because he wanted programming to be a delight. I think if you experience both of those, like developing in both of those languages, you gain a certain appreciation for what that means. It is almost undefinable, indistinguishable, although you can see the effects of it everywhere. In Ruby, everything is an object, including nil. In Elm, it's almost he's taken everything away. Evan's taken everything away that could potentially cause you to stumble. There's a lot to learn with Elm in terms of getting your head around functional mindset and also, working with types but as far as that goes, people often call it like the Haskell Light, which I think those are a disservice to Elm because it's got different goals. CHARLES:: Yeah, you can tell that. You know, my explorations with Elm, the personality of Elm is 100% different than the personality of Haskell, if that is even a programming term that you can apply. For example, the compiler has an identity. It always talks to you in the first person, "I saw that you did this, perhaps you meant this. You should look here or I couldn't understand what you were trying to tell me." Literally that's how the Elm compiler talks to you. It actually talks to you like a person and so, it's very... Sorry, go ahead. PHILIP: No, no, I think the corollary to that is the principle of the surprise in Ruby. You know, is there going to be a method that does this? You type it out and you're like, "Oh, yes it is," which is why things like inject and reduce are both methods in enumerable. You didn't choose one over the other. It was just like, "Let's make it easy for the person who's programming to use what they know best." I think as well, maybe people don't think about this as deeply but the level of thought that Evan has put into designing Elm is crazy, like he's thought this through. I'm not sure if I said this the last time but I went to a workshop in the early days in London, which is my kind of first real exposure to Elm and Evan was giving the workshop. Someone asked him, "Why didn't you do this?" and he was like, "Well, that might be okay for now but I'm not sure that would make so much sense in 10 years," and I was kind of like, "What?" Because JavaScript and that ecosystem is something which is changing like practically hourly and this is a guy that's thinking 10 years into the future. TARAS: You might have answered it already but I'm curious of what you think is the difference, maybe it just comes down to that long term thinking but we see this in JavaScript world a lot, which is this kind of almost indifference to APIs. It almost doesn't really matter what the API is for whatever reason, there seems to be a big correlation between the API that's exposed with the popularity of the tool. I think there are some patterns, like something that's really simple, like jQuery and React have become popular because of the simplicity of their APIs. What the flip side to that? What other ways can APIs be created that we see in JavaScript world. Because we're talking about this beautiful APIs and I can relate to some of the work that Charles has been doing and I've been doing microstates but I wonder like what would be just a brief alternative to that API, so it's kind of a beautiful API. PHILIP: I don't know if anyone is familiar with the series of essays 'Worse is Better' like East Coast versus West Coast, from Richard Gabriel. The problem is, I guess and maybe this is just my understanding over my paraphrase of it, I'm not too familiar with it but I think that good APIs take time and people don't have time. If someone launches a V1 at first and it kind of does the job, people will use that over nothing and then whenever they're happy with that, they'll continue to use it and develop it and ultimately, if she's market share and then that's just the thing everyone uses and the other guy's kind of left behind like, "This is so much better." I guess this is a question, I think it was after Wicked Good Ember, I happened to be on the same trend as Tom Dale on the way back to New York and we started talking about this. I think that's his big question. I think it's also a question that still has to be answered, which is, "Will Elm ever be mainstream? Will it be the most popular thing?" aside from the question of whether it has to be or not. For me, a good APIs good design comes from understanding the problem fully -- CHARLES:: And you can understand the problem fully without time. PHILIP: Exactly and often, what happens -- at least this is what happens in my experience with the production software that I've written -- is that you don't actually understand the problem until you've developed a solution for it. Then when you've developed a solution for it, often the pressures or the commercial pressures or an open source is [inaudible] the pressures of backwards compatibility, mean that you can never refactor your way to what you think the best solution is and often, you start from scratch and the reality is people are too far away with the stuff you wrote in the past about the thing you're writing now. Those are always kind of at odds. I think there are a lot of people that are annoyed with Elm because the updates are too slow, it relies on Evan and we want to have a pool request accepted. All of the things that they don't necessarily recognize like the absence of which make Elm an Elm, if you know what I mean. The very fact that Evan does set such a high standard and does want everything to go through his personal filters because otherwise, you wouldn't gain the benefits that Elm gives you. The attention is very real in terms of I want to shift my software now and it becomes easier then. I think to go to a language like JavaScript, which has all of the escape hatches that you need, to be able to chop and change, to edit, to do what you need to do to get the job done and let's be quite honest, I think, also with Elm, that's the challenge for someone who's not an expert level like me. Once you hit a roadblock, you'll say, "Where do I go from here?" I know if I was using JavaScript, I could just like hack it and then clients are happy and everything's fine and you know there's a bit of stuff in your code that you would rather wasn't but at the end of the day, you go home and the job's done. DAVID:: Have you had to teach Elm to other people? You and I did some work like I've seen you pair with someone and guide them through the work that they needs to get done. If you had a chance to do something like that with Elm and see how that actually happens, like how do developer's mind develops as they're working through in using the tool? PHILIP: Unfortunately not. I would actually love to go through that experience. I hope none of my developers are listening to this podcast but secretly, I want to push them in the direction of Elm on the frontend. But no, but I can at least make from my own perspective. I find it very challenging at first because for me, being a Ruby developer and also, I would never say that I understood JavaScript as much as I would have liked. Coming from dynamic language, no functional experience to functional language with types, it's almost like learning a couple of different things at the same time and that was challenging. I think if I were to take someone through it, I would maybe start with a functional aspects and then move on to the type aspects or vice versa, like try and clearly breakdown and it's difficult because those two are so intertwined at some level. Gary Bernhardt of Destroy All Software Screencast, I watch quite a bit of his stuff and I had sent him an email to ask him some questions about one of the episodes that he did and he told me that he done the programming languages course, I think it's on Coursera from Daniel Grossman, so [inaudible] ML which is kind of the father of all MLs like Haskell and also Elm. I find that really helpful because he broke it down on a very basic level and his goal wasn't to teach you ML. It was to teach you functional programming. It would be a very interesting exercise, I think. I think the benefit that Elm gave you is you get to experience that delight very quickly with, "Oh, it's broken. Here's a nice message. I fix the message. It compiles. Wow, it works," and then there's a very big jump whenever you start talking about the effects. Whenever you want to actually do something like HTTP calls or dealing with the time or I guess, the impure stuff you would call in the Haskell-land and that was also kind of a bit weird. CHARLES:: Also, there's been some churn around that, right? PHILIP: That's right. When I started learning, they had signals, then they kind of pushed that all behind the scenes and made it a lot more straightforward. Then I just mastered it and I was like, "Yes, I know it," and then I was like, "All right. I don't need to know it anymore." This is the interesting thing for me because at work, most of our work now is in Elixir and Phoenix. I'm kind of picking a little bit up as I work with them. I think Elm's architecture behind the scenes is kind of based, I believe on Erlang's process model, so the idea of a mailbox and sending messages and dealing with immutable state. CHARLES:: Which is kind of ironically is very object oriented in a way, right? It's functional but also the concept of mailboxes and sending messages and essentially, if you substitute object for process, you have these thousands and thousands of processes that are sending messages back and forth to each other. PHILIP: Yeah, that's right. It's like on a grand scale, on a distributed scale. Although I wouldn't say that I'm that far with Erlang, Elixir to appreciate the reality of that yet but that's what they say absolutely. CHARLES:: Now, Phoenix and Elixir is a dynamically typed functional language. does it share the simplicity? One of the criticisms you had of Rails was that you couldn't fit it in your head. It was very difficult. Is there anything different about Elixir that kind of makes it a spirit cousin of Elm and the simple Ruby? PHILIP: I think so, yes. Absolutely. I don't think it gets to the same level but I think it's in the right direction and specifically on the framework front, it was designed specifically... I mean, in a sense it's like the anti-type to Rails because it was born out of people's frustrations with Rails. José Valim was pretty much one of Rails top core committers. Basically, every Rails application I wrote at one period, at 80% of the code written by José Valim, if you included all the gems, the device and the resourceful and all the rest of it. Elixir in many ways was born out of the kind of limitations of Ruby with Rails and Phoenix was also born out of frustrations with the complexity of Rails. While it's not as simple as say, Michel Martens' Syro which is like his web framework, which is a successor to Cuba if people have heard of that, it is a step in the right direction. I don't understand it but I certainly feel like I could. They have plug which is kind of analogous but not identical to Rack but then the whole thing is built out of plugs. I remember Yehuda Katz give a presentation like 'The Next Five Years' and essentially about Rails 3.0. This is going way back and Phoenix is in some ways the manifestation of his desire to have like the Russian doll pattern, where you could nest applications inside applications and you could have them side by side and put them inside each other and things like that. Phoenix has this concept called umbrella applications which tells that, like Ecto is a really, really nice obstruction for working with the database. CHARLES:: I see. It feels like, as opposed to being functional or static versus dynamic, the question is how do you generate your complexity? How do you cope with complexity? Because I think you touched on it at the beginning of the conversation where you thought that my problems are complex so the systems that I work with to solve those problems must necessarily also be complex. I think one of the things that I've certainly realized, kind of in the later stages of my career is that first part is true. The problems that we encounter are extremely complex but you're much better served if you actually achieve that complexity by composing radically simple systems and recombining them. To the commonality of your system is going to determine how easy it's going to work with and how well it can cope with complexity. What really drives a good system is the quality of its primitives. PHILIP: Absolutely. After ElmConf, I actually invited Michel to come to my place in the Netherlands. He live in Paris but I think he grew up Buenos Aires in Argentina. To my amazement, he said, "Yes, okay," and we spent a couple of days together and there he talked to me about Christopher Alexander and the patterns book, where patterns and design patterns actually grew out of. One of his biggest things was the code is the easiest part, like you've got to spend 80% of your time thinking deeply about the problem, like literally go outside, take long looks. I'm not sure if this is what Rich Hickey means with Hammock Driven Development. I've never actually got around to watching the talk. CHARLES:: I think it's exactly what he means. PHILIP: And he said like once you get at, the code just comes. I think Michel's work, you should really check it out. I'll send you a link to put in the show notes but everything is built out of really small libraries that do one thing and do it really well. For example, he has a library like a Redis client but the Redis client also has something called Nest, which is a way to generate the keys for nested hashes. Because that's a well-designed, the Redis client is literally just a layer on top. If you understand the primitive then, you can use the library on top really well. You can embed Syro applications within Syro applications. I guess, there you also need the luxury of time and I think this is where maybe my role as VP of engineering, which is kind of my first role of that kind, comes in here which is when you're working on the commercial pressure, try to turn around to a business guy and say, "Yes, we'll solve this problem but can we take three weeks to think about it?" It's never going to happen -- CHARLES:: No. PHILIP: Absolutely, it will never going to happen. Although the small things that I tried to do day to day now is get away from the computer, write on paper, write out the problem as you understand it, attack it from different angles, think about different viewpoints, etcetera. CHARLES:: I think if you are able to quantify the cost of not thinking about it for three weeks, then the business person that you're going to talk to is their ears are going to perk up, right? But that's so hard to do. You know, I try and make like when we're saying like, "What technologies are you going to choose? What are the long term ramifications in terms of dollars or euros or whatever currency you happen to be in for making this decision?" I wish we had more support in thinking about that but it is kind of like a one-off every time. Anyway, I'm getting a little bit off track. PHILIP: No, not at all. This is a subject I love to talk about because we kind of had a few a bit of turbulence because we thought, maybe we should get product people in, maybe we should get them a product team going and what I find was -- and this is maybe unique to the size of the company -- that actually made things a lot more difficult because you got too many heads in many ways. Sometimes, it's better to give the developer all of the context so that he can think about it and come up with the best solution because ultimately, he's the only one who can understand. I wouldn't say understands the dollars and cents but he understands the cost implications of doing it in efficient ways, which often happens when you're working in larger teams. TARAS: One thing I find really interesting about this conversation is the definition of good is really complicated here. I've observed Charles work on microstates and I work with him, like I wrote a lot of the code and we got through like five or six iterations and at every point, he got better but it is so difficult to define that. Then when you start to that conversations outside of that code context and you start to introduce business into the mix, the definition of good becomes extremely complicated. What do you think about that? How do we define it in a way? Are there cultures or engineering cultures or societal cultures that have a better definition for good that is relevant to doing quality work of this? CHARLES:: That's a deep question. PHILIP: Wow. Yeah, a really, really deep question. I think often for business, like purely commercially-driven, money-oriented good is the cheapest thing that gets the job done and often that's very short term, I think. As you alluded to Charles, that people don't think about the cost of not doing the right things, so to speak in our eyes and also, there's a huge philosophical discussion whether our definition of good as programmers and people who care about our craft is even analogous to or equal to a good in a commercial context. CHARLES:: Yes, because ultimately and this is if you have read Zen in the Art of Motorcycle Maintenance, one of the things that Pirsig talks about is what is the definition of quality. How do we define something that's good or something that's bad? One of the definitions that gets put forward is how well something is fit to purpose. Unless you understand the purpose, then you can't determine quality because the purpose defines a very rich texture, a very rich surface and so, quality is going to be the object that maps very evenly and cleanly over that surface. When it comes to what people want in a program, they're going to want very different thing. A developer might need stimulation for this is something that's very new, this is something that's going to keep my interest or it's going to be keeping my CPU max and I'm going to be learning a whole lot. A solution that actually solves for that purpose is going to be a high quality solution. Also, this is going to be fast. We're going to be able to get to market very quickly. It might be one of the purposes and so, a solution that is fast and the purpose fits so it's going to be good. Also, I think developers are just self-indulgent and looking for the next best thing in something that's going to keep their interest, although we're all guilty of that. But at the same time, we're going to be the ones maintaining software, both in our current projects and collectively when we move to a new job and we're going to be responsible for someone else's code, then we're going to be paying the cost of those decisions. We both want to minimize the pain for ourselves and minimize the pain for others who are going to be coming and working in our code to make things long term maintainable. That's one axis of purpose and therefore, an axis of quality. I think in order to measure good and bad, you really have to have a good definition of what is the purpose of that surface is so rich but the more you can map it and find out where the contours lie, the more you're going to be able to determine what's good and what's bad. TARAS: It makes me think of like what is a good hammer. A sledgehammer is a really good hammer but it's not the right hammer for every job. CHARLES:: Right. TARAS: I think what you're saying is understanding what is it that you're actually doing and then matching your solution to what you're actually trying to accomplish. PHILIP: Yeah, absolutely and in my experience, we have a Ruby team building a Rails application. That's our monolith and then, we have a couple of Elixir teams with services that have been spun out of that. This isn't proven. This is just kind of gut feel right now and it is that Elixir is sometimes slower to develop the same feature or ship it but in the long term it's more maintainable. I haven't actually gotten dived into to React and all of the amazing frameworks that it has in terms of getting things up and running quickly but in terms of the full scale application, I still think 10, 11 years on, Rails has no equal in terms of proving a business case in the shortest time possible. CHARLES:: Yeah. I feel very similarly too but the question is does your development team approach the problem as proving a business case or do they approach the problem as I want to solve the set of features? PHILIP: Yes. Where I'm working at the moment, I started out just as a software developer. I guess, we would qualify for 37 signals or sorry... base camps definition of a calm company -- CHARLES:: Of a what company? PHILIP: A calm company. Sorry. They just released a new book and called 'The Calm Company' and 'It Doesn't Have to Be Crazy at Work.' I was given in my first couple of months, a problem. It was business oriented, it had to be solved but it had to be solved well from a technical perspective because we didn't want to have to return to it every time. It was standardizing the way that we exported data from the database to Excel. You know, I was amazed because it was literally, the first time that I'd been given the space to actually dive in on a technical level to do that kind of stuff. But I think even per feature, that varies and that sometimes challenging when handing the work on because you've got to say, "This fit. Literally, we're just trying to prove, whether if we have this feature, the people will use it?" versus, "This is a feature that's going to be used every day and therefore, needs to be at good, technical quality." Those are the tradeoffs that I guess, keep you in a job. Because if it was easy, then you would need anyone to figure it out but it's always a challenge. What I like is that our tools are actually getting better and I think, with Elm for example, it's kind of major selling point is maintainability and yet, with Elm, there haven't been that many companies with Elm over a period of years that exists, that can live to tell the tale. Whereas, we certainly know with Rails applications have done well like Basecamp and GitHub. For sure, they can be super maintainable but the fact that it took GitHub to just moved Elm to Rails 5.0, I belief, the fact that it took them years and years and they were running off at fork of Rails 2.3, I think it shows the scale of the problem in that way. You know, Phoenix also went through a few issues, kind of moving architectures from the classic Rails to a more demand driven design model. I think we're getting there slowly, zig-zagging towards a place where we better understand how to write software to solve business problems. I guess, I was really interested in microstates when you shared it at Wicked Good Ember because that to me was attacking the problem from the right perspective. It's like given the fact that the ecosystem is always changing. How can we extract the business logic such that these changes don't affect the logic of our application? CHARLES:: Man, we got a lot to show you. It has changed quite a bit in the last two years. Hopefully, for the better. TARAS: It's been reduced and it's almost a quarter of its size while maintaining the same feature set and it's faster, it's lazier, it's better in every respect. It's just the ideas have actually been fairly consistent. It's just the implementation that's evolved. CHARLES:: Yeah, it's been quite a journey. It parallels kind of the story that we're talking about here in the sense that it really has been a search for primitives and a search for simplification. One of the things that we've been talking about, having these Ruby gems that do one thing and do it very, very, very well or the way that Elixir being architected has some very, very good primitives or Elm, the same kind of thing being spiritually aligned, even though on the surface, it might share more in common with Haskell. There's actually a deep alignment with a thing like Ruby and that's a very surprising result. I think one of the things that appeals to me about the type of functional programming that is ironically, I guess not present in Elm, where you have the concept of these type classes but I actually think, I love them for their simplicity. I've kind of become disenchanted with things like Lodash, even though they're nominally functional. The fact that you don't have things like monoid and functors and stuff is kind of first class participants in the ecosystems, means you have to have a bunch of throwaway functions. Those API surface area is very large, whereas if you do account for those things, these kind of ways of combining data and that's how you achieve your complexity, is not by a bunch of one-off methods that are like in Lodash, they're all provided for you so you don't have or have to write them yourself. That is one level of convenience but having access to five primitives, I think that's the power of the kind of the deeper functional programming types. PHILIP: And Charles, do you think that that gives you the ability to think at a higher level, about the problems that you're solving? Would you make that link? CHARLES:: Absolutely. PHILIP: So, if we're not doing that, then we're actually doing ourselves a disservice? CHARLES:: I would say so. PHILIP: Because we're actually creating complexity, where it shouldn't exist? CHARLES:: Yeah, I think if you have a more powerful primitive, you can think of things like async functions and generator functions, there's a common thread between async functions, generator functions, promises arrays and they're all functors. For me, that's a very profound realization and there might be a deeper spiritual link between say, an async function and an array in the same way that there's a deep spiritual link between Ruby and Elm, that if you don't see that, then you're doing yourself a disservice and you're able to think at a higher level. Also, you have a smaller tool set where each tool is more powerful. PHILIP: You did a grit, I think it was a repository with a ReadMe, where you boiled down what people would term what I would term, the scary functional language down to a very simple JavaScript. Did you ever finish that? Did you get to the monads? CHARLES:: I did get to the monads, yeah. PHILIP: Okay. I need to check that out again. I find that really, really helpful because I think one of Evan's big things with Elm is he doesn't use those terms ever and he avoids them like the plague because I think he believes they come tinged with the negative experiences of people trying Haskell and essentially getting laughed at, right? CHARLES:: Yes. I think there's something to that. TARAS: But we're doing that in microstates as well, right? In microstates documentation, even though microstates are written completely with these functional primitives, on the outside, there's almost no mention of it. It's just that when you actually go to use it, if you have an idea, one of the thing that's really powerful with microstates is that this idea that you can return another microstate from a transition and what that will do is what you kind of like what a flat map would do, which is replace that particular node with the thing that you returned it with. For a lot of people, they might not know that that's like a flat map would do but a microstate will do exactly what they wanted to do when it didn't realize that's actually should just work like that. I think, a lot of the work that we've done recently is to package all things and it make it powerful and to access the concepts that it is very familiar, something you don't need to learn. You just use it and it just works for you. CHARLES:: Right but it is something that I feel like there's unharvested value for every programmer out there in these type classes: monads and monoids and functors and co-functors or covariant functors, contravariant functors, blah-blah-blah, that entire canon. I wish there was some way to reconcile the negative connotations and baggage that that has because we feel kind of the same way and I think that Evan's absolutely right. You do want to hide that or make it so that the technology is accessible without having to know those things. But in the same way, these concepts are so powerful, both in terms of just having to think less and having to write less code but also, as a tool to say, "I've got this process. Is there any way that could it be a functor? If I can find a way that this thing is a functor, I can just save myself so much time and take so many shortcuts with it." PHILIP: And in order to be able to communicate that, or at least communicate about that, you need to have terms to call these things, right? Because you can't always just refer to the code or the pattern. It's always good to have a name. I'm with you. I see value in both, like making it approachable, so the people who don't know the terms are not frightened away. But I also see value in using the terms that have always existed to refer to those things, so that things are clear and we can communicate about them. CHARLES:: Right. definitely, there's a tradeoff there. I don't know where exactly the line is but it would be nice to be able to have our cake and eat that one too. We didn't get really to talk about the type versus dynamic in the greater context of this whole conversation. We can explore that topic a little bit. PHILIP: Well, I can finish with, I think the future is typed Erlang. Maybe, that's Elm running on BEAM. CHARLES:: Whoa. What a take? Right there, folks. I love it. I love it but what makes you say that? Typed Erlang doesn't exist right now, right? PHILIP: Exactly. CHARLES:: And Elm definitely doesn't run on BEAM. PHILIP: I don't know if I'm allowed to say this. When I was at this workshop with Evan, he mentioned that and I'm not sure whether he mentioned it just as a throwaway comment or whether this is part of his 20-year plan but I think the very fact that Elm is designed around like Erlang, the signal stuff was designed around the way Erlang does communication and processes, it means I know at least he appreciates that model. From my point of view, with my experience with Elixir and Erlang in production usage, it's not huge scale but it's scale enough to need to start doing performance work on Rails and just to see how effortless things are with Elixir and with Erlang. I think Elm in the backend would be amazing but it would have to be a slightly different language, I think because the problems are different. We began this by saying that my story was a little different to the norm because I went back to the dynamic, at the dark side but for example in Elixir, I do miss types hugely. They kind of have a little bit of a hack with Erlang because they return a lot of tuples with OK and then the object. You know, it's almost like wrapping it up in a [inaudible]. There are little things and there's Dialyzer to kind of type check and I think there are a few projects which do add types to Erlang, etcetera. But I think something that works would need to be designed from the ground up to be typed and also run in the BEAM, rather than be like a squashed version of something else to fit somewhere else, if that makes sense. CHARLES:: It makes total sense. PHILIP: I think so. I recently read a book, just to finish which was 'FSharpForFunAndProfit' is his website, Scott Wlaschin, I think. It's written up with F# but it's about designing your program in a type functional language. Using the book, you could probably then just design your programs on paper and only commit to code at the end because you're thinking right down to the level of the types and the process and the pipelines, which to me sounds amazing because I could work outside. CHARLES:: Right. All right-y. I will go ahead and wrap it up. I would just like to say thank you so much, Philip for coming on and talking about your story, as unorthodox as it might be. PHILIP: Thank you. CHARLES:: Thank you, Taras. Thank you, David. TARAS: Thank you for having us. CHARLES:: That's it for Episode 113. We are the Frontside. This is The Frontside Podcast. We build applications that you can stake your future on. If that's something that you're interested in, please get in touch with us. If you have any ideas for a future podcast, things that you'd like to hear us discuss or any feedback on the things that you did here, please just let us know. Once again, thank you Mandy for putting together this wonderful podcast and now we will see you all next time.

Guests: Amanda Hickman: @amandabee | GitHub Amberley Romo: @amberleyjohanna | GitHub | Blog In this episode, Amanda Hickman and Amberley Romo talk about how they paired up to get the safety pin, spool of thread, and the knitting yarn and needles emojis approved by the Unicode Committee so that now they are available for use worldwide. They also talk about how their two path crossed, how you can pitch and get involved in making your own emojis, and detail their quest to get a regular sewing needle approved as well. Resources: Unicode Technical Committee Draft Emoji Candidates The Unicode Consortium Members Sewing-Emoji Repo Proposal for Sewing NEEDLE AND THREAD Emoji This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. Transcript: ROBERT: Hello, everyone. Welcome to Episode 112 of The Frontside Podcast. I'm Robert DeLuca, a software developer here at the Frontside and I'll be your episode host. With me as co-host is Charles Lowell. Hey, Charles. CHARLES: Hello, Robert. Good morning. ROBERT: Good morning. This is an exciting podcast. Today, we're going to be discussing writing a proposal to the Unicode Committee, getting it accepted and rejected. This is basically making emojis which I think is really awesome. We have two guests today who have an amazing story, Amanda Hickman and Amberley Romo. Thank you both for joining us. You two have an amazing story that I would really love to dive into and we're going to do that today. It's about basically creating your own emoji and getting that accepted so everybody can use that and I think that's super, super cool, something that I've always kind of wanted to do as a joke and it seems like that's kind of where your stories began, so you two want to jump in and start telling? I think Amanda has a great beginning to this. AMANDA: Sure. I mean, hi and thanks for having me. I don't know where to begin and really for me, this starts with learning to sew my own clothes which is an incredibly exasperating and frustrating process that involves a lot of ripping stitches back out and starting over and Instagram was a really big part of me finding patterns and finding other people who are sewing their own clothes and learning from the process. I wanted to be able to post stuff on Instagram and it started to drive me absolutely crazy, that there's emojis for wrenches and nuts and hammers and there are no textile emoji. The best I could find was scissors which is great because cutting patterns is a place where I spend a lot of time procrastinating but that was it. I knew a woman, Jennifer 8 Lee or Jenny who had led a campaign to get the dumpling emoji into the Unicode character set. I knew she'd succeeded in that but I didn't really know much more about how that had worked. I started thinking I'm going write a sewing emoji. I can do this. I can lead this campaign. I started researching it and actually reached out to Jenny and I discovered that she has created an entire organization called... What was that called? She's created an entire organization called Emojination, where she supports people who want to develop emoji proposals. CHARLES: Before you actually found the support system, you actually made the decision that you were going to do this and you found it. You know, from my perspective, I kind of see emoji is this thing that is static, it's there, it's something that we use but the idea that I, as an individual, could actually contribute to that. I probably, having come to that fork in the road would have said, "Nah, it's just it is what it is and I can't change it." What was the process in your mind to actually say, "You know what? I'm actually going to see if I can have some effect over this process?" AMANDA: It definitely started with a lot of anger and being just consistently frustrated but I knew that someone else had already done this. It was sort of on my radar that it was actually possible to change the emoji character set. I think that if I didn't know Jenny's story and it turned out I didn't know Jenny story at all but I thought I knew Jenny story but if I didn't know that basic thing that that somebody I knew who was a mere mortal like me had gone to the Emoji Subcommittee of the Unicode Consortium and petition them to add a dumpling emoji, I am sure that I wouldn't bother. But I knew from talking to her that there was basically a process and that there were a format that they want proposal in and it's possible to write them a proposal. I knew that much just because I knew Jenny. I think at that point, when I started thinking about this, the Emoji 9 -- I should be more of an expert on that actually, on emoji releases but a new release of emoji had come out. There were a bunch of things in that release and it got a little bit of traction on Twitter. I knew that the Unicode Consortium had just announced a whole new slate of emoji, so I also was generally aware that there was some kind of process by which emoji were getting released and expanded and updated. ROBERT: That's interesting. Do you know when that started? Because it seems like Apple started to add more emojis around like iOS 7 or something but it was pretty static for a while right? Or am I wrong? AMANDA: I actually am tempted to look this up but the other piece that is not irrelevant here is that at the time, I was working at a news organization called BuzzFeed that you may have heard of -- ROBERT: Maybe, I don't know. It sounds kind of familiar. AMANDA: I do feel like people kind of know who they are. I was surrounded by emoji all the time: in BuzzFeed, in internet native of the highest order and we had to use emoji all the time and I had to figure out how to get emoji into blog post which I didn't really know how to do before that. I can put them on my phone but that was it. I was immersed in emoji already. I knew that there was a project called Emojipedia, that was a whole kind of encyclopedia of emoji. One of my colleagues at BuzzFeed, a woman named Nicole Nguyen had written a really great article about the variation in the dance emoji. If you look at the dance emoji, one of the icons that some devices use is this kind of woman with her skirt flipping out behind her that looks like she's probably dancing a tango and then one of the icons that other character sets use and other devices use is a sort of round, yellow lumping figure with a rose in its mouth that you sort of want to hug but it's definitely not to impress you with its tango skill. She had written this whole article about how funny it was that you might send someone this very cute dumpling man with [inaudible] and what they would see was sexy tango woman. I think there was some discussion, it was around that time also that Apple replaced the gun emoji with a water gun. There was some discussion of the direction that the various emoji's face. One of the things that I learned around that time was that every device manufacturer produces their own character set that's native to their devices and they look very different. That means that there's a really big difference between putting a kind of like frustrated face with a gun pointing at it, which I don't really think of it as very funny but that sort of like, "I'm going to shoot myself" is very different from pointing the gun the other way which is very much like, "I'm shooting someone else," so these distinctions, what it means that the gun emoji can point two different ways when it gets used was also a conversation that was happening. None of that answers your question, though which is when did the kind of rapid expanse of emoji start to happen. ROBERT: I feel like the story is setting in the place there, though because it seems like there's a little bit of tension there that we're all kind of diverging here a little bit and it's sort of driving back towards maybe standardization. AMANDA: There's actually, as far as I know, no real move toward standardization but the Unicode Consortium has this committee that actually has representatives from definitely Apple and Microsoft and Google and I forget who else on the consortium. Jenny 8 Lee is now on the consortium and she's on the Emoji Subcommittee but they actually do get together and debate the merits of adding additional emoji, whether they're going to be representative. One of the criteria is longevity and I tend to think of this as the pager problem. There is indeed a pager emoji and I think that the Unicode Consortium wants to avoid approving a pager emoji because that was definitely a short-lived device. CHARLES: Right. I'm surprised that it actually made it. Emoji must be older than most people realize. AMANDA: My understanding is that very early Japanese computers had lots emoji. There's a lot of different Japanese holidays that are represented in emoji, a lot of Japanese food as well are represented in emoji, so if you look through the foods, there's a handful of things that haven't added recently but a lot of the original emoji definitely covered Japanese cuisine very well. ROBERT: I definitely remember when I got my first iPhone that could install iPhone OS 2, you would install an app from the App Store that then would allow you to go toggle on the emoji keyboard but you had to install an app to do it and that's kind of where the revolution started, for me at least. I remember everybody starting to sending these things around. AMANDA: But if you look at Emojipedia, which has a nice kind of rundown of historical versions of the Unicodes, back in 1999, they added what I think of as the interrobang, which is the exclamation/question mark together and a couple of different Syriac crosses. Over the years, the committee has added a whole series of wording icons and flags that all make sense but then, it is around, I would say 2014, 2015 that you start to get the zipper mouth and rolling your eyes and nerd face and all of the things that are used in conversation now -- the unicorn face. ROBERT: My regular emojis. AMANDA: Exactly. CHARLES: It certainly seems like the push to put more textile emoji ought to clear the hurdle for longevity, seeing there's kind of like, what? Several millennia of history there? And just kind of how tightly woven -- pun intended -- those things are into the human experience, right? AMANDA: Definitely. Although technically, there's still no weaving emojis. CHARLES: There's no loom? AMANDA: There's no loom and I think that a loom would be pretty hard to represent in a little 8-bit graphic but -- CHARLES: What are the constraints around? Because ultimately, we've already kind of touched on that the emoji themselves, their abstract representations and there are a couple of examples like the dancing one where the representation can vary quite widely. How do they put constraints around the representation versus the abstract concept? AMANDA: You don't have to provide a graphic but it definitely kind of smooths the path if you do and it has to be something that's representable in that little bitty square that you get. It has to be something representable in a letter-size square. If it's not something that you can clearly see at that size, it's not going to be approved. If it's not something you can clearly illustrate at that size in a way that's clearly distinct from any other emoji and also that's clearly distinct from anything else of that image could be, it's not going to be approved. Being able to actually represented in that little bitty size and I don't know... One of sort of sad fact of having ultimately worked with Emojination on the approval process is that we were assigned an illustrator and she did some illustrations for us and I never had to look at what the constraints were for the illustration because it wasn't my problem. ROBERT: Sometimes, that's really nice. AMANDA: Yes, it's very nice. I ended up doing a lot of research. What made me really sad and I don't want to jump too far ahead but one of things that made me really sad is we proposed the slate and the one thing that didn't get approved was the sewing needle and it also didn't get rejected, so it's in the sort of strange nether space. That's kind of stuck in purgatory right now. I did all this research and learned that the oldest known sewing needle is a Neanderthal needle so it predates Homo sapiens and it's 50,000 years old. CHARLES: Yeah. Not having a sewing needle just seem absurd. AMANDA: Yeah. We have been sewing with needles since before we were actually human being. ROBERT: That's a strong case. AMANDA: Yes, that's what I thought. If I sort go back to my narrative arc, I wanted to do a sewing needle and started researching it a little bit -- CHARLES: Sorry to keep you interrupting but that's literally the one that started this whole journey. AMANDA: Yes, I wanted a sewing needle and I really wanted a sewing needle. I did a little research and then I reach out to Jenny and to ask her if she had any advice. She said, "You should join my Slack," and I was like, "Oh, okay. That's the kind of advice." She and I talked about it and she said that she thought that it made more sense to propose a kind of bundle of textile emoji and I decided to do that. She and I talked it through and I think the original was probably something closer to knitting than yarn but we said knitting, a safety pin, thread and needle were the ones that kind of made the most sense. I set about writing these four proposals and one of the things that they asked for was frequently requested. One other thing that I will say about the proposal format is that they have this outline structure that is grammatically very wonky. They ask you to assert the images distinctiveness and they also ask you to demonstrate that it is frequently requested. I found a couple of really interesting resources. One, Emojipedia which is this sort of encyclopedia of emoji images and history maintains a list of the top emoji requests. I actually don't know how they generate that list or who's requesting that and where but I think it's things that they get emailed about and things people request in other contexts and sewing and knitting, I've done on that list and I started compiling it in 2016. ROBERT: To be a part of the proposal process, to show that it is requested, without that resource, you just start scouring Facebook and Twitter and history and shouting to people like, "I really want this emoji. Why it didn't exist?" That seems pretty hard. AMANDA: Actually our proposals all have Twitter screen shots of people grousing about the absence of knitting emoji and yarn emoji and sewing emoji. I know that Emojipedia, they do a bunch of research so they go out and look at based what people are grousing about on Twitter. They look at places where people are publicly saying like, "It's crazy that there's no X emoji," and that's part of their process for deciding what kinds of emojis people are asking for. Their research was one resource but we took screenshots of people saying that they needed a safety pin emoji and that was part of making the case. One of the things that I found as I was doing that research was that, I guess at this point it was almost two years ago, when the character set that included the dumpling emoji came out, there was a bunch of grousing from people saying, "Why is there not a yarn emoji?" There was a writing campaign that I think Lion Brand had adopted. Lion Brand yarn had put in this tweet saying like, "Everyone should complain. We needed a yarn emoji," but it doesn't matter how much you yell on Twitter. If you don't actually write a proposal, you're not going to get anywhere. I had been told that the Emoji Subcommittee, they're really disinclined to accept proposals that had a corporate sponsor, so they weren't going to create a yarn emoji because Lion Brand yarns wanted them to create a yarn emoji. ROBERT: Right, so it was like counter-peer proposal. AMANDA: Right. But as I was digging around the other thing I found was this woman in... I actually don't know if you're in Dallas or Austin but I found Amberley, who also put a post on Twitter and had started a petition, asking people to sign her petition for a yarn emoji proposal or a knitting emoji. I don't remember if it was a yarn emoji or a knitting emoji but I found her petition and reached out to her to ask if she was interested in co-authoring the proposal with me because she had clearly done the work. She actually had figured out how the system worked at that point. I think she knew who she was petitioning, at least. I reached out to Amberley and we worked together to refine our proposal and figure out what exactly we wanted to request. I think there were a bunch of things that were on the original list like knitting needles, yarn and needles. I think crocheting would have been on the original list. We were sort of trying to figure out what was the right set of requests that actually made sense. ROBERT: So then, this is where Amberley stories comes in and it is interesting too because she has entirely different angle for this. Maybe not entirely different but different than outright. This kind of ties back to the word software podcast mostly. It kind of ties back to the software aspect, right? AMBERLEY: Yeah. I think, really they're kind of separate stories on parallel tracks. My motivation was also two-fold like Amanda's was, where I started knitting in 2013 and I had a really good group of nerd friends with a little yarn shop up in DC, like a stitch and ditch group -- ROBERT: I love it. AMBERLEY: It was a constant sort of like, where's the insert emoji here, like where's the yarn emoji? Where's the knitting emoji? And we would sort of sarcastically use the spaghetti emoji because it was the most visually similar but that was something that was in the back of my mind but it teaches you a lot about yourself too because I was like, "Oh, this is like fiber art, not really an emoji. It's kind of technical, like on a tech space," and I didn't really connect that it was relevant or that I might have any power to change it. It just didn't occur to me at the time. ROBERT: Interesting. I feel like a lot of people are in that similar situation or maybe not situation, even though you can make change on this. AMBERLEY: Right, so my brain didn't even make it like, "Why isn't this a thing? let me look at how to make the thing." When that happened for me, Amanda mentioned using emoji and everything in the BuzzFeed space. I love how you explained BuzzFeed a while ago, it's my favorite description of BuzzFeed I ever heard. Something similar that happened for me was I was a software developer and in 2016, the Yarn package manager was released and that kind of turned something on in my head. That was like I'm seeing all these software engineers now be like, "Where's the yarn emoji?" and I'm like, "Welcome to the club." ROBERT: "Do you want to join our Slack? We can complain together." AMBERLEY: Right. It has been like a pretty decent amount of time, I'm semi-seriously ranting and complaining to my coworkers who were primarily male software engineers. I remember I went to [inaudible] in the Frost Bank Tower after work and was just like, "I'm going to figure out how this happens," and I spent a couple hours at the coffee shop. I found the Unicode site and I found their proposal process and their structure for the proposal and everything and I just started doing the research and drafting up a proposal specifically for yarn. Maybe it was a bit naive of me but to me it was like, "Okay, here's the process. I follow the process. Cool." I mean, you have to make a case and it has to be compelling and has to be well-written and it has to be supported and all that and that to me it was like, "Okay, there's a process. At the same time, I did read about the dumpling emoji but I didn't connect it to Emojination and they had started the Kickstarter. We should talk about this later but I think the sort of idea the issue of representation on the committee and who gets to define language is really interesting but I saw that they had done the Kickstarter and there was a campaign aspect to it, so I ended up just building up this simple site so that if anyone Google, they would find yarn emoji. It's still up at YarnEmoji.com and that was how Amanda found me. I got this random email, I sort of like had this burst of energy and I did all the research and I wrote the draft, sort of piecemeal, filling out the different sections of the way they have it outlined on the Unicode site and then I feel like a month or two went by and I had kind of not looked at it for a bit and then, I get this random email from this website that I almost forgot about. It was like, "Hey, I'm working on this series of proposals. If you're working on knitting or yarn or whatever, maybe we could work together," and I was like, "Well, that's sweet." Then she opened up this whole world to me. There's this whole Emojination organization, sort of 100% devoted to democratizing the process of language formation through creating emojis and so then, I got really into that. My primary motivator was yarn. CHARLES: So what's the status of the yarn spool, those emoji right now? AMBERLEY: The yarn, the spool of thread and the safety pin, they're all approved emoji for the 2018 released. Amanda and I are actually at the end. Amanda, a couple of months ago when I saw someone used the spool thread emoji for a Twitter thread -- you know how people will be like all caps thread and have a thread of tweets -- I saw someone do that just out of the blue. I was like, "Oh, my God. Is it out?" and the thing about these individual vendors, it sort of gets released piecemeal, so at the time Twitter have I think released their versions of this series of new emoji but others hadn't. CHARLES: How does that work? Because you think the Twitter would be kind of device depending on what browser you're using, like if you're on a Windows or a Mac or a Linux Box, right? ROBERT: -- Emoji set, right? I know Facebook does this too. AMBERLEY: I'm painfully aware that Facebook does it because I can't use the crossed finger emoji on Facebook because it actually gives me nightmares. ROBERT: I have to go look at this now. AMBERLEY: Because it's so creepy-looking. CHARLES: Okay. Also like Slack, for example is another. It's like a software-provided emoji set. AMANDA: Right. AMBERLEY: I'm not totally sure that Slack actually adheres to the standard Unicode set. I think it's kind of its own thing but I might be wrong about that. AMANDA: Sorry, Slack definitely supports the full Unicode set. They also have a bunch of emoji that they've added that aren't part of the set. AMBERLEY: Slack emojis? AMANDA: Yes. CHARLES: Yeah and then every Slack also has its kind of local Slack emoji. AMBERLEY: Right. CHARLES: But how does that work with --? ROBERT: Okay, this crossed-finger Facebook emoji is... yes, I agree with you, Amberley. AMBERLEY: Thank you. I had yet to find someone who disagrees with me about that. AMANDA: I have never seen it before and I'm now like, "What is going on?" CHARLES: Yeah, so how does it work if a vendor like Twitter is using a different emoji set? How does that work with cut and paste, like if I want to copy the content of one tweet into something else? Are they using an image there? AMANDA: They're using an image. I think it's doesn't happen as much anymore but for a long time, I would often get texts from people and the text message would have that little box with a little code point in it and you were like -- AMBERLEY: More like an alien thing? AMANDA: Yeah. Definitely, if you don't have the emoji character set that includes the glyph that you're looking at, you're going to get that little box that has a description of the code point and I think what's happening is that Twitter is using JavaScript or generally programming. There were air quotes but you can't see. Twitter is using their software to sub in their emoji glyph whenever someone enters that code point. Even if you don't have the most up to date Unicode on your computer, you can still see those in Twitter. If I copy and paste it into a text editor on my computer, what I'm going to see is my little box that says '01F9F5' in it but if I get it into Twitter, it shows up. I can see them on Twitter but I can't see them anywhere else. AMBERLEY: Damn, you really have the code point memorized? AMANDA: No, I -- CHARLES: Oh, man. I was really hoping -- AMBERLEY: Oh, man. ROBERT: You live and breathe it. AMANDA: No, I'm not that compulsive. AMBERLEY: We definitely have our emojis on our Twitter bios, though. AMANDA: Absolutely. ROBERT: If you see Amanda's bio, it's pretty great. AMANDA: They started showing up on Twitter and I think that somebody in Emojination probably told me they were out and that was when I first started using them. Amberley might have actually seen it. It sounds like you just saw it in the wild, which is kind of amazing. AMBERLEY: I saw it in the wild with this tweet thread and yeah, it's just [inaudible]. I was like, "Amanda, is it out?" CHARLES: Yeah, I feel like I saw that same usage too, although I obviously did not connect any dots. AMANDA: This last week, October 2nd -- I'm also looking things up. I'm just going to come to the fact that I am on a computer looking things up so I can fact check myself -- after they actually released their emoji glyph set, so by now any updated iOS device should have the full 2018 emoji, which in addition to a kind of amazing chunky yarn and safety pin, there's also a bunch of stuff. There's a broom and a laundry basket. There's a bunch of really basic, kind of household stuff that certainly belongs in the character set alongside wrenches and hammers. AMBERLEY: I think one of the big ones too for this year was the hijab? AMANDA: No, the hijab actually came out with a dumpling. Hijab has been available -- AMBERLEY: It's been up, okay. ROBERT: So did it come with iOS 12 or 12.1? I don't know for sure. I just know -- AMANDA: I'm looking at it and it's 12.1. I really feel that I should be ashamed that I have used the internet and search for this. AMBERLEY: I would say, I have no idea what their release numbers are. AMANDA: [inaudible] as it appeared for the first time in iOS for 2018 with today's release of the iOS 12.1, Beta 2 for developers. ROBERT: That is amazing. Do you get some kind of satisfaction -- like you have to, right? -- from people using the emoji and it's starting to make its way out there? AMANDA: So much. Oh, my God, yeah. AMBERLEY: I didn't really expect it, like saying that random tweet using this spool of thread for a tweet thread. I just thought and I just got so psyched. For me, I'm a knitter. I have knitter friends and it started with yarn and then really, Amanda and through Amanda, Jenny really sort of broadened my idea of what it all really meant. To think someone using it in the wild for a totally different application than I had ever thought of was like, "That's legit." AMANDA: I definitely have a sewing emoji search in my tweet deck and sometimes, when I'm feeling I need a little self-validation, I'll go look over there and find people who are saying things like, "Why is there no sewing emoji?" and I'll just reply with all the sewing emoji, like it is part of my work in this life to make sure that not only do they exist but people know about them. ROBERT: That is awesome. I would do the same thing, though to be honest. You'll be proud of that. AMANDA: Totally. ROBERT: Were there any hitches in the proposal process? I know we're kind of alluded to it but the thing that you started off one thing, Amanda didn't make it. Right? AMANDA: I know. ROBERT: So how did that process happen from you two meet each other and then going through the actual committee and the review process and then being accepted. What would that mean? AMANDA: The process is actually incredibly opaque. We wrote this whole proposal, a bunch of people edited it, which is one of the other nice things about collaborating with Emojination. There was a bunch of people who are just really excited about emoji and the kind of language making that Amberley was talking about. There's a whole bunch of people who just jumped in and gave us copy edits and feedback, which was super helpful and then, there was a deadline and we submitted it to the committee and it actually shows up in the Unicode register which is also a very official kind of document register. I was a little excited about that too but then they have their meeting. They first have a meeting and there's like a rough pass and the Emoji Subcommittee makes formal recommendations to the Unicode Consortium and then the consortium votes to accept or reject the Emoji Subcommittee's recommendations. It's a very long process but unless you're going and checking the document file and meeting minutes from the Unicode Consortium meetings, you'll never going to know that it happen. AMBERLEY: -- You know someone connected through there because one of the things in our first pass, it wasn't that it was rejected. It was that we needed to modify something. We do have art for knitting needles with yarn because at one point, I think we weren't totally sure that a ball of yarn would be visually distinct enough in this emoji size to look like yarn and so, we had put it with sort of knit piece on knitting needles. AMANDA: Oh, that's right. There was a tease of a little bit of knitted fabric. AMBERLEY: Right and I think that, probably through Jenny or the people actually in the room, the feedback I remember is that there is a crocheter in the room who was like, "Yeah, why isn't there a yarn emoji but knitting needle?" so there was a little bit of like that was how I think we ended up from knitting needles with a fiber piece to ball of yarn, maybe. AMANDA: I think that sounds right. I'm actually sure of that. It's just all coincide with my recollection. There were some things that they had questions about and that happened really fast because I feel like we had a couple of days and they have stuck to our guns and said, "No, we're only interested in knitted bit of fabric." Also, we worked with an illustrator and went back and forth with her because the initial piece that she had illustrated, I feel like the knitting needles were crossing in a way. That was not how knitting works and so, there was a little bit of back and forth around that as well. But then once they decided that the they like the thread, yarn and safety pin, we're going to move to the next stage. I actually had to go back and look at the minutes to find out that the two reasons that they didn't move the sewing needle on to the next stage is when they thought it was adequately represented by the thread, which I wholeheartedly disagree with and they thought it wasn't visually distinctive. That's so much harder because a sewing needle, which is really just a very fine piece of metal with an eye at the end, you get down to a really small size and it is maybe a little hard to know what you're looking at. But I think there's such a big difference between the static object which is the spool or the thread which represents a lot of things and is important and the needle, which is the active tool that you use to do the making, to do the mending, to do the cobbling. CHARLES: Yeah. I'm surprised that it almost isn't reversed when certainly in my mind, which I think is more culturally important in terms of the number of places which it appears, it's definitely the needle as being kind of... Yeah. AMANDA: Yeah and I think that the thread and yarn, they're important and I think that the decision to have a ball of yarn rather than a bit of knitting makes sense because there's a lot of things that you can use a ball of yarn that aren't just knitting and they think that -- AMBERLEY: And it's the first step too that doesn't exclude anyone in the fiber art community. AMANDA: But there's so many things like in sutures and closing wounds, you're not using a little spool of cotton thread for that or polyester thread and stuff like embroidery and beadwork, you might be using thread or fiber of some sort that started on a spool but you might not. Embroidery floss was not sold in a spool and there's all these places where we use needles and all kinds of different size and you don't always use thread. Sometimes, you're using yarn. Sometimes, you're using leather cord. Sometimes, you're using new bits of, I would say Yucca. You're using plant fibers to do baskets and in all of these different practices, that process of hooking it through the eye and sewing it is how it's actually made. It still sort of mystifies me why they haven't accepted it but they also didn't reject it, which is really interesting. I don't know how many other emoji are sort of sitting in this weird nether space because sometimes they just reject them outright. I think there was a proposal for a coin that they just said no. ROBERT: They were a like, "A coin?" That would be [inaudible]. AMANDA: Oh, God. ROBERT: They have to add one for every -- AMANDA: [inaudible]. CHARLES: Literally, the pager of 2017. AMANDA: Exactly. CHARLES: So what recourse is now available to you all and to us, by extension, to get the sewing needle? AMANDA: I'm actually working on a revised proposal and I've been trying to figure out what are all the arguments that I'm missing for why sewing and the needle are not adequately represented by the thread and yarn. A bunch of things that a friend of my named, Mari who's half-Japanese, half-American but lives in Guatemala and does all this kind of arts in textile work, pointed out that there's a whole holiday in Japan devoted to bringing your broken needles and thanking them for their service. I thought that was really cool. I've been trying to formulate what are all of the arguments for the necessity of both a needle and a spool. If anybody has interesting ways to phrase that, I would love for arguments. CHARLES: Yeah but it's hard to imagine the arguments is just anything being more compelling than the arguments the you just laid out that you named about seven context: shoemaking, medicine, different fibers where the needle operates completely and totally independent of the thread. It's looming so large in kind of our collective conscious like holidays, being dedicated to them, except I think the Cro-Magnon pager, which is made out of stone, I believe, the being the artifact that pre-dates... AMANDA: There's the idiom landscape as well. Things like finding a needle in a haystack, that has a very specific meaning -- ROBERT: And for puns. I've been resisting saying a pun this whole time. AMANDA: Oh, share your pun with us. AMBERLEY: Yeah, you have to say it. ROBERT: Well, you could say that trying to get this through the committee is like threading a needle. Butchered but -- AMANDA: There's a biblical quote about getting into heaven -- a camel through the eye of a needle. I forget actually how it... CHARLES: To thread a camel through the eye of a needle than for a rich man to get into heaven. AMANDA: Exactly and there's this sort of do-re-mi, saw, a needle pulling thread. There are all these places where it's about the needle and somebody had -- CHARLES: It's primarily ancient. AMANDA: I know. CHARLES: It is the prime actor. Maybe, this is a good segue into kind of talking about the makeup of the committee and the decision making process and these kind of what seem like very clear arguments might not be received as such. AMANDA: I certainly don't want to say anything bad about anybody on the committee. CHARLES: No, no, I don't think that there's anything bad. I think that being receptive to things which are familiar to us versus with things that aren't is a very natural human thing and it can be interesting to see that at work and at play. AMANDA: The Unicode Consortium is also evaluating all of these requests for whole language glyphs sets. Lots of languages and lots of character sets that are kind of obvious, like there has to be a sort of like character set like there has to be an Arabic character set but there are a lot of languages that have been left out of that because they're very small minority languages or they are historical languages, where the actual writing is no longer written the same way but there's historical reasons to be able to represent those characters. One of the reasons why the Emoji Subcommittee cares about what gets into the formal character set is that everybody has to accommodate it and there's already been, I think some grousing. People start to moan and groan about how there's too many emoji, then it's too hard to find things. CHARLES: And there's no take backs. AMANDA: There's no take backs. You can't undo it. The committee is made up of representatives from a lot of tech companies primarily, although there's a couple of other kind of odd additional folks on there. I do try to find the committee list and I can find it right now. AMBERLEY: I have it from Emojination. I don't know if it's up to date but Oracle, IBM, Microsoft, Adobe, Apple, Google, Facebook, Shopify and Netflix. The other voting members -- ROBERT: Shopify? AMBERLEY: Yeah, right? The others being the German software company, SAP and the Chinese telecom company, Huawei and the Government of Oman. AMANDA: Yeah, the Government of Oman is a fascinating one. I don't think they're the ones that are biting us on this. Especially for those tech companies, every time the emoji character set adds 10 or 12 emoji, they don't have to accommodate it on their devices. They have to put illustrators on it, they have to deal with everyone saying that the crossed fingers emoji in Facebook looks like I-don't-even-know-what. AMBERLEY: Hey, Amanda. AMANDA: It's all your fault. There's a whole process and there's non-trivial work associated with every single new emoji, so wanting to put the brakes on a little bit and be intentional about where and when they apply that work, it doesn't seem crazy to me. I just want them to approve the thing that I want. AMBERLEY: I like the way that Emojination captures it. I looked at their website earlier and actually, they take it down but their goal quote "Emojination wants to make emoji approval an inclusive representative process." There has to be a process. There's overhead involved but looking at the makeup of the decision makers are not a trivial question. CHARLES: Right. This is a great example like [inaudible] metaphor but these little artifacts, these emojis are literally being woven to the fabric of a global culture and certainly, everybody uses them and they become part of the collective subconscious. It does seem like very important to be democratic in some way. It sounds like there is a process but making sure that everyone has a stake. AMBERLEY: Yeah. ROBERT: What was the reason that they gave for not accepting the needle and thread? Was it like a soft no? You said it's like just hanging out, not really rejected but not accepted. We're going to drop a link in the show notes for the proposal and your GitHub and everything. I'm looking at the PDF that was put together and it seems like it was all a package deal like we talked about. How do they just draw or they just take like a lawyer would, just like draw or cross it out like, "Well, no but we'll take the other ones." AMANDA: Yes, basically. What they did is they need to discuss and I don't know how long they've been meeting but they need to discuss all of the proposals that have been supplied by a particular deadline and -- ROBERT: That sounds painful. AMANDA: Yeah, I mean, it's -- ROBERT: Just imagine the power of thinking about emojis. AMANDA: One of the things that they rejected, I think because there's the smiling poo face. Somebody wanted a frowning poo face and they rejected that. There's a bunch of things that actually do get rejected. I don't know if they've been really care about a smiling poo face versus a frowning poo face. ROBERT: What about an angry one? AMANDA: We got all the feelings of poo. ROBERT: We got important work to do here. AMANDA: But they go through when they're trying to figure out. I think to some degree, you want to get them when they're not tired but I think the status that it's listed right now is committee pushback, so they've set it aside until we have some concerns. We're not going to reject it outright but we're not really sure why this isn't adequately represented. Then their most recent meeting, they just kind of passed on reconsidering it, which is fine because I think I was traveling and my proposal is not done. I really want to make sure that I have consolidated every imaginable argument in one place so -- ROBERT: And make it strong as possible. AMANDA: Yeah. If people want to help the other thing that would be amazing is any and all idioms that you can think of, especially ones that are not in English or European languages, idioms in Central European languages, idioms in Asian languages that refer to needles, either translations of the kind of classic, 'finding a needle in a haystack,' but also any idioms that are kind of unusual and specific to a culture outside of what I have experience with would be amazing for making the case, so this is an international need. ROBERT: Do they need any specific or actionable feedback or do they just say, "We're going to push back on this. We're just not quite sure?" AMANDA: The two things that we're in the minutes -- there are minutes and they publish the minutes to Unicode.org -- were it was not visually distinct, which is not totally crazy. We actually worked with an illustrator to get a different image. The first image was almost at 90 degrees. It was kind of straight up and down and it is a little hard to see and the second is -- ROBERT: Especially, because it's thin. AMANDA: The second image is actually a kind of stylized needle because it's fairly a little fatter and the eye is bigger but it's much more distinctively a needle. I'm hoping that that will also convince them but you have to be able to tell at a very small size that it's a needle. The other thing that they said was that sewing was already represented by the thread, that we didn't need thread and needle but it was literally one line in the minutes that referenced that and then it sort of like, "Did you have somebody in the room or not?" and so, if there is somebody on the committee who is willing to tell you really what their concerns were, then you have some sense of what they're looking for and why they're pushing back. When you can very much see in the earliest emoji character sets that I have a hammer and I have a wrench and I use them but there's these very conventionally male tools. We have all of the kind of office supplies but all of homemaking and housekeeping and textile production, none of them were there until very, very recently. I think it does reflect the gender of the people who've been making these tools, that sewing and knitting weren't important enough as human practices to be included in this glyph set. AMBERLEY: I guess, that's non-trivial to mention because that wasn't an argument that I made in my original yarn draft and Amanda and Jenny sort of pushing to open it up to this whole slate of craft emoji. I didn't realize until they brought that up. I took a stroll through pretty much the whole slate of emoji and you can count on almost one hand the number that represented the creative endeavors or sort of more traditionally known as creative things like camera or painting palette and stuff like that. It was extremely limited. AMANDA: I think they have stuff like that. I think there's a few different variations on the camera and then there's painting palette and that's it. AMBERLEY: Oh, there's the theater mask. AMANDA: Oh, that's right. There is the theater, the happy and sad -- AMBERLEY: And I don't know it exactly and I haven't read the minutes like Amanda has but I think and I hope that that was a particularly compelling piece of that argument. AMANDA: I think they definitely heard it. AMBERLEY: Yeah. CHARLES: Opening it up then, what else is coming in the way of craft? It sounds like this is historical but these pieces are being filled in not only with the work that you all are doing but by other emoji which you're appearing. AMANDA: Yes. CHARLES: And are you in contact with other people who are kind of associated with maybe craft and textiles and other kind of what you're labeling historically creative spaces? AMANDA: I don't think there are anymore with a possible exception. Someone's working on a vinyl record proposal which I think is great. CHARLES: Yeah, that's awesome. ROBERT: Antiquated, though. AMANDA: Maybe not, I don't know. AMBERLEY: Take a stroll through the Emojination Slack and people discussed that. AMANDA: Yeah. If you click at Emojination.org, the whole Airtable database is on there. There's not a lot of other creative ones. A friend of mine got really bent out of shape about the lack of alliums and wrote a whole slate proposal for leeks and scallions and garlic and onions. ROBERT: Oh, there is a garlic one, right? AMANDA: No. I mean, there is -- AMBERLEY: Actually, I'm looking at the Unicode page for current emoji candidates. They first get listed as... I forget the exact order. They become draft candidates and then provisional candidates or vice versa but I don't see any pending further creative ones but garlic and onion are on there. AMANDA: Yes. ROBERT: That makes my Italian a little happy. AMANDA: I think there's some prosthesis, the mechanical leg and the mechanical arm, a guide dog -- AMBERLEY: Ear with hearing aid, service dog. AMANDA: Yeah, there's a good chunk of interesting things that have been left out. I guess they've been approved by the subcommittee but are still waiting on final approval by the Unicode Consortium. ROBERT: Okay. What are the next steps that we can do to help push the thread and needle proposal through it. You mentioned a couple things like coming up with idioms that are in different languages and whatnot but how can we contact you and push this effort and help? AMANDA: That's such a good question. I don't even know. I mean, I am Amanda@velociraptor.info and you're totally welcome to email me if you want to help with this and I will -- ROBERT: That's a great domain, by the way. AMANDA: Unfortunately, there's no information about velociraptors anywhere on that site. ROBERT: That's the way it should be. AMANDA: But also, if you're excited about working on emoji proposals, Emojination is an incredibly great resource and folks there, including me actually will help you identify things that are on other people's wish lists that you could work on if you just want to work on something and we'll help you refine your proposal if you know what you want and we'll help you figure out whether it's worth putting the time in or not and how to make it compelling. You can definitely check out Emojination.org. I think there's a path to get on to the Slack from there. AMBERLEY: Oh, yeah. The Slack and the Airtable. AMANDA: Yeah. ROBERT: It sounds like there's a whole community that was born out of this, where everybody is trying to help each other and collaborate and get their shared ideas across. AMANDA: Definitely and there's a woman, Melissa Thermidor who is fantastic, who actually is a social media coordinator. It's her actual title but she works for the National Health Service in the UK and was tasked with getting a whole series of health-related emoji passed. There's a bunch of things that she's -- AMBERLEY: Is she's the one doing blood. AMANDA: She's doing blood. AMBERLEY: That's a good one. AMANDA: Because there's a lot of really important health reasons why you need to be able to talk about blood and getting blood and blood borne illnesses and -- AMBERLEY: That one was listed on the emoji candidate page or blood donation medicine administration. AMANDA: Yeah. ROBERT: That's really interesting, so she works for the government, right? and that was part of her job to do that? AMANDA: Yes. ROBERT: That's awesome, actually. I love that. AMANDA: Yeah, I think the drop of blood, the bandage and the stethoscope are the three that are in the current iteration, which is interesting because the existing medical emoji were the pill and that gruesome syringe with a little drops of fluid flying off of it, which do not do a lot to encourage people to go to the doctor. ROBERT: No, not at all. AMANDA: So a few more, we're welcoming medical emoji. ROBERT: You have a GitHub. Is that where you're still doing for the follow up and the prep work for the sewing emoji? AMANDA: Yeah, that's probably the best place. I do have a Google Docs somewhere but that's probably a better place to connect even than my ridiculous Velociraptor email. The GitHub -- ROBERT: But it's still awesome. AMANDA: It is awesome. I won't lie. I'm very proud of it. I am AmandaBee -- like the Bumble Bee -- on GitHub and the sewing emoji, the original proposals are there and I will make sure that there is information about how to plug into the revised needle proposal there as well. You guys are a tech podcast, so if people want to just submit suggestions as issues on that repository, that's awesome. We'll totally take suggestions that way. ROBERT: That would be pretty rad. Well, I appreciate you two being on the podcast. I love hearing your stories and how it ended up converging in parallel tracks but it end up achieving the same goal. Still unfinished, right? Let's see if we can help push this over the finish line and get it done because I would really like to see a needle. I could definitely use that in many of my conversations already now, making all kinds of puns. Thank you, Amanda for coming on and sharing your story. AMANDA: Thanks for having me. ROBERT: And thank you, Amberley for also coming on and sharing your story. This was super awesome. AMBERLEY: Yeah and thank you for connecting us to finally have a voice conversation. AMANDA: I know. It's great to actually talk to you, Amberley. CHARLES: Oh, wow, this is the first time that you actually talked in audio? AMANDA & AMBERLEY: Yeah. ROBERT: We're making things happen here. The next thing we have to do is get this proposal through and accepted. AMANDA: Yes. CHARLES: You've converted two new faithful sewing and needle partisans here and I'm in. AMANDA: Awesome. ROBERT: I know you've already gotten, what? Three through accepted? AMANDA: Yeah. ROBERT: We talked about that, it's got to be really awesome. I think I want to try and jump in and get that same satisfaction because a lot of people use emojis. AMANDA: Exactly. CHARLES: It definitely makes me think like you look at every single emoji and there's definitely a story. Especially for the ones that have been added more recently, there's a lot of work that goes into every single pixel. That represents a lot of human time, which I'm sure you all know, so thank you. AMANDA: Thanks for having us on. AMBERLEY: Yeah, thank you guys. ROBERT: Cool. That is the podcast. We are Frontside. We build UI that you can stick your future on. I really love this podcast because it wasn't necessarily technical but had a lot of interesting conversation about how to work with a proposal and probably make a bigger impact than any of us with software, just because the sheer reach that emojis have are insane and the fact that you can influence this process is new to me and really cool, so I hope a lot of other people learn from that too. If you have any feedback that you would like to give us on the podcast, we're always open to receive feedback. We have our doors and ears open, so if you like to send an email at Contact@Frontside.io or shoot us a tweet or DM us at @TheFrontside on Twitter. We'd love to hear it. Thank you, Mandy for producing the podcast. She always does an amazing job with it. You can follow her on Twitter at @TheRubyRep. Thanks and have a good one.

In this episode, Robert, Charles, and Wil talk about the whys and hows of accessibility, as well as what makes single page applications special, why they are they harder for accessibility, and frameworks that can do this for you. Resources: #SkyQ app on #iOS from a #VoiceOver user's Perspective Rob's Routing Doc Wil's PR Single Page Apps routers are broken Greater Than Code Episode #92: A11y Ally with Rob DeLuca This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. Transcript: ROBERT: Hello everyone. Welcome to The Frontside Podcast. This is Episode 111. I'm Robert DeLuca, a software developer here at the Frontside and I'll be your episode host. Today, we're going to be discussing accessibility in single page apps. With me as co-hosts are Charles Lowell. Hey, Charles. CHARLES: What is up Robert? ROBERT: And Wil Wilsman. Hey, Wil. WIL: Yo, yo, yo, yo. ROBERT: Sounds like we're ready to drop a disc track. We're not going to be dissing anybody here. We're going to be talking about helpful things with accessibility in single page apps. Before we get into the nitty-gritty of accessibility in single page apps because we're getting into some deep stuff, I think I want to cover a lot of 'how' because I know accessibility things are usually about why you should be doing it and then they touch on things like, "You should be using alt attributes for your images," but for single page apps, I think we need to go further. CHARLES: It always ends up like, "Then draw the rest of the accessibility owl." ROBERT: Yeah, here's your two circles and the alt-tags are your circles and then the rest of the freaking owl is focus management and everything else that comes with it. Before we get there, what is accessibility? I guess, if we trim the giant umbrella down a little bit from everything that is accessibility that can be physical space things, like wheelchair accessible ramps or things like that, what about just technology? WIL: People need assistive tech to interact with technology such as switches or keyboards and obviously screen reader is a big one but when it comes to the software itself, you could even talk about colors and people who are color blind, so red might not be red to everybody. ROBERT: Speaking from experience there? WIL: Yeah. I'm colorblind. Red is brown to me. CHARLES: Things like hearing and all of it, right? It really is like just designing it in such a way that it can be used by as many people as possible. ROBERT: Right. WIL: And that includes your mom who may not be the best with technology but she still needs to pay your bills online or something. ROBERT: Exactly, yeah and in for context to listeners that might not know, my mom is 100% blind, so it's kind of where it comes from. CHARLES: But my mom is not but she has all kinds of problems. WIL: Yeah, same with my mom. CHARLES: That also falls under the category of accessibility, right? ROBERT: Absolutely. CHARLES: Right, accounting for age and culture. ROBERT: We're blending into the why of accessibility, which is perfect. One of the things that it's such a good segue because what people are starting to realize and I think why accessibility is really starting to catch wind and get some traction is because a lot of people that grew up in the technology age were open in using technology a lot. Our parents probably did not use technology heavily. That's definitely the case for my dad. He still has a flip phone and he says, he's a low tech man living in a high tech world and just refuses to pick up technology but those are the people that just didn't use technology. But now we have a lot of people that grew up with technology and use a lot and they're aging into more disabilities and they're going to need that accessibility, which I think is really interesting to think about because that's a lot of buying power if you're just going to start moving in that needs in accessibility, right? CHARLES: That's true. I know I may need glasses pretty soon, so colors and fonts were going to be heck a lot more to me in the next five years than they have over the last 20. ROBERT: Yup, exactly and that's going to be huge for that and that's one piece of the why, so what are the other reasons that you'd pick up accessibility other than people saying that it's morally correct. I don't like starting off conversations for accessibility because it's the thing that you should be doing. WIL: I think it goes even to my user experience like power users that don't like using the mice or mouse. That's me. I really prefer to just use my keyboard for everything. When the new Firefox browser came out and I couldn't navigate through the menus the way I was used to, I went back to Chrome. ROBERT: That's interesting. CHARLES: I still have not found a good workflow for navigating tabs with the keyboard without just kind of twisting my wrist all out of shape. You have to share that with me but again, that's another thing. That's an impediment that sits between me and the application, that actually -- ROBERT: Which is really interesting, you're getting into a keyboard navigation and focus management, which is really the crux of accessibility for screen reader users and switch users. CHARLES: What I'm hearing is that in this case, including good keyboard and focus management in your application, e.g. making it accessible to screen readers, you at the same time, enabling your power users. I think a great point is that by introducing these very low friction workflows, you're actually going to be enabling other parts of your customer base and not just catering to one, that there are ripple effects throughout your system. ROBERT: It may set it for everyone. WIL: Yeah, not only people who need it but people who don't know they need it. ROBERT: Yes, exactly. Think about the WCAG spec as user experience guidelines. They're not telling you how to implement a thing specifically for a screen reader. They're telling you how to implement it in a way that works for everyone, regardless of what ailment they have. It could be a temporary ailment. It could be a permanent ailment, where they have to use a screen reader or any kind of ailment that you can think of. They're not considering just one use case. It's a broad thing that shows how you can make your application better for everyone. I think that's a better way to look at the WCAG spec than I need to read through this and make sure that this auto complete works for a screen reader. If you look at the guidelines, they're not telling you just first screen reader. They're telling you how to make it work for a switch, someone who is colorblind or who is using a dictation software. I kind of tend to look at that as UX guideline, which really helps me build a better app overall because when you nail down that user flow, everyone benefits from it because it's pretty well thought out at that point. CHARLES: I like that too and I think that thinking of it as user experience and making sure that you have a complete user experience is a good way to think about it because it kind of separates the concerns of, "I've got HTML but is my application really HTML or is it a set of workflows and the data over which those workflows operate?" It really forces you to think my application is not a set of React opponents or web components or Ember components or what have you but really, there's a deep structure to it and it makes you kind of shine a light on that deep structure and try to map its surface. Then, if you really, really know it and you capture it, then you can represent it in any medium. I think it's just a win not just for one niche group of users but also for all of our users and then also for your future users that you don't have because your application is designed better and is going to work. Who knows? Maybe there's some new interface or some new medium, some new device that comes out that hasn't even been invented yet. But if you really have a strong internal representation of what your application is, you're going to be able to be the first to move to that market. ROBERT: Right and like I said earlier, people are starting to aid in the needing accessibility thing. If you need a dollar to justify it, that's going to be a big reason coming up. As a part of the new WCAG 2.1 spec, there is a zoom. I forget what this access criteria number is but the new criteria says your app basically needs to be responsive. That kind of maps directly to what you said earlier, Charles which is like you're going to need glasses soon and fonts and being able to zoom. It's going to be really important. We see that just pop up everywhere. To give a counter example of not just for accessibility like you need it for glasses but the other side of that could be like when we give demos on low resolution projectors or screencast or anything, when we zoom the screen, the apps just still be usable and you should still be able to demo it and that's just something that you have both sides to that, where accessibility kind of works out for everybody. People on the call probably don't need glasses but to see that tiny screen that's being shared, you should be able to assume that. CHARLES: Right. I'm just kind of restating what you said but I want to make sure to call it out explicitly because it was a definitely an aha moment for me, that you basically if you build yourself an accessible website, you build it so that it works on projectors and mobile devices too, so you kind of killed two birds with one stone and you don't have to make a special effort because zooming in the screen is tantamount to viewing it on a phone or viewing it on a tablet. WIL: Yeah, we mentioned it earlier about accessibility and physical space and one of those things is being able to access something from anywhere no matter the device that person is using. ROBERT: Yes. That's a lot of the 'why' of accessibility and I think we did a pretty good job of staying away from the more argument because morally we don't know if it's the right thing to do but a lot of the times it's not in front of you, it's hard to do and I want to make sure that you don't feel bad if you're not building accessibility in your apps. It's not easy. I don't sort of like people get up and say that, "Accessibility is easy. Just do this," because it's not -- WIL: If it was easy, everybody would do it. ROBERT: Exactly. I always come back to accessibility being just like UX -- user experience -- because if it were easy, everybody would have a really great user experience too. It's a hard thing to boil down and to simplify it, right? CHARLES: Right and like every other aspect, the kind of nonfunctional requirement, I say nonfunctional requirement that's a little bit of a contradiction terms but it becomes very hard if you haven't done it from the beginning. But if you didn't start with an accessible app, you weren't thinking about that or you inherited the app or it just wasn't on your radar for whatever reason. If you got a codebase that's two years old, going back and trying to make it accessible, it was extremely hard. It's expensive, expensive, expense. WIL: Yeah, it's going to cost more in terms of money and time to added it after the fact. ROBERT: Exactly. I spent a year on helping on Visa Checkout and there were some accessibility considered in the designs but a lot of the time my feedback wasn't just like, "Yeah, sprinkle some ARIA attributes on there and you're good." It was like, "Do we really need a carousel here to represent your list of carts because it's really hard to navigate around that?" A lot of the times, it ends up boiling up to is this the best way we can represent this data? Is this the best way we can navigate and build this user flow? A lot of times -- CHARLES: And if the answer is no, it's so painful, right? ROBERT: Yeah, exactly, so all that work that was put into building that carousel and all the components that built that carousel together is thrown out because it's just not a pattern that should be used there. Doing after the fact is really hard and really expensive and usually ends up in refactoring anyways. CHARLES: I just want to point that out because a lot of people find themselves in that situation, where they're staring down at a pretty big cost. Now, the reasons why your app may not be accessible or many and good and you shouldn't feel bad, if you're finally in that situation. ROBERT: I gave a talk at Nodevember a couple of years ago called Accessibility Debt and it's just like any technical debt. Your apps going to have it. It's okay. Don't get beat up about it, especially if anybody is trying to beat you up over it, don't listen to them. It really is just like any other kind of technical form of technical debt. It's something that you'll have to deal with and it is just something you have to work through. It's not world ending. It's just another problem to work through. What are some things like everybody usually talks about like the things you can do, the basics for making your app more accessible like using all its attributes and instead of doing a div with an on-click handler, just use a button or don't overuse ARIA attributes. Are there any other things that I missed there like the basics of accessibility? WIL: The biggest is the HTML structure. Screen readers and other assistive tech were built with the standards in mind, so if you're doing nonstandard things like putting divs in H1 and adding ARIA attributes within there, you're not going to have a great time. ROBERT: Right or splattering ARIA roles all over the place, probably not a good idea. That will be harder to debug. Also fun fact, ARIA roles, while you can implement directly to the spec, you may still have bugs across the different screen reader combinations or assistive tech combinations, so that's fine. CHARLES: Keep it super simple is what I'm hearing, like use semantic markup. If you're going to introduce a custom button, still make sure that it's a button. ROBERT: Use the platform. CHARLES: Yeah, use the platform. Don't fight the platform. Probably the best example of that is people implementing their own select boxes. That's the classic example. ROBERT: Wil and I, our lives has centered around that for a little while. It's so true. Usually, it's the first thing people go to grab to reimplement because selects are just ugly. I think Firefox has the ability for you to style select options now like you can change the color and the font but you can't style that. The pop up that comes, usually that's the system dialogue, which a lot of designers don't really like. That's usually the first thing that people go to implement and that's usually actually the first thing that stop somebody from signing up. A lot of sign up forms that I see, if your date of birth is in a select format, that probably will hinder somebody that uses assistive tech from signing up. CHARLES: Yeah. You just basically bounced that entire person. The thing is people don't appreciate the cost. This gets into the whole concept of accessibility that is how much money would that person or those group of people have actually brought into your site versus the cost that you spent redoing that select box. You might be thinking, "It only took this developer actually two weeks," but when you actually look at the actual cost over the course of your application, you're not factoring that into the decision to go with custom select box. Just in our experience, it's just the truly low cost option per quality of experience, that tradeoff there is almost invariably going with platforms select, right? ROBERT: Yeah. Your secret sauce and the best UX that you're going to provide is not going to be nicely styled select box and seriously, a battle that I had to fight a lot was if you really want to implement a custom widget, decide if this is what you want to spend your time on because custom widgets aren't just quick and easy things you implement. You're not going to implement the select that's fully accessible across all the AT combos in a couple of days. CHARLES: It's a lot of work. ROBERT: Yeah. You're going to fall down on a huge rabbit hole. CHARLES: Yeah. It is just you're committing to that work over the lifetime of your application. ROBERT: Exactly, if you can maintain that now. If you implement custom check boxes and custom radios and custom input that of content edible for some reason, I think -- WIL: I think what I see is like custom date pickers -- ROBERT: Oh, I just had a rant about that. CHARLES: Date pickers are hard and there's not really a good option. You just have to open your eyes to the true cost. ROBERT: Right, exactly. That's what I always try to explain, just like you have ownership over this now and you now have to maintain this and you can't regress. CHARLES: Right and if you do regress, it's your neck that gets choked. ROBERT: Yes. A good way to put it. We've talked a lot about things that are just general accessibility but nothing specific to single page apps. I do want to say like a lot of other things that people recommend is like if you're using React, use like the JSX ES1 plugin to help you analyze if you're writing any JSX that might not be accessible or use like HTML_CodeSniffer or aXe to statically analyze the DOM that you have. Those tools are great. I'd see a lot of people [inaudible] those things as like, "Look, this automated checkers says I'm inaccessible so I'm inaccessible," and that's not the case, especially in a single page apps. You can have 100% passing automated checkers but your app also could be 100% broken and why is that? WIL: I don't know. ROBERT: I'm wobbling the router question here. WIL: Yes. I guess that would just be due to routing. In single page applications, they handle their own routing, whereas static web sites and whatnot, the routing is handled despite URLs and the browser and reloading pages. ROBERT: Right. There are probably other major differences between single page apps but the biggest thing is the routing is managed by the client, the JavaScript and on a server-rendered page, you click a link, it's going to rerender the entire page for that next page and the screen reader and the browser know exactly what to do with that, to put the focus back to the top of the page and start working through the page for you. But with single page apps, you're just replacing a piece of the DOM and the screen reader has no idea of what's going on. An automated checker cannot check for this. They cannot tell you if your routes are accessible. The reason I'm bringing this up is because this has never talked about. I don't see in accessibility talks and this is the thing that actually is most broken and makes your app pretty much unusable to anybody that's using assistive tech. There's ways that people if they're savvy can navigate around it but if they don't know what's going on, they're going to think, "All right, I pressed this button and nothing happened, so I'm just going to leave now because it's not working." CHARLES: There's a great video that you point to me, I believe a guy from the UK who has recorded a bunch of his experiences using websites and apps -- ROBERT: Yes, I want to dig it up and put it in the show notes. CHARLES: Yeah. Those are great to watch because you'll really get it. ROBERT: Yeah and that's a really unique case because he's really savvy. I believe, I could be wrong but I think he might have said that he definitely work in tech somehow -- CHARLES: Right. He knows the workarounds and he knows these things but in some of the cases it's like, "If I didn't work in tech, there's no way that I would be able to use this website." ROBERT: Yeah. This is kind of the crux of like if you ever listen to anybody that is an accessibility consultant, they'll say, "You will never ever be able to automate accessibility," and this why I tie accessibility so much to user experience, would you ever have a user experience test that can tell you in a binary fashion? Yes or no, that your app has a great user experience? No, because it's pretty subjective. CHARLES: Of how does your users feel? right? ROBERT: Yeah. CHARLES: You can rate, "My users feel great." ROBERT: Accessibility is like that because there's a lot of context that you have to carry around. It's all about context. When I transition from this page, the next thing does the user have enough context of where they're coming from to where they're going to be able to operate on that page. Is there enough information to achieve the task they want? That is pretty much the crux of why there is no binary yes or no for that because it's contextual. It varies from person to person but you do your best to make sure that that works and that you provide enough information to do something. That's kind of a single page app as a crux. This is why we have a philosophy of testing as a whole. We don't test components individually because again, you can make all of your components individually accessible there but as a whole, they might not work together because you're not providing enough context on an entire page. We did this in one of the apps that we work on, which is open source, so we can link to the PR that Wil wrote for this and I wrote an entire routing documentation around our philosophy and the different things that we tried. How did we manage the focus in that application? I'm kind of just going to lob it over to you Wil since you did the work. Do you want to give context of the holdings and how that all kind of came together? WIL: Yes. One of the features of the app is these panels. It's like this three panel system. When you click an item in a list, a third panel pops up and this goes back to the context thing where if you're using a keyboard, you can't see the screen. You click an item in a list, how do you know that third panel popped up? The solution isn't for a component. The component can't be responsible for this. The list can't focus the item. It opens or vice versa, so this is definitely an application concern, where we needed to check against the route and see whether or not, the pane is opening or was already open or wasn't open before and when this pane opens for the first time, it will just focus it and that gives a lot of the context that the user needs when they click it, like they click an item in the list, it focuses this third pane and they're on a third pane. ROBERT: Right. We didn't even just focus on the entire div. We focused on the heading of the thing that you selected. WIL: Yeah because focusing the div, it might read something off but not all the time. The main thing we're focusing on that third pane opens is the heading to let them know that the item you click, you are now on that page and reading that heading. This is the same thing that would happen if you loaded up that page statically and the screen readers would usually just focus on that first heading. ROBERT: Right. That helped a lot. For a little bit more context, the middle pane is like kind of a master detail thing going on here and in the middle pane there that we have, it's an infinite scrolling list. You have different things there and one of them is like a package. If you select the package and without the focus management and focusing on that heading, you would have to go through every single package that's in that list which could be a thousand of them before you actually get over to the pane that you just opened because source order. You have to go through each one of those. WIL: And it's the same thing is true for power users, not just screen readers. It's like if they want to use tab once that pane open, they have to tab through the entire list. ROBERT: Exactly. It wasn't keyboard accessible and it made it really hard to navigate around with a keyboard because the focus just wasn't being managed. There was a lot of work that we did there. I want to focus on the routing situation there because if you can't navigate around with a keyboard in your single page app, like you click a link and it's not selecting the next thing that should be focused and you provide the right amount of context, your app probably won't be usable without a lot of trial and error to a user, which depending on what your product is, they may not have a lot of patience for trial and error, right? WIL: Yeah. ROBERT: It's really important to try and nail down the routing situation and there are some frameworks and things out there that can do it. In the app that we're talking about, it's React app and we use React router but we don't actually really hook into the router to handle that and that's because React router doesn't provide very much information. WIL: Yeah. You can think of React router as more of an outlet system, where your routes can render anything, anywhere on the page. That's kind of dangers of accessibility and that's kind of the reason that React router can't handle accessible things very well because at any point, the route can just change the button on the page to look like something else. CHARLES: Yeah. It just pop in and pop out. There's no deeper model, right? There's not -- WIL: Yeah, there's no tree. CHARLES: Right, there's the internal state -- WIL: Like a component tree, yeah. CHARLES: The internal state of what is happening is completely opaque. You can only analyze the second and third order effects of the React tree. ROBERT: Right and especially if you have nested route components, it's really hard to determine. One of the things that I've seen people do is focus on mount, which is a very naive approach because what if you have three nested routes, they're all going to focus on mount and the last one that mounts wins and that's a focus for which nobody wins. CHARLES: You all tried that, right? ROBERT: Yes. That's part of the document that I wrote up. We tried three different approaches and we ended up landing on something because we were using React router that was more of like an application state thing, so we were checking props because we knew what the user flow was. We knew what the user, when they come to this page is trying to accomplish, so we're able to kind of figure out from where they came from or where they're going through props and focus the right things for them. WIL: One of the examples, like we talked about the third pane opening and focusing the heading inside but what you know what happens when you close that third pane, you kind of lose contexts again, so we have to focus the previous list item that was active because if you focus back at the top of the list, they've essentially lost their place and the list of results. In that case, we couldn't use on mount. That list item is already mounted. We have to listen to props and we have to look at the route through these props to determine if that third pane was open. If it just closed and if an item in the list is active, it should have focus. It's a lot of logic going on. You have to really understand your app in order to make it a very good accessible app. You can't just sprinkle in ARIA attributes and focus on mount everywhere and think it'll work fine because you're accessible. You have to really know the flow. CHARLES: Right. All those signposts point towards having a deeper application state, a deeper understanding of your application than just the render tree. At that point, it's too late and so by that, I'm definitely lobbying a Reach/Ember router. WIL: Yes. We talked a lot about the React router and we can't really be too accessible with it but to create a React router to go out and there is now Reach router. Robert, have you heard any good things about that? You're the accessibility expert here. ROBERT: I haven't played with it myself, so [audio glitch] things that were lost from the React router three to four transition, I think and also, it's actually accessible routing which is nice because it comes out of the box and you know how to implement it. I haven't played with it. I know Gatsby has implemented that for their V2, that's their default router now, which makes me really happy because a lot of static sites that were being built with Gatsby were very inaccessible and broken which made me sad but now, they're not with V2. I haven't played with Reach router but one of the things that I think it provides which was missing from React router was transition hooks. It has a concept of like where you're going from and to for your routes, which really helps figure out what you need to focus. The one thing I will say about Reach router and I'm sure it's got to be configurable somewhere but I haven't really looked and the demos that I saw, they were just focusing the div of the content that's being rendered, like it kind of just wraps with a generic div or the tabindex="-1" and then focuses it. If you have a lot of content that's inside of that div, it probably will be confusing but at least, it manages the focus somewhere. If you're now off in a no man's land, you have no idea what's going on, at least it focuses that. But if you want to really nail the experience to be better, I would recommend trying to figure out what you should focus inside of that route that you just transition to, what is the best thing. That's for React. There are other things out there for other frameworks, like I am on Ember accessibility team that's out there and we have Ember A11y, which just provide you a focusing outlet for you to be able to just drop in your app and then when the route changes, it does the same thing. It focuses the wrapping div of that outlet that was just rendered. I want to emphasize more in talking about e-holdings work that we did that we were focusing the heading because that told you exactly where you're at and what package you're on or what thing you're on. You know the name of it and now you know how you can go and navigate through that. CHARLES: But it's conceivable, so how would you do that with the Ember router? Would you just introduce some way to delegate down to a particular component? Like when I'm rendered into an outlet, it focus on this component? ROBERT: That would be interesting. I actually don't know. It's been a long time since I've messed around with the Ember router and Ember accessibility. It's definitely a great first step and that's where it kind of came from. I think there needs probably some work done to help implement on what you want to focus. That's kind of where I was going with when we were first exploring the React router stuff and e-holdings was I wanted to have this like high order component that knew of your routing tree and it knew where you're coming from or where you're going. Then from there, it would just tell you, this is the route that we're going to and there's need to be focus management done, like if this prop exists, then you can pick inside of that component that what you want to focus. It leaves it up to the implementor of what they need to focus but it could be probably just like a fallback to the general div because that's not bad. It's a good first step. There needs to be a little bit of work there to get that done probably. CHARLES: Yeah. But it is worth pointing out that by starting from a position of having an externalized application state via the route structure, in an Ember application, you're starting 95% of the way there. An Ember application, at any point, you know where you are and during your transition, you know where you're starting and where you're going to end up and you know when you leave and you know when you got there. ROBERT: Yeah. Having an Ember route pivot handler there, like where you're pivoting from and to is just so nice and it kind of made it click together a lot easier. CHARLES: Right. Reach router looks interesting but as I understand, it's still couched in React components and it feels to me like this is a problem that ought to be solved, that ought to be framework agnostic. Because if I use something like Reach or I use some other routing library for another framework, some other tool might come out that I want to use and I shouldn't be locked in -- ROBERT: Right like what if I really like Redux little router, then I have to make a choice between Redux state that I like or accessibility. CHARLES: Exactly and that feels like a false dichotomy to me. What I would want to be looking for is a platform independent or a framework independent routing library that really just helps you represent your application state and the concrete state in which your application is in at any moment and then, also be able to represent the transitions between those states fully and completely. Then if you have that, you could embed that into any framework. ROBERT: Right. That would be really nice to have. Just like pull it off the shelf and help you out there. CHARLES: If anybody is listening who wants something to do for the next 18 months, no one will thank you for 18 months but you had to get on that. We'll give you lots of thanks then. ROBERT: [inaudible] compare with you. That would be awesome. CHARLES: I would love to be part of that. ROBERT: Yeah, that would be awesome. To kind of tie back to other things, I don't know too much about Angular but I'm sure there are solutions out there to help with Angular. I know Marcy Sutton used to do a lot of work in the Angular world, so I'm sure there's something out there that helps with that. I just don't know off the top of my head right now. I wrote a Medium 'think piece'... No, I wrote a little medium blog post about how all of single page out for routers are broken. I was pleasantly surprised by Vue. Vue brings its own router and their router has a concept of before each, so before each route transition you can run some code and that really helped with implementing the live, the announcer pattern where you use ARIA live to announce something but even beyond there, if you wanted to dig in there, you could probably figure out where you're transitioning from and to and give that next route some kind of attribute that says, "This needs to be focused. Figure out what you need to focus there and inside that route, you can focus wherever you want." I thought that was a really awesome. That's kind of the crux of what I was missing from React router. I wanted the concept of knowing where I'm coming from or where I'm going and I would help with everything but unfortunately, that kind of doesn't exists because they're just components. For better or for worse, they're just components. CHARLES: The world is so much more than just components. ROBERT: Yeah, a little bit off topic, I think it's kind of funny how React kind of just shoved everything into the Vue layer, just to make it all a component. CHARLES: Yeah, it's very easy. ROBERT: Yeah, until it's not. CHARLES: Yeah, exactly. It's easy but it's not simple. ROBERT: That's a lot of talking about how routes need to be done and what you can kind of do to manage focus. It's really about managing the context and how you can provide the most contexts. For somebody to operate on that information, can they complete this thing? What can you do to make sure that you've done this properly? What steps can you take to make sure that you actually are accessible and that your routes work? WIL: Manually testing those screen reader is probably the biggest thing, you know? CHARLES: Yeah. I think the biggest thing is really watching someone who uses assistive tech on a regular basis use your application and then trying to use it yourself. WIL: Yeah. ROBERT: Right. Yeah, I'll definitely -- CHARLES: If you can get a little bit of this yourself but then it's kind of like someone who is never used a mouse before and trying to learn something new. What really helps is seeing someone who's good at it and see how their expectations are either being met or not being met. ROBERT: Yeah, it's definitely the best way. If you can find somebody that actually relies on assistive tech, there's nothing that beats that kind of feedback. If they get to your app and they're really confused, I see some people that just dismiss it because they just don't understand. That is the best feedback you can actually get. If they don't understand what's going on, you have -- CHARLES: That's on you. ROBERT: Yeah, that is going to be what happens for everybody that uses that. Well, maybe not everybody because everybody has different experiences but it's probably going to be a thing that pops up everywhere. But if you don't have access to people that are actually using assistive tech regularly and are pros at it, WebAIM provides really great tutorials for how to use a screen reader. If you're using a Mac, you have a screen reader built in and you can use that called VoiceOver. If you ever want to turn it on or turn it off, its command F5. WIL: You might have to have that shortcut enabled, though. ROBERT: Really? I'm pretty sure it's quite default. WIL: I thought I had to enable mine but I could be wrong. ROBERT: It's interesting. CHARLES: Yeah. They got great tutorial too. It's like it notices the first time you turn it on, so it tries to help you navigate bullet lists and select boxes and input fields and check boxes and all kinds of good stuff. ROBERT: Yeah. It gives you a little bit of a boot camp but WebAIM also helps with specific to website and stuff because one of the things that we ran into while working on the e-holdings project is they're transitioning from a native app to a web app and there were just things that you can do on a native app that you cannot do on a web app. Just keep that in mind also when you're testing, there's just things that will behave differently. Like you're not going to have a lot of crazy key shortcut commands like you're not going to press command F to get to the search box if your app has a prominent search box because that is going to clobber a bunch of other assistive tech key commands. WebAIM is really a good help with giving you the tutorial of how to test a web app and many different screen readers, so they have it for JAWS, they have it for NVDA, they have it for VoiceOver iOS, they have it for TalkBack, which is the Android screen reader. There's a lot of really good resources there for you to start using as screen reader and test with your app. I highly recommend using it with a screen off. You gain a lot of context by looking ahead of your cursor and -- CHARLES: Yeah. It's true. ROBERT: -- The example that I usually give to people is like, "I worked on Visa Checkout. I went through that checkout flow pretty much every day for a year and eight months in, even then turning off the screen, I was still lost." Even though I knew what each screen was and what each component was there, I would find myself confused of like, what just happened and it's because sometimes, you'll get into something like you have a dropdown that sets the focus inside the dropdown and then the dropdown disappears from the DOM and your focuses in nowhere. You're like, "What is going? what am I doing. I don't even know where I'm at," and I turn the screen back on and I'm like, "Oh, now, I know what's going on." CHARLES: Right. It really is a lot like interacting with your application as though you were interacting with Siri or Alexa but with a keyboard instead, instead of voice commands. That's an excellent point. We would understand if where would Amazon be if Alexa couldn't successfully navigate those situations. The counterpoint or even the flip side of that is if you model your web application in such a way that it can handle that type of serial interaction, instead of the highly parallel environment to being able to perceive huge amounts of information concurrently, like you can on a screen, if that were effectively serialized, that means you could represent your application through nothing but voice commands. ROBERT: Yeah. Did you provide enough context for me to build this mental map is really what I'm going on to? CHARLES: Yup. ROBERT: Which I always thought was really interesting because I always wanted to know how my mom visualizes what a website looks like because it's wildly different than any of us. She's never been able to see what a website looks like. Does it look like a bunch of nodes and graphs and webs connecting to each other and how things pieced together? It's just a different way. But it's not only about screen readers, right? You can use a keyboard to navigate and that's definitely what we did with e-holdings is like can we tab through this [audio glitch] list, hit enter and go into the detail record, edit it, close it and go back and edit another one. Is that something that we can do with just a keyboard, not even a screen reader? WIL: And with the keyboard, we're not talking about shortcuts to hit edit. We're talking about like tabbing over and hitting enter like people with accessibility issues would have to do. ROBERT: Right and that's kind of a good segue into creating use cases. If you want to know if your app actually works, if your screen reader users or if your keyword users or if your dictation users are going to be able to navigate this app, create use cases. Things like actual user flows like how would somebody actually going to use this app? What task are they going want to complete? In that case, e-holding was like an electronic holdings management system for libraries. They probably want to get in there, add a couple of books or whatever you might have to their library and get out. A use case could be like, "Can I search for this thing? I'm going to search for something specific. I'm going to go through the list, find the thing that I want. I'm going to add it, close the pane, go back and then remove one thing and can they complete that flow successfully without running into any issues or any blockers or any showstoppers." I can tell you before we did the routing management stuff, you would hit search and that was it. There was nothing else that you could do. CHARLES: Yeah. They wouldn't even announce that anything can happen. ROBERT: Exactly. WIL: Yeah, and with these librarians, it's not necessarily a matter of they can't see the screen. It's just that they don't use the mouse because they're power users. ROBERT: Right. They don't have any disabilities but that was essential to the workflow. CHARLES: Yeah, exactly. Do that change. You just lowered the activation energy of that workflow by... What? three orders of magnitude? WIL: Yeah, at least. ROBERT: Let me click here right now. I can tab, now I can tab. Right now, let me click here and now I can tab, I can tab, I can tab. It's not as nice as just being able to completely do it through a keyboard. Through us making it super keyboard accessible, that also became super screen reader accessible and the people who use dictation were able to work through and get through the app and use it now, which was really cool. It really helps when you go for those things and create use cases to really figure out how a user is actually going to work through this app. That's the best way. Just get right through it. With Visa Checkout, we're like, "Can somebody buy something?" or if they don't have an account, can they sign up and buy? those were some of the use cases that we had because it turns out, those are actually pretty important to the business. That all have been said, you also can test your components for accessibility individually because even at a smaller level, some of your components might have to manage focus. The best one I can think of is models. When you open a model, you should trap the focus inside of that model and you should be able to hit escape to close the model and when you close the model, it should go back to what triggered the model to be opened. These are all things that are individual that can be tested also. But just because your components are individually accessible, it does not mean your application as a whole was accessible either. I don't want to paint the picture like, you don't have to care about your components accessibility individually because you do. It really does help but I think a lot of people miss the whole of the application, rather than individual with components. CHARLES: Right. The components are the individual stitches but you have to follow the thread throughout the entire garment. ROBERT: Right, exactly. CHARLES: Man, what I really want to do is I want to find out how your mom visualizes website navigation and use that as a visualization technique. ROBERT: That might be a fun webcast or something. CHARLES: Yeah and then see like could we actually use it as a tool because I have to imagine, it's probably pretty simple. It's much simpler than what we think of when we think of a website because it has to be really condensed down to its essence. ROBERT: Right. Yeah and [inaudible] users, they're not dumb. They have different ways. They know the gotchas. They know things that happen. They know there are different ways of getting trolled like my mom knows about focus jumping and she gets irritated what happens but she knows generally of what to do and it depends on her patience level. Like if it's focused jumping, she's like, "I don't need to use this thing. See you," and it's just not worth the frustration but there's different ways to navigate around like if you're a real power user, you might be able to recognize like the routing is inaccessible and you can navigate by headings or by regions or different landmarks. There's many different ways for users to navigate but to use those different navigation methods, you need to have a real strong coherent document structure. Your H1s have to actually be H1s and you have to have some things that they can navigate around to kind of work around those things. It's interesting and basically, do everything you can to help those situations. If you can provide semantic markup and give it a proper structure. If you can do the focus management, it's going to help everybody. It really will. I did see when we went through the use cases and defining those things, we actually learned a little bit more about our product because we had to put ourselves in a different seat and think about it because you get real close to it. You go through that same flow nine million times and you pick up real power user things that you can do like, "I'll work on that. I got this. I'll click this button. All right, we're good." Somebody that's going through it for the first time and you put yourself in that seat, it kind of opens your eyes a little bit and makes the experience better for everyone. I think that's kind of the underlying tone there. That's the message. WIL: Yeah, accessibility makes things better for everybody. ROBERT: That was a lot of content thrown at you there. We covered what is accessibility and why you'd want to do that and then kind of like more the basics things and how automated tools are really helpful and they can help you pick out things like using improper roles and nested things but they're not going to be able to tell you if your application is truly accessible or not and never will, unless we get something like a headless screen reader, where you can write automated tests for in that fashion but you're not to get something that will just run over you app and go, "Yup, you're good." CHARLES: Even so, it's a matter of user experience and that's not something you can get a thumbs up or a thumbs down to. When you as a user, it comes with application, you know when you see it but I would say until we have androids that accurately simulate human beings, I don't think we're going to actually have automated testing. ROBERT: Yeah. There's a joke for accessibility consultants. It's like if you put four accessibility consultants in a room and tell them to give you an alt attribute for an image, you'll get four different alts. We talked about the automated checkers, right? They'll not going to get everything for you and we talked about single page apps specifically in the routing and how we handled the routing in a React app and then how you can probably do it in an Angular app and how you can do it in an Ember app and Vue and different methods of how you can kind of attack that. We're definitely giving the link to the document that I wrote and the PR so you kind of see the real 'how' of how we did it because that would probably be helpful. I think there was a lot of good information there, so I would like to thank both Charles and Wil for being awesome co-host on this and -- WIL: Thanks for being an awesome host. CHARLES: Yeah, thanks for being an awesome host. I should say, you're welcome. ROBERT: I tried, I tried. We are The Frontside. We do accessibility consulting and training. If that's anything that your team needs help with, we're more than happy to jump on a call with you to kind of figure out what your needs are and what you need to do. If you need a WCAG support statement, if you need to audit, if you just need to figure out what's the next steps for you to even do, we're more than happy to help you sort through that. To reach out for that, you can contact us at Contact@Frontside.io or Sales@Frontside.io or Info@Frontside.io. You can contact us at any different ways and we'll be more than happy to help you. As always, thank you Mandy for producing the podcast. You're awesome and the next episode, what we're going to have is also still accessibility related and I'm really excited about this. It's about writing a proposal to the Unicode Committee and getting it accepted so basically, writing a proposal to get an emoji added and that's with Amberley and Amanda. They wrote three or four Unicode specs and actually got them accepted for, I believe it was sign language and deafness. That's really cool and I'm super excited for that because they'd be the first people that I've ever talked to that have actually created an emoji and gotten it accepted. WIL: Yeah, it's pretty cool. CHARLES: Yeah, that must feel great. ROBERT: Yeah, it's going to be awesome. That's our next episode. If you have any ideas or comments or anything, you can tweet us at @TheFrontside on Twitter or you can contact us through any of the emails that I talked about earlier. We're always open to hearing feedback. Thanks for listening. Take it easy, everyone.

Guest: Saron Yitbarek: @saronyitbarek | bloggytoons | CodeNewbie | @CodeNewbies In this episode, Charles and Sam talk to Saron Yitbarek about her idea of mentorship, ideas for distributed learning for businesses to promote individual and company growth, and why it's important to take "digital sabbaths" on the regular. This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. Transcript: SAM: Hey, everyone. Welcome to Episode 110 of The Frontside Podcast. My name is Sam Keathley. I'm a developer here at the Frontside and I will be your episode host. Today, we're here with Saron Yitbarek, discussing mentoring. She is the founder of CodeNewbie and the host of the CodeNewbie Podcast. Also with me as a co-host is Charles Lowell, who is also a developer at Frontside. Welcome Saron and welcome Charles. How are you guys doing? SARON: Thanks for having me. I'm doing pretty well. CHARLES: Hello. SAM: Today is going to be an interesting take on the mentoring talk. I mostly want to know first off, Saron, how do you feel about mentoring? What are your opinions on the mentor-mentee relationship or the value there? SARON: Yeah. I have lots of opinions on this topic. I think that the traditional structure of mentorship was usually looks like someone with less experience going to someone who has a lot more experience and say, "Will you be my mentor?" kind of like the children's book, 'Are You My Mother?' like 'Are you my mentor?' and then that person, that mother figure, that mentor looks after them and checks in on them and they have regular coffees and lunches and kind of steers them in the right, usually career-related direction. I don't think that's very realistic, to be honest. I think about why that might be. There's many different reasons. I think the fact that we're so, so, so networked and there's just so many different ways to get in contact with people and build relationships is a big reason but I think that that traditional mentorship model, that kind of one directional way of doing things is just not really needed and kind of overrated. I think that mentorship nowadays looks more like a mutually beneficial relationship, where I might reach out to someone who has more experience than me, for example in drawing, in art, not something I want to get to do but I know a crap-ton about podcasting, so I help you, you're my mentor in this specific area, in this specific topic but then, I get to be a mentor in this other thing that I'm really good at. I think it's those types of very focused topic-oriented, ideally two-way relationships that are more accurate and frankly, a more effective way of doing mentorship. SAM: Yeah, I actually agree with that. I am pretty new myself to development, only really been in this career for about a year now and I always kind of consider that mentoring relationship as a regression back to school days, where you have these -- SARON: Yeah, yeah. SAM: -- considered superior over you in some way, well, that's really not true in this community of developers and no matter where you are in development, it's all about working together and pairing. Working here has really showed me that value in paring, rather than like I have to look up to someone and regress back to feeling like a teenager in high school, like this person so good at this thing and they're the only one who can teach me. I definitely share that view of mentoring. I went to a boot camp one day when they were telling you like, "Oh, you got to find a mentor. You got to find a mentor," and I was like, "Well, but why? Why do [inaudible] together?" SARON: It's also a huge responsibility, a huge burden on the mentor too. Having to be, in a lot of ways, responsible for someone's career and trajectory and direction, that's a big responsibility. We don't have time for that, you know? On both ends, whether it's feeling like you're back in school or feeling like you have this huge responsibility, I think the traditional model isn't really the best model for either party. I think this idea of let's all learn together, let's be really focused on topics and problems that are very particular to what I'm doing, what I'm learning, what I'm trying to do. I think that works out. It feels healthier for both people. SAM: Absolutely. CHARLES: I wonder also too, it seems like it might be a little bit of a throwback to the days when people would spend 30 years in a single company or 30, 35 years in a single career where you have these people who are really these reservoirs of this intense tribal knowledge. It seems like people move around a lot more in their careers, not only in the company that they work for but also in the things that they're literally doing. I might be podcasting one day and producing a bunch of content and then, I might move into music or writing or other things like that. The careers seem to be broken apart a little bit more is one of the reasons why older models of advancement in those careers might not be as good a fit as they once were. SARON: Yes, absolutely. I'm obviously very biased because I'm in tech but when you think about the different roles that people have in tech, I feel like we're always wearing so many different hats. We have to, obviously code and be technical in that sense but we have to be really good communicators, a lot of us are speakers, we host podcasts, we organize conferences, we go to meetups, we're bloggers, we do so many other things, that this idea of, "I'm going to have a mentor who can help me in my role of being a developer," is just too big. You kind of meet people who are good at those individual pieces and those individual skills to get me to where I want to go. SAM: The whole idea of mentoring to me always reminded me of that whole Mr Miyagi relationship where you have this master of something and you're trying to learn from them. But in software development, I've learned that really no one is a master of anything because it's just changing so much and everything is so different. SARON: Yup, absolutely. CHARLES: The question is obviously, it seems like the idea that you're going to find one person who's going to represent the ideal confluence of every single skill set that you could hope to want, to be at some point in your career. That's looking more and more ludicrous. Is there a model where you can try and distribute those things, where you single out a large range of individuals? I guess the kind of what you were hinting at the beginning is that it is a lot more broken up, a lot more distributed but how do you pursue that, even if it is in a distributed manner? SARON: Yeah, that's a great question. One of the moments where I realize that the distributed model is really the only way that makes sense is, I think it was three years ago maybe. We thought about doing some kind of mentorship program in CodeNewbies, some type of way for people to link up and find people who can be there and guide in a way and we had people fill out this survey that basically said what do you want out of a mentor, what do you look for, what do you hope for to achieve but also asked how do you see yourself. Do you see yourself as a mentor, a mentee or both? It was really surprising that most people checked off both, which I thought was so interesting. It was so interesting to me that the same people who said, "I need help," with the same people who also said, "I also have help to give," and to me, that was such an amazing moment because I said, "Wow, it really isn't about this idea of I am the guide and I'm going to guide you." It's really about, "I have information expertise in one area and not in others." What I realize is there really isn't a mentor model. I think it's more of a culture of being helpful, which probably sounds really cheesy but it's true. I really think it's about saying, "I know how to do a thing. I'm going to go on Stack Overflow and answer questions. I'm going to go on Twitter and answer questions. I'm going to write blog post and share with the world." I think the real model, the real distributed mentoring model is us as individuals saying, "I just learned how to do a thing," or, "I just figured out how to do a thing well. Let me capture that. Let me capture it in a response, in an answer, in a forum, in a post and let me share that," and the more we do that as individuals, the more we have this huge amazing aggregate of knowledge that can serve as mentorship for all of us. SAM: I like the touch on that. That's kind of the idea behind Codeland Conference, where everybody thought they might be new to development and everyone is just sharing their knowledge. You might feel you really knew about it but you know a lot more than you think you do and you could still help. SARON: Yup, exactly and that's the whole idea, even the Twitter chats. When we first started, I don't have all the answers, I don't claim to, I don't really want that responsibility but I know there are a lot of people who do and I know a lot of people who have resources and opinions and who can help out. One thing that people do is they'll DM us and say, "I'm having a hard time with this." Sometimes, it's a very technical problems. Sometimes, it's a general 'I'm having a hard time getting a job,' which I do like high level question and I never answer them. I always say, "Tweet us and we'll retweet it and we'll get the whole community involved and we can have a rich conversation around it," and that's really been our motto, our philosophy. I see what they do with mentorship. If you have a mentor, you're not obligated, I guess to listen to them but the idea is kind of that you should. There is one person and you should listen to them because they know more than you. I think that's just not really fair and instead, I like to think that there are lots of different ways to do things and it's up to you to decide what's best for you and in order for you to decide, you have to have a lot of options. With Codeland, with the Twitter chats, no matter what skill level you're at, no matter how confident you feel in your coding abilities, you do have something to offer. Let's pull that out. Let's put it on the table and let's see who you can help today. SAM: That's a perfect way of introducing someone to the idea of helping or getting help from a community, rather than an individual because you never want to take one person's word as gospel on something you just know nothing about. SARON: Yeah, exactly. SAM: You can sound confident talking about something and it can be the total wrong answer but if you're seen as someone superior or someone who knows what they're doing -- SARON: And you can sell it. SAM: Yeah. You can be the best snake oil salesman in the world. SARON: Absolutely. For the CodeNewbie podcast, we do short questions at the end of each episode and one of the questions -- my favorite question -- is what's the worst advice you've ever received. I love that question because I started by saying, I think that people love giving advice. I think people love asking for advice and the assumption again is if I give you advice, then I probably know more and you should probably listen to me but that's not always true. I want people to be comfortable making their own decisions and deciding for themselves. "This may sound like it could work and this may sound like a good idea, generally speaking, but for me, it's probably not a good fit. It's probably not what's best for me," and to be comfortable rejecting advice and that was kind of the reason why I started that question, it's my favorite question and it's so interesting how much advice is good. It's kind of generic but it's a good advice. It's an advice like, "Don't quit. Keep going," which maybe a good idea and maybe you do need to quit, so being open to rejecting advice, I think is really important and that one of my favorite questions. SAM: In your experience, what is the best way to reject advice? Because when you're a new developer, when you're new at anything and you're seeking advice from a community or from a person, you don't want to come off as rude or maybe you're feeling... I don't know, not very confident but you think the advice that you were given is just bad advice. What would your advice be? I guess, what would your advice be on this advice? For your perspective, what would be a good advice to reject advice that you think is just wrong? SARON: Number one is when I ask people for advice, I try to think about and try to know upfront, do they have the same values that I do? Do they have the same worldview? Do they have the same goals? Because that's the thing too. Oh, my God, I got so much unsolicited advice. It's amazing. I've actually stopped just saying to people my ideas or what's on my mind because I know as soon as I say, "I'm thinking about this," I'll get a whole slew of unsolicited advice and I'm like, "I didn't ask for that." What I've learned is first to kind of figure out are we even on the same page because if my goal is to be a developer, if your goal is to be -- the only thing I could think of is a juggler, I don't know why -- a juggler and I ask you for a career advice, I'm going to go ahead and safely assume that what you have to say is probably not as applicable to me and so, number one is kind of identifying that. But number two is if they say something that just I know is not going to work or I've already tried, I'll just nod and say, "Thank you very much," and kind of go about my day. I don't think you have to declare whether or not you going to take it. I think just acknowledging and I think the people that give advice, I think they're trying to be helpful, they have good intentions, usually. Usually it's, "I'm trying to save you from making mistakes that I made and I'm trying to help you get to learn a little faster." I always appreciate it but knowing that I can say, "Thank you. Thank you for your time. Thank you for your perspective," but know that I don't have to go off [inaudible]. SAM: That's actually very similar to my tactic. Just not like, "Thank you. Thank you so much. Don't talk to me ever again." No. SARON: And disappear, yeah. SAM: I'm like the wind. With this pressure that either new developers or seasoned veterans are feeling about the mentor relationship, because I feel like a lot of senior developers that I've spoken with or people who've been in the business for a long time, feel like they should be mentoring or they need to take on that responsibility but there's always that hint of dread in their voice when they say about like, "Oh, I should be doing this." I always feel like it's okay to not do that. I never really understood why it was so high value to have this one-on-one relationship with an individual when you're not in school because it just feels so like... Not childish but childish. SARON: Yeah, that's one of things, frankly that I love about the tech community and also do not understand about the tech community. It's such a giving knowledge sharing community, whether you do it in a tactful way or not in a tactful way but the idea of giving back and paying it forward is just so deep. It's so, so deep that even when I've been coding for only a couple of months, I still felt this, I don't want to call pressure because pressure kind of sounds a little negative but I definitely felt this expectation that I was supposed to be blogging, supposed to be sharing and shouting and helping and doing these pay it forward type things. I don't really know where that comes from. Maybe that comes from the culture of open source, maybe that's where it kind of penetrate. I'm not sure but there's this huge need, desire, idea that we're supposed to be giving back and for that, I am very, very, very grateful. But I think that acknowledging that if you're someone who wants to be a mentor that you can do it simply by being available, literally being available, the going to be hashtag is super, super active even when we're not doing our Twitter chats and people use it to ask for help, they used it to ask questions. If you're feeling particularly giving or extra helpful that day, go on Twitter and check the hashtag and see what questions people are asking. Things like that or just super helpful and don't require a huge amount of time and effort. There's a lot of small ways to help out. That may not seem like a big deal to you but for the person who's asking for help, who has been banging their head against the wall, it's hugely valuable. SAM: Yeah, absolutely. Up until I went to my first conference this year, I didn't realize how supportive and important Twitter is. You know I always kind of considered it to be like another social media platform that I don't understand because I'm 84 years old and I just don't get it. It's been so uniquely helpful and in ways like Stack Overflow or even issues in GitHub, it just aren't. You can get so many more perspectives, so many different perspectives from people. SARON: Yeah, absolutely. Twitter has been amazing exactly for that. It's just an efficient way to crowdsource opinions and crowdsource perspectives and when you get your question answered and someone else answers it, it doesn't only benefit you the way it would if you email someone but it benefits anyone else who comes across that page, so yeah, it's hugely valuable. CHARLES: I remember the first moment I had kind of like that, a light bulb went off in my head where I was working on some really weird project that was using some strange wiki for its content storage and I was getting frustrated and I just tweeted about it and then the CTO of that company just immediately answered my question and I was like, "What?" You know, it was years ago and definitely, I was like, sound of explosion, that is where my mind exploded. I was not seeking help. I was just literally being kind of a jerk and venting frustration and lo and behold, the answer for my problem descended from the Twitter clouds. It was incredible. SARON: The Twitter clouds are the best clouds, usually CHARLES: Twitter is very... What's the word? It's very split down in the middle. SARON: Noodie? Yeah, there you go. SAM: There's this live feedback, so there's no buffer of emotion there, you know? SARON: Yeah. SAM: It's like, "Oh, this thing that you said, it made me mad. I'm going to tell you about it right now." Charles, I know that you had mentioned before that you think mentoring would be a good idea for Frontside and then, after all this discussion, have your views stayed the same? What are you feeling about that? CHARLES: As kind of the person who's like the grizzled veteran in the software world, it's definitely something that I've kind of whipped myself over the back. It's like feeling like it's something that we should do but I think it comes from the idea that people come here and we want to make sure that they're getting access to the learning that they need and the ways, in which they can level themselves up that they need, that's the kind of the prime motivator there. I always perceived mentorship as some vehicle through which to achieve that. It's something that I've heard. Obviously, we don't have a mentorship program at our company. It's something I felt that we should always be investigating. I've always felt maybe a little bit bad that we didn't have it but it's also something that I really struggled with in my career because I can't really say that I've ever had a mentor, so I don't really know what that relationship would look like but I do have a lot of people that I learned a lot of critical things from. I can look at it as kind of these seminal moments in my career, where like light bulbs went off and a lot of the time, they're associated with an individual and that individual and the thing that they taught me or multiple things that they taught me, are still with me. I have those experiences, which have been phenomenal and critical to my development as a software developer, so I guess it's just part and parcel of that impulse that you're describing to pay it forward, to realize that when you walked into the building, the lights were on and the walls were standing and the air was at a comfortable climate temperature. As you live there, you realize that there are people involved in actually, doing that maintenance and providing the building for you and the space for you to become aware of your world and then, when new people walk into the building, you want to provide them the same experience that you had. I guess that's my take on. That's the kind of thing that I would want to provide, so the question is like what does mentoring or mentoring 3.0, as we're maybe talking about in this conversation, how does that fit into that? How would you implement something like this, some sort of distributed learning in a company? I don't know. Maybe, it's not worthwhile. Maybe it is. SAM: I think it focuses more on that pair programming because when you're thinking back and you have all these people, like you have names of people that taught you something, it's multiple names. It's not just this one guy taught me all of these things. Actually, within a company, that mentoring just comes from pairing with your coworkers, seeing what different hats everybody wears and then, trying them on every now and again but not necessarily taking that individual's word as gospel, you know? CHARLES: Right and hopefully, that's not something that we've advocated for. Maybe, it's how do you introduce structure around that, to make sure that the proper ferment is happening, so that you have novel pairings and make sure the ideas that are flowing are flowing around the entire company and not just through certain set channels. SARON: Yeah. The other part of that is if you create structure around what it looks like to share outside. You know, pair programming is interesting because it's kind of one-on-one and it's hopefully, I have something important or bright to say in our pairing session. Maybe, I don't. Maybe I'm having a dull day. Maybe, I'm having a bad day but this really give you the same opportunity to take a moment and say, "What do I know? What do I want to share? What do I want to put together?" It does really give you a chance to prepare and gather yourself. It's kind of in the moment. I think having another opportunity where you can gather yourself is important and so, that might look like brown bag lunches, where everyone takes a turn and has to do a little lightning talk. That's usually the opportunity to say, "I have five minutes. I'm going to share something." Everyone has a turn, which means that the company's literally saying everyone has something to say. It's only five minutes, only a few minutes, so hopefully it won't be too terrifying if you're not big on public speaking and it's your co-worker so hopefully, it won't be terrifying because it's people you know and not total strangers. You know, a format like that, where there's structure but the company is saying, "We're going to give you time to think about what you're good at and what you know and to share that is good." I think another way to do that is by setting time aside for blogging. If your company can say, "Thirty minutes out of the week, we're going to take some time to write down five things I learned, to write down one cool thing I learned, post it publicly, post it internally about this expectation that everyone should be writing, everyone should be sharing, which also says everyone has something to share." I think those are two ideas and two ways that we can create a culture of sharing and a culture of distributed mentorship, where everyone has an opportunity to find the thing that they're excited about and specific ways to share it. SAM: Yeah, that's excellent. CHARLES: I hope you're grinning as much as I am, Sam because at least in the first half, you basically described our Lunch and Learn process. It's a little bit more than five minutes but I think another thing that clicked for me there is making sure that having a variation of expectations of quality or something like that because I feel like where we do really well is in the Lunch and Learn thing. We have a process very similar to what you describe but where we're not so good is maybe with blogging. I wonder if part of that is we just hold ourselves to such a high standard of what it is. The idea that we could throw together a blog post in 30 minutes, I love that idea but it means you really have to be willing to just say, "You know what? We're going to get it out there and we're going to make it bite-sized and the expectation is that we're not going to be writing some gigantic essay that's going to shake the industry to its core every Friday at three o'clock." There are things and if we can, I shouldn't say a reduction in quality but maybe a reduction in scope so that you can say like, "We're going to carve out 30 minutes or an hour and we're going to pick a topic that scope-appropriately for that." SARON: Absolutely and I think that knowing that you only have 30 minutes or maybe it's an hour -- an hour is probably a little bit more realistic -- but knowing that you only have an hour also forces scope. You can't write a book about JavaScript in an hour. You just can't, so the fact that by the end of that hour, you have to have something to turn in is a great way to force people to focus and do something really small and really bite-sized. The other thing is I don't think that after the hour, you need to publish it publicly. It could be, you need to turn it into someone else, to edit and look over for you and give you feedback. It's not quite ready yet but it's a solid first draft and the ideas that after that first person edits it, then it's on its way to being published. I think there's different ways to manage a scope without also making this scary thing where I have to say something for the Earth to shake. There's different things that we can do around that. CHARLES: I'm wondering what other forms of sharing that we can fit into our workday. I guess, the other baseline is just making sure that you're always... What is it? ABT -- always be tweeting. It's so easy to be write-only, not even engaging conversations but just throwing ideas out into the void. SAM: I think holding a discussion with your coworkers like if you're in an environment where you can work face to face or are constantly online, if you're more of a remote worker, I think just having a conversation with anybody, rather than just putting your idea out there, putting it out there with someone who can actually provide real time feedback in a more friendly way, then I think some people on Twitter could be. Because they're your coworkers and they're not going to call you rude names, you know? CHARLES: Right and you're also going to know, hopefully the whole trust and intent and 'are we on the same page?' that question is answered even before the text is written. SAM: Right. SARON: Yeah, absolutely and with those conversations, this actually mean [inaudible] that we do. We do like a show and tell every week and the idea is that we're both always learning random things, usually related things but sometimes, totally random but still very interesting and it will take 30 minutes to just share what we've learned. Sometimes, they'll also turn it into a blog post. Sometimes, it's just a knowledge sharing opportunity. I'm not really sure but it's a good window of opportunity to say, "We are learning and we're sharing and we have something of value to bring." The great thing about something like a show and tell is that it doesn't necessarily have to come from my brain. It doesn't have to be like, "I have a great idea that I'm going to share." It could be, "I read about this cool idea." It can be, "I heard about this cool thing that we can try and we can apply," but I still kind of credit, you know? Like I get credit for being the value bringer but the burden isn't on me every week to come up with the idea. That's a nice balance, to kind of create space to share and to promote this knowledge share and if it comes from you, it's great but if not, you're still helping other people. CHARLES: I have a question that may or may not be related in here. This has just kind of occurred to me because this is definitely something that I experienced, where I get into a mode where I become overwhelmed by the ideas that people are sharing and so, what's the balance? Because usually, we're out there searching for ideas and we're searching for novel things so that we can include them in our work, in the things that we want to do and accomplish, whether be that in tech or elsewhere. What's the balance of being heads down and being like, "You know what? I'm going to be closed to new ideas right now." Because they can be distracting, right? The [inaudible], that's actually a phenomenon and so, how do you protect yourself from sharing? What's the balance? SARON: My solution was to move to San Diego. That was my solution to that. I actually moved from New Jersey and I worked in New York City... How long has it been? Was it only been a year? Oh, my goodness, and now, I'm in San Diego and it was so interesting because that ended up being a really nice side effect. I didn't move specifically for that reason but when you are commuting in New York City every single day, there's so much going on all the time. There's just so many ideas and events and meet ups and companies and people. It's just so much. I think that it was a great place to be in my early 20s when I really just wanted to soak up everyone else's ideas and I didn't really have opinions of my own at that point and it's a great way to just kind of absorb and be this awesome sponge in the big city but after a while, I kind of realized, maybe I have my own ideas and my own thoughts, so moving to San Diego, which is a much, much, much quieter place, has been a really great way to reflect and sit with my own thoughts and feelings and opinions and just kind of focus on that. I understand that everyone can move to San Diego, although I highly recommend it but I think in that way of carving out like... What do they call it? Do they call it a tech sabbath? A digital sabbath? Am I saying that right? CHARLES: Yeah. Maybe. It sounds about right. SARON: Yeah. I came across that term recently but this idea of one day out of the week, I'm not going to be on the interweb. I'll just not going to do it. Maybe, even the whole weekend, oh, my goodness and saying, "I'm just going to stay away from things. I'm just going to create a little space for me to think and reflect." I think when I don't have the whole day and what I need is just like a moment, I find that writing things down is a great way -- a really, really good way of doing it -- even if I'm taking notes or from doing a strategy session or if I'm trying to make a decision. Usually, I'll start typing and what I've started to do recently is to say, "I'm not going to type. I'm going to plot in a notebook and I'm just going to write things down," and because writing is slower than typing, it forces you to just think. It forces you to be alone with your thoughts, for better or worse and it forces you to really just to think about what you're doing or what you're saying and reflect. It's a really, really great meditative exercise that I found. You know, finding little ways to build in an escape from the noise, ideally on a regular basis, I think is a really healthy thing to do. SAM: Yeah, I definitely agree with that. My normal method of sort of closing everybody off is I just sketch out ideas because I'm an artist. I come from that sort of perspective as I can make a drawing out of feelings more so than I can write them out. If I'm feeling really overwhelmed or if I'm trying to make sense of some information that people have given me, I just sketch it out and I think that's something that doesn't really get talked about a lot in the whole community because sometimes it's always about eat, breathe, live, code, you know? We have different skills. You have other things that you're good at that's not just code. When I was at React Rally, a lot of people were in the music and then finding a way to separate yourself from the entirety of it is definitely one of the better ways to make sense of your situation. SARON: Yup, absolutely. CHARLES: Yeah. Sleep? It's a great way to take a break. SARON: Sleep is so good. SAM: On my top three things: sleep. CHARLES: Yeah. SARON: Yeah. As a kid, I never ever thought I would look forward to my bedtime ever. You know, it's my favorite part of the day. CHARLES: Yeah but like sleep, writing, drawing, I always forget. Like when I'm caught up in a problem, I always forget that engaging in some other activity -- I like to walk in the place next to my house. I like to play my ukulele. Engaging in those activities is almost on the critical path to solving the problem and I always forget it. I think we always forget it but sometimes, you can be so frustrated and you can just shut it off, be completely alone and there is some magical process that's probably going on in your brain. I don't know exactly what it is but you come back and the answer is just sitting there, waiting practically on your desk. SARON: Absolutely. One thing, we recently moved a couple of weeks ago to a new place and it's a little bit bigger and so, I had the opportunity to unpack boxes and buy furniture that we didn't really need before or just trying to make it more of a home. I've been using it as a really great opportunity to be productive and to feel productive but not be in front of a screen. Sometimes, I even like save tasks for myself like, "I'm going to wait till the evening to put together this shelf because I'm going to need a moment to just be away from my computer." I have a plan around it so it ended up being such that, I think about every day, there's one little home activity thing that I can do, whether it's cleaning or cooking or assembly or movie or something with the home, that allows me to, because I'm not really a hobby person. I don't really do hobbies because it just feels, like no judgment on people who do. I know people get really into their hobbies but it just feels like, "Why?" You know, like, "Why?" like, "For what?" But when I put a shelf together, it's like, "I'm going to use this for books." You know what I mean? Like you're very purposeful, so home activities has been my way of carving out that space away from the screen but also feeling like I'm doing something productive, I'm getting things done. SAM: Something that I would recommend to anybody who doesn't really have that, I'm going to step away from the computer and do this thing. Something that's kind of helping me was the Pomodoro Technique, which if you're not familiar with that, it's basically a time management thing where you set your timer for work like for 25 minutes, 30 minutes or whatever and then, when the timer goes off, you take an allotted break for five minutes, 10 minutes, just so you're not doing the thing that you've been doing for the last 30 minutes. SARON: Yes, absolutely. That's a great one. SAM: My advice is to implement that if you don't have a thing, that you use for your cleaner, I guess. SARON: Oh, you want to hear some really terrible but effective advice? SAM: Yes. SARON: And this is what I found out very accidentally is if you have a really, really crappy office chair that hurts -- CHARLES: Oh, man. I'm sitting in one right now. It's literally a rocking chair from the 1800s. SARON: That sounds awesome. CHARLES: Yeah, it does sounds awesome. SARON: But if you have a crappy office chair, that can be a really great way to get away from your screen because after about an hour, your thighs will hurt and your back will hurt and you will be forced to get up and walk around. It's so funny because I have a pretty crappy office chair and my back has been hurting for so long and I just thought, "That's life." That's like my [inaudible] for myself. I'm like, "That's just how life is, my backs hurt," and then it got to a point where I was like, "I need to go and talk to someone," and as soon after that, I spoke to a conference and I was basically up and down away from any type of chair for like four or five days straight and magically, my back pain went away and I was, "Oh, my God. It's that freaking chair," and ever since I realized it, I started noticing it. I would sit down and I would go, "I'm nearing the one- Oh, there are my thighs. There they go. They're in pain." There's another great cheap life hack: get a crappy chair, after about an hour, everything will hurt. You will be forced to go do some jumping jacks for about five, 10 minutes and there's your additional sabbath. There you go. SAM: In Charles' case, probably a haunted office chair. Yeah, that's an excellent advice. I never really noticed that. Now, I'm going to. CHARLES: You know what funny is, I actually didn't even notice it but I do. I never used to get up and go on walks before or spend as much of the day standing as I do and I'm actually completely and totally oblivious to it but I think, I realize like I can attribute a lot of that to the terribly, uncomfortable chair, in which I sit on every day. SARON: I have this awesome habit of sitting cross-legged, which apparently is a very, very bad idea, my physical therapist told me, so I make it worse for myself. If you don't feel like you have enough screen time, just sit cross-legged and that will accelerate the pain process. SAM: As you said that, I am currently sitting cross-legged in my desk chair. SARON: Yeah! SAM: That's just how I sit in chairs. My legs are too short to reach the floor. SARON: Me too. I just find it so comfortable. I don't know and I'm so happy to hear that you also do this because I'm like, "Am I just weird?" because I love sitting cross-legged. It's so comfortable. It makes me really happy. SAM: No, I verified 100%, that's how I sit in all chairs. SARON: Yeah, the same. CHARLES: I can do one leg. SARON: Only one? Man, you need to work on that. SAM: -- [inaudible]. That's going to be the end of our podcast. Again, thank you Saron for coming on and having this awesome conversation about mentoring. Definitely check out the website CodeNewbie.org. We are the Frontside. We build software that you can stick a future on. Hit us up if you're looking for help building your next big thing and also, hit us up on Twitter. Hit Charles, myself, Frontside and Saron on Twitter. SARON: Thank you so much for having me. This is awesome. This is fun. SAM: Also, I want to give another thanks to Mandy, our producer for producing this lovely episode and all of our episodes. If you have any questions for us, hit us up on Twitter. Any ideas for future topics, any thoughts you have on this great conversation, let us know and we'll talk to you all later.

Guests: Brandon Hays: @tehviking | Blog Chris Freeman: @15lettermax In this episode, former Fronsiders, Brandon Hays and Chris Freeman join Charles and Taras to talk about the difference between a framework and a library, whether or not React + Redux a framework in itself, red flags to signal that you're actually building a framework, attributes of a good framework, how can you tell if you created a bad framework, and how you can make a bad framework better. Resources: Test Sizes by Simon Stewart This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. ** Transcript:** CHARLES: Hello everybody and welcome to The Frontside Podcast. My name is Charles. I'm a developer here at Frontside and today, we're going to be talking about the things that go into making a JavaScript framework. Because, hey, there's not enough of those in the world today, so we're going to talk about that and with me is Taras. TARAS: Hello, hello. CHARLES: And we've got two very special guests, who have a lot of experience with this topic. Mr Chris Freeman and Brandon Hays. Hey, guys. CHRIS: Hi, there. BRANDON: Hi, there. We're talking about the poofberry framework, right? CHARLES: What's a poofberry? BRANDON: There's a tweet that's going around right now that one of them says, "I don't know what I should be doing," and the next person says, "Oh, just use poofberry." What is that? It's like fluffnuts but the [inaudible]. Hey, dot, dot, dot. Then, it integrates with log bungler. CHARLES: There's a reason that I'm dying laughing. BRANDON: It's so true. CHARLES: It's so true, laugh, cry, laugh, cry. Let's start with kind of a very basic assessment here. Because there's a lot of different things that you can use to compose the applications that you build but for some reason, some of these things are grouped and considered as libraries and some of them are considered frameworks. I don't know that the boundary is very clear like I'll know it when I see it type thing. Maybe, we can start with what is the difference between a framework and a library? CHRIS: I have some thoughts of these. I feel like this is one of those questions that could easily just turn into an infinite bike-shed but I remember reading something a while ago that stuck with me for a long time. I'm pretty sure it's related to Java but that makes sense because if anyone is going to talking about frameworks, it's Java developers. But it was saying that the difference between a library and a framework is inversion of control and the idea is a thing that's a library is a thing where you are in control. You bring the library code into your code and it's up to you what you do with it. In a framework, the framework code calls you as I think what it said. It's like, you call the library code, the framework code calls you and -- CHARLES: In Soviet framework. CHRIS: Yeah, exactly. A framework says, "Here are a bunch of open spaces for you to put your code in and I will take care of the rest," versus a library is just like... I don't know, "Here's some things that you can use. It's up to you. What do you want to do with them?" CHARLES: Right, so in kind of like [inaudible], that would be basically, a framework would be the thing that's got the main method. I think the same thing in JavaScript and when you call it, does it actually implement the main method. In JavaScript, it'll probably like in node. Under that definition, it would be like, "Are you the main scripts when you invoke node? Do you control the main script?" If you were doing your own command line parsing, for example, you're looking at the process.rb and pulling off the command lines and doing all the things but even if you're using something like Yargs or option parser in Ruby, that's more of like a framework. I guess Yargs is a library because you're still implementing the script. You're instantiating the Yargs thing. TARAS: React calls render to figure out what to convert to DOM. Does that make React the framework? CHARLES: I think React as a library. That's a good question. What's the equivalent of the main method on the web? CHRIS: I think there's a very clear distinction, especially if you look at React versus something like Ember and I'm sure Angular does this as well. In React, by default, to build a React thing, you're going to pull in React, you may write some components, you may import them elsewhere but the main method is that you have an index.html with some div in it and you are the one that has to call ReactDOM.render and you pass it like document.query selector or whatever and then, your top level component and that can be as simple as complicated as you like or you can have a webpack plugin do it or whatever else. But the onus is on you to actually take that React app and get it starting up on the page versus Ember, it's like, "There's an index.html. It's fully wired up." There is one point where you sit down and say, "Start my program here," like Ember abstracted all that away. To me, that's the main method for a frontend application. CHARLES: Right and if you actually look at something that Ember generates, then look at index.html, they generate a script tag for you that instantiate your application and mounts it on an element. If you want to change that element, that's actually a configuration option that you can change but it still a configuration option that's consumed by the framework. In that sense, there is that inversion of control. I see what you mean like in React, you're the one who flicks off the first domino, like who's the prime mover. Is it you or is it the framework that knocks over the first domino? BRANDON: I like Chris's explanation and I think it's elegant to say because I was thinking in terms of structure. If it imposes a structure on you but really, the structure is there, it's like one of those Ikea shelf systems for you to put stuff into. If you're trying to solve a problem, here's a shelving system for you to put stuff into, whereas a library is just the tool that you might get out to put something together. Something that's multi-purpose but doesn't impose any structure on you or a ton of structure on you. My question is what's the usefulness of distinguishing between the two? TARAS: I think what's interesting and I had experienced this in a last couple of projects is that people, especially React when they kind of assume, because a lot of people entering to React not understanding the context within which React emerged and so, they're getting into React assuming that it has everything you need to build application that you need to build. A lot of them haven't necessarily built a single page application from scratch before and so, the jump into building something with React and then, it takes about a year for them to realize the full scope of all of the features that their application actually has and then, they kind of take a retroactive look and look like, "Okay, what do I have now?" and what emerges is that they've actually over the last year, they may be creating a framework without realizing that this is actually happening. CHARLES: They've imposed the structure of saying, "Here's the shelving system. Books about geography go here. Books on English literature go there and so on and so forth." BRANDON: But when you rolled your own framework, that's not how it goes. It's like, "You have to launch this balloon into the stratosphere to put a book on the shelf from geology." Taras, to your point, it sounds like the importance is setting expectations properly for people, so that they know what they're in for because kind of calling back to Ryan Florence's post a few years ago, you can't not have a framework. At some point, you will have a framework in order to ship something. I would actually take it one step further. My friend, Kyle talks about this that library is the smallest unit that you're working within a framework but that still doesn't take your code to production and put it in a debugable state. You need a platform. It's arguable, if you're handling deployment tasks and debugging tasks and operating software in production, you now have a platform and it's fair to say that Rails crossed that threshold at one point. It's fair to say that Ember has probably crossed that threshold, if you combine Ember with CLI deploy and the CLI tooling and all of that stuff. This almost like acts as a platform if you're owning and maintaining the software in production. CHARLES: Now, can I play devil's advocate here and say, the platform, is that necessarily predicated on a framework? Is there a pyramid where it goes library, framework, platform and one is built on top of the other? Why couldn't I have a library? Because what I'm hearing is the scope of concerns is just rendering HTML based on a state is a very small chunk. The actual scope of things that you need to do to get that code in production and have it be reliable and do all of the features that you want to do is just massive but why is that predicated on a framework? For example, one thing you have is a bunch of libraries out there, like routing for managing the title tag, managing all these things that you have to do for managing deployment, for building your application, for compressing it. There's all these different libraries out there. What if there was one massive library that just picked a bunch of other libraries but I was still in control? TARAS: I've actually seen this happen in the last of the projects. When people jump into building, they will eventually realize that they're building a platform but what happens before that is that they take user's requirements and they break those up by sections and then they assign them to a bunch of development teams who go and actually start to build. On one platform, they end up building five or six or 10 of siloed, packaged applications that have, in some cases, have their own dependencies, they have similar architecture, might not have similar architecture. Each team kind of implements thing differently and there's an expectation that once you package these things as npm and then you install them into one package, to one application when you run build, it's just going to work together. That's where I think, with the framework, it does create a foundation for these verticals to be implemented using kind of common foundation. This is what a lot of times that as if you don't realize that what you're trying to set out to build, the way that the projects get managed quite often, especially for big applications, for big platforms is that, it creates this period of about two years, where there's a lot of confusion and there's a lot of duplication and then, you end up seeing code that it's hard to put in production. CHARLES: Yeah, I agree. I'm curious then, because we'd started out talking about library and framework and talked about it takes two years to recognize that you're building a framework or you're building not a framework but a platform. Brandon, you said something very interesting. Rails for example, crossed the threshold of being more than a framework and actually, being a platform. What are the concerns of a platform that are beyond a framework? We talked about and using the kind of loose definition of a framework as being something where the framework create spaces for your code, to run your code so you can just take little dollops of code and they have one concern but the framework manages the coordination of the concerns but what's the next level? BRANDON: For the purposes of this conversation, I may have muddied the waters a little bit because I think it's more interesting to talk about the transition and the level of which you've crossed the threshold from being a library or using libraries or collecting libraries, into maintaining a framework because it's where you're going to experience more pain more than likely, than to me, the idea from works on my machine, to deployed and supported across a lot of users, it sounds like it's more interesting but it's not where we experience most of our pain actually. From my experience, maintaining frontend single page applications most of the pain is actually getting the damn thing to work on your machine and getting the libraries to collectively work together and then, getting that to production, it kind of enters back into an area of more known unknowns. I think that's a surprisingly a more mature ecosystem, still getting from this thing works on my machine to getting it out the door. That wasn't true when Rails was invented and so, Rails had to invent a lot of its own ecosystem around this stuff. Like I said, I don't want to muddy the waters too much. I think to me, the interesting question is how do you know you've crossed this threshold? What pain points are exposed when you start crossing that threshold or when you're pushing the boundaries of that threshold? Because you should not be using a framework if you're using React to do a select dropdown. I think of it as, if you're using it the way to replace something you might do with a jQuery plugin five or 10 years ago, you're using React like it's a library. One of the questions that you brought up was is the combination of React and Redux is a framework and I would argue that it is but I kind of want to throw that out -- CHRIS: Oh, interesting. CHARLES: I would say, it's two libraries stuck together to make a bigger library. It's like a monolithic library. BRANDON: But by the time you're actually using that to do anything, maybe the third thing in there is like transitioning states when you transition routes. At what point is that threshold crossed? I didn't build most of the software that led me to some of the opinions that I have about this. This was actually Chris Freeman's, though. I may defer to you on this. CHRIS: I think React + Redux constitutes if you look at what it does. You have like this view layer and this state layer. There's a set of opinions on there that is useful and there is the foundation for doing quite a bit but in my experience, you've already kind of alluded to this a little bit. I don't think it's a framework because as soon as you start using those two things, suddenly the next thing you hit is, "Wait. How do I handle asynchronous things?" There's a lot of different options for that. "Oh, now, I need to do routing. How do I incorporate routing into my React app but also in a way, that is amenable to state transitions in Redux but also, that is aware of the async stuff that I'm doing, that is going to possibly be triggered by my routes and by my Redux actions or by some other side of things?" Suddenly, you are very quickly pulling out a bunch of other libraries but also, probably starting to build abstractions on top of them because you're already finding a lot of common patterns that you're repeating over and over as you incorporate more and more pieces of the stack and then, you're writing a lot of glue code. I think that's the point where suddenly, you look back and behind you is the footsteps of this framework that's been walking alongside you the whole time. BRANDON: That is where I carried you, then dropped you, then sort of drowned you. CHRIS: Yeah. CHARLES: And then, kicked your core. TARAS: I'd like to suggest a way to think about this. As you guys are talking about, it kind of occurred to me is that it seems to me that libraries concentrate on how and frameworks focus on the 'what.' CHRIS: Oh, I love that. TARAS: Because if you think about for example, React is how geostack efficiently update DOM, then Redux is how do you wire together state across multiple components that might be in different parts of state tree and if you look at, for example, a React router or a kind of a routing component is how do you choose which components you want to render when you navigates specific URL. Because those things by themselves are not a complete solution but when you combine them together, what you get is you have a way of saying, "When I navigate to specific URL, I'm going to load specific data, provide that data to components and then, I would have a way to navigate through a different URL when you click on a link." From that, I think what happens when you get to the framework level is you actually have a kind of a bigger umbrella and under that umbrella, you have ways to address problems that you did not have previously. I think that's what framework does it is over time, it's a way of addressing concerns that cannot be addressed with a solution. They have to address with a collection of solutions and then, they provide a specific solution. I don't know if that's -- CHARLES: That actually sparked off a train of thought in my mind that perhaps what you really want to do is say, "I'm going to go a little bit like Lisp on you all," in the sense of every code at some point is data, that maybe every library, at some point is a framework. It's just that you can look and say, "What is the scope of the 'what' that I'm tackling." For some point, you can say like React is a framework. It creates this space where I can put my JSx, AKA the render function and I'm basically inverting control and so, what it is, it is a framework for efficiently rendering HTML or efficiently mapping an object to a fragment of DOM and then the DOM that gets generated from your render function, patching that into the HTML. You don't have to worry about that. There's that inversion of control. It creates that space but that's the only space that it creates. From that perspective, React is a framework for generating HTML but that's all it is but it is a library for constructing applications. Does that make any sense? I think as you layer on concerns, your framework create spaces for you. You use your library code to put stuff in and so, in the same way, I think one of the key realizations, I'm going to call up like BigTest and I'm not going to take credit for this, which is actually a blog post that I read at Google. I can't remember what it is but we'll link to it in the show notes where he said, "There are no such thing as unit tests. There are no such thing as acceptance tests. There are just tests of varying scope." They're all acceptance tests. To use that one thing, they're all experiments. It's just what is the scope of the test that you're trying to accomplish and his argument was we want to make that scope as big as possible by default and then, where appropriate, you narrow down. Maybe, the framework library distinction is a little bit constructed, kind of a construction of our own minds and what really is there, there's just frameworks of varying scope. BRANDON: Agreeing on a shared scope is actually probably the most important part of this conversation. We're referring to building end-to-end an application from data access to rendering to testing -- CHARLES: To deployment to routing. BRANDON: Yeah. CHARLES: To one day accessibility. BRANDON: Yeah. Adding that into the discussion is like a baseline of what constitutes an application. It's the percentage of people that are able to actually use it, the people that are locked out from using it by ability. That's a very useful frame for the discussion. Let's agree on the scope of what an application is and then, coming back to what Taras was saying is basically, when you're talking about the 'how,' that's a decision point. You hear a lot of people talk about decision fatigue in JavaScript and it's almost a played out trope at this point but it hasn't gone away as a problem, so what frameworks are doing is they're making a series of decisions for you that allow you to basically connect the pieces from end to end. Basically, somebody threw a rope bridge across the canyon and it doesn't have to be the best solution to get end to end but we have to solve the problem end to end. If we agree on the places end to end and the problem is when you're building your own series of libraries, you're like, "I'm going to choose best in class of A, best in class of B, best in class of C," and that sounds really good but if you're trying to build a bridge across a canyon and you're building in 10 best of class sections, for the type of connection we're trying to make here in the middle, we're going to use the best in class here. The weak point is in the connections, so you had better be the world's foremost engineer if you're going to be the person connecting all these disparate pieces that are each best in class, in order to bridge this canyon. That's the thing that's interesting to me and it's not even agreed in our industry that JavaScript-based web applications are a good thing or that the browser is web application runtime, those are things that are up for debate. But I think if we make that assumption, this is sort of the founding principle of where Ember came from and it executed to the best of its ability at the time and that philosophy is, I think you can prove it out in terms of results based on if you have two different applications, one of them is built by somebody trying to jam together best in class components and the other person is starting with an end to end solution with a community of people rallied around that solution. It's been interesting to watch those approaches play out over time. I know Chris has a very specific hands-on experience of having done both of these. I'm curious to get your hot take. CHRIS: There's actually a concept that I think about a lot in relation to this question. It's something that I actually heard come up again recently so the timing was great but it's called hypocognition. The idea is hypocognition is when you either just like can't see or can't understand some kind of cognitive representation of something because you don't have the words for it. An example is in Western cultures, especially like in English speaking cultures, there are not that many words for the color blue but in a lot of other cultures, they have many, many words for the color blue. After doing a big study they found that these English speakers actually have a harder time recognizing different shades of blue, like more of them just look the same versus other cultures where their brains are actually wired to see all this variety because they actually have the linguistic representations for these ideas already. When you were talking about maybe a library is a framework at some point, I think that's right on. I think one of the things that I think about a lot when talking about frameworks and seeing these debates happen on the internet about, "What is a framework?" but also like, "Do you even need a framework?" is obviously, there's a lot of people who absolutely... Like Ryan Florence. Ryan Florence clearly knows what a framework is. He knows what it takes to build a web application and he does not lack the words to define a framework versus a library. He's just made that choice and it's a very informed choice but I wonder if there's also a lot of people who are getting into web development for the first time and they look at something like a framework and it seems just absurd to anyone would want all of the things that like in Ember or in Angular is talking about, when they can make a basic UI with React and it's easy and fun and really cool. But then this two-year path happens and they look back and they've learned a whole bunch and now it's like, "Ooh, you couldn't even have explained this to me before," because all of the words would have fallen on deaf ears but now suddenly, it makes a staggering amount of sense. CHARLES: Right. I love that. BRANDON: You have to make a bad one. CHARLES: Just so that you can inherit the vocabulary to understand why you made a bad one. Now, you guys actually have some experience with this. Brandon, you gave a talk about it, which I think you should give more widely because it's fantastic but for those folks who may or may not be aware that they are walking this to your path, I want to talk first about what are the signs that you're walking along this path and then two, what are the consequences in terms of the cost you're paying for walking this path. Let's start with that first thing. What are the signs? How can you tell that I am building a framework? CHRIS: I think one of the telltale signs and one of the biggest red flags that caused me and Brandon to have a very serious heart to heart about our own personal framework was when we hit the point where you could look at a set of tickets for features and all you saw was 'framework features' that you needed to write before you could build the feature itself. You know like, "Oh, we have basic routing setting and we have it set up so that if you have a route transition and you would like a data request to happen when a certain route transition happens, that will happen," but then someone would like infinite scroll and we want to use a query param. When a query param changes, I want to update the query and fetch more records, except that the glue code that we wrote to tie our router to our redux async stuff is not aware of query params. It has no concept of what a query param is or what to do when it changes. Also, it has no concept of refetching the data without a full route transition, so what do we do, this person wants infinite scroll but I first have to implement several layers of framework code before I build the UI feature that you want? CHARLES: The basic heuristic there is ratio of direct feature code to code that supports the direct feature code and code that supports the code that supports direct feature code. It's anytime you're anywhere above that first layer on the stack. CHRIS: Yeah, I think Taras nailed it like what's the 'what' versus the 'how.' If you're asked a question that is concerned with the 'what' and you spend more time focused on the 'how,' then you might have a framework. BRANDON: I think people will think of building an application like a recipe. If you think of it in those terms, people think of frameworks as very restrictive but I'm a big fan of Blue Apron, a sponsor of this podcast. Thank you. They pre-select the ingredients and they give you the instructions and you know what to do, you still have to do the effort but you know if you connect these pieces together properly that you're going to wind up having a good experience and then, it gives you a lot of freedom to experiment and be creative beyond that, should you choose to. I think one of the signs that you've done a crummy job is that you're staring down, like Chris Freeman said, you actually starting to restrict your choices like, "I can't actually build you that feature because we don't have time to take on the amount of work necessary to build the support structure, to build you that feature," or if you find yourself writing a test framework. CHRIS: Oh, yeah we did that too. BRANDON: You know, we were real deep in this. There are developers that are like, "I really want to feel like I'm walking into a grocery store and selecting all the things necessary for my recipe," and so it really depends on what the problem actually is. If you're working at a giant megacorp and you have a two-year timeline to deliver something and their goals are not about delivering stuff on a tight turnaround, that's usually a recipe for a software failure anyway but let's say, that you're in the 5% of those types of projects that's going to succeed, that might be a good place where you can say, "What we're trying to do here is so custom and we have such a long lead time and a long leash and such a high level of internal expertise here that we should be shopping in the grocery store and we should be selecting all these things and we should be solving these problems." Basically, when is it time to use a framework? Well, when you don't have 10 times the time you think you do, when you don't have the ability to spend 80% to 90% of your time in the first three to four months of your project, maybe six months, debugging you're glue code in between the different libraries that you're gluing together and then coming back and realizing that you've painted yourself into a corner and you have to re-architect your whole framework, then you could be so proud of this baby, 18 months to two years from now, when you actually have delivered both a framework that took about 70% of your time and an application that took 25% or 30% of your time. CHARLES: Yeah. I think it's important to realize that people think we'll do it and we'll build it as we go but I want to call out right there, you will be spending 80% of your time and you have to be upfront about it. Of this two years, 18 months of it is going to be spent building this framework and six months of it is going to be spent actually writing the feature code and you have to be 75% of your tickets or your issues, whoever track the work, 75% of that has to be dedicated to the framework. BRANDON: If you're going to bake in that kind of overhead purely for the satisfaction of a single or one or two developers that like inventing things, that is literally the worst possible reason you could do that. That is almost like a guaranteed recipe for failure. It has to be for some other business reason like, "We want to be the company that owns this." There has to be business value attached in making that kind of investment. If you can't justify that at the outset, then you should probably just go ahead and lean on an existing framework and join a community of people. CHARLES: Yeah and I think one good litmus test for that is, "Is this a 'what' for which there is currently no 'how?' One of the reasons we're writing BigTest is because for the general JavaScript community, there are a number of acceptance test frameworks out there but the market is very, very limited. When we look to actually acceptance tests, our React application, this thing does not exist. Now, we had experience with something that was very like Ember specific and so, we kind of knew what the 'what' was, we experienced the 'what' but there was no 'how' for our current situation. That's like a place where you might be called upon where makes business sense to actually invest in a framework. I'll tell you another thing too is if you have made the decision to kind of follow the beaten path on the other areas, then when a framework is called for, you have the bandwidth. You've allowed for the buffer, for the margin, for you to write in with that framework, whereas if you're already just by default, maintaining all the glue code in every single thing, then if some unique 'what' comes along, for which there is no 'how,' you're not going to have a bandwidth to tackle it. BRANDON: Yeah. That's a real bad situation to be in. TARAS: There's something else that I find interesting is because there's a certain point, like this two-year mark where everyone's like, "We want to fix this now." I think what is interesting what comes next which is the three years of undoing all the stuff that you made because the biggest challenge, especially in really big projects. When your projects has to borderline into platforms and a platform threshold is when you have a multiple teams working separately to write separate modules that run, maybe in a separate Git repo and maybe, packaged in separate npm package and assembled together. Then what happens at that point, the question arises like how do you actually make this changes in this environment. Answering that question is actually really difficult. I think if you look at frameworks like Ember, Ember has made it their business to figure out exactly how to make this happen and I think they've done it really well but it's a really challenging endeavor, especially in incorporate environments where they don't have an update. You have like upgrades are like a curse. It's like a thing that you don't really want to ever do and because most quite often, they don't have the right testing habits in place to be able to support the change if necessary. I think what a lot of times happens is that the team that made the framework in the first place, they end up trying to maintain a fort but you won't have like 10 people and they only have machetes, you know? All you can do is run around and try to chop down little twigs but at the end of day, the trees is still going to keep growing. I think that's the really challenging part of being two years into a project, where you realize that you actually need something much more comprehensive than initially thought you needed. CHARLES: On that, assuming that you have decided that you are going to make a framework, it's a good business decision for you. Based on the criteria of this discussion, how can you assess whether it's good? Chris, you talked about needing to integrate query params with routing and asynchronous data loading and making sure all of that coordination happened and worked together easily. What's the difference between your framework just missing features kind of having holes in it that can be filled in, versus something that's not good and it's going to cost you lots of money down the road. CHRIS: Yeah. TARAS: One thing, if you look at what makes a good library of any kind, it tends to be like how effectively and how much words to take the address the use cases that you need. The problem is that to build a good framework, you need to understand the use cases. This is what usually happens over time. Two years in, you've actually understood the use cases and now, it's time to change and so, I think if you want to build a good framework, you actually need to understand those use cases quite early on or account for understanding use cases over time and that's a big question -- how do you figure out how to know what you don't know. CHRIS: Yeah, I think that's exactly right. I think about what you were just saying Charles and Taras like one of the things that I think has a big impact on and what this process looks like is the completeness of vision for what's your project actually is. If you have a very, very clear idea of what the entire product you're building is going to be or, at least what the key money-making feature is going to be and you can understand the ins and outs of that, then I think that's the point where you can look at what you have and say, "Have I created a good or bad framework? Does this framework have the ability to solve this one very important thing that I have to be able to do? If the framework doesn't do it, then I need to build my own but I now know what very important features I need to front load my framework with." I kind of think of it as imagine that you're like Jeremiah Johnson, the Reverend Jeremiah Johnson and you're going to go trekking through the woods for some unknown amount of time and you have no idea yet. You don't actually know where you're going. You don't know what you're going to see. You don't even know what's out there because you haven't done the research or whatever and you need to be prepared for anything, so you bring just a hodgepodge of stuff. If that's you at the beginning of your company or the beginning of your product and yours is kind of like... I don't know, we got to get product market fit and that means that we may have to kind of pivot once or twice or we need to be very flexible, then I would think long and hard before you commit to writing your own framework because you don't even know what framework to build and you might as well take a broad array of tools and use what you need. There will be times where that's frustrating and there won't be exactly the right tool for the job but 80% of the time, it's going to do just fine but if you know you have to do this one very special thing and you know that a framework is going to give you a lot of stuff that you won't need and it doesn't really excel at the one thing you do need, then don't force the framework. There may be time to build your own but just know that you need to go in with a very clear idea of what you're doing before you start building the abstractions that constitute a framework, rather than just like a constellation of libraries. CHARLES: I have a question on that then. Going back to one of the things we were talking about like React plus Redux. Your opinion, Chris that it is not a framework, so the question is does a framework actually exist for React? CHRIS: My guess is that many frameworks exist for React. CHARLES: Is there a public framework? TARAS: There is one called Fusion but it's [inaudible] what you would have imagine. It is essentially Redux and React together conventionalized. They addressed a bunch of concerns around service rendering and such but it does exist. CHRIS: How about Next? Next.js? TARAS: I'm not familiar with its features from a single page application perspective. CHARLES: I think it does have a router. It does bundle with Redux and this is one of the things that when you first started using Redux, it's like, "How do I even get my store to my components?" Yes, I can connect them but there's actually a lot of stuff that you have to do. First, you have to say, "I'm going to put my reducers here and then when I create my store, I'm going to fold all my reducers. If I've got a whole bunch of reducers in my application, I've got to fold them all together. I've got to pass them off to the store. When I create the store, I have to inject the middleware and then, everybody else just imports my store and then, I have to put in a provider and then, I can connect my components." That's actually a lot of stuff that you have to do and I think that, for example, Fusion just says, "Put your reducers here and we'll take care of all that process," and so it makes that decision for you, right? It says, "For state management, you're going to use Redux. For your reducers, they're going to go here. For your actions, they're going to go here." I don't know exactly how it's laid out but I remember reading the ReadMe, it was basically layering conventions over that. That's definitely going into framework territory but that's the only one that I know of, which is really, really odd. TARAS: There's something interesting that's happening also and this goes to what Brandon was saying earlier is that choosing the best in class, there's this 10 things but then, what if one of the best in class stops being the best in class. The fact that the creators of Redux was essentially saying that we needed to basically provide a way to do Flux that was better than 10 different options that were available, so here's Redux. We've created Redux but we don't really think it's ultimately the solution. We need to have something else in React that provides a foundation for us to be able to deliver a better state management than what Redux is, so what happens when one of the best in class is no longer the best in class? The bridge is already standing. There's people walking across the bridge already. How do you replace one of the chains in it? CHRIS: Over the course of six months while you figure out the differences in API between Flux and Redux and all the custom route transition data loading stuff you did with your Ajax library in your state management software that you put in a case statement inside there that you now have to change over. It's easy. It's no big deal. Don't worry about it. BRANDON: Just a simple matter of programming. TARAS: At least 25 years of collective frontend development experience is laughing like hyenas about the simplicity of building a -- BRANDON: Yeah, I'm actually looking at some of the old code that Chris wrote for trying to glue together, Redux Saga. I've been out of the game long enough to not know whether that's been superseded by some new or best in class piece of technology and even then, it was really challenging. This is true for frameworks too, is they don't really optimize for best in class. They optimize, hopefully for best fit for purpose but the world has moved on since Ember launched obviously. A lot of things have changed and it's, at least as difficult to try to keep that up to date with evolving trends and technologies and updates for a core team at a framework level as it is for you, as an engineer on the team. The difference is you get to outsource that work to a core team for a framework. Ember has not done a fantastic job in keeping up with. They've done a good job and they've tried their best but if there were more people working on it or if there was more effort applied to it or if it was a higher priority, you would see Ember being a more up-to-date framework using more modern tools. As a framework author, if you stay too close to the bleeding edge, all you're going to do is change out your build system. You're going to replace a Broccoli with webpack, with Rollup, whatever's after that. What's new in Packer? CHRIS: Parcel? CHARLES: Parcel. BRANDON: Parcel. You should immediately go build your framework with that and have fun. I am excited by the new and interesting stuff that's happening in these ecosystems and I think it's important not to get lulled into the siren song if your goal is to actually ship a piece of software on a timetable or a budget. TARAS: One thing, if it's a red flag, if you think this is easy, if you think your decisions can be made in this isolation without talking to somebody else and actually kind of flashing it out, then you're probably doing something wrong because a lot of these things are not trivial. There's a lot of thought, there's a lot of considerations used to go into decisions that you make, especially when you're creating something that is going to be used by more than a few people. I think that's really one of those things where it's hard to know what you don't know but if you think you know and you haven't done this before, you haven't done this a few times before, you're probably missing some pieces. BRANDON: Yeah, I agree with that. CHRIS: I think one of the things that's really enticing about React and Taras, you just hit on it but I've never felt as clever as when I was writing a React app. If I'm clever, I mean, clever in the same way that I felt really clever when I wrote some unbelievably convoluted Scala one-liner that six months later, neither me nor anyone else could decipher what it meant but at that time, I felt like a god of programming. That's how it felt like, "Well, a lot of the React stuff is addicting." It felt so much fun. It was so much fun until I really had to do something and it mattered for my job and there was a deadline and people were depending on me and I've realized that the clever thing I had done a month later was not the right clever thing but I can see how, if you're like what Taras was saying, where you are at the point where these decisions are easy. These decisions make sense. We're going to be fine and you haven't done it enough to kind of like know where all of the pitfalls are. That cleverness that you feel is fantastic and I can see why it takes two years before you look back and if the cleverness was finally worn off and then, you're just mortified at what you've done. CHARLES: Pride cometh before the fall. CHRIS: Yeah. BRANDON: It's like being a dungeon master in Dungeons and Dragons, where you're like, "Oh, look at this fiendish world." All right, cool. Now, you actually live there though. I have to move into an apartment on Mordor. TARAS: You know what's the funny flipside to that is that coming from Ember world where it's so normal to leverage the work of other clever people, like really smart people who've invested a lot of time to solve a particular problem, is that there's no stronger sense of being dumb than having to write it from scratch in React. That first feeling of like, "I've actually never had to implement this from scratch," and I feel like a bunch of applications before but because I've leaned on for accessibility, I've leaned on something that someone else has done and it worked really well for me and it was perfect. But now, I need to implement autocomplete from scratch in React and I have no support. I'm basically learning as I'm going on this and it's that sense of discomfort that you get from having to do it from scratch and then, comes the euphoria of having to figure it out. But if you figured it out, you figured it out in the last month. You've written it for the first time in the last month and you now understand what all the things that the Ember implementation does for you. It's an interesting psychology of doing this -- CHARLES: Yeah, it gives you a lot of perspective but you have to ask as a business owner, who may or may not be technical and this is the hardest thing for technical people who are business owners is to be able to not see things through a tactical lens. Is what you really want to pay for is to basically give your programmers this kind of a-ha moment of their own shortcomings because that what you want to be buying. BRANDON: Yeah, you want to maximize leverage. Your goal with technology is to maximize leverage. It's like being hired as a chef and you walk in and then you're like, "I'm a terrific chef. I worked in these fancy kitchens in New York and I'm known as a great chef," and they're like, "Okay, cool. Here's some flint and steel and a spear." CHARLES: Go hunt. BRANDON: You're like, "Wait, what?" Yeah, yeah, yeah. Show me what you can do. TARAS: We had a conversation in one of the previous podcast with Michael Jackson and we asked him, "What is the one thing you wish like React community would do more of?" and he's like, "I really wish React community have more conventions." All of this is to kind of say as like, there is a place for frameworks in React world. There's a very strong place for it. The question is how and what it said and how do you actually build it and when do you --? BRANDON: So we need a framework for making framework. TARAS: Getting really meta here. BRANDON: I totally agree with that and that's a great observation and that was actually the point of my talk as well, which is if I could convince people just to use Ember and improve Ember, that would be great because I think it's a really great starting point. But the React community is much larger because it had such a great adoption story. The adoption of Ember was very difficult and the adoption of React was very easy and it expanded to include the scope of full end to end applications in terms of what people thought the problem spaces they were thinking of with React. Ember was built to solve that but it was hard to get into. React was really easy to get into but it's actually hard to build applications with. I would love to see a dedicated subset of the React community, except the idea of shared solutions and the philosophies that made Ember into sort of a powerhouse of value delivery but built out of tools that satisfy the React community and a little more modular and a little more available for people to customize and built in that ecosystem. I'd really love to see that that included all of the main components of what we accept as, "This is an application framework. It handles testing. It handles accessibility. It handles data loading," and it doesn't have to be best in class in every scenario but it does have to be a reasonable bridge across that chasm and have a group of people look at this the same way. I would love to see a collective subset of the React community dedicate themselves to this idea. I don't know if that's too culturally opposed or even orthogonal to what the value system inside the React community. I haven't been able to fish that out but I would really love to see that emerge. this is something I would love to push for and I'd love to see other people jump in and push for as like, "What if 20 of us got together and decided we're all building our applications in similar ways, instead of one person saying, 'I'm going to use --'" Even create React app is kind of a Band-Aid on that, it isn't useful past a certain stage of life. I would love to see a group of people, though, get together that are sort of like-minded like that, the Michael Jacksons and maybe even Dan Abramov or a group of people that shared that set of values or came into React from the Ember community. That's actually one piece of advice I would give to people. You said, "How do you convince this engineer that they've built a bad framework?" Use a decent one. That's the biggest guide. Use a decent one. Build something in Ember and ship it to production and go, "Oh, I get it." If you've used a good framework, you can't go back to rolling a crappy one. Your standards have been ratcheted up. CHRIS: I wholeheartedly agree that you should try something else and Ember is a great option but I don't want to dismiss just like, "React is cool as hell," and there's a lot of stuff in React that's really, really awesome and things that I wish that will show up in Ember and they are starting to show up in Ember but they're taking a while and it'll be nice in there but who knows when that will be but I would encourage even more so is both sides, like Ember folks who are listening to this podcast, if you have never messed around with React because you feel some kind of tribal affiliation that you can't betray, please set that aside and go do something in React because you will learn a lot about why Ember does what it does and you will see a lot of really interesting things that will probably jostle some ideas loose in your brain. The same thing goes for React developers. You, 100% should spend a weekend building something in Ember and nothing about that means that you have to switch or it's going to change the path that you're going on at work but I guarantee you, you will go back to your React application with some new and fair useful perspective that you didn't have before and that's okay. That's great. There's no identity crisis that will come about as a result of that. CHARLES: That is a fantastic advice, Chris. It will only stretch you. CHRIS: Yeah. BRANDON: I think developers have been sold this idea of a competitive landscape by authors of these frameworks because it helps sell the framework. You can build and strengthen a community by leaning into the tribalism that can surround the usage of a tool. The older I've gotten as a person who was deeply tribalistic about Ruby on Rails when I got into it and Ember when I got into it, because I love tribes, I think tribes are awesome and it's a way to make friends but when you really lean into that, the costs are too high and experimenting with other technologies and noticing flaws in your own technology is not only not a betrayal, it's actually critical to your growth as a developer. The more people that do that, like Chris was saying, the better both of those ecosystems will get. CHARLES: Absolutely because having spent as much time in React as I have, I really appreciate the precious things about Ember. It will make you appreciate the things that you hold dear. It will make you appreciate the really, really, really special things about the tool that you're using and at the same time, it will highlight the weaknesses which you can immediately use to feedback and make your tool better. It really is a win-win situation. TARAS: I just want to do a little plug before we close up. I think the feels of working with Ember is actually gone into microstates and we're still getting our things together to make microstates look accessible and usable by everyone but that feeling of pleasure that you get from working with Ember and just things just being there for you, like we really want to reproduce that and make that available in React community and the stuff that we do in microstates is actually really designed for that. CHRIS: Yeah, I see that in BigTest too as well. That's definitely another place where it's like, "These people definitely used to spend time in Ember and they're now in React-land." It's cool to see that stuff getting ported over. CHARLES: Absolutely because it fundamentally changes your taste. Working with an application that doesn't have like a bolted on testing framework is like eating water soup. You just can't enjoy your life. It really is flavored everything that we do. On that note, we can go ahead and wrap up. There actually is some pretty exciting news. We're actually going to be launching a BigTest launcher. Up until this point, you kind of had to roll your own using BigTest for your assertions but using something like Karma to actually launch the browsers and we're actually launching our own launcher. I guess we've written our own launcher and we're going to be pushing it to NDM, not to overload the word launch. You can look for that in the next couple of weeks. There's going to be a CLI that ships with BigTest to help you do even more set up, to make it so that you can just drop BigTest right into your application, whether it's jQuery, React, Ember, you name it. That should be really, really fun. Be looking for that and with that, if anybody has any other remarks... BRANDON: If people are coming through RubyConf this year, I'll be there talking about management stuff. That's my only near-future conference stuff coming up. Hope to see some of the more Ruby-flavored folks out there. CHARLES: All right-y. Definitely, go to every single talk that Brandon ever gives. You won't regret it. I can base that on very dear personal experience. You won't be disappointed. You know, not to put the pressure on or anything like that but you could never put any more pressure on Brandon than he puts on himself. With that, we will say good bye. Bye Chris, bye Brandon. Thank you so much. This is a great conversation. It certainly clarified a lot in my mind -- TARAS: Yes, same here. CHARLES: -- About these problems. With that, we will say goodbye. Thank you for listening The Frontside Podcast. Please get in touch with us at @TheFrontside on Twitter or contact at Frontside.io on email. We do a range of custom services from full stack project development to JavaScript mentoring, to as you go JavaScript help desks kind of stuff. If you need to reach out to an expert, please get in touch. Our podcast as always is produced by the inimitable, Mandy Moore. Thank you very much and we'll see you all next time.

In this episode, Kristian Freeman tells us about ByteConf React: why he decided to start the conference, unique challenges of putting an online conference together, what he expects in terms of viewership and his hope for sponsors, and supporting speakers who haven't recorded videos or maybe haven't ever even given a talk before. ByteConf will take place on Friday, August 31, 2018! Grab your ticket! References: Twitter Facebook Twitch This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. TRANSCRIPT: ROBERT: Hello, everyone. Welcome to Episode #108 of The Frontside Podcast. I'm Robert De Luca, the president here at Frontside and I'll be your episode host. Today, we're going to be discussing ByteConf, which is an online stream conference with Kristian Freeman. Kristian is a developer at Product Hunt. With me today as co-host is Wil Wilsman who is a software developer here at The Frontside. Before we jump into the discussion, I would like to make a little announcement. I'm going to be speaking at ByteConf and JSConf for BigTest. If you're interested in hearing about the next generation of UI testing for single page apps, you don't want to check that one out, I think. Without further ado, let's just jump into it. Hey Kristian, how are you doing? KRISTIAN: Hey. Thank you for having me. It's a pleasure to be here. ROBERT: You recently just moved to Austin and you moved from LA. How's that thing going? KRISTIAN: It's actually going really good. We packed all our stuff up and drove all the way across the US. I guess that's not really across the whole US but it was a good move. I've been here for about two weeks or so and I'm really happy with it so far. I have a full office and recording space now, so it's glorious. I can actually sprawl out and have mikes and all kinds of other gear out. I'm very excited. WIL: Nice. Now, what is it you do at Product Hunt? KRISTIAN: I'm a software developer there, kind of full stack engineer. Product Hunt is architected basically as a Rails app in the backend and then, a React app in the frontend, so I'm doing React obviously and GraphQL stuff and then Rails on the backend. It's been really cool. It's a really neat product and there's a really cool community there as well, so I'm been very happy with the transition. I've been there since the beginning of the year, so February or so. ROBERT: That's awesome. Are those projects split so it's like the React app separate from the Rails apps or is it like a monolith? KRISTIAN: It's a good question. I am not sure if it's planned to be split up eventually but as it stands right now, it's actually all one big application. We just have kind of that classic frontend folder where everything is dropped in and that's React related. We have some interesting stuff around server side rendering and things like that. All of that stuff was kind of there before I started working at Product Hunt but it's been really interesting coming from my previous gig where it was just very straightforward Ruby on Rails and then React like it's a totally separate thing, like they weren't really related at all as working on a couple of different projects. Coming to this, it has been really interesting. It kind of gives me a better sense of what Rails projects might look like in 2018, if that makes sense. ROBERT: Right, yeah. Are you using like the webpacker gem? KRISTIAN: I don't. I mean, it's not old really, in terms of web projects overall than in terms of like a Rails app. It's still running on an older version. I think we're pretty homegrown set up. We're not using webpacker. We kind of set things up and run them as like two different processes and stuff like that. It's been really interesting. There is a bit of onboarding stuff that it took me a while because I came from doing, like I said, kind of standard Rail stuff and I would say that they are really kind of pushing what you can do with half Rails, half React set up. There's a bit of time of me kind of flailing around and figuring out what was going on but it's been really cool. I definitely feel like I leveled up in my understanding of how all that stuff can fit together over the last couple of months or so. ROBERT: That's awesome. You're running a conference, right? You're running an online-only conference for free and streamed on Twitch. That's pretty bold thing, right? That's a new concept. How do that come together? I guess before we even discuss that, what is ByteConf? KRISTIAN: I'll start with the synopsis of what it is and I can talk a little bit about history. It's actually a conference series. I have other events planned and the first of those is ByteConf React, which Robert is speaking at. It's a React in JavaScript conference that is streamed on twitch. If listeners aren't familiar with Twitch, it's a live streaming platform. Kind of the primary use for it is for gaming. There's a lot of people who will stream themselves playing like Fortnight or whatever other things. I usually just watch Fortnight. It is primarily for gaming and in the last couple of years, they've started doing this separate section called the creative section, where there is non-gaming stuff. I've watched people paint. I've watched people play guitar or take song requests on the stream and all kinds of interesting stuff. What I can say is the idea for the conference wasn't a thing that I just came up with on the spot. I actually attended a really small conference that was streamed on Twitch. It was a game development conference. I wish I could remember what it was called but it was really interesting. There wasn't too many people in the stream but I was impressed with the way that it was put together, especially for a topic like game development, which is I would say, I kind of took a stab at it last year and I found it to be pretty difficult to get into and fairly opaque in terms of understanding how to get started and then make that progression into a career. I watched it and I enjoyed the format of the conference but I didn't really get learned too much from it because it was just too kind of complicated, I think unless you already were in the field. I started thinking about that and I thought it was a really interesting model and I felt like web development in particular, would be a really neat way to approach that format because web development has, I think a reputation if you want to get into programming. I don't know if I would say, entirely approachable. There's still a learning curve and there is a lot of work that you have to put into it too to get like junior dev role somewhere but I thought it was really interesting. I thought it would be interesting to take that format and apply it. My background, besides doing straightforward full stack engineering, I've done some courses for Pluralsight. I've done some in-person technical training, so I had a background in teaching and I felt like it would be an interesting just to try it and see what would happen. So far, we're a couple of weeks out from the conference and honestly it's been pretty wild how many people are excited about the conference. I don't think I've ever done a project on my own, like a side project that has had people that just tweeting about it without me prompting it or anything like that. I saw something on YouTube today like a Spanish YouTuber who does tech news and he was talking about the conference. I don't know Spanish. I was curious and wanted to see what he's talking about but I couldn't really understand what he's saying but I saw the logo and I was like, "What? This is crazy." I'm really excited about it and I'm sure we'll kind of get into this but there's some really interesting implications and ways that this format, I think it will be a different approach to the usual tech conference format. I'm really excited about it and I think it's going to be really neat. ROBERT: It's awesome that it's free and available for anybody at any time. KRISTIAN: Right and a part of that, I think when I started thinking about it like, "Can I make it free? What are the implications of that?" I think that the main thing is that when it comes to running a conference, getting the location or whatever, I would say is probably by far, the most expensive component of that. For me, I'm a remote developer and there's a lot of people that I talk to day-to-day. I think a lot of kind of my audience and people that I know online are also working remotely, so for the conference to just be online, it wasn't too crazy of an idea for me because most of the interactions I've done in development stuff like that have been through the context of remote work. But also, for people who aren't remote workers, who are getting into the field or even just have a small interest in web development, I think it removes some barriers of being able to access this kind of stuff. You know, if we can look back at the future and say, "This sounds very ambitious," but kind of a democratizing force of anyone can view this content and get access to it regardless of their skill level or economic level or things like that. WIL: I frequent trips sometimes and I know this is obviously, a free conference but are you expecting any donations? KRISTIAN: That's a good question. That's kind of one of those things I haven't really figured out the best way to do yet. In Twitch, I think there's the concept like 'Bits.' Is that what they're called? They're like microdonations. I genuinely don't know how that's going to work out. I know the plan is to take the talks after the fact and get any kind of additional slides and stuff like that. We're doing a couple of pre-conference events that I can talk about. I guess I should plug those as well before we wrap up but the way that that's going to work, I am not quite sure because I would like to sell the packages after the fact and actually, being able to pay the speakers. But in terms of little of bits and stuff like that, I'm not actually sure. I'm genuinely curious how that's going to work out. I don't know if people will do that. I guess it just kind of depends on the audience. ROBERT: -- In my talk now. KRISTIAN: Yeah, give me Bits please. ROBERT: "I intended to write, don't move to the next slide." KRISTIAN: Yeah. I use Twitch every once in a while. I said, I generally just watch like one game on Twitch but I don't watch Twitch all day, every day. I think it'll be interesting to see because this is probably a different audience than, say the average Twitch user. It'll be really interesting to see how that shakes out. I don't really have a great answer for you there. WIL: Do you have any guests like a number of people that are going to be attending? I see on the ByteConf site, there are 1500 subscribers. KRISTIAN: That's a good question. I guess I can talk about this stuff that we have planned before the conference. We've been building an audience for really, it hasn't been that long. It's been like four or five months since I announced it. We're using kind of dogfooding a thing that I've worked on a Project Hunt, which is this Ship product, which is for collecting emails and sending out newsletters and stuff like that. ROBERT: I saw that get announced not too long ago. KRISTIAN: Yeah, it's really cool. I'm really happy with the product and they have some built-in promotional tools on the site, which is pretty neat. But we have, I think like 1500 people on the mailing list. I think, we have, the last I checked, about like 4200 followers on Twitter. It's hard to convert that like how many people are actually going to attend. What we are doing and this is like A, because I think it be interesting and B, to kind of gauge this, hopefully as best we can. We're going to be doing some preconference like 'Ask me anything' interviews with some of the speakers and I'm hoping I can get a better sense of how many people will actually start attending any of the events that happen. The way the Twitch works is you can follow and subscribe. You'll get a notification when a channel goes live. The first time that we go live will be tomorrow, actually and so, we'll see how many people will turn out and it should be interesting. But in terms of actual numbers, I genuinely am not sure. I would hope that a lot of people who are on the mailing list will be there but it's been pretty neat. I've already been hearing of people who are trying to setup like in-person events, viewing parties and stuff like that. I've tried to help coordinate that as best I can without taking over the limited amount of time I have before the conference actually happens. Also, people in Europe and vastly in different time zones are actually kind of grilling me about, "When can I watch this? Will you do every broadcast so I can actually attend this because I don't want to see the conference at two in the morning," and I'm like, "Yeah, I know. I understand that." We're kind of figuring out those details as well. Like I said, I very much consider this like the MVP of a longer term event series so I'm excited. I think it will end up building something that a lot of people will attend multiple times and hopefully, we can expose people to new stuff as that happens. ROBERT: You mentioned that somebody in Europe like wanting to know like, "When I can watch this?" which actually makes me wonder like that's one of the unique challenges that you have for an online-only conference because no one's going to be asking that question if the conference is in... I don't know, LA, right? Everybody knows where it's going to be because it's all co-located, so what are some other unique challenges that come with running an online conference? KRISTIAN: That's a great question. We don't have the explicit location and time that it starts to kind of point people to. In some ways, it's a positive thing. We have a lot of people who can attend that normally wouldn't be able to. They're excluded by price, location and stuff like that but there are some things that you, I think kind of give up when you do the online format instead. One of those is just being there at the conference and running into people that may be you know or having sponsors with booths set up, where you can make a connection in that way. Some of that, we're trying to solve by building an active community. We have a Discord server that we started a couple of weeks ago, where people are chatting about this kind of stuff. I really think of it as like there's probably many ways to solve this problem and I'm trying a couple of different ways to see what sticks. Building a community where people can continue to talk before the conference, obviously they already are doing that and then continue after the fact and build the kind of connections and relationships and community that would maybe happen organically or at least, have a chance to happen organically in an actual physical conference. Some of the stuff, I genuinely still trying to figure it out like how best to give people the sense that they are welcomed and I guess, kind of feel like they're part of a community of developers. I remember when I started the conference, one of the first things I thought about was when I first went to a conference in San Francisco that Heroku put on and I remember being there, I was very, very junior as a developer and I remember sitting there being like, "Whoa. This is probably the first time I've ever been in a room with a bunch of other hundreds and hundreds of developers," and it was real interesting. It's one of the first times that I was like, "I actually am for real, doing this. This is pretty cool." I'm trying to figure and imagine and we'll iterate on this in the future like how best to give people that experience. Maybe that means doing a couple of physical ByteConf events. I'm thinking about that definitely but also, how do we keep the original idea of the format but also, how people feel like they're part of a community. It's very much a work in progress. WIL: I could see a future where you have a physical, smaller conference but you still stream it on Twitch and everybody could still attend. KRISTIAN: Exactly, yeah. I think that's probably the format. ROBERT: That'd be rad. KRISTIAN: One of the things that I thought would be interesting would be to do some kind of and actually, I think about this before when I moved out to Austin, like doing some kind of West Coast tour where we go up the West Coast and do events, maybe every a couple of nights in a really small format. The same kind of conferencing they have people from that area, come and give a talk but still stream that on Twitch. It's kind of a hybrid approach that the people who are already part of the community can still attend but for people who want that physical experience, they can do that as well. ROBERT: That's awesome and if you did that, then you wouldn't necessarily lose the whole way track. One of the things that I really love about attending conference is like the talks are great but I usually always find those online afterwards. But what I can't find online afterwards is the communication and the talk that I have with people that are there. That's an interesting challenge to have, maybe you could have... I don't know, like not to tell you how or what to do, maybe like a channel in the Discord for a Hallway Track channel or something that encourages conversation, maybe outside of but in connection with the talk. But I would just say, maybe that's just one of the tradeoffs that you're willing to have for an online-only conference. There are a plethora of things that you just shed by not having it out at a physical location, like a bunch of cost for one and AV setup and worrying about people connecting and getting and presenting properly -- KRISTIAN: Via conference Wi-Fi. If they have problems, that will be their house Wi-Fi. ROBERT: Yeah, exactly. KRISTIAN: I totally agree. I think it's not the worst problem to have because we're in a lot of ways kind of simplifying and really, it's the kind of thing that we can iterate on over time. When I was talking about the European time zone thing, I may be sounded like I was bothered by people reaching out or whatever. It's actually quite the opposite. It's really exciting and I have actively kind of sought feedback out and been like, "How can I do this better? How can I communicate this decision or that decision?" or even help me make this decision so that it's best that I do whatever works best for the community and I expect that will, as the community grows, just be more and more a factor. I think that's the kind of thing that tying up like the Hallway Track or something like that. I'm confident that people will have opinions on that and they'll say like, "This is what would work for me best to feel like I'm part of this community," and we're going to definitely try those things and iterate on them. It's not the worst problem to have because there is really nowhere to go but up, in terms of how we do it well and stuff like that. ROBERT: Good problems to have. KRISTIAN: Yeah, exactly. ROBERT: All the talks are prerecorded, right? What have you done or have you done anything to help support people who haven't recorded a video for speakers? What are you doing to kind of ease speakers into this new style? KRISTIAN: Yeah, it's interesting because in terms of a speaking lineup, there is clearly, some people who have experience both as conference speakers, also in particular in this format. It's basically recording like a Screencast. It's more or less the same thing. It's slightly a different format, maybe condensed to a shorter, like an hour talk. There are some people: Kentcdodds, Tracy Lee, they're two of our keynote speakers, I guess you could say. They have a ton of experience. They're pretty much giving talks regularly all the time, so for them, this is this is no biggie. But there are a couple of people I've tried to, like in terms of once we got our CFP, our call-for-paper, we were accepting talks submissions and also getting information about the speakers themselves like, "What is your experience of speaking at conferences? Do you have any experience speaking at conferences?" What I thought would be in the spirit of the conference itself and kind of our ideals and even, I would say like the ethics of how we think about this kind of stuff, I do actually think about it in that term. We want to have speakers that represent that. You know, bringing anyone from any experience level, in any location and stuff like that and having them be able to attend the conference and also speak at the conference. There's a couple people that just don't have a ton of experience speaking at conferences or even keep doing this kind of Screencast format and so, for those people like kind of the silly one, I've just been reaching out to them like, "If you need any help on the stuff, let me know. I've done this a couple of times, at least the Screencast part of it, I have a ton of experience with them, so if you need help, let me know of that." Also, if someone needed it, we bought them a mike and a webcam and we sent it to them and be like, "Don't need to worry about that because that's potentially --" ROBERT: Woah, that's awesome. KRISTIAN: Yeah. That can be like an economic kind of thing to make people feel uncomfortable, like maybe, you can't afford a mike or something like that but we will cover you and no strings attached. That kind of stuff, I think is really important. I think, the kind of the main thing is we just want people to feel comfortable. There is no reason that because someone hasn't given a talk at a conference before, there's no reason they can't start. Everyone has something interesting to say, I think and everyone's experiences is really interesting and brings a perspective. Especially in the conference format, I think it will bring a perspective that you're not used to seeing at a conference. Not to say that the kind of perspective of people who are super experienced and things like that. As a developer, as a conference speaker, that's obviously really useful but it's also useful to see things from the perspective of someone who is just getting into the industry. I think that being able to amplify those voices is really interesting and exciting to me. I'm sure there's probably ways that we could do this better in the future but for now, it's been just kind of like supporting them whenever they need it and trying to be encouraging and then any kind of small things like buying a make or something, we can provide that. ROBERT: That's awesome. There are some tradeoffs you could make always with anything but it's almost, I want to say better, to give your first conference talk or one of your first conference talk in this way. I know I was really excited about it when I first heard about it because I get pretty nervous getting up in front of people. At JSConf, I don't know how many people. It kind of gives me anxiety but with ByteConf, it's pre-recorded so I have the ability to go back and polish everything that I want and I can remove those odds and things like, "Oh, wait. That didn't quite slow right. Let me fix that real quick," or, "I didn't really like what I said there. I can go back and fix it." It does come with the added complexity of like, "Now, I have to go and cut it together and make sure that there's this whole post-production aspect of it," but it makes me feel a lot better because I feel that I can deliver something that I feel really good about and I know because I've watched it six times and gone over it with a fine-tooth comb, you know? KRISTIAN: Right. One of the things that I am hoping that we can do in the future is in terms of the editing and stuff like that. If someone feels comfortable like really fine-tuning their talk and stuff like that and giving almost a finished product to us, we're happy to obviously accept that but for people who just don't have that ability or needs some help in refining, I don't want to say the quality of their talk but just kind of the delivery of it, we can definitely help with that. In terms of refining, say you're going to give the talk again in your case, I think it's really interesting also. We're trying to coordinate as with as many of the speakers as we can, kind of like time zone permitting and things like that, having them in the 'attending the conference,' or 'viewing the conference,' and also being available in the Twitch chat and not necessarily having an interview there but maybe, if something comes up or someone is like, "I don't get what this slide means," or something like that and that's both an opportunity for, we're not going to like pause the talk or anything but the speaker can be there to clarify and add that additional, I guess dimension of understanding of what's going on in the talk. I think it's actually really interesting. I'm really curious to see how it turned out. WIL: Yeah, I'm curious too because I [inaudible] with Twitch sometimes and most videos, like you mentioned before, you want like some small conference and it was a very small chat and a lot of Twitch, for me is interacting with the person that I'm watching, through the chat. It's interesting to me that it'll be pre-recorded but the speakers are still going to be interacting through the chat, so it's going to be real cool to see. KRISTIAN: Yeah, I'm trying to -- ROBERT: I'm pretty excited about that. KRISTIAN: Yeah and I hope that you would be interested in being mixed. I'm sure people will have questions about that kind of stuff. I've already talked to a couple of speakers and I'm trying to reach out individually and see how many people can be there for that because it's really interesting. In your case, if there's enough people to say like, "This part kind of confused me," or, "You lost me here," that's an opportunity for you to refine the talk and get really explicit timed feedback. I think if someone came to you after, say your JSConf talk and was like, "You know, there's this part that I don't really understand," like you don't have the immediate understanding of literally, at what point in the talk are they exactly talking about. I think that will be really interesting. That's -- ROBERT: That's absolutely could be bigger. KRISTIAN: Yeah and I'm trying to figure out the best way to do this. If you've ever been in Twitch chat before, it can get a little rowdy and I'm trying to figure out the best way to manage that because I have literally zero tolerance for whatever kind of the most toxic of Twitch chat as I have zero tolerance for that in the conference. I'm trying to figure out the best way to make that happen. But if you're a speaker and especially, it's your first time ever giving a talk or something, if you get that kind of feedback, hopefully it's delivered in a way that doesn't suck and we're going to try and mediate that as best as we can. That's a great opportunity for really effective improvement on your presentation style and stuff like that, so I'm really excited about that. ROBERT: It's actually interesting to think about is what kind of trolls you might run into and -- KRISTIAN: And you have people who are like, "Vue is better..." ROBERT: Yeah or any kind of cross-trolling that might happen. That'd be interesting to see how that plays out and how you might enforce that code of conduct. KRISTIAN: We definitely do have a code of conduct in Discord and so far, I'm happy that we haven't had to enforce that in any way or there hasn't been anyone that has brought the quality of the chat down. I've seen people answering questions about different open source projects and stuff like that. I think, Robert you wrote up a solution to someone's problem in the TypeScript channel or something. Did I see that? ROBERT: Yeah. KRISTIAN: Yeah, stuff like that is really cool. ROBERT: Someone who was asking how to do radio buttons in React and I was like, "I'll just write a quick code chain box example that kind of showing this." KRISTIAN: Yeah and if I could pick one long term goal of where I want to see the conference in the community in a year or two, I want to be able to scale that up to, say like 10x or 100x the amount of people but still keep that quality of conversation. I think that is really looking at producing a conference. That part, honestly isn't the most complicated part. It's if you can use Adobe Premiere, you can pretty much make a pre-recorded conference work. It's keeping that quality and making people feel like they are a community, especially for people who know that they want to be a web developer, maybe they have no idea where to go or how to start. If people can join the ByteConf community and feel like this is a good place, that you can call this place home, I guess online and learn in that way. That's kind of the larger goal. The conference is just one aspect of getting there. WIL: This is all very exciting. I'm looking forward to attending. KRISTIAN: Yeah. I am really looking forward to it as well. It's pretty wild that it's August, that it's actually happening soon. I'm really excited. It's going to be sweet. ROBERT: Yeah and I'm seriously working on my talk right now, to try and get it together. The cool thing that I found about that, I'm talking about BigTest and Wil is a person that's writing BigTest and he's the mastermind behind it. It'd be great to have him in there and answering any questions alongside with me as the talk goes on. I didn't even consider it before you said it. It is really powerful because I'm going to be introducing something that might be foreign to a lot of people, this testing style and how you do it in single page apps. There will be a question and I know I won't be able to cover everything and hit all the bases and make sure that's not confusing because it is a complicated topic. I'm going to do my best but the added benefit of me being able to clarify things on the spot is kind of mind blowing there. KRISTIAN: It's huge and I'm trying to figure out the best way to archive that kind of dimension of the conference. I'm really interested to see it tomorrow. We're doing an 'Ask me anything,' but I'll plug that at the end. It's going to be an interesting to see what the kind of ratio, like signal-to-noise is in the chat and if it's good, especially at the conference itself, if people are asking really good questions and that kind of stuff and the speakers are responding, that is a really valuable thing to try and save. I'm trying to figure out how to do that as well, even save the most requested questions or maybe the most detailed answers that the speakers have and making that available in some way, I think it would be really valuable to people. WIL: Yeah, for sure. ROBERT: The other thing I just have some of thought of too, with all this being pre-recorded, you are able to schedule this out pretty well. At a normal conference, if somebody had a 45-minute slide and they finished in, say 30, the conference organizers will then have to go and figure out what they're going to do with that spare time but with all pre-recorded, you can just kind of spot it together and have a plan going forward. KRISTIAN: I think most of the talks, I've kind of ask people to keep them around 45, 50 minutes and we'll have some space between the talks for people to continue to ask questions in the chat or I can plug things like the Discord server in those spaces and sponsorship infos and stuff like that. But I'm also constantly thinking of these little formatic conference allows so many different little things to be tested. One thing we're thinking about doing is like at noon, there's going to be a break, kind of a lunch break, but ideally and I need to start thinking this out, getting some lightning talk style things from people who submitted a talk and didn't get accepted or something that and those are -- ROBERT: Would those be live? KRISTIAN: That's a good question. ROBERT: Or pre-recorded? KRISTIAN: That's a great question. I think the thing with live is that I would have to figure out how to get people to hop onto the stream. That might be possible but I'm not quite sure. I think we'll probably do pre-recorded, kind of across the board for this one but there's all of these little opportunities to do interesting things with the format. One thing that, I will take you on kind of a journey here like where my mind goes and I think about stuff like this over the last year or two. This is going to seem like such a tangent but I'll tie it up, I promise. Over the last year or two, actually longer, it's probably the last couple of years, I was really into Anthony Bourdain and all of his shows and I was really interested in, again this is going to sound really bizarre but I was interested in taking that idea and applying it to conferences. For say, the keynote speakers, I was thinking like it would be cool actually to go and meet them wherever they're working and stuff like that and introduce them in that format and maybe even sit down with them and do an interview or do some kind of live coding with them and have that available as a bonus material to the conference itself. Maybe air some of it in-between talks as part of the preface for their talk. There's all these kind of interesting things. I think that one thing that always kind of bothered me about the developer world is, I guess I always feel like it's really hard to visualize how to get started as a developer and then what is the day in the life of a developer and what do you actually do. I think I've been really interested in this idea of giving people a holistic view of how to get into this industry and to show people. At least in my opinion, there is a lot of hype and maybe, not intentional but it makes a scene a lot more difficult than it really is. That takes a lot of time but there are a lot of people who probably have been, whether that's kind of the Steve Job's worship of tech people or this other thing that no one can be like them unless you're whatever, if that makes sense. Basically, everyone in this industry is just a normal person. Maybe, there are some crazy personalities out there who are really dominating or stuff like that but for the most part, I think especially in the web developer world, everyone is, at least in my perspective, very welcoming and just like normal people and I want the aspect of the community to be letting people into that world and say like, "This is not as impenetrable as you may think," and there's a lot of different ways to -- ROBERT: Amplify the kindness and amplify the welcoming. KRISTIAN: Exactly, yeah. ROBERT: I like it. You did mention like around lunch time, there would be a break. At other conference, they usually cater lunch. Is there anything offered for that or is it like go on and find your own lunch? KRISTIAN: That's actually a really interesting question. No, there isn't anything planned but now, I wonder if I should find a company that's like... Is that DoorDash? Is that grocery delivery, is that restaurant delivery? You know what I'm talking about like -- ROBERT: Restaurant delivery. KRISTIAN: -- Or something like that. It'd be cool to have a coupon like if they're React, they might want to sponsor the conference. That would be interesting. That's a super -- WIL: -- Delivery fee. KRISTIAN: Yeah, that would be super cool. ROBERT: It might be too late for that now since we're a couple of weeks out but some of those companies do used React or in the future, for the future of ByteConf series, like if it's a SwiftConf and I know you've mentioned that before, you might be able to be like, "We're doing a Swift online conference. You guys use Swift. Do you want to sponsor?" KRISTIAN: Yeah. I think there are so many opportunities to do really cool things. That's a really cool idea. I haven't thought about that before. I'm going to write that down. That's a very cool idea. ROBERT: Could you tell I've been thinking about like a ByteConf accessibility conference? Because I have. KRISTIAN: Yeah. Let's do it. For real, that would be sweet. The format, we can tweak it in so many ways. It's like a full-day conference plan but there's definitely the opportunity to do really small form, like just an evening or something like that, where you get a couple of people together. The way that I visualize it in the future is there are these longer conferences but also, it just be really neat to do kind of continuous -- WIL: Online meetup essentially. KRISTIAN: Yeah, like broadcasting. I don't want to say like a TV channel but like this place that we're going to be airing new stuff to the people who are working on and we're going to be airing old talks from conference and stuff like that and giving people a space to constantly be learning. ROBERT: You can do like a nightly techcast. "Tonight, in JavaScript news, there are 15 new frameworks." KRISTIAN: Yeah. The thing is like with Twitch, they've done a lot of tools recently that I've become kind of aware of as I'm trying to figure out the best way to broadcast the conference. They have a lot of stuff around scheduling and stuff now that actually gives us the opportunity to basically run, maybe not nightly but weekly or monthly thing without having to explicitly setup... I don't know if either of you've ever done Twitch streaming but you have this broadcast software that you have to run on your computer and stuff like that. They're working on tools to remove that aspect of it and really just make it almost like a YouTube competitor in some ways and maybe like a more live aspect. That's stuff is really interesting to me because that totally fits in with the kind of aspect of what we want to do but there's all kinds of other opportunities too. I know there are a growing number of people who are doing live programming streams and it would be really cool to be able to share our audience -- ROBERT: Coordinate that? KRISTIAN: -- And stuff like that, so I'm trying to figure that out as well. ROBERT: Are you familiar with the Ember community at all? KRISTIAN: I was familiar with the Ember community a couple years ago. It's actually is what I learned before I learned React but I think I'm pretty out of date now. ROBERT: One of the team members here at Frontside, Taras, he started something similar like that two years ago called Global Ember Meetup and it was just an online meetup that would happen at night and people would come on and give their talks. It was actually really cool because there's a lot of engagement from all across the world, which was super neat. I would love to see that idea to continue live on. KRISTIAN: I know for our mailing list, we have a sense of where people are located and this is the nature to advertise and stuff because I think our most of our audience is still in the US, Canada, UK and stuff like that but there is growing numbers of places like Africa and South America and stuff like that, where I'm not as exposed to that community but I would to make it available to all of those people. I genuinely just haven't been exposed to those communities as much and I would both like to understand the unique problems of being a web developer in those areas and also, do my best to adapt the format of the conference and stuff to those groups. I imagine that people are really excited about it but I think after the conferences, really one of a lot of the interesting stuff happens because we can take a look back and say, what could we have done better to include all kinds of groups that are historically disenfranchised from attending this kind of stuff, if that makes sense. ROBERT: Even for me, I really want to go to conferences that are in Europe but that's a big investment. It's like breaking down those barriers. I'm pretty privileged in that regard but for somebody that isn't, even just attending a conference inside the States or somewhere that even kind of close for them, just the price of the conference ticket puts them out, so I'm really excited about this idea. Why not leverage the web and make everybody available to learn in conferences and have access to that community. KRISTIAN: Yeah. I think I actually saw that Facebook just announced. They're doing another React Conference and it was interesting speaking of ticket prices, I think a lot of you were saying it was super expensive. I don't know what the exact number was, maybe you know but I actually had some people tweet like, "This is why I'm excited for my ByteConf," and I was like, "What?" WIL: That's awesome. ROBERT: I don't know what their prices but when Facebook throws ReactConf, you have to enter into a lottery. You wouldn't even actually get a chance to buy a ticket. You have to enter a chance into winning a ticket for you to buy. KRISTIAN: Yeah and that kind of stuff, I mean that won't get too deep into my politics in general but generally, that's the kind of thing that I am extremely allergic to. Even the idea of having a lottery and stuff like that, there's a lot of people who, to make the decision and say they have the opportunity to attend the conference, like if they say they get a lottery email like you have a ticket, there are some people who will be able to swing that on the spot and say, "I want to buy a ticket and start to book my flights and stuff or whatever," but there's a lot of people who that's going to be a thing they need to plan for a really long time. They don't have the opportunity to wait on the email and say, "Yes, I can go to this to what I'm being paid." That's just a different dimension of financial and I think the ticket was like $600 or something, maybe $700. It was expensive but there are much more expensive conferences. Especially, if you don't work at a company that covers your conference costs, like I am fortunate to the both places I've been at for a longer period of time like say, two plus years, have both sponsored conferences, they allowed any of their employees to go to conferences with some budget in the thousands of dollars every year and for someone to pay that, say they want to get into web development, that's a huge financial burden if you're working minimum wage or something like that. I feel like I sounds I just came down very hard on the React Conference but it's fine. It's cool that they're going to get really cool speakers and stuff like that but I think it's something. ROBERT: It's the job position of online-only versus co-located, right? There's talk there. KRISTIAN: Yeah and we talked earlier, maybe there's a hybrid approach of doing ByteConf physically, I think the one thing I will never compromise on in terms of how we put on the conference is like if we're going to do a physical thing, it needs to still be available for people who can't attend it. I think even at this point, the first conference hasn't quite happened yet but I do strongly believe that's already in the DNA of the idea and kind of ideals of the conferences I want to allow people to always attend, whatever we're doing, regardless of their situation. WIL: That's huge. I never attended a conference until last year when the company I'm currently working for, Frontside, paid for it. Before this, I had never been to a conference. It's awesome to see, they're like free [inaudible] by now. KRISTIAN: The conference I talked about earlier, the San Francisco one, I just straight up put that on a credit card, like I could not afford it. I did it because I guess I felt like -- ROBERT: That thing -- KRISTIAN: Yeah, exactly but there are people who just straight up can't do that. By that point, I was interning at a web development place but I still was basically getting paid like minimum wage. It was like under paid but I did it because I felt like it would be an investment. I didn't actually get a job from any one of that conference or anything like that, so who can say what the actual value of that was but it was important kind of in a motivational way but I don't ever want people to go into debt to go to ByteConf. That sucks. There's no way I'll allows them to do that. ROBERT: Yeah, because it's not only the conference ticket. Depending on what conference you're going to, I've seen as low as $150 and as high as $2000, just for the conference tickets and then you have to get your hotel for a week and fly out there and food. It quickly turns into a really expensive endeavor. KRISTIAN: It is in a lot of ways. I think for people who are fortunate in tech, it's somewhat of a vacation because you get to go somewhere. Usually, the tech conferences, I think are held in pretty cool locations, unless it's some kind of indie conf that doesn't have a lot of sponsorships or something like that. I went to a conference a couple of years ago that was at Disney World and it was very much a vacation. I went to the conference and I had a lot of fun. It was an Elixir Conference. I learned a lot of stuff there but after the conference was done, I went to like... I'm trying to think what it is called. It's like Downtown Disney, basically or whatever, so I went like -- ROBERT: Oh, they renamed it to Disney Springs. KRISTIAN: Oh, really? Disney Springs, wow. That sounds very -- ROBERT: Yeah, I [inaudible] for two years. KRISTIAN: Actually that does sound right. Coming from LA, I used to go Disneyland all the time. Even if the conference is just on a hotel or whatever, usually the area around it is pretty nice but that definitely limits a lot of people, unless you're fortunate enough to be making a tech salary or have a company that will cover that conference budget for you. ROBERT: We're sending two people to JSConf Hawaii. We were able to snag the early bird tickets which are so much cheaper. Then I was shocked that the hotel cost on Waikiki Beach was cheaper than my Portland hotel, so I'm actually super jealous and it was a super awesome vacation on the beach in Hawaii for less than what probably took for me to get to Portland. KRISTIAN: Is that where JSConf is? It's just in Portland? ROBERT: That's JSConf Hawaii. Portland was Chain React, so shout out to React Native Conference. The JSConf US one is in San Diego which is coming up in two weeks. Oh, my God. KRISTIAN: Nice. If you are a conference speaker and stuff, I think you get some stuff cover. I don't know. Every conference is different or whatever, so if you go in that format, if you go to conferences as a speaker, I think it's a little bit different situation but I can think of a lot of times that I looked at a conference and there's been a couple of talks that I found interesting but just the amount of money that I would spend to see one or two talks that really interested me, it wasn't worth it. ROBERT: At least ByteConf kind of shed that and absolutely drops the barrier of entry of to nothing. I mean, nothing mean you have to have an internet connection. KRISTIAN: Yeah but there's still a couple of things. This is why I'm trying to deal the rebroadcasting and making it available after the fact is there are some people who still can't take a day off and watch a full seven or eight-hour conference, so it's important to make it available after the fact too. I think I mentioned, I want to sell the conference talks with the slides and with the bonus materials and stuff after the fact. There's people that are actual, like practicing React developers who would feel fine paying like $30 for those or something and that way, we can hopefully, ideally, I hope I'm not totally speaking out of this to make totally go wrong but ideally, pay the speakers to some degree. That's another kind of aspect of it that I eventually would to do well in the future. But like you said, lowering the barrier to entry to literally as close to zero as we can get is what's really important to me. Then they feel everything else, we can work back up to something, putting on the really big conference events that a lot of other people are doing but still keep those ideals that we had from the first place. ROBERT: I love it. ByteConf sounds super awesome. I'm very excited to be selected to be a part of it. I really appreciate that. Is there anything else that you want to plug about ByteConf? KRISTIAN: Yeah. A couple of things, tomorrow depending on when this comes out, August 10th at 5 PM PST, we have our first 'Ask me anything' with Kyle Shevlin, who is a speaker at ByteConf React this year. It's just going to be Twitch.tv/ByteConf. If you're on the mailing list or you're in the Discord server or stuff like that, you probably already know about this but I will obviously tweet about it as well. A couple of other things. The 24th of August, we're actually doing and this hasn't been announced yet. This is the first time I'm talking about it. ROBERT: You heard it here first. KRISTIAN: Yay! We got an 'Ask me anything' with Kentcdodds, one of our keynote speakers. I'm very excited about that. That hasn't been announced yet but I imagine that's going to be really cool. I think people are going to be very into that. Finally of course, the conference itself. ByteConf React is August 31st. It's one day, starts at 9 AM PST. You should join the mailing list and follow us on Twitter. It's just at @ByteConf. You'll see a link to the mailing list there as well and you'll get some more information there but it starts at 9 AM. On Twitch, it's Twitch.tv/ByteConf. That's all you need to do to attend. I would love for people to follow us on Twitter and join the mailing list but if you are allergic to following the people on Twitter or getting emails, you don't have to do any of that. You can just find us that day on Twitch. We have some more things that we're probably going to announce as kind of preconference events in between now and then but those are kind of the two or I guess three, main things. Thank you for having me on. It's been really awesome. I think it's maybe the first time I've talked about the bigger picture stuff with the conference so it's been really cool to get to talk about that. I'm excited for your talk as well. I think it's going to be really neat. ROBERT: Awesome. Thank you for coming on. Like I said, I'm really excited for ByteConf. When I saw this pop up as an idea, I was all over it. I think I actually submitted the CFP before you officially announced that there was CFP. I'm like, "I'm in it. I'm going for it." KRISTIAN: One more thing, I think I didn't mention just kind of organically is that all of the CFP submissions are actually reviewed by people in the community. I'm really proud to say that the talks they have selected, including Robert's were generally, because people were just super interested in them. I think that's going to really show when we air the conference. People are going to be really excited about this stuff. It's going to be super cool. I'm beyond hyped. I'm extremely nervous, extremely hyped and it's going to be great. WIL: I'll also going to say that if you have an Amazon Prime account, you get a free Twitch subscription, so you can go ahead and subscribe to ByteConf on Twitch. KRISTIAN: Yes. That is very true. I should do a better job of plugging that. Oh, one more thing. I guess I should be a good podcast guest and also say like, if you want to follow me on Twitter, my name is Kristian Freeman, it's at @imkmf on Twitter. For the most part, I just tweet about ByteConf stuff and Product Hunt stuff and then get mad about politics sometimes but I should do a better job of plugging my stuff. Again, thank you for having me on this. This has been really, really great and I'm looking forward to seeing you both at the conference. ROBERT: Cool. Thank you Kristian. This is a great conversation. I'm really excited about it. We are the Frontside. We build software that you can stake your future on. If your team needs any help with single page app testing, accessibility especially in single page apps, I'm really, really open to helping anybody. If you or your team need help in that or leveling up, be sure to reach out. We're open to pair. We're open to start a new engagement, anything that kind of helps you and your team to move forward, we're super interested in. As always, you can reach out to us Info@Frontside.io for any feedback on the podcast and thank you Mandy for producing our podcast. Thanks everyone. Have a good day. WIL: Yup. Thanks guys. See you at ByteConf.

In the last episode, we spoke a lot about the "why?" behind microstates. This time we wanted to cover how ideas in Microstates map to different patterns used to build JavaScript applications using frameworks like React, Ember, Vue and Angular. We discussed what you need to know about Microstates if you prefer component state architecture or global store architecture like Redux, as well as how setState in React can be refactored to use Microstates. We closed off with the comments about the trade offs that component heavy frameworks make by overemphasizing the view layer at expense of other aspects of the MVC pattern. This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. Upcoming Conference Talks: Manhattan.js - August 8th (Taras) ReactJS Austin August 6th (Charles) Transcript: CHARLES: Hello everybody and welcome to The Frontside Podcast, Episode #107. My name is Charles Lowell and we are going to be following up with our Episode 106 with the exciting conclusion of 'Microstates, the Podcast.' With me today to wrap this subject up, at least for the near term, obviously we're going to be talking about it a lot in the days and months to come, are David and Taras, also co-developers here at Frontside. Hello guys. TARAS: Hello, hello. DAVID: Hi everyone. CHARLES: Before we get into that, we'll just make a few quick announcements. First off is here at Frontside, we are going to be having some availability at the end of August. If you or anyone on your team is looking to level up in the area of testing, especially testing single page applications, acceptance testing single page applications or if you need React or general JavaScript consulting, please get in touch. We would love to work with you and love to do some of the great things together. Second, we finally released this and this is germane to the topic at hand. We released a major version of microstates this week that's based on a new and simpler architecture. That's really exciting. It doesn't really change the content of the conversation because the API hasn't changed that much. It just means that the library, which was already very small is even smaller. I think we shaved almost 40% of the size off and it's just much simpler and it's much more harmonious with the underlying JavaScript runtime. It's even more just simple JavaScript objects. Unless there's any other news that you want to cover, we'll jump right into it. TARAS: All right. Let's do it then. Let's do microstates. CHARLES: I Love to talk about microstates. This is obviously the second podcast that we're doing on microstates, just because we ended up, I think it was two weeks ago and we'd been speaking for almost an hour and really, we're just laying out the problems that microstates solve -- the problems of state management and why you often run up against complexity when you have a single state management tool that doesn't account for a bunch of different use cases. We got into microstates a little bit but we left it as kind of a cliffhanger. We talked about transactionality and laziness and immutability, ease of composition, simplicity of API, performance, memory footprint, things like this. Those were all the problems and then we're like, "Yeah, microstates. It's awesome." We're going to talk a little bit more about actual microstates proper and what is involved like the adjustments in mindset that you need to make or don't need to make when adopting a tool like microstates. TARAS: Actually, it's been really interesting for me because I just gave a talk at Toronto JS on microstates and it was pretty cool to see. There was a panel at the end. I think the people that are representing just using component state as a way of architect implication. It was managing state and application and it's representing redux. It's really interesting to see, first of all, like how curious people were. Most of questions were directed to the discussion around microstate, simply because it is a new tool but it was also just interesting to see how curious people who really loves redux, he was like, "I really love redux but I really like that fact that you guys have types," so I was like, "Oh, that's cool." Even though you really liked your solution, you actually found something in microstates interesting but at the same time, I think he was kind of missing that what aspects of Microstates overlap with redux. I think one of the things that we can do today is talk about what microstates has and what API does microstates provides and how similar they are to what people already know. I think that's one of the things that I was thinking about this conversation is that there are some things around microstates API like how to use microstates but the architectural concepts that power a microstates, they are already part of the development process and development architectures for most people that building single page applications. I think we can do is try to map these ideas not what people really know and patterns that they use to ideas in microstates, just to show how close we actually are. I think that's a good way for us to go. CHARLES: I guess we can start with, maybe one particular mode of managing state and then, how that maps to uses with microstates. TARAS: Let's start with components state because that was kind of our starting point. One of the original starting points for microstates because so much of our work was in Ember and component state is where most of Ember development happens. I think that's a good place to start. If you're someone who uses components state, if you're using Ember or Vue without something that is redux-ish which is pretty common these days, if you use components state, then one of the features of -- CHARLES: And also, if you're using React, right? Ember, Vue and React pretty much are all the same in this regard. TARAS: Yeah. Well, Ember and Vue are a little bit different in that behalf. Their particular pattern is you have data coming into the component, then you're doing some work with the data so that in Ember and Vue, you might use computed properties to derive state on that and then you might pass that state further down into other components. In React, you would do something a little bit different because there is no memorized computed properties by default, so you, a lot of times, write your computations that you pass into other computers that are written as expressions in JSX. If you're using this pattern, then one of the challenges around using this pattern is the process of lifting state becomes quite complicated. Because if you have like a bunch of state that lives in the component, it's attached to your derivation of state and the properties that you can invoke are attached to a component object. In React world, it's called lifting state. In Ember world, it'd be essential refactoring it into the parent component and passing that state in. in microstates, you're essentially moving this state into the microstate and now, what you have is this object that instead of having the state live on the component, the state now lives on the microstate object and in all frameworks, when you do that in microstates, you can use the getters to get the memorized computed properties behavior on the microstate. If you use your computed properties, computed properties are available for you on the microstate. You can do it the same way. CHARLES: It's kind of like a wrap up of that idea, not a wrap up but a summary of that idea is that the microstate essentially is a substitute for your component properties. If you're working with a component, a component has state associated with it, so you're setting properties on a component and then using the computed properties of that component. Microstates is you have all those benefits where you can set properties directly. You can define methods that like set properties as part of a transaction and you can then have simple JavaScript getters, which are just like computed properties except there's no enumeration of dependent keys. It's just, that's all they do for you. TARAS: Then the next question that arises is like, "Do I make like one huge microstate or a bunch of smaller microstates?" I think the question is really depends on the role of your component and what's nice thing about microstate is in microstates, there's two things that are kind of cool. One is when you represent your status in microstate, it makes it very easy. If you need to lift that state from the component up into the parent, you can lift that type. You can take that type and you can compose that type into the parent's microstate, if you have to and in that case, you would pass this new object that's created from that type. You'd pass the state for that component to the component that would otherwise have its own state. Essentially now, the parent has the state that would otherwise be in a child component and now, the parent is passing a state from the parent to the child. There's two kind of benefits to that. One is it gives a parent a way to control the state of the child and it also gives you a really easy way to serialize and deserialize state. You start off with having ability to serialize state for a particular component and all of his children, now you can represent that serialized state as part of the components state, so you have an easy way to restore the state of the component tree at the parent component. CHARLES: Right and that's possible because a microstate is really just encapsulating a value within a type but the value is just a simple serializable plain old JavaScript object, right? TARAS: Yeah. CHARLES: I'm just not trying to say like this is the feature that microstate provide, so when you're creating a microstate, you just pass it a POJO and it off to the races with that POJO and you can access that POJO at any single point. TARAS: Yeah. This is part of the architecture that I think are helpful for people that are using components. Is there anything else for people who are using components state that we think would be helpful? CHARLES: I think we're just realizing that you get the benefit of the laziness and immutability and the transactionality and all kind of things that we talked about last time but actually, the mental model is very similar. I think the thing to realize is that if that's the way that you used to working with state, the bridge is not too far. It's actually quite a short one. TARAS: Yeah, that's good. CHARLES: It feels very natural. It's not a huge mental shift. It's just more about a very small mental shift, a very small shift in mechanics for a very large payoff. DAVID: For instance, say you're in an Ember application using component state, what I'm used to is you're going to be passing actions around between components. Whenever you're trading that out for microstates, is it more that these actions are just bundle in with the microstate as the transition? TARAS: Yeah. One nice thing with the microstate, because the transitions that you can perform on any data type are intrinsic to the actual data type, the part of the data type, when you instantiate a microstate, you get these transitions for free. When you pass the object into a component, you can now invoke transitions on that object, that are part of that object's type. It's hard to imagine how awesome that really is because the closest pattern that you would see for that would be like, for example, if you're using Ember, you have Ember object or Ember model that you pass into a component and then, you can work a transition and because it goes through the Ember data store, your component is going to update. With microstates, there's no observation of any kind but what you're doing is when you are invoking the transition, transitions going up to the top of the root and gives you the next microstate, which updates your component tree. You get that functionality of 'data-down, actions-up'' built into the entire system of the microstates and all of that is hooked into these transitions that you can invoke on the objects that you pass into your components. CHARLES: Right, maybe David and Taras, you all could provide a concrete example of what that looks like. Because I think it is something that when I was giving talks on microstates at Ember meetups, one of the first things I like to show is you've got all these actions that you're writing either on your router or your controller but they're really actions manipulating the same type of data. It really comes down to like you're pushing and popping things off of arrays, you're toggling Booleans, you're incrementing counters, you're setting this property on this object. These are actions that we're writing and in the kind of microstates world, that's boilerplate. Because the transitions are intrinsic to the data type of the microstate that you're working with, maybe we could provide an example, like what's an action that you describe that you would pass around. TARAS: I think an easier one would be like if you have a model for example, you're composing a model into your application state or you're compose a model into your components state, so you have this model type that gets instantiated and you pass it into component that represents a model. Off that model, you might have open status like is it open or not that gets consumed by the model component, to know whether the model should be visible. Then along with that is model is open state, there is a toggle transition that you can find inside your component that is going to automatically flip the visibility of open. What will happen when you bind that transition to some kind of action handler that you invoke inside your component, when you invoke that action, it will then trigger transition at the top of that microstates and then, it will create the next state, push it through which will cause your model to change the visibility. David, that does answer the question? DAVID: Yeah, it does and that's actually the example that I have come up with whenever asked. A very simple sort of Boolean toggle. CHARLES: In a couple of my earlier demos, I should dust off that talk that I gave to the Austin Ember meetup where I was able to create an input with a dropdown menu with basically, some pretty advanced mouse behavior, all without having any component state or like storing it all in microstates and a lot of pushback was, "Aren't you putting logic in the template?" and the answer is, "Absolutely not." The logic is in the models but what I'm doing is I'm composing the actions that operate on those models inside the templates but at the template level, the action is data. If you're thinking about, we always want to have, we always want to lift state and then push that state down to the application, what this is really saying is that the actions are part of the data. It is implicit to the value that you've got. TARAS: I think that's really powerful because we do think about actions as being something that you can invoke. Like with closures, it's an action that you can invoke but considering that operation, that piece of data that invokes a transition in microstates and is derived from the type of the data, I think it's a cool concept. That's probably, one of the things that is kind of new for microstates but how you use it in your application should feel pretty much the same as you would if you were creating action handlers on your components. CHARLES: Yeah, just like kind of bundled action handlers for free. TARAS: Right. CHARLES: Maybe the way that you wrap up on this one when talking about kind of what microstates has to offer for the Ember developer, the Vue developer is really, I would say someone who's used to working with like MobX. Would be another good example is when I first started using Ember back in 2012, Ember Object solved a whole lot of issues in a really profound fundamental way and I really, really was drawn to that as the best API to be working with state. What kind of became apparent was perhaps not the best implementation. This is not like bag on Ember Object. I think it was actually amazing technology for seven years ago when most of it was written and probably, even came from before like SproutCore. It's almost 10 years old, which is incredible but it's still under heavy use in 2018 and it speaks well of how well it was constructed. I actually don't have much of a problem with Ember Object API. I just think that the way in which it's implemented that API means that there are some problems that require a different paradigm, so very much I kind of see microstates as heavily inspired by those types of APIs but with, I want to say a modern implementation but an implementation that is designed to solve the problems that have come to light over the last five years of developing single page JavaScript application. TARAS: I want to add that, when you describe with Ember Object, it exists in every framework that uses immutability. In MobX, the observation introduces the necessity to wait for a bunch of things to resolve. In Angular zones, I saw a similar purpose to ensure that if there are kind of cascading or streams or all of that stuff gets settled in into kind of restful state before you can start to assert on what's going on and with Vue, their computed properties has something similar. I think because of the complexity of having to track lots of objects and then recomputing things based on the result, that complexity in microstates is simplified by the fact that you describe you transitions. When you write them, you describe exactly what changes, so we don't need to wait for things to settle down. When you invoke a transition, that transition is explicit and based on the path of where that transition is invoked, we know exactly what needs to change, so we need to only perform one operation to compute the next state. There is no other things that we need to wait. There's no other side effects that we need to resolve before we know what the final status. I think that simplicity can be applied to all the frameworks that are using immutable APIs and derived computations from those immutable APIs. CHARLES: Uhm-mm [affirmative]. TARAS: Should we jump into talking about what microstates has to offer for people familiar with redux? CHARLES: Yeah, okay. Let's jump right into it. For folks who are familiar with immutable APIs like Ember, like Vue, like other ecosystem using MobX, a shift to microstates might feel like what you can expect. What about people who are just using often React-land and they're just using like set state. TARAS: Set state is a little bit tricky because it does get complicated over time and then you get this kind of funky things going on after a while where you start seeing things like, "I'm going to set state on this component and then once the change happens, I'm going to change some states some other state," so this kind of cascading state changes. The other part that I find particularly more challenging in set state world than it is in, I think in regular components state, like what you have with Vue and Ember. I feel like the way that the transitions, the state change handlers become part of the component, I find that part particularly kind of fragile. When you start doing refactoring, when you need to lift up state, it's like -- CHARLES: What's an example of this? TARAS: When you start off and your component owned old state and now, you need to have that state being controlled by the parent, we get into a situation where like, "What am I going to do? Am I going to do something with props as they're coming into the component or I should probably just flip that state from the child to the parent?" and now, you're refactoring the internals of a component to lift up that state so you can then combined those operation with the parent's state transitions. But then, you have this kind of added complexity of the fact that you're working with immutable data in that place. You've got these three things going on: you're refactoring your child component, you are moving these things into the parent components, you modify a parent component and now, you're also managing the complexity of forming immutable objects. I think the fact that people make it work is kind of a testament to human resilience. The people are able to solve such challenging problems and this is not a super hard one but when a component is complex enough and when stakes are high enough, these changes can become fragile. This is what I think microstates simplifies is that by taking care of the model, it makes components much simpler and makes it possible for you to just render a lot of components that are functional components without their own state. CHARLES: It actually reminds me and like I said, we're talking about what it's like to use them, not so much the implementation but that's the exact same sentiment that the author expressed in his book that was very helpful in writing microstates, which is Brian Marick. He has this book called 'Lenses for Mere Mortals' and he's talking about the practical use cases of using lenses, which microstates leans very heavily on. What he says right in the introduction is it gives you by abstracting the location and compose it the way in which your data structures are composed, it makes them very refactorable. You can say, "I want to change where the state lives. I want to change the structure of this object and I want to move it somewhere else," and it's sounds weird to say this, because the structure of the object is not coupled to the structure of the object, it means that it's very malleable, so you can move and you can say, "You know what? I don't want this address to be embedded in this person. I want this address to be inside this address book and then my person has an address book entry." Being able to make those refactors make those changes is very easy because your method of accessing the data is abstracted. It's something that applies beyond even user interface as a component state but it's something that this is a problem that's very salient when you are authoring complex UIs and compose components. It's something that you can benefit very greatly from. DAVID: Taras, you were telling us earlier about a friend of yours who is learning React for the first time over the past few months and he's mainly been using set state as you would just starting out in this world. You said that he didn't really get why you might have to go and learn some other way to manage your state and I think Charles said it, whenever your application starts to get more complex and you've got a lot of different moving parts, that's really where microstates will come out and shine for you. TARAS: Yeah. My friend kind of feel bad because he's spent the last three months learning how to work with React and how to use set state, "And now, you're telling me, I need to learn something else? Like I'm going to start from scratch?" I kind of reassured him that especially when you're a beginner, when you're learning, you learn how to address very specific challenges but you don't know how complicated these things can get when your UI gets really intricate. In those intricate scenarios is when you have to leverage your experience and you'll be able to solve these problems but if you're learning, you encounter these challenges head on and your tool does not really, like set state. Although it can be used to build really complex UIs very effectively but the complexity that over time increases as you start to manage the state, increases much more later on than does early in the beginning. Basically, with microstates, even if you have some basic proficiency with set state, when you start working with microstate, the things that you can do with microstates, you will be able to do more sophisticated things easier when you need to use them than you might realize when you start. Because microstates API is so consistent, there are a very few concepts to actually learn and they cover a broader range of use cases so when you actually starts using it, you'll encounter these challenges. What you already know it will just work for you and always just continue to work for you. That was part of kind of fundamental design on what Charles and I have spent, a lot of time putting into place from a point when we started two years ago. CHARLES: I think, and you touched on this and this is kind of what I was speaking to earlier is that when you're working with a simple system, it's simple, it's easy to work with and then the complexity starts to grow, it's never a good thing when you have to reach for a more complex tool to manage the complexity. It's a hallmark of a good system and that it can scale with you from a very simple and easy use cases to actually being able to handle very large and complex use cases but your API doesn't have to change and your usage doesn't have to change. If you have a tool that can actually scale with you from a one liner to a hundred thousand liner, that reflects very well on the design of the tool. I think you see that with things like Ruby, you see it with things like Clojure. You see it with, I would say there's even people writing like Haskell, shell scripts now but not so much with like C++. I think the analogy is very similar with microstates in the sense that when I'm looking for evaluating a language, it's not the only thing I look for but I really am looking like how is this going to work for me as a one liner? How's it going to work for me as a hundred thousand liner? If the answer is pretty consistently, then that's something that is going to get a lot of bang for your buck, so to speak. If I invest the time in learning it, I'm actually able to reap the reward of having a tool that's got my back on a whole bunch of different use cases. We talked about, in React, if I'm using set state and that's kind of a sub case of component state because I would say that in the previous systems we talked about -- Ember, Vue -- they have components state but the component state is a little bit more rich in the sense that you've got computed properties and what have you. But if we look at a system that externalize a state like redux, where you have a global state atom in your application, at least that's the way most of the redux applications that I've encountered behave, what does it look like for you? TARAS: I think it's a little bit challenging when talking about redux is redux conflate a few different things together. I think it's helpful to split those things up, so we can talk about them separately. One of the parts is how the state is delivered to components. The way that redux does is the instance is this created so every time you use connect, you essentially wired together. You can connect through the context or... I'm not sure. I think they have an observation mechanism that's internal to redux as well but they essentially connect components to the store and then they view that to deliver the state. It's kind of worth pointing out that for that like microstates bindings for React actually give you something similar out of the box through context. For those who really like the ergonomics of connect, I think it would be pretty easy to make that available for microstates. Usually, we don't even know why they wouldn't be available. CHARLES: But I do think that connect can be problematic like you can encourage you to not make components that are reusable and have isolated state. It's very easy to hitch yourself to the redux store and now, you don't have components that are shareable. TARAS: I think I would personally prefer to pass microstates instances through props because of the stability that's built into microstates and structural sharing, you can get some free optimizations and allow it to use functional components in more cases like out of the box but some people who are really like redux, they really like connect. Although it might be my personal preference, there's no reason why that wouldn't work if somebody wants me to connect function and make it available for people. CHARLES: Right and I think, there's a happy medium there too, right? TARAS: Yeah, I think -- CHARLES: -- You can connect components and then fan out that state to a set of functional components that are not connected. TARAS: There are some places where microstates and redux are very similar. When you're using in redux, you have this dispatch mechanism, where you essential saying, "I'm going to dispatch this action and your action creator is going to create an action for you with the payload," and then you're going to match that action to a reducer. One of the things that I kind of hear redux people really enjoy about redux is the one with data flow that dispatch the action and then, it reduces this next state and pushes that through and you have this kind of ingress point where everything is going through this one point. I think what's really interesting with microstates is you essentially get that. That's exactly how microstates works, in a way. The only difference is that the API is different. Any action that you invoke is going to go through a single entry point, which is going for you. Because we know the structure of the data, we know how to transition that state for you, so you don't need to reducers, so you're just defining your transitions or use the built-in transitions and then when you invoke them, we know the path. We're essentially forming for you. The path where you invoke the transition, conceptually, it kind of forms the name of the action that you are invoking. The path refers to a place where the state is going to be transitioned, then you have your transition, which is the actual reducer for that part but it's contextualize, so you don't have to think about how you need to transition that state in a great, global redux state. CHARLES: There's no matching. The matching is automatic. TARAS: Yeah, the matching is automatic, so you get that same single ingress and one directional dataflow. You get those mechanics except the APIs that you use, instead of writing the actions yourself. Instead of writing, we just use yourself. You get to use microstate types. I've heard some people who use redux who are like, "I really love the fact that microstates has types," but other people don't like types for whatever reason. Microstates comes with 'from,' which allows you to take a POJO and from that POJO, it creates a microstate and then you can invoke transitions the same way as if you had a type microstate. The only thing you don't get is you don't get to create your own transitions. You have to use the transitions that are provided for the primitive types. CHARLES: I think that there's a couple of benefits that you'll realize for free. There's laziness, reducers by default, or eager. When you dispatch an action, it will run against every reducer in the store. If the reducer matches, you're going to run the computation that's associated with that reducer. Microstates by contrast is lazy. Basically, until you try and read the property that is affected by that reducer, the reducer won't run. There's some ways that you can get around this. When you're using redux and first of all, you can actually have your reducers return objects that have getters on them. You can realize some of that laziness but again, it's work that you have to explicitly put in. I think, didn't you actually say that there is a package of plugins? There are plugins. There's basically a set of libraries that you could bundle together, which would give us an experience similar to using microstates. TARAS: Yeah. If you wanted to combine redux and reselect and immutable JS together, you can get some of the benefits, except this benefits are not integrated that well because they're still separate systems that you are essentially using together. Also, like microstates, it's four times smaller than redux and reselect and immutable JS combined together. If size matters to you and ergonomics matter to you, you actually can get a lot of ease out of using microstates while still maintaining the benefits of having redux. CHARLES: But if those things, those packages, like reselect and immutable JS, are things that are familiar and you naturally gravitate for it, then you'll probably absolutely love microstates. Because honestly, one of the ways that I think about microstates is like, what if you could have immutable JS, if there was no cost for composing the types like list and record. Immutable JS has come a long way since I've last used it or it's evolved since I've last used it but I think there's still only about four or five basic types and actually, making your own new immutable structure, your own custom type with its own custom methods that still get the benefits of structural sharing and laziness that you have on immutable JS is not something that you can do. But you could think one way to think about microstates is an immutable JS where you can make any type that you want. You're not just constrained to the record types and to their list types and set types or map types. TARAS: It's worth pointing out that at the moment, microstates doesn't map perfectly to immutable JS simply because immutable JS has certain optimizations for managing lists that kind of a great value of immutable JS and microstates doesn't have some of those pieces but -- CHARLES: They're definitely on the road map. TARAS: Yeah. Because microstates is abstracted high enough, that we can actually change internals and some people who are using microstates now, they will get benefits of ergonomics and there's already performance benefits from the stability that microstates offers but there will be a time when by upgrading to newer versions, you will not basically, need to change anything in your op but you will get the benefit of improvements to performance that we will introduce over time. Some of those improvements might come from what we will learn from immutable JS. CHARLES: Right. I have a couple of thoughts before we wrap up. What are the ramifications no matter who you are? What kind of development background, some of the benefits that you'll experience with microstates that might be a pain point or something you hadn't thought about where you are currently? A couple of things that I have jotted down is first, and this is what brought it to mind is talking about stability. Something that you see in a bunch of frameworks is having to manually track the keys that are associated with data. If I've got a list or I've got an object, being able to say, if that object changes, then I need to actually have some sort of key object, which effectively amounts to a hashing function to say, "Did it really change?" Because no matter what system you're in, you need to know how you're going to re-render. If the reference to this object changes, then the default thing needs to be, "I need to re-render it," right? You see this in Ember, in Vue. In React, there's this ability to pass a key or ask the question, "Should this component actually update?" and with microstates, that's much less of an issue because basically, it keeps the key tracking instability for you. If you are using a model with microstates, if you think of an object as a graph of nodes, no node in the graph will change unless it absolutely has to, at least that's the goal. We still actually have some work to do when you're running queries against the state of a microstate. We can cover that later but that's most largely the way it is now and definitely, the way it's going to be going forward is you don't have to do any extra work as a programmer to figure out what has actually changed. TARAS: That opens up some interesting opportunities. Imagine if you had a rendering engine that did not expect side effects to be significant and you could just say, "If I know whence they'd changes, I will then re-render that." That will be really interesting exercise, seeing like what would that look like for a performance perspective, if you have a very clear picture of what has actually changed and what part of the DOM as a result, need to be updated without having to do diffing. The [inaudible] you could actually do a diffing at much higher up. Actually, [inaudible] to diffing because you know what's changed but you can push a lot of assumptions higher up on the architecture stack. CHARLES: Right. That's actually one of my favorite thing about microstates and one of the unwritten values is like triple equals used to work everywhere and by and large, it does. It's usually simplifying but when you don't have to manually tell the computer what equivalence looks like, you can just say, "Look, are they the same object? They're not the same. If not, then they're not," and keeping that consistent is huge. TARAS: I think this is a good segue for us to kind of bring this conversation to a close and also, kind of set up potentially a third full-on conversation, which we could talk about actual architecture of microstates and design decisions because for people who just want to use microstates, they don't need to know all these details but for people who are curious, they might actually want to understand what are the considerations that remain when we were designing microstates, so maybe in a next conversation about microstates, it could be about architecture and the pieces in microstates today and then where we are going with microstates and what it could give us long term. CHARLES: Yeah, I like that idea. It is a plannable subject that we've been talking about internally for the past two years, so it makes sense that there would be plenty to speak about on the podcast. There is one other thing that I did want to bring up and that is, I think enabling to have a state solution that is composable because it allows you to think about your state first. Because really, if you do have a functional UI, where your view is a pure function of the model, that your view follows the model and so, if the view follows the model, then really, the thing that you should be thinking about first is the model that's going to be required to drive your view. I shouldn't drive. I should say derive your view because that's the primary artifact of which the view is nothing more than a function. It's a reflection onto a surface. I don't think that we have a state management solution yet, that enables that mode of thought, where I'm thinking about my prime artifact first and working forward rather than thinking about my secondary artifact and trying to kind of wishy-washy way, work backwards and reconstruct the primary artifact. I think that we've talked about all the development ergonomics and I think there's a mechanic of thought there that's enabled by this that I hope to see in more and more applications. TARAS: I think that's a really well put. I think that's something that I've been thinking about as well, as how do you convey this shift that microstates allows in terms of how we're thinking about architects and the application. For some people that value, the model, like they'll find that shift easier but regardless, I think that making that shift has a potential of simplifying your view dramatically and I'm very excited about exploring this further and having more conversations about this. CHARLES: Yeah, that's where we really kind of open up the conversation about state machines, which is also central to the conceit of microstates and using state machines as an incredible design tool but anyway, we can all get into that later. You heard it here folks, Episode 3 is coming out, although probably not for a while. We're going to be mixing up and we will be talking about microstates at least for a while. I understand that next time, we actually teased it but we based on how much material there was on microstates, we ended up packing in a second episode. We teased it last time, we're going to be talking next time about running an online conference with Twitch, so definitely look for that. Thank you, Taras. Thank you, David. TARAS: Thank you. DAVID: Yeah, it's been great. CHARLES: This is a wonderful conversation and as always, we are Frontside. As I mentioned at the top of the show, we have availability coming in August, so if working with us is something that you would like to do, we have a range of services, please get in touch. You can get in touch with us at @TheFrontside on Twitter or Contact@Frontside.io. That's it for now. I guess we should also mention that Taras that you are going to be giving a talk on microstates at ManhattanJS. When is that? TARAS: On August 8th. CHARLES: I will be giving a talk on microstates at ReactJS Austin on Monday, the 6th, so that is right around the corner. I'm excited about both of those talks, especially following so closely on the heels of the TorontoJS meetup talk, which I understand is... Is that posted online yet? TARAS: It's recorded but it should be coming out soon. We'll definitely tweet it out. CHARLES: Okay. All right. Look for that and we will see you next time.

In part I of The Frontside's microstates series, Charles Lowell, Taras Mankovski, and David Keathley talk about state management that's easy and fun and transactionality. This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. Upcoming Conference Talks: Toronto.js - July 30th (Taras) Manhattan.js - August 8th (Taras) ReactJS Austin August 6th (Charles) Transcript: CHARLES: Hello, everybody and welcome to The Frontside Podcast, Episode 106. My name is Charles Lowell, a developer here at Frontside and I'm going to be hosting today's episode and we're going to be talking about microstates with fellow Frontside developers, David Keathley and Taras Mankovski. Welcome you all. DAVID: Hello. TARAS: Hello, hello. CHARLES: I'm really, really excited that we finally get to talk about this but before we jump into this, just wanted to let everybody know that last week, we published our roadmap for Big Test, which is a JavaScript acceptance testing framework that we've been building right here, in-house at Frontside, which can work with both Ember applications and React applications, Vue applications. It's in alpha phase and we're looking for feedback but we're really excited about it, so we're going to leave a link to that in the show notes and please go check it out. All right. Finally, the moment has come that we actually get to talk about this publicly. We've been publishing things about microstates for a while now but we feel that it's ready to share with the world and that's really, really exciting. We should probably like wind the clock back a few years because that's about how long we've been working on this and talk about, kind of the why and the what of what microstates are. It's kind of a weird word. What are we doing here? TARAS: Yes. That's interesting question because we were working for so long and after all this time, what is this specifically we're working on. I can speak personally from my personal motivations because we have conversations over the last two years to why we were doing this and I think for me personally, it's always been that I've been mentoring building complex applications for almost five years now and one of the things that I find consistently is that there are patterns for how to build complex stateful UIs that the required solutions, that are fairly reliably consistent. I can teach people certain patterns and then they can use patterns to build complex applications and those patterns scale really well, the challenge is that, there's not an easy way to express them and it's different for every framework. The way that I would teach somebody how to do it, for example in Ember versus how they would do it in React, even though the pattern itself is the same but the implementation of the pattern is different but it's different in such a way that it's very difficult to see where the consistency is, what is the same about these two patterns. It shows me that there is room for improvement. In the same way that if there is an opportunity that in the future, components will get unified under one umbrella or what component spec. The fact that we do states differently across every implementation, across every opinion, it suggests that what we might be missing is something that would unify across frameworks, how we actually do statement management. CHARLES: Yeah, that makes sense because as much as people try to buck the trend of MVC, it can't keep coming back in the rear view. Hopefully, not like a horror villain but like a caring friend, like a reliable pattern. I remember when I was in writing Java applications back in the day, the most important thing when you were writing a swing application was making sure that you had your models right. If you were modeling either a form or a dialogue or a set of pages, the most important thing was to have those models. Back in the day, we modeled those things with event listeners, very similar to where the way like a Backbone application used to work or if you're familiar with Ember Object, the way that it worked basically, adding observers to some model so that when that model change, you were able to react to that. Your representation was able to be 100% dependent on this model. That was like a Java Swing Application, which was then inherited from this pattern from Smalltalk but we keep on seeing this again and again and again. There is this piece, this state, that if you're going to be representing something, then you need to be able to get at the meat of it. That's ultimately what the model is in the MVC pattern. I remember when React first came out, everybody was like, "Oh, MVC is dead," but now in terms of state management and all of the state management solutions that we see, what that really is, is the model trying to reassert itself. Because it's such an important piece of your application, it's going to rear its head and you cannot escape it. I make it sound like a problem. I guess, it is a problem but there's a proliferation of solutions out there that are attacking the problem in order to represent something you have to know what that something is. DAVID: I think when you only work with the one framework, your perspective might be very much shaded by your experience. If you're really good at building React applications and you have subscribed to POJO is king and you don't need anything beyond that, then you're going to get good at handling POJOs and you might not realize that there is actually limitations to what you're doing. I think the same applies to people who use Ember and they're using Ember Object and they have computer properties and now, in Angular, there is observables and subscribing to streams. What was really interesting with the work that we've been doing with microstates is that and one of things that have notice with other state management solutions is that a lot of them emerge as, "Oh, I had this insight. I spend a couple of days working on it. Here's my proposal, in a way, for managing state." I think that's great because insight and understanding is really important but sometimes, taking the time to design something can be really helpful. I think that one of the things that are really interesting with microstates is that because we've been iterating on it for so long, they've been testing it for so long, we've been able to do something in microstates that's really difficult to do when you have a production application that you're like, "I want something different. Something is not enough but I don't have the time to really make it better," or, "I have an idea. I want to release this thing but I don't have any other collaborators that I can talk to and really flush out these ideas," so what you get is a solution but it doesn't always touch on a lot of the angles that you want to touch on. It's really interesting when you think about -- CHARLES: I'd say, we should talk about those angles, like what are some of those angles? DAVID: Yes. You could think about this from a few different perspectives. I would say, from Ember perspective because that's been my personal starting point. For anyone who was listening. Ember and Vue, for all intents and purposes, in this scenario are very similar because they have the same primitive, which is the computer property. The computer property in Ember, the computer property in Vue is almost identical from the way that you actually consume. In Ember, you would use computer properties and in Vue, it's just computer properties to create derived state from data that's passed into the component and then, you would use is derived state to basically, decorate information that comes with the props, so you could present it in your component. This pattern is very nice. This pattern is being used to build LinkedIn, to build applications at Apple. There are huge implications being built by Ember community using this patterns and I'm sure something similar has happening now with Vue but this pattern doesn't really exist in React in the same way. You might opt in to start using this pattern if you start using [inaudible] but it's not quite the same. Writing computer properties in Ember and in Vue is essentially free because it's so effortless, where in React, if you're using [inaudible] to create cache computer properties and memorize computer properties, you are really opting into doing a particular way of computing this properties and writing selectors that's not trivial. I think that's one thing that is available in Vue and Ember and it's a really effective pattern and you can do it in React but it's actually not possible in Angular, unless you're using, I'm guessing something like ngrx, where you can do similar things to what you would do if with redux in React. Creating derived state that you can consume in your component is one of the things that is possible but it's not possible consistently across all frameworks. That's just one of the things. CHARLES: I agree. I think that was one of the things that drew me to Ember at first coming from Backbone as I did and honestly, from the models in Java was that, in order to compute anything, you had to install a listener and then eagerly make that computation and store it somewhere, where as those frameworks, it made you feel effortless where you can just decorate some state and derive it and the information is there, the computation is there when you need to reach for it but you don't have to do expend any extra effort aside from say, "This state is derived off this other state." I think another case that I came across was immutability. Immutability is a means but the end is to have consistency inside your application. I first really started running up against this when I was working with forms and since then, I've realized that actually, a lot of the pain that I was feeling was because things were not being immutable but this is where the fact that things weren't immutable ended up causing me a lot of pain and headaches and I was having the code into being a lot more complex. Essentially, when I wanted to make something transactional if you're editing a form or something like that, then you need to essentially store off a copy of your current state. If you're editing some object, I want to say, open up a dialogue and I'm going to edit it and then I'm going to commit the changes back to that dialogue. What I'm modeling there is a transaction so I kind of need to shave off a copy or make a Xerox copy of the object and then make the changes inside to that object and then somehow, try and merge those changes upstream and then, get them back into the main lines. Really, like a very Git-like operation. It wasn't just at the object level. When you're doing things like dirty checking, you need to make what is the original value of a field versus what does the input currently have. I might be doing any number of transformations in between the actual physical representation of the field, maybe it's a date but the user interacts with it in as a string, so there has to be this kind of parsing serialize thing sitting in between that value and you need to basically, keep a copy of that field as a mini transaction within your macro transaction because you want to say like, "Is it dirty? is it really the same object?" because if you can just do an object comparison, you have to get into all that hairy like equality checking and it gets really complex really fast but it turns out that a very clean solution to this is if you just never actually make the changes in place and whenever you are making changes, you're generating new information without destroying the old information because when a form is the case where we really come up against this, where we're modeling the same object over time, so a form explicitly models the change of an object. Change is part of what it is and so, the definition of change is being able to compare something in its prior state to be able to compare it to its current state. If you're making that change by destroying the prior state, then you're going to run into a lot of trouble. It just turns out that when you're working with forms and it turns out there are a lot of use cases like this but this was the one where I first really just couldn't even... Without immutability, you want to model your change as always rolling forward and not ever destroying prior states but being able to pick and choose and can always be able to look back to where you were and compare where you are now to where you were. That's where immutability comes when you're modeling change. It's actually much easier to model change when you have explicit states that represents what was and what is. But when you start doing that, then things get a little bit messy. You have to do the compromise. There's a tradeoff, like when I say, "Set this property on this object," that is easy and that's something that I think that we can all understand. We're not idiots. That's why immutable models are very conceptually easy to grasp, to wrap your head around. If I'm modeling myself and I'm saying, "Set hand position two feet up in the air and now I'm raising my hand." That's easy to understand and it turns out that when you're changing a data structure immutably, then you have to, for example model a simple set operation as duplicate and swap. Or if say like you're swapping out one property in an array, you actually model that as a map, where you swap out the one element at that index, rather than just saying, "Just set the thing at that index," or if I'm changing a property of an object, I want to copy over all the fields except for that one which I'm going to change. If you start to do that then, you realize the benefit of always being able to look backwards but you've now introduced overhead and complexity in your code, so you've made a tradeoff. I think that's a lot of people look at saying, "I want this to be immutable," then they actually have to come to the grips with the complexity that's going to introduce. There are libraries like Immutable.js that do make that a lot better but even they have the problems. I can talk about those but one of the cases is like being able to reason about a series of states for your object, rather than just only having one copy of it ever and being stuck with having to deal with it as it is. DAVID: Yeah and I think that really important too is being able to mentally track where you've been because in my experience on other large Ember applications, I've run into so many different bugs that I could really track down to people or rather, places in the code where things are just reaching in and mutating your model, where it wouldn't make sense, so as you're writing your code, you're in a completely different state than you might have expected. CHARLES: Yes and it's easy, right? But essentially, when you have some model, you've basically got a global object. There's a lot of recomputations that needs to happen when those things change. As a result, if you look at the actual code that supports computer properties in Ember and I'm sure other frameworks as well, some of the most hairy and complex. If you look at the chain watchers and the chain nodes and all the stuff that's required to support things like computer properties, it's amazing that it works as well as it does. I've tried to actually open that up and understand that code on a number of occasions and every single time, I had to walk away in defeat. TARAS: This problem exists in React applications and Angular applications because if you think of the challenges of working with set state, I think one of the problems with working with set state in React is that your complexity increases very quickly. It starts off kind of simple when you're using immutability with set state in React. They're kind of very light and then, as you start to add features, your complexity starts to grow very quickly in a way that you really start to run into limitations. You start to confront your personal limitations of your abilities to work with immutability, so you're limited to doing immutable structures that only a few levels deep because your ability to use destructuring to create new immutable object is limited by expressiveness of JavaScript. It's gotten a lot better with the destructuring but if you do it a few levels deep, I think everybody's familiar with what happens. It starts to get really hairy. You kind of loose track of where you are. CHARLES: Yeah. The signal-to-noise ratio increases because after probably, two or three levels, the majority of your code has to do with destructuring and restructuring and very little of the actual change that you want to make. TARAS: And this is kind of interesting because people talk about declarativeness versus imperativeness but at certain point when you have a complex immutable state change, if you're doing with destructuring your code, even though it looks declarative but there is so much processing that you're doing, it's actually kind of losing the benefit of its declarativeness. It's actually starts to look more like imperative code than it does what you would expect a declarative system to look like. I think this touches on the other aspect. It kind of compromise that when you work immutably, when it compromises, you make a serialization. Your ability to represent your state as a POJO becomes restricted by the fact that you have this complex system that's wired together and you have systems like zones in Angular and in Ember Object that are able to keep track of changes in these objects but you don't have a way to restore those objects. You don't have a way to do more sophisticated things that you might want to do, especially in situations where if a service feeding you, what do UIs going to look like. In that situation, it's really helpful to be able to say, "Here's a POJO that I got from the server. I'm going to use this POJO to build this component tree that the user is going to interact with and then, as the user interacts with it, I'm going to then, capture that state, serialize it and put it back in the server." When you're using something like Ember Object or if you're doing this kind of stuff in Angular or even if you're doing this stuff with React but without using something like redux, you essentially end up doing so much wiring to accomplish that. By the time you finished writing your application, you've written a ton of code just to handle this particular use case and if you have to do this again in another application, you just rewritten that kind of code in a new application as well. CHARLES: It reminds me of the concept of a Smalltalk image, like there's no way to really get at the state of a Smalltalk thing. It's almost like you're dealing in docker containers and not actually being able to write that state down into JSON or something like that. I'm trying to casting about for an appropriate analogy. Maybe that's not a good one but what's actually happening cannot be made orthogonal. It can only exist in that one run time that you're currently running. If something wrong manifests itself, reproducing it can be extraordinarily difficult, right? TARAS: Yeah. CHARLES: Imagine if there's some render cycle that's making a bunch of mutations and there's this process that you stood up and it runs in completion, some signal comes in and those effects are like ripple through the system, there's no way at any point to have any other representation of that system than the running system itself. TARAS: There's another element to this, which I find really interesting. When you're thinking about how to architect complex UIs, it's actually helpful to get really clear about what kind of changes are happening. A lot of times when I want to see beginners are writing, especially if you task someone who is a fairly junior at building a single page applications, a lot of times what will happen is because they don't have a clear mental model of what is going on in regards to state. They end up setting a lot of properties. Every operation, every time you have an event handler because they don't have a clear model of what's going on, they end up setting like five or six or seven properties. That kind of signals that they don't have a clear picture but what that also does is a lot of times, they usually comes together with cascading state changes. Usually they're not representing a single operation as a single state change because there might actually be a bunch of things that are happening because what they're doing is they're massaging the system into submission. Not like they're not in control of the state transitions, so they use, essentially time and their dedication to kind of sort it out and make it work and eventually, would that ends up looking like that if it works for most of the cases that they are able to test for or that they able to manually see. But then they AR, not accounting for problems that they're not able to understand right now, they become discovered by users when users start to interact with the system and with the components or with that application and the application is there to get into some funky states. The tools that we have, they don't prevent that from happening. They just -- CHARLES: Right. They don't force you. I think what you're saying is that ideally, you want to model your UI as a set of transactions on your state, that you want transactionality to your state so that I basically am saying, "I'm not going around and setting seven properties in reaction to this one event." I'm saying, "This event triggers this transaction and that transaction clearly bundles up every single operation that needs to happen and the tools don't enforce that." Is that a fair --? TARAS: Yeah and then, the problem is we're working with component trees. You start off with having a set of requirements and over time, the requirements change. As the business unit understands your application better, they give you more direction of how accounting should work and then you find out that there's more interconnect at stake but then what's happening now is that the cost of refactoring those state that spread throughout the components, whether that be with set state, whether that be with the actions in Ember or even in Angular. What you end up doing is you start to change the system but change is not trivial because the actual process of changing where that state lives is not linear. It depends on the complexity of the code that you wrote and it just gets really hairy very quickly. That's where companies end up losing a lot of time. A developer could start off with a requirement, you build something and then a new requirement comes in and instead of it being a simple change, it turns into a week or two weeks refactor because you now understand the state ownership should be different. The state that you have should be in a different place. You have to manually make that change. You have no obstruction to help you express that in an easy way. CHARLES: Another thing, because we are doing a kind of a roundup of all the things that you need to account for when you actually embark on managing your state. Another is actually constraining the amount of computation that happens. If your system is based on listeners and large chain reactions of things where it's like, "This property changes so I need to notify these other 10 dependent properties that change." You can do a lot of unnecessary computation, especially if nobody is going to render. That's kind of the thing that you have to do if you're going to be immutable. You have to eagerly walk those change to see which objects are affected so that you can then invalidate those caches. A system like Ember Object, I don't know exactly how Vue works, it mitigates this somewhat by the fact that the computer properties are lazy but you still have to walk all of those chains. That can actually get out of hand. They're eager, not lazy. Then the other concern that you have, where you normally have to make a trade off around is around composability. One of the things that's really nice about immutable systems is they're very composable. If I've got some object that does one thing, I can then just set that object onto another object just by mutating one of its properties and I've effectively composed them. I can then install listeners onto that thing or I can compute properties off of that property and they can post pretty well. That's something that you get but then of course, you're losing all of the benefits of immutability, so things like Immutable.js don't really compose very well or redux doesn't really compose very well. The concept of taking a redux store and embedding it into another redux store, you just don't see that. I would never distribute and I think ultimately, the litmus test there is would I be able to share it on something like npm. Nobody shares an npm package that's just a redux store that you can dispatch actions to and observe and use it with your other redux stores. When it comes to a system like Immutable.js, that does make transitions a little bit easier over lists and arrays and maps but you still run into the exact same set of problems that you have when you have lists of maps of records and you don't really get any help there, so you have to make this tradeoff between immutability and composability, whereas a system like MobX or Ember object actually quite composable. Before we start talking about microstates, I want to say that you just throw those in there because there is just a lot of concerns out there, a lot of edge cases that actually build up but through the course of a real application, you will encounter them all. You might be making tradeoffs at the beginning that you're not realizing that you're going to need or are going to get you into trouble later on. DAVID: This actually happens in the Angular community as well because there's something really great that's happening with observables in the Angular community. I think everyone's embracing them wholeheartedly and I think that's really been pretty great to see but observable streams of composable, but objects that have on them observable stream providers of some kind, like if you have something that you can subscribe to and that is part of a property in a class, composing multiple classes together and consuming properties from those classes, there is no mechanism for composing that. That kind of composition has to be done manually. Again, you're kind of manually wiring together a bunch of objects and the big challenge is that you are manually subscribing to all those streams and unlike what you have with components. Components have lifecycle hooks. When your components is being torn down, you know you can perform some operations. If you need to remove an event listener, you have a hook where you can do that when you have a class instances like JavaScript class instances that have on them properties that have observables that you subscribe to. There are no tear down hooks for class instances, so there are no obstructions from managing unsubscribing from those streams. You essentially end up having the foundation that you can use to build complex reactive systems and you can subscribe in there really fast but wiring those things together at a bigger scale is simply not there. It's something that you have to create and enforce yourself. CHARLES: Right and I think that's probably a perfect segue into talking about microstates, which is the project that we've been alluding to for the past 30 minutes, that is I think in attempts to solve these problems and make sure that you don't actually have to compromise on those things, so you can reason about things locally but have those things be composed into a greater state. But also have them be immutable so that you can look at past states and reason over a data structure over time as opposed to just in one instance. Also, have an intuitive interface that when you're making these changes, doesn't look like half of your code is unpacking some data structure, flipping some bit in it and then repackaging it back up again. That's the context. Should we start talking about what microstates is and how it addresses those? TARAS: That's a good next step and when I start working in the ReadMe, I end up actually, I think I wrote about 40 pages. One thing that's interesting about microstates is that and this was part of the design of microstates from the work that we've done is that we intentionally wanted to make the number of ideas that you haven't microstates very little, so when you use microstates, the number of concepts that you need to remember in your mind is very few. It is a conceptually a different way of thinking about organizing your state in the same way that shifting from managing DOM elements directly to having an obstruction like component that declaratively applies changes to your DOM tree. In a same way, microstates is kind of an abstraction that allows you to declaratively describe how your state will change and it will manage the transitions for you and allows you to give the state transitions names and it allows you to give your states names as well, so you can actually name things. You don't get a POJO that has a shape but has no name. You actually get to give things names. CHARLES: So, why don't we start? I have a list in my mind. I should probably write it down of the things that we just talked about but I think the things that we talked about are ease of representation, like conceptually easy, transactional, basically serializable and immutable, lazy and composable. Those are like five or six things. But I think there are kind of aligning principle around which we gathers that the state management should feel easy. It should feel fun. One of the things that is awesome about working with components, whether you're using web components or React components or Ember components is when you get it right, you're just snapping these things to feel together and it feels great. It's like I'm just passing properties and render blocks down the tree and the framework is just doing all of the grunt work for me and I'm just operating at a very high level. That's what organizing principle with microstates as it needs to feel easy. Maybe we should start there and just say, how does that easy and fun line up with each one of those kind of unique problems around which we typically have to make tradeoffs? We could start with the interface of making a change. TARAS: I'll go back to the starting point. I remember what got me first interested in microstates is Charles, when you said that, when you have a number, there are certain operations that a number can do. We really don't need to be writing an increment operation for every... Like if you a have a number, you can increment it and decrement it. CHARLES: Honestly, every time I see state management tutorial and they tell you how to increment and decrement a number with it and you write the increment code and you track the thing and you store it back into the store, at this point I'm still annoyed with those tutorials because I'm like, "It's a number. We know we can increment it. Just show me where to plug in the code. I should not have to be writing an increment method." TARAS: Yes. And that's the kind of starting point. There are certain operations that you can perform with the primitive types. If you need to add a number or if you increment a number, we already know how to increment the number. It's part of microstates. But that in itself is nice but that's not that important. I think what's really important is that when you need to put a number into another data structure, let's say you have a nap and you're like, "I need to..." I don't know -- CHARLES: Let's say, like a click tracker that has a number of clicks. TARAS: Right. By itself, you can increment the click tracker but if you need to put a quick tracker into another app, essentially you can compose it in and you don't need to figure out how to wire the actual mechanism of how to make sure that you can update the property, like it's part of another class, for example, you don't need the way you would increment the number. When it's a part of another class versus how you would do it when you're working with it by itself is an approximately the same. The amount of work that you need to do to actually perform that operation is the same. Your complexity doesn't increase as you compose one data structure into another. CHARLES: Right. You can just say, "This is an app. It's got a click tracker and this property is a click tracker and I have to do nothing else. I can register clicks on that thing. It doesn't increase the complexity of application at all." TARAS: There's no wiring. Now, you added some new state, that state is very explicit and it's really clear that it is not impacting other parts of the state. You can operate with this thing. If you change it, it's going to work properly with all the other things that are in the type that you are adding this counter to. Those things are just going to fit well together and it's not going to break if you need to transition one more thing. All of the other transitions will work the same way. I think that kind of consistency is really meaningful, over time especially when you start to increase the amount of state that you manage in your application. CHARLES: Just the ability to work with types and just have kind of those implicit operations and have those things compose, kind of indefinitely. Moving down, we talked about easy and the other thing I would put on that is that the way in which you express those transitions, for example if microstates comes bundled with numbers and Booleans and strings and arrays and objects and kind of the stable of types that you would expect in any JavaScript application but those types are expressed using this way for expressing types, essentially. When you actually do make a transition, it feels very object-oriented, I would say, even though it's not. It feels like you're mutating but you're actually doing a transition. Does that make sense? TARAS: I think for anyone who is familiar with what it's like to write queries for GraphQL, if you're not familiar with it, it's fine. You can get a sense of that from microstates but if you're familiar with the ease of just writing a query and if your backend knows how to retrieve the data, then your queries will just give you the data that you want. That feeling is really powerful and just being able to write the query and just gives you what you want. Microstates is kind of like that. Actually, the inspiration came from experiences with GraphQL, which is that sense of ease is what we wanted to have in microstates and so you get that seems sense of like, "I can just do what I want and it just going to work and with this other thing and it just going to work," and you're just like flying through, like adding states to your application and it's just working for you and working for you and working for you and you don't have to do monkey work like gluing things together. It'll change how you are working before because you have a way of opt working with these things at a higher level. CHARLES: Right. Let's talk about transactionally or should we talk about immutability? How does this make immutability easy and fun? TARAS: I think one thing is that you don't have to write reducers and you don't have to do destructuring by hand. I think you have a way of expressing. Thinking about this, if you have a component tree and let's say you have redux and then a bunch of components like your parent component, your root component has some state using sets date and then components further down the tree also have state. You could actually express that as a microstate. What you would do is, essentially the parent component state would be the root and then the children's component states would be composed into it. The nice things about doing that is that at the root level, you have access to transition state of the children declaratively. You know where the states for those children is on the route type and you can write transitions that are going to declaratively perform multiple operations on the children state and I suppose it'll restructure to what happens with components but if you don't use this, you might have multiple sets date operations. The process of wiring data from the root down to the children is kind of complicated, where here, you have a way to represent that and perform a lot of transitions in the way that is going to be just easy at whatever level you need to operate at. CHARLES: Right. I think, for people familiar with redux, in redux you act globally and then you react locally, if that makes sense, so you dispatch an action to the entire store and every single reducer can see that action. There's ways to manage that but effectively, you have this one atom and then you have the reducers that kind of act on local state, whereas with microstates, you're basically acting locally. You're reacting locally but the effect is global. TARAS: You're participating globally. CHARLES: Yeah, participating globally but you never have to consider the context that is above your own, so you never have to be mindful or cognizant of the context in which you're enclosed because from your perspective, it just doesn't exist. DAVID: Every microstate has a set transition which is the basic transition that you can invoke, essentially in any type, so what's interesting is that it's amazing how powerful -- CHARLES: So, we should break it down really simple. Basically, when you create a microstate, with a type, you say like, "I want to create a number with the value five," and then I can just say, "That returns a microstate," and I can say, "microstate.set 10," and that will return a new microstate who's also a number but the value is 10 and that's available on all microstates. DAVID: Yes. If you have a tree of components and your state is presented by a microstate at the root level, then what you can do is you can invoke the transitions on any part of the microstate and it will just know how to properly create the next microstate for you. The example that Charles you gave of one number so that number can be inside of a class that represents state for a particular component and then that can be a part of another class or represents a state for another component but then, when you invoke a transition on one of the leaf nodes, an equal sets state on one of the leaf nodes or equal set on one leaf nodes, it will respond locally but it will actually reflect the changes globally. At the root level you're going to get a new object that causes the components to update. CHARLES: You know what? I have another concern that actually just popped into my head, which is something that I've certainly struggled with in every single application of notable complexity is stability of value. We should put that on the list. We're almost out of time to talk about this. We spend too much time... Well, not too much time, of the issues of state management, which I think you can't spend enough time talking about but I did want to pile one more on there is when you're making that transition, where you're acting locally but you're participating globally. For things that are unaffected by your action, remain unchanged. That is a super power. It's actually very hard to do with a lot of state management, especially when you're cloning a bunch of stuff, being very judicious about what you don't want to clone. Where this really comes into play is if I want to re-render something. A lot of times you have to jump through a bunch of hoops to tell, did my model really change or did only referentially change? With microstates, when you make that local change, if you're embedded in a very large graph of objects, obviously all of the objects above you are going to be changed but what about things that are off to the side of your siblings. They're outside of that scope of that change. They shouldn't be cloned. They shouldn't be copied over. They should remain the same. If you're doing a re-rendering based on the changes that are happening, that's going to be a key feature because you're not going to have to write, basically any hooks to say, should I have to re-render my component. You can always rely on triple equals. DAVID: This quality is going to describe the structural sharing and some of the other tools that are available but I think that's how it's accomplished. I think one thing that's a little bit different with microstates is that when it comes to structural sharing, it's not that difficult to do if you just do structural sharing and value. Meaning that you can do structural sharing on a value using something like lenses and not a lot of people are familiar with lenses in JavaScript but it's actually only three functions that you can use and it can give structural sharing on complex POJOs. It's pretty easy to use relative to how little people know about it but what that doesn't do is it doesn't allow you to graph of objects that have their own operation that you can invoke and that will perform structural sharing. That part, I know that is not available in any solution, I think at the level of completeness and luxury that microstate provides when you write things. CHARLES: Right because every piece of the tree kind of comes bundled with its own things that you can do with it. DAVID: I don't think you should jam everything into this podcast because there's a lot to talk about it. I think one of the things and we've talked about this a lot, which what we want to do is create an implementation for an idea of what it would look like. What would it be like if we had a composable state primitive that we could use to describe state and share state solutions in the same way that we share components like react-virtualized or whatever your particular frameworks or popular component may be. What it would look like if we had solutions to state problems that we could share and we could -- CHARLES: I think the litmus test of an awesome solution is like you look at this current crop of MVC frameworks and what's so awesome about it is they're sharing. That can happen, right? If I'm writing a React component, I can publish it on npm and other people can use it. If I have an Ember add-on, I can publish it and people can use it. They can consume those components. That's awesome and I think it's the hallmark of a great system. What would it look like if I wrote just the state piece of a file upload and I could publish it npm and then anybody, in any framework, could actually use it with their framework without paying any penalty. What would it look like if there was some transactional data store that could be built and shared and the hooks into any framework were minimal. The possibilities are really exciting around that idea, whether it is realizes microstates or not. But clearly, we feel this is something you should be able to do. DAVID: I'll add one more use kind of use case that I personally find really motivating is that there's a lot of companies that are investing into building single page applications and a lot of times, what you see happening is they're building a very similar application to what they had before because their business hasn't changed. The technology has moved on so solutions have improved. The demands for better user experience have increased but the actual business and how the things that people have to do on day to day within their company hasn't changed. What we're seeing right now is we're seeing the same, like whatever was written before in jQuery or an AngularJS is now being rewritten in React or Angular or whatever you might Choose. But whatever you like to see is it has a situation where the domain specific logic of your business is represented as a data structure that knows how to, potentially, in the future talk to the server and retrieve data from the API because that's likely not going to change. But you can use that like that's been tested and published as an npm module within your enterprise and then you can then consumed that in any framework and it's actually easier to do it this way, than to implement it in a framework-specific version of their state management. That's the part that I find the most exciting. I think one of the things, just to connect to the goal is that, we would like to keep this conversation going. If you're interested and I think this is a kind of a call to our audience that if you're interested in this, we would love to have in our podcast to talk about these things because I think there's a lot of things that microstates really is a beginning of a conversation. It's not meant to be a statement. It's meant to be a proposal that we can just talk about this. CHARLES: I agree and that's one of the reasons we're keeping it very small at this point. The core library of microstates is not setting out to accomplish too much. In the core library, there aren't even any side effects. It's actually impossible to have side effects. It means that it cannot be used for anything except for the model but that's very liberating and it let us focus on what would a system like this actually look like. DAVID: It's really exciting because this has been the biggest metric but we have microstates in the JavaScript weekly and that was great and then it got circled around when people know and it's like 800 stars now, which is not a really big deal. It's funny because somebody commented like, "How can you put something in production that only had 100 stars?" CHARLES: I think it's just important to realize that this really is the beginning of a conversation. There's a really exciting set of things to come. We haven't even talked about how we're going to model side effects, although we're going to use microstates to do it. We haven't even talked about what the various framework integrations will look like and what are the best practices for using this to organize state in your application. We've had some lively discussions internally about what that looks like. There's still a lot of questions but it's going to be a really, really exciting and edifying experience to get to answer them. DAVID: Yeah, it's pretty exciting. I'm excited too. There's been a lot of interest from people in microstates, so it's going to really great. I'm looking forward to meeting people and having conversations about how we can use microstates because I'd love to have someone create a really great solution that I could just take off the shelf and just use and not have to implement them myself. CHARLES: All right. Well, I think we could talk about microstates for at least the next three hours but we have to give everybody an opportunity to, at least like go to the bathroom or something. Microstates will return but if you're interested in learning more about microstates and you happen to be in one of the many places on which we're going to be presenting on microstates in the future, who knows? Maybe you can come in and join the conversation in person. Taras is going to be speaking at Toronto.js on July 30th. He's also going to be presenting at Manhattan.js on August 8th and then, yours truly will be presenting on microstates at React.js Austin on the 6th of August. Come out and see us. We'll drop those in the show notes and it's guaranteed to be a good time and we'll have that conversation. Until then, we are the Frontside. We lead with the why, the how and then the what, if you're interested in working with us and that helps us that we guarantee the lowest total cost of ownership for your application. We're always looking for feedback. If you have news items that you'd like to see at the head of the show or just any feedback or questions, we would be happy to answer them. Thanks today to Mandy Moore for producing our show and next time, we'll be talking with Kristian Freeman about what it's like to run an online conference with Twitch, so I'll be looking forward to that. Bye David. Bye Taras. DAVID: Yeah, thanks for having us. TARAS: Bye. CHARLES: Yup, and bye everybody. See you next time. Next Time: Running An Online-Only, Free Conference on Twitch with Kristian Freeman

Special Guests: Brian Douglas and Bex Warner of GitHub. In this episode, the panelists talk about automating GitHub with Probot. The origins of Probot are discussed, as well as making GitHub apps with the GitHub API, automating workflows with Probot, must-have Probots for every repo, and GitHub's V4 GraphQL API. References: Microstates README Probot github.com/integrations/slack github.com/marketplace/pull-reminders platform.github.community/c/integrations probot.github.io/apps/unfurl-links/ probot.github.io/docs/deployment/ probot.github.io/docs/extensions/#scheduler probot.github.io/community This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. TRANSCRIPT: ROBERT: Hello everyone and welcome to Episode 105 of The Frontside Podcast. I'm Robert DeLuca, the director of open source here at the Frontside and I'll be your episode host. Today, we're going to be discussing automating GitHub with Probot with Brian Douglas and Bex Warner. I'm really excited about this topic. The idea of automating GitHub workflows with bots is amazing. This is something that I've been wishing the GitHub have the platform support for since I even started using GitHub for open source. Just being able to have a bot to take care of certain things like somebody doesn't leave enough of a PR description and they open up a PR, you can have a bot that just responds to it and saying, "Can you provide more information?" It's pretty awesome. With me as co-host today is Charles Lowell, who is also a developer here at the Frontside. Hey, Charles. CHARLES: Hey, Robert. ROBERT: Before we get into the discussion, I like to make a tiny little announcement. We've been building a composable and an immutable state container called Microstates. I'm sure Charles can talk about this more at length, then we will in the next podcast episode -- 106, but I would like to make a small announcement that Taras who is an awesome developer here just wrapped up a month's worth of work, creating a new ReadMe to describe the vision of Microstates and what you can do with them and everything about Microstates. If you're interested in that, I highly recommend checking out the ReadMe. I'll drop a link in the show notes for you that are interested. CHARLES: If I can add, it really is [inaudible] because it isn't like any other state management solution out there. ROBERT: No, absolutely not. I've been building something with it in React Native over the weekend of the 4th of July and it's amazing. But enough about that, you'll hear about that next episode. For this episode, I want to talk about Probot with Brian and Bex. Hi are you two doing? BRIAN: I'm well. BEX: I'm good. Thanks for having us. ROBERT: No, thank you for joining. This is really exciting. Like I said in the intro, I've been really excited about this project. I do a good amount of open source, I would say and this has been really helpful in all of our repos. We have, I think like 78 open source repos on the Frontside. We have Microstates, like we just talked about and Big Test and all of those repos use some combination of Probots that people have built and it's really nice, especially with the new Checks API that has just come out. You can integrate Probot into that, right? BEX: Yes. I, actually am currently working on shifting one of our bots from using the commits Statuses API to the Checks API. ROBERT: That's awesome. Before we go too deep into it because I want to come back to that because that sounds really cool and what the integration of that is like and what changes because I'm not even really that familiar with it. I just know it was released. I kind of want to go from the beginning here. Where did Probot come from and can we get a little bit of a history for everybody that might not know what Probot is? BEX: Sure. Probot originally started out as this simple idea to make GitHub scriptable. The original idea was you have a single file in your repository that would be like a JavaScript file and it would essentially spell out how the bot would act on your repository and the goal was to make GitHub apps accessible to people because if you ever look through our GitHub apps documentation, I think it can be a little tough to get started. There's, honestly, a lot of nonsense that you have to go through in order to get set up. For one thing, the way our GitHub app authentication works is it requires a JSON web token followed by using that JSON web token to request an installation access token and that process would be really tough for new people to get started. ROBERT: Yeah, it sounds like it. BEX: Yeah, so Probot was created to abstract all of that away and handle all of that authentication automatically and simply leave you with the payload that you get from listening on web token events and in authenticated GitHub client to make authenticated API requests while authenticating as an app. ROBERT: Cool, so that's where it started like a flat JavaScript file in the root but today, you use like EMO files and a .GitHub folder. How do that kind of progress? BEX: Originally, their use case was much simpler and it quickly became clear that a single JavaScript file in the GitHub repo was not scriptable enough and not easy enough to understand. The goal was to make like an API that could make that JavaScript file really, really easy to customize for every API of GitHub and it quickly became clear that that was not really a feasible thing to do. as time went on, it turned into this way to build Node JS applications and essentially, what the configuration files you're referring to are the way in which we make it customizable because right now, there's no way to be officially supported GitHub apps channels to pass secrets because it means you're a [inaudible] and the owners of GitHub apps, so that was just a way to kind of stop that problem. ROBERT: Gotcha, okay. BEX: The actual code for GitHub apps still lives in a Node JS module basically and the configuration file just specifies how that module runs. ROBERT: Right, so they're deployed like Heroku instances, if you want, like anywhere you can host a node app. BEX: Yup. Heroku, Now, yeah. ROBERT: Interesting. BRIAN: As a reason to that, some explorations of doing serverless deployments for Probot, I think there's a couple of issues of them. I'm not sure if anybody's shipped anything like the way they at but it's pretty much it's possible to. BEX: Just a week ago, we even released a new version in which we update our core from Node JS to TypeScript and now that things are typed, we have big plans for serverless. ROBERT: Nice. That's awesome, so then you'll be able to deploy to a Lambda and off to [inaudible]. BEX: Exactly. CHARLES: Can I actually interject here, as kind of a person who doesn't really know the relationship between GitHub apps and the GitHub marketplace and what exactly a Probot is before we hear the origin story. I would love to hear a very high level view of how this ecosystem fits together. BRIAN: I think a lot of people are pretty familiar with interacting with the GitHub API and OAuth integrations. I think I've just spent a lot of time at different companies previously to GitHub, just like making calls, either to cURL or through Node JS or more recently, [inaudible]. GitHub apps itself are a way to take all the things that you had to do to make an integration to GitHub much easier. It has a lot of cool things like OAuth, scopings, so you no longer have ask for all your repos ask access whenever someone logs in with GitHub and the connection between like, "Now have gone from OAuth to Now to GitHub apps," there was a lot of, as Bex mentioned earlier, ceremony that happens to getting set up with GitHub apps and integrations that Probot is like this tool to speed up the process of getting to the point where you just want to script some automation or some sort of workflow and it gives you all that bullet play for you. I don't know if that was a good high level for you Charles. CHARLES: Yeah. I've kind of witnessed this second hand with Robert installing a bunch of things here, so let's use an example, like you did some sort of automation on our repos, Robert, where when someone files a ticket, there's this workflow that automatically adds a triage label, so that we know that this thing hasn't even been dealt with, so we really need to address that issue. It doesn't need to be as a high priority. It doesn't need to be closed as a duplicate of something. One of the different aspects that you described there, how do they fit in terms of serving this workflow onto the end user? Or was that a good example, even? BRIAN: One of the cool thing about GitHub apps and what Probot does for you is that normally, if you want to add a label to an issue, either you Charles or Robert, would have to be admin or maintainer on the team for the Frontside and you could add labels. But somebody who opens up an issue, doesn't have that ability to have write access to your content, which is adding a label. What a GitHub app does, it actually takes a spot as if you would have another user on your platform, instead of creating a dummy account or a dummy user. Probot is basically building a bot for you to then, give you the ability to add that issue. That's sort of workflow that normally would have to happen through an actual real human could not happen through a bot without taking up a spot of like, "I guess, I probably shouldn't speak so ignorant about our platform and what we actually pay for nowadays for GitHub," but I know we used to have like a limited amount of seats for organization, like that seat no longer has now taken up and now, it could be just be used a bot can do something that normally us would take. ROBERT: Right. You no longer have to create a user to do these things. BRIAN: Correct. BEX: [inaudible] within GitHub. It's sort of built in a way that apps can take a lot of power in your repositories. CHARLES: So then, what is the relationship between Probot and an app? BEX: Probot is essentially the framework for building an app. You can definitely make the equivalent of any Probot app outside of Probot. It abstracts away all of, basically, the horrible parts and leave the easy part. CHARLES: Now, I think I'm ready to participate in this discussion. ROBERT: That was perfect, though. That's a great intro because I actually didn't have a total grasp or understanding of the relationship between GitHub apps and Probots. That's really good. BEX: Yeah. Additionally, going back a second. You mentioned the marketplace before. One thing to note that is that there actually are several Probot apps on the marketplace right now. The marketplace is essentially the home for any larger, usually third-party companies that have made apps and Probot is essentially supporting some of those. ROBERT: Interesting, so then my question would then be, do you know anybody selling their Probots. Does the marketplace charge? I'm going to assume it does. BEX: Yes. ROBERT: Okay. Is there anybody charging for their Probot? BEX: Yes. There is a quite a few, in-fact, charging for it. Recently, a pretty popular example is the GitHub Slack integration, which is if you open new issues, you can have them appear in your Slack channel. That whole application was recently rewritten by GitHub. It was previously owned by Slack and that was built on top of Probot. CHARLES: And I actually remember, we upgraded to that version. It's actually way, way, way better. BEX: I'm glad you feel that way. CHARLES: I didn't know the story behind there. I was like, "Oh, I just got a lot of... Awesome," you know? Although I don't know what's the costing. BEX: Yeah, I think that integration is actually free, so that wasn't the best example. I think it's for open source projects, at the very least. BRIAN: Brandon, one of the maintainers for the Slack integration and work at GitHub, also did a really cool talk at the SlackDev Conference a couple of weeks ago, so if you're interested what were the behind the scenes. That integration is all open source as well, so if you have request or you have features that you would like to add to the Slack integration, you can pop into the repo that hopefully will show up on the show notes because I'm not sure if it's like GitHub/Slack, but I guess we'll find that out in the show notes later on. BEX: It's Integration/Slack. BRIAN: But for an example of a paid app of a non-third party, we're not talking like Travis or Circle or another one with the big names but rather, a solo dev created. It's Pull Reminders, which is on the marketplace as of today and essentially, this gives you reminders of your pull quest, so you can actually ping inside the comments and tell Pull Reminders to say, "Tell me about the pull request like next week because it's Friday and I don't have time to look at this." ROBERT: That's awesome. I've also seen the one that's kind of related, that is like you can set your out of office at GitHub, which is actually kind of a neat concept. BEX: Was that the one where we are already changing that profile photos to have the overlay or the one where is just auto-replying to messages because I've seen a couple of -- ROBERT: I think, it's just auto-replies. BEX: Okay. CHARLES: So, it can change like your profile pictures and really, not just related to repo and history related activities but everything? BEX: Anything that you can access via the GitHub API, you can almost access via GitHub apps. There's a list of end points that I specifically enable for GitHub apps because there's something such as delete a repository that there's basically, a very few circumstances under which you want to give that permission to an app. Also, to things very specific like your profile or your personal page. About a year ago, there was an official internal audit of all of the API endpoints because there are lots of inconsistencies over what was and what wasn't enabled for GitHub apps, so they went there and kind of decided, what endpoints should be enabled and what endpoints actually get enabled. Now, that list is much longer than it was a year ago. Now, it's much more comprehensive. ROBERT: That's awesome and is this for the Rest API and the GraphQL API? BEX: Yes. Probot does support both. The Rest API is the one that specifically had all of these endpoints audited. The GraphQL, since it's a bit newer, we sort of built those and more. ROBERT: Cool. I really like working with the GraphQL API with GitHub. It makes it easier than trying to do a bunch of Rest calls. BRIAN: Yeah, there's a community form, it's like a discourse form that the API team actually manages and sort of pipes in there. Again, going back to like, if there's not something in the Slack integration that you would like to have, the form, that community is actually in there, if there's something not in the GraphQL API, that you would like to see. No promises on shipping it within an x amount of time but if enough people are requesting it obviously, there's going to be some resources [inaudible] at. ROBERT: What do you mean? We're doing open source. It has to be done yesterday. BRIAN: Yeah, exactly. And that form is at Platform.GitHub.Community, just a URL to get there. ROBERT: Awesome, that will be helpful to look through and get some recommendations in there. One of my favorite things I was going to say about the new integration for Slack and GitHub is the fact that I can highlight line numbers, paste that linked in and then it just expands it and the chat in Slack. That is so nice and I use it all the time. BEX: Yeah, I love that they built that feature. Actually, the original feature that was built on GitHub to allow those line expansions in the first place, like on GitHub itself, was actually built last summer by some folks who were also a part of my intern class at GitHub last year. ROBERT: Hey, intern power. That's awesome. BEX: Yeah. ROBERT: Everyone there is doing amazing work. I'm also following along with somebody that is also an intern and it's building a weekly digest program. BEX: Oh, yeah. That's actually a Google Summer of Code student. ROBERT: Oh, interesting. BEX: So, being sponsored through Google Summer of Code by Probot as an open source support. ROBERT: Is there anything more to unpack there? That sounds really interesting. BEX: Essentially, we submitted an application for Google Summer of Code because we thought it'd be a cool way to get more people, more students, a mentorship opportunity for the maintainers, basically and we were honestly overwhelmed. We got like almost 100 applications and it ended up being a huge of a deal but we're -- ROBERT: That's a great problem. BEX: Yeah, definitely a good problem but we were really happy. We, initially wanted to accept more students but Google limited us to only two students, so we have two Google Summer of Code students working on projects and one team of women from Rails Girls Summer of Code working on Probot. ROBERT: That would be awesome. What do they working on? BEX: I'm not sure yet. They actually just started a couple of days ago but the other Google Summer of Code student is working on a background checks API to eventually do sentiment analysis of comment history of someone new to your repository. ROBERT: That's interesting. That sounds like there will be some machine learning in there. I might just throwing out buzzwords? BEX: Most likely, I think they're just using some sentiment analysis API, like the perspective API. I don't think they're actually doing that themselves. ROBERT: Okay. CHARLES: Actually, I have a couple questions. Back on the subject of Probot. How does this square with the classic mode of integration because there was a lot out there? I think the first one that I remember that stuck in my mind was like Travis and I don't know if there had to be like a special relationship between the Travis developers and the GitHub developers, that's like, they was able to make that integration happen so many years ago. I don't know how that happened. I just remember it popped up and I was like, "Woah. This is incredible," and we see kind of the integrations gets more and more rich. For someone who's got, like you mentioned a couple of the big names, is the idea that eventually those would be able to be completely supported is GitHub apps or is it they're always going to be kind of a separate track for kind of the really deep integrations? BRIAN: I wasn't around when Travis first integrated with Lyft GitHub and I think that's a really cool integration and I know they have a very nice sized team that's able to do that. I think if we zoom back out like Probot, the way to get started with Probot is that we have the CLI command, which is to create Probot app. I believe it was intentionally copied off of create React app and the cool thing about create React app and create Probot app is that they abstract all the ceremony and boilerplate to get started really quickly. It was like, what developers or smaller teams can get started with integrating with GitHub apps. I highly doubt that Travis is going to rewrite their entire application with something like create Probot app but they're definitely going to be moving towards the new API calls, which would have been like GitHub apps. Part of the Checks API that we had launched at the end of May, Travis had blog post on how their integration with the Checks API works. They're making, though they have a lot of what Legacy endpoints and a lot of Legacy integrations in the way they integrate with GitHub, they are actively moving towards a GitHub app. I don't know if I could actually comment on their status of where they are today, to be honest but actively, we want all new apps and new integrations to follow the model of being a GitHub app, so that way, out of the box, you have access to all the newer features. You have all the access to all the newer GraphQL endpoints, if you want to use GraphQL and that way, we can serve one market, as opposed to everybody who had a GitHub integration from five or six years ago, that was all piecemeal together and sort of duct tape, like we run move away from duct tape everything together. CHARLES: I see. BEX: I definitely agree that I don't think Travis is going to switch to using Probot anytime soon and I don't think most of the large companies will be doing that but I do think, there will be shift towards GitHub apps in general. For those companies that don't already have the buildings of the GitHub app started, I think that Probot could be, in time to free some of them. BRIAN: In addition to that too, Travis and Circle and all the CI integrations, they're doing a really good job. I think the cool thing about GitHub apps is what you take away all that ceremony of getting your checks to work, now we can start opening up the door of like what's the next sort of CICD thing like? There's another term or another, I guess category of applications that can now be built to improve GitHub. CHARLES: The most amazing thing about having a great platform is the apps that you don't foresee, like it just come completely out of left field and you're like, "Woah. I can't believe that's actually a possibility now." When you have started to see some of those, some Probot or GitHub apps, you're like, "Man, I didn't see that coming. That's awesome." BEX: A hundred percent. I think it's the most exciting part of Probot because I think GitHub as a platform, we all know GitHub is the largest developer platform in the world and I think the idea that developers can build on top of this platform is the most exciting idea right now. I have honestly already seen apps that really excites me. The other day, I saw this app that was definitely not near completion but it was essentially updating and issue a comment box over and over and taking response through like checking a box and then listening on that common edit, in order to specify your coffee order. ROBERT: Woah. BEX: I was like, "Do you want an ice coffee or regular? Do you want milk or sugar and cream?" and it was going one at a time. It didn't actually order you your coffee at the end but it was super exciting to watch that. You're just editing the comment. I had never seen that before. ROBERT: That's pretty slick and that's taking the API pretty far. I'm sure there were some parsing in there and each Webhook response are like, "Was this box edited or not." That interesting. CHARLES: Yeah. Actually, now that we're having this discussion is kind of like changing my mind a little bit. Robert and I were actually talking yesterday about trying to standardize on our release management and our plan was basically to have some software that was going to run inside of our CI provider and have kind of a shared library, just a little ntm package that was shared by all of our repos but I'm thinking now, man, we should really explore doing this as a GitHub app. ROBERT: Yes, please. I've had three ideas that I really want to build out as a Probot. I'm just going to list them off and then we can build them all together and take equity and you know. I'm kidding. But the two that really excite me, that I kind of want to do is one concept that we work on this open source project for our clients and if somebody from the outside that doesn't have commit bits to be able to push to master, it would be really cool if we had a Probot that after it had an approved on the PR, from the maintainer, that the person that open the PR could then tell a Probot say, "This is approved by somebody that manages this project. Can we merge?" and then the Probot would then actually merge. I don't know if that's possible. That's something that I definitely wanted to explore. Then the other one, which is less cool, would just be like if we have a couple branches on some of our projects that we want to continue and we're not ready to put it back into master but we want to continuously run the test suite against it, so the idea there would be to have a Probot that would watch for changes on master and rebase as needed and continue to run the test suite and see where you're at. Those are the two things that I'm really excited about to do with Probot but I just want to automate everything with GitHub now. CHARLES: Right. BEX: Yeah, definitely, that first idea was actually pretty viable. I'm curious to know like how you actually get those commit links -- is that what you called it? ROBERT: Commit bits are more like commit permissions, I guess. BEX: Oh, I see. ROBERT: An outside contributor. CHARLES: Yeah, we want to push responsibility to the person who is the maintainer who can approve it but actually, the way we do it at Frontside is the person who actually is making the change is responsible for merging it. Once you get approval, you still have to hit the go button and that's just going to make sure that you're taking responsibility for saying it's done but that doesn't work for open source because people coming off the internet are going to have the right to push but we would like to give it to them, maybe via an app, if there is a maintainer who's approved it. BEX: Yeah. That's definitely something you can do. I've seen quite a few apps that, essentially add outside collaborators to the repo. Are you familiar with the... I forgot what it is called, like the all contributor section, where you cite everyone in your repo and everything and who's worked on it. There was a GitHub app that would add someone automatically after they merge their first change. CHARLES: That's awesome. ROBERT: I may have seen that on React State Museum but I'm not sure. It's a repo that we've contributed to and it has all the contributors at the bottom. It seemingly just kind of popped up there. BRIAN: There's an app that, I would like to mention too that I'm pretty excited about, that it sounds trivial too and it's almost similar... Not similar but it's sort of related to what you were talking about, Rob, with your first app, which is the WIP bot, which is the work-in-progress bot. This is a pattern of whenever I open a PR and I might not ready for a merge but I want to share my code so I can get feedback earlier on, I'll type in WIP so that append to my title of my PR. What this engineer did was every time you do WIP, it's going to go into the GitHub API and actually block the PR for merging, which is a feature available to GitHub. It's nested in your settings but the cool thing about this it actually blocks the PR for merging, so you don't have to worry about getting your, sort of like show and tell code merging the master without being ready. ROBERT: That's one of the first bots that I installed on all of our repos and then you can correct me if I'm wrong, it didn't always have the ability to block the PR from being merged but with the new Checks API, is that something that was introduced? BEX: Not exactly. The way that blocking of merging works is if you set it as the required status, so you can install any sort of CI on your account and have it not being required and ignore it whenever you feel like it, so it's really up to you to make it required. Otherwise, it just isn't checked and that's true for anyone who uses the Statuses or the Checks API. ROBERT: Okay, so that's a Statuses API. Okay, sorry. BEX: Yes. ROBERT: Also, the cool thing about that that I noticed when that was rolled out was I was now able to pick and choose and use workflows on Circle CI and each workflow is broken out as a different status check. I am now required like linting and the build and the test have to pass for these browsers before it can merge, which is really cool to be able to pick and choose. BEX: Yeah. It's awesome. I know personally on some of my repos, I have a few checks that I just don't require because I know I have to make them pass. ROBERT: Yeah. Speaking specifically about the work-in-progress bot, do you know how that works? It's open source, so I am sure I can go look. I think we want to go make a PR. We had some back and forth about this, Charles. CHARLES: I actually just [inaudible] we disagree. ROBERT: Yes. Charles opened a PR and one of his first commits in the PR had work in progress and the title had work in progress and we have this this Probot on our website and it was a blog post. You know, you make a couple more commits and you're further down, you move the work in progress in the title but the PR were still blocked because the first commit on a PR have work in progress in it. I think if it's the most recent commit or if it's in your PR title with work in progress, it should block but otherwise, it should not and Charles feels differently. CHARLES: I have about six commits and the very first one have WIP in the title or in the commit message and it blocked the whole thing but I kind of felt like it actually made me go back and I had to squash it down to two commits because I actually feel that your commit history should tell the story of the development, not like it should an absolute one-to-one journal of what happens but what you are intending. I actually felt that it could help me out because there's six commits that we're kind of all over the place and just kind of slapdash together have made me kind of go back, rethink it and tell a coherent story. I think it did me a service but it was not obvious. I definitely agree with that but I was like, "Why? Why were you still blocking?" ROBERT: Do I really [inaudible] admin privileges? BEX: I would say, I am friends with the creator of the web app. His name is Gregory Mantis and he is actually got a huge work in progress PR shifting work in progress over to using the Checks API and one of the features that he's using with the Checks API is essentially this mark as now work in progress button that will add the special line, like feel free to merge or something like that into your original PR description at the bottom. If that is there, the work in progress app will no longer be blocking. It's essentially like a hard override and honestly, that's the power at the Checks API versus the Statuses API. That's really exciting. ROBERT: Because I have seen the work in progress bot to get into a weird state, where I did remove the work in progress from the title but it didn't quite update and I'm still blocked. It's okay for me because I have admin privileges but other people on the team maybe not and they might be blocked from something that's actually work in progress. It's a lot like that hard override will be probably pretty helpful. BEX: Yeah, definitely. I think sometimes, there's some confusion with that just because of the way what perks work on GitHub and the way our pages are rendered, that you may need to refresh the page before you actually see it take effect. ROBERT: Right, yeah. Overall though, I love that bot. I go weekly, probably to the Probot apps listing and just go shopping. BEX: Wow. I'm actually the person who approves all the Probot apps to the listings so that's pretty motivating there. ROBERT: It's really nice. I am not even joking when I say shopping, I go through and I open up a bunch of tabs, I read through them, "Oh, this could be useful," that kind of thing. BEX: The first app you mentioned, which was like the one that requests more info is actually one that I built, so that was kind of funny. I guess you got that from the Probot apps too. ROBERT: Yup. That one, we definitely use on a couple of our organizations and repos. It has yelled at me a couple of times because of a blank PR. BEX: It yells at me all the time. I think I get yelled at more than people who are actually doing it wrong. ROBERT: I'm a little embarrassed like, "I should do better. I need to set an example." BEX: Definitely. ROBERT: Cool. I'm curious what both of your favorite Probot app is. This ought to be interesting. BRIAN: The app that I'm really impressed with so far, that I actually only use on a junk project at the moment, is the weekly digest one and it's mainly because I built something for this in my previous role at the company but then we shift it, which is basically go through every single repo. I worked at a company called Netlify previously and we had way too many repos to maintain... Oh, sorry, to keep track of and I was moving further and further away from the backend at the time so I was unable to keep up to date with all that was changing. I built a Lambda to watch Webhooks and then give me a digest of what was shipped like issues and PRs closed. It was way over-engineered and I never actually shipped that to actually make it work. But then the weekly digesting came out maybe a couple of weeks ago and it blew me away because I was like, "This is exactly what I needed," and I was trying to make it overly complicated through like a Lambda and like a bunch of Webhooks and this person, with only a few weeks, has the scaffolding of what I needed. That's the one thing I'm pretty excited about. It was already mentioned earlier too, as well. BEX: I guess, I would say one of my favorite ones is the unfurl a link app. I think that one it so simple but so nice. I don't know. I think having that unfurl link preview is just beautiful. Essentially what it does is it listens on issue comment creation or pull request comment creation or issues your pull request or whatever and read through the text or whatever was that issue or pull request and looks for links and then, essentially unfurls them so you can get a really nice preview of what you're going to. I think that's really beautiful and just so simple. ROBERT: Yeah. I love that one too. I have that added to all of our repos. BEX: It's so much nicer. Why would you not unfurl your links when you could unfurl your links? ROBERT: Exactly. CHARLES: I actually have a question. I think it's been touched on, probably at least twice throughout the conversation. I want to actually create a Probot, how do I actually go about deploying it? What does that look like? What does it look like to deploy and maintain it? BEX: We have a page on our docs about deployment and essentially the TL;DR is you can deploy it on any normal cloud hosting service that you wanted to deploy it. There are a few things you need to specify. For example, GitHub gives you a private key that you need to create your JWT and that private key means to be passed into your hosting service however you do that and then, there's a few bits of information that need to be pass in. We have pretty intense docs about it. Honestly, I'm not a deployment person. I usually try to let other people do that and I have never had a problem going through our docs and just getting it working immediately. BRIAN: It's also mentioned that there are examples like Heroku and Now and a couple of other ones. If you have a service that you already like, it's possible it's already in the docs, like steps to how to get that deployed. BEX: Yup and any other services are more than welcome to be added to the docs. Pull request are welcome. ROBERT: Sweet. It sounds like we need to set up a hack date to create a Probot, Charles. CHARLES: Seriously, my mind is brewing. ROBERT: I guess it's not directly related to GraphQL but there's something that I've always wanted to build. For prior history to everybody [inaudible], then the podcast, Brian and I used to work at a company called IZEA and one of the things that we built and I worked on a lot was we would create a collect metrics on people's social accounts that they're connected and do that and graph it over time. This idea came from when I was building up that feature all the way back in 2013, I want to graph the change in GitHub stars. Is there an API available for me to see like weekly GitHub stars or is that something that I still have to manually store and track? BEX: There's definitely an API endpoint to get the amount of stars and I don't see why you couldn't just do that on weekly basis and compare but I don't think there's any track that change API. ROBERT: Gotcha, like a history of it. I could do this by just stealing and looking at what the weekly digest Probot is doing because there is a change in stars section in there. I was just curious if there was now an API that was available. BRIAN: Yeah, that's more unlikely. I'm going to say no without looking at all the reference documentation. I think as far as that database, it's something you'd probably have to collect on your own but it's also a good candidate for a GitHub app, where you build a service that you can actually track stars once you've installed it and then if you want to monetize it, you can actually pay for private repo or whatever stuff like that, if you wanted to. But it sounds like a great opportunity to see this in the GitHub/Probot listings. BEX: I actually just look this app really quick in our docs because I was curious but apparently, you can receive the star creation timestamps. That could be doable through timestamp usage. ROBERT: Oh, and then I just kind of loop through back and build your graph in there. BEX: Yeah. ROBERT: Interesting. All right. Well, [inaudible] I was going to do today. BEX: Yeah. But I think it's exciting to bot the weekly digest and then what you could extract from that into stargazing is that Probot scheduler, which is essentially this all Probot extension we made that triggers a Webhook on a scheduled time period because right now, the way GitHub apps works are so centered around Webhooks. It can be difficult to find a way to trigger an action on something outside of a Webhook, like on a schedule basis. ROBERT: Yeah, that would be really helpful. I can definitely see how that would be a problem, if it's very, very central to reacting to Webhooks and events that happen on the system. BEX: Exactly. ROBERT: You're just hoping that somebody comes through and creates an event at a specific time. CHARLES: Can I ask you a question about, it's definitely on topic of extending GitHub but currently, just a question about, where the line is between what you can and cannot extend? You mentioned, for example in the rewrite of the WIP bot, being able to throw out a big button that says override this merge. Are there any plans to be able to actually extend the UI in novel ways? Everything there right now is happening with API calls, with I assume, UI elements that are related but the UI elements are static. If someone wants to put a novel piece of the UI, that button is going to require an extension of the GitHub UI by GitHub itself. Are there any plans to be able to, I know it's a dangerous waters, perhaps at a limited fashion at first but maybe more so, add different interactions and the actual application. BEX: I think this is actually the most exciting future of GitHub as a platform. In the past, GitHub APIs have only specifically supporting things that you can do through the command line or you can do through GitHub's UI itself. The Checks API introduced the very first non-integration specific UI element essentially and the merge button that I was referring to in WIP is exactly that. It's essentially this button that you can change the text of it to be whatever you want and you can listen on that action and then you can do as an integration or an app, anything that you want based on that. I think that's the most exciting direction for GitHub. Because if you look at Slack, Slack is a platform that has sort of really impressive integrations in that response. Your apps on Slack can really do all of these things, use custom UI elements, so I think the most exciting features for GitHub as a platform is all of this customization and giving the power to the apps. ROBERT: Yeah, that sounds an awesome way to be able to extend GitHub without having to try and throw the feature on to GitHub developers. BEX: Exactly. I feel that a lot of the struggle right now is that there aren't these nice ways of communicating via apps because I feel lot of the apps and bots end up just commenting on issues and pull requests and taking up a ton of screen real estate as a result and I just think that that's not the way that bot should ideally interact with the GitHub platform. They should have their own space to exist and that's the feature I'm most excited for. CHARLES: Yeah. I can think of having like progress bars for CI checks and your various appointments. It's too exciting. I'm glad. That's definitely the response I was hoping to hear. BEX: Yeah. We're excited for it too. ROBERT: Basically, you all have a massive community of a bunch of developers that would want to do this and are willing to get their hands dirty on it. Enabling that community is probably the root of all Probot is about. That's super awesome. BEX: Yup. CHARLES: That's a good place to end, because gosh, it's going to be so exciting to have the millions of developers on the planet, just like surgeon to the APIs that you're developing. BRIAN: One thing to add to that too, about the whole million developers, there's a number that's been thrown out from Stack Overflow and also, some other people who are saying like there's 50 million developers, there's 24 million developers. As far as GitHub, our public user number is 28 million, the cool thing about Probot and GitHub apps is that there's a good chance that all those people that are using GitHub today are not actually developers. They're like PMs or designers and what's really cool about this, like having interactions with that kind of platform in this way is that you can now enable all the non-developers to be able to interact with your GitHub repos and start bringing more designers and PMs onto to the GitHub platform to interact with the developers. ROBERT: That is an interesting point. That is awesome and something that I'm always looking for is a different ways to collaborate with non-developers on my team because... I don't know, developers tend to think everything is always centered around code but it's not. The shifting at work that are awesome, needs a lot of collaboration from non-devs and non-dev skills. That would be really interesting to see. I'm excited for that to play out. BRIAN: Yeah. There's a blog post that was published a month ago, I think about where the design team, design system teams rather, built the integration to Figma to update their icons effectively. I just posted that in the chat to look into but they also built this as a Probot app as well. ROBERT: That is awesome. BEX: Yeah, that one is super exciting. You would have the app comment, the diff between what the old icon versus what the new icon look like and it's just such a beautiful design change to be able to see that shift. ROBERT: Man, I'm happy that this is happening. The future seems super bright. Where can we direct people to get resources to contribute, to get involved and start really going at this? BEX: Basically, Probot.GitHub.io has all the Probot stuff, /app has all the listings for apps you can install today, /docs is where the docs are, if you want to get started and hopefully from there, we link up to the necessary things that you need to do. BRIAN: Also, what I mentioned too via Probot Slack channel, there's a Slack channel as well and they do a weekly call. I think, it's weekly or bi-weekly call to actually chat with the Probot community. If you have questions, you can actually bring your questions to the team. BEX: Yeah, we call it 'Office Hours' and it's once a week and it's under our community page, where we also have a link to our Slack. We have a link to another podcast we run and basically, how to get involved in the Probot community. ROBERT: Those are really helpful resources. I do remember seeing that Office Hours. It's on Thursdays, right? BEX: Yes. ROBERT: I was going to drop in for one and then, I actually forgot. Actually, it might be going on as we talk right now in this podcast. BEX: It starts in half an hour, I think. ROBERT: That's awesome. Cool. Well, thank you Brian and Bex for having a conversation about Probot. This is really awesome. Is there anything that you would like to plug for yourselves? How people can get in contact with you? BRIAN: Yeah, I am BdougieYO on Twitter. Everything you need to know about me is there and I am happy to say hello. I'm also helping with the GitHub developer program, which is sort of getting a soon-to-be announced rebranding. If you go to Develop.GitHub.com/Program and you want to have more conversation about the API and GitHub apps on the GitHub side, you can go there to sign up. BEX: And I am HiImBexo on Twitter. You can ping me in any Probot stuff. I'd be happy to look at any Probot code. I've been looking at it for a while now so I'm happy to do that. ROBERT: That's awesome. Thank you all for having a conversation with us. This was really fun. I'm so excited about everything you can do with Probot. This is a really fun project. I'm happy that this is happening and I will make a Probot in the future. CHARLES: I'm looking forward too. Robert has been excited for quite some time and he definitely talks a lot about it and now, I have some insight as to what -- ROBERT: It's happening, I'm telling you. Well. Thank you for being here and we are the Frontside. We build UI that you can stake your future on. We are specializing in JavaScript. We can build anything that you want throw at us. We do functional programming, React testing, Vue, anything in JavaScript, we specialize in. As always if you want to suggest anything for us to have on the podcast or talk about, you can reach out to us at Contact@Frontside.io and like I teased earlier in the podcast, next episode is going to be all about Microstates, the immutable and functional state container, composable model system that we've been building, it's controls as a brainchild for the past two years. That is next episode and I'm really excited about that. It's a really fun API and expressive to build models with. Thank you, Mandy for producing our podcast and we'll see you next episode.

In this episode, Chris Martin of Type Classes and Joe LaSala of The Frontside talk about blockchain development. Do you have opinions on this show? Want to hear about a specific topic in the future? Reach out to us at contact@frontside.io or on Twitter at @thefrontside. This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. TRANSCRIPT: JOE: Hey, everybody. Welcome to Episode 104 of The Frontside Podcast. I'm Joe LaSala. I'm a developer here at the Frontside and I'm going to be hosting today's episode. Today we're going to be associating blockchains and other cryptographically secure technologies and everything that has to do with the web and the future of the web. We have with us Chris Martin and he's currently with Type Classes. What do you do over there, Chris? CHRIS: Our goal is to teach functional programming with Types, specifically with Haskell and a little bit Nix. We do subscription video service. JOE: There seems to be, I guess a bit of an overlap between people who are into functional programming and people who are involved in this new space that has opened up, this new web, I guess and that's something that I want to talk about based on a tweet that I saw you made recently. You mentioned that there's a big section of the Haskell community that is being drawn into whatever the hot ICO is at that moment there, something along those lines. CHRIS: Some of it are bitcoin people or something else but there's definitely a weird overlap that I can't fully explain. JOE: It seems like strange bedfellows, right? CHRIS: Well, there's a couple of things that make sense, which I think the distributed systems in cryptography are kind of these notoriously hard problems. I think when somebody wants to convince their boss that they really need to use Haskell for this problem, I think they can make a persuasive argument in this case. JOE: That's interesting. There's actually, a lot of technology around blockchains around bitcoins, specifically being written in Haskell. I didn't know they were technologically overlapped like that. I guess I just thought they were two very kind of passionate communities but you're saying that a lot of the bitcoin startups that you might see coming out in any given week are actually being written with an eye towards functional programming. Is that accurate? CHRIS: I don't know about bitcoin along this bit but I think some of the people who are working for banks and trying to develop their own sort of novel internal blockchains and stuff, I think those are the people who see this. Although in the case of banks, we don't necessarily see what's coming out of them, so we can't verify whether they're actually shipping things or not. JOE: Yeah. That means there's a lot to touch on there. I would agree with you on your initial sentiment, also just to extend to say that I think personally that both communities are really evangelical. Functional programmers, people who are into functional programming, for me it hasn't clicked yet and I know that it will come into my heart. I've asked functional programming to kind of where things are starting to fall into line where I'm certain to see the world in that way but for people who have seen the light fully, I'm sure believers once monads and functors kind of enter the conversation. They don't leave. It's similar like when bitcoin first started and everybody's running about the gold standard. Really, it's just nothing. It was hard to find resources on it that did the most of the amount of screaming. CHRIS: Yeah, you're absolutely right, that culturally, they're going to attract the same group of people or the people who are willing to adopt something that's not fully fleshed out yet, people who want to take what they believe and sit in this community and try and spread it to the rest of the world. I think it's the same kind of people. JOE: The early adoption, I think is something I can consider too. I guess it's a very risk-oriented group. CHRIS: Yeah, kind of. I mean, Haskell is pretty old, I guess but -- JOE: That's fair, yeah. CHRIS: -- Some of the changes that really make it, it great and usable lately are pretty [inaudible]. JOE: That's interesting. You mentioned this idea -- we kind of skipped over a little bit but thanks, having their own blockchains and that's something that I think that maybe people not actively following this space, which is I will say, a very hard space to keep up for those of us who are actively following it. But those who may just know blockchain through the name of an iced tea company changing or some sensational news article or what have you or just through bitcoin even, but I know that it's not the blockchain. It's not a singular blockchain. It's very easy to implement the fundamental structure. It's a linked list, essentially, with the kind of a cryptographic thing that keeps from breaking that link. Those links are inserting new history, I guess the further you go back. I guess people are even exploring different data structures like directed acyclic graphs and stuff and how that could be used to map other domains but the reality is it's a linked list and you can spend up as many of them as you want and you can mine blocks based on all this different criteria. Bitcoin is a proof of work associated with the minting of a new block and that's been a problem for them as they scale as a currency but it could be a history of anything and the minting of those blocks can be based on anything. You mentioned banks, the financial kind of sector is certainly interested in these smaller private chains but do you think there's a use for that consumer market as well? How do you think that your personal blockchain or set of blockchains might be a factor in the hobbyist of the futurist life? CHRIS: Oh, wow. That's a different question than I thought. [inaudible] where you're going with that -- JOE: Where do you think I was going? CHRIS: Well, we're talking about banks and so, the question is now everybody other than banks -- JOE: Well, it could be everybody, including banks too, however you want to take it. CHRIS: Yeah. There's a much harder question, I think of what in the world we're actually saying when we are talking about blockchain, right? The notion obviously has started with bitcoin but if what you want to do is bitcoin, then you should just be as in bitcoin, so what are we talking about similar bitcoin and the general phrase people have they like to throw in here is Byzantine fault tolerance. I'm talking about any kind of system that can have multiple participants. We're used to talking about clusters of computers and making systems that can work if one of them fails, if one of them just stops working but now, we're starting to talk about how do we make systems work if one of them gets hacked, then we still have some assurances that the whole system works together as a whole. JOE: Would you consider Byzantine fault tolerance to be the defining factor of a blockchain because I feel like there's the timestamping element that goes along with it. I feel like they're kind of part and parcel, right? CHRIS: Kind of but if you're not considering Byzantine faults, if you're only talking about systems where you have benign faults, which is a machine goes down sometimes, then timestamping isn't really a problem because we can just use NTP and we all have a pretty sensible idea of what time it is. JOE: Time specifically, even just like, I guess order. I always considered sequence to be a massive part of what a blockchain fundamentally was. You have the distributed aspect of the network that gives this sort of resilience to malicious intent but not only is it protected, I guess against demolition and malicious intent by this crowd strength but also just fundamentally through the cryptographic side of it, you can't go in and insert things that didn't happen. Once that order has been said, it's been written in stone, basically, right? Because the way I understood is there were papers coming out of Bell Labs in the early 90s and those two things set as approaches to this independently and it wasn't until the internet advance so we put them together and we're able to achieve Byzantine fault tolerance through that. Is that, I mean...? CHRIS: It does help a lot, I think to buck up and think about what the state of research was in the 90s because I think that's something that a lot of people in blockchain space kind of lose sight of. You have a whole lot of people writing papers now who didn't used to be academics until a couple of years ago. It was the early 90s where we started having faxes and we started having what later turned into what's kind of known as raft. Like you said, they solved the ordering problem. Even something as simple as what we call Lamport clocks which is you have sort of a virtual timestamps and as long as nobody's malicious, if you remove the timestamp forward, then we can all have something that resembles the deterministic forward flow of time. Then, that milestone that I was like to remind people of this in 1999 is when we had the paper practical Byzantine fault tolerance. JOE: That was '99. You're talking about the... was it Castro and --? CHRIS: Liskov, yeah. JOE: Okay. I didn't know it was '99. CHRIS: Interestingly, the same Liskov that the Liskov substitution principles named for, Barbara Liskov. It's also a distributed systems research. JOE: That's swell as well. I kind of heard the concept of Byzantine fault tolerance but I never read this paper. I'm also surprised to find that it didn't come out of that same period of the early 90s and it was as far as '99. I haven't read its entirety but I did fall asleep reading it last night. You mentioned this specifically, I guess, when we're talking today, as a paper that is important. It's the work that we're trying to do at... was it Hijro, I think? CHRIS: Yeah. JOE: Yeah, so what kind of work were you doing there and what is important to you, I guess about this paper specifically, when you look at all the research that went into priming the community for the space that we are now in? CHRIS: When I joined Hijro, I got kind of a difficult and nebulous mission, which was that everyone in and around that space that was trying to sell to banks was if you said the word blockchain, you could get your foot in the door because all the banks were looking at bitcoin and saying, "Well, look, this is clearly something that's going to be big and we don't want to be missing out, so we have to figure out how this applies to us." JOE: What year is this? They were working this in 2014-ish, is that right? CHRIS: '15 or '16, I think. The question was trying to figure out what aspect of it was actually what they wanted here. What Hijro is trying to sell them, the details aren't even important for this conversation but we need an interbank solution. We needed a ledger of accounts that 'we weren't a bank so we couldn't be the one holding everyone's money and keeping track of the flow of money in our network.' We were on something that the banks were truly in charge of but we didn't want to necessarily have our platform be owned by a particular bank. We wanted to be the sort of consortium of all of our partners. JOE: Consortium is a keystone word I think here, that we should definitely come back to that. CHRIS: Yeah and people talk about, if I use the word consortium blockchain, I think sometimes in contrast with the public blockchain, with the 'free anyone can join' blockchain. JOE: Yeah. I'm particularly fascinated by this concept. That is a term that is used. I can confirm this. But you're doing that pretty early then because I feel like that concept didn't make it out into, I guess the public understanding, until recently or maybe I'm just behind at times. CHRIS: Yeah, I guess so. I don't know. When I start working on this, I just spent a couple of months trying to read papers about what was in space and I guess, the only big name that was trying to do something like this was Tendermint. JOE: Tendermint? Interesting. CHRIS: You can pick out technologies like this because the magic number is always one-third. They can tolerate Byzantine failure up to one-third of the nodes. That was a theoretical result that was reached, just sort of the best you can do. Before BFT and then BFT is one of those solutions in that category and Tendermint does something similar. JOE: That, I guess is sort of the background to this paper and it's impacting your life. I guess, what is put forth in this paper is to solve for higher tolerance. Would that be the right way to put it? CHRIS: Did you say higher tolerance? JOE: Yes. You're talking about the Byzantine tolerance is 30%, right? With Tendermint? But you're saying that they're doing something similar to that's before in the paper? CHRIS: The most interesting thing to me, I think is probably, hopefully possible to convey concisely is the rationale behind the one-third number because that took a while for me to really appreciate but I think it really clicked when it did. One of the hardest intuitions to get people to break, I don't know, way of thinking to shift, I guess is convincing people that consensus is even a hard problem because I had this conversation a lot with people that'd say, "I've got this JavaScript library here, for instance that just lets me broadcast a message to all the nodes in a cluster, so why can I just do that?" Why can't we just use one at a time to do it and if I detected someone's trying to cheat, if I get two different messages from someone that are conflicting, maybe I can just ignore them. JOE: Not in finance. That's kind of ironic, I guess that you found it difficult to get people to come to a consensus about the importance of consensus. CHRIS: Right. The basic flow of all these things is we describe them as voting systems. We have voting rounds where each time, like you said the blockchain of the ledger or whatever it is, just a linked list, so the problem of using consensus build database is we're just going to iteratively try to vote or come to consensus on what the next block is. What the next ledger entry should be? Obviously, since we don't have a synchronized wall clock to go by, we have to assume messages can come in any order. We might all sort of speak up simultaneously and propose different blocks as the next one, at which point we have to start over and retry that. But furthermore, I can send different votes to different people if I'm trying to be malicious and that's where the tricky part comes on. The rationale for the one-third number, maybe I can just try to come around to that and say it directly then, is that when we take a vote for what the next block is going to be, we need the supermajority. We need two-thirds of the participants to have all said the same thing and the rationale for that is it's actually easier to think of it backwards. Rather than saying, two thirds of the total, what we say is, "If we're going to allow some fixed number of nodes to fail, to behave maliciously --" you know, we traditionally call that number 'F' in the paper, then what we say is we need 3F+1 total nodes to be participating. JOE: I didn't know that was sort of codified into how conflict is resolved on things like bitcoin during blockchain. It's inherent, I guess. CHRIS: No. This is the total opposite of what bitcoin and Ethereum are going to do. JOE: Because I always thought it was just going to be like a majority, I guess but what you are talking about is more like how the Senate would were to pass a resolution to the constitution, like it has to be an exceptional majority. I'm starting to understand why one-third, specifically. It's 3F+1, I guess. CHRIS: The reason is because for each vote, every time I look at the results of a vote, I have to be able to assume that some number that we called F, of the people that I've heard back from are trying to cheat me. It turns out I need to be sure that the majority of the votes that I've heard back are from people who are actually following the protocol correctly and not lying. We need to be tolerant to two kinds of failure. One is that a node simply goes down and we don't hear from them and we don't receive a vote from them and then the other kind of failure is the Byzantine failure, that they're not following protocol in some way. The reason I need 3F+1 nodes is because we need to be able to make progress, even if F of these number is we didn't hear from at all because they're down and then, I need 2F+1 votes because I need to take into account the possibility that some F of these votes were from cheaters and then we need to have more honest votes than lying votes. JOE: That's pretty profound. I definitely going to finish the rest of this paper while conscious later today. I guess we're a little off with regard to math at this point and it's when you said, you spent I guess a month or so just reading papers around the time you started with Hijro and I guess did you stop because I feel like I've read just more white papers than ever thought I would outside of the academic setting, just trying to keep pace with what's been going on, particularly with regard to the web. I don't if you're familiar with like IPFS but these sort of directed acyclic graph things are popping up all over the place and platforms are even now being built on this concept. I guess, Ethereum feels impractical in a lot of ways. These dime-a-dozen tutorials, when you started talking about the global computer that is Ethereum and the blockchain and it's going to change everything in the internet and you won't have to pay Comcast like some central authority or you just pay for each transactions. The reality of it is every time you do a write against a data store have, first of all, thousands of computers go and verify that and also, you don't want to store your information on a linked list. It's not feasible for storing large data structures and it becomes very expensive for the user and for the person, if you're maintaining a smart contract for the contract itself. These are volatile, all little points of value. It's impractical. CHRIS: It's definitely a cost that you don't want to incur. In all cases, just a confirmation time is a cost you don't want to incur. JOE: Absolutely. CHRIS: There is one nice thing that that you can do in some cases, which is that people is talking about the piggybacking on these blockchains like if I have a system and I just want some extra assurance to keep it honest, then I can do things like periodically publish a hash of my database onto something like bitcoin or Ethereum. JOE: Yeah. That actually happen with anyone in financial... They do publish stuff in the paper and this was before cryptographic ledgers but to basically prove that this was the state of something, I remembered seeing this somewhere, like there would be in financial news, like there'd be some crazy number or string at the top to verify what was on the string. CHRIS: Yes. Of course, the irony there is that you really don't need some kind of blockchain if you want to do that because the fact that we're doing that before the blockchain has existed and doubly, it's funny because the first block of the bitcoin blockchain, the genesis block includes in it, I think a New York Times headline, which was intended as proof that Satoshi or whoever didn't spend years mining bitcoin prior to releasing it. It's supposed to be a proof of the time of the first genesis actually was. It's funny that we are actually already had this verification system and what that demonstrates is sort of a principle of consensus that I like to talk about which is that as you increase the time scale, consensus becomes an easier and easier problem. I think the reason why something like newspaper headlines are reliable means of a timestamp is just mostly because they're big and slow, because there's only one every day. I think the whole challenge like you said of, how a lot of systems kind of boiled down to having the white paper for bitcoin refers it describes bitcoin as a distributed timestamp server, something along those lines. The reason why you need a new technology to do that, I think so that you could have timestamps that are every of couple minutes, rather than every 24 hours. JOE: That's a very interesting take on it. I guess, the more time there is, it is easier to reach a consensus. It's just interesting to think about. It's funny as humans like the longer time passes, the less reliable memory is, I guess, less reliable history as we conceive of it, I guess. It's different when you record something than the way that you hold in the brain that sometimes I wonder how much impact that's had on. It's a little ephemeral, I guess but it's interesting. CHRIS: Yeah. I guess my statement is limited to the on-scale where we can actually fit into memory. JOE: Right, that most of the times, it's the only relevant scale, I guess, like a blockchain doesn't have use outside of our use of it, inherent to it, so it's going to be seen through that lens, I guess of our use of it. I think it is kind of profound, a thing to think about that I definitely considered. You mentioned using blockchains as adding a little bit of... how do you put it? Like truthiness, I guess, we'll say. I know that's not how you put it but adding a little bit of security, maybe around something else but the reality is you can get away with that on a number of other levels. I think that's important and interesting to think about. There seems to be this trend now talking about a blockchain as part of a bigger picture or consortium blockchain or a consortia of blockchains, right? Because a consortia would be multiple and then a consortium would be... No, a consortium would be a single grouping, consortia would be multiple groups. Basically, going back to the problem you're trying to solve with Hijro, you have multiple banks and I believe eventually, I don't know if you work on it, there was a protocol that came out of that company to unify these blockchains, like a few of them. They demoed and everything. That, I think gives you some power with regard to access control but again, I guess, that's not a thing that you really need consensus for. So, where does it fit in? Aside from things like voting and transparent finance for maybe a political cause or in the case of bitcoin, just finance in general. In bitcoin, I feel like we got Mongo DB super hard in the sense that it just got applied to every domain and it applies to very, very few. CHRIS: My boss at Hijro, Lamar Wilson really like to say that people talked about blockchain like it was hot sauce and they sort of sprinkle it on everything to make it better. JOE: That's sad. CHRIS: I guess, two answer to that one. One of the places where it absolutely captivates people's imaginations too far and doesn't work and then places where it doesn't work, so I want to start with the first here because the biggest mistake that people make is that there was this notion of tokenization that came out of Ethereum, where anyone could make a smart contract that represented something and now, also that I can trade digitally. Just like it's money or some kind of digital asset, so people want to talk about putting your car, putting your house on the blockchain or selling it there. But it's just shocking how many times I had to remind people that if I make a smart contract that represents cars and I put my VIN number on it and I transfer you my car, at Ethereum contract in an exchange for a bitcoin, if I call the police and report my car stolen, they're not going to look at the Ethereum contract, right? JOE: Yeah. Man, you're really right. People don't think about that enough. If your car is in the blockchain, your car still on the block. CHRIS: What we had realize when we're selling solutions like this is that they're great for some reasons but you need actual legal agreements to underpin things when you actually make connection to the real world. The magic of bitcoin that can't really be replicated is that the coin actually didn't need a pinning to the real world because the thing bitcoin was running was itself. It just depended on hoping that people were going to find the coin and ledger valuable intrinsically and bitcoin never really purported to control things in the real world. JOE: I guess, definitely not in the paper. There are some place that can buy in from some very specific elements of society that sort of cemented its place as useful but we don't really need to go to that road, I guess. I don't know. You know, my roommate is a lawyer and we have this conversation often and I feel like if we go down law and cryptography, we're going to be talking for too long, where we are at currently. CHRIS: Right and that wasn't your question anyway. It was just what I respond to easiest because being a critic is always the easier thing to do. JOE: I can feel you there. CHRIS: One of the interesting things that I never even found too much about but I noticed this in a couple of passing references as I was reading stuff about Byzantine fault tolerance in general is that it seems to have some application in things like flight control systems and space ships because when you think about a computer that you're going to send into space, you have two things that Byzantine fault tolerance applies to directly. One is you need a lot of redundancy. You need these control systems, maybe you have a dozen things computing the same result because you can't replace the hardware when once you shot something to space. The second thing is once you've sent something outside the atmosphere, all of the sudden, you're being bombarded with a lot more cosmic rays than you were before. Now, you actually really do this idea that computers can fail, not just by stopping but by producing wrong results. All of a sudden, it becomes a lot more real because you actually have physics slipping a bit at your computers. JOE: I don't even think you have to go as far as space if you talk about just like a fleet of something, like self-trading cards. I suppose, in domain where there is an interplanetary file system, it's good to specify the planet we're talking about. Just having worked a little bit with robots in college, they lie all the time and they produce bad data constantly, so not even bad actors just incompetent actors, I guess could definitely... This is something that has to be, I guess on our minds as we move forward as the society that has more connected devices, which I think as much as I would love to have left this conversation off in outer space, I think bringing it around to the internet of things, which is sort of where this all began months and months ago is probably a good place to stop meandering through these cryptographic weeds. You can probably put a pin in this. I think we've been talking about for a while now, I guess and just kind of trying to see what it is and where the applications are. It's constantly changing and never clear, I think is the conclusion that I've come to. I don't know. I think, just kind of shooting the breeze about it is a fitting end to a series of Frontside engagements in this space, for the time being. CHRIS: I've seen several people try to tackle the space of how to stop relying on things like Google Drive to store our data because I think a lot of us have realized that we're tired of losing all of our family photos every time a hard drive dies but a lot of people are uncomfortable with trusting Google with everything. This to me seems like a perfect opportunity for people to start building redundant systems among their home and friends. JOE: Yeah, I completely agree. I'm actively trying to do exactly that right now. CHRIS: Oh, cool. And you don't necessarily want your cluster of machines that's running on all of your family's computers to be able to go down if your 10-year old get some virus, right? JOE: Right and also, there's definitely things that you want just within your home or even just within your section of the home. I guess you could layer chains, to kind of manage those interactions? CHRIS: Sure. I'm not exactly sure what you mean by layering chains. JOE: You could have consortia in this case. If you had like a hypervisor, almost like a control notice, essentially or some type of view from above of this situation, you could say, think of it as a family scenario. We have three different houses on this call that all belong to our immediate family and cousins and whatever and it's like, me and my siblings, we have information that we all want just within the siblings. We don't want Mom and Dad to know. We don't want the cousins to know, so you could basically use like a blockchain to kind of date access to data that is held within that consortium and then the consortia could communicate amongst each other. Only the pertinent information that they wanted to allow access to at that time and then, internally of course, you could have all these different mechanisms for how you actually store that data or how you actually serve it up. It's pretty complicated. CHRIS: Yeah, I think you made a lot of sense, though. JOE: Yeah, cool. I'm hoping so. There's been some work on it out of Microsoft, actually. CHRIS: On the files storage problem, specifically? JOE: I guess this is like with a smart home and kind of just teaching devices to cooperate and ask each other. If you had a section of connected devices that maybe were related to the workflow that a human being might go through to get groceries or something and then a section that's related to doing laundry or whatever, eventually, they would learn to communicate in the laundry grouping and could say, "Hey, grocery people. We're out of soup," or something like that. It's sort of almost happened organically, I guess. I had not actually felt like I found that paper. I've only found references to it. This is where I need to get something like academic access but that was interesting stuff. I don't know how I end up here, either. This has always happening when you're talking about this domain. Anyway -- CHRIS: People's ideas, it's just sort of generally inspiring concept so people is following you everywhere. JOE: Yeah, it's heartwarming. You know, with my ICT, I could look back and see exactly where I usually came from than [inaudible], the name of the farmer who grew with. I don't know. It'd be so much easier to fake most things, really when you think about it. On that note, I hope that this conversation was... I know that there was no JavaScript and I apologize for that but I hope that our audience finds it interesting on some level and I want to thank you for your time. Chris, it was really great talking to you and getting your take on these things as somebody who's been in the industry for a while. Definitely, some fascinating points to consider and definitely, I will finish that white paper, probably this evening because it's pretty cool. If anybody in the audience has anything they'd like to ask you about pertinent to this conversation or anything else, where is a good place to get a hold of you? CHRIS: For me, it's mostly Twitter. I'm @chris__martin. I'm also at Type Classes, if you want to talk to me about our new business. JOE: Cool. This has been Episode 104, I believe of The Frontside Podcast. Frontside, we're a consultancy based in Austin, Texas and we love writing elegant, sustainable code and just producing good stuff, really. I think that's what we're all about. I think, we can agree at least, that's a core tenet of what we do and if you would like us to produce some good stuff for you, feel free to get in touch with us. Also, feel free to reach out via email if you have any ideas for future topics or any feedback about this episode. I also want to thank Mandy for producing this episode. You can catch us next week, I believe for our talk with Brian Douglas on Probot and Robert will be hosting that one, as far as I know. Thank you all for your time and feel free to reach out. This has been The Frontside Podcast. I'm Joe LaSala. Chris Martin, thank you for joining us and have a good day, everybody.

In this episode, Michael Jackson of React Training and Rob DeLuca and Taras Mankovski of The Frontside talk about what is a component, and what a component is specifically in the context of React. They also discuss when it stops being a component and becomes something larger, how it has changed the way we develop UI, and thoughts on container and presentational components being synonyms for controller and view. References The Tweet that started it all Wil Wilsman: Does my application work in real life? Do you have opinions on this show? Want to hear about a specific topic in the future? Reach out to us at contact@frontside.io or on Twitter at @thefrontside. This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. TRANSCRIPT: ROBERT: Hello, everyone. Welcome to Episode 103 of The Frontside Podcast. I'm Robert DeLuca, a developer here at the Frontside and I'll be today's episode host. We're going to be discussing what is a React component or what is a component with Michael Jackson. I'm pretty excited about this topic, a sort of off from a tweet that I sent out after a long workday, when it is something being called "component" because it was built with React components but it was more like a mini application? Then Michael replied to my tweet, as they're happening and he said, "What is your definition of a component?" which is exactly what we're going to be discussing today. I thought that was a really great question. With me, as a co-host is Taras Mankovski. TARAS: Hello, hello. I'm also a developer at Frontside. ROBERT: Before we get into the discussion, I would like to make a little announcement. We've been working on building a suite of testing tools to make acceptance testing JavaScript apps like React or Vue or Ember or anything else kind of JavaScript, fast and easy to maintain over the past few months. We call it BigTest. If you follow me on Twitter, I'm kind of loud about it. I love this project that we were working and one of my coworkers just published a blog post, giving an introduction to acceptance testing on React applications with BigTest. If that sounds interesting to you, you should check it out. We'll leave a link in the show notes. Now, let's jump right into it. Hi, Michael. How are you doing? MICHAEL: Hi Robert, I'm doing well. Thank you for inviting me to be on the podcast. ROBERT: Oh, no. Thank you for joining. I feel like this going to be really fun conversation. We went back and forth on Twitter a little bit and I was like, "You know what? This serves really well like a free flowing conversation." MICHAEL: Yeah, it's something I really like to talk about, specifically, when we're talking about components or when we're talking about React components. I think there's a lot stuff there to discuss and I think that React, specifically has really, at least defined that word for me. When you were talking about your mini application, I was like, "Oh, yeah. That's cool," and to me, that's kind of what these components have the potential to be, what these React components have the potential to be. It's kind of like almost these miniature applications that you stitched together to make a bigger one. ROBERT: Yeah, that's really cool. I was frustrated at the end of that day, just because that had so much logic crammed into it and you can kind of see that come through in the tweet but I'm not going to lie. After I sent that tweet, I was a little disappointed with myself because it came off a little bit like flangy but I guess we could just kind of jump right into defining what is a component. MICHAEL: When I saw your tweet, I was like, "Oh, you know like --" and of course, I don't know the component that you're looking at. It could have been terrible. Totally, it could have been terrible. I'm totally willing to believe that it was not a good component. But writing React is hard, especially writing good React is actually hard. At first, I thought it was easy but I think it's easy to solve your immediate problems but to write React to that is generic enough to solve lots of different problems, I think it's actually very, very hard and it's something that I've spent the last couple years learning. Anyway, that's kind of a tangent. But regardless of the component that you were looking at, the way that I've tend to think about these components is we used to have a model for thinking about how to build frontend applications -- ROBERT: Like MVC? MICHAEL: MVC, that's the one. That's the one that talking about. You have, here's your model, here's your view, here's your controller. You know, there are separate logical entities and lots of times, those even live in separate files on the file system and we keep them sort of separate and spaced out. As far as I can tell, that's a sort of paradigm. I learned it in school. By the time universities catch up with industry is at least, like twenty years, so it's probably something that was invented back in the 60s or 70s. ROBERT: I think it was like the 80s because Charles talks about it a lot and sort of Taras, actually because we're building something that's like a composable model, just like the Vue of React is a composable view, its components. I think it's interesting because I was listening to the Changelog Podcast and it sounds like you worked with Ember for a while. MICHAEL: I did, yeah. ROBERT: I forget, which podcast that was but we can find it and we'll link it to the show notes but I thought that was really interesting that you also came... Oh, maybe not came from Ember. You had experience with for, at least a couple years and that's where I came from. That's kind of where that tweet came from and was like, "Man, I don't like what happened to the word component," because when I think of component, it's like something that is encapsulated in a small piece of UI. But I'm okay with letting go of that. MICHAEL: Well, yes. Ember came out of something called SproutCore, which actually originated at Apple. Tom Dale used to be a developer at Apple. I'm sure you all know all of this but just for some sort of back story for others who might not. Apple was a place where MVC was really, really big. The whole Cocoa frameworks. They're all very much like, here's your controller, here's your view, core data has all of your models, that kind of thing. It's a software development philosophy that goes back a long time and has deep roots and I'm not here to suggest that because it's old it's bad, because lots of old things are actually very, very nice and good, especially in programming. We're starting to see a resurgence of functional programming ideas, for example, which have been around forever, since at least, the 60s or even earlier than that. It's not necessarily that I think that because it's old, that it's bad. I don't. ROBERT: Right, right. MICHAEL: But the thing that I do see in these React components is they kind of go very much against the grain. If you had, for example your model, your view and your controller as these vertical silos, just imagine turning that whole thing on its side and then cutting a slices of that, so that each of your React components has elements of MVC in it. They're all sort of mixed together. I run a training company called React Training and whenever I'm doing one of my training workshops, the way that I talk about it is a component is able to encapsulate three things: markup, which is the view essentially; state, which is essentially your data or your model; and behavior, which is essentially the controller, so what happens when the user is interacting with the view and clicking on the buttons? What do we do in response to the user input? You've got all three of those MVC elements in the React component. In fact, I've been meaning to kind of coin the term for a while. Instead of MVC, I call it MSB -- markup, state and behavior. ROBERT: You heard it here first. MICHAEL: That's right, here on The Frontside Podcast, we're coining new terms. ROBERT: That's pretty great. I guess coining off of that, is it fair to say that MVC kind of still lives on, except it's now in React? Or it's kind of more sliced up into smaller, more digestible pieces? We can always come back to this composability element, right? Would you say like really React would made it possible to compose many little MVC apps? MICHAEL: Yeah, exactly. The thing that people really, really love about MVC, I think you still have it in React. MVC, the thing that people always come back to is the separation of concerns. Let's just say, "I don't want everything sort of mingled together. I want it to be very clear. What is my model? What is my controller? And what is my view?" The cool thing about a React component is if you look at it, you still have very clear separation within the single component. You have a very clear separation of all three of those concerns: your state lives in an object called 'this.state,' then your view is whatever you return from your render method. That's in a completely separate spot, since still in the same component and then your behavior or your controller usually has its own method, so you can inline those in your render method or you can pull them out if this is how you'd like to do it. You can pull them out into instance methods. You can still have kind of that separation of the concerns within a single component. I think to your point, the React component model is all about composition, so this is kind of a term that we've actually adopted in other areas in technology. It's sort of just coming to frontend. The idea is you've probably heard of a service-oriented architecture for design in backend systems. These servers over here are application servers, these are database servers, here's our caching tier, here's our Ingress or LoadBalancer and maybe, you stitch them all together on the backend into one coherent system. You know, here's our storage and our image-resizing services. You stitch them all together on the backend into one coherent system but it benefits you to not build them all as one part of sort of monolith. We've got a lot of benefits from the decoupling. For one, if one piece of the system fails, the whole thing doesn't go down, so you can like swap it out or upgrade it and replace it with something else. I feel like that is kind of a similar idea to what is going on the frontend with these React components. ROBERT: Oh, microservice components. MICHAEL: Yeah, exactly. Each component is kind of like you said. It could be like a miniature application and it could be totally self-sufficient and I could drop it onto a page and it could just do its own thing or what I could do is I could take that app, that component and I could drop it into a larger app that can speak to it and can communicate state to it via the props that it sees and then can get state back out of it via these callbacks. I kind of view it as like all of these miniature systems that they can either be completely encapsulated and self-governing on their own or you can take them and compose them with a larger app. ROBERT: That's really interesting. I guess what is a component specifically in the context of a React. I guess, we kind of just answered that, that you can bundle all these things together, right? Does that make sense? MICHAEL: Yeah. The beautiful thing about this composition model is you can have all three of those things in the same component: markup, state and behavior. You can have all three of them right there, encapsulated in one component or once you discover more advanced patterns with React, you can actually make it so that a component only has two of the three or one of the three. ROBERT: And then it kind of like passes callbacks or props or state around? MICHAEL: Exactly. It can essentially delegate the responsibilities for handling the other pieces that it misses to other components around it. There's a pattern that I kind of coined last year called render props. ROBERT: Oh, yes. I remember hearing this on Changelog Podcast. I was actually one of the people that at first thought it literally was a prop name render. MICHAEL: Well, I mean it's not a bad way to think about it. It could be a prop name render. ROBERT: But then I found out it actually is a prop that you send JSX too and render whatever is there. MICHAEL: Yeah, exactly, a prop that you used to render stuff. It's the way that I describe it. When you think about it, any component, for example that accepts a render prop is really just delegating it's rendering to some other function. It can contain the two pieces: the state and the behavior, but then it can sort of delegate this third piece, which is the markup and say, "You know what? I don't really care about, specifically what I have to render." I don't have an opinion about that. You just give me a function and I'll give you my state and possibly, some behavior callbacks and then you can tell me what to render. I'm like a component that I have two of those three pieces. I don't have the markup, you just give me a function, that I will call when I need the markup. ROBERT: That's really powerful. MICHAEL: Yeah, exactly. I can have all three in one place or I can also with this component model, just delegate one piece or even two pieces, using these kinds of patterns to other components, so it's really nice. ROBERT: I'm always have to come back to this because this is where I kind of cut my teeth in the frontend world. I feel like render props are very similar to the yield pattern in Ember, where you can yield out a component and place it anywhere inside that template block. I think it's really similar. Because we have a large base of Ember developers that listen and they needed a draw like a parallel, I would say that's almost a parallel. I might be off on that. Taras, would you agree with that? TARAS: Yeah, it's very similar. There is one little 'gotcha' with render props in React, which I think Ember deals with a little bit more gracefully. I do use React all the time now. I use it much more often than I use Ember but -- ROBERT: Same. TARAS: Yeah and I'm kind of curious, Michael if you would think about this. Because there are certain things that, for example, JSX does, like one of the attributes of using render prop is that you want to pass stable functions, like something that is actually defined in init hook, so you're not necessarily recreating a new identity for this function every time that you pass into the render prop. Is that right? I think for someone who is might not be familiar with this, it's a little bit counterintuitive because it just kind of like, you just pass something and then you later realize, "Actually, I can't really do that because now your components were rendering over and over again." I'm curious like where can we kind of innovate in that? Because in Ember world, for example, it's really safe to use render props. If you're delegating to the context to do the rendering, you can very easily pass a block of template into the component and then the component will know how to render that in a performant way. My experience with render props in React was that the pattern in Ember is it's a very common behavior to delegate to the context of how the children of a component are rendered. This is the primary purpose of the yield in Ember and this is used by a lot of the components to provide APIs, so essentially, you're yielding, you're sending into the children function behavior that you want the developer to use while consuming your component. The parallel with React is that you would have, essentially a function, like you have a children render prop, which is a function that receives so that when a component is rendering the children render prop, it's passing some data into that function and it's invoking the children function and it's passing to it some data and then that data is being used inside of the children render function. Now, this mechanism works really well in Ember because Ember is using its own templating engine, so it manages every placeholder in the entire template as being passed into the component. It knows exactly what has changed in the function. Looking at that, the value just passing through component as a whole thing. It's actually managing each individual spot where the dynamic elements needs to be rendered. What that does is it allows it to manage the entire children function so you don't need to worry about stabilizing the render props. You don't need to assign them. Because there's also other challenges that kind of arise when you have to move the actual function that you pass into children. You have to move into the body of the actual component. Your ability to compose the state that is within the render function is kind of limit because you now have kind of broken up into different places. I'm a little bit curious what are your thoughts about this? And if there's any innovation that's going to require or improvements that will require in the way that React handles render props. ROBERT: I definitely remember seeing some tweets from you, Michael about this, like very recently. MICHAEL: Yeah, just yesterday in fact, I was talking about this on my Twitter. First of all, thanks for the question, Taras. I appreciate it. I think you kind of nailed it with the difference between Ember and for example, just JSX -- Handlebars, really and JSX is that Handlebars is not JavaScript. Handlebars is a language of its own, really. As the implementers of that language, the Ember team or the Handlebars team has complete control over how things work. For example, when functions are allocated. If they have a yield block, for example, that yield block does not have access to things like JavaScript Scope. It really is just a block within the templating language and so, they can decide when they're implementing that language. They can say, "We're only going to ever allocate one function for this yield block." JSX on the other hand, it really is just sort of some syntax sugar over JavaScript, which is nice lots of times because you can use things like JavaScript functions scope. You can do things like refactor it, like you would any other piece of JavaScript and pull that function out into its own variable or something like that. In a lot of ways, it's more flexible but with that flexibility comes this extra concern of what happens with the prop passing and it's a real concern but I don't think it's as big of a problem as most people have heard or might think it is. The real problem, the root of the problem is that if you have a pure component in React, the pure component is optimization tool in React, if you find that your component really is a pure component of just a pure function of its props and state, and that it will never render anything different, if it receives the exact same props and state, you can declare that that component is a pure component. Instead of just a regular component, it's now a pure component and so, it will actually opt out of the render cycle if it receives the exact same props and state on subsequent render. By exact same, what I mean is like identity triple -- ROBERT: Yeah. MICHAEL: -- Same. That's an optimization tool that you can use in React to say for example, maybe I've got a component and for some reason, it's always receiving the exact same props and it's rendering a lot. I can say, "You know what, React? Don't even waste your time re-rendering this component and doing the reconciliation around it and everything like that. If it receives the exact same props, don't even waste your time. This is a pure component. It received the same props. It's going to render the exact same stuff as it rendered last time, so it will completely opt out of the render cycle," so React won't even call the render method, won't even bother doing the reconciliation for that component or its descendants. It's actually a really, really nice way to get a little performance boost in that situation. ROBERT: Interesting. Does that mean, rather than passing some JSX that returns out an anonymous function, you would rather have like a pure component over that? MICHAEL: All it really means is that when you identify a place in your app where this is happening, instead of extending React component, you can extend React.PureComponent and now you component opts out of the render cycle. ROBERT: Gotcha. Because I see a lot in React applications, it will just return some JSX out of anonymous function, kind of think, Taras was getting at, when each re-render happens, it's recreating a new function and all of it were done inside of it, from the JSX because it's just a new function that's being bound each time. I think my takeaway from that and correct me if I'm wrong, is don't pass JSX and an anonymous function or I guess, from your tweets yesterday, that's kind of a premature optimization. Maybe, not pass as anonymous function but pass as pure component. MICHAEL: I hadn't yet gotten to directly addressing Taras's concern but the idea is -- ROBERT: -- Jumping the gun. MICHAEL: Yeah, it's fine. If you do have a pure component, that is when you're going to have a problem with this render prop pattern because you can't pass a render prop to a pure component. They just don't go together. Well, sorry. I should actually qualify that. You can pass around a prop to pure component but you need to make sure that it is always the exact same function, otherwise why are you making this thing a pure component, so there's a little bit of a problem there. I actually wrote the documentation on React's website on render props and there's a little caveat section where I discuss how you could get around this, you could convert it into an instance method or something else but then the concern at this point and this is why I said it's not such a huge problem because we're getting very, very specific now. Let's assume that we've got a pure component that is the child that we want to render and it accepts a render prop and that render prop depends on state and/or some sort of scope that it needs in the render method. Now, we've got a problem because we need to generate a new function in the render method and the fact that we're generating a new function and passing it to a pure component means that we are essentially negating all the benefits of extending pure component in the first place. First of all, in order to experience the problem, you have to get very, very specific and it's not easy to actually get into a point where the problem manifests itself. But then for the second part, fixing the problem is actually not tricky, so the way that you would fix this issue, there's a couple of ways to fix it. The most common way, I think to fix it would be to just say that pure component, let's just move that down one level and we'll put it inside the render method of the component that takes the render prop. The component that takes the render prop is no longer pure but the pure component has dropped down one level and we can still get all of the benefits of a pure component and again, its whole descendant tree lives underneath it. ROBERT: That make sense. MICHAEL: So, we can still get all the benefits of having a pure component, while still being able to use the pattern. But again, just getting back to what we were saying at the beginning. The render props is just one example of an advanced pattern that actually lets you delegate one of these three pieces to somewhere else. It's just one pattern. It's not like there are others as well. ROBERT: Yeah, so getting back to what is a component, is there ever a line that's drawn, like when does it stop becoming a component and it becomes something larger. MICHAEL: It's actually interesting because for a long time, we didn't have a component model on the web and I think -- ROBERT: It's very true. MICHAEL: -- still, like we mix to all three things together in our jQuery code. It was just like there was some state mixed in with some markup, mixed in with some behavior. It was all kind of mixed into one place. ROBERT: Nice plate of spaghetti. MICHAEL: Yeah, exactly. As soon React came along and actually gave us a real component model that really worked where we could actually identify these pieces, then all of a sudden, everything is a component and you really can make if you want, like if you really wanted to, you could make your entire application out of components. It could just even behavior stuff like, "I want to fetch some data," that could be a component or, "I want to listen to the window scroll or the window size," that could be another component. "I want to render an image or a bit of text or do some measuring or I want to do some parsing of some data or something like that," all of this stuff could be represented as components if you wanted it to. Of course, you can still extract those functions out into their own bag of utility methods and lots of times when I write a React Apple, I have a little bag of utils and I'll go and reach in there and grab those as I need them. But for the most part, most of my React apps are just components all the way down, so there's not much that I can't do with a component. The cool thing about that is what do you get when you have components, where you get reusability and sharability. I can make a component and I can just share it with you, Robert. I could say, "You want a component for listening to the windows size? Here you go. Here's the components. Got to render --" ROBERT: Drop it in. MICHAEL: "-- It will give you the size when you need it," so that ability to share code is, I think super important. ROBERT: Yeah, I think so too and it really helps you build applications rapidly because you can just start dropping these little components that are contained in composable throughout your entire app. I guess, I'm still clinging onto like there is maybe a line that is drawn and I think from the React community, there were a couple of terms coined called the container component and the presentational component, in which I immensely mapped those to basically being synonyms for controller and view. It's like the container components is kind of like a controller, like it delegates the view... Or decorates the view, not delegates. And the presentational component is just like we're talking earlier -- pure component. It's just taking props and rendering that. Is that fair to say? MICHAEL: Yeah. As far as I can tell, that's a pattern that was first talked about by the redux community, specifically I think Dan is the first to discuss that that pattern and it's a pattern that I think can work well. I haven't ever found it to be especially useful in my code but I can see how other people could like it for their code. I really just think it has to do with how your brain works. If you prefer structuring things like that, if you prefer like the vast majority of your components to just be these little view layers and then your main container components to hold all of the logic, I want fault you for that. I'd say, that's fine you can build your app like that. But I do think that when you're thinking about components like that, you are thinking about the much less like these miniature applications. Basically, in my opinion it's kind of a false separation. It's a separation that doesn't really need to exist because essentially, the thing that I really love about these components is you have encapsulation. You encapsulate everything. As soon as you adopt this pattern of containers and essentially, just these presentational leaf nodes, they're not as useful on their own. They just encapsulate some of the rendering logic. ROBERT: That's very true and they're not reusable. I mean, it's very, very hard to make those reusable. MICHAEL: Well, yeah. Most of the time, they're designed to be used with a very specific container. It is like, "Why couldn't we just put that in a container." Like I said, I really don't fault anybody for using them. It's just not how my brain works. I tend to think more about components as if they were these miniature applications and they're going to be useful on their own for the most part. One place where I have started to do that, the fourth thing that your components can encapsulate is style information, which is really just kind of the markup but it's the fourth thing and I think that I've really started to employ that kind of pattern when styling comes in. But it's not for this separation of containers and leaf nodes. It's just because I would like to encapsulate some of the styles down at a lower level or be able to reuse some of the styles down at a lower level, so that's when I'll typically start breaking up one of my components into multiple kind of tightly-coupled components but it's just for the sake of the styling, not really in any of the control flow or the logic. ROBERT: Right. When I first got into doing React, I was first like a little befuddled and then, I fell in love. I was like head over heels. When I hear about separation of concerns, it's always talked about like we need to separate HTML, JavaScript and CSS. They're all different things. They just need to be separate. I understand that. I kind of come from there but for me, my personal opinion is coming from the context of single-page apps, there is no feature that you can ship without the three. I don't think there is a complete feature that exists. Sure, there's edge cases that's like a blanket statement but I think those lines of separation is way more blurred these days and I think that's what React just unlocked. That's kind of where all of this is sort of rolling downhill and it's not a train. It's really powerful to be able to encapsulate all of those things together and then just think of your UI as these components. That's why we call them components. They componentized. Like when I have a butt-in and I drop it on the page, I don't want to have to go and make sure that I have the right class that's wired up in some other file. I can just drop the button in there and it's going to do its thing. I can pass the props, it'll change it, that's fine. It lives within itself. MICHAEL: Yeah, I would say you're absolutely right. You can't ship anything meaningful without having, at least some styling in there, unless you're shipping very, very basic behavioral style components. It's kind of one piece, I think like the React components are able to encapsulate their own styles but there's no, I guess officially recommended way to do it, which is where I think React get some of the critique, like some people come in and say like, "There are a million different ways to do styling in React and therefore, React is supposed to something like --" I guess, yeah, some of the other frontend frameworks have like, "This is how you do styling in our framework." ROBERT: Right, like the lack of convention. MICHAEL: Yeah, exactly. It's a blessing and a curse, though. If there's a lack of convention, it's harder for newcomers to know exactly what to do. ROBERT: Yeah, or like what's best for their use case. MICHAEL: Yeah, exactly but then on the other side, it's kind of a blessing because people are free to experiment and they don't have to feel like if they're experimenting, they're going sort of against the grain of what the community has already accepted to be the answer. I think that freedom to get out there and experiment is really valuable as well. ROBERT: You can see but I'm nodding. TARAS: I'm just curious what your thoughts were. This Suspense API is something that's... React kind of lead the way with components and now, the changes that are coming in with Suspense API coming in and it'd be great, if you can, are you able to explain a little bit about what it does for people that are not familiar and I'm curious what your thoughts about it. MICHAEL: Yeah, for sure. The Suspense API, I think is actually, with regards to what we've been talking about, with the encapsulation of markup, state and behavior and possibly styles, I think the Suspense API is kind of the React teams attempt to sort of encapsulate or generalize, at least one more thing, which is asynchronous behavior. The truth is, the most common async behavior that we know of, that most people are familiar with is a fetch, a data fetch -- you go and fetch some data. But there are others, there are things like loading of images and animations and navigation and things like that, where you have some time that passes in between when you initiate an operation and when you're actually able to render as a result of completing that operation. How do we manage or how do we generalize that and how do we think about it? The Suspense API is basically a way to declaratively say, "I need this async operation to be done." Let's just use the example of a fetch. I need this fetch. I need to have fetched this data in order to render this view and don't even attempt to render this view, unless the data has been fetched. Before the async API or Suspense, as they're calling it, you would basically have to manage this asynchronous behavior yourself. You would basically have to say, "Go and do the fetch," and while we're doing the fetch, maybe I'll set a loading flag in my state and say, "Loading is true," and then when the fetch gets back, I'll say, "Loading is false," and then I'll render. Maybe, while loading is true, I'll show like a spinner or some loading dots or something. What the Suspense API allows us to do or what they're hoping to do, again it's all very like... I don't know if I made this clear. It's still very, very, very, very early days for this API and I don't actually know when it'll be ready. But anyway, the idea is to be able to say -- ROBERT: Would you say the Suspense is killing the community? MICHAEL: The suspense around suspense. ROBERT: A terrible pun. MICHAEL: Anyway, the API is basically just a more declarative way to indicate where this asynchronous stuff is happening and give React sort of clues, if you will, to say, "There's some asynchronous stuff happening here, just so you know." That has many applications in the real world. Let's say for example, you were navigating in a master detail view and you're tapping down the master view and loading these detail views, if you tap a master view and you haven't yet loaded the information for the detail view, instead of sliding to the detail and showing the data still loading there in the detail view, you might want to show some sort of loading indicator while you're still in the master view. Then, if it's taking a while to load, they can actually select something else from the master view and in that case, you would cancel the old fetch and start another fetch. It's just supposed to make things like that, kind of feel a little bit more fluid and a little bit more intuitive to the programmer, so you don't have to think so hard about managing a lot of that complexity in your own head. ROBERT: Yeah. By the way, I love the mobile first language that you used. You tapped here, you tapped here, that slides in. MICHAEL: Yeah, for sure. It's all we're building these days. ROBERT: Yeah, it's very true. TARAS: It's interesting because the Suspense API, you wouldn't really imagine it to be within the scope of a view in a traditional sense, to [inaudible] but considering the scope of a component in React, it makes sense but I think people might not imagine as being something that would become part of core, potentially in React. I'm curious like what do you think other areas of React and I'm glad that the scope of components in React is so broad because it kind of opens up this question to be pretty much anything in React ecosystem. But is there an area of React building or the aspects of applications that you would like to see an improvement in or some kind of a change or something that hasn't seen the right amount of innovation in the same way that we've seen in other areas? MICHAEL: Basically, is there a place where I would like to see the React ecosystem improve? TARAS: Yes. MICHAEL: I know this goes kind of contrary to what I just said but I would like to see a little bit more cohesiveness. I feel like in some ways, the freedom is good but I also feel like in some ways, for the last couple of years, we've been just sort of like rehashing the exact same old problems again and again because React really made everything else so easy, like a lot of the stuff that was hard about building web apps, just got really easy when React came along. I'm not saying that because I have React training company. Of course, my livelihood right now depends on React but honestly, if it wasn't for React, I think I probably would have quit and done something else because it was actually getting really, really difficult for me. I was trying to build an app in Ember before I started with React and I just couldn't do it, you know? I just couldn't get the level of polish that I wanted in my app and get all the bugs out and get it working exactly how I want it to. It frustrated me, actually that I've spent a lot of time. I spent about 18 months, about $100,000 of my own money, just down the drain trying to build this app, trying to get it out the door and I wasn't able to finish it off. At the end of the day, I really started to question like, "Was it me? Do I just suck? Am I not a great developer? Is that the problem?" You really have to start asking ourselves hard questions like that when you put everything you have into something and it doesn't work out and then React came along and I was like, "I was just using the wrong tools." The tools were the problem. It wasn't me. There is a different way to think about this about building stuff. I thought the way that I was doing it was the only way that existed to build things but it turns out, it's not. I used to not understand what were people talking about when they were talking about functional programming or what do they talk about when they're talking about composition, solving problems with composition, instead of inheritance. I didn't even understand what that meant. But all I knew is really smart people would say stuff like that and I was like, "What are you talking about?" Now I feel like, because of React, I've gotten it. Again, to get back to your question, Taras, React made a lot of the stuff that we used to worry about, that we used to think about. It made it easy and so, we could build amazing things now a lot more easily. But now, for the last couple of years, I feel like we've just been sort of rehashing a lot of the same stuff. I would really like to see us, as a community sort of tackle the next level problems. I think the React Suspense stuff is definitely getting there. That's a problem that I really don't see anybody else sort of address, which is how do I make it easy to deal with the fact that applications by their nature -- these networked applications -- are asynchronous. How do I deal with that in a declarative way? That's why I'm encouraged by that work because I do think that it's kind of one of the more forward thinking things right now that's going on in the React community but I would like to see us in general, sort of like get past talking about styling and get past talking about service and rendering and move on. ROBERT: Right and I'm going to assume you have this really unique experience since you do a lot of trainings. I follow you on Twitter and it's just always talking about where you're flying to and who you're training, so the things have to come up there where you see these things in pattern over pattern over pattern. MICHAEL: Yeah, exactly. I see like a lot of and this again, gets back to Taras's question. I train a lot of people who are very new to React and I see people who are new to React, they really don't have a ton to go on besides just like blog posts and Medium pieces and podcast like this one and who am I? A lot of people in their React community don't even know or care who I am. It's like, "There's a guy out here who are saying stuff about React. Maybe I should listen to him. Maybe I should listen to somebody else," and it's confusing. It's confusing, I feel like, for people who are just getting into React. It's confusing. Like I said, experimentation is good but I guess, I wish the experience of coming to React was a little bit more like the experience of coming to Vue because -- ROBERT: Oh, like it was a beaten path. MICHAEL: Exactly. I don't actually think there's not a whole lot about the Vue technology that is compelling to me and let's not dig on it. It's just -- ROBERT: A personal preference? MICHAEL: Yeah, exactly, just preference but I do think the thing that is very compelling about the Vue experience is just the cohesiveness of it all. You go to Vue and there's like a way to do server-side rendering with data fetching and styling and -- ROBERT: They have a CLI and -- MICHAEL: And a CLI and a component and all these stuff. Yeah, and a router built right in. Anyway, I think that's a way that the React community could improve. I don't know if that'll ever happen because the React community at heart is artists and hackers and those people are traditionally very reluctant to be corralled and like I said, it's a blessing but I think from the perspective of people who are new to the community, it does tend to cause some confusion. TARAS: I want to add to what Michael is saying. What's interesting and I'm sure Michael sees this in his training but the kind of people that use React is very diverse. There's this kind of original group or there's kind of mentality that prompted the early adopters of React and now, we're seeing these companies that are traditionally enterprise-y with MSE backgrounds and coming into React. It's really interesting to see all this kind of worlds collide and then see what's happening as a result. It's definitely interesting on what's going on now. ROBERT: This is an excellent stopping point. I really agree. I come from Ember background so I would like to see a little bit more convention but I think it's okay. It would be nice to see that. Thank you, Michael for coming on. Is there anything that you would like to give a quick plug for and where people can reach you? MICHAEL: If you want to support the work we're doing, you can sign up for one of our upcoming workshops. We've got one right now on ReactTraining.com. We've got a workshop coming up, actually in my hometown of Carlsbad. You can come out here in July and hang out with us and do some React training. We got a really awesome host here who's right here in town. We're doing some React training workshops on July 25th through 27th, I think and then, other ways that you can support what we're doing is publish in all my open source code at GitHub.com/ReactTraining or you can follow us on Twitter at Twitter.com/ReactTraining or at my personal account at @MJackson -- Michael Jackson. ROBERT: Awesome. I really appreciate you taking the time to come on. MICHAEL: Yeah, thank you so much. It's been a pleasure. ROBERT: This is a great talk. Cool. Thank you everybody for listening. We are The Frontside and we build UI that you can stick your feature on. If you would like to give us feedback, you can always reach out to us on Twitter at @TheFrontside or you could email us at Contact@Frontside.io. We're always looking for any new topics or things you would like to talk about or things that are interesting to you. As always, thank you Mandy for producing our podcast. It's @therubyrep on Twitter and on June 28th, we're going to have Chris Martin on to discuss blockchain development.

In this episode, Harry Kaplanian of EBSCO joins the show to talk about the FOLIO project: a community collaboration to develop an open source Library Services Platform (LSP) designed for innovation. He discusses the why behind the the decision made to embark on this project, the benefits it brings to the market, and what makes it different from current offerings and other open source projects. Harry also shares where he sees FOLIO going in the future, challenging and unexpected outcomes of making the choice to go open source, and advice for other businesses that are considering embarking on a similar journey today. Do you have opinions on this show? Want to hear about a specific topic in the future? Reach out to us at contact@frontside.io or on Twitter at @thefrontside. This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. TRANSCRIPT CHARLES: Hello everybody and welcome to The Frontside Podcast Episode 102. My name is Charles Lowell. I'm a founder and developer here at The Frontside and I'm going to be today's host. So today, we're going to be talking about the business of open source with Harry Kaplanian. Harry's someone that we've had the pleasure of working with over the past year and collaborating with on a very unique project, which we are going to get into later. With me is Robert DeLuca. Hey, Robert. ROBERT: Hello. CHARLES: And before we start out today, we just wanted to take care of some quick news. Just so you know, we released BigTest React. This is actually something that grew out of the work that we've been doing with Harry. It's a set of React helpers to acceptance test your React applications so that you can make sure that you know that your application is actually working on actual browsers in actual life. So, that's exciting. You can go check it out at BigTestJS.io. And with that said, let's get on with the project. So first if all, welcome, Harry. HARRY: Thank you. CHARLES: I teased this in the intro, but we'll go ahead and say it outright. The project that we've been working on with you and with EBSCO is a project called FOLIO. And it's one of the more unique projects we've ever had the pleasure of working on. So, maybe you could just give us a brief overview of what exactly FOLIO is and how it came to be. HARRY: Sure. So, FOLIO I think can best be described as an open source platform for services. Our focus has always been the library market and so, currently this platform exists in the library space. That said, there's nothing about this platform that is library-specific. It can actually be used anywhere, anytime, for anything. The platform itself, all communication occurs basically via HTTP. It's microservice-based. It's language-agnostic. And it is really the core of what we're building. However of course, a platform on its own is not very useful. And at least for right now, libraries are overall expecting to get to a point in terms of features, functionality, apps, and services that are being built. So FOLIO can actually take over the enterprise-wide day-to-day operations of what it takes to actually run a library. CHARLES: And these are libraries like university libraries, the Library of Congress, the library at Alexandria. I mean really, when you think of library, like library [3:00 inaudible]. HARRY: Yes. However, we decided to focus on a particular market, at least initially or I guess a subset of the library market. And so, that subset we've actually chosen to focus on is the academic library market. And so, really mainly colleges and universities all around the world. That said, there is no reason why others can't use this software. And it is all open source. So, if any changes are ever needed, modifications or additional applications are needed to support those additional markets, that can certainly happen at any time. CHARLES: Historically, it's a very complex problem, right? There is a lot that goes on at these libraries. But historically, the software that's driving it has not been open source. HARRY: That is correct. So, it's sort of an interesting history. And I guess you're leading into what I think was maybe your second question [Chuckles] which was: What are the circumstances or the thinking that lead to the creation of a project such as this? And really, libraries have been pretty much operating the same way, to be honest, for probably about the last 40 years. And I'm sure we either all know or have heard, most libraries are facing shrinking budgets. They're really challenged in terms of space and storage requirements. And just in general, users have changed dramatically in the last 40 years, the last 20 years, maybe even the last 10 years. Most people don't really ever expect to walk into a library and actually conduct research anymore. They expect that to happen elsewhere. And so, in many ways, they're really facing what amounts to Google and others, because that's where people expect to go. The vendors that typically created the systems and software that supported these libraries, the day-to-day operations over the years, many of them have consolidated. And there are just actually very few options left. These are really old, monolithic systems. They were built initially with the best intent. And for the first couple of years, many features and functionalities were added. But over time, these systems grew to be so large. There really was never any one person anymore that fully understood it. They find themselves in situations where if they add a little feature here or modify something there, it breaks something somewhere else within the system. And so really, they've gotten to a point where libraries have generated lists of features and functionality that they need that are frankly years long. And the vendors just can't deliver any of it. They just don't have the ability anymore. The systems are too old. In addition, really they're sort of building these walled gardens where they really try to take over all the day-to-day operations of the library but are actually starting to try and expand out in other areas of the university as well. And as they do this, they really tend to lock the system down. They're not really open to integration with other systems or other vendors. And they really see it as they are the one company you should go to that can completely manage and operate everything that goes on in your library. And when we talk about innovation in these systems, really they've been operating the same way, as I mentioned earlier, for the last 40 years. Really, the only major, major innovations is there's maybe one company now that's offering these services on the cloud. One size fits all. And the real issue there is, all of these libraries we talked to, all of them see themselves as being unique in some certain and specific way or sometimes multiple ways. All of them believe they're bringing something special to the world in terms of knowledge, research, understanding and learning. And they tend to cater or specialize often to the institution they're a part of. And so for example, if a library belongs to a medical school, of course their collections are tailored that way. A lot of their workflows and operations tailor to students that conduct research in that field. And oftentimes, even to a hospital that may be associated as well. And this will oftentimes be very different from what might exist for example in an engineering library. And so, for most of them, they look around at what's available to them. Not only has nothing changed, not only has there been no innovation, but on top of it, they're forced to choose essentially a single system. And that's not really getting them what they want or where they want to be. So I, as well as some others, spent quite a bit of time looking around at the market and trying to assess and understand all of this. And one of the things we felt was, as we look at all of this, really we're living in a world, these libraries are living in a world of closed systems, lack of transparency, lack of change, lack of innovation. How do we change that? How do we spark that innovation? And it seemed like what we really need is some sort of an ecosystem, a sustainable ecosystem or really, openness and transparency. And it seems like the only way to get that really is open source. And in fact, we actually conducted a survey and out of that survey, 100% of the libraries we spoke to really believe in open source. And when you think about it… ROBERT: Whoa. HARRY: Open source really in many ways aligns itself really nicely with the mission of the library, right? The library is there to not only promote learning but to make sure information and knowledge that normally you may not have access to is accessible to everyone. They're really leveling the playing field. And that really is a really nice analogy for what open source is doing in the marketplace as well. And so, this seemed like really, and at least, an initial ideal path to head down. And so, we did. CHARLES: Yeah. And so, you did. And I'm curious to explore more about that decisions. Because – now this is a little bit of a dated quote – but Steve Balmer, when he was the head of Microsoft, called open source communistic and un-American, then how do you reconcile that? Because obviously you represent a business that ultimately needs to be profitable and is engaging in this in order to make a profit. And so, where does the balance lie and how do you reconcile that? And obviously you all perceive this as a strong business move. And so, yeah, how do you balance those two concerns? HARRY: I think from our perspective, we looked at it as if a single vendor or maybe a group of two or three vendors end up locking up the entire market with systems that operate the entire library that are not open and they're not willing to integrate with other systems and services that exist out there, in essence that's an attack on what is the basis of our business as a company. We provide services and we provide content that libraries use and ingest and make available to their users. And essentially, it looks like they're trying to lock us out. And how does that really help in terms of creating an open and fair marketplace where people get to make choice? It doesn't. And so, what can we do to disrupt that? And building a system that allows everyone to play at least puts us on a level playing field where if an organization or a library chooses to adopt it, we're now back in a very competitive situation where really, the vendor or the organization or the individual that provides the services that best fit that library's needs wins. And that's what we really want to happen. And so, that's in essence how we win. We believe we offer better services than most others in the industry, if not everyone else in the industry. We think there's a sizable percentage of the market that truly believes that. And we just need to make sure that the mechanisms or the systems are in place that allow us to compete in that manner. And in the end, if what we end up with is a platform that allows 3, 5, 10, or 20 companies to actually compete in a reasonable manner fairly, really it's the customer in the end that wins. It's these libraries and it's the users and everyone else conducting research. ROBERT: So, I'm still kind of reeling from 100% of libraries said that they would like to have an open source platform. How many did you survey? HARRY: So to be truthful, it wasn't a large number. ROBERT: That's still insane to me. [Laughs] HARRY: Right. We focused on, however, surveying different segments of I guess really representation within the library. So, the people that do the work, the middle managers, and then really, the folks at the director level that are really responsible for running and organizing a library. ROBERT: The ones that will experience that pain of being locked in, right? HARRY: Right, right. And trying to get decision-makers at many different levels. And we tried to also cover both very small libraries to very large, enormous libraries as well. And all of them saw a benefit. One of the pieces where things sort of maybe went in the other direction as far as the statistics are concerned, if you ask them though “How many are willing to adopt open source?” the numbers aren't quite the same. In fact, they're rather dramatic. And so, at that point it's roughly about 33% said they'd be willing to adopt. CHARLES: So, why is that? HARRY: So, we asked. And interesting question. And what most of them came back and stated was, “Well, we love the idea of open source. We fully support it. We're willing to do what we can to help it and make sure this happens. However, we ourselves don't believe we've got the staffing, the resourcing, or the knowledge to host this ourselves.” And so, this leads to another really interesting thought or idea as far as open source is concerned, is so this really implies that there really needs to be sort of an ecosystem of organizations that are providing these types of services. And to be honest, the faster we can make that happen, the better off the market is. Libraries are now getting to choose vendors that they get to get their services from, which is rather interesting. Because for the first time, not only do they get to choose the vendor but if they're not happy with the vendor, they should be able to switch vendors but keep the software and the data the same. Today, in the library market, if a library chooses to switch vendors, that means essentially they're completely starting over in their library. They completely have to migrate to a fully new system which means enormous amounts of training, enormous amounts of data transfer, and all the pain that goes along with it. I had a librarian tell me once, “The next time we migrate to a fully new system, I either want to die or retire.” I just want [14:30 inaudible]. ROBERT: Man, that sounds painful. HARRY: That's how painful it was. CHARLES: Yeah, wow. Because the complexity intrinsic to these systems, it does, it boggles the mind sometimes. Just even the little slices of the ecosystem in which we're operating, there's just a lot at play there. ROBERT: Yeah. Like trying to work with other vendors and be agnostic so anybody can adopt these things that we're building. And since we had that vendor lock-in, it's hard for you to adapt the two systems, because they were built in complete silos. So, trying to merge that is an interesting challenge. HARRY: And I think everything I'm saying here, I'm talking about it from the library perspective. But if you look at other industries, I don't think it's really any different. If you look at any of the enterprise-wide management systems that are used in corporate organizations today, most of these corporate entities will stick with the system they chose for a very long time. Because it's incredibly painful, it's disruptive, and it's expensive to switch from one system to another. And so, the idea of you're not happy with a vendor and hey, I can go to somewhere else and maybe keep the system I have in place is really kind of interesting. And even better, if the vendor's not able to provide some of those features that I need but for the most part I like the system, wouldn't it be great if I can hire someone to do that work and to have those pieces built for me? And those are some of the things that open source offers. CHARLES: Right, right. So now, I understand that for example to kind of put a concrete bow on this, [Chuckles] to move away from abstraction of what it means and provide an example, so we have this open source platform, this FOLIO platform. Which if I'm a library, I could in-house, I could either use it on premises on my own servers or I could run them on AWS or I could run them on Google Cloud – there are all these different deployment options and some libraries are doing that, but that's something that they might not want to do. So, I understand that EBSCO is actually going to be for example one of the services that y'all are going to be offering around FOLIO, is actually hosting the platform so that the overhead of managing all of the servers and provisioning accounts and doing all that stuff is going to be taken care of for you. So, I can just sign up. HARRY: Yes. So, that really fed into the results of the survey. For those libraries or organizations that felt they couldn't do it themselves, where would they go? And who could provide those services? So, we're absolutely going into the business of providing these services for libraries that choose to. But at the same time, we're really strongly encouraging other vendors and organizations to provide these services as well. Because for instance, you may represent a series of libraries in let's say Hungary. You may have a vendor there you enjoy working with in the past. It's provided great services for you. There really ought to be no reason why you should not choose that vendor to provide those services for you. And again, it's an open system. There's nothing there that's excluding me and the organization I work for to provide additional services, to provide content or what have you. It's an open system. And so, we should strongly encourage that. And the reality is, if we're going to have a disruption in the market, it seems we can either have one organization or one company trying to usher this disruption along. But what if we could get 10? Or what if we could get 20? Or more companies out there ushering this disruption along simultaneously worldwide. It feels like we've got a lot more of a disruption happening in that case. And so, that's what we're strongly encouraging. CHARLES: Right. But at the same time, I can see it from if I were considering making a play like this, one of the things that would worry me is, “Okay, so we're going to disrupt the market by allowing a large tent where lots of vendors could compete.” But then, I immediately experience fear of, “Well then, how do I differentiate myself?” Because ultimately I want to be competitive. How do I make it so that I'm not a commodity in this new market? HARRY: So really, I think that's key for any vendor that chooses to compete in a market like this. Do you believe as an organization you have not only what's needed to compete but those key differentiators that at least a certain segment of the market would be interested in you and the services you offer? We believe we do. Another company that just recently announced that they're going to be supporting FOLIO and providing services around it is a company called ByWater Solutions. They've actually been in another segment of the library market for many years providing services to libraries as well. Longstanding, actually, with open source codebases. And so, this very much appeals to them. They believe… ROBERT: Oh, interesting. HARRY: They have a niche that they can provide and they are going to be doing that. And we strongly encourage it. What's again great about it is if they set up a library with FOLIO, they're not building the walled garden. They're building or providing the open platform that we can connect our services to. So in many ways, yeah there are some areas we're going to compete. But there are actually many areas that we're also going to be complementary. And I think that's what's really, really interesting to us. CHARLES: So, it sounds like you're also counting on just by having the ecosystem in place and having this sea as it were filling the ocean to enable trade, one of the bets too is that you're going to open lines of business that weren't even possible before. HARRY: Correct. So, another way to look at it – we've got our iPhones or Android devices. Apple built a platform is really what that phone is. When it was released, you could make voice calls. You could do some simple text messaging, some basic email, and I'm sure there are a couple other of things in there that I've forgotten. I do not believe that Apple had any idea that they'd actually be able to create a marketplace that had hundreds of thousands of people and vendors providing features and functionality for that platform. I am pretty sure no one at Apple had ever imagined the guitar tuner would be released. I don't know, a compass app, flashlight apps, whatever. You name it. And not only that, as a user of that platform, you're not happy with the standard email app? Well you know what? I can get a different one. In fact, I've got a choice of many and I can find the one that suits me best. And that's fine. It's an open marketplace. And so, with FOLIO today, we're in early stages. We're building out that platform. We're building out some of the basic functionality for that library that they need to operate as a library. But once this gets released, it's at that point when we believe the more interesting things are going to happen where we build a cataloging app for FOLIO. Well, we know already that there's other organizations that are starting to look at and think about, “You know what? We've got a whole other way or a whole other interest in terms of how we'd like to support those workflows on the FOLIO platform.” Or even better, librarians starting to think about, “Hey, if I've got this open platform just like Apple does, this means I now have the option to build features and functionality that take me where I want to be five years from now, 10 years from now. And I've never had that ability on the existing systems that we use today.” And so, one of the other I think advantages or benefits that something like FOLIO or a platform like this brings to the market is the ability to create that marketplace very much like Apple has done and Google has done with Android as well. ROBERT: So, we're pretty far in, but you mentioned FOLIO a couple of times. Do we want to unpack what FOLIO actually means? HARRY: Yes. FOLIO actually stands for the Future Of Libraries Is Open. ROBERT: Right there in the name. I love it. HARRY: Absolutely. CHARLES: So, we've talked about this platform. We've talked about this marketplace. We've talked about the business incentive of why you would want to do this, why it makes a lot of sense. Clearly, this takes an enormous investment. So, we think, I think, a lot of people think of open source as a bunch of wild developers running around. ROBERT: Pushing commits wherever they want. CHARLES: Pushing out code and doing stuff. And then it kind of organically grows into something that someone then picks up. ROBERT: And then you burn out. CHARLES: Yeah. [Chuckles] But that's I think a lot of people's experience with open source. But an initiative like this takes a staggering investment both in terms of capital resources but also in terms of will, in terms of management, and putting all these enormous forces in motion. There's developing the software, developing the awareness, and I think right now there are hundreds if not thousands of people working on this right now. How do you even go from zero to 6,000 like that? And I'm not talking about people. I'm talking about velocity. [Laughs] HARRY: Right. Early on, when we were conducting research, market research, one of the things we did, we spent some time looking at what we believed were successful open source projects. And I think what was interesting was in many cases it took a single individual or a company to actually create what amounts to that first piece, that first core building block that others could start to expand upon. And very often, they literally just created it themselves, made it available out there as open source, and basically told the world, “Here. Have at it. Have fun. Do with this whatever you wish.” So, we actually thought we'd like to do that. However, building a system on this scale is not that easy. Understanding the operations of a library day-to-day is not that simple. We need help. And so, what we decided early on was: could we kick off and start this project while at the same time [25:42 inaudible] that community as early as possible? To get people excited, interested, and to get this project in a place where we're not the only contributors, where there are many others contributing as well. And when I say ‘contribute' I don't necessarily mean software developers. But that is certainly one aspect of it. But one of the key pieces of building software is gaining access to subject matter experts as well. And it's absolutely key and critical. And so, our goal of course was to build this community, have that community start to provide that in-depth knowledge, those subject matter experts that we need so we can determine what it is we need to build, how we need to build it, and really go from there. And by including those people as early as possible, I think one of the things we find that happens with this project which is really sort of incredible is the excitement that starts to build. The word of mouth advertising, marketing, as librarians, libraries, vendors and individuals start to talk to others and really spread the word about this project. In many ways, it starts to compound or snowball and build on itself. But that was really challenging. And it was really actually kind of slow going in the beginning. Because we started with nothing. And one of the issues you face is, “Well, you want me to contribute my time to this project, yet I see nothing.” Right now it's just a dream. And so, one of those early, early issues was, “Can we build a team small enough, focused enough, where we can start to build some of the basic, basic, basic core pieces, to really prove to the world that this project is real? This project is actually moving forward and this project is actually delivering.” And now that we're in a state where people can actually see working software, the excitement is just starting to expand and compound rapidly. And we see it everywhere. CHARLES: So, do you find libraries or people in the target market, there's that key point where they start to feel empowered? It's real and then the thing that I think that you're going for is to have the realization dawn that not only is it real, I can put my hands on the wheel and I could control my destiny. And I can contribute now because there's this critical mass. ROBERT: You have the power to make change, which never existed in the library world before. HARRY: Or at least, not in this specific area. Because to be truthful, I think libraries as a whole believe they're making change around the world all the time. But it's really related to content. But this is actually making changes in the actual operations of the library. And they are actually empowered to get involved, to contribute, and to help us get this done. And it seems to really resonate with folks. Because it's something they haven't been able to do previously. And it seems genuinely exciting. And we announced this really, well the market learned that this was happening I'd say a little over two years ago. Maybe more like two and a half years ago. And it was sort of a gentle drip in terms of a roll out. People started to learn because we started talking to them. There was no real active marketing going on. But of course, those people started to talk to other people. And so, it happened. And when we were able to provide those first demonstrations, no matter how rudimentary it was, that's when things seemed to really kick into a much higher gear where people started talking to each other. Librarians started talking to each other and say, “Hey, this is real. This is exciting.” ROBERT: It's not vaporware. HARRY: And this is happening. Right. It's not vaporware. And we need to get involved. And so, they have. CHARLES: Yeah. And the energy and involvement just in the course of the time that we've been a part of the project is apparent. Every kind of gathering is larger and the diversity of topics that are discussed is increasing. And yeah, the conversations have burned from “What can we do with it?” to the last time the project got together as a group, it had much more of a conference-y feel. And the topics being discussed were like, “How do we actually deploy this to our real systems?” Which has been fantastic to see. And so, just to actually feel the traction is just, it's so gratifying. So, I guess one of the questions that is on my mind is – so you mentioned that you had been at this for about two and a half years. What is in your mind the most pleasant, unexpected outcome of this project that you didn't foresee two and a half years ago that you're experiencing now? HARRY: So, I think one of the one's that's surprising is I've never actually personally been involved in an open source project before. And I think one of the fears constantly working for organizations that really have been closed source – in fact, I've been that my adult working life – and you sort of walk into this with some [trepidation]. Because oh my gosh, everything I'm doing, everything I say, everything that gets documented is going to be out in the open for everyone to see? Including competition and everyone else? And I have to say, one of the most interesting things, or one of my favorite pieces here, is: it is so freeing. It's a release. It's actually amazing to not care about what your competition or anyone is doing or thinking. And it's even more amazing whenever you've got a question and someone's out there asking you for access to documentation, you can simply point them to the website, to the URL and say, “It's there. It's all there. Anything you want is there. Go take a look.” I think that's pretty amazing. The other one of course are the relationships between many, many different people who I've just never really been – or I never would have had the chance to work with before. And really hearing all these different perspectives and points of view as far as what people think are right, correct, or what we should be doing – it's great to see. And it's great to see this wildfire word of mouth message that seems to be moving everywhere. We're hearing back from people not just in North America, not just from the EU, but from the Middle East, from South and Central America. There's really just people interested everywhere. And it's amazing to me, because I don't think I spoke to anyone over there. So, it's happening. And that's just an exciting thing to see. CHARLES: Yeah, yeah. ROBERT: It's really cool that open source can connect you with the rest of the world like that. It's just so powerful. HARRY: It does. And it's an amazing thing. ROBERT: And the amount of collaboration that happens, I've never experienced it in any other way. It's really cool. [Chuckles] I don't know how else to put it. It's just, it's almost mind-blowing that you are able to across timezones, across the world, collaborate with somebody on something that you both feel passionate about and you're pushing it forward in an open manner. HARRY: Right. And it's also amazing when you have these meetings and there's people you've been collaborating with for months if not a year or more. And getting to meet them face-to-face for the first time. That's really a pretty amazing experience. ROBERT: Yeah. And it's pretty awesome. At our last gathering that we had at WolfCon, I had the absolute pleasure of chatting up a bunch of different librarians. And hearing their experiences and what they're looking for out of an experience, it's just really cool to be able to talk to those people and see how they work and help build this into the platform that they're wanting to use. HARRY: Right, right. CHARLES: Yeah. I would say on the whole, there's been a lot of those wonderful, unexpected outcomes. Were there any that presented a particular challenge or wasn't quite so much a walk along the path of roses? HARRY: So, you know I think the biggest one is – I mentioned earlier we embarked on this project a little differently versus what we saw as normally successful open source projects, right? We did not have that core, beautiful, shiny object to unleash onto the world and let them know, “Hey, have at it.” And so, it wasn't so hard in terms of getting interest. But what's been hard is the amount of interest has generated enormous numbers of organizations that are interested in contributing to the project. Which also implies then that now, not only are they coming to us asking us, “Hey, how can we help? What can we do here?” but then there's a certain aspect, because this is a first version of, “How do you manage the building and construction of that very first version of this software project when the reality of the matter is, you're not really responsible for most everyone working on the project?” And how do you get everyone aligned and organized? How do you get everyone onto that same path along with those same milestones that we've all agreed on? And even worse, how do you get agreement on those milestones so we can all walk that same path and deliver that first version? And so, I get asked sometimes by folks that are involved in other open source projects, “How do you handle this situation? Or how do you manage all the work that goes on?” And when we tell them, it's kind of surprising because it's not what they're used to seeing. And a lot of it is because of the way we started, the interest that was generated early on, and the fact that we did not have an initial version. And I don't want that to sound like, “Oh, this is a huge problem,” or, “If I were to do it again, I would not do it this way again.” I in fact would do it this way again. I think there are a lot of benefits and it's been a great and interesting way to go. It's just the management aspects of it are definitely challenging. And I think more than maybe we had initially anticipated. Going into this next time, though, I definitely know what I'm facing and I'll be prepared. CHARLES: Yeah. So with that in mind, given that you've been through this, do you have any advice to offer someone who might be considering embarking on a similar journey? And undertaking a similar Herculean task. HARRY: Well, I think you'll find it's harder than you think. And be sure to plan for that. But I guess at the same time, I think my biggest advice or my best advice would be: you need to keep an open mind. And when I mean an open mind, you need to be open to other thoughts and ideas. And I think you need to put yourself in the perspective of what if you were one of these other people that are working on the project or trying to contribute to this project? What are the things that you're doing or rather the things that you're doing now, the way you're acting, the way you're reacting, how does that look to others on the project? Because you're not the only one contributing. You're not the only organization contributing. And are the things you're doing reflecting badly in terms of others' perception or optics as to what this looks like as an open source project? And I think for me and others, there was a huge learning experience. Because your first thought is, “I'm going to tackle this like any other software project,” and it's not. It's not like that at all. CHARLES: I think that's… ROBERT: That makes sense. CHARLES: Yes, that makes sense. That's excellent. That's excellent advice. And I think that's one of the things that makes open source so powerful, is because there's that aspect just baked into it from the get go, you have to be mindful of a bunch of different perspectives. And that ultimately results in a solution that's going to be workable for a bunch of different perspectives, that's going to be flexible to accommodate for all the different use cases that all the different participants in the community might bring to the table. HARRY: And you have to be very mindful of how your actions are perceived. Because normally, your actions are perceived by the company you work for, what tends to have a particular culture. But on an open source project, you're actually dealing with many different cultures. CHARLES: Yes. HARRY: And so, that's a tight line that you have to walk. CHARLES: That was a powerful note to end on. Thank you everybody for listening. We are The Frontside and we build software that you can stake your future on. If you want to get in touch with us, continue the conversation, you can reach out to us on Twitter at @TheFrontside or drop us an email at contact@frontside.io. Thank you so much Harry, for being on the show today and talking to us about FOLIO. HARRY: And thank you very much for having me. This has been great. CHARLES: And if you want to, if there's something that you want to learn about or a topic that you want to discuss, please, please let us know. We always are open for your feedback, especially when it comes for things that you would want to hear. And if you want to get involved or learn more about the FOLIO platform, you can just head on over to FOLIO.org. There are resources for librarians, developers, and anyone who is curious about becoming a community member. Thanks as always to Mandy, our producer. And we've got a really great podcast coming up on the 14th of June. We're going to have Michael Jackson on the podcast to talk about separations of concerns in React. So again, thank you Robert. Thank you Harry. And we'll see you all next time.

In this episode of The Frontside Podcast, panelists Charles Lowell, Rob DeLuca, and Sam Keathley, discuss how much the distinction between frontend, backend, and fullstack developers matter in both personal and professional senses. The differences in mindset between these categories are also discussed, for example, how does a fullstack developer solve a problem vs how a backend or frontend developer? And also, how clear (or fuzzy) is the line that separates them? What are the skills a frontend or backend developer needs to consider themselves fully fullstack? And finally, is there any sort of tribal separation between the factions? Do you have opinions on this show? Want to hear about a specific topic in the future? Reach out to us at contact@frontside.io or on Twitter at @thefrontside. This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. TRANSCRIPT CHARLES: Hello everybody and welcome to The Frontside Podcast Episode 101. My name is Charles Lowell. I'm a developer here at The Frontside and I'll be hosting today's episode. Today we're going to be talking about the different developer profiles that are out there. You might have heard mention of them on Stack Overflow, on job boards, on the Twitterverse. Things like frontend, backend, full stack, half stack, short stack. [Chuckles] Things like that. With me to talk about these are Rob and Sam. Rob is our director of open source here at Frontside and Sam is a software engineer here at Frontside. So, hey y'all. SAM: Hey. ROB: Hey yo. CHARLES: Alright. So, before we get into the meat of the topic, there are a few announcements that we wanted to make. First and foremost, this is some really exciting news. This week, Taras Mankovski officially joined the team at The Frontside. And we are really, really excited about that. [Cheers] ROB: That's exciting. CHARLES: Yeah. It's super exciting. Not only is Taras an exceptional engineer, he's got amazing code skills. He's extremely empathetic and a great person to work with. He's great working with clients and he's also fantastic at really working with developers who are climbing that ladder and helping them level up. So, he's been involved in mentorship literally since I've known him. I think that was the first time that we ever came into contact with him was through his mentorship work in the Ember community. And we've been collaborating with him for years on several open source projects. And so, we just are so excited and know that he's going to come on and do some amazing things by helping us be better developers, helping us be better community emissaries. And so, lots of good stuff coming out of that. ROB: Absolutely. CHARLES: Yeah, yeah. No, that's really exciting. So, definitely wanted to throw that out there. ROB: Too bad he didn't join a little bit earlier. We would have had a good time with him on this podcast. CHARLES: I know. Well he's actually, I think he's been on the podcast twice. ROB: [Laughs] That's true. CHARLES: But yeah, there's definitely a couple of topics where his input would have been absolutely critical. I think it would have been great to have him along when we were trying to run our own apprenticeship programs. Those would have worked out I think a lot better. Well, not to say that the outcomes haven't been stellar in some cases. But I think having his expertise would have made it a lot smoother. That said, before we jump in then, just wanted to let everyone know that as always, if you want to work with us you can get to us at contact@frontside.io or send a shout-out on Twitter @TheFrontside. Okay. Y'all, let's jump into it. Like what is even a stack? ROB: [Laughs] So, do you want to kind of set up where this came from, Charles? I think it came from you and I going back and forth on Twitter. And I was like, “You know what? We need to talk about this on the podcast.” CHARLES: Yeah. I think I was actually on vacation. ROB: [Laughs] That's right. You were trying to ski. CHARLES: [Laughs] I was on the ski lift. And I saw you tweeting and I was like, “No, no, no. I got to respond to that.” Probably because there was maybe a little bit of disagreement but also because I just needed something to do. Looking out the majestic scenery of the Rocky Mountains wasn't enough. So yeah, what's the backstory there? ROB: So, I can't remember what I was actually tweeting about. But I kind of feel like there is a distinction between a frontend developer, a backend developer, and at this point I don't know if a full stack developer truly exists. I mean, it does. But it's used more often than it really implies what they're doing. Usually when you hear of a full stack developer, it's like, “Man, I can sling some code on the backend and I can do a little bit of architecture stuff on the frontend. But CSS, keep that away from me.” And in my opinion, if you're a full stack developer, you need to sling CSS, too. CHARLES: Really? ROB: Yeah. CHARLES: No, no. I certainly agree. Because I was going to ask, is it fair to say that full stack developer is code for like, startup developer? ROB: Yeah. That's a good – like a utilitarian, like a jack of all trades but not a master of any? CHARLES: Right, exactly. ROB: I can take that. So, I guess the way we can start to dig into this one is like, how much distinction is there between a frontend, a backend, and a full stack developer? And does that matter in a professional sense? CHARLES: Right. And so, I think my take was that the skillsets that you need in both places are actually much more convergent than you think, than people might think. In other words, the skills that you utilize on the backend actually translate very well to the frontend. I think my stance is that what makes a frontend developer and a backend developer is more the problem sets that you gravitate towards naturally, the things that interest you. I actually was a backend developer. And now I consider myself to be primarily a frontend developer. But the skills that I had developed writing backend systems in Java translated shockingly well into frontend development with JavaScript. But it was working on the portion of the program that interfaced with the human was, that was the thing that fascinated me. And so, it's what I gravitated towards. And so, I guess in my [6:15 inaudible] or the way that I have been thinking about it is that what makes a frontend developer and a backend developer is more what gravitational pulls you respond to, rather than one particular skillset. But I know that you've got a perhaps more refined take on that, or a different take. ROB: Yeah. Before I do, I want to know what Sam's thoughts are on this. This would be a good one. SAM: Where I come from in the understanding of frontend, backend, and full stack is I – background on me: I did a bootcamp before working here. So, I did a development bootcamp where they were really excited to tell you at the end of it that, “Oh, you're a full stack developer because we've introduced you to everything. Like, touched on every sort of little topic. You're full stack. Tell everyone you're full stack.” And it's like, “Well, you're not really a specialist in any of the sense.” So, what makes a difference to me, I'm going to specifically talk about full stack, but it's like before you can say you're full stack, having knowledge of frontend and at backend, you need to have expertise in at least one of them first, or understanding of that mindset first. Like if I say that I like frontend work more then I need to understand frontend work more before I can say that I understand backend work, you know? ROB: Yeah. CHARLES: So, do you feel like the bootcamp was doing you a disservice? Or do you think they got… SAM: I don't think they did a disservice at all. So, they did a really good job. So, bootcamps are really for introducing you to everything. And then whatever you take from that is your own. So, they introduce you to all the little things like database working and then JavaScript, Ruby, whatever it is. And then it's your job to figure out in those introductions, what you resonate most with. So, I think they did a really good job in that. I think that they shouldn't tell you what you are, like tell you that because they touched on whatever it was, that you're full stack. CHARLES: Right. SAM: So, I don't think they did a disservice at all. I learned a lot. But I wouldn't say I'm full stack at all. CHARLES: So, if you feel as though you're not full stack, what do you feel that you are? SAM: Definitely frontend. CHARLES: Definitely frontend. SAM: I gravitate more towards frontend. ROB: Why would you feel that way? Is it because you're more pulled towards the UI of things? Or is that where you feel like you've gotten most of your experience now? SAM: Actually a little bit of both. I tend to like the UI of things a little bit more. It's interesting. Whenever I was going to the bootcamp, I thought I was going to be more backend because I thought I really like working with the behind the scenes sort of. You don't see this work but you know it's there. But I definitely like the UI of frontend a lot more. Just personal. ROB: I see that. SAM: And it's where my skillset is, because this is all frontend work that I'm doing here at The Frontside. So, it's what I know most at this point. CHARLES: Yeah. I'm curious too. Isn't it true that there's also a conception in the wider tech world that frontend is limited to CSS and HTML? ROB: Ooh. Yeah, that could be. CHARLES: Like when you see job posting for frontend developers, typically it's like, we want someone… ROB: Yeah. It's kind of morphed from – JavaScript developer is now a single-page app developer instead of frontend. Yeah. Yeah, I see what you're saying. It's now, it could be considered as HTML and CSS and not JavaScript anymore. [Chuckles] CHARLES: Which is so bizarre to me. SAM: [9:42 inaudible] a lot of places when a frontend developer who is also a UX designer. Someone who has the knowledge of designing things from the ground up and then only working in things that look good, rather than logic. ROB: Yeah. This can be probably an entirely other topic. But that's like the frontend of the frontend and the backend of the frontend, is like [10:03 inaudible] call it. Because there are [10:06 inaudible] developers. And single-page apps have gotten so complex these days, between your data layer and your testing and managing state on the frontend. There is a backend, an architecture, that you need to consider when you're developing a single-page app. And I've worked on teams where there are developers that are working on the frontend but they still don't like CSS. They don't do CSS things and they don't consider layout or any design. They just take the ticket and they get the data going. Get the data from the server, request it, display it on the page. And then another person would come through and style it up to the spec or the mock. But yeah, that's interesting. So for my take on this, and where this actually really came from, is we were going through some hiring. And we were trying to figure out how we can place somebody or hire someone that can be effective in the role that we immediately needed. And all of our immediate needs were in frontend-specific things. And we can actually talk about if they're frontend-frontend or frontend of the backend. But where this kind of came from or was born was: if we're hiring somebody that needs to fill a frontend role and they're primarily a backend developer, there are gotchas and things you have to know to be an effective frontend developer. Like quirky browser things that happen or like how to set up a Babel transpilation setup. And what are the gotchas here? Kind of the history of the frontend web, because there's a lot of things that are there that can cause a lot of headache if you don't know the backstory. And I know from your perspective Charles, you think – and this is isn't wrong – but you think that if somebody has a really talented backend developer, it's really just concepts that you can apply to the frontend. And I agree with that. CHARLES: Right. But you have to want to do it. [Laughs] ROB: Yes. That is also a key. CHARLES: You have to be gripped by the problem set of the frontend. You want it to inspire you to leverage those solutions. ROB: Yeah. So, would you say there's a different mindset between a frontend, backend developer, and full stack? Does a full stack developer solve a problem that's different from how a backend developer or a frontend developer would? CHARLES: That's a subtle question and probably one that might offend people, my answer to it. So, I'll go ahead and answer it. [Chuckles] ROB: Hashtag [12:23 inaudible] CHARLES: Actually, maybe not. Just in my experience historically – I want to underline the fact historically – I think that it is people who gravitate towards the outside-in mindset, in other words the very first thing they're thinking of is, “How is this going to feel to a person who's going to use this?” If that's your first question that you ask, then you probably are going to gravitate towards the frontend. Then in terms of the backend, I'd say historically – and like I said, I want to underline that, and I'll get to that later – I think it's people who are more drawn to: I want to attack the most complex problem that exists. Like distributing state over 10,000 nodes, managing huge deployment infrastructures that drive these massive websites that happen at this mind-boggling scale. And so, there you really are thinking computer to computer and, “What's the ideal interface for doing that?” And then on the frontend you're thinking more like human to computer, because that's where the interface lies. Now the reason I say… ROB: I feel so let down. Where's the controversy? CHARLES: Well, okay. Well, because I think that basically – I think the controversy is saying like people who are more naturally empathetic. I don't know. That would be the controversy thing. But I want to say historical. ROB: Yeah. I can see that. [Laughs] CHARLES: So, I want to say historical too, because I feel like one of the things that's been magical about the last two decades of software development is the dawning realization that no software you write exists in a vacuum and that it's all interconnected. And so, I think that I for example feel very strongly while I try to think about the user experience first. And the developer, I'm also thinking about developer experience first. The way that you want a system to feel is going to inform the design of the UX workflow and that's going to affect the architecture of your frontend. And if you're interfacing with it through different media, like websites, phone, I don't know, even text messages, Slack, what have you – that's going to inform then the next level of how your APIs are shaped, your public-facing APIs. Which then inform the structure of your internal deployments. So, recognizing that the decisions you make in one end of the stack and ripple through completely and that they're not what we hold as dear these concepts of separation of concerns and complete and total isolation of layers. That really is false. But what it does is it enables us to change the individual layers to more – ironically the reason you want to have separation of the layers is so that it more easily lets you change them provide a more integrated experience. So, I think that's a long way of saying that I think frontend developers are more aware of what's going on in the backend and that they might be drawn into the backend. And the same thing goes for backend developers, realizing that the decisions they make are affected by the frontend and affect the frontend. And so, I think there's been a dawning more of a collective responsibility for design, for operations, for developer experience. And I think it's great. ROB: Yeah. And to [think back off] that. So, I was kind of wondering where the controversy was, because that's kind of my exact answer, too. Where do you gravitate towards on? The difference of mindset is like, if you've got a problem and let's say you're a full stack developer, so whatever that manes, but if you're given a task where you have to implement the frontend and the backend, where you gravitate towards first I would say is how you would attack solving a problem. For me, I would immediately go to the frontend and see how a user would immediately interact with it and then work out that way. Mostly also because I like having a tight feedback loop. And the frontend provides that nicely. I can change something and immediately see the difference there. And in the backend you can spend a little bit of time, unless you're TDD which you should be – you can spend a little bit of time piecing together things and figuring out the architecture. And then you'll have something that you could show for. It's just nice to see UI get thrown on the page. And it's real and you click a button and something happens. That's a really nice feeling. And that's kind of where I gravitate towards. And if yeah, if I had to attack the problem, I'm going there first. I get the endorphins from it. [Chuckles] CHARLES: Right. So, if we want to add controversy to the other side, because you got to always be controversial, right? So, we said the really blunt horrible oversimplification is that people who are more naturally empathetic gravitate towards the frontend. You could also say that from the other perspective, people who are more internally validated will gravitate to the backend. Because if you need to get those endorphins from working on the frontend, basically what you're saying is, “I want to put it in front of people and get the feedback from those people and know whether it's good or bad.” Whereas you could say, “Actually, I don't need validation from other people. I've got this concept of this architecture that is going to be the ideal thing. I know it. I've got this vision and I'm going to chase it, regardless of what's going on.” I think you can more readily do that on the backend because you're not as open to the feedback of a whole bunch of different users. And like I said, I think those are gross oversimplifications. ROB: Yeah. I agree. And so, as a devil's advocate towards the backend developer that is less empathetic, I think actually some of the best backend developers I've ever worked with were the most empathetic people ever. Because they know that software is for humans. And humans are going to be consuming the API that they build. And they take that into consideration. CHARLES: Absolutely. And I actually think that empathy pervades good software development just wherever you find it. Because yeah, someone's going to be using your APIs whether it's software as a service or it's a library. If it's software as a service, it's got to be easy to work with. It's got to be performant. It's got to have a nice command line. And so, you have to be thinking at all times, “What is it like to use this software? What is it like to consume it? What is it like to link it into my library? What is it like to call it from a web client?” So yeah, I think you're absolutely right. I think it's actually one of the great virtues of great software developers. SAM: Yeah. So, something that I've always kind fo had a question with, especially coming from that bootcamp setting: is there any sort of tribal separation between what you would consider a frontend or a backend developer or even full stack? Like, “I'm better because I understand data layers than I understand how a button fits on a page,” that sort of thing. CHARLES: I would say absolutely yes. If you at the, just do a quick poll of the Twitterverse, it seems like people tend to hang out in little circles of similar interest. So, there's definitely a bunch of people that I follow that are always tweeting about backend stuff. ROB: Yeah. Twitter is ‘build your own echo chamber'. CHARLES: Yeah. And there are people who I follow that are tweeting nothing but mostly frontend – when I say frontend, I mean frontend of the frontend. Well basically, from CSS down to nothing deeper than React components. And then there's people who are talking primarily about the backend of the frontend. So yeah, I do think that people – there are clearly, there are different areas of the stack and I think that people do tribalize around them. So, the question is: Who are the border agents? Because always cross-border trade. Any time you have borders, there's an exchange of ideas and information and things that happen at those layers. And actually, one of the things I'm really curious about is: How does that happen? ROB: Yeah. You might have the best perspective on this because you did – you were using Java Swing back in the early 2000s building UI. And that's like backend things. And then you had a lot of experience with Rails and you've moved into the JavaScript world. And the thing that I've noticed with frontend is, we're applying a lot of concepts that existed on the backend. And we're moving all that complexity into the frontend. And that's kind of where that fuzzy line of, “Are you the frontend of the frontend? Are you the backend of the frontend?” comes from. And it's interesting that – like this is going to be another podcast. We'll end up talking about this – but we have – MVC lives on the frontend. Like in React, your C is a container component. The controller is a container component. The model is probably your Redux layer if you have it or if you're using micro-states. And the view is your components, your view layer components that you're rendering down. So, these concepts are moving from the backend to the frontend. We're just kind of renaming them and making them our own concept. But they're there. So, how does that knowledge sharing happen? Is it really just a mass exodus of backend developers interested in frontend developers now? CHARLES: I don't know. I think it goes both ways. So, I can only speak to my own experience. And that was when I first started doing frontend development was back in 2003 I think. So almost 15 years ago. We were building a touchscreen interface for an electronics vendor in the UK. And it was just so much fun, because we were getting to build these interfaces that literally looked like nothing else. And you could touch them and you had support for rudimentary gestures and there was no keyboard. And basically the only interface was your fingers and like a price scanning gun. And everything, all the entire UI had to be developed out of that. And so, it was such a novel system and it was so fun to implement that I just gravitated towards it. I think the thing that was particularly compelling was it was very interactive UI. It was high-touch, literally. This is a clerk who's sitting there using this thing as rapidly as they can to sell stuff and move people through the line. So, it was like a unique problem. But the thing that was cool about it was I realized so many – we kind of already touched on this in this conversation – so many things that I had learned on the backend applied there. And there was in order to provide that experience, there had to be a pretty complex machine behind it. And that was fascinating, that machine. And so, we were able to apply a lot of the concepts. And so, in that time on the backend I'd just come off working off a backend that had basically an event bus – we would probably call them microservices now, although we didn't call them that then – were coordinating based on all these events. And that translated over into the frontend really, really well. And I remember using a lot of the techniques for exception handling – doing it on the frontend and doing a lot of the multithreaded stuff that we were doing on the backend, trying to reconcile that with the frontend and really understanding. So, there were all these analogous concepts. And it could be – so, there was an original exodus of backend development, for me personally. And so for me, it was like I felt like I moved onto the web pretty much exclusively in 2006. And it was a good – I would say 2013 maybe is when web UI development finally caught up with Java Swing. Like it was like, that whole time I was like, “I'm using the web because it's an awesome deployment platform. And it's got all this great stuff,” but man, the developer experience is not as good as like a stateful fat GUI was back then. And now, I would say it's actually surpassed quite significantly. But now, I think there are a lot of people maybe who either – I think there's a casting back of people who are in frontend development and casting back to the web. So right now, my love affair with immutability and functional programming started by using a Clojure web framework which borrowed a lot of the ideas from Clojure. So, I guess there was an exodus there – people from Clojure wanting to develop apps, wanted to have their goodies. But then I found that and I was like, “Whoa, that's awesome.” But then that really set the hook for me. And so now, I try and go back to the well, the backend well or the shared computational well, as much as I can. So, all of that stuff of basically discovering all this functional programming stuff, this highly formalized functional programming, all came from saying, “Wow. I got a taste of that. Let me get more.” So, it's very much like trying to run through the village of the backend and ransack the stores and take as much as I can and bring it back, because I know that it's good. ROB: So earlier in the podcast, you said that you were a frontend developer. You described yourself as a frontend developer. And that's funny, because I actually consider you a full stack developer. It's because you can jump in on the backend and sling any kind of code as well as anybody could, and then you can also do the same thing on the frontend. So, the question I have here is: Do you think actually someone that truly is a full stack developer – and we can define that in a second – but do you think they're at an advantage here, because they have all that experience? You can answer yes. But why? Why do you think they have an advantage? CHARLES: I think they have an advantage in the same way that a person who's bilingual has an advantage. So, if you are living in North America, it behooves you to speak Spanish because you are now open to an entire set of markets that were previously closed to you. Well, not entirely closed but like you can trade with Mexico and you can trade with South America. You can buy and sell goods. You have access to yeah, emerging technologies that might – something special might be happening in Mexico City, or in Medellín, Colombia, and you're going to have first access to that. And so, if you don't, if you're not bilingual, then you're going to have to go through an intermediary who is. And so, there's a premium: you get to be the middle man so to speak. Or you get to be, maybe that has kind of a negative connotation, but you get to be the broker of new ideas and new technologies. If you can, if you are fluent in backend so to speak, then you can go into backend-landia and you can talk with the developers there and see what kind of cool stuff they're doing. And then you can take it across the border into frontend-land and sell it. And so, that's – I think we're very much first to the gun on – not first to the gun but we're ahead of the pack in terms of functional programming. Because we've seen that. We've seen it proved out and we've now actually started to integrate it into your day-to-day routines. And we're ahead of the pack on that, right? And so, I think that's kind of a keystone analogy for me that I think really, really captures what is the advantage in being a full stack engineer. ROB: So, well how do you define a full stack engineer or developer? What things do you have to possess to actually claim that title? In order to be a full stack developer I personally believe you have to know, you have to be comfortable with CSS. You can hate it. You can yell at it. But I think you have to be able to put together a layout if a designer gives you a comp in order to claim full stack. And in the same token, if you're a full stack developer and you mainly came from the frontend, you should be able to implement backend features. I don't actually have – so, I'm strong in the frontend, not so strong in the backend. What would be the analogous of CSS in the backend? Would it be like mocking your controller actions? [Chuckles] CHARLES: I would say you should have a familiarity with HTTP and REST, would probably be the equivalent. Kind of like a foundational technology that just every single technology is oriented around it, or most. Regardless of the protocol you're using – there are things like GraphQL which are not really RESTful, although it's a blurry line there – but they're still delivered over HTTP. And so, understanding things like CORS, understanding the things that are going to be universal to APIs. ROB: I think a lot of people try to claim full stack. I try to claim it. And I know I'm not. I can put together a pretty crummy Rails API on my own for personal projects. But I'm not going to be the one that's setting up indexes on a database to optimize a database or anything. I think that does come in time, but you have – borrowing from Brandon Haye's talks about career development – if you're a full stack developer, I think you end up having to be in the industry for a long time. Longer than what we consider a senior developer these days, I think. Because you could be a senior developer and be specialized. And we see that all the time, and that's okay. But in order to amass that knowledge and experience, I think you have to be in the industry for a long time and done those things a couple of times to really know. CHARLES: Yeah. I agree. So, can I add something there? To continue the analogy of being full stack is like being bilingual or multilingual. I go back to those analogies a lot because that's what I – linguistics is what I studied in school. But when I was studying linguistics, one of the things that was going on there was they were kind of redefining what… ROB: That makes so much more sense of your vocabulary. [Laughs] CHARLES: What it means to be bilingual. There was kind of a reorientation of that definition that was going on while I was in school. And that was being bilingual doesn't necessarily mean you're able to converse about poetry or like deep technology or give speeches in another language. It really is, it's as spectrum of… ROB: Ooh, that's quite interesting. CHARLES: The definition of bilingual is like: Do you use another language in your life? So, if you are let's say someone who is a recent immigrant to a country and let's say you've got less than 1,000 words but you're using them to buy groceries, to pay bills, to do things like that, then you are bilingual. Bilinguility is not an exclusive club. It's really, are you actually using a language? So, if… ROB: Ooh, so that actually gets my wheels turning. Does that mean that you can have a junior full stack developer? Because like, if you just merely speak the both languages, technically by that definition, I jive with that. You can speak two languages. You are bilingual. You can write in two different languages. You might be full stack. But does that mean in the software industry you are a junior full stack developer and then as you go on and get tasks that are full-stack-like, you get better on both sides? Or is it as an industry, we really have to specialize? CHARLES: To start on the first point, I think that if you – let's just restrict it to one language – if you're doing JavaScript on the frontend and using Node.js on the backend, if you're writing Node code you are I would say by definition, and certainly by the definition of bilingual that I just proffered, you would be considered full stack, a junior full stack developer like I was saying. And I think that it's just been my experience that as you go on in your career, you will cross multiple layers of the stack just because you can't keep your hands off. If you have an ownership mentality of, “Here's this thing that needs fixed,” and it's on the backend, you know what? I'm going to go ahead and learn a little bit of how to do this, because I need it to work with the thing that I'm working on. ROB: Yeah, that makes sense. CHARLES: You will just naturally be drawn over borders throughout your career just because someone's got to do it, to do the thing that… ROB: You got to solve a problem. CHARLES: Right, when you've got to solve a problem. So, I think that people become more and more full stack over the course of their careers. But that said, I do think that there are clearly areas where functional specialization is absolutely a requirement. Like if you say, “You know what? We've got this API and we need to support 600 queries per second and we've got this huge, crazy deployment…” ROB: I will not be your person to do that. CHARLES: Yeah, exactly. That's something you're very much – that is something that you're very much hiring for. And you want to be hiring for like you said, someone who has the skill and someone who, that's what they want to do. ROB: Yeah. If you need better state management and rendering performance and testability on the frontend, I'm your person. [Chuckles] If you need me to scale your API, to handle hundreds of thousands of requests a second, nope. [Laughs] So yeah, I agree with the specialization. So Sam, I want to know. Has this conversation swayed you any way? Are you more interested in being full stack or are you leaning to a side more? [Chuckles] SAM: So, I think full stack is, it's as much about skillset as it is about personality. So, like you were talking a little bit earlier on how someone who's frontend might have more empathy towards the user. And someone who's backend might have more empathy towards the computer and the developer, rather than the end user. I think someone who's full stack has to have a wider range or empathy so they can empathize with all rather than just one or the other sets. I think personality-wise, I'd be a fine full stack developer. But I think professional-wise, I do gravitate towards the frontend because that's just how I am. As a person, I like to see the visual rather than the concept. I do a lot of painting. I do a lot of drawings. So, I'm a very visual person. So, it's really helpful to me, especially for someone who's junior and who's still learning and who came from that bootcamp experience, to be able to see what I'm changing. So, I think it does kind of go a little bit into experience as well. So, I think over time when I start touching on maybe some more backend work and I see some stuff that interests me, definitely I'll gravitate towards it, just because I want to learn. But I don't know that it's like an intrinsic quality, you would want to be full stack or be backend or be frontend, you know? ROB: That's interesting. Yeah, I agree with that. CHARLES: Yeah. That is actually a really interesting point. Because I think what I heard you say there is that when you have – software is a hidden world in many ways. You talked about it in the beginning of the conversation, the areas of the site unseen. Like you know it's there but it's hidden from view. So, there's a certain amount of vision that needs to develop. So, you kind of internalize what software, like this hidden software, looks like. So what does a deployment of some load-balanced microservice clustered thing look like? Most people would not be able to answer that question. But the more time you're exposed to it, the more it becomes burned into your inner vision. ROB: Mental model? CHARLES: Yeah. You develop a mental model. Your brain literally wraps around that. It takes time. But on the frontend, especially if you're a visual person – but I would say even if you're not – I think the same would apply to someone who's using assistive technology. It's still something that you can feel with your sense and you don't have to develop a sixth sense to perceive it. So, there's literally a problem of perception there. And so, maybe frontend work is a great on-ramp for everybody, because it's so perceptual. ROB: That's exactly why I picked it. I was trying to do iOS dev before. And I could not do it for those reasons. I didn't have that nice tight feedback loop of even though it was UI, in Objective-C you had to construct your UI and buttons out of Objective-C. Unless you wanted to use Apple's Interface Builder and that was, meh. Everybody built it procedurally. And what I loved about HTML and CSS, was I could instantly throw something on the page, attach some CSS to it, and see it change immediately. And that felt so good. [Chuckles] CHARLES: Yeah, yeah, no. And it's important to get those really tight feedback loops, especially when you're starting out. ROB: So, if we had to tie this back together, did we decide that there is a distinction between frontend, backend, and full stack? And if there is, or isn't, why? SAM: I think there's like… CHARLES: I don't know if we've… go ahead. SAM: Kind of a distinction. But it's also really fuzzy, I would say. So, if I'm going to explain what the difference is between frontend and backend to someone who maybe isn't a developer, I would always go back to a video game. So, when you see your guy… ROB: Ooh, this is interesting. SAM: Yeah. [Chuckles] When you see your guy running around and you're like, “Oh, this game looks really good. I'm really into these graphics,” but do you ever think about what it takes to save the game or what is actually being saved when you go to save it? So, if you're more interested in making the guy run and making the guy and the environment look good, you might gravitate more towards frontend. But if you're really interested in saving the data, like well what is being saved when I hit ‘save to this slot' or whatever? Then you might gravitate more towards backend. ROB: Or like the network layer of the online multiplayer? SAM: Yeah, yeah. ROB: [Laughs] That's interesting. I've never used video games as an analogy. I always go like, you know, if you're a frontend developer, you know that button that you click? That's the button I create. And then the action that happens is usually what the backend developer does. I think I like the game analogy better. [Laughs] SAM: I think the game analogy is something that most people can grasp. And most people can grasp. Like I'll tell people the difference between frontend and backend if I'm talking about Facebook. Like what I do for a job is I'll show you everything that's on your Facebook page. And when somebody, or the backend is somebody who makes all of that come into play, kind of, you know? But I think video games are easier to conceptualize that abstraction of data being saved rather than trying to explain the intricacies of Facebook to somebody. ROB: [Chuckles] CHARLES: Yeah. No, I like that. I like that. So, I guess if we achieved consensus, would we say that these do exist as broad functional categories? And a full stack developer is someone who will move in between those functional categories through the course of their career. And that generally, the trend is that the longer the career goes, the more you will do that. ROB: Yeah. I can get on board with that. I'm going to use your career as like the guiding post of that. The way you just explained it, it kind of made something click for me. You describe yourself as a frontend developer. And I think in our industry with software development and the way teams work, I think you end up specializing no matter if you call yourself a full stack developer or not. Because I do think you're a full stack developer. But you've mainly been working on frontend for the past what, three years, four years? So, I think it's okay to call yourself a frontend developer. But you are a full stack developer. But as you specialize around and amass that knowledge, that's where you become that full stack developer, right? And so, at one point in your career, you were a backend developer. And then now at this point in your career, you're a frontend developer. But now you have those experiences and you can draw on both of them and apply them across the fields, right? That's super interesting to me. So, maybe it is that you have to specialize in order to achieve the full stack. And you're never truly a full stack developer in your role. But it's possible, depending on the team and the team dynamics. It's just interesting that you can be a full stack developer, also at the same time be specializing as a frontend developer at that current time, right? Yeah, that's interesting. CHARLES: Alright, so it sounds like consensus achieved. Let's all virtually hug. [Chuckles] ROB: Send the hug emoji. CHARLES: Alright everybody. That is our show for today. We are The Frontside and we build software that you can stake your future on. We would love to hear from you, especially if you have an idea for a future podcast topic. Any news that you think is something that we should discuss, even if it's not the theme of the whole episode. And we will see you next time. As always, you can give us a shout on Twitter at @TheFrontside or just drop us an email at contact@frontside.io. If you enjoyed today's podcast, it was produced by Mandy Moore, the inimitable @TheRubyRep. So thank you, Mandy. And see you all next time.

Frontside alum and original podcast host, Brandon Hays, makes a special guest appearance to talk with Charles about the evolution of The Frontside as a company: where it's been, where it's going, and more hopes, dreams, and goals for the future! Transcript CHARLES: Hello everybody. Welcome to The Frontside Podcast Episode 100. Here we are. Episode 100. My name is Charles Lowell. I'm a developer here at The Frontside and I think it's safe to say, your official podcast host. With me to celebrate the 100th episode, he was also here a few episodes ago but also was here on our first episode I believe, is the [inaudible] Hays. Hello Brandon. BRANDON: Hi. CHARLES: Welcome back to the podcast. BRANDON: Actually, are you going to light your trainee badge on fire now in a bucket, in a ceremonial pyre? CHARLES: I live in New Mexico, so I think I'm going to just after this, grab my shotgun and give myself a 21 gun salute. Just in my front yard. BRANDON: There goes old man Lowell again, with the shotgun. CHARLES: I'm just going to [gun shot sounds] in my own honor. BRANDON: I was at the Alamo this weekend, actually. And I don't know if it was just because it was fiesta in San Antonio but they had a demonstration, like a musket firing demonstration where those things are basically little cannons. They're just small cannons. It's very interesting. They're very loud. CHARLES: Yeah. They're small, handheld cannons, yeah. So wait, were you – what is fiesta? Now, as someone who grew up in Central [inaudible], I feel like I ought to know this. BRANDON: I don't know. We found out by accident because we were planning a weekend to go hang out and get drunk on the riverwalk and we took our families down with some friends and then they're like, “Oh, it's fiesta,” which is like a 10-day celebration of the history and establishment of San Antonio – which I did not know is a 300-year-old institution. So, it's like one of the oldest things in this entire western United States. So, it's pretty neat. It's different. It's weird. It's like 90 minutes from Austin. There's nothing in Austin that's older than six months. Every six months we must demolish something and then build a condo skyscraper in its place. So, it's kind of neat to be in a city where it has – walking around the Alamo, I'm realizing, “Wow. Setting aside any of the historical significance of Texas independence or whatever, this is just like a really interesting very old building. This is hundreds of years old in an area where there's nothing that's hundreds of years old.” So yeah, it was pretty cool. It was a good weekend and we got to see muskets being fired. And we saw a doctor gross my kids out by talking about the medicine of the day, in full costume and showing all of the procedures and threatening my kids with amputation. And it was a good time. We all had a good time. My nine-year-old thought it was the coolest damn thing he'd ever seen. CHARLES: Really? Did the have bloody saws and everything? BRANDON: Oh, yeah. CHARLES: Was it like a reenactment of 300-year-old surgery? BRANDON: It wasn't a full reenactment. But it was a graphic description using the tools of the time. CHARLES: Wow. BRANDON: Highly recommend, check out the Alamo. Super fun. CHARLES: That does sound really cool. BRANDON: I did not expect to have a good time and it was a good time. CHARLES: Yeah. Yeah, I know the whole reenactment with the musket firing is fun. And it is, it's actually an incredible building. Although there's been a big kerfuffle about something about how they're going to preserve the lawn. But I haven't really followed that too much. BRANDON: Yeah. Yeah, I don't care about the lawn. I care about – no offense, lawn, if lawn is listening. This is not weird, how Stanley broke our brains with the word ‘lawn'. CHARLES: That's true. BRANDON: Yeah. He broke us real good. CHARLES: Yeah. I can't see a lawn without a beard. BRANDON: So yeah. So, life has been pretty good, man. Let's see. I left Frontside September, October. CHARLES: 2016. BRANDON: 2016. CHARLES: So, it's been months. BRANDON: 18. Yeah, thereabouts, right? So, I assume that nothing happened since then and if I came back to The Frontside now, everything would be exactly as I left it. My posters are still up in my room. My Bon Jovi poster. You left my bed just as I made it, like kind of unmade. Everything is just preserved as a shrine to me. CHARLES: Pretty much. I mean, we did give away the mics to Goodwill. BRANDON: No. CHARLES: We actually did not give away those mics. BRANDON: I never even got to use them. CHARLES: I know. Well, you know part of the problem is we don't even get to use them that much either. It looks really cool and it plays really well, like our podcast studio. But you know, I'm now spending 75% of my time in Corrales, New Mexico. And at any given time, people are either working from home, or working remotely. So, a lot of times the podcast room tragically does not get used. But it looks so cool. People come in there and they're like, “Wow, you guys must be really smart and technical people.” BRANDON: I realize this is probably a rote stereotype at this point, but I am assuming the only reason that you moved is that you are dabbling in the production of meth. CHARLES: Pretty much. BRANDON: It's like, I want to learn a new trade. Programming, it's just – programming, how interesting does it stay honestly for 25 years? CHARLES: Right. Yeah, and you know, we've got some good techniques. Continuous integration, deployment, things like that. Test-first. These are things that can be applied to different verticals. And I was looking… BRANDON: [Laughs] We ship meth to production on the first day. CHARLES: Right. [Laughter] Exactly. So, I figured it was a market ripe for disruption. BRANDON: [Laughs] It's probably true. So yeah, I wanted to ask you about that. You all kind of scattered to the four winds in some ways. You have Elrich in Boston and you're in New Mexico most of the time. CHARLES: Joe is in [inaudible]. BRANDON: Oh yeah, Joe moved to New York. CHARLES: Yup. And honestly, the traffic is so bad in Austin that I'd say 50% of the time, people stay home rather than drive into our centrally-located office. So, that's actually something that we're struggling with right now because the bulk of the team is still in Austin. But the office space is underutilized. Our team size now, we have eight engineers. And five of them are in Austin. Our other staff is also in Austin. So, what do we do with the office? It's a big question. BRANDON: And that's quite a cultural change, too. Because when I was there, we would tell people, “We want to be able to do remote someday. But we just don't know how to get into that culture to change the way that we do our meetings and change the way that we do standups and coordination and communication.” I didn't feel like we had the tooling at the time. So, something – I knew that at some point there would be probably a forcing function to basically catalyze something to allow that to work. And I'm curious to know what that process was like there. CHARLES: I wish I could say that there was a process other than experiencing the force of the forcing function and then being forced into it and then just kind of dealing with it. I have not taken a poll of the other remote employees of which now I am one, at least for the time being. So, I don't want to speak for them. But it was less painful than you might imagine. And the reason is because – and it's one of those things you actually gave me this analogy back, probably three or four years ago and I love it – is sometimes you're hanging off of a precipice and you don't realize that you're toes are two inches off the ground. And then all you can perceive is the precipice and you feel the weight of your own body concentrated on your fingers gripped to the ledge. And you don't focus on the fact that you're actually, the fall is only two inches long. And that's kind of what we experience with the remote culture. Now, I don't want to say we were Pollyanna about it and didn't realize that this was the step that we were taking and making sure to check in with the remote employees. But one of the things is our communication styles were already very asynchronous both for our client work, for our internal work, using mostly Slack and GitHub pull requests and issues – certainly for the development portion, very little changed. What we didn't realize is that because of our involvement in open source, we were already acclimated to a distributed work style. We just didn't really realize it. We didn't have to change much. I think where we have a lot more work to do is kind of integrating people socially and making sure that conversations don't happen that aren't available for other people to consume asynchronously. So, if you're having some architecture problem and you're sitting next to somebody, you'll take that avenue rather than let it play out in chat or over email. And there is definitely a certain portion of that, but I think we still do a lot of pair programming. That's still our major mode. I'd say 75% of our code gets written as people collaborating. And so, while those in-office discussions do happen, the ramifications circulate rather quickly. And most of those are in the context of people pairing inside the office. Does that make sense? BRANDON: Mmhmm. CHARLES: So, I don't think the office and the physical space were as much of a bottleneck as we thought they might be. And so, because of the – a lot of people did work from home already because of the traffic. And we were involved in open source. And our communication with our clients is usually – we don't currently have any clients in Austin. So, that's all to say that the transition was actually quite natural. And I think there's some strong analogies between collaborating in open source and having a remote culture in your office. I think what we need to get better about is making sure that we get the team together at least twice a year, everybody together. Making sure that people are able to understand their priorities and get to circulate around and get introduced to a bunch of different people. And yeah, I don't know. There's definitely a lot of work to be done on the non-development front. BRANDON: It's interesting. The agile approach to things is to try something. I'm starting to think the agile and the scientific method are related where it's like, “Here's a hypothesis. Here's the experiment. Here's what we think we want to learn,” and then you learn it and you take the next step based on that information. And that failure is an option. I think that's the point of agile, is to make failure safe because it's small and you're guaranteed to learn from it. Like, the point is to learn. And so, I really, I'm starting to think that those are just basically the same thing. That agile is like the application of the scientific method to product development. And it sounds like you're being agile or experimental about your work. And the trick is, like any scientific discovery, the trick is in coming back around to it and analyzing it and deciding whether this was successful or a failure based on feedback and finding what the measurement was that you were trying to improve. So, the lesson there was, “Oh, people become disconnected from each other. We need to gather everybody for an all-hands periodically.” We didn't use to have to do that because all-hands was every week, at least. CHARLES: Right. Yeah, everybody was constantly – there was a constant chatter and you could just kind of, the context was just all sitting at that one table, in that one room on 38th Street. And all you needed to do was dip your ear into that pool of context and you're set. Whereas that's just not an option right now. So yeah. I think the danger with agile is not being concentrated in your experimentation. I think what gave us our fear about saying we're going to do remote work – because I remember we always talked about it. We danced around the issue – was are we going to lose who we are? We have a set of way that we do things. And there is power in kind of sticking to the framework of the way that you do things. Because you understand it and you know it. So, when you're pushing and you're experimenting, being able to say, “We're going to – push and we're going to focus on this one area and we're going to iterate on it and we're going to keep everything else static,” it's going to be the wall that we can walk along. But we are going to push in this area. And so, I think the dangers of you doing that in all the areas of your business or all the areas of your project, you're iterating and refining, nothing ever gets done. And so, it's kind of like once you get to some ground that's solid, when you do start iterating it, you start introducing instability. So, when you go remote you have to start thinking about remote work, whereas we didn't have to think about that before. We were essentially, the feature of saying that we were a one-office company and an on-site company is we didn't have to think about that problem. BRANDON: One thing that you were just taking about is this idea of concentrating so that your experiments are happening one or two or maybe three at a time instead of trying to run five experiments at a time. And yeah, there's another danger I think in agile of seeking local optimization where you're basically like – it's like taking a bacteria and running it through many, many, many iterations that's targeting one thing and it mutates into this weird thing that only does this one thing. Or a dog breed that the whole – did you see that, I don't know where this came from but there was some scientific findings that there was a dog that was bred in ancient prehistoric times that was bred to turn a spit to roast meat over. So, they bread a dog that the whole point of this dog was to turn a spit so that people could roast meat and go to sleep and let their dog serve it, cook for them I guess? CHARLES: Wow. BRANDON: That's pretty impressive. CHARLES: I would say like their dystopia is in the past. Or certainly canine dystopias. I guess we live in a canine dystopia. BRANDON: Not in my house. CHARLES: Not at your house. BRANDON: This place is known as a canine paradise. So yeah, I think that's a really interesting point though, that limiting the number of concurrent experiments so that you can actually respond to them in a meaningful way instead of just being like, “Wow, we learned a bunch of stuff we're doing wrong. Anyway, back to the grind.” CHARLES: [Laughs] Yeah. BRANDON: Back to sucking at everything. CHARLES: Right, right. BRANDON: That kind of feeds into a lesson that I have learned very, very, very recently in the interview process for looking for my first real job in over a decade. And that process is very humbling. And one of the humbling experiences was being rejected for a job from a very notable larger former startup here in Austin. And their interview process is really buttoned up. I got really deep into the interview process and at the end of it they're like, “Oh, you're not technical enough.” And it was really, it was like, I don't know. It was hard for me to process at the time but it's super easy now to look back and go, “Oh, I was definitely not a fit for that type of job if being able to write JavaScript on a whiteboard without the aid of Google to solve problems and refactor code is like a fundamental part of what is valued in a manager there.” That's just not going to be me. But one thing I – and it wasn't a colossal waste of time. There was a ton of time and energy I invested into that specific process, but I actually derived a ton of value out of it. Because every person I met there was focused on the same thing: their culture of making experimentation inexpensive so that everything there is framed in terms of an experiment. What's the experiment here? What's the hypothesis? What's the expected outcome? How soon can we get to a place where we can validate that outcome? So, it's kind of like everything is really lean. And yes, it does – like I asked, “What's the dark side of that?” and it can lead to optimizing for a local maximum. So, you have to pause every once in a while and reflect at a larger scale. But it changed my attitude about a lot of stuff. I tend to walk around fearing failure. That's more my speed. I'm afraid of failing because failure can be catastrophic. But that's because I take big swings at stuff. When I go give a conference talk, it better be the best conference talk of my life. When somebody's like, “Oh, that was the best conference talk I have ever seen,” I'm like, “Ah. I'm so glad you said that because if you'd said literally anything else I would have collapsed internally.” You know? The stakes are so high for everything. And making it safe for yourself to fail by treating things like an experiment and working with my teammates. And so, two or three scenarios over the phone in a week when I was managing the team at my last company, somebody would bring something to me and I'm like, I instantly went to all the reasons this probably won't work. “Here's the problem with this.” And I thought, and I immediately turn around and went, “Wait a minute. Bring me a hypothesis and the experiment and how we can experiment with this thing.” And he's like, “Well, we could try this next week and we'll know whether or not this is a good idea.” And we tried it the next week. It was like organizing an architecture team because we were waiting to hire an architect. And the results were mixed for reasons I won't get too deep into. But the fact was, it gave us the freedom to try things. And I'm trying to carry that spirit around with me now. It's been really eye-opening. So, completely like, just a 4% alteration in the way that I think about problems, but it has the ability to dramatically alter the trajectory of how I solve things in the future. CHARLES: So, do you include now inside the planning process experiments? Like, a certain number. BRANDON: Absolutely. CHARLES: So, the typical “enterprise” development is we have our features, we're going to do them in this order because they're this priority. And then agile comes along and it's like, “You need to take these things and you need to break them up into small chunks so that they can be accomplished in small time slices,” so that you don't basically bark up wrong trees. Or explore [inaudible]. BRANDON: Yeah, but that's almost like a stupider version of waterfall. CHARLES: But exactly. That's exactly my point. Whereas the problem is, there's no avenue for experimentation in there. Rather than saying the entire team is marching in this one direction that meanders around and focuses in on the local maximum, which hopefully is relative to the market landscape is the absolute maximum, saying, “We're actually going to be marching in one major direction but we're going to be sending out scouts at all points.” If you were actually – I've actually been reading a lot of ancient military history. And It's just insane that an army, or even a detachment, would go all in one clump. They're constantly sending out people. Information is really, really, really important. BRANDON: That's an extremely, extremely good point. I've actually – it's so funny, because I've used a very similar description where we are trying to chart a course to this ocean of opportunity somewhere. And we can't just send the whole team in a direction hoping that the ocean is in that direction. We have to have our Lewis and Clark. Somebody has to be the cartographer. Somebody has to be the explorer. And that means that there has to be a little bit more freedom for those explorers. I don't yet know how to translate that into software terms. I just know that that's a collaboration usually at most companies between product and development. That product is doing some of the exploring of the space and then development is doing some of the exploring of the technical capabilities and possibilities there. CHARLES: So, you see it. What's interesting is you see it in product planning, kind of in the large, with the waterfall. You see it in huge organizations. They have a research and development department. And I wonder if agile kind of saw the Balkanization of your feature set into very small component parts. Can you take the exact same principle and Balkanize your research and development and integrate it into micro-iterations? We have this R&D but we're going to integrate it into our day-to-day and week-to-week process. BRANDON: I think that is a really noble goal and I think I see some people making progress toward that. The company I interviewed with does it almost to a pathological degree where there is a point of diminishing returns where you're sort of bound to this process of experimentation. And at a certain point you can only achieve incremental results. CHARLES: Some of these problems, you just need to be able to think about them for a long, long time. I actually didn't read, I actually didn't see the talk. But everything from the title, Rich Hickey's ‘Hammock Driven Development', just that title resonated with me so much. I was like, “Yes,” because sometimes you just need to be in the hammock for six hours at a time. Or in the shower. Or hiking. Or doing whatever it is that you need to do to put yourself in a zen state where you're just, your brain is slowly turning its wheels. And it can follow every lead to its conclusion without any interruption. And sometimes that process can take hours. Sometimes it needs to take weeks. BRANDON: Right. I want to kind of pivot on that. Because that's actually one of the biggest things that I've learned in the intervening time since leaving Frontside, which is creating space instead of trying to maximize – one thing that I did when I was at Frontside and then did again at my next place and I'm realizing is really has long-term negative implications is cram as much into a work day, as much output out as possible. I'm very output-oriented. I want to jam as much into my day as possible. I want to jam as much software out the door as possible. And people describe working at Frontside while I was there as one of the most intense work experiences they'd ever had. Literally, I can project that, literally just from my own intensity of trying to cram all that stuff. And providing that space for developers to ruminate on hard problems, on some of the harder problems they encounter, providing space for managers, I've learned that a big chunk of what it is to be a manager is to be available. And so, I actually want to write a sign – I was on the fence about doing this but I think I'm actually doing to do this – I have an office and I'm going to write a sign and put it up on the door that says, “If I look busy, interrupt me and remind me I'm not doing this right.” So, creating the space to ruminate or to be available for discussion, people that protect their breathing room sometimes are made fun of, especially in American corporate culture. I walked in and they were just reading a newspaper. What the heck are you doing at work if you're just going to read a newspaper? Like no, this is actually really important time. CHARLES: I think it's, yeah, it's something that I think about a lot. And I know I've shared this analogy with you before. I don't know if I've done it on the podcast. But I saw and I can't take credit for it. I actually saw it at DevOpsDays I think in 2013. There was a woman giving a talk and she was just talking about managing developers. But one of the things that she was saying was that if you looked at a microservices architecture or you looked at just even your operating system, and if your CPU was constantly pegged, you were squeezing out 100% of every time slice, instructions were just flowing through that, you're going to have a very unhealthy, very brittle, very prone to failure software system. If our microservices were not available to actually service requests, and service excess requests, and service spikes of requests, then something is fundamentally wrong. BRANDON: I want to add to that a little bit, because the thing that I noticed in managing a team where I received a ton of pressure to peg everybody out at 100% – and it jived with my philosophy at the time of, “Hey, I'm 100% guy. Everybody I work with is 100% type people. And then, let's peg everybody at 100%. This is a startup. Let's get everything going,” and I realized very, very quickly that if you don't preserve a little buffer, 20% buffer in that level of intensity, there is no ability to share resources. Everything is now a silo. So, if you're going to peg all your CPUs out, part of that thrashing is that there's no time for people to share things with each other. And people become very protective over their little silo all of a sudden. And it causes us – it's actually like the first stage of a catastrophic cultural collapse if everybody's pegged out at 100%. And literally, just dialing down the intensity is often the only thing that's necessary to get people to feel comfortable sharing some of their time with each other. You do a really good job of that with the lunch and learns. You mentioned that y'all are doing better thoughtful lunch and learns and stuff like that. It's one of those forcing ways that you can force that and say, “Hold on. Stop the development and do some stuff where you're actually sharing things with your teammates.” CHARLES: Yeah. And we do that. My biggest concern is that that actually increases the intensity. So, one of the things we've done is we used to actually be very formal about our lunch and learns. It's like, “We've got to generate content and put it out on the web so that people can see us.” We backed away from saying – we're not going to do them as often and make sure that people can actually do them. Yeah, making sure that people don't feel overwhelmed by, “I've got a lunch and learn coming up.” The point is to share something that you're passionate about and maybe introduce some really cool ideas to ferment in people's head. Rather, that's kind of the goal. There are certain things that we do very much feel interested in generating content. But I think, we've kind of been dancing around the ideas of distributed computing and IoT and what are some of the others? BRANDON: If you say blockchain, I'm going to just virtually punch you in the face. CHARLES: [Laughs] I actually didn't. Did I say blockchain? BRANDON: No. I just was waiting for you to say it. CHARLES: Okay, no. I haven't. Well, because that's – but it is distributed computing in Web 3.0, right? These problems – and we're actually going to be podcasting about this next, so in two weeks you can tune in to listen to us talk about blockchain but in the context of distributed computing – and one of the things that we're seeing is now we're starting to pay the price of outsourcing all of our lives to these central services like Facebook and Google and Amazon. And I think now they're starting to build a credible and more mainstream movement to wrestle back that control and say, “What would it mean to have software as a service that wasn't actually dependent on some central thing?” What would it look like to have Slack where it's Slack that looked like email? Where everybody had their own email server, maybe not a bad example. But you've got an email at Gmail or Microsoft or Yahoo or your company-run that's big enough its running its own Outlook client or something like that. Email is actually a really great example. Now probably people are going to crucify me for saying this, but I think it's actually a good example of a distributed system that's worked well. I own all of my email. All the messages that you send to me, I own, and all the messages I send to you, I also own. But you also own the messages that I send to you. Information is duplicated. And it's fine. If I send you an image, yes it's on your hard drive or it's on your Google Drive. You send a message to me, it's got an attachment, I also have that attachment. But the point is that we can each own our email and we each own our email service. And we can change it up. That's not possible with Slack. That's not possible with Facebook. That's not possible with all these other sharing platforms. All of them are controlled by this one thing. And so, I think that that's something that we've been exposed to through the lunch and learns and I'm actually certainly very excited about it. It's not something that we're going to be investing in immediately. We're kind of dancing around that idea. But that's something that's come out of that. So yeah, we've kind of refocused it on, what is something that you feel good about? But back to the original point, I think that this is something that applies on all fronts. If you have a business where you can't actually take opportunities because you don't actually have people – so there's maxing out at the individual level, filling up people's workspace with client work or filling it up with what have you or having them work nights and weekends. There's individual maxing out but then there's like maxing out of your business. So, if you have – we're a consultancy – if you have 100% utilization or you're shooting for 100% utilization, that everybody is placed on a project, that is a brittle and unsustainable system. BRANDON: I wish you would have told me that 18 months before I left there. There were like two years where we were at 100% for two solid years. CHARLES: Yeah, yeah. We're still at 100%. BRANDON: Yeah. I wonder what would have happened if we'd had a little, if we had figured out how to build in space. CHARLES: Part of the problem – so, here's the thing though. Space, nice space costs nice money. BRANDON: Yeah. CHARLES: And so, that's the thing, is you have to charge more. And you have to say, “We are going to be more expensive than other people.” You have to be dedicated to be at the forefront of a cultural battle, essentially. In the same way that people were with testing, where it was very [controversial]. BRANDON: Yeah. You were with CI. CI is a given now, right? CI is… CHARLES: Yeah, like [inaudible]. BRANDON: This idea was semi-revolutionary when you and I were talking about this in 2012, 2013, that we ship to production on the first day. We don't even start building software until the CI system is set up. The first thing we do is set up Jenkins and tests and get everything, the pipeline working. And now, that's just what people do. By and large, that's how software is expected to be built. And the tooling has really come up around that. But that was an expensive way to sell software five years ago, that, “Hey, this is going to cost more than bringing in Cowboy Bob and having them come jam in your console for 40 days and ship a bunch of stuff that then will most likely collapse and you won't know about it and Cowboy Bob has ridden off into Juarez, Mexico.” CHARLES: Right, with his saddlebag stuffed with your cash. BRANDON: Yup. CHARLES: Yeah, no. So, you have to – the problem is, you know when you pick these battles, you need to be prepared to fight the war of attrition of they're not going to be able to perceive the value for six months, a year, right? You're going to have to ask your clients to bet on this strategy. And it's a bet. And you're going to have to say, “It's going to pay off in six months. It's going to pay off in a year.” And you're really going to start raking in like five years. That's when… BRANDON: Yeah. Try making that pitch to a startup founder that is borderline, that is on the verge of an anxiety attack, and you can kind of just figure out what my last year was like. And the… CHARLES: So, that's one of the reasons we don't really work with startups anymore. They have a five-year plan, but not really. BRANDON: Yeah. CHARLES: They're fighting for their survival. And they're fighting for the opportunity to have a legitimate five-year plan. And so, in that sense, it's maybe not a good fit for the way that we develop software, because you either need an extraordinarily prescient founder who has been through this before, knows the true costs of software development, and is pretty well-funded so that they can actually – because we're more expensive upfront, like a lot more expensive upfront and so sometimes they flat out don't even have the cash. And that's something that you can make a quick, “It's not a good fit,” but then there also needs to be this understanding and an acknowledgment that what you're really shooting for is your five-year dividend. BRANDON: Yeah. It is really interesting, the turn that occurs when a company finds product/market fit. By then it's too late to fix the problems. So, it's really tricky to find the balance of: how much energy do you put into the success case for a company before they have product/market fit? How much time and energy do you invest in betting that this is going to be successful versus betting that if it is successful, hopefully we'll have the time, money, and resources to redo a bunch of the things that we are going to have to apologize for later? And I think that's what makes… CHARLES: Right. Like, where do those two lines cross on that graph? BRANDON: Yeah. Because you and I have both seen startups completely sunk by somebody who was overly focused on building a scaleable architecture in a company pre-product/market fit. That is a common story where an engineer that doesn't understand the business value of what they're doing and only focused on “quality” will absolutely torpedo, they'll chew up your first million and a half of funding and leave the place in just a smoldering pile of ashes at the end. So, it is tricky. It's totally a difficult thing. But I think coming back to your point of being sort of a vanguard of cultural, the tip of the spear on somebody's cultural changes – DevOps would be one. People that were really investing in DevOps culture in 2010, 2012 saying, “Hey, this, automation, is the future of how software gets shipped, maintained, observed, supported.” And so, now it sounds like, so what is your big bet for the future? CHARLES: Boy. That's a great question. There are two bets. One you're going to like, one you're going to vomit. BRANDON: [Laughs] CHARLES: But that's okay. BRANDON: Yeah. I don't work for you. CHARLES: You need to serve, what is it? You need to serve the spiny urchin with the yellow tail. BRANDON: Is that a Sonic the Hedgehog reference? CHARLES: It's just a sushi reference. BRANDON: Oh, okay. CHARLES: Some people don't like urchin. Or maybe they don't like eggs. What it like, the roe that come with sushi. But they're on the same plate. So, I would say the first one that I've been thinking about a lot is optimizing for capacity and being able to handle spikes and not being at 100% both for people and for utilization. I think that's something that is – I don't see how you could have a healthy software development process if people are completely spiked on delivering, heads down delivering features for product. That is something that I'm betting on. Essentially, you could call it the 25% time but it's really about having excess capacity to exploit opportunities as they arise. And then being protective of that excess capacity. Because you can exploit an opportunity. Your CPU has a spike load up to 100%. But then make sure you [inaudible] down to 50% at some point, or 75%. And so, I would very much like to see Frontside have a bench where people can rotate out and they're working on different stuff that are not even client-related. They can recharge their creative tanks. They're not going to be idle. BRANDON: Yeah, I've really come around on – and I really hated this at the time – but I've actually come around to the thoughtbot style of working on a product where – because owning and managing a product and developing it as a side quest, the goal is not necessarily for that product to catch fire and become the world's next big thing and to replace your consulting revenue. The goal is to give people a sense of – think about all the stuff that you've learned in your side projects that you went back and brought to your work. And some of my biggest gains as a developer have come from having a side gig of some kind, some side project that that's how I learned Ember. That changed my life. And I would never have gotten to try it if I was waiting for somebody at work to tell me it was okay to do it. So, it's about taking that permission back for yourself and giving yourself permission to try stuff. So, it could be something like that, or it could be the content stuff that y'all do. Or it could be conference talks. It could be whatever. But the goal isn't necessarily to produce things that have a direct return. It is to create the space to allow people to flex some muscles of creativity that you may not get in your day-to-day work. And that's very difficult to offer to people in any company. Now having explored startups and larger companies, but I would say especially in a consultancy where the exchange rate is dollars for days. It's sort of like when I was freelancing. I could feel every vacation I took draining both real money and opportunity money out of my bank account. That's such a hard, difficult thing to do. And so, you actually have to create the budget ahead of time and say, “This budget is allocated to these things and it's already spent.” Anyway, that's really tough to do. CHARLES: It is hard. BRANDON: If you can exercise the discipline necessary to do that and create the environment for that, I would say you're ahead of 90% of companies in the industry. CHARLES: Yeah. Yeah, so that's something I definitely want to bet on, because I think that's where the best things come from. BRANDON: Okay. So, what's the thing I'm going to hate? CHARLES: Functional programming. BRANDON: Oh, Charles. Okay, I have to stop you. Do you know what I'm doing? Did I tell you this yet? That I am participating. When I told them this, I was like, “Charles is going to have a field day with this,” but I am participating in a Haskell study group. CHARLES: No way. BRANDON: And I'm like four exercises into this thing. I have to do four more for next week. And I'm like, “This is bizarrely easy, actually,” after as much JavaScript as you and I did in sort of a functional style and then learning Elixir. And I was like, “Wait a minute. The case statement is, Elixir just stole Haskell's case statements.” So like, so far I'm not finding functional programming to be onerous. Or anyway, but we'll see when we get to the static typing. But so far, I'm not getting any of that in the earlier lessons of the book. CHARLES: Yeah, the static typing. But the thing is, you can do – it's not 100% necessary. It isn't in Haskell, for sure. But I'm surprised. What inspired you? BRANDON: We have an architect at the office that was like, “Hey, I want to do sort of a functional programming book club.” So, we have a Slack group for FP study group. CHARLES: Are you doing ‘Haskell: From First Principles'? BRANDON: No. That one was a little actually intimidating. CHARLES: Really? BRANDON: Yeah. It gets into the lingo a little early. And we're doing one called ‘Get Programming with Haskell' that is a little more – ‘Haskell: From First Principles' is kind of math-oriented. So, for somebody with a math background but not necessarily a programming background, it's perfect. But for somebody with a programming background that is just trying to understand functional programming principles using Haskell, ‘Get Programming with Haskell' is actually a really great option. CHARLES: Okay. Actually, I have not heard of that one. BRANDON: The stuff that I'm looking at looks just like Elixir. So, it's early. But it's very comfortable so far. CHARLES: Yeah. So, this is the thing. It's all a matter of messaging and marketing. Because I really feel – so, it is like there are a lot of behaviors that you see sometimes in currently entrenched functional programming communities that I think are, well I think they're objectively repulsive. But I think they're also pragmatically repulsive and that they repulse potential community members. But I think a lot of it too is people talk about these things that are, they use abstruse terminology. And they're kind of chattering back and it's very jargon-oriented. And there's just – people operate with a different set of concrete things. So, when you and I are talking, for example we might talk about a Rails controller and that's a very concrete thing. You know exactly what I'm talking about. It's something that you have held in your hand, literally. Remember when we got that Rails codebase that came as a thumb drive? BRANDON: Yes I do. CHARLES: But the point is you knew that this had a Rails codebase on it. There were any number of controllers. And when I say controller to you, a controller is an abstraction, but not really. Once you work with an abstraction long enough, it becomes concrete. And so, part of the problem is just a mismatch in language where people are talking in their world about concrete things, things that you can touch and you can feel and you can exchange and they're very relatable. But from another person's perspective, they're talking about something that's totally abstract and totally opaque and totally what have you. And so, I feel like yeah there's a huge mismatch there. And that's been one of the big bets. The other big bet that I'm making is on this trying to make what is currently abstract to JavaScript and Ruby developers be concrete. And I think that we're going to see type classes like functor and monoid and semigroup and all these things, they're abstract to you now, become concrete over the next five years. And so, that's something that I'm betting on. BRANDON: Check out this – and I know that you have a good relationship with the people that did the other book, but it really does tend to come from more of a mathematical background. And this one actually does speak to people with JavaScript, Ruby, Python experience. Like, “Hey, here is how you will perceive these things.” And so, it's much more approachable. I'm still in the first unit of the book. But having sort of tasted it a little bit, it's like, “Wait a minute. This is actually extremely familiar and not super intimidating.” CHARLES: Exactly. And that was kind of – so, I read the other book. And I think I was also aided by the fact that I tried to learn Haskell probably for five times in the past. And so, I also had the benefit of jumping against the wall with the velcro suit and bouncing off four times. And fifth time, it stuck. So, I had just temerity on my side and a general feeling. But that's definitely – the lesson that I actually came away from reading that book was like, “Oh, there's a mismatch in concrete concepts.” It's using concrete concepts that are concrete to people with a CS background or mathematics background, or people who are brand new. Honestly, people who are brand new to programming who don't actually have JavaScript or Elixir or Ruby or any other thing to lean on, I think that the First Principles book is actually pretty decent for them, too. Because they don't have anything to compare to. BRANDON: They don't have anything to unlearn. CHARLES: Yeah, they don't have anything to unlearn whereas one of the things I took away was I was like, “Oh, man. I'm using semigroups all the time. This is something that I do constantly.” When I'm coding, I might do it eight times in a day. I just didn't have a name for it. BRANDON: Right. They're like design patterns, just at a micro level. CHARLES: Yes, micro-design patterns. Yeah, it's like a RESTful architecture for your code. In REST you only get five verbs. There's five methods, man. That's all you got. BRANDON: Okay, so those are two bets. And I want to cover one more thing because I know we're super overtime. But the last thing I want to be able to say about talking about what we've learned since I left Frontside but I want to put a bow on that. So, the two things that you're betting heavily on are functional programming as a basis for solid architectures in the future, like the work that you all are doing. And… CHARLES: I would also like to say, and this is something – let me just add one more thought. What I don't understand, and this is in no way like, I don't understand people who do the, “Saying goodbye to framework X.” That's not me with object-oriented programming. BRANDON: Often abstractions are like oversimplifications but they're really useful, sort of like Rich Hickey's Simple versus Easy. Like, “Hey, there's a lot of promise with that metaphor. It's a leaky abstraction but it's a useful abstraction.” And Gary Bernhardt's ‘Functional Core, Imperative Shell' is a leaky abstraction but it's a useful abstraction. If people haven't seen or experienced that, it's pretty good. The subtlety is that these are tools that are suited to certain situations a little better. And those same situations can exist in the same codebase, can exist in the same program. CHARLES: Yeah. I still, I love Ruby. I adore it. And in some ways, I've been researching functional programming and it's been going on for the last four years. So many times, people are like, “Oh, I just can't stand this tool anymore.” And I'm like, “Man, I still love Java.” I don't understand how learning to love something decreases your love for something else. BRANDON: That happens the first two times that you fall in love, is that you feel like you have the old thing less in order to love the new thing. And then you start realizing, “No, you are allowed to fall in love with new things without falling out of love with the old things.” I would almost use that as an interview question. Is there some way to use that as a way to gauge somebody's actual real concrete maturity as a developer? Because that is a mark of maturity. CHARLES: Yeah. I mean, you could say, “What's some tool that you no longer use that still informs your day-to-day routine?” BRANDON: Yeah. I guarantee you, people that were doing Smalltalk in the 80s think about it all the time. CHARLES: [Laughs] Yup. Yeah, exactly. Exactly. BRANDON: Alright. So, I want to cover one last thing. CHARLES: It's part of growing, right? If you're going to grow as a developer, you can't be shrinking at the same time as you're growing. Otherwise, you're like the same size, just in a different place. BRANDON: However, you don't get any Medium think piece points. Nobody does the one clap, two clap, forty, for blog posts that are like, “Why I'm still using some programming language but using one a little more than I used to use and this one a little less.” CHARLES: [Laughs] Zero claps. BRANDON: Yeah, zero claps on that think piece. I just want to cover one last thing before we wrap this up, and it is the fact that Frontside, the biggest gift that Frontside gave me was the mission for the next 20 years of my career. I think it could change, but I'm pretty confident about this, at this point. Being approximately 20 years into my career, I feel like I kind of have a feeling for what the next 20 years is about. And the Frontside really drilled that into me and helped me focus it and helped me dial it in. And it is this idea that there is an incoming generation of programmer that thinks about things differently than the previous generation in a pretty radical way. Because the previous generation all came out of the same schools. They all look the same. They all have a similar shared set of values in general. They created the Sil- – you know, I'm not actually going to be overly critical of the Silicon Valley culture that exists now. It is a result of the type of people that came out at the time that value innovation over almost anything else. People talk about ripe for disruption. The fact is, that has been an engine of economic growth and progress for society in a lot of ways that has a lot of costs that weren't factored in by a bunch of people who all thought the same way. And now, with people coming through code schools and people coming from different backgrounds and people coming from different environments, they're looking at programming and software as either an economic opportunity or something they didn't see that they could possibly do. Those doors were not open to that group of people before. There is a natural influx of people but many of them are bouncing out because they're not finding that group of people, they don't have a shared enough set of values that the people that are new are coming in and finding job opportunities, finding promotions, finding leadership positions. And so, I know now that my mission over the next 20 years of my career is to create those opportunities for people that have different backgrounds from me and different experiences. The career tracks, the promotions, endorsing and supporting and kind of sponsoring this incoming group of freshmen into our industry that come from different places, different backgrounds, different problems that they care about solving. They want to figure out how to solve the Flint Michigan water crisis instead of delivering socks to people in Silicon Valley, you know? So, I feel like we're at the beginning of a seed change in the value system potentially of our entire industry. But that's going to require training up the next generation of technical leadership. And I felt like the best thing I could do right now is learn to be a better manager, because I really like that job. And it provides the opportunity to find, hire, sponsor, promote and encourage those people to move into their own leadership positions. There are lots of other things that a person, you could be a VC and care about that stuff. You could have lots of different positions and put yourself in a position to do that. You could be a consultancy owner. You know what I mean? There are jobs that you can do that you can accomplish that goal. But it gives me such a sense of direction that when I'm looking for a job, I was looking for a home for that mission rather than just the thing that I felt like doing. Like okay, this job is important to me because I need it to house me and this mission so that I can support my family but have enough emotional overhead to participate in community stuff, but enough ability to lead within an organization, enough influence to actually push that agenda. So that the next generation of people are making better companies. So anyway, all of that came out of my time at Frontside where you and I sat around talking about: how do we build a place that is like a monastery? These were your words. You remember this? We want a monastery for code where people can just focus on becoming better developers. And underneath that though was the sense that this was a place of opportunity for people that might go somewhere else and stagnate as a developer. This will be a place to accelerate them. And so, that kind of spun me out and accelerated me into my mission. So anyway, I just wanted to point out that that was like, with a bullet, is the most important thing that came out for me in my time at Frontside, was that it clarified for me what I was trying to accomplish with the next couple of decades of my career. CHARLES: Wow. Well, that's fantastic. You definitely did a lot of that both here at Frontside and I mean you're continuing to do that. I definitely want to see more public speaking from you. Maybe some [inaudible] perfect. [Inaudible] at EmberConf was actually fantastic. But I mean, you're also able to help people find their mission, too. Like the talks you have at Keep Ruby Weird and even really, the first talk you gave at LoneStarRuby about moving Ember. It's always, how do I adapt what I'm feeling to my overall mission and then relate that back to technology? Man, I just can't wait. I can't wait. When are you going to hit the road again? BRANDON: I think this is the year. I'm going to start thinking about this stuff. I'm looking at the stuff that I wanted to talk about on this podcast and I was like, “Oh no, wait. That's like a dozen podcasts.” Like, no. Absolutely not. Not possible. I will say, I miss so much, this time that I spend with you. I don't want to let it go. I really miss working with you. I really miss having these conversations whenever I want. This has been a very, very special privilege for me to be able to do this with you today. And congratulations on Frontside continuing to thrive and grow and become more of its own entity and more of its own special flavor. And it makes me really happy to see the people coming out of there, that it's still doing its mission of making great software by making great developers. It makes me real happy. CHARLES: Yeah, yeah. Hopefully we can keep on keeping on. I do miss working with you. I miss the conversations that we would have in the kitchen which are basically an extension of this podcast. But I also, man, I really, really, really, really like working with the group of people that are here today. I've just seen them producing just some absolutely amazing things. And honestly, there's a selfish aspect to it, too. I get stimulated. My own thinking and learning is stimulated by the people that I work with. And like I said, the whole side note we had about distributed systems and IoT and just a constant ferment of things. So, I still really, really, really enjoy it. BRANDON: That makes me happy. CHARLES: And I'm really glad that we got to kick it today. BRANDON: Yeah, me too. CHARLES: I thought you were going to say that your 20-year mission was to have your perfect Emacs initialization setup. BRANDON: Oh my gosh. Some of these days, I'm going to figure out RuboCop. CHARLES: Actually, do you want to pair on that? BRANDON: Yeah, let's do that. CHARLES: Alright, everybody. I'm going to sign off. If anyone wants to continue the conversation, obviously you can get in touch with Brandon. He is misspelled @tehviking on Twitter. T-E-H-V-I-K-I-N-G. Always come at him. BRANDON: Don't @ me. CHARLES: [Laughs] BRANDON: I work for a really cool company and if you ask me about it on Twitter, I'll tell you all about it. CHARLES: Awesome. And we of course are Frontside. You can get us on Twitter at @TheFrontside or just drop us a line to contact@frontside.io. And we would love to talk to you more about this podcast and all the wonderful things that we do here, which includes building custom software that you can stake your future on, that's going to be good for the five-year outlook. So with that, goodbye Brandon. Goodbye everybody. And we will see you… BRANDON: Bye Charles. I love you. CHARLES: Me too.

Taras Mankovski: tarasm In this episode, Taras and Charles talk about a project that they work on together: Funcadelic - a Functional Programming and Category Theory for Everyday JavaScript Development. Funcadelic takes the simple idea that a single function can operate on many different data structures (typeclass oriented programming) and brings it to JavaScript. Yes, there are a lot of FP libraries out there, but this one is geared towards unlocking the magical powers of functional programming while always maintaining a tangible and accessible experience for JavaScript developers. Because if you're a JavaScript developer today, you're already using most of the structures in funcadelic! Transcript: CHARLES: Hello everybody and welcome to The Frontside Podcast Episode 99. My name is Charles Lowell, developer here at The Frontside and your podcast host-in-training. And with me today is Mr. Taras Mankovski. Welcome. TARAS: Thank you, Charles. It's a pleasure to be here. CHARLES: Yeah. So, you are ubiquitous in the JavaScript world. You do a lot of stuff with mentoring and you are involved in a bunch of different interesting projects. I think you're one of those developers who's difficult to classify, which is – that's definitely one of my favorite kind of developers. I wanted to have you on the show today because there's been a project that we've been collaborating on. And there have been some interesting ideas to come out of that and solidify through that project, at least in my head. And yeah, I thought we could maybe just talk about that a little bit. TARAS: Yeah, sounds good. It's going to be fun. : The thing that we are going to be talking about is a project called Funcadelic. It's more than really just a library, a JavaScript library on GitHub. It's kind of a different way of thinking about your code. And so, I know for me, where this really became part of my workflow was, when was it? It was about three months ago or something like that? Six months ago? TARAS: Oh, I think yeah, I think it's probably more six months ago. I think it's probably what, two months, I think probably December maybe? CHARLES: Okay. But it's hard now to imagine working without this tool on my workbench. It's been probably the biggest game-changer for me in the last, I don't know, definitely in the last several years. TARAS: Yeah, it's pretty impressive how little, how small of a library can have such a big impact in what we do day-to-day. Because it definitely makes me think differently about how I can solve problems that I solve on a daily basis when I work with React. So, it's been pretty interesting. I think for me, having worked with this library, I think what I'm getting is an understanding of how things work in a way, and a perspective on how React works, in a way that I don't think was available to [inaudible] Funcadelic. The funny thing is it's not a React library, right? It's not designed for React. It's just that… CHARLES: I don't even think that – it helps you think about React, but I don't even think it's the way that the React developers think about React, right? TARAS: Yeah, I don't think so, either. I think a lot of people are on the spectrum of understanding functional programming. And I think a lot of people use, people learn how to use React, but they don't really – I don't think a lot of people have traveled very far. I'm talking about general, majority. There's definitely people who know functional programming really well. And there's a lot of really good libraries in the JavaScript space for doing functional programming in JavaScript. But I don't think the general public, the general people that on a daily basis go into – write a render function and do ‘this.' or like ‘product.map' and then return an array of components. I don't think those people necessarily think about or get the context within which they use this tool. CHARLES: Right. And I think that's actually kind of one of the reasons I think a library like Funcadelic is so important and fills kind of a missing piece in the ecosystem, is because it really is predicated on the idea that programmers use these concepts all the time. They really are, they're foundational. But we only kind of see them out of the corner of our eye, in our peripheral vision, as being like a formal concept, like mapping. And giving a name to that. You know what I mean? Like you do the mapping, but you're not thinking about: how do I generalize over it? And I think that that for me, certainly in my journey with functional programming, I thought that it was mostly about functions. Not to say that it isn't, but that was kind of the end of the story. It's like, keep your functions pure so that the outputs are only dependent on the inputs. And away you go. And understand closures and higher-order functions, functions that return functions or take functions, and that's it. But I really feel that that's only half the story. TARAS: Part of it I think is that for people, even if you look at the kind of content that's available around functional programming, it tends to be – trying to kind of [reach] people into this idea of thinking of map, filter, reduce, kind of operations. And I think that's a place where everybody starts. But I think what happens is that you really are missing – and I think for most people. And it wasn't for me, it wasn't until you wrote the readme for Funcadelic and then I read it – up until that point I didn't really, I was also the same. I didn't know how these things were related to each other. Because there's this history and wealth of conceptual depth that connects all these things together. And these things are well-understood by people who don't – they're probably not writing JavaScript on a daily basis. They might be like Haskell programmers or Lisp programmers or ClojureScript or something like it. In other worlds, not JavaScript world. So there is all this understanding about how functional programming works but I don't think it's really leaked to the general masses of the JavaScript community. CHARLES: Yeah. TARAS: You know? And it wasn't until I started reading the readme – I'm like, “There's so many answered questions that I didn't even know these questions were asked.” You know? CHARLES: Yeah, yeah. Yeah, no. And I think you're absolutely right. It isn't accessible to the general JavaScript community. And part of that is because one person's – like when you read about these things, when I would go read about these kind of higher-order concepts, of basically classifying functions, not just of saying, “Yeah, what is the essence of a map operation? What's the essence of an apply operation?” you know, “What's the essence of concatenation?” things like that, I go read the documentation in Haskell or in Clojure. And first of all, it's hard to distinguish when you're not programming in those day-to-day, am I reading reference documentation or explanatory documentation? But even in the explanatory documentation, they're using what seems like incredibly self-referential and abstract examples. And I don't think that's necessarily a knock against those communities. I think what it is, is what's concrete to one person is abstract to another. And it's like, if you're working with those things, you're working with those sets of analogies, and then you're working with those abstractions every day, then they're concrete to you. In a sense that once it clicks in your mind and your mind kind of accepts it and rationalizes over it, then it moves from being, “Yes, it's an abstraction. But it's a concrete abstraction,” in the sense that you can have a conversation with somebody and use that abstraction as an example, as a counterpoint, and a method triangulate and reveal other abstractions. But if you're talking to somebody for whom those abstractions haven't clicked yet, then it's just, it's opaque. And it's not helpful. And so, I think that one of the things that I realized is like we are using these abstractions, we just don't have names for them. And so, I wanted to give them names and put them in the hands. And the names are weird, but they are really useful. And so yeah, maybe we could talk about some of those right now. Because I think that maybe now's a good time to actually introduce some of those abstractions. So for example, if you don't know what a functor is, it's worthless to talk about a monad, in my opinion. So, that was critical piece of information for me. Because that is like missing in every monad tutorial you ever read. At least, I must have read a thousand monad tutorials. And they kind of glossed over functor or didn't mention it. Whatever. Maybe they did but I wasn't looking. And that needs to be put front and center, that there is a natural sequence to these things. It's like, some of these abstractions are built on other abstractions and you have to – you can't skip to monad. You have to start with functor. Again, I realize that's probably gibberish gobbledy-goop to a lot of people. So really, this is what Funcadelic is about, what this conversation is about, is just saying – talking through it in real-world examples to make those abstractions concrete, so it doesn't feel strange anymore. TARAS: Yeah. It's definitely giving names to things. I think it's really helpful. One thing I really like about Funcadelic is that you're not giving names that you made up. These are names that existed for 50 years. They're historic. And so, when you talk to somebody who is familiar with functional programming and if you say ‘functor', all of a sudden, I feel much smarter. And we're actually referencing the same thing, because we are – because alternatively, you can say something like, “Something that is mappable,” right? Like a functor is essentially describing something that you can map over. CHARLES: Right. That's all it is. TARAS: Right. But you know, having a name for it, it allows you to just describe it exactly as it is. CHARLES: So yeah, there are tons of things that you can map over, right? Most of the time, we think about arrays as something that we can map over. TARAS: One thing I found really interesting in starting to use Funcadelic is that when you start thinking about things as they are like abilities – like you know with an array you can map over an array. It's something we've all been doing for a while – but then something that you end up doing a lot of. When you get familiar with an array, being able to use object map, mapping an object, becomes something you want to do at some point. Most of the time, what happens is that you're like, “Oh, I don't actually – well, I have to write this thing. How do I write this thing?” and then by the time you do this, you're like, “I'm just going to go to Lodash and I'll just get the map thing that will map an object.” At the end of the day, it doesn't quite necessarily feel right because a lot of these libraries – like, Lodash has map but it feels like there is always some kind of a compromise with how these things are implemented. It's not consistent. CHARLES: Right. TARAS: And I don't remember exactly what the problem with Lodash map was. I know for a fact there are, like there's different ways that you can map things. There are different functions available for mapping different things in Lodash. CHARLES: There's ‘map to' and ‘map in' and blah, blah, blah, blah, blah. TARAS: Yeah. All those different variations. But I think it's been really interesting. We've been using Funcadelic on a project we've been working on, on microstates. And just being able to use one function map that is going to map an array or is going to map an object and it's going to do it the same, use the same function, and it's going to do the same thing. CHARLES: Yeah. You have one function that maps over the values. And that's the thing, is you realize you can map over arrays. You can map over objects. You can map over observables. You can map over promises. You can map over trees. You can map – there's literally thousands of things that you can map over. And realizing that all of these fragmented interfaces can be coalesced into a single interface. And so, it really is, I think the biggest thing is like the power of polymorphism for a function. Because that's basically the problem that I think – it's not I want to say basically the problem. I think it is a problem that a lot of the functional programming libraries suffer from, is that the only polymorphism is object polymorphism, which is kind of the native polymorphism in JavaScript. Whereas in systems like Haskell and like Clojure, you can have a function be polymorphic. And so, one function can operate on many different kinds of data, provided it has an interface. So, when we're working in microstates, we use literally one function: map. The same, the actual same function, reference to the same function we use everywhere when we map. And we're just mapping a bunch of different things. So, I think that that's one of the reasons that I prefer Rambda, for example, over Lodash, is because it has a form of polymorphism. Most of the things like maps and lenses and applicatives and stuff, almost everything works on both objects and arrays. That's actually kind of nice. So, Rambda has a basic polymorphism. But I think one of the other things that is really empowering about Funcadelic is that it allows you to make the map function work on any data structure that you happen to come up with. Anything that you want. You can make it mappable and map will work on it. TARAS: Yeah. I think for people, it's probably quite abstract, what that actually looks like. I think one thing that's interesting is that – so for listeners, the way that would look is you have a class, you have an ES6 class, which gives name to a certain piece of data that you have in your application. And then what you can do with Funcadelic, you can then say, “This is how you would implement – if you were to map this kind of an object, if you had an instance of this object, if you wanted to map that object, you can specify: what is the function you would use?” So, even though you would use the same – you would import map from Funcadelic and you would use that map to map whatever that object is and whatever type it is, but there's a way for you to say, “For instances of this type, I would like to use this specific implementation of a map to map that object,” but use one function to do it. So, it's going to use one function that you call. But you can specify, under the hood it can specify how the actual, what is actually used to map that instance. CHARLES: Right. And then that's nice, because then you can – anything that uses a mappable object, there's a couple of reasons that that's nice. Any time you can have some sort of higher-order mechanism that just requires that something be mappable, that it be a functor. And then it's really nice because then you can have higher-order operations that they just need something that's mappable. But you don't have to use that one shot of actually having a map function as a property of the object. You can actually, you can kind of define wrapper classes or whatever, that then introduce a unique way that this object can be mapped. So, you can have the same structure and map it three different ways. Whereas you're kind of constrained by that using normal OO inheritance because you have to have a map property on your object. TARAS: Yeah. There's something else actually, when you start thinking about this. For me personally, I think the first step when we start working with objects, having mappable objects, it was the first thing that was really helpful. But then I think really quickly, right after that, I think my second thing that I started using and I think is probably my favorite feature now, is append. I think it's actually – yeah, so append is an implementation of a semigroup but I think it's simpler. A simplest metaphor would be something like object assign. So, object assign is an implementation of a semigroup, except that object assign mutates the object that you pass into it. Right? CHARLES: Right. TARAS: It doesn't create a new object for you. CHARLES: Right. So again, getting to this idea that there are some universal operations. Like there's this idea that you can take two things – I don't want to say object because that's an actual concrete type in JavaScript – two things and you can smoosh them together. And I think there was actually – wasn't there literally a big controversy about this? TARAS: Yes. CHARLES: About like, array smoosh? I think this is nice – smooshing, right? But you can smoosh things together. And with addition, I can smoosh two integers together or two numbers together and get a third number, right? Or with objects, I can take two objects and smoosh them together by merging their keys. Or I can take two strings and I can smoosh them together and I can end up with a string that's concatenated. Or I can take two arrays and smoosh them together and I've got now an array that's been concatenated. So, what's interesting is these are all very, very different types that I've just described. And yet, there is some fundamental operation about them. And I think this is actually something that's bad about JavaScript is there's five different names for all those operations that we talked about, but it's really one unifying concept. And that might seem like a small thing, but when you have five different interfaces for one fundamental construct, that leads to fragmentation and you can't treat that data uniformly. And it ends up like, paper cuts, paper cuts everywhere. Whereas if you can unify all of these into a small, one thing, which is like we can append two objects, and then we're going to have an implementation of append for array. And behind the scenes it's going to call smoosh. And we've got a universal – we've got an implementation of append for object, which is going to assign the keys in an immutable way and return a new object. Or we're going to have an implementation of append for string which is going to concatenate the strings. You might even, I don't know a hardcore FP nerd would have to probably correct me because this is just totally conjecture, although I know it's probably a solved problem – is maybe you could say we append two functions together. And that returns a function which is composed, right? That might be a way that you could say – what would it look like? Is function a semigroup? Can we append two functions together? And maybe you end up with like a pipe or a function composition or something like that. But I think that highlights, when you have these universal interfaces, because literally I feel like most of the stuff that we have been working with, there's literally five, there's like five interfaces. And everything is one of these five interfaces. And it kind of flips you on your head, because the classic programming wisdom that I have certainly have espoused for at least the last 10 years is that you don't want to race to find abstractions. That's dangerous. Because you can get locked into the wrong abstraction, right? Wait. Let the abstraction emerge. And I'm a lot less bullish on that concept now that I've discovered these things, because that wisdom is cultivated in a world where there are [billions] of abstractions, if you're giving unique names to everything and the combinations between them. But if you are coming up in a world where there's five basic abstractions, then it actually pays off to ask the question, “Is this thing a functor? Is it a semigroup? Is it a monad? What would that look like?” It's a nice thought experiment that doesn't require that much investment. You can think about it for a couple of minutes. And usually, you can come up and say, “Yes, yes. It totally is,” and I can start using it. And now I've introduced this really powerful abstraction. Or you say like, “No, no it's not.” And then that information is just as valuable. And so, it's very low-cost to experiment with abstractions. And so, I kind of think of it as – I know I'm on a little bit of a rant here but this has kind of been a major revelation for me – is that when you have very few abstractions which you compose in myriad ways versus having a whole bunch of abstractions that can't be composed very much, the cost for experimenting with abstraction and making the wrong abstraction is several orders of magnitude lower. And so, you don't have to be as cautious. And you can actually use trying on abstractions as a tool, rather than a very, very high risky undertaking. And just to kind of close that thought out, I think that – I don't know if anybody else but me remembers the world before we decided we were going to make all of our web services RESTful – like when people first started building all these web services, we were just going crazy with the endpoints. And there was no rhyme or reason. Kind of weird arbitrary levels of nesting. Sometimes, you'd throw in an ID as a query param, sometimes you'd throw it in as part of the path. And then I definitely remember, it was probably around, I don't know, 2010 for me where I listened to a podcast where James Edward Gray was talking about S3 and ‘Was it a RESTful interface?' and the O'Reilly book. And it really clicked for me. And realizing that if you constrain yourself to thinking about your API at least as these fundamental operation of manipulating resources, and you were constrained to four verbs and everything, you want to have ID-based URLs and resources and as flat as possible, those constraints actually are very enabling for consumers of your API and for actually authoring an API. And I think it's the same principle at work here. Anyway, so I'll end that rant. [Laughs] TARAS: Yeah. I think it's, I think people could probably, for those who haven't been in programming as long as Charles has been, it's probably easier – Charles I think people could probably relate to what's happening with components now, I'd imagine. Because having components essentially look the same across every framework, they all have props and they all render, return some DOM, or some variation of that. But it's kind of the same thing. You take some data and then you return something that is going to become DOM. And I think having that as a rule for what a component is, you can then make really complicated applications using these fundamental building blocks. And then you don't have to – there's not really much thinking on, “Well, how am I? What interface is this component going to have?” Okay, well you know it's going to accept props. And you know it's going to render some DOM when you actually render it, right? CHARLES: Right. TARAS: That simplicity I think is really helpful. And I think it's one of the things that – I think one of the things about Funcadelic that I really like is there's kind of a really small set of rules that are really helpful. And these rules are actually, they make it predictable. Because one thing that I find really challenging with using Lodash or using Rambda is that because there are so many functions, it's difficult to know what is actually going to happen when I do something. So, a good example would be like if you use omit from Lodash, Lodash omit will then, it will actually – one thing you can do is you can materialize your getters. So, if you have getters in your objects, those getters can become values on the new object that's created. So, that's one thing that could happen. Or if you use omit, your symbol, if you have values… CHARLES: What does omit do, by the way? I'm actually… TARAS: Omit is a pretty popular way to exclude functions. Basically like a filter equivalent for an object. It's usually used to remove some props that are coming into a component. But it can do some – it can actually change the type of the object. One thing you know for sure is if you use something like omit or if you use assign, if you have an instance of something, guaranteed, working with that object in a mutable way is going to cause some really strange things. I think with omit, if you were to have an instance, it would definitely not give you an instance of the same type, like an object of the same type. It will give you just a regular [inaudible]. And there is no real way – you could create an instance. I don't know what assign would do. I'm guessing that it would just take an instance and would put things on it. But it's really not – I think this kind of ambiguity doesn't work very well when you're trying to build something, when you really just need to know exactly how your tool behaves. I think that's one thing that with Funcadelic, because it's such a small API and because you know for a fact that the library is designed to be immutable and it's designed to preserve type information, then you know that if you use one of the operations, you will get most likely the same kind of object. And you're not – well, you will the same kind of object and it's not going to mutate that object. And so, there are some of these guarantees that are actually really helpful and [inaudible] is liberating, I think. CHARLES: Right. TARAS: Especially when you're trying to do more challenging things, not trivial, just copy some properties from one object to another. But when you're actually doing more sophisticated things, in those use cases, having these kind of rules is extremely powerful. CHARLES: Yeah. And I've definitely resisted and I think will continue to resist expanding the API surface area that much. Because it is, I think there's only five or six functions in there. But what you get for those functions is extremely powerful and extremely predictable. I think it might help to give a concrete example. Like when you were talking about object assignment. Like if you have a car class in your application and you want to do an immutable object assign, well the kind of stereotypical way to do that or the typical way to do that is you assign. You create an empty object and then you assign one car to it and then you assign the next car to it. And now you've merged these two car things, right? But then the problem is, you're going to get an object out of that, not a car. It's just going to be a vanilla object. Now, it's going to have all the properties of both, but it's not going to be a car. And that could be a problem if you've got any custom methods on the prototype, any computed properties on the prototype. It's going to be a problem. Whereas Funcadelic, you can append two cars together and you're going to know that it's going to have both of the properties. You're going to know that it's going to be of type car. And you don't have to worry about running the constructor or anything like that. It's just going to have – the properties are going to be carried over properly. If you're using a library like Lodash or Rambda or something that doesn't account for type, because in order to append two things or map something, the implementation actually lives with the type, not with the function. The function is polymorphic, but the implementation lives with the type. You can then actually, you can always return the proper type. Because it's ultimately – like if your map operation or your filter operation or your what have you operation doesn't take type into account, then there's no way to actually preserve type. But because we delegate in Funcadelic, it's core to the concept. And so, it's actually a very trivial thing to do, which is why you get that repeatability. TARAS: One of my favorite things about append and semigroup implementation for object is that you can overload getters on an existing instance. So, what you get is you get a new instance with the getter that you passed to append applied to that instance. And this is kind of trippy but it's actually really powerful because – so, let's say you have an instance of an object and that instance has some getters on it. And those getters use some properties of this object to compute their value. And so, when you want – if I need to create a copy of that object in such a way that the getters still continue to work properly but I need to override how one specific getter works for one specific instance, one specific scenario, one specific use case in my code, then I can just use append. So, the first argument is the original instance, second argument is an object that has the getters that I want that I want to overload the getters on the original object. And then append will squish those things together, smoosh those things together, and give me a new instance that has the getters that I passed in, in the second argument. And all the same things that the first object had. And that object will work. This new object will be a fully-functional object just like the original object that I still have a copy of, that I can use. But this is really interesting because one thing that I'm finding with having these kind of tools in my toolset is that I've had features that I needed to implement on a project. And the people that I work with are really technical. So, they know where problems are going to be. And so, the would write requirements for how something – for a feature that needs to be implemented. And knowing the problems, they're like, “Oh, by the way. You're going to have a problem in this area when implementing this kind of specific functionality.” And for me, I'm like, having this toolset, I'm like, “I don't see it as a problem at all.” It's so easy for me. Because I know that if I need to implement – so if I have something that has expected behavior, but I need to create something that behaves very similarly but slightly different in one particular use case, I can always just copy that object and then overload its behavior. And it's still going to be a fully-functional object. And I think that alone is just not something that you can do usually. It's not something that's available. CHARLES: Yeah. It's a technique that I think was discovered. Maybe it's not original to us. But it's just, the tools enabled it in the sense that it's like having a flashlight in a dark room. It's like, that technique was always there. It's just when it becomes so concise that it's so easy to just append one more computed property to a thing, then you just wouldn't have thought to do it otherwise. TARAS: What's interesting about this too, for me, is that – and this goes back to the context conversation that we had earlier – is that I think React brought into our lives functional programming kind of, in a big way. Because part of programming React applications is working with functional components and working with functional concepts. And a lot of the things, like Redux, a lot of these things are powered by functional programming. And they work together. They compose well together because they're all from the same paradigm. But the problem is that there are all these concepts developed together over time. And they've been tested together and they've been formulated together in languages like Haskell. But the ideas that make all of these stuff work really well together, they haven't really become available in the JavaScript community. And it feels like, it's like we're all using a language that we don't fully understand. And it's like we're all – a lot of people in the JavaScript world who works, when it comes to functional programming, it's kind of like having English as a second language. It's like, you can use the words but you don't understand the humor. And it's kind of like, you can't make a joke. It's kind of like that. You can't really express yourself correctly when you don't have full grasp of the language. And I feel like how we use functional programming in JavaScript is a little bit like that. And by starting to bring these ideas, moving the wealth of knowledge from the source into the realm where we actually need to use it now, we can actually start to take advantage – leverage all these insights that actually enables all of this. It's not like – so, the idea of how to write React and think in a reactive way or think in a functional way, those things are not just owned by the React core team or they're not owned by an elite group of developers who really understand how functional programming works. It's just available to everyone. And all you need to do is just learn some of these concepts that glue all these ideas together that are fundamental pieces of how functional programming works. CHARLES: Right. That's a great point. And it points back to kind of the original reason that I wrote Funcadelic and then started and then continued to work on it with you, is that it really was – it was actually meant as a – it started out as an educational exercise. What would these things look like if they were translated into JavaScript? And it turned out that it rapidly became core to my workflow and way of thinking. And so, it really is, there are weird names to these universal concepts. It is true. It is a foreign language. I really like that analogy. But foreign languages sound weird, and when people are talking in a foreign language, you can feel excluded. And the really, the reason that we wrote Funcadelic and the reason it's there is to make them accessible, these things accessible to you. So that those abstract foreign words can over time turn into concrete concepts that you're completely comfortable with, just like any word in any language. At some point, you approached it having no idea what that sound represented. And so, it really is trying to – the emphasis there is not to noodle about and dwell on the names of the concepts but to take the real things that you are actually doing, give them names and formalize them, to enable you to participate in this new functional world that you're describing. Because I love that sentiment. It does not belong to the React core team. It doesn't belong to an elite set of developers on this project or that project. It literally is a universal tool that is 100% achievable. And people don't even realize how close – if they've been working with JavaScript for a couple of years, how close they actually are. TARAS: Yeah. I think they're really – for most people, if they were to read the readme and then – well, I think one of the problems, it kind of works with the language metaphor, is that you need an immersion, right? I think one of the reasons why Funcadelic really stuck and functors and semigroups and [filterable] and all of these things really stuck for me, and I'm thinking about how using monads and monadic operations or applicatives and all that stuff – the reason why it all stuck for me is because I've been able to talk to you about it. And I think for people, finding a network that will allow them to practice immersively, to think about functional programming, not just occasionally – I mean, you could do it occasionally as well, it just takes much longer. But once you really, once you have a few conversations where you try to dissect a problem in a functional way and you think about what these pieces are made of, it becomes very natural, very quickly. CHARLES: Yeah. I think that's actually a really great point. And the thing is, you can immerse yourself incrementally. So you can just say, “You know what? I'm just going to start using append.” Anywhere that I would concat two strings or I concat two arrays or I would do object assign – screw that. You can even just say, “Instead of using object.assign, I'm going to use append,” and start there. Or to say, “I'm just going to start with mapping.” I think also the thing that's nice about it too, is the buy-in can be incremental. But you're right. You do need immersion. You do need practice. You need to actually use, you need to use the functions and you need to be able to use them one at a time, so that your mind can close over them. So then, you can kind of move onto the next one. TARAS: Yeah. CHARLES: So, that might be one way, is to say – because you know, I don't think I really understood. I was already using append and map ubiquitously before I really understand applicative/apply. So, you don't have to grok it all at once. You can definitely bite it in chunks. And the best way to do that is to start with one concept and really just attack it mercilessly. And then also understand that there's a natural sequence there. TARAS: I would add a little bit of a caveat to that. I think there's a thing about using – doing something for learning purposes and there's another thing about shipping things. What's interesting with Funcadelic and what's interesting about a lot of these ideas from functional programming is that I think they give you benefits that you might have not previously thought. Like for example, if you're going to concat two strings together, doing it with append is probably the most robust way of doing it, relative to just being able to use [inaudible]. CHARLES: Yeah, that's true. You wouldn't want to use a string [inaudible], wouldn't you? TARAS: Yeah. But there are areas when using – there are times when things are just not possible otherwise. Like for example, if you wanted to treat an instance of a class in an immutable way, this is simply not possible in any way. So, if you're going to say, “I'm going to work with a bunch of these instances of ES6. And I want to keep them as instances, because they have certain behaviors that I want to have. I want to have a getter, or I want to have a method on it. And I want to keep these things fully full instances, not broken, not be turned into objects. I want them to be normal instances but I want to work with them immutably. When I need to make a change, I want to get a new object and not modify that object.” So, if you set yourself that goal and you say, “This is what I'm going to do,” then you really are not left with very many options. You only really have, you have to use append from Funcadelic. And because alternatively, you're going to implement something yourself. You might as well just use append. And [I think] that's a good place. I think if you're starting to, if you need to make something lazy, if you need to delay an execution of something – so, instead of pushing that execution, instead of using object assign and then computing everything ahead of time, you can use append. And you can create a getter and you can delay the computation of that value until the point when the user actually reaches for that value. If you want to start doing that kind of stuff, you really are not left with very many options. And append is the way to do it. But that's the thing, is when you start to set these kinds of standards for yourself, you level up in a way that is very significant. I think it's like a natural progression of learning. You start off learning and anything goes, as long as you can make this website work, it's like, “I'm happy.” And then over time you get better and better at it. And then when you get good at building applications, your next step might be like, “What if I was stricter about it? What if I could actually – what would that open up for me? What would that allow me to do?” I personally think about it that way. CHARLES: Yeah. I think that's a good way to think about it. You mentioned having a network for discussing these concepts and trying to internalize them. Let me first and foremost offer myself as somebody. If this is a hill that you are interested in climbing, and I think it is a very worthwhile hill to climb because of the perspective that you will gain from its summit, please reach out to me. You can contact me at cowboyd on Twitter or cowboyd@frontside.io. I'd be happy to discuss these kinds of things, because I think that these tools are just incredibly powerful and will improve you. So, if folks want to get in touch with you, Taras, where would that be? TARAS: I'm tarasm@gmail.com and tarasm on Twitter. CHARLES: Alright. Well, thank you everybody for listening. And as always, if you want to get in contact with us at Frontside, you can just email us at contact@frontside.io or give us a shout on Twitter at @TheFrontside. Thanks everybody. We'll see you all next time for episode 100.

This Frontsider panel episode explores what virtues go into making quality software, such as having tests, making sure software is performant and accessible, and why you should try to avoid technical debt. Transcript: CHARLES: Hello everybody and welcome to The Frontside Podcast Episode 98. My name is Charles Lowell, developer here at The Frontside and your podcast host-in-training. With me today, we're going to have a round table, a Frontside round table. With me today is Elrick. ELRICK: Hey. CHARLES: Joe. JOE: How you doing? CHARLES: And of course, Will. WIL: Hello, hello. CHARLES: Welcome, y'all. We're going to be talking today about some of the things that we do around here, aside from trimming the shrubs and making coffee and snacking on Altoids. Like, way too many of them. Yeah. I was thinking we could talk a little bit about software qualities of relative things, like this software has these qualities. And I think that that kind of lofty goal of software quality is comprised of having a bunch of little qualities. The quality of having fewer bugs or the quality of having these things. And so, talking about all these things that we do and kind of what we do to make sure that we continue to do them. Or the ways that we can ensure that our software has these things. So yeah, we can just start really anywhere. WIL: Yeah. So, one core thing is obviously tests. CHARLES: That kind of falls under we want to have – really, there's two qualities there that we want, right? Is we want to have… WIL: Maintainable software. CHARLES: We want it to be maintainable. We want it to be resilient to change. And we want it to work properly, right? Yeah, so we put tests in place to make sure that that happens. JOE: Tests also inform design in a really positive way. A lot of the time, anyway. WIL: Another thing that we like to include in our apps is responsiveness. CHARLES: Yeah. And just making sure that you have – that it works on a multiplicity of devices, right? WIL: Yeah. And not just the devices, but browsers as well. CHARLES: Yeah. And it turns out it's actually really hard to do that after the fact. WIL: Right. JOE: Yeah. CHARLES: Making sure that lots of browsers, lots of devices. Because yeah, sometimes you have some weird screen width that is on some weird device, and making sure that that works. I guess there's some overlap with testing there, too, isn't it, right? Like you want to be running your tests on those devices at those resolutions to make sure that they're going to work. This is something that we aspire to but I don't think we're quite there yet. It was making sure that our applications are accessible. WIL: Yeah. JOE: I'm very excited to learn more about this as we get into this, yeah. CHARLES: Right, right. And asking the question, how is it that we actually can ensure our applications are accessible? We have very paved roads for making sure that our applications are resilient to change and that they have low bug rates and that they're well-designed via testing. But what is the analog of testing for accessibility? What's the way that you can put those guardrails in for accessibility? I have no idea. And that's an ongoing conversation here at Frontside. JOE: So, I guess I'm curious as to what technologies are actually involved in accessing a web application in – would it be reasonable to say a non-traditional way? I know there's such things as screen readers, but is that all we're talking about? Or what is the ecosystem that we have to consider supporting? CHARLES: I'm certainly not an expert on this. We'd have to get Rob in here to chew our ears off this. JOE: Yes. CHARLES: But from what I've picked up from him and from our conventions with Marcy Sutton and some other folks that we've had on the podcast, it's a big umbrella. So, it's anyone using an application in a non-traditional way. So, whether that can have to do with limited vision, hearing, movement, range of movement, cognitive ability, it's a gigantic whale of a domain. WIL: Yeah. The topic of accessibility can definitely be several podcasts on its own. CHARLES: Yeah. One thing that we've talked about is it would be great if you could drive your test suite through a screen reader or something like that. What would that even look like? There are a couple of open source ones out there, but they're Windows-only. I think it was NVDA was the big one. And then you have a screen reader that then drives the applications in your operating system, so it's going to vary per operating system. So, making sure that it's accessible on Windows, at least as I understand it, is very different from making sure that it's accessible on a Mac. JOE: Yeah, it's like a whole other layer. And it's like BrowserStack outside of the browser. CHARLES: Right. WIL: There are things that you can do from the beginning that will make it easier when you get to that point. It's just like using semantic HTML, knowing when and how to use proper aria labels. All these things, if you do it from the beginning, it's not as big of a task as bolting it on afterwards. CHARLES: Right. And I think we do have a leg up when it comes to web applications. It's within our power to change. There are cross-platform of those technologies. But as you said, it's important to put them in from the beginning. Because as we've seen, for each one of those categories, you're accumulating debt if you don't address it. So, there's technical debt. But I think that technical debt can [inaudible] into a bunch of different areas. So, there's technical debt in terms of the internal quality of your architecture, the way your software components talk to each other. And I think that that's what people mostly think of when they talk about technical debt. But I think in terms of responsiveness debt, there's a slice of the technical debt pie that has to do with making your application responsive. And so, if you don't address making your application responsive, you're accumulating debt and you might not know it. And if you're not making your application accessible, then from the beginning you're accumulating debt. So that if you have to go and try and figure out your accessibility story six months, a year, two years, you might actually uncover and say, “Whoops. I've been swiping the accessibility credit card. And holy crap, with all this. All my fines and penalties and compounded interest. Now I'm accessibility bankrupt.” And that can be scary, right? WIL: Yeah. And a lot of people don't realize with all this debt after the fact is they think they're going in and adding things like responsiveness and accessibility and tests. But really, you're also taking away previous work that's already there, things that need to be refactored. If you put these things off, you're not just adding a few hours of time. You're inflating your time exponentially. CHARLES: Right. Right, exactly. It can be intimidating but I think it's also empowering, because technical debt is like a scary subject. But if you're like, “Oh, we can actually slice our technical debt into a bunch of different categories and address them individually,” just knowing that this is an area where debt can accumulate, that's half the battle. Because the worst thing is debt you don't even see. ELRICK: Yeah. WIL: I mean, [inaudible] is big. That's a big part of accessibility, that, is most people don't think of accessibility. So, that is a huge debt that a lot of companies don't see. JOE: What about something like internationalization where I feel like I've never been in an application where that wasn't punted on to some degree. That's kind of a well-known problem, but it still takes a back burner. Do you think that if accessibility had more exposure as a concern, would it actually get the attention it deserves or is it kind of destined to, “Oh, we'll get to those yaml files later. We'll send those off for translation later,” that type of thing. ELRICK: I don't know. Sometimes I feel as though people feel as though they're trading speed away when they're building applications when they go to implement these things. Like, “Okay, well we're not really going to touch on these right now because that's going to slow us down from pushing out features.” Which is not really true. Because if you don't settle on these things early, you're not really building a solid foundation for your application in the long haul. So, I think people are like, “Oh, we'll just do it later.” CHARLES: Right. ELRICK: And, “We'll just ship features now.” CHARLES: Right. I think that's exactly right. It has this kind of secondary effect where not only do you develop the debt but you develop a culture of accumulating debt, right? Like when it comes to people getting a hold of their finances, the first thing that they have to change is they have to change their spending habits. And that can be the hardest thing. It's not just balancing the equation. It's like saying, “I need to readjust my thinking about this.” ELRICK: Yup. CHARLES: So that I'm not consistently put in this situation again. JOE: So, there's an operative word there, right, in personal finance in that usually if a company is addressing technical debt especially down the road, something that they've punted on for a while, it's far from personal. There's a board of directors or there's a special interest group involved. There's people who want features that are putting money into it. There's a lot of pressure as the company grows and more people are involved. Priorities are more likely to be lost, I guess. CHARLES: Are you saying it can be hard when your culture is spread over that many people, it can be hard to shift? JOE: Absolutely, yeah. And I guess to keep with the dash-first thing, ideally were we starting a company, we would want to start a culture for this company. A culture that recognizes the vulnerability that we all have to technical debt as applications grow. We want that upfront. But the reality is, you know, startups are eager to get things out. Companies that have been around for a long time have high-paying clients that they depend on that want certain things. And yeah, I guess I'm just saying that it has to come in from the beginning. CHARLES: Yeah. And I think that – I don't want to completely disparage technical debt entirely, because technical debt like actual debt, like financial debt, is a powerful tool that you can wield. But it's also, it's like a table saw. You can also easily slice your finger off. It doesn't mean that it's not a useful tool, right? If anyone's bought a house, it's really great that you can borrow money to buy a house. It's great that businesses can borrow money and get small business loans to get bootstrapped. And that benefits us all to have that community. I don't think that – yeah, startups definitely, they need to have technical debt as a tool that's available to them. But they just need to understand the consequences of it and be able to get a hold on it. JOE: That's a super interesting take. I never considered it that way before. CHARLES: Yeah. It's definitely not my take. I actually think the person who coined the term ‘technical debt', that was the original idea. But then people realized that technical debt can also get way out of hand. WIL: It's just like real debt. If you're not paying down a certain amount every so often, it's going to keep growing. CHARLES: Yup. You're going to have to declare bankruptcy at some point and throw out the piece of software if you don't pay a down. And that's going to be more expensive. ELRICK: Yup. That's definitely true. So, I have a question. And we see this all the time repeating itself at various companies, whether it's a startup, a large company, where they put off testing and mobile-first, user-first, accessibility-first. Like all the firsts, they just toss it to the side. Why do you all think that that happens so frequently? CHARLES: I think it comes into people not understanding that if you don't address it from the start, it won't happen naturally. There is a prime motivator that has to happen. If you don't imbue something with those qualities when it's tiny, when it's a tiny seed, a tiny crystal, you're going to have to drill through layers and layers and layers of core to put it at the crystal to begin with. I like to think of software as kind of like a tree. And we eat the fruit of the tree, and that's the features that users use. And we can tell that a fruit is delicious merely by placing it in our mouths. And we can tell what fruit is bad. But we can't really look at the fruit itself to say what caused this fruit to be good, what caused this fruit to be bad. We have to look at the tree. And I think that that's what people miss when they're developing software, is that what you really want to do is you want to build a tree that builds good fruit. You can't just take the fruit off the vine and say like, “Hey, I've got this peach but it doesn't have enough sweetness. So, I'm going to take a syringe and I'm going to inject glucose around it and make it less tart.” You say, “I want a sweet fruit,” right? ELRICK: Yeah. JOE: You could probably actually do that. CHARLES: You could. And that might be a strategy. And we see a lot of software that has those qualities of, “Oh, we're going to make this accessible,” or, “We're going to try and make this beautiful.” I happen to think that pigs are adorable animals and look great in lipstick. But that [laughs]… you could put lipstick on a pig but people can tell. And you can say, “Oh, this peach needs to have softer fruit,” and you can whack it with a mallet to actually make the meat more tender. But people are going to be able to tell. So, what you really need to do is you need to care for the peach tree rather than worry so much about the fruit. Because if you have a healthy tree, then you will have healthy fruit, right? ELRICK: Yeah. So, you want to plant good seeds. JOE: Yeah. WIL: Back to you question, Elrick, about what motivates startups and other companies to put off these things. I think the biggest thing is just time and money. They have this misconception where they're saving a little time and saving a little money now just to add it back later. But in reality, it's going to cost them tenfold time and money for adding it later, versus just spending that little bit of time and money and all that to begin with. CHARLES: That's true. JOE: It could also boil down, as far as just personal intimidation. Not so much like a business side of a thing but maybe just, think of all of the things that you listed, Elrick. It was almost a dozen dash-firsts in there. If you're sitting down at a startup that you started with three friends and just approaching these things for the first time, that's a lot to tack on right upfront. It's intimidating. CHARLES: It is intimidating. I think my message to those people is I've felt intimidated by that. I think my message to those people is like, the nice thing about it is if you attack those, if you tack all of those things from the get go, the features will take care of themselves and feel more effortless as you go on. You say like, “Oh, well actually, I don't worry about a high rate of bugs.” I want to say recidivism, but that's not the right word. A high rate of return, not on money but on – or high rate of bouncing your users. You don't want that. And if you bake that in from the beginning, parts of the software development cycle that were stressful before just aren't stressful anymore. So, if we say, “We want to have a system that is easily maintainable, well let's put that in from the very beginning.” We say that a lot. We deploy to production on day one. But what that means is, we say we have this value that we want the system to be easily maintainable. And so, we're going to do it from day one. That means that we actually – it's not something that we worry about so much on down the road. Whereas that used to be very stressful. I don't know. I remember when I started my career, there were these long release cycles where every six months, you'd release software. And the last month was just absolutely terrible as you try to stand this thing up and get it into production and then realize it's not monitored. There's no one checking the health of this thing. So, it's pissing off users at one in the morning. And… WIL: Beepers. CHARLES: What's that? WIL: Beepers. CHARLES: That's actually a great – there's a story there. The one time I got a beeper, I went canoeing in the canals of London and I tipped over my canoe and I dropped both my cellphone and the beeper that they've given me. ELRICK: What? CHARLES: I never got put on pager duty again. [Laughter] JOE: I'm going to use that next time [inaudible] with an on-call position. That's a good move. CHARLES: I remember, I definitely remember how sour my manager's face was when I turned [inaudible] the cellphone that was like, dripping with water. JOE: He was eating bad fruit, probably. CHARLES: Yeah. [Laughs] So, the other thing is we like to build beautiful applications, right? So, you have to – that match the user experience. You have to spend that time on design and beauty upfront. You will not have a beautiful application after the fact. You just need to bake it in. ELRICK: And accessible design. CHARLES: Exactly. ELRICK: Don't forget that one. CHARLES: Don't forget that, right? A responsive design. WIL: Yeah, accessibility-first in design. Yeah, responsive and all that starts in design phase, yeah. CHARLES: Yeah, all that, right? So, you want a great experience. You want an accessible experience. You want a responsive experience. You want a quality experience. You want a performant experience. That's another quality that you say. Like, “We're going to make sure that this is performant.” If you want that – and that's something that we're not always great about, right? We don't actually put in benchmarks for our software from the get go. But maybe we should. But there's perhaps a hidden cost there that we might be actually accumulating performance debt that we don't even know about. JOE: That's true. ELRICK: Interesting. JOE: So, things that pop up that are new. Like, accessibility wasn't probably always a thing in computing. Internationalization probably wasn't always a concern. Beautiful certainly wasn't a concern if you look on Wayback machine. You will see that to be true, right? [Laughter] JOE: So, all code is tech debt, I would argue. Or at least has the potential to be. And yeah, as the ecosystem as a whole evolves, being responsive to that, having plasticity in that respect, sort of like meta-first. CHARLES: Right. JOE: That could be the real challenge. WIL: Yeah, Charles is mentioning all these experience things. And so, I was thinking X-first is simply experience-first. You want you users to experience a certain quality of your app. That experience needs to start in the conception phase. CHARLES: Yeah. ELRICK: That's true. And even your developers coming in, developer experience. JOE: Yeah. CHARLES: Right. And I think the core of that X-first, that experience-first, is you need to pick which experiences. Because you can't have everything. JOE: Right, yeah. CHARLES: One, there is going to be too much. You have to say, “I'm going to sacrifice on knowing that this is a performance thing. I'm not going to include that in the core DNA of my application.” And there's just going to be things that you don't know about yet that are just unsolvable problems or that don't necessarily work. And you can say, “You know what? Hypothetically, I'm not going to make this an accessible – I'm not going to focus on accessibility.” But then you need to own that. And you need to know that you're accumulating a huge amount of debt around that. And then I think that is a particularly bad trade-off because someone's always going to come along and you're going to have to know that your application is accessible. I think once we clamp down on that, that's going to be something that we have a strategy for and we include at the beginning on every single application, right? ELRICK: Yeah. CHARLES: But I think you need to have, almost like holding the cards in your hand, say, “These are the cards. These are the X's that I'm going to have in my hand. And they are going to be core to my app.” And they're going to be part of the DNA of that tree. So that I know that the fruit is then going to have those qualities. JOE: And then you as an engineer, that goes through an iterative process as well. Just starting out, you have no idea what that DNA should look like. And short of learning from people who are wiser than you who are around you, and reading blog posts and whatnot, really the only way to know the pain of strong-arming internationalization for instance into a 15-year-old Perl application, is to go through it. And then, you know, future trees will not have this DNA. CHARLES: Right. Right. And that's the other thing. Is if you are going to include, if you are going to try and splice something into the DNA, there's a lot of work. And you just need to go for it. You acknowledge that it's going to be a lot of work. And you need to, you just need to own it and go for it. And pay that expense of actually getting it deep, deep, deep into your application's core values. So that then, you don't have to worry about it anymore. Otherwise, you're going to be paying – you're just basically signing up for a lifetime of debt. Right? WIL: Yeah. And then to make the debt analogy even more, it's like people don't understand the total debt. The end debt. People get a $30,000 loan with a 4% interest and they think they're paying back that $30,000 loan. But really, they're paying back $36,400 after all the amortization of their interest. The debt is higher than you can see, always. CHARLES: Right. WIL: And it's true in tech debt, too. React is the new hot thing now, but in 10 years we're going to be on React debt that we're migrating away from. JOE: I hope so. [Laughter] CHARLES: Maybe less, I think less than 10. WIL: Yeah. The debt is always there. And people don't realize how much they have to pay on top of what's visible. JOE: Yeah. It's an invisible vig. CHARLES: What's a vig? JOE: It's interest, in the mafia. CHARLES: Oh. JOE: Sorry. Yeah. CHARLES: I forgot you're Italian. JOE: Yeah. ELRICK: So, for people that are listening, they might be in a situation where they need to advocate to the powers that may be these X-first values. What do you all think that some of the approaches that they should take to say to whomever it is that, “We need to do this first”? Because there's times where you might say, “Hey, we need to do this first,” and people just look and say, “Oh, maybe not.” Then you need to push back on that. CHARLES: In my experience, I find that the tech debt argument is a good one. Because I think it can be, it's both limiting and empowering. Because sometimes it really is the right call to pull out your credit card and put something on it. If you need to buy water and you need to buy food and you don't have any other means, man, put it on the credit card. Right? Seriously. Even if you have no idea how you're going to pay it back. Like, whip that sucker out and stick the chip in. And it doesn't matter how much it costs. And so, sometimes that is the right call. But I think draining it of a moral or a value as a human person thing, and approaching it from a business decision and saying, really trying to attach a cost to it. Because then I think if you can drain out the emotion of it, because people really want something. They're striving to go get it and trying, give them tools to think about it rationally. That I think is a good strategy, to just say, let them know that there is a debt that's being paid here or that's being accumulated here. And it's really large. And maybe even say, “Look, if we were to put this off by six months, this might cost not twice as much. It might cost ten or even a hundred times as much.” So, by saving $5,000 now, you might actually be accumulating $50,000 worth of debt. It's [bigger] than you think. But I do like – so, I think that's one important tool. But I think then also the other important tool is to say, “If we are going to attack this, let's drive it home. Let's put it at the core. Let's make this a value that we hold so that the tree can take care of the fruit itself.” So, if we say that we're going to put in accessibility – because not all projects are greenfields. JOE: Absolutely not. CHARLES: So, what's the message to them? Sorry. You're just SOL. I think if you're a year into a project, two years into a project, and you realize, “Oh no. We need to do internationalization,” recognize that that might be something that's – that's a pillar of your architecture. Or, “We're going to make this application accessible,” don't half-ass it. WIL: Weave it in. CHARLES: Say, “We're going to transform this. We're not going to add accessibility. We're going to transform what we have into an accessible application.” Or, “We're going to transform what we have into a beautiful application.” Otherwise… WIL: Yeah [inaudible]. CHARLES: I would say leave it ugly and focus your efforts elsewhere on things where you do have your values straight. Because you're never going to have everything in line. JOE: No. WIL: Treat software like immutably. You don't add something to it. When you want to add accessibility, you're creating a whole new accessible app. ELRICK: Ooh. That's deep. CHARLES: Yeah. JOE: So, having seen – I don't know. I think it was very apt, looking at it as a business decision. I've seen it go the other way. Because at least among engineers and people on the technical side of it, this can become a very strong moral issue that people feel very strongly about. CHARLES: Because we have to live with the consequences quite honestly, right? JOE: Exactly. And that's a hard thing to translate to say an executive board that may be three levels abstracted away from you and is making those decisions. I've seen people attack or approach this I guess with that emotion built in, with the, “This is the right way to do it. Everybody else is doing it wrong.” It gets nowhere, basically. What needs to happen I think, so you talk about having this beautiful tree. But that also requires beautiful gardeners. And so, where the moral thing or the interpersonal thing comes in is there needs to be kind of an inclusive and encouraging environment that is fostered among the people tending to the tree. And that's a totally separate thing than selling the business value of it. Those things should be completely divorced. CHARLES: Yeah. It's funny. It's always hard to reconcile those two things, right? Because on one you have, “You have to take care of the raw consumption of material and the output of product.” But then also trying to – so, there's some baseline math that has to happen but making sure that that goal, it doesn't slice people. And can enable them to be happy and feel like they're doing good work. And that the things that they're doing is having meaning. It's probably an insoluble problem that we're going to be dancing around for as long as people are around. If there's one thing that we've come to recognize around here, and we've stated it many different ways from a bunch of different angles through the course of this conversation, and I would say through the course of this podcast, but that is if you want to see something in your software, make sure that you attack it from the get go. ELRICK: Intertwine it in your DNA. CHARLES: Exactly. And then you can actually experience the fruit, rather than trying to always, always trying to jam it and change it and get it into the taste you want after the fact. So, I guess that's it. Thank you so much y'all, for this conversation. I really, really enjoyed it. For those of y'all listening, if you want to continue the conversation, you can get in touch with us we are @TheFrontside on Twitter. Or you can drop us an email. We're contact@frontside.io. So, thanks Elrick. Thanks, Joe. Thanks, Will. JOE: Thank you. WIL: Thank you. ELRICK: Yup. It was great. JOE: It was fruitful. [Chuckles] ELRICK: Frontside-first. CHARLES: And well, we'll see y'all around.

Erich Gamma: @ErichGamme Dirk Baeumer: @dbktw Show Notes 01:11 - The Design Patterns Book 02:45 - The Eclipse Project 09:24 - Language Server Protocol: Overview 15:16 - What can you do with a server that implements the LSP? Incremental usage? 20:12 - Keeping the Tools in Sync and Refactoring Support 24:33 - Keeping it Performant 29:41 - What kind of proliferation of codesmart tools are there that implement the LSP? 34:51 - What are the challenges encountered trying to build abstractions that work for 40 different languages? Resources Visual Studio Code Transcript CHARLES: Hello everybody and welcome to The Frontside Podcast Episode 97. My name is Charles Lowell. I'm a developer here at The Frontside and your podcast host-in-training. And with me today, we have two very special guests. They have been working on technologies that have run very parallel to my entire career as a software developer. And we're going to talk about that. So with us today are Erich Gamma and Dirk Baeumer who are developers on the team developing VS Code, which if you're in the frontend space is taking that area of development by storm. It's just amazing, some of the things they can do. Lots of people are using it every day. Lots of people are trying it. And so, we're going to talk about the technologies that underlie that and the story of how it came to be. So, welcome Erich and welcome Dirk. ERICH: Hello from Zurich. CHARLES: Alright. Zurich to Albuquerque. Here we go. As a first start, I would have to say my first contact with this story, I at least have to mention it because – and this is for Erich – you wrote a book that was very, very instrumental in my formation as a young developer. I think I was about 22 years old when I read ‘Design Patterns'. And I don't know. I still carry a lot of those things with me to this day, even though a lot of things have changed about the way that we do development. I still carry a lot of those lessons, I think especially things like the state pattern and the strategy pattern, and stuff like that. I want to move onto other things, but I was hoping that we could talk just a little bit about, what are the things that you find still kind of relevant today? ERICH: Well, now as you said, some of the things are kind of timeless and we're lucky to have found these things. And I still love all the patterns. But I must say, things have changed, right? So, at that time, we thought objects are very cool. And as we have evolved, all of a sudden we think, “Oh, functions are actually very cool, too,” right? Closures and so on. So, I think we got more broader and of course if you use functional programming, you have many more patterns available as you program. So, I feel some of the object thinking still applies. But that's not the only thing that counts anymore. Today it's functions, stateless, immutability, and all those things within functional programming which is [straight] and which [inaudible] in our team. CHARLES: Yeah, yeah. I would love to see an update to how do these concepts transfer into functional programming. But anyway, just wanted to say thank you for that. And it was about the same time that, a few years after, I don't know the exact same timing, I want to wind back. Because we're going to talk about VS Code but before VS Code, there was a project that both of you all worked on called Eclipse, which I also used. Because at the very beginning of my career, I did a lot of Java development. And it really opened my eyes into a level of what tooling could do for you that I didn't see before. And I was wondering how did you arrive to there? Because before that, I was using Emacs and Vim and Joe's Editor and things that were editing the text files. And how did you kind of arrive at that problem? Because I feel like it's very similar to the one that VS Code solves, but this was what, 15 years ago? ERICH: I think it's older, right? CHARLES: Really? DIRK: It's 17, 18, yeah. Yeah, yeah. It was end of the millennium, right? So to be honest, Eclipse wasn't the first development tool we worked on. Then, we worked on the company ObjectTechnologyNational. They worked on Smalltalk tools. And of course, Smalltalk had a great IDE experience, right? So back then, Java became popular. One idea was, how can you preserve the great Smalltalk coding experience? [Inaudible] CHARLES: Ah, okay. DIRK: [Inaudible] and find all references, method-level history versioning, and so on. So, that was the input that got Eclipse kicked off. And one idea we had at that time, Eclipse is our opportunity to make everything right. And as we have seen now, when we did VS Code, we could even improve what we have [inaudible] at that time. So an example, in Eclipse we thought plugins are very cool and we have kind of a microkernel. And you load all of the plugins in the same process, they have a rich API, and so on, which is great. But we found over time, if you have lots of plugins and they do bad things and they run in the same process, it's not the best thing. CHARLES: Ah. Right. And so… DIRK: [Inaudible] have a different architecture. We believe now in isolation, separation. So, we now run extensions in separate process that communicates through RPC with the IDE so that we are in full control. And we can always say you can save the tool, save the document, no matter how bad a plugin behaves and decides to do an endless loop. Because in a separate process, the hope is still one CPU is open, available for you, that it can be safe from the other process. So, that's some example, right? Eclipse has done many things right, but the multi-process architecture I think is a major switch. And the other major switch is at Eclipse time you think Java is cool. Everything has to be in Java. CHARLES: Right. DIRK: No longer think like that, and that brings up this other topic of then the language servers that we can also talk at some point. CHARLES: Right, because that's the thing, is VS Code – now I've primarily been exposed to it through JavaScript and TypeScript development. But it really, it's designed to support all kinds of different languages. So, the C++ support is really good. The C support is really good. And I assume the Java support is really good. Is it safe to say? Because I only ever used Eclipse in the context of Java. Did Eclipse gain kind of a wider acceptance further beyond Java and C++? ERICH: Yeah. I think it's fair to say Eclipse has a rich ecosystem. Yeah. I think with all the tools. And it will be interesting to see that you can close the loop, because for Visual Studio Code, when you do Java development, you actually run Eclipse behind the scenes. That's how we kind of smiled at each other, Dirk and I, when he said, “Now we close the loop.” We started with all JavaScript and then we integrate Eclipse using this language server protocol and that's how we close the loop. CHARLES: Ah. DIRK: So maybe one thing I would like to add is that when you look at Eclipse and the tool and framework landscape that existed in the Java time, at that point in time when we started with Eclipse, it was very well-defined. There was Java. It was a well-defined set of libraries you were using and frameworks you were using. And if you look at the programming and tool landscape you have today, in months you see a new framework for JavaScript popping up or there's something else or another cool X, Y, Z thing. So, the tooling you build today has to be a lot more open to these new inventions, especially since they occur in a higher frequency than they did in the past. And that had influence on how we architected Visual Studio Code to give people a lower barrier of integrating their stuff into Visual Studio Code than you typically have in Eclipse. In Eclipse you needed to program in Java. With the LSP you can program in any programming language. In Eclipse, if you really want to try to do something nice with code complete and stuff like that, you had to hook up a lot of stuff. So, we raised that to another abstraction layer where we more talked about what people provide on data and we do a lot more for them in the user interface than compared for example to Eclipse, which lowers the barrier for people to integrate languages in Visual Studio Code than the barrier you had to integrate something in Eclipse. And so, [inaudible] for that one was that there are a lot more tools and programming languages out there that have importance than 10 years ago. ERICH: I'll give you an example. So, when we did C support in Eclipse, and it was also the team that seeded it. Of course, it took over and has now a great community behind it in Eclipse. But you wrote the C tooling in Java. And of course, that means you built the parser in Java and then of course, there are great C parsers around, C frameworks. But also it means you cannot dogfood what you write. You write Java but you don't program in C++. I think which is what makes VS Code so appealing is we are a very aggressive dogfooder. We want to use ourself and of course [inaudible]. That's why [inaudible] is very good. The C++ guide, they programmed C++ and they write in C++ so that's how they make it very good, that you have this feedback loop. CHARLES: And so, what's an example? We've talked about this low barrier of entry. So, if I were wanting to say, I do mostly programming in JavaScript. Let's say I wanted to add, I know all of this already exists, this infrastructure already exists, but let's say I wanted to add smart editing to JavaScript source files. What would that process look like for me as a JavaScript developer? DIRK: To be fair, whenever it comes to language services, it's never easy. But [inaudible] lower the bar. A language always means you have to do parsing, you have to do [9:59], type bindings. You have to make it fast, scale high up, and so on. So, this is never easy. But I think if you think about the different steps you can do, the first thing, let's not take JavaScript. Let's take a new language. CHARLES: Okay. DIRK: Your new cool language. CHARLES: Or maybe we take a Lisp or something where writing the parser is very easy. DIRK: Even that, you have to resolve symbols and so on. CHARLES: Okay, okay. DIRK: Even the parsing [inaudible]. But yeah, let's take a fancy language like Lisp or whatever. So, the first level I think is you want to get some nice coloring. That's the first level. CHARLES: Yes. DIRK: So, you get some coloring. And what we do there actually in VS Code is we tap into the community from TextMate. So, we use TextMate grammars to support colors in languages, which gives us access to a long [10:51 tail] of languages. So, to change the [10:54], if your language is not too exotic, you will find the grammar that describes how to color, what the tokens are in your language, and then you can get your language colored. That's step one. The next step is of course you want to get smarts like IntelliSense and so on. Ideally of course you can say, “Well, maybe there is something already around that has abstracted the parser and you can use this library.” CHARLES: Right. Because there actually are a bunch of JavaScript parsers written in JavaScript. I know I keep coming back to JavaScript, but let's assume with this language that we've got. I may not have to write a parser but I've got one. ERICH: You've got one, exactly. You've got one, right, and then technically it's not in the same language as the tool. So, that's why I don't want to go too much into JavaScript because for instance VS Code is written in TypeScript, which [transpiles] to JavaScript, which moves a little bit, makes it not as convincing as it could be. So, let's say it's a different language. Your fancy language is written, has a parser in your fancy language, which is different than the language of VS Code which is JavaScript. CHARLES: Right. ERICH: So, then the next level is to say, “Okay, well you have your code you encapsulate it in a server that you can talk to through some protocol.” And now the challenge is what protocol do you talk to? Typically in the language, the library you get, it will use some ASTs, symbols, type bindings. And what Dirk mentioned with lowering the bar is that assuming you have those ASTs, the way you talk then with our tool is through a protocol that is not at the level of the ASTs but at a higher level. CHARLES: A higher level than the ASTs. ERICH: No, yeah. A higher or simpler level. Let's give you an example. You want to find the definition of a symbol in your fancy language. The way the protocol works is you only tell it, in this document with the URI, at this position, I want to find the definition of the symbol that is this position. The request goes over the wire to the other process. Document URI, and the textual position. And what comes back of course now in the server you used AST, you find the symbol, you find the binding of the symbol which means it gives a definition for it. Of course you use your AST to analyze it. But then what gets back to send over the wire is yet another document, the reference, and the position. CHARLES: I see. So, you're really like pinpointing a point in just the raw bytes of the document. And you're saying, “Look, what is here?” And you just want to delegate that completely and totally to this other process. So, the IDE itself doesn't know anything about the document? ERICH: It knows about the document, right? CHARLES: I mean, it knows about the textual positions of the documents and the stream of characters, but not the meaning. DIRK: True. The smarts are in the server. And you talk to the smarts at the level of documents and positions. And the [good thing is] it's a protocol, is at this level it makes it easy to integrate into one editor, which is VS Code, but also into other editors. So, that's why we came up with the idea to have a common language server protocol which allows to provide a language not only for one editor but also for many editors. That was a challenge we had in VS Code. Remember when we started, we were kind of late to the game. We said, “VS Code should be in between an IDE and an editor.” But what we liked from an IDE is of course code understanding, IntelliSense. Go to definition, find all references. But how do you get that for a long tail of languages? We cannot do it all ourselves. So, we need to get a community to tap into. [Similar to] like TextMate grammars are kind of a lingua franca for coloring. So, we are looking for the lingua franca for language smarts. And that's what the language server protocol is, which means you can integrate it in different IDEs and once you've written a language server you can reuse it. CHARLES: I guess I've got two questions. What are the kind of things that I can do with a server that implements the language server protocol? And then I guess the – so we've talked about being able to find a reference. And is there a way you can incrementally implement certain parts of the protocol as you go along? ERICH: Yeah. DIRK: Yeah, basically you can. The protocol on the server and the client side talks about capabilities. The server can for example say, “I am only supporting code complete and go to definition and find all references.” And for example, something like, “Implementation hierarchy or document symbols or outline view is not supported.” And then the client adapts dynamically to the capabilities of the server. CHARLES: Okay. DIRK: That's one thing. And the set of capabilities is not fixed. So, we add them. We just added four or five new capabilities to the protocol last week. So of course, we listen to requests that come from other IDEs, what they would like to see in the protocols that we see in Visual Studio Code, we would like to extend. And that's the way we move the protocol forward. CHARLES: Okay. DIRK: It's capability-based and not so to speak version-based. So, [inaudible] versioning at the end of today. CHARLES: Right. You can incrementally say, “I'm going to have,” if I'm starting to write a server, I can say, “Well, I'm going to only start with just find definition at point.” And that's the only thing that my server can do. ERICH: Well, there are some basics, right? Keep in mind you have two processes. And once the user opens an editor, the truth is in the buffering memory on the one process. The basic thing you have to in a language, so you have to support the synchronization of [inaudible]. Once you open a file in the editor, then the truth in the buffer, and then you have to sync it over. CHARLES: Right. ERICH: [Inaudible] close the truth on the file system and you also have to tell this to the server. Because the server has to know where the truth is. DIRK: That's correct. These two open/close handshake methods and change methods, this is the minimum you have to implement. But for example, for Node itself, we provide libraries that help you with this. And the protocol is not very complicated. It's a buffer. Then it's change events. Either it's an insert, a delete, or an edit. CHARLES: So, let me try and get this straight in my head. I think I understand. The problem is that the VS Code, or your code editor, it's actually making changes to the buffer, and it needs to communicate those changes to the server. Or does the server actually make the changes itself? DIRK: The editor does make the changes. So, the protocol is spec'd in a way that as soon as an editor opens a document, the ownership travels from the server for the content to the tool. And the server is basically not allowed to read the state of that content from disk anymore, or get it [inaudible]. CHARLES: Aha. DIRK: Therefore, the client guarantees that everything the user does in that document is notified to the server, so that the server can move the document forward. CHARLES: Okay. DIRK: [Inaudible] we see the close event, that basically with the close, transfers the ownership of the document back to the language server. And it is allowed to re-read that content from disk if it wants. CHARLES: Okay. ERICH: Here, the protocol is really data-driven. Dirk mentioned that earlier, right? So, basically what flows between the server and the tool is data. So, what do we mean by data? You ask for IntelliSense or completions at the line. What follows is just the data. A list of completions that flows then from the server to the client. And then the client decides what to do with this data and decides to modify the document by inserting the completion proposed that the user selected. CHARLES: Right. And then if it decides to make any updates, it needs to send those to the server. DIRK: Exactly. CHARLES: So, if I actually insert the method that I want to call there, I'm going to be inserting nine characters, and I need to tell the server, “Hey, I just inserted nine characters to this document,” something like that? ERICH: Exactly. CHARLES: Okay. And so now how, because I remember now one of the coolest things about the class of tools of Eclipse that I hadn't really seen in the more lightweight editors – I went from Java, like so many of my generation went from Java to Ruby and then to JavaScript – once I moved out of the Java world, one of the things that I had come to expect from my tools was that they would help me make modifications to my codebase at a very high level. So, I would be able, if I had some class that was imported into say five modules in my codebase, I could say, “I want to change the name of this class,” and then it would find the references and then make the updates to those things. So, how do you manage that? So, if I have a class called ‘Person' that I want to change to ‘User', if I change it to ‘User' then it's going to break in those five different places unless I rename it to ‘User'. That's something that was very doable in the Java world. How do you keep the code editor, the tool I guess is what you were calling it, in sync? Like the server is going to make that change or does it just come back with data and says, “Here's the references if you wanted it to change”? ERICH: Yeah, yeah, yeah. So, two things you mentioned, right? Java and JavaScript or course. Java is a typed language which means you have better understanding of the code and what the reference is. In JavaScript which is typeless, you cannot know it as much, so that's actually why we developed also, we're using TypeScript. VS Code is actually [written] in TypeScript which allows you to do these kinds of things like refactorings. But if you look at the language server protocol, it has support for rename. And the way how rename is done is again it just documents positions. You say, “At this position, I want to rename the symbol with this other name.” And then you tell this to server and the server will handle the rename by giving you back a list of positions that need to be updated. CHARLES: Ah, okay. So now, I'm starting to understand what you're talking about when you say data-driven. It's literally just telling the tool – the tool proposes, “I want to do this rename.” And then the server provides all of the information that is required to actually do the rename. But it doesn't actually do the rename itself. It just provides the data. DIRK: A couple of reasons for it. The data effects, at the end of day, it's again, edit, and it's more or less the same edits the client sends to the server when the user types in the document. This is the protocol. On top of it, something that you can create a file or rename a file, this comes as a result back to the client. And then there, since it is a client/server architecture, the whole process is async. So, we have to give the client the change to revalidate if that edit structure that comes is still valid. If it is still valid, the client basically applies it. And by applying these edits to these documents, they will automatically flow back to the server until the client either closes these documents again or saves them. So, the reason being is that some of the tools may even show you a preview. You can only select some of them and apply them. So, there's always an interaction in these refactorings and to make that possible, as Erich mentioned, the whole protocol is data-driven. We don't go the server and say, “Okay, do that rename,” and he writes that back to disk. It computes a set of transformations to bring the current state of the workspace into that new state after the refactoring. CHARLES: I see. ERICH: [Inaudible] be fully transparent. Actually, no. Refactorings, Dirk [inaudible] refactorings for Eclipse so we can go deep on that. What we don't support right now in the protocol, we support edits in the buffer but when you want to rename a class in Java, you also want to rename the file. And that's something we're currently working on to support in the specification of the language server protocol. So, we don't have that yet. But we support code actions, quick fixes, that you like from Eclipse probably. And you can use then to do refactorings like extract method, extract constant or extract local variable, things like that you can do at the level of the language server protocol. CHARLES: Wow. That is… ERICH: I think [inaudible] right now. Let me go back to the Java thing. The Java language server actually has the support for refactorings. And there is now a language server protocol implementation of this Java provided by Eclipse. So, all the support you had in Eclipse for Java or most of the support is now also enabled in VS Code. CHARLES: Right. ERICH: [We don't] really have to reimplement it because you can reuse. And that's the big thought we have. You want to reuse language smarts as much as possible because they are so hard to implement. CHARLES: Right. And so, you can do that because you're providing this abstraction between the tool and the actual smarts, which is really, really cool. I do have to… how do you make it fast? Because you're describing this tool, this client and this server, and they're syncing. They're keeping this distributed state in sync and you know, how do you keep that from coming too chatty? Or is it something that you have to consider? Or is it just, maybe I'm overthinking it because I haven't dealt with it? DIRK: So, at the end of the day, it is chatty. But it is made performant in the way that it's very incremental and partly event-based. So for example, if you type in the document in the editor, you can either decide to [inaudible] sync the full content of the document, which we do not recommend but for some basic exploration, that is something people do. And we have [inaudible] the delta-encoded mechanism. So, we sync the buffer once and then after that you only get the edits the user does. These are chatty of course since the user types them, we debounce them and collapse them on the client side and only send them if we know that the server really needs to know them because we have another request we are asking the server or after a certain timeout. So, there are smarts behind it. But the protocol is kept performant by making it an incremental protocol at the end of the day, and not sending too much data back and forth. ERICH: Right. We don't serialize ASTs. We serialize positions, a list of items for completions. And actually, the transport is just JSON RPC. CHARLES: Okay. ERICH: And actually, someone, there is different usage now for language server protocol. And there is one host, Eclipse J, which brings it again back to Eclipse. They actually run language servers remote. CHARLES: Interesting. ERICH: And if you use it, you can run it on the browser, you get IntelliSense, and of course I guess it depends on how far away you are from the server. But it seems to work, according to feedback we've heard. CHARLES: Really? ERICH: The feedback we heard from them [is pleasant]. So, they use many of the language servers. CHARLES: So, is this a product that they have where the language server is running in the cloud and you send – your entire codebase essentially goes over to the language server and you can export the smarts to the cloud? ERICH: It's one step at a time. So, Eclipse J is kind of, they have what they call cloud workspace, which means the workspace is in the cloud. And [inaudible] code smarts of the workspace in the cloud, they can run the language servers in the cloud. It's a [inaudible]. One user has one workspace, has one language server. CHARLES: That sounds amazing. And if they can make it performant. ERICH: We have done cloud IDEs, right? If you look at the history from Visual Studio Code, you also had our stuff running in the cloud at some point. That's how we started. Before we pivoted to VS Code, we built – our exploration was, that's why the project is six years old. The first two years, we explored how far you can get coding done in the browser. CHARLES: Right. ERICH: And we had some [inaudible] there. CHARLES: So, I've played around with a lot of cloud IDEs and I've found them to be neat, because every few years it comes along. But yeah, it does seem that there are certain challenges that it's nice to have a client running and just be able to have the files locally. And is that a performance thing or if VS Code is written in TypeScript, theoretically it could run in a browser, right? ERICH: Of course. The [inaudible] there still runs in the browser. Then it's used by many tools that run in the browser. Like actually, if you want to edit your source code in the browser, there it's using the same editor that's running VS Code. So, that's how we started. Cloud IDEs, yeah we were at this point. We had our cloud IDE. We could edit websites in the browser, source control them, have a command line, deploy them. What we found is it's great for some scenarios like code reviews or doing small tweaks to files. But when it comes to really development, you use so many other tools. And you want to just have them. And [inaudible] a long tool chain problem. So, as a developer, you just want to use other tools as well. And that's why you can't have them all in the cloud. CHARLES: Right. ERICH: And [inaudible] we said at some point, it was a great lesson we had that you can program in the browser. But now we want to go to have a really [seven by 24] coding, you want to have a desktop experience. So, what we then did, we moved over the code we had run in the browser using a shell, the Electron shell, and can run it on the desktop. CHARLES: But there's theoretically, you could be running your language server for example in the cloud, but everything else on the desktop. ERICH: Yeah. Some people do that. DIRK: Right. CHARLES: Okay. Wow. It's crazy. It's heady stuff. We've talked about the barrier to implement the code smarts is much lower than it has been in the past. What kind of proliferation of code smart tools are there now that implement the language server protocol? Like how many different languages would you say have airtight…? DIRK: So now, [inaudible] time where we don't count anymore. You tell us a language and I can look it up, whether it's supported. Tell me a language and I can tell you whether – no, we have a website. CHARLES: Okay. DIRK: And when I look at it, we have about 40 languages. CHARLES: Wow. That's probably about, pretty much every mainstream language. DIRK: Yeah. I cannot find what isn't there. CHARLES: Yeah. It almost kind of begs the question, is this going to be the new bar for a language? Because I remember when I was starting out, really you just needed to have some interpreter or some compiler to have “a language”. And nowadays, it's not just the language. You need to have a command line tool for managing your dependencies. And you need to have a package system with a public repository where people can publish reusable units of code. And what's become expected out of a language to succeed has upped. Is having a language server implementation going to be part of the bar, the new bar, for “Hey, I'm thinking about creating a language”? I haven't really arrived until I have a package manager, I have a command line for resolving dependencies, I have documentation, and I have a language server. DIRK: I personally think that is our dream at the end of the day, to get there. We know about languages that do so. So, a lot of these language servers come for example from the people that developed the language. For example, the WASP guys, they do the compiler and they actively work on their language server as well. So, at the end of the day, the advantage of that approach since the WASP language server is written in WASP and runs in WASP, they can reuse so much code that they already have written in WASP. That's easy for them to package that up in the server and basically the people that maintain the compiler, at least the same team, maintains the language server at the end of the day. ERICH: And that's why we call [those] a win-win for the language provider. Because if you implement the language server using the language server protocol, then it can be integrated easily by the tool provider. And it's a win for the tool provider since there is a common protocol across all these languages you have to support. You can write an implementation once and again benefit and support many different languages, which makes the matrix problem one language support for each tool into more a vector, right? It reduces the matrix into a vector. You only write language servers that get integrated into different tools. CHARLES: Right. DIRK: And [inaudible] especially I think appealing for new languages that come out, because it lowers the bar for them to get into existing tools. Because if they write a language server speaking the language server protocol integrating that at the end of the day in Visual Studio Code is basically packaging up an extension for Visual Studio Code and writing 20 lines of code. CHARLES: Yeah. DIRK: And same [inaudible] for other IDEs that exist where people implemented the language protocol client side for the tool, for example. For vim or for Atom. CHARLES: Yeah. DIRK: So, new languages I think definitely, we see that trend go onto the language server protocol because that gives them an entry point into a large tool community. CHARLES: Yeah. I'm really excited about it. I'm actually an Emacs user. And that's actually how I found out about LSP, was in my Emacs newsfeed I saw that someone was starting on LSP support, and got digging into it. And I think that one of the problems that has plagued not only Emacs but all these editors is what you're describing where for example the JavaScript support was really great – is really great – in Emacs. There's refactorings. There's IntelliSense, code completion, all that stuff. But that's because someone wrote an entire JavaScript parser and code smart system in Elisp, which is just an absurd hurdle to jump over, to expected. And so, what you expect out of your editing experience, like when I went to try – if I were to go to try Python, well it's not nearly as good as what I'm expecting. And so yeah, I think it's exciting to hear what you're describing where with having some shared set of abstractions, you can offload all of that code smart onto the community that's building these new tools so that they're really easy to integrate into your environment. I think it's really exciting. Although it does make me ask – and I think we've got time for one more question – is we've been talking about all these different languages. Java, C++, JavaScript, TypeScript, Ruby, Python, et cetera, all these, the 40 languages that you talked about that have this implementation. What are the challenges that you've encountered trying to build abstractions that work for 40 different languages? All with their different syntax, all with their different conventions. It sounds like when aside from the fact that you've actually done it, I would say it's impossible. So, I'm curious. What were the unique challenges to solve there? DIRK: I think we already touched that at the beginning, the appealing stuff of the LSP is that it's not talking about the programming language itself. It's talking about things I can do with source code. For example, requesting code complete, go to definition, find all references. And the data that flows between the client and the server is not in terms of the programming language itself. It's about editor abstractions. We talk about documents and positions. We talk about edits that are applied to documents. We talk about snippets and stuff like that. And these abstractions, since they are programing-language-neutral, are a lot easier to implement for different editors. And the [inaudible] where the [inaudible] would speak AST nodes and symbols and functions and classes and methods, that at the end of the day, would not work. Because if I ask go to definition, the result is not a function or a variable definition. It's simply a position in the document with a hint which range to select. CHARLES: Okay. Yeah. ERICH: [Inaudible] places. In only a few places, we have to really abstract across languages. Like for instance, completions. When you do completions, you don't know, is it a variable? Is it a function or a method? That's where we have to abstract. But that's one of the few places. But again, it's an enumeration. DIRK: Yeah. And that's only to present an [icon]. ERICH: Yes. DIRK: It's only to give you a nice icon in front, because when you insert it, what comes back for completion item is basically a textual edit or a bunch of textual edits that when you select that completion item, we take these edits and apply them to the document buffer. And whether you edit a functional programming language or some other stuff, Prolog or whatsoever, it does not matter at the end of the day. CHARLES: Yeah. That simplicity, and treating it at that simple of a level is what unlocks all those superpowers. ERICH: It unlocks lowering the bar. But of course, if you look at some [of the demands], refactorings, whatever, they cannot easily be funneled. Not all of them can funnel to this low-level abstraction. Then of course, the criticism of the LSP protocol is that if you have already a very rich language service, you might not get it all through the LSP. DIRK: That's true. ERICH: And the [inaudible], that criticism we see of the LSP. But it's a tradeoff, like so many things in software. DIRK: Yeah. But what we learned there looking at different types of refactorings, it's more the set of input parameters that vary much between languages. The result of a refactoring can for every programming language that is at least document-based, [inaudible] in that lingo the LSP speaks. Because at the end of the day, it's textual edits to a document, right? ERICH: So, many people like LSP but there are people that don't like it. And people that have rich language services like IntelliJ, [Cool Tool], and [inaudible], even with LSP we would only get 20% of [our cool] features. Which is a little bit downgraded and not really true. But you see, it's a tradeoff. CHARLES: Right. ERICH: And if you want to [inaudible] language available broadly, I highly recommend it packaged as a language server. Your chances that it gets used, supported by different tools, is much higher than anything else. CHARLES: Right, right. So, it's kind of like, what's the UNIX thing? The universal text interface and how it seems counterintuitive but it actually just means you can literally compose anything. Because so few assumptions are made. ERICH: I would just recommend, [inaudible], go to the website that we have about the language server protocol. I'm pretty sure it will be in the introduction or whatever. It's microsoft.github.io/language-server-protocol and then you see the implementations, all the implementation of languages, who integrates language servers, and also what kind of libraries are available, if you want to implement your language server. DIRK: And a full specification. ERICH: And the specification is there as well. Yeah. CHARLES: Yeah. If you want to go ahead and do it yourself. Well, thank you so much, Erich. Thank you so much, Dirk, for coming on the show to talk about the language server protocol. It's very exciting to me and I think it's exciting for development in general because I just think by having – even if it's 20, 30, 50% code smarts for ever single language, just the billions and billions of hours that you are going to save developers over the next, over the coming years, it's a great feeling to think about. So, thank you for all your work and thank you for coming on the show. ERICH: You're welcome. DIRK: Yeah. It was fun talking to you. ERICH: Yeah. [Inaudible] CHARLES: Yeah. If people want to continue the conversation, is there a good way that they can get in touch with you? DIRK: Usually GitHub Issues. So, where the language protocol is, it's a project on GitHub. Simply find issues. We accept pull requests. I think that's the way we communicate. CHARLES: Awesome. Again, if you want to get in touch with us, you can get in touch with us at contact@frontside.io or you can reach out to us on Twitter. We're @TheFrontside. So, thank you everybody for listening. And we will see you next time.

Show Notes: 01:02 - Why outside-in development? 05:50 - Best Practices and Implementation 09:35 - API Iteration and Design 18:31 - Is outside-in development a timeless approach to software development? 24:10 - Outside-in Creation 28:37 - Summarizing Outside-in Development and What it means to the Frontside Resources: Sketch InVision Balsamiq pretender Ember CLI Mirage bigtest Transcript: CHARLES: Hello everybody and welcome to The Frontside Podcast episode 96. My name is Charles, I'm a developer here at the Frontside. And your podcast host in-training with me today is Jeffrey. JEFFREY: Hello. CHARLES: Hello, Jeffrey. And Arash. ARASH: Good morning. CHARLES: Yeah, how are you doing? ARASH: Doing great. Thank you. CHARLES: All right. So, today this is going to be a little bit of an internal powwow where we're just going to talk about some of the patterns that we see as we develop software and just kind of a little bit of a chat on the way we do it. So, we're going to be talking today, I guess we could do like a little bit of a spoiler and just kind of co lead with what we're talking about. ARASH: Which is... CHARLES: Which is... ARASH: Drum roll... CHARLES: Outside-In development. And it's kind of the way that we've just naturally gravitated towards, towards developing software here. ARASH: So first question is why we gravitated toward that. CHARLES: I could just tell you kind of our personal history on this. Is that, I guess it was, would you say like around 2007, something like that kind of the pattern of like REST services got like really well established. ARASH: Sure, that's accurate enough. CHARLES: Yeah. It's like around mid-2000's. And so people just went kind of, you know, cuckoo for Cocoa Puffs except APIs instead of Cocoa Puffs. So it was like all the talk was about your API, what's going to be your API, and how are you going to present this to the world? And oh my goodness, you're opening up your system for extension by developers and it's really fantastic. So, it kind of became the norm, I feel, from that point forward to be thinking about what is the API that I'm going to be presenting rather than what's the application. ARASH: That's your starting point. CHARLES: Yeah, that's your starting point. So a lot of development and getting like a lot of frameworks came along to make API development really simple, regardless of run time. It was really just really simple to just set up API. And so, a lot of people did that and I think there was value in that and that you're thinking kind of about your external domain model, like how you're going to present yourself to the world. And so you really are focused on the constraints and the abstractions that you want to present and how you want to hide all the messy complexity behind your system. But the problem that you get into is in those days, especially on the web, the clients were very closely aligned with the APIs. If you're coming like from a Rails backend, you almost had one for one between your pages and your API. If I've got a user's end point, I'm going to have a user's management page. And I think as clients became more stateful and more of the kind of interactions were ported over to the client, that synchronization and one to one mapping between what your API look like and what your interface look like, that started to crumble and disintegrate. And now I think that it's actually quite, I mean there are still clearly analogs because you're talking about the same entities. But your client is really now a full-featured application. And it's got complex rendering logic, it's got complex state management, it's got all these things that happened completely and totally separate from your API endpoints. JEFFREY: And your UI ends up being aware of relationships between models. So yeah, it's just so much more sophisticated than what we used to need to do when everything was server-side rendered. When you're dealing with a heavy client side app, your API needs to be quite a bit more flexible really. CHARLES: Right. And so what we were finding was that we'd often have to change the API significantly in order to support complex interaction on the client. But the problem is, is if you've started with your API, you've invested a lot of development, you've invested a lot of design and you've really laid down not just that design, but also the infrastructure and the operations to make it real, it can be very hard to change. And so, it can exclude a lot of the desirable interactions that you would like to have merely by virtue of the fact that it's set in stone. So I think that's kind of why we found ourselves... JEFFREY: Why we discovered that inside out wasn't quite working for what we're doing anymore. CHARLES: Yeah, exactly. It was like having a rock solid API was actually a drawback rather than an asset, whereas 10 years earlier it was an asset. And so it really, kind of having that insight made us step back and wind it back and say, "OK, where is the natural starting point? Where do we want to start?" ARASH: So philosophically, what does Outside-In development mean to the Frontside? JEFFREY: It means that instead of starting with the API, looking at actually the workflow is instead we start with what's the UI. What is the client side app going to look like and what are the needs of that client side app? And that drives the development of the API rather than solely, I guess you could say, the business models that would have previously been the initial driver for what that would look like. CHARLES: Right, exactly. Letting the business models be kind of a function of the desired interaction or the desired experience and then letting it proceed from there. So really is yeah, it's starting from kind of what does the person using your system going to touch rather than what is the computer using your system going to touch. ARASH: And so if one of our listeners is interested in Outside-in development, what kind of tenets would you recommend they follow and what are some of the best practices you can put into place to align well with this kind of thinking that we're talking about here? CHARLES: I think that's a good question because there's obviously the philosophy of it, but then there's the practice and the implementation and what does that mean. And there's a lot. There's a lot of just kind of the way you approach the problem and then the tools that you're going to use. I don't know, Jeffrey, which ones should we tackle first? JEFFREY: Let's tackle the tools. Why not? We'll go backwards. We probably should cover other things first, but why not? CHARLES: We'll tackle the tools? All right, and this is actually a question I want to throw your way too is at what point do you...because you want to start really with having a good design, like a good wireframe? Do you prefer to start with say like something in InVision or Sketch or do you like to proceed straight to a working implementation? JEFFREY: For me, it usually depends on how complicated of an app it's going to be. If it is something that's, I'm just dealing with like some crud operations, a lot of times I'll just go straight to the HTML and like starting to build that in actual JavaScript and we'll probably talk about the fake backend for that. But yeah, it versus something that's a little more complicated that has maybe a more novel approach or particularly interesting workflow or something that's going to be pretty difficult to code. That's when I'll start on in Sketcher, InVision, or a tool like that. CHARLES: I think in the last project, we used Balsamiq. I mean, we don't want to get caught up on like tools, but I think that's a great tool because the other thing we did is it wasn't just one designer doing it. We actually had all of our developers going and writing Balsamiq sketches for the features that they were going to be implementing because it really means that they're going to actually put themselves in the shoes of someone using it and they're going to be thinking about what it's like to be a user. Because they've got to start drawing the boxes and lines and the buttons that people are going to click in and the text fields in which they're going to communicate with the system. JEFFREY: And what was awesome about using a sketchy type of style like Balsamiq is that it removes the developers from thinking too much about having to make sure this headline is exactly the right font size, that I'm using all the exact right colors. Like, don't worry about that yet. Let's just get the wireframes in this sketchy style first and then we can work from there. CHARLES: Because it really is. It's very valuable to have this process for yourself as one single developer to be able to view it from the outside in. And I think that looking at it one way, that's truly what it means to be full stack is that you can put yourself in the shoes of every piece of the system at each point. So I can put myself in the user's shoes, I can put myself in the client's shoes. When I say 'client', like the actual browser, the browser's shoes. I can put myself in the server shoes and I can have that perspective of each part. JEFFREY: So that got me thinking about what browsers would wear what kind of shoes, but we'll leave that conversation for another time. Who'd wear the Chrome shoes? ARASH: Chrome is definitely Nike, for sure. Speed. JEFFREY: So back to the tool. CHARLES: Just like Safari, like Hugo Boss. ARASH: It's like, yeah, that's pretty close. Netscape is like K-Swiss or something like that. CHARLES: It's got to be some defunct brand. ARASH: BK Knights. JEFFREY: Mosaic is the LA Lights. CHARLES: Those were great. JEFFREY: So once I have this design, I'm starting to move to code and I'm starting to build some UI, how can I iterate on my API from there? Because I'm starting to actually build a real app, I want to connect to something, I need a data store. What can I do there? CHARLES: We've talked about this a lot, I think on this podcast, really over the years. And this is a technique that we got from the Ember community. There's a stack of fantastic libraries. There's Pretender which allows you to stub out XMLHttpRequests using a DSL, kind of like Node Express. And then based on top of that, there's basically an entire fake API layer which allows you to build, I would say it's not even really a stub at this point. It's like actually a prototype server implemented entirely in your browser. It's called Mirage. JEFFREY: So, you think about it, almost like starting in Sketch and then going to HTML. You're starting in some pretty simple JavaScript and then you're working your way up to what that API design looks like. CHARLES: Right. And so that lets you experience all of the nuance of API designs. So it comes with like you can experiment with different serialization formats, what are my property's going to look like? You can experiment with how do I load related data? So, if I've got a list of users and I want to get all of the comments that they've ever posted on my site, you might want to load those at the same time. So whether your API's going to support that and how it's going to support that, you can't really know until you actually have a consumer of that API. So what Mirage lets you do is it lets you consume an API you haven't written yet and you get to feel what it's like on the client and you get to make changes to that interface at a very, very low cost. So there's no deployed infrastructure, there's no automatically generated documentation. There's no real consumers yet. There's no multiple clusters running in the Cloud and it's very, very, very lightweight. It's just running right there in the browser. And so what that lets you do is if you have an insight about the shape of the API, it lets you validate that insight. Or if you have an assumption that you have about your API that actually is going to be detrimental to your experience, you get to have that assumption invalidated very quickly. And so at the backend of this process, you end up with a backend that is optimally shaped to serve the experience that you're shooting for. ARASH: So once you've got that, how do you reconcile it against is this a good API that I want to expose externally because maybe it's not. CHARLES: Like what do you mean? ARASH: Imagine that we've built the UI and we've iterated on the API in a fake stubbed way with Mirage or some similar tool and we've got to a state that we're pretty happy with, with the API design that works for this UI. Now, imagine we want that API to also be available to other things. CHARLES: Right. ARASH: How do we reconcile that? CHARLES: That's actually, I think that's kind of an open question right now, right? I mean, we don't really have a good answer to it because it is a stubbed API. And ultimately you're going to have a real API. So at some point, you do have to say like, this is the shape that I want. I'm going to go ahead and create these things and I need to make sure that those two APIs actually line up. Is that what you're asking is like, how do I make sure that they line up? ARASH: Sure. That's one piece of it. CHARLES: Oh, that's one piece of it. OK, so this is actually...we'll explore this process. This is actually like a problem, I think, with the stubbed approach is you have your set of stubs and then you have kind of the real McCoy which eventually will come along. So, it's great to have a server running inside the browser, but at some point you're going to need an actual server that's running in the Cloud. ARASH: Will you? Like what does the future hold? CHARLES: I mean, yes because the data needs to move, right? The data needs to move off your laptop. Honestly, Arash asked a good point if you're making an offline application. So if you're making an offline application, this actually allows you to bring a persistence architecture to the browser that is like every other... you don't have to make a special one off case. You can treat your backend like a backend. It's just running on the frontend. So that's probably a corner case, but it's worth pointing out that there used to be like back in the day, if you're using something like Microsoft Word or Excel or some other offline app, you can go a long way without having any internet connectivity, but you get to build it in exactly the same way. And then when you are ready to have internet connectivity, you've got the architecture in place. So, the future might not hold that and you might actually be able to use this in production. I would say it's a minority of cases. Most time you actually are going to want to have a server component. Let's say that you do, now you've got your real API and then you've got your prototype API that you run your tests against or that you're running against, and they need to line up because your frontend is going to be using both of them kind of throughout the development process. So there are a couple of strategies to deal with this. I'd say one is you run automated tests against both versions and that's a whole another subject of having testable APIs. Like most APIs that we have that we develop aren't set up to do end to end testing including a lot of the ones that we've written, although I think going forward now that we have a better handle on it, we can do that. We can definitely unpack that subject at some later point like making testable APIs. But I would say that the other strategy that you can use is use some sort of third party verification mechanism where you essentially record a bunch of interactions between your application and the fake API. And then you take those recorded applications and have them in some sort of repository and you can play them back on your real API and make sure that the responses generated by your real API for those recorded interactions to match up. So it's kind of like, I don't know, it's just a machine to make sure that the APIs stay in sync. ARASH: That's a domain that we need to get better at. CHARLES: Yeah. ARASH: We're only scratching the surface there. CHARLES: We're definitely only scratching the surface there. But the power of being able to develop without a real API first is enough so I think it makes the exploration warranted. There are some established tools in this space. The biggest one that I know of is called Pact. It's both a library and a repository. So it's kind of like got a...I don't know if there's a central server for it, but it's kind you record interactions, you upload them to a Pact server and then you can verify. So you can actually have a lot of consumers. And so, it's designed not only for making sure that your stubbed API works, but it's also for making sure that you just don't make breaking API changes. You can actually collect a lot of data about how people are using your API and then you can use that as a repository for when you make a change in your API to verify that'll work for all these other people. So if I'm just some random person or some random developer using your API, I can...kind of like how you record [inaudible] statistics and stuff to make the Apple experience better or whatever. It's very common for developers to ask, to record for certain anonymous data. You can submit anonymous data in the form of like Pacts. And so that can help an API developer verify whether the change they're going to make is going to break clients out there. So that's a little bit of a sidetrack there, but it's something that you start to think about a lot more when you start to develop in this fashion. ARASH: Did you feel like Outside-In development is a timeless approach to software development? CHARLES: I do. I feel like Outside-In, it actually even pervades more of software development than actually like the building the software. Like I feel that the longer you go in software development, you realize that the highest value activity is to front load understanding. So whether you're submitting a pull request, whether you're working on a work ticket, whether you're documenting code or even writing a method, the highest value activity that you can be engaged in, in each one of those points is understanding what the hell you're doing. ARASH: And why, too. Right? CHARLES: Yeah, exactly. Why? Understanding the motivation. What's your prime directive? What is the context on which you're entering into this piece of work? JEFFREY: I'm going to go with a bit of a contrarian approach. CHARLES: OK. JEFFREY: I'm going to go with a "maybe"... CHARLES: Maybe? JEFFREY: Maybe this is timeless. CHARLES: Maybe it's timeless? JEFFREY: Because I think historically, so much of software development was working with constraints. And the particular types of software that we're working on, the most important constraints are on the frontend. What does the end-user experience look like? But historically, that hasn't been the case. Historically, the primary constraints have been what can this technology stack that I'm working on actually do? CHARLES: Like what is the computer experience that I can support. JEFFREY: Yeah. And so that was actually the bulk of the work was like how can I actually get something that I want out of this giant machine in a room with me? So, maybe it is timeless, maybe not. But I think it is the way going forward. CHARLES: OK. So I agree that there's an interplay. There is kind of a yin and yang cycle. And I don't mean the cycle of we go to where we have to think about this and then we go to where we have to think about that. But I take your point, Jeffrey, but I think of something like Super Mario Brothers, which is I would say both a miracle of experience but also a miracle of technology in the sense that I think it was hand coded in Assembler on an 8-bit controller with who knows how much memory, the whole thing. So taking that into account, they had to think very strongly about what the computer could do at that point. They were operating under some just incredible constraints. But at the same time, they were thinking about what...they very clearly were focused on what is the coolest game that we can make at this point. So you're absolutely right. You have to hold both inside your head. You don't want to go crazy. I mean, we could sit down on the Balsamiq thing and be like, the first thing we're going to do is we want a VR room with...okay, no. Let's dial it back. JEFFREY: How do you wireframe VR? I've never seen anybody try do that. CHARLES: The Frontside Podcast episode 97: Wireframing VR. JEFFREY: Interesting. CHARLES: How we wireframe VR. So yeah. I agree. I take your point. You do have to hold both in your head at the same time. But I would say start with one a little bit and then see where that can take you, that the task of gaining understanding about what you're trying to build [inaudible]. JEFFREY: Yeah. CHARLES: If you try to understand what you're trying to accomplish, then you can try to push the tech stack towards that direction and maybe push a little progress forward. And then if you've accomplished something that you couldn't accomplish before, then you can start dreaming about even better experiences. And so kind of moving...like the wheels on He-Man's magic bus or whatever. JEFFREY: This is out of my domain. ARASH: I'm so [inaudible] backwards right now. JEFFREY: Our references are not timeless. CHARLES: Haven't you seen the...what's the one where he's like singing the song like the 4 Non Blondes song? And I said, hey, yeah, yeah. You haven't seen the He-Man, like the cover of 4 Non Blondes? ARASH: Oh, like they dubbed his... CHARLES: Yeah, no one's seen it? Oh, man. ARASH: It's starting to sound familiar. CHARLES: OK. I mean, this is like we're talking... ARASH: This is like 2006 internet? CHARLES: No, this is like 2010 internet. It's not the ancient past here. ARASH: I'm going through my internet filing cabinet in my brain right now. CHARLES: Okay. It had particular significance because the friends from attorney were also part of my childhood and I realized I'm unique in that aspect. But they basically, in those days they had the Hanna-Barbera cartoons were very cookie cutter, so they basically took the Mystery Machine from Scooby-Doo and they kind of re-colored it and they put like weird tank treads that like they put really funky kind of weird futuristic He-Man wheels on it. And it went from the Mystery Machine to He-Man's kind of a ride. ARASH: They repurposed all the illustrations. CHARLES: Exactly. Hey, you got to save time. ARASH: Save time somewhere. CHARLES: Doing your framework. ARASH: Kids need cartoons. CHARLES: That's right. So anyway, that's how the wheels on He-Man's Mystery Machine worked. ARASH: This has been interesting for me because when we first started this conversation, we're looking at it as Outside-In development. But we've talked so much here about design and creation that it almost starts to feel like Outside-In development is almost too narrow. Like it's really what we're talking about here is Outside-In creation of anything useful and valuable to people. CHARLES: Yeah. It's easy to overlook like whenever you want to create something, you get so focused on the actual building of it or the making of it happen that you kind of lose sight of where it is that you're going. And I think that sometimes it's important to be able to create without knowing where you're going to kind of push the paint around the canvas, so to speak. I think that's a valuable activity, like sometimes it's important to just code without really having any clear direction. You're just kind of following your instincts. Then it's the same way you look at Picasso's study in charcoal or whatever. There'll be like a picture of a horse leg and Picasso's like, "Oh, there's a horse leg. Man, how do I capture a horse leg?" And not really having much intent. And so that's important. But I think when you're building systems or you're creating something that you want to have a real lasting impact, you do need to consider it holistically and you really need to understand why it is that you're doing what you're doing. JEFFREY: Part of what's awesome about working in software is that unlike Picasso who had to start a new piece, we can keep working on the same piece and be able to paint over it multiple times. And you can start with that outline of like, "Hey, this is the vision that we see this particular thing going." But then don't make space for like, "Let's just push the paint around and see what happens in this little section," and that's fine. CHARLES: Yeah. And I think that because software is so malleable that it can end up capturing a lot of what is typically considered design. I think this has kind of also been a theme that we've talked about kind of over the last year or so is that over the last 20 years we've kind of seen software capture a lot of what were once separate practices. So when I was first starting out, it was quality. The line between QA and development was very stark certainly at the beginning of my career. And there was a fusion that was happening right as I joined of QA and development. And we saw basically the testing revolution, realizing the testing needs to be brought into the center of development, which needs to be put forth first if you want to have quality, be something that you want. And then, a revolution that I saw kind of in the middle of my career is seeing operations. It used to be certainly for many years into the beginning of my career, like the people who maintain the software were very different from the people who wrote it and there was a big divide there. And with the Dev Ops movement, it's kind of the realization that no, if we want to have healthy operations, we need to understand what healthy operations are at the very beginning and we need to like put them at the beginning of the process. And I think that what we're seeing now, maybe a revolution that hasn't quite happened yet, but we're on the cusp of is seeing design burrow its way to the heart of development where it's like we want something to be beautiful. We want it to be frictionless. We want it to be delightful to its end users. It needs to be there from the get go and developers need to be thinking about it. It's not someone else's responsibility. It's you as the primary creator. JEFFREY: It doesn't matter how elegant of an API you've created if the end user experience is just not there. CHARLES: Yeah. So it's all about tearing down. There's been a series of walls that have come down that I've witnessed. And so I think this is one that's in the process of crumbling. ARASH: So, this has been an awesome talk. How would you summarize Outside-In development and what it means to Frontside today and in the future? CHARLES: I think that it really is about making sure that the understanding is solid. That's the core piece, frontloading understanding, realized through a stack of tools. So the tools we use are some sketch and design tool. We'll sketch out the idea we really like, ask questions and try and understand it and firm up in our idea, like have a vision of what the experience is going to be like. Then the next step is to write some acceptance tests. So to define done, define what your criteria are going to be so that if I run these automated tests, the code will in fact realize the experience that I'm dreaming about. To do that, we've actually been assembling a suite of tools over the past year to do that and we've actually started releasing them to the world. So, it's still in a...I don't want to say alpha because we're using them in production systems, but it's much more of a work bench at this point than a framework, if that makes any sense. So we have about four related tools under the big test tent. We have interaction library for stubbing out, gestures, mouse clicks, keyboard events, scrolling. We have assertion library that works with any other source. It's really an assertion extension or extension wrappers to make your assertions impervious to asynchrony. We've extracted Mirage from Ember CLI Mirage so that you can use it in any project, React, Vue, Ember, what have you. You can check it out at big test. It is for early adopters right now, but it is in production systems and if you want to acceptance test anything, command line, back client, what have you, if you're willing to write some JavaScript, you can write those acceptance tests and have them be robust. And whatever experience you're trying to create, you can validate it with that. And so, that's why we created it. JEFFREY: And that's really a good marker of where we are in our experience with Outside-In development right now is we've been doing this in practice for awhile at this point and we're crafting tools to help us be even better at it and sharing those. CHARLES: Right. And so we're pretty excited about these tools because I think they bring solutions to a lot of the problems you're going to encounter. So there's a lot more work to do and we're excited to do it. Head on, check those out. And we'll go ahead and wrap up. Just a few quick announcements. If you're going to be at Assert(js) tomorrow, the Frontside is going to be there. I'm going to be there. Mr. Wil Wilsman is going to be there. So if you're there, please do give us a shout and we'll hang out. And then on the next podcast, we are really, really excited. I'm both nervous and I just can't keep a lid on it. We're going to have Erich Gamma on the podcast to talk about the language server protocol that he's been working on. If you don't know Erich Gamma, he's one of the Gang of Four that wrote The Design Patterns Book. He was the primary architect behind Eclipse and most lately VS Code. So, if you've used any of those projects or benefited from them, which I have benefited immensely from all three, it's going to be really, really exciting. So, that's something to look forward to. And with that, I will say goodbye to you, Jeffrey. JEFFREY: Ciao! CHARLES: And to you, Arash. ARASH: Adios! CHARLES: And everybody listening along at home or in your cars or cleaning your kitchen as I do when you listen to podcasts, we'll see you next time. If you want to get in touch with us, you can always give us a shout on Twitter, we're @thefrontside or you can drop us a line, contact@frontside.io.

Kevin Nater, CEO & Co-Founder | Chas Studor, Founder & CTO Show Notes: 02:43 - The Evolution of Briggo 04:23 - “The Briggo Experience” 08:22 - Quality Control 12:44 - Source Origin 15:13 - Growth Plan 16:40 - Ramp Up Time 17:37 - Operation, Supply, and Usage Patterns 21:51 - Customer Feedback 25:02 - Consumer Perception and Design 27:41 - The Name “Briggo” 30:04 - The Future of Briggo Resources: @DrinkBriggo Transcript: Coming Soon!

Katharine Beaumont: @katharinecodes Show Notes: In this episode, we hit the topic of machine learning from a 101 perspective: what it is, why it is important for us to know about it, and what it can be used for. Transcript: CHARLES: Hello everybody and welcome to The Frontside Podcast, Episode 94. My name is Charles Lowell, a developer here at The Frontside and your podcast host-in-training. Today I'm going to be flying it alone, but that's okay because we have a fantastic guest who's going to talk about a subject that I've been dying to learn about. But you know, given the number of things in the world, I haven't had a chance to get around it. But with us today is Katharine Beaumont who is a machine learning consultant. And she's going to talk to us, not surprisingly, about machine learning. So welcome, Katharine. KATHARINE: Hello. Thank you very much for having me. CHARLES: No, no, it's our pleasure. So, I guess my first question is, because I'm very much approaching this from first principles here, is what is machine learning as a discipline and how does it fit into the greater picture of technology? KATHARINE: Okay. Well, if you think about artificial intelligence which is one of those slightly undefinable fields because it encompasses so much, so it encompasses elements of robotics, linguistics, math, probability, philosophy, it has six main elements. So, a really basic definition of machine learning is getting, and this comes from Arthur Samuel in 1959, it's about getting computers to learn without being explicitly programmed. And that's hugely paraphrasing. But machine learning is an element that sits under the wider discipline of artificial intelligence. Artificial intelligence is one of those tricky to define fields because people have different opinions about what it is. And obviously philosophers can't agree what intelligence is, which makes it slightly complicated. But artificial intelligence as a broad brush is a discipline that borrows from philosophy, math, probability, statistics, linguistics, robotics, and spawned subfields like natural language processing, knowledge representation, automated reasoning, computer vision robotics, and machine learning. Machine learning is the, in a sense, the mathematical component of artificial intelligence in that from a basic point of view, even though you're looking at it from the perspective of computer science, you're utilizing algorithms that a lot of mathematicians will say, “Look, we've been doing this for years. And you've just stolen that from us,” that try and find patterns in data. And that pattern could be as basic as mapping, say, the square footage of a house to the price that it will sell at and making a prediction based on that for future examples, or it could be looking for patterns in images. CHARLES: Okay. You mentioned something that I love to do. I love stealing ideas from other disciplines. It feels great. KATHARINE: Who doesn't? CHARLES: Yeah. It's like free stuff. And the best part of ideas is the person who had it still has it after you've lifted it off of them. KATHARINE: Yeah. You just have to reference and then it's not plagiarizing. CHARLES: Yeah. So, how did you actually get into this? KATHARINE: Well, a few years ago, I was desperately bored in my job. CHARLES: So, what was that job that you were working on that was so desperately boring? You don't have to name a company. KATHARINE: Oh, I won't name the company but I will – I have to make a confession now which links back to something that we were saying off recording earlier, which was that it was doing web development. So, I'm sorry. And that's not to say that web development is boring at all. It's just that I wasn't particularly engaged, which is not a reflection on web development. CHARLES: No, no, no. I actually came – I was doing, before I got into web development, I was actually doing backend stuff for years. That was all I did. KATHARINE: Yeah, me too. I would have described myself as a server-side Java developer who then cross-trained into Ruby. And I thought I'd be doing exciting backend things in Ruby. But unfortunately, it was more, “We'd like you to move this component from this part of the page to this part of the page.” And I didn't really connect with that. And I started to wonder if I even should be a developer. CHARLES: Wow. KATHARINE: Larger forces than myself were at work to try and push me into management or analysis. And as happens, I think, after a few years. So, I started doing, in my spare time, looking at a website (and I'm sure you've heard of it) Coursera. CHARLES: Yeah. KATHARINE: So, this is the birthplace of the massive online, I can't remember what the second O is, MOOCs. Massive Online something learning. Maybe a Q in there. I'm not sure what. Do you know what the acronym is? CHARLES: I actually don't know. KATHARINE: Well, MOOCs anyway. Massive online learning courses. And there was one offered Andrew Ng from Stanford on machine learning. So, I took that and I just loved it. I really enjoyed it. And I really connected with the programming. I really enjoyed the programming. It was very fulfilling. So, it grew from that, really. And now, I've decided to go back to university. So, I'm a mature postgraduate student and I'm just currently weighing up my PhD options. So, whether to sacrifice four years for the greater good and the pursuit of knowledge or go back into an employment. So, we'll see. We'll see. And I'm quite enjoying not being employed, I have to admit. Or being employed on a freelance basis. It's wonderful. CHARLES: Right, right, right. Now, a couple of things stuck me when you were talking about – so obviously, you're studying a lot of the mathematics behind it. And you said that machine learning involves a lot of the – it's the mathematical component of artificial intelligence. But what strikes me is learning, to me, implies a lot of statefulness where you're accumulating state. Whereas my experience with mathematics is usually you're solving equations. You start from some set of facts and whether it's a dataset or some other thing, and you derive, boom, boom, boom, boom, boom, you get your answer. Whereas with learning, at least when I think about school learning, like spelling or, I don't know, paleontology or something, you're accumulating facts over a very long time. And the inferences that you make are not necessarily – they're drawn from all fo the sources that you got over all that period of time rather than some one set of facts that then you make this logical argument and poof, presto, you've got your answer. How does that square? I guess it's just a little bit off from my experience with mathematics. KATHARINE: So, I am being a bit reductionist. So probably, one way to explain it is that essentially, behind a lot of the machine learning algorithms, you're inputting numbers. And that might be the percentage of red, green, blue in a pixel for example. Or it might be the diameter of a wheel, for example, if you're looking at a component of a car. Or it might be a binary configuration if you want to input the configuration of a control panel, for example, and you're looking for anomalies. And you're running these numbers through an algorithm. And what you're getting out is either a continuous value, if you're looking at a problem with continuous data like house prices, or you're getting a probabilistic output like 60% certain these pixels together make a cat, for example. So, I am simplifying by saying it's math because what you're really doing is looking for patterns in data but a way to get a computer to understand it is to somehow input it as numbers, essentially, and to get numbers out of it. CHARLES: Oh. KATHARINE: Yeah, it's more algorithms, really. And I shouldn't have said that it was essentially math because I'm sure I'm going to get shouted at on the internet. CHARLES: Well, I certainly don't want to get you in trouble. But maybe that's a point that we should shy away a little bit, the high theoretical stuff, and bring it back. If I'm excited, not even if I'm excited, why should I be excited about it? I've heard that it's a hot topic. I've heard that a lot of people are excited about it. Is there a way that I as someone who has no specialization in this might actually be able to bring some of these techniques to bear on the problems that I'm working on? Perhaps even without understanding them first, like understanding how it works. What are some problems that I might be able to attack with these techniques? KATHARINE: Yeah, absolutely. So actually, and one of the things about machine learning that I should say is don't think, “Oh, it's not for me. I'm rubbish at math. I don't understand these concepts. I'm not willing to get my head around an algorithm,” because there are so many pre-configured APIs available from big companies like Google and Amazon and Microsoft and IBM, and I'm sure many, many more. And I'm not paid by any of them, I should say. So, you don't need to understand the inner workings of an algorithm to use it. So, one example is speech-to-text. So, if you imagine that you're working on a website and you want to make it accessible, maybe you could have a component to your navigation bar that allows users to record their voice and say, “I want to navigate to the shopping cart,” for example. And machine learning would be behind that processing. So maybe, behind it you'd have an API, I've used a few of them before just to play it, where you make a call to, say, IBM service and it returns you the text. And in your program you match on keywords like shopping cart and then change the menu bar for them. So, that's one really simple way you could do it. Another more complicated way to do it is to implement something like a recommender system. So, say you have a website where you offer customers products of some description. And the most famous example of this is Amazon and Netflix. Amazon, the shopping site, rather than now the big, big corporation. And you see what other customers like you bought, or Netflix, what you might enjoy. And that's based on taking your information, comparing your viewing habits to other people's viewing habits, and then drawing some kind of correlation between the programs you watch and trying to find programs that other people have watched that you haven't, that you might enjoy. That's more complicated, to be honest. CHARLES: But that is an example. Machine learning is what underlies all that. KATHARINE: Absolutely. And at the heart of some recommender systems, the mathematics behind it is finding a way to quantify people's preferences and measuring distances between them. But you don't need to understand that to understand the basics of how a recommender system works. CHARLES: Okay. And so, how does a recommender system work? KATHARINE: So, imagine me, yourself, and Mandy each read four books. And we rate them. But I read four books, you read three of the same books, and one different one, and Mandy reads three of the same books as me and one different one, for example. So, we've got a little gap but we're not really sure what the other person will think. And we know the genres of the books. And you can compare the genres and the ratings. So, you might rate sci-fi 6 out of 10 and romance 7 out of 10. And I might rate sci-fi and romance in equal ways. So then you might say, okay, there's a similarity between our preferences. So for this book, that Charles read, Katharine might like it. CHARLES: That makes so much sense. KATHARINE: Yeah. And maybe Mandy only likes romance, only rates it 0.3 for example. So we think, “Okay, well Mandy might not be able to recommend a book to Katharine and Charles.” CHARLES: Right, I see. Implicit in this though is there's this step of the actual learning, I guess. Or the actual teaching. How do you actually teach? Again, and this is kind of me trying to wrap my head around the concept, is I've got these set of facts and I'm inferring and I'm pattern-matching and I'm trying to draw conclusions with some certainty from this set of data. But is there this distinct actual teaching phase where you have to actually teach the computer and then it takes new facts and gives stuff? How does that work? How does it incorporate the different – I guess what I'm saying is, are there distinct phases? Or is it… KATHARINE: Yes, and it's not the same for every algorithm. So, I'm going to try and give you two examples of training. So, there's something called online learning where as a new example comes in, for example it might be – I'll just explain what a classifier is. So, this is a brief diversion. So, a classifier is a machine learning process where you're trying to put information in and you're trying to get discrete information out. So, discrete meaning like it's a cat or it's a dog or a weasel or a minion or something like that. Or, it's cancer or it's not cancer. Whereas continuous output might be the price of a car. So, in a classifier you're trying to work out what type something is. So, a really good example is there's a Google project where they've clustered artworks. So, they've taken lots of different artwork and their algorithms, which I won't explain now, have determined, “This is a ballet dancer. So, we're going to group all of these ballet dancer paintings together. This is a landscape, so we're going to group all the landscapes together.” So, online learning, you might get a bit if information in, like a picture, and you will classify it and you add it to your existing information. Whereas another type of learning is you take all of the information you have, you train the algorithm, and then you make predictions. So, you either make predictions as you go along with online learning, or you do all of the work upfront. So, one algorithm – have you heard of decision trees? CHARLES: No, I haven't. KATHARINE: So, do you ever read those rubbish teen magazines where you have a flowchart and it starts at the top like, “Do you like cats? Yes or no?” CHARLES: Oh right, yeah. KATHARINE: “Do you likes dogs? Yes or no,” and it tells you what kind of a person you are or what make up you should wear or something like that. CHARLES: Right, right, right, yeah. KATHARINE: Yeah, so decision trees are kind of like that. One example is you might get information about, it's a famous toy dataset, information about passengers on the Titanic about gender and age. And we all know the techniques on the Titanic, women and children first. I've lost my use of normal English. I'm sorry. CHARLES: Right. Maybe like a trope? KATHARINE: Yeah. So, women and children first. So, you put this data in a decision tree. And what happens is at the beginning you have all of this data. So, person A is a male. They're in their 50s. This is the type of ticket they had. And this is their income. I don't think income is one of them, but just as an example. And then you have person B, person C. So, you might have a hundred people. And the decision tree algorithm goes, “Okay, if I just looked at one of these features like gender, would that differentiate the people the most?” So, it already has the answer as to whether or not they survived or did not survive. And it's looking for the one feature that gives it the most information. And then it will split on that feature. So, you go form your thing at the top and the first question might be, “Were they male or female?” And then the decision tree will split down a level. And then your algorithm will go, “Alright, what's the next feature?” Maybe the next feature is, “Were they under the age of 30?” for example. And it works down. And you end up with this sort of flowchart. And once it's trained, you then get a new example you have, person X. and you basically just work your way through the decision tree to make the prediction of whether they survived or did not survive. CHARLES: I see. And so, do you do it with some sort of certainty? Because there's going to be variation, right? You're going to have some people who are a poor fellow in his 70s who survived. Like, it's not certain that he went down but there's some probability at the end? KATHARINE: Yes. Because you're using it to make a prediction, there is always a probabilistic element. And it depends on the decision tree algorithm. So, there are some decision tree algorithms that really don't work with contradictory data, for example. CHARLES: I see. KATHARINE: There's an element of picking your algorithm. If you're approaching a machine learning problem, you have three elements. The first one is choosing your feature and your algorithm. Then you have evaluating it, so you need a way of saying how good or bad the algorithm's doing on your data, how accurate is it for example. And then you have optimizing it, which is the dark art of machine learning. CHARLES: Right. That's the wrap across the knuckles. It's coming up with wrong numbers. KATHARINE: Yeah. That's – oh no, this took two weeks to run and I need it to take 20 seconds. Or, this is only 60% accurate and I need it to be more accurate. And it's easy, just as an example I'm just working on a course that I'm giving in a few weeks' time. And I just took a day to set up a website called Kaggle, K-A-G-G-L-E, for wine quality. So, it's got acidity, citric acid, residual sugar, chlorides. They're the features. They're the components that you're going to put in. And it has a quality score. So, what you're trying to do is you're trying to find a relationship between the features and the quality. And it's very easy to get it to work. So, I now have it working. But I have it working with a really low accuracy. So, it took maybe five minutes to get it to work and it's going to take me about half an hour to make it more accurate, and that's the optimization element. CHARLES: I see. How stable are these processes? Is it finicky and fragile so that if you get new types of features it just throws things way off? Or are there ways you could control for that? KATHARINE: Well, that's a particular type of problem and that all links in with the evaluation and optimization. And that's to do with something called overfitting. So, decision trees is an algorithm notorious for overfitting, depending on the data, that is. So, overfitting is when you get the algorithm to perform really well on the training data. And then you feed it in a new example and it might grossly misclassify it because it hasn't learned to generalize beyond the examples that you've given it. CHARLES: I see. Okay. So, it's just too concrete. It hasn't recognized deep patterns. It's only recognized something superficial. KATHARINE: Yes. So typically, if you have a finite amount of data, you only train your algorithm on a certain percentage. And then you test it on the rest. CHARLES: I see. KATHARINE: So, you hold back some data. But back to our conversation earlier about when does the training happen? Another example is something called K-nearest neighbors. You could just imagine that means three nearest neighbors, for example. So, we're trying to find who we're most similar to in a room. So, it's a room full of people. And all the people are standing next to already similar people, for example. So, you might have a room where marketing's in one corner and the software developers are in another corner and the project managers are in another corner. And you go it and you're looking for the three people who are the most similar to you and you're going to go and stand in that group, for example. So, in that type of machine learning, the training is happening as each new sample comes in, rather than upfront. And there are drawbacks to both methods and there are positives to both methods. And really, in a horrible, unsexy way, it's to do with the data. And that's normally where most people because it's the most boring part when you're talking about machine learning. CHARLES: Is the data? KATHARINE: Yeah. It's this backlash from data scientists, from the golden age of data scientists where it was the hottest job on the internet to now everyone cringing going, “Oh, I really don't want to deal with my data.” But with any machine learning problem, you can't just go, “Okay. Here you go, Charles. Here's a dataset. Learn something from it.” You need context. You need to understand it. You need to have an idea of what you're looking for. So, you're getting the machine to learn but you're using it as a tool to complement your knowledge, really. And you're feeding in your knowledge to it. And part of that are the decisions that you make on the algorithms to use. CHARLES: Okay. KATHARINE: And what you're looking for. CHARLES: So, here's something that I'm wondering is related, and again I have no idea – for some reason I always associate when people talk about neural nets as being related to teaching a computer something. Is that part of the discipline of machine learning? Or no? KATHARINE: I would say it is. But it has its own cool and trendy title of Deep Learning. But it's very much powerful machine learning. So, let's go back to this. This is a classic example of machine learning. It's probably the first example you'll come across if you do any course. House prices. So, imagine that you have a piece of paper and you're going to draw one line at one side and that is the price of a house, and you're going to draw a line at the bottom which is the size in square feet, and you're going to plot examples that you have. And you might find that there's a linear correlation between the two. So, you'll draw a line and that line of best fit is the human equivalent of doing linear regression on a computer, for example, where you're just trying to find a linear correlation between two things. And a lot of the principles in linear regression, which is a very simple learning algorithm, are found in some examples of neural networks, so some basic examples of neural networks. But instead of having one input, the size in square feet, you might have 20. And you might be repeating that process of trying to find the line of best fit with different combinations of features in different places again and again. And it scales up in complexity very quickly. But it's very similar basic principles. I'm hesitating to say ‘very similar' because they're notoriously more complicated. CHARLES: Yeah. I guess I'm not really divining what exactly, what makes it – why is it called a neural net? What makes it special? It sounds like if I'm just comparing the regressions of house prices, I'm comparing those datasets over and over again, how is that different from just a loop? What are you getting out of it? KATHARINE: Historically, neural networks come from a very simplified idea about how the brain works. So, in the early 20th century people had performed autopsies and divined the inner workings of kidneys and hearts and livers. And the brain was still a bit of a mystery. And then 2 men jointly won the Nobel Prize for Physiology, and I'm going to pronounce these names wrong, so I'm sorry. I think it's Santiago Ramón y Cajal is one and Camillo Golgi is another. And they completely disagreed about the brain but they used a staining technique from Golgi to look, using silver nitrate, at the cells in the brain. And the idea of the neuron doctrine was borne out of that, that the simplest unit to look at the brain at in order to understand it is the level of the neuron, this cell in the brain. And from there, several – well, everybody was a polyglot really, back then, so I don't want to say computer scientists. So, you had people like Frank Rosenblatt with perceptrons, Marvin Minsky and so many other people looking at a very simple idea which is that a neuron either fires or doesn't. And then you're linking boolean algebra to this cell. So, you're saying it either fires or it doesn't. And from that principle, people started drawing similarities between neurons and a basic function machine. And when I say function machine, I mean imagine when you were in elementary school and you're learning how adding up works. You might have a box with a plus on it and your teacher says, “I want you to put a three in a box and a four in a box and I want you to add them together. And what do you get out?” And obviously the answer is seven. And you can think about that little box with a plus as a function machine. So, now you could think of a little mathematical function machine where you put in some inputs and there something happens in the box. And then you'll either get a one or a zero out of it. CHARLES: And so, that's like your neuron, is the little box? KATHARINE: Yes. CHARLES: Okay. KATHARINE: Yes. So since then, the neuron doctrine is pretty much contested. There are several other elements of the brain that compose thinking and the circuitry. So, any cognitive scientist listening to this will say, “That's really not how the brain works.” You have to say it with a lot of disclaimers. But the whole idea of neural networks was borne out of this idea of thinking of a neuron as like a function machine. CHARLES: Right. And also, it doesn't actually discount the usefulness of neural networks. There are a lot of things where people didn't find what they set out to find but what they found was useful. KATHARINE: Absolutely. Yeah, and they are incredibly powerful, especially with multilayer networks which are deep networks or deep learning. CHARLES: Okay. So, I didn't want derail you from your explanation. So, you've got these little function boxes and those are the kind of neurons inside the neural network? KATHARINE: Yes. So, what happens is you might have a layer of 10 of them and you might have another layer after that of another 10. And each of them are connected in a simple neural network. And then you might have an output layer of five because you're looking to classify, I don't know, an apple into five different types of apple, for example. CHARLES: Now, when you're talking about a later, you're talking about, I've got the outputs of one layer of the network are the inputs to the next layer. KATHARINE: Yes. And in different neural network architectures they'll be connected differently. But in a simple neural network, you can assume that every neuron is connected to every neuron in the next layer. So, if you have two 10-layer, two layers each with 10 neurons in, the top neuron in one layer is going to have 10 connections going out of it into the next layer. CHARLES: Oh really? Wow, that's interesting. KATHARINE: Yes. And each connection has its own sort of configuration. CHARLES: So, you're like cross-wiring all the – okay. Wow, that's kind of… KATHARINE: And yeah, that's where the complexity comes in. CHARLES: Yeah. I was thinking it was like a simple exponential fan-out. But it's even more complicated, the number of combinations you can get. KATHARINE: Yes. But in theory, it's very simple because each neuron is like this function machine with inputs coming in and something going out. It just might go out to several different locations. But it scales up in complexity very quickly. So say we're classifying apples. So, we have five different attributes of an apple like its color, its texture, its weight, acidity. I don't know how else you would measure an apple. How shiny it is, for example. And we're putting all of that information in. And each one of those things that I've just listed is a feature. So, we might have an input per feature and we'll link them all up to the neurons in a layer and we'll move that information onto the next layer. And the really important thing is what's happening on those connections between them. Because on the connections you have weights, which is a way of changing the input from one neuron to another. So, the weight might be like 0.5 for example. So, it squashes the input. Or it might magnify it. And then you get the output. Now, once you have the output you can compare the output with what you know the right answer is. And then you have this idea of an error. So, you might be like, “You got this so wrong. It's not a Braeburn apple at all. It's actually a Granny Smith apple. And what you do is with each example that you train in, you use your information about the error to train the network. Because what you're trying to do is get the error to be as small as possible. And one of the techniques of that is called back propagation. And it's notoriously difficult to understand because it involves partial derivatives and a large element of calculus. But essentially, what you're doing is comparing the right answer with the answer that the network gave it and asking it to go back and change it, change those connection weights. CHARLES: And so, do you make them fluctuate at random or is there some – is there a method to the madness of changing the weights? KATHARINE: There is a method to the madness and it's called back propagation. And the reason I linked linear regression in early, so our really simple map of house size in square feet, and the price, is because it uses a similar technique called gradient descent which is an algorithm for looking at the error and changing the weight, those numbers on the connections, to try and get it to a minimal point. So, if we go back to our house price problem, I just want you to imagine in your mind that we've got this one axis going up which is the price, and one going across which is the size in square feet. And you've got line drawn, a diagonal line. If you just imagine now in your mind moving that line down till it's completely flat at the bottom, and then moving it up so it's vertical, so it's aligned with either axis in every point in between, what you could do is you could take the error on each of those lines. So, if we imagine we have these two axes, we have the price of a house on one side and we have the size in square feet in another. And we're going to draw a line from the top axis and we're going to sweep it down and draw a line at each point as it goes down until it's aligned with the bottom axis. And at each point we draw that line, we measure the error. So, what we'd do for that is all of the little points, all of the example data, we'd measure the difference between them and the line and we're going to, say, add them all up. So, what you'd end up with is you'd end up with a graph mapping the error against the gradient at all fo those different points. And it would look like a bowl. You'd have a lowest point. You'd have a point for some gradient where the error is the lowest. CHARLES: Right, yeah. Okay. I'm seeing it. I think I'm seeing it. So, you want to take that error function and you want to, what is it? Now this is – boy, I'm going back to high school math. You would take the derivative and find out where the tangent is and that's your min point? That's the root of the equation and that's the point where your error is lowest? KATHARINE: Yeah, exactly. Or there's an algorithm called gradient descent that does it automatically by taking little steps. So, it looks at the tangent of the gradient at a point and says, “If I move the gradient down is the error going to decrease. And if so, move in that direction.” So automatically, it tries to take steps to get to the bottom. And optimization problem with that is that you can configure the step size. So, you could take tiny, tiny steps and take forever to get to the bottom or you could take massive steps and completely miss the bottom. So, you can imagine it like walking down a hill. If you're a minion then it will take a really long time because you're tiny. And if you're a giant, you might never get to the valley. CHARLES: Right. You might just leap right across the chasm. KATHARINE: Yeah, just miss it completely. So, that's linear regression where we're looking. And that's really, even though we have a two-dimensional graph that's a one-dimensional problem because we just have the one input feature. Now, when we're looking at neural networks and we're looking at gradient descent in neural networks, each one of those connections is something that we're trying to configure to reduce the error. So suddenly, you have maybe a hundred-dimension landscape and you're trying to get to the bottom of a hill. And there might be several local optimas and one deepest valley, but you might have lots of other valleys that you could get stuck in. So, it becomes a very difficult problem. Does that make sense? CHARLES: Yes. No, that does make sense. I'm just trying to let it sink in. KATHARINE: It's completely impossible to visualize a hundred dimensions, yeah. CHARLES: I actually had to sit back and kind of close my eyes and stare up at the ceiling. KATHARINE: I think the trick is not to think about it. I heard someone say, there's another fantastic course on Coursera by a famous computer scientist studying neural networks called Geoffrey Hinton. And his advice in one of the videos on visualizing these multidimensional landscapes, say it's 15 dimensions, is to close your eyes and shout, “15!” And that hasn't worked for me, but I'm sure it's worked for some people. CHARLES: But it certainly probably makes you feel better. KATHARINE: Yeah. I think it's one of those things that's just beyond comprehension. But we can just quietly accept it. CHARLES: Right. It's just – yeah, what's nice about I guess math is you just don't have to understand it. I mean, you do. You just understand that there really is no mapping to our physical experience. And that's okay. And just let that go. It's just like, this is just some… KATHARINE: Yeah. CHARLES: We had some numbers that existed in the domain of understanding, that we can understand. And there are just some rules that we follow and if you look at the intermediate steps, well the points don't really exist in that domain of physical experience and understanding. And that's okay. We just accept and let it go. We just hope that at some point, we can translate that model back into the domain of ‘we can understand it'. KATHARINE: Oh, completely. There's a lot of faith. CHARLES: Yeah. KATHARINE: And also, I remember coming into machine learning and thinking, “This is like magic. It's amazing.” And then you study it a bit and you're like, “This is so easy. This is just basic – this is just functions. These are just glorified function machines.” And then you look into it some more and you're like, “Nope. It's definitely magic.” CHARLES: Yeah. It's a phase of, every time you come up against the wall, right? And then you realize, “Oh no, it's actually something that I can close my mind over,” until you come to the next hurdle of magic. KATHARINE: Yeah. You think you've got a grasp on things and you think, “I know the landscape,” and then you suddenly realize how much more there is to learn. And that sinking feeling that you'll never learn it all. CHARLES: Yeah, yeah. Yup. Unfortunately, it seems like in tech that's like, that's just the condition. KATHARINE: Yeah, like all of my sad, unread books. CHARLES: If I wanted to get into just really start experimenting with this stuff and start saying, “Maybe I can utilize some of these techniques for some of these problems that I'm encountering,” Where would be a good place to get started? What libraries? What online resources? What people are good to follow and ask questions of? KATHARINE: Okay. Let's start with websites. So for a start, this website called Kaggle which I mentioned earlier, and that is K-A-G-G-L-E dot com. And that has a lot of dataset resources. It also has a community of people discussing how they use the datasets. It has competitions. And it has a lot of links. I discovered recently as well a really good blog. It's on Medium and it is called ‘Machine Learning for Humans'. And that's really well-written. I really like it, actually, and it has a good section on resources called ‘The Best Machine Learning Resources'. And I should probably plug my own blog, but this one's so much better. CHARLES: But please do. KATHARINE: No, no. I have to actually write stuff for it. But there's a lot of things there about, well, if you want to learn linear algebra, what if you want to learn probability and statistics, calculus, and then just go straight to machine learning and pick up the math on the way. I would say go on Coursera, because there are courses like Andrew Ng's course on machine learning from Stanford. And Geoffrey Hinton from the University of Toronto. But there are also courses there on things like calculus and probability and statistics if you want to level up your math. If you don't want to do anything to do with the math, I would say go to the vendor websites, like AWS, Google. If you're into Java, go on deep learning for J. And a lot of them have tutorials that complement their products. Deep learning for J is one of my favorites at the moment. It's an open source Java library. It's pretty plug and play, actually. You don't need to understand a lot of it to get started with it. But it helps. And obviously then there's TensorFlow although personally, I find just other Python libraries like SciPy a lot easier than using TensorFlow. And I think naturally you'll find the resources and the people to follow on Twitter from that. But the crucial thing I'd say is don't get hung up on which language to start playing around with. So, a lot of people say, “Oh, I must need to know Python. I must need to know math.” And really, you don't. It just depends on what level you want to approach things at. So, if you want to write your own gradient descent algorithms, then Python is probably more for you, or Matlab or R or something like that. But there are libraries where you can do it in Java. I've heard rumors that there's a JavaScript library and I wouldn't be surprised. So, I would just have a look at what's out there. But try and get a grasp of the fundamentals just from an intuition point of view, because it will make your life so much easier. You might for example realize that you're using the completely wrong algorithm for the problem that you're looking at. And that's invaluable. CHARLES: Yeah. Knowing what not to do certainly is. Alright. Well, thank you so much for that, Katharine. Thank you for being on the show. Thank you for curing us of at least a small portion of our ignorance. And if people want to get in touch with you perhaps and continue the conversation, or follow you, what's a good place to get in touch? KATHARINE: Sure. Probably tweet me on Twitter. I'm @KatharineCodes but it's spelled like Katharine Hepburn. It's K-A-T-H-A-R-I-N-E codes. Because when I joined Twitter, I didn't have much of an imagination. I still don't. So, it's not particularly clever. But it's there. CHARLES: Alright. Well, fantastic. And for everybody listening at home, you can also get in touch with us. We're @TheFrontside on Twitter. Or you can just drop us a line at info@frontside.io. Thanks for listening and we will see you all next time.

Julie Moronuki: @argumatronic | argumatronic.com Show Notes: This episode is a follow-up episode to the one we did with Julie in September: Learn Haskell, Think Less. We talk a whole lot about monoids, and learning programming languages untraditionally. Transcript: CHARLES: Hello everybody and welcome to The Frontside Podcast, Episode 93. My name is Charles Lowell, a developer here at The Frontside and I am your podcast host-in-training. With me today from The Frontside is Elrick also. Hello, Elrick. ELRICK: Hey. CHARLES: How are you doing? ELRICK: I'm doing great. CHARLES: Alright. Are you ready? ELRICK: Oh yeah, I'm excited. CHARLES: You ready to do some podcasting? Alright. Because we actually have a repeat guest on today. It was a very popular episode from last year. We have with us the author of ‘Learning Haskell: From First Principles' and a book that is coming out but is not out yet but one that we're eagerly looking forward to, Julie Moronuki. Welcome. JULIE: Hi. It's great to be back. CHARLES: What was it about, was it last October? JULIE: I think it was right before I went to London to Haskell [inaudible]. CHARLES: Yeah. JULIE: Which was in early October. So yeah… CHARLES: Okay. JULIE: Late or early October, somewhere in there. CHARLES: Okay. You went to Haskell eXchange. You gave a talk on Monoids. What have you been up to since then? JULIE: Oh wow. It's been a really busy time. I moved to Atlanta and so I've had all this stuff going on. And so, I was telling a friend last night “I'm going to be on this podcast tomorrow and I don't think I have anything to talk about.” [Laughter] JULIE: Because I feel like everything has just been like, all my energy has been sucked up with the move and stuff. But I guess… CHARLES: Is it true that everybody calls it ‘Fatlanta' there? JULIE: Yeah. [Laughs] CHARLES: I've heard the term. But do people actually be like “Yes, I'm from Fatlanta.” JULIE: I've heard it a couple of times. CHARLES: Okay. JULIE: Maybe it's mostly outsiders. I'm not sure. CHARLES: [Chuckles] JULIE: But yeah, it's a real cool city and I'm real happy to be here. But yeah, I did go in October. I went to London and I spoke at Haskell eXchange which was really amazing. It was a great experience and I hope to be able to go back. I got to meet Simon Payton Jones which was incredible. Yeah, and I gave a talk on monoids, monoids and semirings. And… CHARLES: Ooh, a semiring. JULIE: Semiring. So, a semiring is a structure where there's two monoids. So, both of them have an identity element. And the identity element of one of them is an annihilator. Isn't that a great word? It's an annihilator… CHARLES: Whoa. JULIE: Of the other. So, if you think of addition and multiplication, the identity element for addition is zero, right? But if you multiply times zero, you're always going to get to zero, so it's the annihilator of multiplication. CHARLES: Whoa. I think my mind is like annihilated. [Laughter] JULIE: So, it's a structure where you're got two monoids and one of them distributes over the other, the distributive property of addition and multiplication. And the identity of one of them is the annihilator of the other. Anyway, but yeah, I gave a history of where monoids come from and that was really fun. CHARLES: Yeah. I would actually like to get a summary of that, because I think since we last talked, I've been getting a little bit deeper and deeper into these formal type classes. I'm still not doing Haskell day-to-day but I've been importing these ideas into just plain vanilla JavaScript. And it turns out, it's actually a pretty straightforward thing to do. There's definitely nothing stopping these things from existing in JavaScript. It's just, I think people find type class programming can be a tough hill to climb or something like that, or find it intimidating. JULIE: Yeah. CHARLES: But I think it's actually quite powerful. And I think one of the things that I'm coming to realize is that these are well-worn pathways for composing things. JULIE: Right. CHARLES: So, what you encounter in the wild is people generating these one-off ways of composing things. And so, for a shop like ours, we did a lot of Ruby on Rails, a lot of Ember, and both of those frameworks have very strong philosophical underpinnings that's like “You shouldn't be reinventing the wheel if you don't have to.” I think that all of these patterns even though they have crazy quixotic esoteric names, they are the wheels, the gold standard of wheel. [Laughs] They're like… JULIE: Right. CHARLES: We should not be reinventing. And so, that's what I'm coming to realize, is I'm into this. And last time you were talking, you were saying “I find monoids so fascinating.” I think it took a little bit while to seep in. But now, I feel like it's like when you look at one of those stereo vision things, like I'm seeing monoids everywhere. It's like sometimes they won't leave me alone. JULIE: In ‘Real World Haskell' there's a line I've always liked. And I'm going to misquote it slightly but paraphrasing at least. “Monoids are ubiquitous in programming. It's just in Haskell we have the ability to just talk about them as monoids.” CHARLES: Yeah, yeah. JULIE: Because we have a name and we have a framework for gathering all these similar things together. CHARLES: Right. And it helps you. I feel like it helps you because if you understand the mechanics of a monoid, you can then when you encounter a new one, you're 90% there. JULIE: Right. CHARLES: Instead of having to learn the whole thing from scratch. JULIE: Right. And as you see them over and over again, you develop a kind of intuition for when something is monoidal or something looks like a semiring. And so, you get a certain intuition where you think, “Oh, this thing is like a… this is a monad.” And so, what do I know about monads? All of a sudden, this new situation like all these things that I know about monads, I can apply to this new situation. And so, you gain some intuition for novel situations just by being able to relate them to things you already do know. CHARLES: Exactly. I want to pause here for people. The other thing that I think I've come in the last three months to embrace is just embrace the terminology. JULIE: Yeah. CHARLES: You got to just get over it. JULIE: [Chuckles] CHARLES: Think about it like learning a foreign language. The example I give is like tasku is the Finnish word for pocket. JULIE: Right. CHARLES: It sounds weird, right? Tasku. But if you say it 10 times and you think “Pocket, pocket, pocket, pocket, pocket.” JULIE: Yes, yeah. [Laughs] CHARLES: Then it's like, this is a very simple, very useful concept. JULIE: Right. CHARLES: And it's two-sided. There on the one hand, the terminology is obtuse. But at the same time, it's not. It's just, it is what it is. And it's just a symbol that's referencing a concept. JULIE: Right, right. CHARLES: It's a simple concept. So, I just want to be… I know for our listeners, I know that there's a general admonition. Don't worry about the terminology. It's… JULIE: Right, right. Like what I just said, I said the word ‘monad'. I just threw that out there at everybody, but [chuckles] it doesn't matter which one of these words we'd be talking about or whatever I call them. We could give monads a different name and it's still this concept that once you understand the concept itself, and then you can apply it in new situations, it doesn't matter then what it's called. But it does take getting used to. The words are… well, I think functor is a pretty good word for what it is. If you know the history of functor and how it came to mean what it means, I think it's a pretty good word. CHARLES: Really? So, I would love to know the history. Because functor is mystifying to me. It sounds like, I think the analogy I use is like if George Clinton and a funk parliament had an empire, the provinces, the governors of the provinces would be functors. ELRICK: [Laughs] JULIE: Yes. CHARLES: But [Laughs] that's the closest thing to an explanation I can come up with. JULIE: I might use that. I'm about to give a talk on functors. I might use that. [Laughter] ELRICK: Isn't that the name of the library? Funkadelic? CHARLES: Well, that's the name of the library that I've been… JULIE: [Could be], yeah. ELRICK: That you'd been… CHARLES: That I'd been [writing] for JavaScript. ELRICK: Yeah. CHARLES: That imports all these concepts. JULIE: [Laughs] ELRICK: Yeah. JULIE: Yeah. ELRICK: So awesome. JULIE: Yeah. Yeah, I have… CHARLES: So, what is the etymology of functor? JULIE: Well, as far as I can tell, Rudolf Carnap, the logician, invented the word. I don't know if he got it from somewhere else. But the first time I can find a reference to it is in, he wrote a book about… he was a logician but this is sort of a linguistics book. It's called ‘The Logical Syntax of Language'. And that's the first reference I know of to the word functor. And he was trying to really make language very logically systematic, which natural language is and isn't, right? [Chuckles] CHARLES: Right. JULIE: But he was only concerned with really logically systematizing everything. And so, he used the word functor to describe some kinds of function words in language that relate one part of a sentence to another part of a sentence. CHARLES: Huh. So, what's an example? JULIE: So, the example that I've used in the past is, as far as I know this is not one that Carnap himself actually uses but it's the clearest one outside of that book… well the ones inside the book I don't really think are very good examples because they're not really how people talk. So, the one that I've used to try to explain it is the word ‘not' in English where ‘not' gets applied to the whole sentence. It doesn't really change the logical structure of the sentence. It doesn't change the meaning of the sentence except for now it negates the whole thing. CHARLES: I see. JULIE: And so, it relates this sentence with this structure to a different context, which is now the whole thing has been negated. CHARLES: I see. So, the meaning changes, but the structure really doesn't. JULIE: Right. And it changes the whole meaning. CHARLES: Right. JULIE: Not just part of the sentence. So, if you imagine ‘not' applying to an entire sentence because of course we can apply it just to a single word or just to a single phrase and change the meaning just of that word or that phrase, but if you imagine a context where you've applied ‘not' to a whole sentence, to an entire proposition, because of course he's a logician. So, if you've applied ‘not' to an entire proposition, then it doesn't change the structure or the meaning of that proposition per se except for it just relates it to the category of negated propositions. CHARLES: Mmhmm. JULIE: So, that's where it comes from. And… CHARLES: But I still don't understand why he called it functor. JULIE: He's sort of making up… well, actually I think the German might be the same word. CHARLES: Ah, okay. JULIE: Because he was writing in German. Because he's looking for something that evokes the idea of ‘function word'. CHARLES: Oh. JULIE: So, if you were to take the ‘func' of ‘function' [Laughs] and the, I don't know, maybe in German there's some better explanation for making this into a particular word. But that's how I think of it. So, it's ‘function word'. And then category theorists took it from Carnap to mean a way to map a function in this category or when we're talking about Haskell, a function of this type, to a function of another type. CHARLES: Okay. JULIE: And so, it takes the entire function, preserves the structure of the function just like negation preserves the structure of the sentence, and maps the whole thing to just a different context. So, if you had a function from A to B, functor can give you a function from maybe A to maybe B. CHARLES: Right. JULIE: So, it takes the function and just maps it into a different context. CHARLES: Right. So, a JavaScript example is if I've got an array of ints and a function of ints to strings, I can take any array of ints and get an array of strings. JULIE: Right. CHARLES: Or if I have a promise that has an int in it, I can take that same function to get a promise of a string. JULIE: Yeah. CHARLES: Yeah. I had no idea that it actually came from linguistics. JULIE: Yeah. [Laughs] CHARLES: So actually, the category theorists even… it digs deeper than category theory. They were actually borrowing concepts. JULIE: They were, yes. CHARLES: We just always are borrowing concepts. ELRICK: I like the borrowing of concepts. JULIE: Yeah. ELRICK: I think where people struggle with certain things, it's tying it back to something that they're familiar with. So, that's where I get… my mind is like [makes exploding sound] “I now get it,” is when someone ties it back to something that I am… CHARLES: Right. ELRICK: Familiar with. Like Charles' work with the JavaScript, tying it with JavaScript. I'm like, “Oh, now I see what they're talking about.” JULIE: Right. CHARLES: because you realize, you're using these concepts. People are using them, just they're using them anonymously. JULIE: Right. ELRICK: True. CHARLES: They don't have names for them. JULIE: Right. ELRICK: True. CHARLES: It's literally like an anonymous function and you're just taking that lambda and assigning it to a symbol. JULIE: Yeah. CHARLES: You're like “Oh wait. I've been using this anonymous function all over the place for years. I didn't realize. Boom. This is actually a formal concept.” ELRICK: True. And I think when people say like “Don't reinvent the wheel” it's a great statement for someone that has seen a wheel already. [Laughter] ELRICK: You know what I'm saying? If you never saw a wheel, then your'e going to reinvent the wheel because you're like “Aw man. This doesn't exist.” [Chuckles] JULIE: Yeah. ELRICK: But if people are exposed to these concepts, then they wouldn't reinvent the wheel. CHARLES: Right. JULIE: Right. Yeah. CHARLES: Instead of calling in some context, calling it a roller. [Chuckles] It's a round thingy. [Laughter] JULIE: Right. Yeah, so that's a little bit what I tried to do in my monoid talk in London. I tried to give some history of monoid, where this idea comes from and why it's worth talking about these things. CHARLES: Yeah. JULIE: Why it's worth talking about the structure. CHARLES: So, why is it worth the… where did it come from and why is it worth talking about? JULIE: Oh, so back when Boole, George Boole, when he decided to start formalizing logic… CHARLES: George Boole also, he was a career-switcher too, right? He was a primary school teacher. JULIE: Right, yeah. CHARLES: If I recall. He actually, he was basically teaching. Primary school is like elementary school in England, right? JULIE: I believe so, yes. CHARLES: Yeah. I think he was like, he was basically the US equivalent of an elementary school teacher who then went on to a second and probably, thankfully a big career that left a big legacy. JULIE: Right. Although no one knew exactly how big the legacy was really, until Claude Shannon picked it up and then just changed the whole world.[Laughs] Anyway, so Boole, when he was trying to come up with a formal algebra of logic so that we could not care so much about the semantic content of arguments (we could just symbolize them and just by manipulating symbols we could determine if an argument was logically valid or not), he was… well, for disjunction and conjunction which is AND and OR – well, disjunction would be the OR and conjunction the AND – he had prior art. He had addition and multiplication to look at. So, addition is like disjunction in some important ways. And multiplication is like conjunction in some important ways. And I think it took me a while to see how addition and disjunction were like each other, but there are some important ways that they're like each other. One of them is that they share their identity values. If you think of, it's sort of like binary addition and binary multiplication because in boolean logic there's only two values: true or false. So, you have a zero and a one. So, if you think of them as being like binary addition and binary multiplication then it's easier to see the connection. Because when we think of addition of just integers in a normal base 10 or whatever, it doesn't seem that much like an OR. [Laughs] CHARLES: Mmhmm. No, it doesn't. JULIE: [Inaudible] like a logical OR. So, it took me a while to see that. But they're also related then to set intersection and union where intersect-… CHARLES: So can… Let's just stop on that for a little bit, because let me parse that. So, for OR I've got two values, like in an ‘if' statement. This OR that. If I've got a true value then I can OR that with anything and I'll get the same anything. JULIE: Right. CHARLES: So, true is the identity value of OR, right? Is that what you're saying? So, one… JULIE: Well, it's false that's the identity of OR. CHARLES: Oh, it is? JULIE: Zero is the identity of addition. CHARLES: Wait, but if I take ‘false OR one' I get… oh, I get one. JULIE: Right. CHARLES: Okay. So, if I get ‘false OR true', I get true. Okay, so false is the identity. JULIE: Yeah. CHARLES: Oh right. You're right. You're right. Because… okay, sorry. JULIE: So, just like in addition, zero is the identity. So, whatever you add to zero, that's the result, right? You're going to get [the same] CHARLES: Right. JULIE: Value back. So, with OR false is the identity and false is equivalent to zero. CHARLES: [Inaudible] ‘False OR anything' and you're getting the anything. JULIE: Right. So, the only time you'll get a false back is if it's ‘false OR false', right? CHARLES: Right. Mmhmm. JULIE: Yeah. So, false is the identity there. And then it's sort of the same for conjunction where one is the identity of multiplication and one is also the… I mean, true is then the identity of logical conjunction. CHARLES: Right. Because one AND… JULIE: ‘True AND false' will get the false back. [Inaudible] CHARLES: Right. ‘True And true' you can get the true back. JULIE: Yeah. CHARLES: Okay. JULIE: And it's also then true, getting back to what we were talking about, semirings, it's also true that false is a kind of annihilator for conjunction. That's sort of trivial, because… CHARLES: Oh, because you annihilate the value. JULIE: Right. When there's only two values it's a little bit trivial. But it is [inaudible]. So… CHARLES: But it's [inaudible]. Yeah. It demonstrates the point. JULIE: Right. CHARLES: So, if I have yeah, ‘false AND anything' is just going to be false. So, I annihilate whatever is in that position. JULIE: Right. CHARLES: And the same thing as zero is the annihilator for multiplication, right? JULIE: Right. CHARLES: Because zero times anything and you annihilate the value. JULIE: Yeah. CHARLES: And now I've got… okay, I'm seeing it. I don't know where you're going with this. [Laughter] ELRICK: Yeah. CHARLES: But I'm there with you. ELRICK: Yup. JULIE: And then it turns out there are some operations from set theory that work really similarly. So, intersection and union are similar but the ones that are closer to conjunction/disjunction are disjoint unions and cartesian products. So we don't need to talk about those a whole lot if you're not into set theory. But anyway… CHARLES: I like set theory although it's so hard to describe without pictures, without Venn diagrams. JULIE: It is. It really is, yeah. So anyway, all of these things are monoids. And they're all binary associative operations with identity elements. So, they're all monoids. And so, we've taken operations on sets, operations on logical propositions, operations on many kinds of numbers (because not all kinds of addition and multiplication I guess are associative), and we can kind of unify all of those into the same framework. And then once we have done that, then we can see that there's all these other ‘sets'. Because most of the kinds of numbers are sets and there are operations on generic sets with set theory. So, now we can say “Oh. We can do these same kinds of operations on many other kinds of sets, many other varieties of sets.” And we can see that same pattern. And then we can get a kind of intuition for “Well, if I have a disjunctive monoid where I'm adding two things or I'm OR-ing two things…” Because even though those are logically very similar, intuitively and in terms of what it means to concatenate lists versus choosing one or the other, those obviously have different practical effects. CHARLES: So, I'm going to try and come up with some concrete examples to maybe… JULIE: Okay, yeah. CHARLES: A part of them will probably be like in JavaScript, right? So, to capture the idea of a disjunctive monoid versus a conjunctive monoid. So, a disjunctive monoid is like, so in JavaScript we're got two objects. You concat them together and it's like two maps or two hashes. So, you mash them together and you get… so, for the disjunctive one you'd have all the keys from both of the hashes inside the resulting object. You take two objects. Basically we call it object assign in JavaScript where you have basically the empty object. You can take the empty object and then take any number of objects. And so, we talked about… JULIE: That would become a disjunctive monoid, right? CHARLES: That would be a disjunctive monoid because you're like basically, you're OR-ing. Yeah. JULIE: You're kind of, [inaudible] CHARLES: Hard to find the terminology. JULIE: Yeah. CHARLES: But like object assign would be a disjunctive monoid because you're like mashing these two objects. And the resulting object has all of the things from both of them. JULIE: Right. So, it's like a sum of the two, right? CHARLES: Right, right. Okay, so then another one would be like min or max where you've got this list of integers and you can basically take any two integers and you can mash them together and if you're using min, you get the one that's smaller. Basically, you're collapsing them into one value but you're actually just choosing one of them. Is that like… JULIE: Yeah. CHARLES: Would that be like a conjunctive monoid? JULIE: No, that's also disjunctive but that's more like an OR than like a sum. CHARLES: Okay. JULIE: Right. So, that's what I said. It's hard to think of disjunctive monoids I think because there's really two varieties. There's some underlying logical similarity, like the similarity in the identity values. But they're also different. Summing two things versus choosing one or the other are also very different things in a lot of ways. CHARLES: Right. Okay. JULIE: And so, I think the conjunctive monoids are all a little bit more similar, I think. [Chuckles] But the disjunctive monoids are two broad categories. And we don't really have a monoid in Haskell of lists where you're choosing one or the other. The basic list monoid is you're concatenating them. So, you're adding two lists or taking the union of them. But for maybe, the maybe type, we do have monoids in Haskell where you're just choosing either the first just value that comes up or the last just value that comes up. So, we do have a monoid of choice over the maybe type. And then we have a type class called alternative which is monoids of choice for… so, they're disjunctive monoids but instead of adding the two things together, they're choosing one or the other. CHARLES: Okay. JULIE: Though we have a type class for that. [Laughs] CHARLES: [Sighs] Oh wow. Yeah. JULIE: Mmhmm, yeah. CHARLES: I'l have to go read up on that one. JULIE: That type class comes up the most when you're parsing, because you can then parse… like if you found this thing, then parse this thing. But if you haven't found this thing, then you can keep going. And if you find this other thing later, then you can take that thing. So, you allow the possibility of choice. The first thing that you come to that matches, take that thing or parse that thing. So, that type class gets mostly used for parsing but it's not only useful for parsing. CHARLES: Okay. JULIE: So yeah. That's the most of the time when I've used it. CHARLES: Is this when you're like parsing JSON? Or is this when you're just searching some stream for some value? Like you just want to run through it until you encounter this value? Or how does that…? JULIE: Right. Say you want to run through it until you find either this value or this value. I've used it when I've been parsing command line arguments. So, let's say I have some flags that can be passed in on my command line command. There are some flags that could be passed in. So, we'll parse until we find this thing or this thing. This flag or this flag. So, if you find this flag, then we're going to go ahead and parse that and do whatever that flag says to do. If you don't find that first flag then we can keep parsing and see if you find this other flag, in which case we'll do something different. CHARLES: Okay. JULIE: It'll take the first match that it finds. Does that make sense? CHARLES: Yeah, yeah, yeah. It does. But I'm not connecting how it's a monoid. [Laughs] JULIE: How is that a monoid? Well, because it's a monoid of OR-ing CHARLES: What's the identity value or the empty value in that case? JULIE: Well, the empty value would be… let's say you have maybes. Let's say you have some kind of maybe thing, so you're parser is going to return maybe this thing, maybe whatever you're parsing. Like maybe string. CHARLES: Yeah, yeah. JULIE: So, it's going to return a maybe string. So well, nothing would be the empty. CHARLES: Okay. JULIE: But nothing is like the zero because it's a disjunction, logical OR. So, only when you have two nothings will you get back a nothing. Otherwise, it will take the first thing that it finds. CHARLES: Okay. I see. JULIE: Yeah. So, the identity then is the nothing, like false is the identity for disjunction. CHARLES: Mmhmm. Okay. JULIE: Yeah. CHARLES: [Inaudible] JULIE: Yeah. If you have nothing or this other thing, then you return this other thing. Then you return the maybe string. If you have two nothings, then you get in fact nothing. Your parsing has failed. CHARLES: Right, because you've got nothing. JULIE: Because you've got nothing. There was nothing to give you back. CHARLES: So, you concatenated all of the things together and you ended up with nothing. JULIE: Right, because there was nothing there. CHARLES: Right. [Laughs] JULIE: You found nothing. So, it's useful when you've got some possibilities that could be present and you just want to keep parsing until you find the first one that matches. And then it'll just return whatever. It'll just parse the first thing that it matches on. CHARLES: Okay, okay. JULIE: Does that make sense? CHARLES: Yeah. No, I think it makes sense. JULIE: I'm not sure. Because I feel like I kind of went down a rabbit hole there. [Laughs] CHARLES: Yeah. [Laughs] No, no. I think it makes sense. And as a quick aside, I think… so, I was, when we were talking about min and max, are min and max also like a semiring? Because negative infinity is the annihilator of min and it's the identity of max. and positive infinity is the annihilator of max but it's the identity of min. JULIE: I guess. I don't really think of min and max as having identities. Is that how [inaudible]? CHARLES: I'm just, I don't know. Well, I think if you have negative infinity and you max it with anything, you're going to get the anything, right? Negative infinity max one is one. Negative infinity/minus a billion is minus a billion. JULIE: Yeah, okay. CHARLES: I don't know. Just off the cuff. I'm just trying to… annihilators sound cool. And so… [Laughter] CHARLES: And so I'm like, I'm trying to find annihilators. JULIE: Yeah, they are cool. CHARLES: [Laughs] JULIE: One of my friends on Twitter was just talking about how he used the intuition at least of a semiring at work because he had this sort of monoid to concatenate schedules. So, he's got all these different schedules and he's got this kind of monoid to concatenate them, to merge the schedules together. But then he's got this one schedule that is special. And whenever something is in this schedule, it needs to hard override every other schedule. CHARLES: Right. JULIE: And so, that was like the annihilator. So, he was thinking of it as a semiring, because that hard override schedule is like the annihilator of all the other schedules. CHARLES: Yeah. JULIE: If anything else exists on this day or whatever, then it'd just get a hard override. So, there's a real world use. [Laughs] CHARLES: Yeah, a real world example. That's the thing that I'm finding, is that all these really very crystalline abstractions, they still play out very well I think in the real world. And they're useful as a took in terms of casting a net over a problem. Because you're like… when I'm faced with something new, I'm like “Well, let's see. Can I make it a functor?” And if I can, then I've unlocked all these goodies. I've unlocked every single composition pattern that works with functor. JULIE: Right, right. CHARLES: And it's like sometimes it fits. It almost feels like when you're working on something at home and you've got some bolt and you're trying on different diameters. So you're like, “Oh, is it 15 millimeter? Is it 8 millimeter?” JULIE: Right. [Laughs] CHARLES: “Like no, okay. Maybe it'll work with this.” But then when it clicks, then you can really ratchet with some serious torque. JULIE: Right, right. Yeah. CHARLES: So, yeah. Definitely trying to look for semirings [Laughs] is definitely beyond my [can] at this point. But I hope to get there where it can be like, if it's a fit, it's a fit. That's awesome. JULIE: Right. Yeah, it's kind of beyond my can too. Semirings are still a little bit new for me and I can't say that I find them in the wild as it were, as often as monoids or something. But I think it just takes seeing some concrete examples. So, now you know this idea exists. If you just have some concrete examples of it, then over time you develop that intuition, right? CHARLES: Right. JULIE: Like “Okay, I've seen this pattern before.” [Chuckles] CHARLES: yeah. Basically, every time now I want to fold a list, or like in JavaScript, any time you want to reduce something I'm like “There's a monoid here that I'm not seeing. Let me look for it.” JULIE: Yeah. Oh, that's cool, yeah. CHARLES: Because like, that's basically, most of the time you're doing a reduce, then like I said that's the terminology for fold in JavaScript, is you start with some reducible thing. Then you have an initial value and a function to actually concatenate two things together. JULIE: Right. CHARLES: And so, usually that initial state, that's your identity. And then that function is just your concat function from your monoid. And so, usually anytime I do a reduce, there's the three pieces. Boom. Identity value, concatenation function, it's usually right there. And so, that's the way I've found of extracting these things, is I'm very suspicious every time I'm tempted to… JULIE: [Laughs] CHARLES: A fold. I'm like “Hmm. Where's the monoid I'm missing? Is it [under the] couch?” Like, where is it? [Laughs] Because it just, it cleans it up and it makes it so much more concise. JULIE: Oh yeah, that's awesome. CHARLES: So anyhow. JULIE: Have we totally lost Elrick? ELRICK: Nope, I'm still here. JULIE: Okay. [Laughs] ELRICK: I'm sitting in and listening to you two break down these complex topics is really good. Because you guys break them down to a level where it's consumable by people that barely understand it. So, I'm just sitting here just soaking everything in like “Oh, that's awesome.” Taking notes. Yes, okay, okay. [Laughter] JULIE: Cool. ELRICK: So, I'm like riding the train in the back just hanging out, feeling the cool breeze while you guys just pull the train ahead in… [Laughter] ELRICK: In the engine department, you know? It's awesome. CHARLES: Yeah. ELRICK: I don't know if they're related. But you were talking about semirings and I heard of semigroups or semigroups. I have no idea if those two things are related. Are they related or [inaudible]? JULIE: They're kind of related. So, a semigroup is like a monoid but doesn't have an identity value. CHARLES: What is an example of a semigroup out there in the wild? Because every time I find a semigroup, I feel like it's actually a monoid. JULIE: Well, you know I feel like that a lot, too. We do have a data type in Haskell that is a non-empty list. So, there is no empty list CHARLES: Ah, right. Okay. JULIE: So then you can concatenate those lists, but there's never an identity value for it. CHARLES: I see. JULIE: Yeah. So, that's a case. There's actually a lot of comparison functions, greater than and less than. I think those are semigroups because they're binary, they're associative, but they don't have an identity value. Like if you're comparing two numbers, there's not really an identity value there. CHARLES: Right. Well, would the negative infinity work there? Let's see. Like, negative infinity greater than anything would be the anything. Well, okay wait. But greater than, that takes numbers and yields a boolean, right? JULIE: Yeah, CHARLES: Right. So, it couldn't be… could it be a semigroup? Don't semigroups have to… Doesn't the [inaudible] function have to yield the same type as the operands? JULIE: Yes. CHARLES: But a non-empty list, that's a good one. Sometimes it's basically not valid for you to have a list that doesn't have any elements, right? Because it's like the null value or the empty value and it could be like a shopping cart on Amazon. You can't have a shopping cart without at least something in it. JULIE: Right. CHARLES: Or, you can't check out without something. So, you might want to say like the shopping cart that I'm going to check out is a non-empty list. And so, you can put two non-empty lists together. But yeah, there's no value you can mash together, you can concat with anything, that isn't empty. JULIE: Right. CHARLES: So, I guess going back to your question Elrick, I don't know if it's related to semiring. But semigroup is just, it's like one-half of monoid. It's the part that concats two values together. JULIE: Right. Well, yeah. And so, it's supposed to be half a group, right? But I don't remember… CHARLES: [Laughs] JULIE: [Inaudible] all of the group stuff is, all the stuff that these types have to have to be a group. And similarly, I forget what the difference between semiring and ring is. [Chuckles] Because a ring and a group I know are not the same thing. But I forget what the difference is, too. So, I kind of got a handle on what semigroups are, and I know all my Haskell friends are going to, when they hear this podcast they're going to tweet all these examples of semigroups at me, especially my coauthor for ‘Joy of Haskell', Chris Martin. He's really into semigroups. And so, I know he's going to be very disappointed in my inability to think… [Laughter] JULIE: To think of any good examples. But it's not something that I find myself using a lot, whereas semirings are something that I have started noticing a little bit more often. So, how a monoid relates to a group is something that I can't remember off the top of my head. And I know how semirings relate to monoids, but how monoids then relate to rings and groups, I can't really remember. And so, these things are sort of all related. But the relation is not something I can spill out off the top of my head. Sorry. [Laughs] CHARLES: No, It's no worries. You know, I feel like… ELRICK: It's all good. CHARLES: What's funny is I feel like having these discussions is exactly like the discussions people have with any framework of using one that we use a lot, which is EmberJS. But if you could do with React or something, it's like, how does the model relate to the controller, relate to the router, relate to the middleware, relate to the services? You just have these things, these moving parts that fit together. And part of… I feel like exploring this space is really, absolutely no different than exploring any other software framework where you just have these things, these cooperating concepts, and they do click together. But you just have to map out the space in your head. JULIE: Yeah. This is going to sound stupid because everybody thinks that because I know Haskell I must know all these other things. But I just had to ask people to recommend me a book that could explain the relationship of HTML and CSS, because that was completely opaque to me. CHARLES: [Laughs] Yeah. JULIE: I've been involved in the making now of several websites because of the books and stuff like that. And I have a blog. It's not WordPress or anything. I did that sort of myself. So, I've done a little bit with that. But CSS is really terrifying. And… CHARLES: Right. Like query selectors, rules, properties. JULIE: Yeah. ELRICK: [Laughs] CHARLES: Again, might as well be groups and semigroups and monoids, right? JULIE: Right, right. ELRICK: Yeah. CHARLES: [Laughs] ELRICK: That is really interesting. [Chuckles] I've never heard anyone make that comparison before. But it's totally true, now that I'm thinking about it. JULIE: Yeah, yeah. CHARLES: Yeah. In the tech world we are so steeped in our own jargon that we could be… we can reject one set of jargon and be totally fine with another set. Or be like, suspicious of one set of concepts working together and be totally fine with these other designations which are somewhat arbitrary but they work. JULIE: Right. CHARLES: So, people use them. JULIE: So, it's like what you've gotten used to and what you're familiar with and that seems normal and natural to you. [Chuckles] So, the Haskell stuff, most of it seems normal and natural to me. And then I don't understand HTML and CSS. So, I bought a book. [Laughter] CHARLES: Learning HTML and CSS from first principles. JULIE: Yes, yeah. I just wanted to understand. I could tell that they do relate to each other, that there is some way that they click together. I can tell that by banging my head against them repeatedly. But I didn't really understand how, and so yeah. So, i've been reading this book to [Laughs] [learn] HTML and CSS and how they relate together. That's so important, just figuring out how things relate to each other, you know? CHARLES: Yeah. ELRICK: Yeah. That is very true. JULIE: Yeah. ELRICK: We can trade. I can teach you HTML and CSS and you can teach me Haskell. JULIE: Absolutely. ELRICK: [Laughs] CHARLES: There you go JULIE: [Laughs] ELRICK: Because I'm like, “Ooh.” I'm like, “Oh, CSS. Great. No problem.” [Laughter] ELRICK: Haskell, I'm like “Oh, I don't know.” JULIE: Yeah. CHARLES: Yeah. ELRICK: [Laughs] CHARLES: No, it's amazing [inaudible] CSS. ELRICK: Yeah. CHARLES: It is, it's a complicated system. And it's actually, it's in many ways, it's actually a pretty… it's a pretty functional system, CSS is at least. The DOM APIs are very much imperative and about mutable state. But CSS is basically yeah, completely declarative. JULIE: Right. CHARLES: Completely immutable. And yeah, the workings of the interpreter are a mystery. [Laughs] ELRICK: Yup. JULIE: YEs. And you know, for the Joy of Haskell website we use Bootstrap. And so, there was just like… there's all this magic, you know? [Laughs] ELRICK: Oh, yeah. CHARLES: Yeah. JULIE: Oh look, if I just change this little thing, suddenly it's perfectly responsive and mobile. Cool. [Laughter] JULIE: I don't know how it's doing this, but this is great. [Laughs] CHARLES: Yeah. Oh, yeah. It's an infinite space. And yeah, people forget what is so easy and intuitive is not and that there's actually a lot of learning that happened there that they're just taking for granted. JULIE: I think so many people start from HTML and CSS. That's one of their first introductions to programming, or JavaScript or some combination of all three of those. And so, to them the idea that you would be learning Haskell first and then coming around and being like “Oaky, I have to figure out HTML,” that [seems very] strange, right? [Laughter] CHARLES: Yeah. Well, definitely probably stepping into bizarro world. JULIE: And I went backwards. But [Laughs] CHARLES: Yeah. JULIE: Not that it's backwards in terms of… just backwards in terms of the normal way, progression of [inaudible] CHARLES: Yeah. It's definitely the back door. Like coming in through the catering kitchen or something. JULIE: Yes. CHARLES: Instead of the front door. Because you know the browser, you can just open up the Dev Tools and there you are. JULIE: Exactly, yeah. CHARLES: The level of accessibility is pretty astounding. And so, I think t's why it's one of the most popular avenues. JULIE: Oh, definitely. Yeah. ELRICK: It's the back door probably for web development but not the back door for programming in general. JULIE: Mm, yeah. Yeah. CHARLES: Yeah. It seems like Haskell programming has really started taking off and that the ecosystem is starting to get some of the trappings of a really less fricative developer experience in terms of the package management and a command line experience and being able to not make all of the tiny little decisions that need to be made before you're actually writing ‘hello world'. JULIE: Right. ELRICK: Interesting. Haskell has a package manager now? CHARLES: Oh, it has for a while. ELRICK: Oh, really? What is it called? I have no idea? Do you know the name off the top of your head? CHARLES: So, I actually, I'm not that familiar with the ecosystem other than every time I try it out. So I definitely will defer this question to you, Julie. JULIE: This is going to be a dumb question, I guess. What do we mean by package manager? CHARLES: So, in JavaScript, we have npm. The concept of these packages. It's code that you can download, a module that you can import, basically import symbols from. And Ruby has RubyGems. And Python has pip. JULIE: Okay, okay. CHARLES: Emacs has Emacs Packages. And usually, there's some repository and people could publish to them and you can specify dependencies. JULIE: Right, yeah. Okay, so we have a few things. Hackage is sort of the main package repository. And then we have another one called Stackage and the packages that are in Stackage are all guaranteed to work with each other. CHARLES: Mm, okay. JULIE: So, on Hackage, some of the packages that are on Hackage are not really maintained or they only work with some old versions of dependencies and stuff like that, so the people who made Stackage were like “well, if we had this set of packages that were all guaranteed to work together, the dependencies were all kept updated and they all can be made to work together, then that would be really convenient.” And then we have Cabal and we have Stack are the main… and a lot of people use Nix for the same purpose that you would use Cabal or Stack for building projects and importing dependencies and all of that. CHARLES: Right. So, Cabal and Stack would be roughly equivalent then to the way we use Yarn or JavaScript and Bundler in Ruby. You're solving the equation for, here's my root set of dependencies. Go out and solve for the set of packages that satisfy. Give me at least one solution and then download those packages and [you can] run them. JULIE: Yeah, yeah. Right, so managing your dependencies and building your project. Because Haskell's compiled, so you've got to build things. And so yeah, we have both of those. CHARLES: And now there's like web frameworks and REST frameworks. JULIE: Oh there are, yeah. We have… CHARLES: All kinds of stuff now. JULIE: We had this big proliferation of web frameworks lately. And I guess some of them are very good. I don't really do web development. But the people I know who do web development in Haskell say that some of these are very good. Yesod is supposed to be very good. Servant is sort of the new hotness. And I haven't used Servant at all though, so don't ask me questions about it. [Laughter] JULIE: But yeah, we have several big web frameworks now. There are still some probably big holes in the Haskell ecosystem in terms of what people want to see. So, that's one thing that people complain about Haskell for, is that we don't have some of the libraries they'd like to see. I'd like to see something… I would really like to see in Haskell something along the lines of like NLTK from Python. CHARLES: What is that? JULIE: Natural language toolkit. CHARLES: Oh, okay. JULIE: So yeah, Python has this… CHARLES: Yeah, Python's got all the nice science things. JULIE: They really do. And Haskell has some natural language processing libraries available but nothing along the lines of, nothing as big or easy to use and stuff as NLTK yet. So, I'd really like to see that hole get filled a little bit better. And you know… CHARLES: Well, there you go. If anyone out there is seeking fame and fortune in the Haskell community. JULIE: That's actually why I started learning Python, was just so that I could figure out NLTK well enough to start writing it in Haskell. [Laughter] JULIE: So, that's sort of my ambitious long-term project. We'll see how that goes. [Laughs] CHARLES: Nice. Before we wrap up, is there anything going on, coming up, that you want to give a shoutout to or mention or just anything exciting in general? JULIE: Yeah, so on March 30th I'm going to be giving a talk at lambda-squared which is going to be in Knoxville and is a new conference. I think it's just a single-day conference and I'm going to be giving a talk about functors. So, I'm going to try to get through all the exciting varieties of functors in a 50-minute talk. CHARLES: Ooh. JULIE: So, we'll see how that goes. Yeah. And I am still working with Chris Martin on ‘The Joy of Haskell' which should be finished this year, sometime. I'm not going to… [Laughter] JULIE: Give any more specific deadline than that. And in the process of writing Joy of Haskell, I was telling him about some things that, some things that I think are really difficult. Like in my experience, teaching Haskell some places where I find people have the biggest stumbling blocks. And I said, “What if we could do a beginner video course where instead of throwing all of these things at people at once, we separated them out?” And so, you can just worry about this set of stumbling blocks at one time and then later we can talk about this set of stumbling blocks. And so, we're doing… we're going to start a video course, a beginner Haskell video course. I think we'll be starting later this month. So, I'm pretty excited… CHARLES: Nice. JULIE: About that. Yeah. CHARLES: Yeah, I know a lot of people learn really, really well from videos. There's just some… JULIE: Yeah. [Inaudible] for me, so I'm a little nervous. But [Laughs] CHARLES: Yeah, especially if you can do… are you going to be doing live coding examples? Building out things with folks? JULIE: Yeah. CHARLES: Yeah. Well, you just needn't look no further than the popular things like RailsCasts and some of the… yeah, there's just so many good video content out there. Yeah, we'll definitely be looking for the. JULIE: Cool. CHARLIE: Alright. Well, thank you so much, Julie, for coming on. JULIE: Well, thank you for having me on. Sorry I went down some… I went kind of down some rabbit holes. Sorry about that. [Laughs] CHARLES: You know what? You go down the rabbit holes, we spend time walking around the rabbit holes. JULIE: [Laughs] CHARLES: There's something for everybody. So… [Laughter] CHARLES: And ultimately we're strolling through the meadow. So, it's all good. JULIE: [Laughs] Yeah. CHARLES: Thank you too, Elrick. JULIE: It was nice talking to you guys again. CHARLES: Yeah. ELRICK: Yeah, thank you. CHARLES: If folks want to follow up with you or reach out to you, what's the best way to get in contact with you? JULIE: I'm @argumatronic on Twitter and my blog is argumatronic.com which has an email address and some other contact information for me. So, I'd love to hear questions, comments. [Laughs] Yeah. I always [inaudible]. CHARLES: Alright, fantastic. JULIE: To talk to new people. CHARLES: Alright. And if you want to get in touch with us, we are @TheFrontside on Twitter. Or you can just drop us an email at contact@frontside.io. Thanks everybody for listening. And we will see you all later.

Sam Cates: @SamCates | GE Ventures Show Notes: 02:01 - What Corporate Investing Looks Like 03:48 - Presenting Ideas For Funding 09:01 - Democratizing Venture Capital 10:17 - ICOs and Cryptocurrency 13:53 - Evaluating Companies to Fund 21:09 - Investing in Potential Competitors 24:42 - Looking For Funding as a Company 28:04 - “Mentoring” Ideas/Companies 30:07 - Monitoring/Evaluating Company Metrics 32:47 - Putting Together a Basic Business Plan 36:05 - Making Choices: Investor and Company-wise Resources: Resin.io Series A, B, C Funding Angel Investor Seed Money Initial Coin Offering (ICO) AngelList Crunchbase Fred Wilson's Blog: (AVC.com) Transcript: CHARLES: Hello everybody and welcome to The Frontside Podcast, Episode 92. My name is Charles Lowell, a developer here at The Frontside and I am your podcast host-in-training kicking it off in 2018. [Inaudible] of our first episode. We've got Elrick also joining us. Hello, Elrick. ERICK: Hey. How you doing, Charles? CHARLES: I'm doing well. I'm doing well. You having a good new year so far? ERICK: Yeah, it's great. There's a snowstorm passing through today. So, I'm going to break in the New Year shoveling. CHARLES: Let us know if we need to parachute in some shovels for you. ERICK: [Laughs] CHARLES: And then with us today, we have Sam Cates on the show who is… a lot of times we have developers on the show. He's actually a venture… what would you describe yourself as? SAM: Yeah, I'd say I'm a venture investor with GE Ventures. So, on the corporate investing side. CHARLES: Okay. Now, I didn't even know that GE actually had a corporate investing side. Is that pretty common for a large company? SAM: You know, it's becoming increasingly common. I think in 2015 there was actually a peak of activity coming from corporate venture capital groups. And I've only seen the number of firms escalate since then. Although the dollars invested stays pretty consistent. But if you look at a lot of big companies, particularly in the common tech world like Cisco, Google, Intel, they have historically had large venture firms inside of themselves. And then GE and a lot of other industrials have since followed suit. We've been at it for about five years and we see it increasingly. CHARLES: And so, have you been with them since the beginning? SAM: Yeah, just about. I've actually been with GE for about nine years now. So, I was on the operating side in a number of the industrial businesses before I joined GE Digital and then GE Ventures. And so, it was just after GE Ventures got kicked off. CHARLES: Oh, that's exciting. So, what is it… now, we actually got connected to you through one of the companies that you actually invested in. It's something that we use and we're very interested in. Why don't you tell us a little bit about what your job looks like on a day-to-day basis and what companies you invest in? SAM: Sure. I really focus a lot of my time on Internet of Things companies. So, that's a really big trend that GE has been a part of and a leader in over the past few years. And so, we spend time investing in companies that are directly working with GE or playing in similar spaces to us. And so, Elrick and I actually met at a hackathon for one of those companies. And I always like to use that as an example because it's a good one, to demonstrate the kinds of investments we make. And that's Resin.io. I know you guys have done an episode or two talking with them. But that for example was a ‘Series A' investment that we made about two years ago. And then company essentially helps developers build connected products. And so, that's something that GE cares a lot about. We had people inside the company who found the product and loved it and that's actually how we met. CHARLES: When you say ‘Series A', can you give a brief overview of what the different stages of funding of a startup might be? SAM: Yeah, yeah, certainly. So, maybe if I take a step back and answer your original question on what I do on a day-to-day basis. A lot of my job is meeting with all kinds of new companies, whether they be early stage, usually things that would be seed funding – and we'll go into what some of those things mean – all the way through the late stage which would be companies that are maybe on the border of going public or are already profitable. And so, if we go into what kinds of investors there are, I think that's probably an interesting subject to talk more about. But they're a whole wide variety. When I said ‘Series A' I just meant a company that was at what we would call the ‘Series A' stage, and the letters act just like you'd expect. So, there's ‘Series A', ‘Series B', ‘Series C', and so on. And they all, they tend to look similar at those stages in terms of sizes and progress. But there is a range, and no two company is the same. ERICK: In today's world, it's very easy for people to create a startup. They can write some code and they can either come up, get a Raspberry Pi or some microcontrollers or whatever it is, and either do an IoT startup or a software startup. Now, when you get to the point where you have an idea and you kick it off initially, how do you go about then saying, “Let me get some funding.” How do you even get funding? SAM: Sure, yeah. And to your point, there's a huge range of technologies that are making it easier to start almost any kind of company. It's a great time to be an entrepreneur, whether it be 3D printing for hardware products, all the technologies that you were mentioning, AWS, all this stuff is contributing to reducing the cost to allow companies or people to create companies. And so, once people have gone out and experimented with some of these things and built what they think is a product the market wants, often if they require more money which may be for acquiring customers through things like Facebook Ads or simply doing further product development to make sure the product is somewhere that more customers could use it, often they can't finance it just through their own revenue. And so, there are typical stages and types of investors that people go approach looking for money. ERICK: Okay. What are those Series? I remember you mentioned something like a ‘Series A' investment. So, initially when you're looking for an investment, is that where you would… category you would be in as a startup looking for investment? They would consider you a ‘Series A' startup? SAM: Well, I want to caveat and just say every company is different. So, I see companies that… ERICK: Gotcha. SAM: Start out at a much later stage because they're able to bootstrap to that point. And bootstrap is the word that I use for a company that funds its own investment. They get paid by customers and they use that money to continue building the product. But if I talk about the range of types of financing a company may go for, I think the way that most people categorize this, first people often raise from friends and family or angels. And so, it's just money to get off the ground and maybe to pay the rent while you're doing some of that experimenting we were talking about. And then commonly after that is a seed round. And a seed round tends to be a little more institutional. So, it's maybe a more formal set of funds who exclusively invest in companies that are often pre-revenue but they have a product, or at least the beginnings of a product. And so, that's a really common category of investors. And then you get to ‘Series A' and the letters can escalate from there to the point where… ERICK: Gotcha. SAM: There can be some later rounds when they'd be ‘Series F' or even beyond, I guess. CHARLES: Right. So now, what are generally the terms on these? So, for my angel investments or my seed investments, I assume what distinguishes these is essentially how much ownership of the company you're getting for how much money. And those kind of, those change as the product solidifies. SAM: Yeah. CHARLES: And the potential becomes more visible. SAM: Yeah, it's a wide… and again, these are all… the venture is a world of ranges. There's a really wide difference between the two ends of any spectrum. So, I'll just talk in generalities though. So, I think the latest report that I've seen at least for an annual basis was PitchBook's 2016 report. And they were laying out some of the medians. So, for seed stage deals I believe it was something like one and a half million dollars raised was the median on a pre-money valuation of six and a half million. And that just means the company is worth, investors say the company is worth six and a half million dollars today. And we're going to give you a million and a half dollars invested at that price. CHARLES: So roughly, a sixth… they would take a sixth of the company then in return? SAM: Yeah. CHARLES: Okay. ERICK: Ah. CHARLES: Okay. ERICK: Okay. CHARLES: I see. That makes sense. So now, back to Elrick's original question. If I'm, I've got my product. Or I've got this idea. I've written some code. I've turned it into a prototype product. Maybe I'm moving through these various stages. What type of VC am I going to be looking for? How do I actually find the right type to be talking to? I guess what types are there even? SAM: Yeah. And one part… we mentioned a lot of the technologies that are making it easier to start companies. One part that also makes it easier is the proliferation of financing options, whether it be even more investors in these traditional structures we talked about like seed and A. And then there are other options that are emerging, things like you see a lot of people raising through what they call Initial Coin Offerings or ICOs. And then there are also things like AngelList which are attempting to democratize the investing process, make it more accessible. So traditionally, a lot of the seed A, B investors, they tend to be network-based, which can be a challenge for a lot of people that are maybe not in Silicon Valley or not a part of that network already. And so, one thing you can do is obviously go search databases that are on the web, things like Crunchbase. It's a free resource. It has a lot of deal history for investments that people have made. And it's a great resource for knowing, “Okay, this investor cares about these things.” And then in addition to that, there are also platforms that people can put their companies on. Like I mentioned, AngelList. And that's somewhere that you can list your company, you can meet investors, and they actually have some backend to actually support the investing process as well. CHARLES: So, there were two acronyms in there, or two specific technologies. [Chuckles] CHARLES: You talked about ICOs which I assumed that you said it was Initial Coin Offering. Not like insane clown offering. [Laughter] CHARLES: Which I would love to see. And then AngelList. So traditionally, these had been very network-based which brings to mind the capitalists of Old England or whatever where there's a bunch of people with cigars in a room and I realize it's not actually like that. What are each of these things? The AngelList and the ICOs? And how do they democratize that process? SAM: It's funny you should mention the old times. I think a good example of that is there are a lot of stories about the founding of General Electric. It's a 126-year-old company and back then it was largely, it was Thomas Edison working with I believe was JP Morgan to get it off the ground. And so, today there's still a bit of the network piece you're mentioning. But I think of AngelList as a place that you can essentially market to investors. If you think about the types of people that are on there, it's people that are looking to invest money in early stages in startups. And I'm not a big user of AngelList because I tend to be investing a little bit later. So, I really recommend anybody who's interested, just go check it out. It's I believe just Angel.co. CHARLES: And what about an ICO? SAM: So, an ICO is a more modern one. And it's kind of fraught with some concerns around regulations and transparency today. But I think since Thanksgiving there's been a massive wave of conversation about cryptocurrencies. And an ICO is essentially a way of creating your own cryptocurrency. The way I always explain to people, I love the analogy that people make around, think of it like I want to go build an amusement park. And in that amusement park, everything, rides, food, everything, is going to be denominated and payable in Sam-bucks. CHARLES: Ah, right. SAM: And… [Chuckles] And so, my options… CHARLES: [Laughs] That makes sense. SAM: Yeah. And my options are I can go to a bank and borrow money, I can go to investors and say, “Hey, give me the 10 million dollars it's going to take to build it,” or I can just go to the people in the place where I'm building it and say, “You want this amusement park to exist? Why don't you pre-buy these Sam-bucks?” And each one is going to cost a dollar today. And we create this universe of Sam-bucks and they're essentially valuable once you can use them in the park. And there are certainly exceptions. There are other versions of cryptocurrencies and other uses for them. But that's a conversation for another day. CHARLES: Ah, mm. SAM: I think that's just a good, easy way to understand it. CHARLES: Oh no, I like that. It's like, well not quite like carnival tickets. But yeah, that's something that everyone's familiar with. Same thing as the Xbox Marketplace. Very similar thing. So, the idea is you would buy a bunch of Sam-bucks… you would get them at pennies on the dollar, so to speak, today. SAM: Yeah, right. By the time it opens, maybe a hotdog would cost just one Sam-buck. CHARLES: Right. SAM: Whereas, when it's coming in, we'd have to spend five dollars to get that one Sam-buck. Right, the idea being those people who got in early will be rewarded. And you can see it's like a further extension of a Kickstarter or something else that you're allowing people to pre-buy into a network. CHARLES: Right. Right, okay. I can see that. ERICK: That's very interesting. [Laughs] CHARLES: And so, it's got a range of options too, because if you're really interested in the services you can go ahead and spend them on the services and get a lot of value that way or you can actually trade for someone who does want the services if you don't. SAM: I think that's exactly right. And it's just, the one that I think I would just caveat is there is a huge amount of concern at the moment, and maybe concern is too strong a word, but uncertainty around one, what are the value of these coins, these tokens? And two, how will governments react to something that looks potentially like a security or a currency? And so, that's something that still is being worked through. And even though they haven't figured that out there's still a massive amount of money being raised through these ICOs. CHARLES: [Laughs] So, it does beg the question. Why is a cryptocurrency necessary? Why not just use Xbox Marketplace points? Why not just say, “Here are Sam-bucks.” SAM: [Chuckles] CHARLES: And there's a row in my database. [Laughter] CHARLES: That's your balance of Sam-bucks. SAM: So, I think we're about to get way beyond the [inaudible] [Laughter] SAM: But I think the argument would be that some of these things are better decentralized. So in my example, you're right. That might just make more sense. But I think there are some examples around cryptocurrencies that are supporting a network of decentralized services where a centralized database historically was inconvenient or didn't provide the amount of transparency that people were looking for. CHARLES: Right, right. SAM: And so, that's a topic for a whole other podcast. CHARLES: Yeah, right. No, it makes sense. SAM: [Laughs] CHARLES: I think it's a matter of scale, right? If you're going to be just buying services but if you're going to have secondary markets where you're trading in this currency, I can see that. So, let's… [Chuckles] We'll reel that back in. SAM: [Chuckles] CHARLES: And ask a question that occurred to me. So now, we talked about your day-to-day. What exactly, when you're looking at a company to basically give money to, what are you looking for? What are the things you're like, “Oh man, I want to throw dollars at this company,” versus, “Mm. I'm going to keep them and give them some feedback and send them on their way.” SAM: There's always a set of factors that we evaluate. And I think the waiting is probably different for different types of investors. And then there's I'd say for me as a corporate VC being a part of GE, there's an extra lens which is, how is this relevant to GE? What does it mean for GE to be an investor? But if I think about just the kind of general industry lines it's: team is a really big one. So, who's building this company? Do I believe in their ability to reach this vision that they're laying out for me? Another one would be technology. What have they actually built? Is that hard to build? Do the things they want to build in the future, will those be hard to build? And do they have the skills and the people to do it? Then their technology, maybe an extension of that would be intellectual property. And besides intellectual property, just defensibility of a business in general. So then, you start thinking about, can somebody else just come along and to the same thing? Because if so, then maybe there's not a strong advantage in what the company has done so far. And then lastly, it's also just traction. How far along are they? How much have they proven the ability to execute on the plan that they're laying out? CHARLES: Right. ERICK: So, you're a corporate investor. So, there's other types of investor like an institutional VC? What are the differences between an institutional VC and a corporate VC and the other types of VC? Potentially what they'd be looking for, in terms of what they wanted best. SAM: Yeah. So, I think generally I categorize investors as institutional or corporate. And corporate [inaudible]… ERICK: Yeah. SAM: Corporate or strategic. And then there are people who exist on a spectrum there. But generally, an institutional means this is a group that is raising money from a set of limited partners who are the people who invest in the fund that are pension funds or wealthy individuals. They're large pools of institutional capital and their pure purpose is to earn return. And they may have a certain focus because they believe in this part of the market, or they like this kind of company or the stage of company. But essentially, their job is to return more money to the limited partners of that fund that were put in. That's their role in the world. And then on the corporate side, if we go the most extreme version of corporate VC, this is a group that is a part of a larger corporate. They're investing that company's money. So, in this case for me it's GE. I'm investing GE's money into these startups. And that means that I only have a single backer being GE. And I also maybe have a different lens, because my purpose is one, to earn financial return. I want to go out and I want to find good companies. I want to earn returns just like the other institutional venture capitalists. But I also have the goal of, and the strategic goal may differ by company, but for me it's about how can I help GE advance? How can I help GE understand a market? And how can GE be helpful to this company in achieving their goals? And so, for each company we use that lens as well, as a corporate. CHARLES: What I'm hearing is that you want to invest… I guess the thing is you can experience return that's not just cash. It's not just dollars. You'll experience return in raising the ocean of the business that GE is in, right? So… SAM: You said it much better than I did. [Laughter] CHARLES: Well, it's all… paraphrasing is actually easy. [Laughter] ERICK: Oh, yeah. SAM: An important skill. CHARLES: That makes a lot of sense. So, the question I have then is, you said you were looking for companies that kind of swim in a specific ocean. And each company is farther along. Are you usually finding this company I want to work with, like you are going out and finding them? Or they're coming to you looking for investment? Or is it really just, depends. SAM: So, we call that part of the process sourcing, sourcing investments. And they come from all over. So for us, there are a few different ways. One is we tend to be thesis-driven. Meaning we go out and we say, the world is changing in this way and therefore we're interested in this kind of company. And so, we'll proactively go out and research. We're also, I mentioned, a little later stage. So, I don't tend to do seed investments. I tend to do ‘Series A' and more often ‘Series B' and later. So, companies that have often already raised a seed round or raised a ‘Series A' round. So, I can actually search databases to say, “Okay, in the last two years who has raised a seed round or ‘Series A' round and these other things I'm looking for whether it be location or tied to investors or other things.” So, that's one way of being proactive is saying I want to go out and look for companies in this space that look like this. And that can be either like I mentioned, desktop research like searching the web, searching databases. Or it can be just going to conferences, right? So, on thing we spend a lot of time on in the IoT world is artificial intelligence and machine learning. It's been a big, big topic over the last year that a lot of people have invested in. So, we may go to different conferences that focus on that topic, meet lots of people that are working on it. Some companies, some individuals that are either investing in or advising their companies. And we'll talk to them. What companies are rising out of that space that we should be looking at? What technologies are changing in that space that we should be thinking about? And just trying to get smarter so that we can make the right investments and help the right companies find their way to work with GE and make our products better and help them advance their own enterprise. CHARLES: Are you investing with a mind that eventually GE might acquire this company and integrate it into GE itself? Or is it really just, “Hey, we're just going to take a part of it. We're going to have maybe a seat on the board to be able to steer a little bit. But we're pretty much going to let it be its own thing with its own autonomy and go where it was and just benefit through those secondary and tertiary effects.” SAM: Yeah, acquisitions from our portfolio by GE happened. But they're certainly not the explicit goal or our focus. I know we've had one, maybe two of our portfolio companies acquired by GE, one that I was directly working with called Bit Stew. So, we made the investment in the company. It was with the goal of using their data management platform for a lot of our applications. And at some point in working with GE and GE Digital, they decided, you know, this would make sense to be a part of GE. That wasn't why we made the investment. But it did end up being acquired by GE. And I know the team is doing really well. And it's been at GE for about a year now. So, it does happen. But when I said one or two, that's versus a portfolio of a hundred plus companies. CHARLES: Right. SAM: Since we started investing. And so, that's not what we're looking to do every time. Much more often it's about again, how does the company make GE more competitive and a better company, a better place to work. And then how do we help them advance their goals? Whether it be bringing them developers, or finding them other routes to market, or just being a customer. CHARLES: Right. SAM: So, that's really how we think about strategic value. There's a lot of different ways to create it. CHARLES: Yeah, I'm curious. Because it seems like also in a lot of these companies you're investing in potential competitors. Extensively you're operating if not in the exact same market, maybe very similar markets. There's a little bit of overlap. And so, you're kind of investing in potential competitors, right? So, where's the balance of here we're funding our competitors versus we're going to move into these markets ourselves. SAM: Yeah, and funding of “competitors” can happen. I think that we talk about that more in theory and say, “Oh sure, we'd be willing to fund a company that's out disrupting the space that we're playing in.” And we do that. It's rare that you see startups that are directly head-on competing with much more established companies like GE or other industrials or even other consumer companies. They don't take these companies head-on because that's not a way that startups have been successful in the past, right? We talk much more about disruption and saying, how is this company doing something that may indirectly compete with GE? So, you think about things like, for anybody that's not familiar with GE… actually, a lot of people associate us with our appliances which we actually don't manufacture anymore. That's [inaudible]. [Chuckles] SAM: We sold that business a few years ago. Almost everything we sell is like big, heavy industrial equipment. So, we sell aircraft engines, locomotives. We sell gas turbines, wind turbines. So, here and there a couple of things that do power generation. One trend that's affecting that industry is distributed generation of energy, energy storage. And those are parts of the market that are a less significant part of GE's business than say, heavy-duty gas turbines that sit in a power plant and generate a massive amount of power. And so, if you look at that and say, “Wow, GE Ventures is out funding storage companies. Does that mean they're funding competitors?” Well, it means that we're funding innovation that may disrupt the future of our business, but that's part of being a VC and that's part of the value that GE Ventures brings to GE. CHARLES: Right. SAM: We're out there looking at markets before they're large enough or in scope for GE. CHARLES: Mmhmm, right. And so, yes you're disrupting the space but then you're going to be a part of that disruption and have strong connections to those markets if you need to actually migrate your business completely over to them. That's kind of what I'm hearing. SAM: Yeah, absolutely. Better to disrupt yourself, right? CHARLES: [Chuckles] SAM: And be a part of the ecosystem in the future because I think the future happens with or without you. And it's really key that we get out in front of it and a part of that, a part of that discussion, a part of that process. CHARLES: And so now, you've been saying that this is, GE, this has been pretty explosive? There's a lot more happening through GE Ventures. There's a lot more happening in other companies globally, having these corporate ventures. Where do you think the balance is going to lie to say, “Hey,” I'm just going to throw out some numbers, just for theory here, it's like, “10% of our business is essentially this distributed network of semi-autonomous or mostly autonomous startups. And then we have our core business.” Does that stabilize at 50/50? Does it stabilize at 75% the other way with GE essentially becoming a capital management company? Or is it somewhere in the middle? SAM: So, GE Ventures will never be a meaningful part of GE's revenue, a meaningful part of its business as a percentage. The overall venture industry is full of funds that are on the order of like, bigger funds are on the order of, in the billions. The single-digit billions. And GE itself is a much, much larger company. Well over a hundred billion dollars in enterprise value. So, I think GE Ventures will always be a small part of the company financially. And the impact will be largely felt through how we help the rest of GE navigate the future. ERICK: You said that sometimes you go and look for companies, startups to invest in or sometimes startups come to you or come to a VC looking for funding. Now, I'm a developer or a startup founder. And I'm going to look for funding. What are some of the mistakes or pitfalls that you see that startup founders or people with an idea fall into when looking for funding that you can help them avoid? SAM: Yeah, and we do see companies that come to us. So, I mentioned a lot about how I go out looking for companies based on a thesis or a set of relevant factors or relevant things for GE. But we do have a number of inbound requests. People know some of the bigger VC brands. They know GE the big company. So, we do get inbound interest and we also get referrals from networks of VCs and some are employees and other things. But for the companies that are seeking us out, the ones that are going out looking for funding, there are some things that are really well-known in Silicon Valley and other places, or you could research online and find, but may not be obvious at first. And so, I think the first one is, who are you talking to? What investors are you seeking out? Depending on what stage you're at, what kind of business you're in, you have to understand what the landscape of potential investors are and which ones might be interested in a company like yours. So, I think there are tons of good mentors that can help people navigate that. Maybe less commonly outside of Silicon Valley, in Boston, New York, in the places where you have traditional venture ecosystems. But you see a ton of resources available online whether it be things like Fred Wilson's blog, AVC.com, or Crunchbase, TechCrunch. You can read and understand and from headlines tell what people care about. And I think that's fundamentally a really important first step. You don't want to waste an hour talking to somebody who will never… this is somebody that invests in really late-stage growth equity companies and I'm coming to them for my first investment. That's not going to work. So, I think finding the right people, step one. I think when you're going through the process of pitching and talking about your business, the pitfalls are all about understanding the strengths and weaknesses of your business and where you are today. And so, for every company, that's different. But I think just being open and honest versus glossing over a lot of the risks, these are all really risky companies. If they were easy, then you'd have a lot more competition. And so… [Laughter] SAM: I think that's one thing that I see, too. You have some company that comes in and say, “Look, here are the parts I've figured out and here are the parts I still have to figure out.” And that's a really good conversation to have. There are other companies where they say, “Look, we've figured the whole thing out. We just want you to give us some money.” And I don't think a lot of investors necessarily buy into that. And certainly, there are investors of every stripe. So, I may be speaking too broadly. But I think that's a really important part of the venture investment process, right? You're looking not just for money but also for counsel and for somebody that you're going to work with over the next, sometimes seven years or longer. CHARLES: Yeah. SAM: [Inaudible] going to be on your board and participating. So, it's a really important part. CHARLES: So, you're looking, you're actually looking not necessarily for all the answers but you're looking for the questions that they're asking, too. SAM: Yeah, absolutely. And demonstrating they understand the ins and outs of the business. And that they have the capacity to carry this onto that next stage and hopefully beyond. CHARLES: Mmhmm. So, now you said something that caught my interest there that you work with some people sometimes seven years. You enter into these long relationships. Do you generally ever do any type of, I want to say almost like… mentoring might be too strong of a word, but in the pre-investment, in other words before you actually invest in a company, do you ever work with them to prepare them for investment to say, “Hey, I think there's potential here. Work on A, B, and C and then let's talk.” And you have this image in your mind. You go, you pitch to an investor, and it's either thumbs up or it's like thumbs down and you never talk to them again. Versus, is there some ground in between where there's a conversation that evolves that eventually ends up in an investment being made? SAM: Absolutely. I think one of the parts of this industry is even when I'm not an investor in a company, I may know a company and say, “It's not a fit for me for GE Ventures but I still think that we can provide help.” It's one of the things I love about tech and about venture in general, is that people are often willing to pitch in, even when they don't have a direct financial incentive. And so, I see that a lot whether it's helping a company where we've met them and we later see an opportunity and say, “Oh, you should go and talk to this company or that company.” And then often, we may see a company that's pitching us ahead of where we would typically invest. Maybe they're looking for a ‘Series A' but given the space that they're in or what we're doing at the moment, it may not be the right time for us. And so, we'll continue to track along and keep up and get updates. Some companies do a really good job of actually providing proactive updates and sending out monthly or quarterly reports to investors they've met with before. I think there's a wide range of ways that founders do this. But it is a really good way to keep people interested in the prize. And then when you come back and say, “Hey, now I'm out raising my ‘Series B',” that's not a surprise. I knew that you were hitting these milestones, that you were doing everything you said you were going to do. And you've demonstrated a level of credibility that really adds to the pitch that you made the first time around. ERLICK: You said something, metrics. So, a venture capitalist, after they make an investment, what are some of the expectations that they may hold this startup that they just invested in… what are those expectations that they may hold them accountable for? Or those metrics that they'll be looking at? SAM: Yeah, so I think some of the really high-level ones that are common across businesses, generally growth is a really big one. So, I almost said revenue. But I wanted to caveat… [Laughter] SAM: And say growth could mean different things. It could mean number of developers. It could mean number of downloads if you're an app. It depends on what the business is. But I think growth is a huge one. Growth is a really important, that top line, that's what's going to drive a lot of the value in the business. And then below that, demonstrating that you can hit the milestones around things like margins. So, how profitable is each unit you're selling? Or how profitable is each customer? And lastly, how are you doing managing your spend? So, that's great that you're earning the right amount of money for each customer, but are you doing it by… do you have a massive number of employees and offices and all the things that are too expensive to allow you to use your money wisely as you reach the next stage? And so, those are the big milestones. It's really just growth, margins, and operating cost or burn rate as we call it. CHARLES: Mmhmm. So, that sounds like a lot of work to actually evaluate these companies. Do you do your due diligence once you've already moved in pretty solidly into the process? SAM: Yeah, these processes can move really fast. And depending on the timing, generally it's, you jump in, you learn as much as you can, as fast as you can, and you make a decision so the company can move on. I'll say there's a lot of work that goes into considering and deciding which companies to spend more time on, both for us and for them. We don't want to waste a company's time evaluating, going through more meetings, if it's not a really strong candidate for us. Because they could be spending that time better with other investors who are a better fit. And I'm not going to pretend to like the evaluation part. I have a lot of respect for the amount of not just work but of a person's energy and really, their life goes into these companies. And so, I think the hard part is building the company. And so, it's hard for me to say that evaluating is a hard part. I'm trying to understand as much as I possibly can in a month or two. I'm not going to know as much about the business as the founder does. And I'll be wrong a lot. I may miss something and not understand, whether it's because I don't see the market but it's there or because I have some underlying assumption about the way things should work that they don't meet. And I think that that's something that investors have to come to grips with. You try and get as smart as you can as fast as you can, but you're not always going to get to the right answer. ERLICK: You said that it was growth, spend, and profits were some of the metrics. That is almost all of the essential components of a business plan. I remember one time, one of our previous conversations, you emphasized how important it was for companies, or even at just a simple startup, to put together a basic business plan. Is that something that you can elaborate on a little? SAM: Yeah. So, most companies show up with a pitch deck. So, they have a set of PowerPoint slides and then they have a set of materials behind that where if you go deeper into an area they may have a white paper about their technology and they may have an Excel financial model that explains why they have these expectations about what growth and margins and all those things will look like. So, there are all of those pieces that come together into a business plan. The business plan could be written or it could be that PowerPoint. But very traditionally, it's a PowerPoint or some kind of presentation that is shared in person. There's usually a version that's sent in advance to confirm that the company and the investors should meet. And then once you clear that bar, there's a deeper presentation that often you'll give to either one or a set or a whole team of investors. And you'll go through and explain why it is you think this is a good investment opportunity for them and why you'd like to work together. And then you have a discussion about whether that's a good fit, about some of the underlying assumptions, and come to either a set of next steps for the diligence or a decision that it's not the right fit, it's not the right time to take the relationship further with more diligence and that kind of stuff. ERLICK: Yeah, because I see… well, I know a few people that have startup ideas and they kind of put the business plan on the back burner and put the actual prototype more at the forefront. They say, “Oh, we can worry about the business plan later.” [Laughs] SAM: [Chuckles] Well, I think… there's something to be said to that. There's something to be said for product and growth winning. So if you… Let's start at the early stages. If you have something that's working and that's really obvious, you may not need a… ERLICK: True. SAM: To go raise money. It all comes down to, do you have enough to get enough investors interested to raise the round that you want to raise? Because you want to have enough investors involved, enough demand, that you can be selective about who you want to work with and on what terms, right? So, what valuation and how much of the company am I giving them, and all of those things. So, if you can do all of those things with nothing but an app and one chart that shows a hockey stick of growth, that's awesome. CHARLES: You're hot. [Laughter] SAM: Often it does require much more and a much longer plan. So, even if you say, “Look, it's growing like crazy,” there's usually some set of questions behind that. So, that's great. Your free app is growing like crazy. How are you going…? [Laughter] SAM: To get paid for that? And you'll talk about that. And you'll say, “Here are the things we're planning on doing,” or here are the assumptions that we're making. And the more original, the more unique the business model is, the more discussion and explanation that may require. And that's where the business plan and a pitch deck come in handy, because it's a really good presentation aide or pre-reading to get to that answer faster. ERLICK: So, this evaluation it seems, is a two-way street. The VCs evaluating the company and also the company or the startup evaluating the VC to know whether it's going to be a good relationship. SAM: Oh, absolutely. Yeah, the best companies have choice. They have a number of investors who are interested in funding them. And certainly, that might be different at different stages or at different times, depending on what's going on in the economy and in tech and in other places. But generally, VC is a very competitive industry. I'm trying to sell my money and services as an investor versus other options that you have. And so, while it's maybe not as competitive as only one of us can buy the company like in an M&A situation. There are often more than one investor. There's still a very intense set of competition around, okay, who's going to be involved in the deal? How much money will they be able to invest? So, that's something that really can come in handy for founders. ERLICK: And what was that you just said there? M&A situation? SAM: Oh, sorry. When a company is being bought. So, when a company is being bought, it can look kind of like a fundraising process, but instead of selling a part of a company, you're selling the whole thing. And so, in that case, obviously it's a competitive situation where there's only one winner. And this is a different process. Often, the rounds that we're a part of, we're not… we're buying a minority stake just like any VC. We may be buying 5% of a company, 10% of a company. And often we're being joined by other venture investors. We really actually commonly partner with the institutional firms and they'll take a board seat. We'll invest alongside them and be an observer on the board and provide counsel. And so, it is a very competitive process. And that, while M&A is a winner-take-all, there is one buyer who is ultimately going to own this company going forward, the investing process for a venture is much more collaborative. But it is still competitive, because there can only be so many investors in one company. CHARLES: And you want to choose the right one on both… the right set. Alright. Well, I think we're running up against time. This has been a fascinating conversation into an aspect of our industry that really is providing the fuel that drives so much of this forward. So, I guess I'll close by asking you, already talked about Resin. We had them on the podcast. We love them. Are there any conferences or products that you're investing in that you feel like our audience might want to know about or anything like that? SAM: Well one, you mentioned Resin. CHARLES: Yeah. SAM: I know you guys have been a good friend to them and Elrick and I met at their hackathon. I would recommend to anybody, go try it out. It's a really cool way to play with hardware products. I am not a developer and I required a lot of help from Elrick at the hackathon. [Laughter] SAM: But at the same time, it is something that almost anybody can pull out of a box and start playing with. So, I think that's a great one. The episode you did on them were fantastic. So, I really enjoyed those ones. I'd say in general, I'm always out looking to meet new companies that are going to benefit from working with GE. I spend a lot of my time not just trying to invest but also trying to find partnerships for companies that we're looking at within GE, either selling to us or working with us. And so, if somebody thinks that there's an opportunity to do that, then I encourage them to reach out. Because I think there's a ton of opportunity. It's a really big company that really has a ton of opportunity for other partners. CHARLES: Alright. If they wanted to reach out, how would they get in touch with you? SAM: Yeah, I think maybe the best way to initially make contact, I tend to be pretty active on Twitter. So, my handle is just @SamCates. S-A-M-C-A-T-E-S. And you can also learn more through our website. If you're curious about some of the businesses I mentioned, so just GEVentures.com. And it's about to go through a whole refresh. So, go check it out. CHARLES: Alright. Well, fantastic. We will definitely look for that. And for everybody else, you can get in touch with us on Twitter at @TheFrontside or send us a line at info@frontside.io. Thank you everybody for listening. Thank so, so much Sam, for being on the podcast. SAM: Yeah, of course. It was a blast. I'm a big podcast fan and I've really enjoyed catching up on your episodes. CHARLES: Ah, and thank you Elrick, always. ERICK: It was great. SAM: Elrick, when you finish building your Raspberry Pi Battleships, I want to play. CHARLES: [Laughs] ERICK: Oh, yes. Yes. It's in the works, man. It's in progress. SAM: Alright, I'm waiting. CHARLES: Alright. Well, take it easy, everybody.

Tracy Lee: @ladyleet | ladyleet.com Ben Lesh: @benlesh | medium.com/@benlesh Show Notes: 00:50 - What is This Dot? 03:26 - The RxJS 5.5.4 Release and Characterizing RxJS 05:14 - Observable 07:06 - Operators 09:52 - Learning RxJS 11:10 - Making RxJS Functional Programming Friendly 12:52 - Lettable Operators 15:14 - Pipeline Operators 21:33 - The Concept of Mappable 23:58 - Struggles While Learning RxJS 33:09 - Documentation 36:52 - Surprising Uses of Observables 40:27 - Weird Uses of RxJS 45:25 - Announcements: WHATWG to Include Observables and RxJS 6 Resources: this.media RxJS RX Workshop Ben Lesh: Hot vs Cold Observables learnrxjs.io RxMarbles Jewelbots Transcript: CHARLES: Hello everybody and welcome to The Frontside Podcast, Episode 91. My name is Charles Lowell, a developer here at The Frontside and your podcast host-in-training. Joining me today on the podcast is Elrick Ryan. Hello, Elrick. ELRICK: Hey, what's up? CHARLES: Not much. How are you doing? ELRICK: I'm great. Very excited to have these two folks on the podcast today. I feel like I know them… CHARLES: [Laughs] ELRICK: Very well, from Twitter. CHARLES: I feel like I know them well from Twitter, too. ELRICK: [Laughs] CHARLES: But I also feel like this is a fantastic company that is doing a lot of great stuff. ELRICK: Yup. CHARLES: Also not in Twitter. It should be pointed out. We have with us Tracy Lee and Ben Lesh from This Dot company. TRACY: Hey. CHARLES: So first of all, why don't we start, for those who don't know, what exactly is This Dot? What is it that you all do and what are you hoping to accomplish? TRACY: This Dot was created about a year ago. And it was founded by myself and Taras who work on it full-time. And we have amazing people like Ben, who's also one of our co-founders, and really amazing mentors. A lot of our friends, when they refer to what we actually do, they like to call it celebrity consulting. [Laughter] TRACY: Which I think is hilarious. But it's basically core contributors of different frameworks and libraries who work with us and lend their time to mentor and consult with different companies. So, I think the beautiful part about what we're trying to do is bring together the web. And we sort of do that as well not only through consulting and trying to help people succeed, but also through This Dot Media where it's basically a big playground of JavaScripting all the things. Ben and I do Modern Web podcast together. We do RX Workshop which is RxJS training together. And Ben also has a full-time job at Google. CHARLES: What do they got you doing over there at Google? BEN: Well, I work on a project called Alkali which is an internal platform as a service built on top of Angular. That's my day job. CHARLES: So, you've been actually involved in all the major front-end frameworks, right, at some point? BEN: Yeah, yes. I got my start with Angular 1 or AngularJS now, when I was working as a web developer in Pittsburgh, Pennsylvania at a company called Aesynt which was formerly McKesson Automation. And then I was noticed by Netflix who was starting to do some Angular 1 work and they hired me to come help them. And then they decided to do Ember which is fine. And I worked on a large Ember app there. Then I worked on a couple of large React apps at Netflix. And now I'm at Google building Angular apps. CHARLES: Alright. BEN: Which is Angular 5 now, I believe. CHARLES: So, you've come the full circle. BEN: Yeah. Yeah, definitely. CHARLES: [Chuckles] I have to imagine Angular's changed a lot since you were working on it the first time. BEN: Yeah. It was completely rewritten. TRACY: I feel like Angular's the new Ember. CHARLES: Angular is the new Ember? TRACY: [Laughs] BEN: You think? TRACY: Angular is the new Ember and Vue is the new AngularJS, is basically. [Laughs] CHARLES: Okay. [Laughter] CHARLES: What's the new React then? BEN: Preact would be the React. CHARLES: Preact? Okay, or is Glimmer… BEN: [Laughs] I'm just… CHARLES: Is Glimmer the new React? BEN: Oh, sure. [Laughs] CHARLES: It's important to keep these things straight in your head. BEN: Yeah, yeah. CHARLES: Saves on confusion. TRACY: Which came first? [Chuckles] BEN: Too late. I'm already confused. CHARLES: So now, before the show you were saying that you had just, literally just released RxJS, was it 5.5.4? BEN: That's right. That's right. The patch release, yeah. CHARLES: Okay. Am I also correct in understanding that RxJS has kind of come to very front and center position in Angular? Like they've built large portions of framework around it? BEN: Yeah, it's the only dependency for Angular. It is being used in a lot of official space for Angular. For example, Angular Material's Data Table uses observables which are coming from RxJS. They've got reactive forms. The router makes use of Observable. So, the integration started kind of small which HTTPClient being written around Observable. And it's grown from there as people seem to be grabbing on and enjoying more the React programming side of things. So, it's definitely the one framework that's really embraced reactive programming outside of say, Cycle.js or something like that. CHARLES: Mmhmm. So, just to give a general background, how would you characterize RxJS? BEN: It's a library built around Observable. And Observable is a push-based primitive that gives you sets of events, really. CHARLES: Mmhmm. BEN: So, that's like Lodash for events would be a good way to put it. You can take anything that you can get pushed at you, which is pretty much value type you can imagine, and wrap it in an observable and have it pushed out of the observable. And from there, you have a set of things that you can combine. And you can concatenate them, you can filter them, you can transform them, you can combine them with other sets, and so on. So, you've got this ability to query and manipulate in a declarative way, events. CHARLES: Now, Observable is also… So, when Jay was on the podcast we were talking about Redux observable. But there was outside of the context of RxJS, it was just observables were this standalone entity. But I understand that they actually came from the RxJS project. That was the progenitor of observables even though there's talk of maybe making them part of the JavaScript spec. BEN: Yeah, that's right. That's right. So, RxJS as it stands is a reference implementation for what could land in JavaScript or what could even land in the DOM as far as an observable type. Observable itself is very primitive but RxJS has a lot of operators and optimizations and things written around Observable. That's the entire purpose of the library. CHARLES: Mmhmm. So, what kind of value-adds does it provide on top of Observable? If Observable was the primitive, what are the combinators, so to speak? BEN: Oh, right. So, similar to what Lodash would add on top of say, an iterable or arrays, you would have the same sorts of things and more inside of RxJS. So, you've got zip which you would maybe have seen in Lodash or different means of combines. Of course, map and ‘merge map' which is like a flattening sort of operation. You can concatenate them together. But you also have these time-based things. You can do debouncing or throttling of events as they're coming over in observable and you create a new observable of that. So, the value-add is the ability to compose these primitive actions. You can take on an observable and make a new observable. We call it operators. And you can use those operators to build pretty much anything you can imagine as far as an app would go. CHARLES: So, do you find that most of the time all of the operators are contained right there inside RxJS? Or if you're going to be doing reactive programming, one of your tasks is going to be defining your own operators? BEN: No, pretty much everything you'd need will be defined within RxJS. There's 60 operators or so. CHARLES: Whoa, that's a lot. BEN: It's unlikely that someone's going to come up with one. And in fact, I would say the majority of those, probably 75% of those, you can create from the other 25%. So, some of the much more primitive operators could be used… TRACY: Which is sort of what Ben did in this last release, RxJS 5…. I don't know remember when you introduced the lettable operators but you… BEN: Yeah, 5.5. TRACY: Implemented [inaudible] operators. BEN: Yeah, so a good portion of them I started implementing in terms of other operators. CHARLES: Right. So, what was that? I didn't quite catch that, Tracy. You said that, what was the operator that was introduced? TRACY: So, in one of the latest releases of RxJS, one of the more significant releases where pipeable operators were introduced, what Ben did was he went ahead and implemented a lot of operators that were currently in the library in terms of other operators, which was able to give way to reduce the size of the library from, I think it was what, 30KB bundled, gzipped, and minified, to about 30KB, which was about 60 to 70% of the operators. Right, Ben? BEN: Yeah. So, the size reduction was in part that there's a lot of factors that went into the size reduction. It would be kind of hard to pin it down to a specific operator. But I know that some of the operators like the individual operators themselves, by reimplementing reduce which is the same as doing as scan and then take last, implementing it in terms of that is going to reduce the size of it probably 90% of that one particular file. So, there's a variety of things like that that have already started and that we're going to continue to do. We didn't do it with every operator that we could have. Some operators are very, very common and consequently we want them to be as optimized as possible. For example, map. You can implement map in terms of ‘merge map' but it would be very slow to do so. It might be smaller but it would be slower. We don't want that. So, there are certain areas we're always going to try to keep fairly a hot path to optimize them as much as possible. But in other spots like reduce which is less common and isn't usually considered to be a performance bottleneck, we can cut some corners. Or ‘to array' or other things like that. CHARLES: Mmhmm. TRACY: And I think another really interesting thing is a lot of people when learning RxJS, they… it's funny because we just gave an RX Workshop course this past weekend and the people that were there just were like, “Oh, we've heard of RxJS. We think it's a cool new thing. We have no plans to implement it in real life but let's just play around with it and let me learn it.” I think as people are starting to learn RxJS, one of the things that gets them really overwhelmed is this whole idea that they're having to learn a completely new language on top of JavaScript or what operators to use. And one of our friends, Brian Troncone who is on the Learning Team, the RxJS Learning Team, he pulled up the top 15 operators that were most commonly searched on his site. And some of them were ‘switch map', ‘merge map', ‘fork join', merge, et cetera. So, you can sort of tell that even though the library has quite a few… it's funny because Ben, I think the last RX Workshop you were using pairs and you had never used it before. BEN: Yeah. TRACY: So, it's always amusing for me how many people can be on the core team but have never implemented RxJS… CHARLES: [Laughs] TRACY: A certain way. BEN: Right. Right, right, right. CHARLES: You had said one of the recent releases was about making it more friendly for functional programming. Is that a subject that we can explore? Because using observables is already pretty FP-like. BEN: What it was before is we had dot chaining. So, you would do ‘dot map' and then call a method and then you get an observable back. And then you'd say ‘dot merge' and then you'd call a method on that, and so on and so forth. Now what you have is kind of a Ramda JS style pipe function that just takes a comma-separated list of other functions that are going to act upon the observable. So, it reads pretty much the same with a little more ceremony around it I guess. But the upside is that you can develop your operators as just higher-order functions. CHARLES: Right. And you don't have to do any monkey-patching of prototypes. BEN: Exactly, exactly. CHARLES: Because actually, okay, I see. This is actually pretty exciting, I think. Because we actually ran into this problem when we were using Redux Observable where we wanted to use some operators that were used by some library but we had to basically make a pull request upstream, or fork the upstream library to include the operators so that we could use them in our application. It was really weird. BEN: Yeah. CHARLES: The reason was because it was extending the observable prototype. BEN: Yeah. And there's so many… and that's one way to add that, is you extend the observable prototype and then you override lift so you return the same type of observable everywhere. And there are so many things that lettable operators solved for us. For example… CHARLES: So, lettable operators. So, that's the word that Tracy used and you just used it. What are lettable operators? BEN: Well, I've been trying to say pipeable and get that going instead of lettable. But basically there's an operator on RxJS that's been there forever called let. And let is an operator and what you do is you give it a function. And the function gives you the source observable and you're expected to return a new observable. And the idea is that you can then write a function elsewhere that you can then compose in as though it were an operator, anywhere you want, along with your other dot-chained operators. And the realization I had a few months ago was, “Well, why don't we just make all operators like this?” And then we can use functional programming to compose them with like a reduce or whatever. And that's exactly what the lettable operators are. And that's why I started calling them lettable operators. And I kind of regret it now, because so many people are saying it and it confuses new people. Because what in the world does lettable even mean? CHARLES: Right. [Laughs] BEN: So, they are pipeable operators or functional operators. But the point is that you have a higher-order function that returns a function of a specific shape. And that function shape is, it's a function that receives an observable and returns an observable, and that's it. So, basically it's a function that transforms an observable into a new observable. That's all an operator. That's all an operator's ever been. It's just this is in a different flavor. CHARLES: Now, I'm curious. Why does it do an observable into an observable and not a stream item into an observable? Because when you're actually chaining these things together, like with a map or with a ‘flat map' or all these things, you're actually getting an individual item and then returning an observable. Well, I guess in this case of a map you're getting an item and returning an item. But like… BEN: Right, but that's not what the entire operation is. So, you've got an operation you're performing whenever you say, if you're to just even dot-chain it, you'd say ‘observable dot map'. And when you say ‘dot map', it returns a new observable. And then you say ‘dot filter' and it returns another new observable. CHARLES: Oh, gotcha, gotcha, gotcha. Okay, yeah, yeah, yeah. Yeah, yeah. BEN: So, this function just embodies that step. CHARLES: I see, I see. And isn't there some special… I feel like there's some proposal for some special JavaScript syntax to make this type of chaining? BEN: Yeah, yeah, the pipeline operator. CHARLES: Okay. BEN: I don't know. I think that's still at stage one. I don't know that it's got a lot of headway. My sources and friends that are in the TC39 seem to think that it doesn't have a lot of headway. But I really think it's important. Because if you look at… the problem is we're using a language where the most common use case is you have to build it, get the size as small as possible because you need to send it over the wire to the browser. And understandably, browsers don't want to implement every possible method they could on say, Array, right? CHARLES: Mmhmm, right. BEN: There's a proposal in for ‘flat map'. They could add zip to Array. They could add all sorts of interesting things to Array just by itself. And that's why Lodash exists, right? CHARLES: Right. BEN: Is because not everything is on Array. And then so, the onus is then put on the community to come up with these solutions and the community has to build libraries that have these constraints in size. And what stinks about that is then you have say, an older version of Lodash where you'd be like, “Okay, well it has 36 different functions in it and I'm only using 3 of them. And I have to ship them all to the browser.” CHARLES: Mmhmm. BEN: And that's not what you want. So, then we have these other solutions around tree-shaking and this and that. And the real thing is what you want is you want to be able to compose things left to right and you want to be able to have these functions that you can use on a particular type in an ad hoc way. And there's been two proposals to try to address this. One was the ‘function bind' operator, CHARLES: Mmhmm. BEN: Which is colon colon. And what that did is it said, “You can use this function as a method, as though it were a method on an object. And we'll make sure that the ‘this' inside that function comes from the instance that's on the left-hand side of colon colon.” CHARLES: Right. BEN: That had a bunch of other problems. Like there's some real debate I guess on how they would tie that down to a specific type. So, that kind of fell dead in the water even though it had made some traction. And then the pipeline operator is different. And then what it says is, “Okay, whatever is on the…” And what it looks like is a pipe and a greater than right next to each other. And whatever's on the left-hand side of that operand gets passed as the first argument to the function on the right-hand side of that operand. CHARLES: Mmhmm. BEN: And so, what that means is for the pipeable operators, instead of having to use a pipe method on observable, you can just say, “instance of observable, pipeline operator and an operator, and then pipeline operator, and then the Rx operator, and then pipeline operator and the Rx operator, and so on.” And it would just be built-in. And the reason I think that JavaScript really needs it is that means that libraries like Lodash can be written in terms of simple functions and shipped piece-meal to the browser exactly as you need them. And people would just use the pipeline operator to use them, instead of having to wrap something in a big object so you can dot-chain things together or come up with your own functional pipe thing like RxJS had to. CHARLES: Right. Because it seems it happens again and again, right? Lodash, RxJS, jQuery. You just see this pattern of chaining, which is, you know… BEN: Yeah, yeah. People want chaining. People want left to right composition. CHARLES: Mmhmm. BEN: And it's problematic in a world where you want to shake off as much unused garbage as possible. And the only way to get dot chaining is by augmenting a prototype. There's all sorts of weird problems that can come with that. And so, the functional programming approach is one method. But then people look at it and they say, “Ooh, yuck. I've got to wrap things in a function named pipe. Wouldn't it be nicer if there was just some syntax to do this?” And yeah, it would be nicer. But I have less control over that. CHARLES: Right. But the other alternative is to have right to left function composition. BEN: Right, yeah. CHARLES: But there's not any special syntax for that, either. BEN: Not very readable. CHARLES: Yeah. BEN: So, you just wrap everything. And the innermost call is the first one and then you wrap it in another function and you wrap that in another function, and so on. Yeah, that's not [inaudible]. But I will say that the pipe function itself is pretty simple. It's basically a function that takes a rest of arguments that are all functions. CHARLES: Mmhmm. BEN: And so, you have this array of functions and you just reduce over it and call them. Well, you return a function. So, it's a higher function. You return a function that takes an argument then you reduce over the functions that came in as arguments and you call each one of them with whatever result was from the previous. CHARLES: Right. Like Tracy mentioned in the pre-show, I'm an aspiring student of functional programming. So, would this be kind of like a monoid here where you're mashing all these functions together? Is your empty value? I'm just going to throw it out there. I don't know if it's true or not, but that's my conjecture. BEN: Yes. Technically, it's a monoid because it wouldn't work unless it was a monoid. Because monoids, I believe the category theory I think for monoid is that monoids can be concatenated because they definitely have an end. CHARLES: Right. BEN: So, you would not be able to reduce over all those functions and build something with that, like that, unless it was a monoid. So yeah, the fact that there's reduction involved is a cue that it's a monoid. CHARLES: Woohoo! Alright. [Laughter] CHARLES: Have you found yourself wanting to apply some of these more “rigorous” formalisms that you find out there in the development of RxJS or is that just really a secondary concern? BEN: It's a secondary concern. It's not something that I like. It's something I think about from time to time, when really, debating any kind of heavy issue, sometimes it's helpful. But when it comes to teaching anybody anything, honestly the Haskell-isms and category theory names, all they do is just confuse people. And if you tell somebody something is a functor, they're like, “What?” And if you just say it's mappable, they're like, “Oh, okay. I can map that.” CHARLES: [Laughs] Right, right. BEN: And then the purists would be like, “But they're not the same thing.” And I would be like, “But the world doesn't care. I'm sorry.” CHARLES: Yeah, yeah. I'm kind of experiencing this debate myself. I'm not quite sure which side I fall on, because on the one hand it is arbitrary. Functor is a weird name. But I wish the concept of mappable existed. It does, but I feel like it would be handy if people… because there's literally five things that are super handy, right? Like mappable, if we could have a name for monoid. But it's like, really, you just need to think in terms of these five constructs for 99% of the stuff that you do. And so, I always wonder, where does that line lie? And how… mappable, is that really more accessible than functor? Or is that only because I was exposed to the concept of mapping for 10 years before I ever heard the F word. BEN: Yes, and yes. I mean, that's… CHARLES: [Laughs] BEN: Things that are more accessible are usually more accessible because of some pre-given knowledge, right? What works in JavaScript probably isn't going to work in Haskell or Scala or something, right? CHARLES: Mmhmm. BEN: If someone's a Java developer, certain idioms might not make sense to them that come from the JavaScript world. CHARLES: Right. But if I was learning like a student, I would think mappable, I'd be thinking like, I would literally be thinking like Google Maps or something like that. I don't know. BEN: Right, right. I mean, look at C#. C#, a mapping function is always going to be called select, right, because that's C#. That's their idiom for the same thing. CHARLES: Select? BEN: Yeah. CHARLES: Really? BEN: Yeah, select. So, they'll… CHARLES: Which in Ruby is like find. BEN: Yeah. there's select and then, what's the other one, ‘select many' or something like that. [Chuckles] BEN: So, that's C#. CHARLES: Oh, like it's select from SQL. Okay. BEN: Yeah, I think that's kind of where it came from because people had link and then they had link to SQL and then they're like, well I want to do this with regular code, with just using some more… less nuanced expressions. So, I want to be able to do method calls and chain those together. And so, you end up with select functions. And I think that that exists even in Rx.NET, although I haven't used Rx.NET. CHARLES: Hmm, okay. ELRICK: So, I know you do a lot of training with Rx. What are some of the concepts that people struggle with initially? TRACY: I think when we're teaching RX Workshop, a lot of the people sort of… I'll even see senior level people struggle with explaining it, is the difference between observables and observers and then wrapping their head around the idea that, “Hey, observables are just functions in JavaScript.” So, they're always thinking observables are going to do something for you. Actually, it's not just in Angular but also in React, but whenever someone's having issues with their Rx applications, it's usually something that they're like nesting observables or they're not subscribing to something or they've sort of hot-messed themselves into a tangle. And I'm sure you've debugged a bunch of this stuff before. The first thing I always ask people is, “Have you subscribed?” Or maybe they're using an Angular… they're using pipes async but they're also calling ‘dot subscribe' on their observable. BEN: Yeah. So, like in Angular they'll do both. Yeah. There's that. I think that, yeah, that relates to the problem of people not understanding that observables are really just functions. I keep saying that over and over again and people really don't seem to take it to heart for whatever reason. [Chuckles] BEN: But you get an observable and when you're chaining all those operators together, you're making another observable or whatever, observables don't do anything until you subscribe to them. They do nothing. CHARLES: Shouldn't they be called like subscribable? BEN: Yes. [Chuckles] BEN: They probably should. But we do hand them an observer. So, you are observing something. But the point being is that they don't do anything at all until you subscribe to them. And in that regard, they're like functions, where functions don't do anything unless you call them. So, what ends up happening with an observable is you subscribe to it. You give it an observer, three callbacks which are then coerced into an observer. And it takes that observer and it hands it to the body of this observable definition and literally has an observer inside of there. And then you basically execute that function synchronously and do things, whatever those things are, to set up some sort of observation. Maybe you spin up a WebSocket and tie into some events on it and call next on the observer to get values out of your observable. The point being that if you subscribe to an observable twice, it's the same thing as calling a function twice. And for some reason, people have a hard time with that. They think, if I subscribe to the observable twice, I've only called the function once. CHARLES: I experienced this confusion. And I remember the first time that that… like, I was playing with observables and the first time I actually discovered that, that it was actually calling my… now what do you call the function that you pass to the constructor that actually does, that calls next or that gets passed the observer? TRACY: [Inaudible] BEN: I like to call it an initialization function or something. But the official name from the TC39 proposal is subscriber function. CHARLES: Subscriber function. So, like… BEN: Yeah. CHARLES: I definitely remember it was one of those [makes explosion sound] mind-blowing moments when I realized when I call my subscribe method, the entire observable got run from the very beginning. But my intuition was that this is an object. It's got some shared state, like it's this quasar that I'm now observing and I'm seeing the flashes of light coming off of it. But it's still the same object. You think of it as having yeah, not as a function. Okay. No one ever described it to me as just a function. But I think I can see it now. ELRICK: Yeah, me neither. CHARLES: But yeah, you think of it in the same way that most people think of objects, as like, “I have this object. I have a reference to it.” Let observable equal new observable. It's a single thing. It's a single identity. And so, that's the thing that I'm observing. It's not that I'm invoking this observable to observe things. And I think that's, yeah, that's a subtle nuance there. I wish I had taken y'all's course, I guess is what I'm saying. ELRICK: Yeah. BEN: Yeah. Well, I've done a few talks on it. CHARLES: [Laughs] BEN: I always try to tell people, “It's just a function. It's just a function.” I think what happens to a lot of people too is there's the fact that it's an object. But I think what it is, is people's familiarity with promises does this. Because promises are always multicast. They are always “hot”. And the reason for this is because they're eager. So, by the time you have a promise, whatever is producing value to the promise has already started. And that means that they're inherently a multicast. CHARLES: Right. BEN: So, people are used to that behavior of, I can ‘then' off of this promise and it always means one thing. And it's like, yeah, because the one thing has nothing to do with the promise. It wasn't [Chuckles] CHARLES: Right. BEN: This promise is just an interface for you to view something that happened in the past, where an observable is more low-level than that and more simple than that. It just states, “I'm a function that you call. I'm going to be able to do anything a function can do. And by the way, you're giving me an observer and I'm going to do some stuff with that too and notify you via this observer that you handed me.” Because of that you could take an observable and close over something that had already started. Say you had a WebSocket that was already running. You could create a new observable and just like any function, close over that, externally create a WebSocket. And then everyone that subscribes to that observable is tying an observer to that same WebSocket. Then you're multicast. Then you're “hot”. ELRICK: [Inaudible] CHARLES: Right. So, I was going to say that's the distinction that Jay was talking about. He was talking about we're going to just talk about… he said at the very beginning, “We're just going to talk about hot observable.” ELRICK: Yup. CHARLES: But even a hot observable is still theoretically evaluating every single time you subscribe. You're getting a new observable. You're evaluating that observable afresh each time. It just so happens that in the lexical scope of that observable subscriber function, there is this WebSocket? BEN: Yeah. So, it's the same thing. Imagine you wrote a function that when you called it created a new WebSocket and then… say, you wrote a new function that you gave an observer object to, right? An observer object has next, error, and complete. And in that function, when you called it, it created a new WebSocket and then it tied the ‘on message' and ‘on close' and whatever to your observer's next method and your observer's error message and so on. When you call that function, you would expect a new WebSocket to be created every single time. Now, let's just say alternately you create a WebSocket and then you write a new function that that function closes over that WebSocket. So, you reference the WebSocket that you externally created inside of your function. When you call that function, it's not going to create a new WebSocket every time. It's just closing over it, right? So, even though they both are basically doing the same thing, now the latter one of those two things is basically a hot observable and the former is a cold observable. Because one is multicast which is, “I'm sharing this one WebSocket with everybody,” and the other one is unicast which is, “I am going to create a new WebSocket for each person that calls me.” And that's the [inaudible] people have a hard time with. CHARLES: Right. But really, it's just a matter of scope. BEN: Yeah. The thing people have a hard time with, with observables, is not realizing that they're actually just functions. CHARLES: Yeah. I just think that maybe… see, when I hear things like multicast and unicast, that makes me think of shared state, whereas when you say it's just a matter of scope, well then I'm thinking more in terms of it being just a function. It just happens that this WebSocket was already [scoped]. BEN: Well, shared state is a matter of scope, right? CHARLES: Yes, it is. It is. Oh, sorry. Shared state associated with some object identity, right? BEN: Right. CHARLES: But again, again, it's just preconceptions, really. It's just me thinking that I've had to manage lists of listeners and have multicast observers and single-cast observers and having to manage those lists and call notify on all of them. And that's really not what's happening at all. BEN: Yeah. Well, I guess the real point is observables can have shared state or they could not have shared state. I think the most common version and the most composable version of them, they do not have any shared state. It's just one of those things where just like a function can have shared state or it could be pure, right? There's nothing wrong with either one of those two uses of a function. And there's nothing wrong with either one of those two uses of Observable. So, honest to god, that is the biggest stumbling block I think that I see people have. That and if I had to characterize it I would say fear and loathing over the number of operators. People are like… CHARLES: [Chuckles] BEN: And they really think because everyone's used to dealing with these frameworks where there's an idiomatic way to do everything, they think there's going to be an RxJS idiomatic way to do things. And that's just patently false. That's like saying there's an idiomatic way to use functions. There's not. Use it however it works. The end. It's not… CHARLES: Mmhmm, mmhmm. BEN: You don't have to use every operator in a specific way. You can use it however works for you and it's fine. ELRICK: I see that you guys are doing some fantastic work with your documentation. Was that part of RxJS 2.0 docs? TRACY: I was trying to inspire people to take on the docs initiative because I think when I was starting to learn RxJS I would get really frustrated with the docs. BEN: Yeah. TRACY: I think the docs are greatly documented but at the same time if you're not a senior developer who understands Rx already, then it's not really helpful. Because it provides more of a reference point that the guys can go back and look at, or girls. So anyways, after many attempts of trying to get somebody to lead the project I just decided to lead the project myself. [Laughter] TRACY: And try to get… the community is interesting because I think because the docs can be sometimes confusing… Brian Troncone created LearnRxJS.io. There's these other visualization projects like RxMarbles, RxViz, et cetera. And we just needed to stick everybody together. So, it's been a project that I think has been going on for the past two months or so. We have… it's just an Angular app so it's probably one of the most easiest projects to contribute to. I remember the first time I tried to contribute to the Ember docs. It literally took me an hour to sit there with a learning team, Ember Learning Team member and… actually, maybe it was two hours, just to figure out how the heck… like all the things I had to download to get my environment set up so that I could actually even contribute to the darn documentation. But with the Rx, the current RxJS docs right now is just an Angular app. You can pull it down. It's really easy. We even have people who are just working on accessibility, which is super cool, right? So, it's a very friendly place for beginners. BEN: I'm super pleased with all the people that have been working on that. Brian and everybody, especially on the accessibility front. Jen Luker [inaudible] came in and voluntarily… she's like the stopgap for all accessibility to make sure everything is accessible before we release. So, that's pretty exciting. TRACY: Yeah. ELRICK: Mmhmm. TRACY: So funny because when me and Jen started talking, she was talking about something and then I was like, “Oh my god, I'm so excited about the docs.” She's like, “I'm so excited, too! But I don't really know why I'm excited. But you're excited, so I'm excited. Why are you excited?” [Laughter] TRACY: I was like, “I don't know. But I'm excited, too!” [Chuckles] TRACY: And then all of a sudden we have accessibility. [Laughs] ELRICK: Mmhmm. Yeah, I saw some amazing screenshots. Has the new docs, have they been pushed up to the URL yet? TRACY: Nah, they are about to. We were… we want to do one more accessibility run-through before we publish it. And then we're going to document. We want to document the top 15 most viewed operators. But we should probably see that in the next two weeks or so, that the new docs will be… I mean, it'll say “Beta, beta, beta” all over everything. But actually also, some of our friends, [Dmitri] from [Valas] Software, he is working on the translation portion to make it really easy for people to translate the docs. CHARLES: Ah. TRACY: So, a lot of that came from the inspiration from the Vue.js docs. we're taking the versioning examples that Ember has done with their docs as inspiration to make sure that our versioning is really great. So, it's great that we can lend upon all the other amazing ideas in the industry. ELRICK: Oh, yeah. CHARLES: Yeah, it's fantastic. I can't wait to see them. ELRICK: Yeah, me neither. The screenshots look amazing. I was like, “Wow. These are some fabulous documentation that's going to be coming out.” I can't wait. TRACY: Yeah. Thank you. CHARLES: Setting the bar. ELRICK: Really high. [Laughter] CHARLES: Actually, I'm curious. Because observables are so low-level, is there some use of them that… what's the use of them that you found most surprising? Or, “Whoa, this was a crazy hack.” BEN: The weirdest use of observables, there's been quite a few odd ones. One of the ones that I did one time that is maybe in RxJS's wheelhouse, it was just that RxJS already existed. So, I didn't want to pull in another transducer library, was using RxJS as a transducer. Basically… in Netflix we had a situation where we had these huge, huge arrays of very large objects. And if you try to take something like that and then map it and then filter it and then map it and then filter it, we're using Array map and filter, what ends up happening is you create all sorts of intermediary arrays in-memory. And then garbage collection has to come through and clean that up. And that locks your thread. And over time, we were experiencing slowness with this app. And it would just build up until eventually it ground to a halt. And I used RxJS because it was an available tool there to wrap these arrays in an observable and then perform operations on them step-by-step, the same map, filter, and so on. But when you do that, it doesn't create intermediary arrays because it passes each value along step to step instead of producing an entire array and then doing another step and producing an entire array, and so on. So… CHARLES: So, will you just… BEN: It saved garbage collection and it increased the performance of the app. But that's just in an extreme case. I would never do that with just regular arrays. If anything, it was because it was huge, huge arrays of very large objects. CHARLES: So, you would create an observable our of the array and then just feed each element into the observable one at a time? BEN: Well, no. If you say ‘observable from' and you give it an array, that's basically what it does. CHARLES: Okay. BEN: It loops over the array and nexts those values out of the array synchronously. CHARLES: I see, I see. BEN: So, it's like having a for loop and then inside of that for loop saying, “Apply the map. Apply the filter,” whatever, to each value as they're going through. But when you look at it, if you had array map, filter, reduce, it's literally just taking the first step and saying ‘observable from' and wrapping that array and then the rest of it's still the same. CHARLES: Right. Yeah. No, that's really cool. BEN: That was a weirder use of it. I've heard tell of other things where people used observables to do audio synchronization, which is pretty interesting. Because you have to be very precise with audio synchronization. So, hooking into some of the Web Audio APIs and that sort of thing. That's pretty interesting. The WebSocket multiplexing is something I did at Netflix that's a little bit avant-garde for observable use because you essentially have an observable that is your WebSocket. And then you create another observable that closes over that observable and sends messages over the WebSocket for what you're subscribed to and not subscribed to. And it enables you to very easily retry connections and these sorts of things. I did a whole talk on that. That one's pretty weird. CHARLES: Yeah. Man, I [inaudible] to see that. BEN: But in the general use case, you click a button, you make an AJAX request, and then you get that back and maybe you make another AJAX request. Or like drag and drop and these sorts of things where you're coordinating multiple events together, is the general use case. The non-weird use case for RxJS. Tracy does weird stuff with RxJS though. [Laughter] CHARLES: Yeah, what's some weird uses of RxJS? TRACY: I think my favorite thing to do right now is to figure out how many different IoT-related things I can make work with RxJS. So, how many random things can I connect to an application using that? BEN: Tracy's projects are the best. They're so good. [Laughter] TRACY: Well, Ben and I created an application where you can take pictures of things using the Google Image API and it'll spit back a set of puns for you. So, you take a picture of a banana, it'll give you banana puns. Or you can talk to it using the speech recognition API. My latest thing is I really want to figure out how to… I haven't figured out if Bluetooth Low Energy is actually enabled on Google Home Minis. But I want to get my Google Home Mini to say ‘booty'. [Inaudible] [Laughter] CHARLES: RxJS to the rescue. [Laughter] BEN: Oh, there was, you remember Ng-Cruise. We did Ng-Cruise and on there, Alex Castillo brought… TRACY: Oh, that was so cool. BEN: All sorts of interesting… you could read your brain waves. Or there was another one that was, what is it, the Microsoft, that band put around your wrist that would sense what direction your arm was in and whether or not your hand was flexed. And people… TRACY: Yeah, so you could flip through things. BEN: Yeah. And people were using reactive programming with that to do things like grab a ball on the screen. Or you could concentrate on an image and see if it went blurry or not. ELRICK: Well, for like, Minority Report. BEN: Oh, yeah, yeah. Literally, watching a machine read your mind with observables. That was pretty cool. That's got to be the weirdest. TRACY: Yeah, or we had somebody play the piano while they were wearing one of the brainwave… it's called the OpenBCI project is what it is. And what you can do is you can actually get the instructions to 3D print out your own headset and then buy the technology that allows you to read brain waves. And so with that, it's like… I mean, it was really awesome to watch her play the piano and just see how her brain waves were going super crazy. But there's also these really cool… I don't know if you guys have heard of Jewelbots, but they're these programmable friendship bracelets that are just little Arduino devices that light up. I have two of them. I haven't even opened them. CHARLES: [Laughs] TRACY: I've been waiting to play with them with you. I don't know what we're going to do, but I just want to send you lights. Flashing lights. [Laughter] TRACY: Morse code ask you questions about RxJS while you're working. [Laughter] CHARLES: Yeah. Critical bug. Toot-toot-toot-too-too-too-too-toot-toot. [Laughter] CHARLES: RxJS Justice League. TRACY: That would actually be really fun. [Laughter] TRACY: That would be really fun. I actually really want to do that. But… CHARLES: I'm sure the next time we talk, you will have. TRACY: [Laughs] Yes. Yes, yes, yes, I know. I know. we'll do it soon. We just need to find some time while we're not going crazy with conferences and stuff like that. CHARLES: So, before we head out, is there any upcoming events, talks, releases, anything that we ought to be, we or the listeners, ought to be aware of? TRACY: Yeah, so one of the things is that Ben and I this weekend actually just recorded the latest version of RX Workshop. So, if you want to learn all about the latest, latest, newest new, you can go ahead and take that course. We go through a lot of different things like multiplex WebSockets, building an application. Everywhere from the fundamentals to the more real world implementations of RxJS. BEN: Yeah. Even in the fundamentals area, we've had friends of ours that are definitely seasoned Rx veterans come to the workshop. And most of them ask the most questions while talking about the fundamentals. Because I tend to dig into, either deep into the internals or into the why's and how's thing. Why and how things work. Even when it comes to how to subscribe to an observable. Deep detailed information about what happens if you don't provide an error handler and certain cases and how that's going to change in upcoming versions, and why that's changing in upcoming versions, and what the TC39's thoughts are on that, and so on and so forth. So, I try to get into some deeper stuff and we have a lot of fun. And we tend to be a little goofier at the workshops from time to time than we were in this podcast. Tracy and I get silly when we're together. TRACY: It's very true. [Laughter] TRACY: But I think also, soon I think there are people that are going to be championing an Observable proposal on what [inaudible]. So, aside from the TC39 Observable proposal that's currently still at stage one, I don't know Ben if you want to talk a little bit about that. BEN: Oh, yeah. So, I've been involved in conversations with folks from Netflix and Google as well, Chrome team and TC39 members, about getting the WHATWG, the ‘what wig', they're a standards body similar to W3C, to include observables as part of the DOM. The post has not been made yet. But the post is going to be made soon as long as everybody's okay with it. And what it boils down to is the idea of using observables as part of event targets. An event target is the API we're all familiar with for ‘add event listener', ‘remove event listener'. So, pretty much anywhere you'd see those methods, there might also someday be an on method that would return an observable of events. So, it's really, really interesting thing because it would bring at least the primitives of reactive programming to the browser. And at the very least it would provide maybe a nicer API for people to subscribe to events coming from different DOM elements. Because ‘add event listener' and ‘remove event listener' are a little unergonomic at times, right? CHARLES: Yeah. They're the worst. BEN: Yeah. CHARLES: That's a very polite way of putting it. BEN: [Chuckles] So, that's one thing that's coming down the pipe. Other things, RxJS 6 is in the works. We recently tied off 5.5 in a stable branch. And master is now our alpha that we're working on. So, there's going to be a lot of refactoring and changes there, trying to make the library smaller and smaller. And trying to eliminate some of the footprints that maybe people had in previous versions. So, moving things around so people aren't importing stuff that were meant to be implementation details, reducing the size of the library, trying to eliminate some bloat, that sort of thing. I'm pretty excited about that. But that's going to be in alpha ongoing for a while. And then hopefully we'll be able to move into beta mid first quarter next year. And then when that'll be out of beta, who knows? It all depends on how well people like the beta and the alpha, right? CHARLES: Alright. Well, so if folks do want to follow up with y'all either in regards to the course or to upcoming releases or any of the other great stuff that's coming along, how would they get in touch with y'all? TRACY: You can find me on Twitter @ladyleet. But Ben is @BenLesh. RX Workshop is RXWorkshop.com. I think in January we're going to be doing state of JavaScript under This Dot Media again. So, that's where all the core contributors of different frameworks and libraries come together. So, we'll definitely be giving a state of RxJS at that time. And next year also Contributor Days will be happening. So, if you go to ContributorDays.com you can see the previous RxJS Contributor Days and figure out how to get involved. So, we're always open and happy and willing to teach everybody. And again, if you want to get involved it doesn't matter whether you have little experience or lots of experience. We are always willing to show you how you can play. BEN: Yeah. You can always find us on Twitter. And don't forget that if you don't find Tracy or I on Twitter, you can always message Jay Phelps on Twitter. That's important. @_JayPhelps. Really. TRACY: Yeah. [Laughter] BEN: You'll find us. CHARLES: [Chuckles] Look for Jay in the show notes. [Laughter] CHARLES: Alright. Well, thank you so much for all the stuff that y'all do, code and otherwise. And thank you so much Ben, thank you so much Tracy, for coming on the show. BEN: Thank you. CHARLES: Bye Elrick and bye everybody. If you want to reach out to us, you can always get in touch with us at @TheFrontside or send us an email at contact@frontside.io. Alright everybody, we'll see you next week.

How do we ensure a high level of quality and maximize the refactorability of our code? Frontsiders, Wil and Charles, talk about their battle tested techniques for testing web applications, not only in React JS, but in any JavaScript framework. Links Acceptance Testing Integration Testing jest Cypress.io Assertion Ember CLI Mirage mirage-server react-trigger-change Transcript CHARLES: Hello everybody and welcome to The Frontside Podcast, Episode 90. My name is Charles Lowell, a developer here at The Frontside and your podcast host-in-training. And with me today is Mr. Wil Wilsman. Wil, who just got back from Nodevember just walked straight into the office and is ready to podcast with us on a very, very, very interesting subject, I think, today. We're going to be talking about acceptance testing in JavaScript applications, especially some of the techniques that we've developed here around testing React applications based on the lessons that we learned from the Ember community. But really, more than just React applications. Really, testing any JavaScript application from the inside out, making acceptance tests for that. So, I think we're going to talk about some of the challenges that you encounter and some of the really novel solutions that are out there that we had nothing to do with. And I guess we really didn't have, just more of cobbling together of various techniques for a powerful witch's brew for acceptance testing. Anyway, so Wil, just to round out the problem space or explore the problem space, what are some of the challenges that you encounter with an acceptance test? Actually, let me back it up even further. What is an acceptance test in a JavaScript application compared to what people normally encounter? WIL: Acceptance testing or end-to-end testing is just a problem that every JavaScript app should face. Not everyone does, but they definitely should. And basically, it's how the user interacts with your app through the browser. And every part of that we want to test, from the browser triggering browser events, interacting with the app, not calling functions or clicking buttons, and we're pretending we're a user. CHARLES: Yeah. You know, I know that when we showed up in the React space, that was not really the way that most people tested their applications. WIL: No, not at all. they're all about unit testing. Make sure every small piece of your code works, and to some degree integration testing, making sure your components work with other components. But, nothing is out there really for those big acceptance tests that you want the user to click a button and expect them be brought to a page or these fields to be filled out, et cetera. CHARLES: Mmhmm. Yeah. And there certainly was a very high level of maturity around unit testing, like you said. There are tools like Enzyme and… WIL: Jest. CHARLES: Yeah, Jest. But I was actually shocked to find out that Jest didn't even run in the browser. WIL: Yeah, it's all virtual. CHARLES: It's all virtual. It's completely and totally simulated and stubbed. And that presents some problems. WIL: Yeah. The main problem is cross-browser testing. Some people might consider that to be separate from their acceptance testing but you should be able to just run your acceptance tests in multiple browsers and be able to also test cross-browser support. CHARLES: Mmhmm. Yeah. And so, if you're using something like Jest, you're never actually running the code inside Safari. You're never actually running it inside Internet Explorer. You're actually running it in NodeJS. WIL: And you know, your user is not going to run it in Node. [Laughter] WIL: They're going to use a browser. CHARLES: I don't know about your users. WIL: [Chuckles] CHARLES: [Laughs] You know, we like to stick to the pretty advanced. It's like, go to getNodeJSBrowser.com. WIL: [Laughs] CHARLES: Enough of this Firefox BS. But not seriously, it was certainly a problem. We were looking around, because we never like to build anything ourselves if we can avoid it. But it really just seemed like there was not an off-the-shelf solution for writing these big style acceptance tests in JavaScript. There are some services out there. There's a couple now. What was the… WIL: I think the main one here is Cypress. CHARLES: Cypress, yeah. So, there's Cypress now. I've watched the instructional videos but never actually tried to integrate it into my application. WIL: Yeah. I think at its core it takes the same approach that we've been doing with how we're interacting with our tests. CHARLES: Mmhmm. Okay. The main difference is, is it that it's a service? Like you have to edit your tests through… WIL: Yeah. CHARLES: Their web browser, their web interface, and use their assertion library? WIL: Yeah. I'm not sure about the editing part. But yeah, it's their assertion library. I'm pretty sure it's their test runner and it's their testing environment. Really, the only control through that is through their UI, or through settings, basically. And you're stuck with those. You can't use other… I don't think you can use Mocha with Cypress. CHARLES: Right. WIL: Although it's very much like Mocha… CHARLES: Right. WIL: It's not. CHARLES: Right, right. And I also noticed, we'll touch on this later, the assertions, most of the side effects that were happening were happening right there inline inside your assertions. And that might be an opaque statement, but we will actually get into that later. WIL: Yeah. And I think one of the things about their side effects so to speak is everything leading up to a side effect is a promise with Cypress. CHARLES: Mmhmm. WIL: So, when you select a button and click it, Cypress is going to wait for that button to actually exist before it clicks it. CHARLES: Right, right, which is actually pretty cool. So, that's actually a perfect into into one of the primary challenges with doing acceptance testing in general in a JavaScript application. This is a problem when you're doing it in Ember. It's a problem in React. It's really a problem anywhere. And that is, how do you know when the effects of a user's interaction have been realized? Right? WIL: Yeah. And in Ember you take advantage of the run loop. Once that action happens, you wait for the run loop to complete and then your tests run. CHARLES: Right. So, the idea is that I've clicked some button or I've typed some key or I've moved the mouse. And then I listen for the run loop and when it's “settled” then I can now run my assertions because I know that the side effects that I was looking for have now been realized. WIL: Yeah, hopefully, if you're... CHARLES: Hopefully. WIL: Writing your app right. [Chuckles] CHARLES: Right, right. But that actually presents some problems in itself because it requires visibility into the internals of the framework. WIL: Yeah, so Ember is built with testing in mind. CHARLES: Mmhmm. WIL: And other libraries like React just being a view library might not be built with testing in mind. So, we don't have those hooks to wait for this loop to complete, wait for all of these things to be rendered before you continue. CHARLES: Exactly. And so, this is, I think it's actually kind of both a blessing and a curse. Because there are such strong conventions in Ember, they were able to build this wonderful acceptance testing regimen from the get go. WIL: Yeah. CHARLES: But like you said, that doesn't exist at all in the React ecosystem. And so, what do you do? There's no run loop. You're cobbling together a bunch of different components. And maybe you're using Redux, maybe you're using MobX, maybe you're using… you're certainly using React. And all of these things have their own asynchronies built-in. And there's not one unifying abstraction that's keeping track of all the asynchrony in the system. And so that presents a challenge. So, the question is then, if you're trying to not actually check and observe the state of a system until the right moment, how do you know when that right moment is? WIL: Yeah. And in early testing of a side React project I had, I would basically wait for a state to be complete before I continued my ‘before each'. And in the testing we're doing now, it's essentially what we're doing except the state is what the browser sees, or what the user would see in the browser. CHARLES: So you were actually querying the... WIL: Yeah. So, I was using Redux. So, in my app I was saying, when the Redux app is done loading... CHARLES: Mmhmm. WIL: The instance is set to true or false, then continue the test. CHARLES: Mmhmm. And so, what that means, what we're doing, is doing the same thing except observing at the DOM... WIL: Yeah, exactly. CHARLES: Level. And what it means is we actually… I would love to set this up and have a big reveal but I guess we'll just have a big reveal, is that essentially what we do is polling. WIL: Yeah. CHARLES: Right? So, when we run an assertion, let's say you click a button and you want the button to become disabled, there's an inherent asynchrony there. But what we will do is we'll actually run the assertion to see if it's disabled not one time. We'll run it a thousand times. WIL: Yeah, as many times as needed until it passes. CHARLES: Right, exactly. As many times as needed until it passes. And I think that is, at least to most programmer instincts, an odious idea. WIL: Yeah. CHARLES: [Laughs] WIL: It's like, “Oh wait, you're just looping over every single assertion how many times?” CHARLES: Yeah, exactly. And it feels, yeah, it feels weird as an idea. But when you actually see the code that it produces, it just sweeps away so much complexity. WIL: Yeah. And… CHARLES: Because you don't worry about asynchrony at all. WIL: Yeah. And it's pretty genius. If I'm a user and I click a button, it's loading when I see that it's loading. So, our tests are going to wait until that button says it's loading. And then the test passes. CHARLES: Right. And so, what we do is we essentially, we use Mocha but you could do it with QUnit or anything else, is that when you run your assertion, you declare, you have an ‘it' block or I guess, what would it be in QUnit? WIL: A test? CHARLES: A test? WIL: I think it's just a test. CHARLES: You have your test block. And so that function that actually runs the assertion and checks the state will actually run, yeah it could run three times. It could run a thousand times. It's just sitting there waiting. And it will time out. And it will only fail if that assertion has failed a thousand times or it has failed through, I think two seconds is our default. WIL: Yeah, yeah. I think we default to the runner's default timeout. CHARLES: To the runner's default timeout, yeah. WIL: Yeah. Or you can set that yourself with how we have it set up. And the other thing that comes from that is if your tests are only failing when they time out, how do you know what's actually failing? And our solution to that was we catch the error every time it fails and right before the timeout actually happens we throw the real error. CHARLES: Yeah. Exactly. But the net effect is that you're able to write your assertions completely and totally oblivious of asynchrony. You don't, we don't have to worry about asynchrony pretty much at all. I mean, we do, and we'll get into that. So, I made a global statement and then immediately contradicted it. WIL: [Chuckles] CHARLES: But hey, you got to be controversial. But for the most part, asynchrony just disappears because asynchrony is baked into the fabric. So, rather than thinking about it as a one-off concern or a onesie-twosie, it's just every single assertion is just assumed to be asynchronous. And so, that actually means you don't have to deal with promises. You don't have to deal with run loops. You don't have to deal with anything. You just write your assertion and when it passes, it passes. And there are some really unique benefits for this. And there are some challenges. So, I think one of the first benefits is that it's actually way faster. WIL: Right. CHARLES: Which is counterintuitive. WIL: It's incredibly fast. CHARLES: It's very fast. WIL: Yeah, for all the loops that's happening you might think every loop is going to slow it down slightly. But it really doesn't. Our tests, each test, even though it asserts five or six times, it takes milliseconds. CHARLES: Mmhmm. WIL: The test itself might only loop twice. CHARLES: Right. Exactly. Whereas if you're waiting for a run loop to settle, you might have some… you click a button, it disables, it also fires off an Ajax request and does all this stuff. But if all my assertion wants to know is “Is this button disabled?” then I only need to assert until that has happened. I don't need to wait until all the side effects have settled... WIL: Mmhmm. CHARLES: And then do the assertion. I just know, “Hey, my assertion, the thing that I was waiting for - that happened. Let's move on.” Yeah. And so, it's so, so fast. And that was actually, I didn't predict that. But I was definitely pleasantly surprised. WIL: Yeah, that was a very nice surprise. And all of our tests ran so much quicker than they would have in a run loop environment with Ember or something. CHARLES: Right. Yeah, yeah. That was, we actually had just come off a project where we were having that thing, that exact problem, which was that yeah, our animations were slow. Or, the animations were fine. They were perfect. [Laughs] But there were slowing down the tests. WIL: Yes. I think in that project, it was like, 30-minute tests... CHARLES: Mmhmm WIL: For the whole suite to run. CHARLES: Yeah. To which I'll add a public service announcement. I think this is a conjecture but I do believe that animations are best applied not to individual components but by the thing that uses a component. So, I shouldn't have an animation that's like, implicit to a dialog. It should be the thing that's showing the dialog that gets to decide the animation to use. Anyway, just throwing that out there. WIL: [Laughs] CHARLES: Because animations are about context. And so, the context should provide the animation, not the individual atom. Anyway, moving on. WIL: Some other podcast. CHARLES: Yeah. [Laughter] CHARLES: That's another podcast right there. But there's also, this does present some challenges or requires code to be structured in a way that facilitates this. So, there are some challenges with this approach, some things you need to be aware of if you're using this kind of system. We've kind of settled on a name for what we call these types of assertions and these types of systems. WIL: Yeah. We call them convergent assertions, because you're converging on something to happen. It's going over and over until it happens. CHARLES: Right. WIL: And yeah, a lot of these challenges that we've come across are things that you might not think of, like there are a few instances of false positives... CHARLES: Mmhmm. WIL: That happen with these convergent assertions. CHARLES: Right. So, what would be an example there? WIL: So, the most common example that I'm seeing so far is when you're asserting that something didn't happen. CHARLES: Mm, right. WIL: That would immediately pass. But if it takes your app... CHARLES: [Laughs] WIL: A few seconds for it to actually happen, then you could still have an actual failure but your test passed immediately. CHARLES: Right, right. So, what's the countermeasure then? WIL: We invert our assertions. So, we make sure they fail for a certain amount of time. CHARLES: Right. So, the normal case where you just want to say, “I want to make sure that my state converges to this particular state.” WIL: Alright. I said fail at first. I meant, pass. We have to make sure it passes for a certain period of time. CHARLES: Right, exactly. WIL: So yeah, the normal way is it fails until it passes, and then it passes. When you invert one of these convergent assertions, you're just making sure it passes repeatedly and if it fails at any point, you throw a failure. CHARLES: Right, okay. And so, that's like, if I want to check that the button is not disabled, I need to check again and again and again and again. WIL: Until you're comfortable with saying, “Alright. It's probably not going to be disabled.” CHARLES: Yeah, exactly. And so there, it's kind of weird because it is dependent on a timeout. WIL: Mmhmm. CHARLES: You could go for two seconds and then at the very end it becomes disabled. So, you just kind of have to take that on faith. But... WIL: Yeah. CHARLES: In practice, I don't think that's been much of a problem. WIL: No. CHARLES: It's more indicative of, if your button disables after... WIL: A few seconds. CHARLES: A few seconds, what's up with your... WIL: Yeah, what's up with your app? CHARLES: Yeah, exactly. WIL: If you're waiting for an Ajax request or something, an example, then you should be using something like Mirage Server. CHARLES: Right. Which is, man, we got to get into that, too. There are a couple of other things that I wanted to talk about too, with these convergent assertions. And that is, typically when you look through the READMEs for most testing frameworks, you see the simple case of the entire test, the setup, the teardown, and the actual assertions, are in the actual test. WIL: Yeah. The ‘it' block in Mocha or the test block in QUnit. You click a button, and then make sure it's disabled, and it moves onto the next test. CHARLES: Mmhmm. WIL: Then you click a different button or the same button and you assert something else in the next test. CHARLES: Mmhmm. Right. WIL: And yeah, you can't do that with convergent assertions because they're looping. So, if you click a button in a loop it's going to keep clicking that button over and over and over again. CHARLES: Yeah. [Laughs] Right, right. So, it means that you need to be very conscientious about separating the parts of your tests that actually do the things that actually act the part of the user from the part of your test that's about observation. WIL: Yeah. So, our solution to that is we move out all of our things that have side effects like clicking a button or filling in a form, all that stuff happens in ‘before each's. And all of our actual assertions happen in these convergent ‘it' blocks that loop over and over again. So, our ‘before each' runs and clicks the button and then we have 10 or so tests that will loop and wait for various states to... CHARLES: Yeah. WIL: Be true. CHARLES: Right. That means that yeah, all these assertions do is they read state. And you just have to, you do have to be conscientious. You're not allowed to have any side effects inside your tests, your actual assertion blocks. WIL: Mmhmm. CHARLES: But that's actually, it's a good use case for the whole Act-Arrange-Assert, which has been around way, way, way before these techniques. But here, we're doing Act and Arrange in our ‘before each'... WIL: Mmhmm. CHARLES: And then we're doing Assert later. And I think it actually leads for more readable... WIL: Yeah, definitely. CHARLES: Things. WIL: And it also opens the door to something that we can't really take advantage of yet but if you have 10 assertions with one ‘before each' side effect, you could run all of those assertions in parallel. CHARLES: That's right. WIL: And your tests would be 10 times more faster. CHARLES: Mmhmm. Yeah, exactly. Or you could run them in parallel or you could just run them one after the other but you wouldn't have to run that ‘before each' 10 times. WIL: Yeah. But something with that, that I found, is if we move all of our side effects to ‘before' blocks instead of ‘before each' blocks, sometimes a test three tests down that's waiting for something to happen... CHARLES: Yeah. WIL: That thing might have already happened earlier... CHARLES: Yeah. WIL: And it already went away. A loading state is the best example of that. CHARLES: Mmhmm. WIL: You show the loading state, the loading state goes away. So, if you move that button click into a ‘before' and that loading state test is three tests in, that loading state is already going to be gone. CHARLES: Yeah, so I think the long story short is we've kind of come to the conclusion that we would have to write our own runner. WIL: Yeah. CHARLES: Essentially to take advantage of this. But that said, we've done some sketching about what we would gain by writing our own runner. And the speed, we're talking about exponential speedups. WIL: Yeah. CHARLES: Maybe taking an entire acceptance test suite and having it run in five or six seconds. WIL: Yeah. We're talking about these tests that are already extremely fast. CHARLES: Mmhmm. WIL: Each test takes a few milliseconds or tens of milliseconds to complete. But then if you can run all of those at the same time, all of your tests for that entire ‘describe' block just ran tens of milliseconds. CHARLES: Right. Yeah. So, it's really exciting and pretty tantalizing. And we would love to invest the time in that. I've always wanted to write our own test runner. But never had, [chuckles] never really had a reason. WIL: Yeah. CHARLES: Certainly not just for the sheer joy of it. Although I'm sure there is joy in writing it. But that, yeah, we'll have to wait on that. But I am actually really excited about the idea of being able to maybe bring this back to the Ember community. WIL: Yeah. CHARLES: Because acceptance tests getting out of control in terms of the speed is I think a problem with Ember applications. And I think this would do a lot to address that. WIL: Yeah. CHARLES: I think, how long, if we were just using a stock Ember acceptance testing setup for this, I think we have about 250 tests in this React app... WIL: Yeah. CHARLES: How long does it take to run? WIL: Right now I think our tests take something like 20 seconds. And that's also somewhat due to they have to print the tests on the screen on Travis so that takes a little time. In an Ember setup, that could maybe take a few minutes. I mean, that's not that big of a deal, a few minutes. CHARLES: Right. WIL: But compared to 20 seconds. CHARLES: Right. you're still talking about an order of magnitude... WIL: Yeah, exactly. CHARLES: Difference. And using this, I think you could get a 30-minute test suite... WIL: Yeah. CHARLES: Down to the order of 3 minutes. WIL: Now when we're talking about those times, we're talking about the tests themselves. Of course, the CI would have to download stuff and set up the [inaudible]. CHARLES: Mmhmm, right, right. WIL: And that of course all adds to the time. CHARLES: Yeah, mmhmm, yeah. Earlier you mentioned, we talked about Ember CLI Mirage. This is actually something that is having now been using it for what, 2 years or something like that, it's just… it's impossible... WIL: To go back, yeah. CHARLES: To go back. It is. It's like [chuckles] you come outside the Ember community and you're like, “How is anybody ever dealing without this?” WIL: Yeah. CHARLES: [Laughs] WIL: A lot of the mocks are usually mocking the function that makes the request and it returns it in that function. That's what's out there currently, minus the Mirage stuff. CHARLES: Mmhmm. WIL: But once you use Mirage, you're mocking the requests themselves. CHARLES: Yeah. And you've got such great support for the whole factories. I love factories. It's something that is very prevalent in the Ruby community, and maybe not so much elsewhere. But the ability to very, very quickly crank out high-fidelity production data... WIL: Yeah. And you don't have to have files upon files of fixtures. CHARLES: Yeah, exactly. And you can change, if something about your schema changes, you can change the factory and now your test data is up and running. So, Ember has this tool called Mirage which is just, like I said, it's so fantastic. Oh yeah, it's also go support for running your application. Not just in your tests, but you can actually run your application with Mirage on and... WIL: Oh right, yeah. CHARLES: And you've got now the most incredible rapid prototyping tool. WIL: Yeah. You don't need to connect to a server to see fake data. CHARLES: Right, right. And we were even talking about this yesterday to a potential client. they're trying to, they've got to present something to investors. And how wonderful is it to just be like, “You know what? We just don't want to invest in all, we don't want to move the inertia, invest the money to generate the force to move the inertia of a backend.” Especially in this particular use case, the backend was going to be really, really heavy. WIL: Yeah. And there were some questions about the backend that we couldn't address quite yet but we wanted to start working on something that we could show, something demo-able. CHARLES: Right, exactly. And so, Mirage is just so wonderful for that. But again, Mirage is, it's an Ember-specific project. WIL: Right. CHARLES: So, the question was, “How are we going to use that?” WIL: And you actually took this on yourself. CHARLES: Mmhmm. WIL: I just saw this pop-up one day and boom, you converted Ember Mirage to vanilla JavaScript. [Chuckles] CHARLES: So, I did extract it. But the lion's share of the credit goes to the developers of Mirage themselves. Sam Selikoff and the Mirage community, they built Mirage not using much of Ember. There were some utilities that they were using, but mainly things like string helpers to convert between camel-case and dash-case, and using a Broccoli build, or using an Ember CLI build. WIL: Yeah. That was one of the challenges that we came across using Mirage outside of Ember, was how do we autoload this Mirage folder with all this Mirage config and Mirage factories and models, et cetera. CHARLES: Right. The internals were all just straight up JavaScript classes, for the most part. And so, extracting it, it was a lot of work. But 90% of the work was already done. It only took three or four days to do it. WIL: Amazing. CHARLES: Yeah. So, it was actually a really pleasant experience. I was able to swap out all of the Ember string helpers for Lodash. So now, it's good to go. It shares a Git history with Ember CLI Mirage, so it's basically a fork. WIL: Mmhmm. CHARLES: Like, a very heavily patched Ember CLI Mirage. But I keep it up-to-date so that it doesn't... WIL: Good. [Inaudible] CHARLES: Yeah, so I think the last time I merged in from master was about a month ago, something like that. Because it's got all the features that we need but it's not a big deal to rebase or just to merge it on over in. Because yeah, it's a really straightforward set of patches. WIL: Was there any talk with the creators of Ember Mirage about getting this upstream? CHARLES: So, I've talked a little bit with Sam about it. And from what I can tell, his feeling on it is like, “Hey, my goal right now is to focus on this being the best testing and data stubbing platform for Ember. Anything that happens out there, outside of that scope, that's great. And I certainly won't get in the way of it. But I'm pretty maxed out in terms of the open source credits that I have to spend.” WIL: [Chuckles] CHARLES: And there hasn't been much motion there. I'm happy where it is right now. I would like to see it merged into upstream. I think it would be great to have basically this Mirage Server and then have Ember CLI bindings for it. WIL: Yeah, yeah. I was going to say, either another Ember CLI specific package for Mirage or maybe to make it a non-breaking change or something. CHARLES: Yeah. WIL: Just like an Ember-specific entry point. CHARLES: Right, exactly. And I think that's definitely doable, if someone wants to take it on. I will say, we have been using this extracted plain vanilla JavaScript Mirage Server now for what, almost six months? WIL: Yeah. CHARLES: And it really hasn't... WIL: Yeah, I don't think I ran into one issue with that. CHARLES: Yeah. It's solid. It's really, really good. So, kudos to the Mirage team for doing that. And if anybody is interested in using Mirage in their projects, it's definitely there and we'll put it in the show notes. WIL: Yeah. We call it Mirage Server. CHARLES: Mirage Server, yeah. So, I don't know. Maybe it's time to reopen that conversation. But it has become a very integral and critical piece of the way that we test our JavaScript applications now. So, what are the foundations of it? We've got, we're using Mirage. We're using these convergent assertions. We're using Mocha, although that's really... WIL: Yeah, we have jQuery and Chai jQuery just to help us out with interacting with the browser as a user would. And I think one of the big challenges with that actually, I just remembered, was triggering changes in React. CHARLES: Yeah. WIL: I think this is pretty specific to React. You might run into problems with the view. I don't know how to mess with view. But in React, at least I think 15 or React 16, one of them, they changed the descriptor of the value property on an element so that they can appropriately interact with it, make changes, watch for changes, et cetera. So, when you set this value property using jQuery or just straight up ‘.value()', that change event isn't triggered in React. Your on-change handlers are never called. CHARLES: Wait, they actually update the JavaScript property descriptor of the DOM element. WIL: Yes. CHARLES: Boo. WIL: Yeah. So, there's a nice little helper out there called React Trigger Change. I dug through it and I've stripped some of it down to just be for more modern browsers, more modern React. But there's a lot of good code in there. And basically, it just caches that descriptor, updates the value, triggers the change, and then adds the descriptor back. And that ends up triggering that React element. CHARLES: Okay. [Laughs] WIL: [Chuckles] Yeah, so that calls your handlers and that's how we get around that. CHARLES: Right. There's a few fun little hacks there. But I think it is good to tie that into a larger point, is that the amount of touch that you have with the framework is actually very low. WIL: Yeah. CHARLES: So, the amount of affordances that we've had to make just for React, there's that, that you just mentioned. And is there… there's not much else. We had to write a test harness to mount the app. But that's like... WIL: Yeah, yeah. Our describe application helper is pretty React-specific. CHARLES: Right. WIL: So, you'd have to render it and set up a Mirage server, et cetera. CHARLES: Right. But that's application-specific setup. WIL: Yeah, that's one file. CHARLES: Right. WIL: So, your goal for acceptance tests is you want to be able to have a refactor and your acceptance tests still pass. CHARLES: Right. WIL: So, what if that refactor involves switching libraries? CHARLES: Right. WIL: If you're writing Ember acceptance tests, you're going to have to rewrite all your acceptance tests. CHARLES: Right. WIL: That's a huge downside. CHARLES: Right. WIL: So, with this method of interacting with the actual library very little, we have that one file that sets up our app and then we have that one trigger change helper, we remove those, we can use whatever framework we want underneath this. And our tests would still work. CHARLES: Yeah, exactly. And I think that we actually could theoretically, and honestly I have enough confidence in this style that we're developing the tests now, we could refactor this application to Ember and not have to rewrite our tests. WIL: Yeah, exactly. CHARLES: In fact, the tests would be an aide to do that. WIL: Yeah, and the tests would be faster than Ember testing with that run loop problem. CHARLES: Yeah, exactly. that's really something to think about or to think on, is like, “Wow. You're really at this point completely, not completely, but very loosely coupled to the actual internal library code.” Which is one of the goals of a nice, big acceptance test, is to be able to make major changes, break big bones, and be able to set them and have your acceptance test suite be the bulwark that holds it all together. WIL: Yeah. CHARLES: So, I actually don't know what a bulwark is. WIL: [Laughs] CHARLES: I just know that it's a really strong thing. [Laughter] CHARLES: Maybe we could put that in the show notes. [Laughter] WIL: A link to what that is. CHARLES: [Laughs] So, alright. Well, I'm trying to think if there is anything else that we wanted to mention. Any challenges? Any next things? WIL: So, one of our next steps is something we mentioned that Cypress does, is they wait for elements to exist before they interact with them. And we're actually not doing that in our app currently. And we don't have helpers out there for it yet. CHARLES: Right. WIL: But that's very much the next step. When we go to click an element in our ‘before each', we have these describes that are nested. Say, you have nested describes and you get down three levels into a ‘before each' when you're clicking a button. That button might not exist yet. CHARLES: Right. WIL: And especially since we're using jQuery, if you trigger a change on an empty jQuery element, it's not going to throw an error. It's just not going to tell you that it triggered anything. CHARLES: Right. WIL: So, we get those skips where that button's not getting clicked and we should really be waiting for that button to exist. CHARLES: Right. So, what we've done right now is we're converging on our assertions at the backend of a test. But at the frontend of a test we need to also be converging at some state before we can actually interact with the application. WIL: Right, yeah. CHARLES: So yeah, so that part is missing. And that actually brings up, we are very slowly but nevertheless doing, we're collecting these convergent assertions and convergent helpers in a repository on our GitHub account. We're going to be adding these things so that you can either use them out of the box or use them to make your own testing library. WIL: Yeah. And one of the other next steps that goes along with waiting for the element to exist is when you need to chain convergences. Like, wait for this element to exist and then click it and then wait for this thing to happen before actually running a test. And that presents the problem of our convergences are waiting for that timeout and those timeouts will accumulate. So if you have three chained convergences, that's now a six thousand millisecond timeout as opposed to a two thousand millisecond timeout. CHARLES: Right. WIL: So, one of the next steps is getting that tracking under control so if you chain three convergences together, they're smart about it and they still fail under the two thousand millisecond timeout. CHARLES: Right, right. So yeah, so we're going to be collecting all this stuff that we're learning into some publicly available code. We have a repository set up. I don't know if I want to announce it just yet, because it's really early days. WIL: Yeah. CHARLES: But that definitely is the plan. And that way, whether you're using Mocha or whether you're using QUnit or whether you're using Chai or jQuery, you've got these underlying primitives that help you converge on a state, whether that state is to interact with some piece of the DOM or to just assert some observation is made about that state. We'll be continuing on that. But by all means, get in touch if this is something that is of interest to you. Let's make something happen, because it's something that we're pretty excited about. And honestly, it's pretty comfortable living inside the four walls of this test suite. WIL: Yeah. CHARLES: It feels pretty good. WIL: It does, yeah. They're very fast. And some places in the test might need a little reworking, but for the most part all of our tests are very well-written, very well-readable. And you can just open up a test and know exactly what's going on. CHARLES: Yeah. Alright. Well, I think that about does it for Episode 90. Wow, Episode 90. WIL: Man, coming up on that 100. CHARLES: Yeah. We're going to have to have a birthday cake or something. WIL: Do we celebrate Episode 100 or Episode 104? CHARLES: What's 104? WIL: 104 would be 2 years. CHARLES: Oh really? WIL: Well, I mean 2 years' worth of podcasts. CHARLES: Oh, right. 2 years' worth of podcasts. Yeah. WIL: Yeah, like if you go every week. CHARLES: Maybe we should celebrate a hundred hours or something like that. WIL: Oh yeah. CHARLES: We can add up the thing or celebrate… I don't know, be like, “You've literally wasted 2 years of your life.” WIL: [Laughs] CHARLES: “2 weeks of your life listening to the podcast.” Anyway, so that's it for Episode 90. and thank you so much, Wil. WIL: Thanks for having me. CHARLES: It's always a pleasure to talk about these topics with you. And as always, if you need to get in touch with us, please reach out to us on Twitter. We are @TheFrontside. Or you can send an email to contact@frontside.io.

Toran Billups: @toranb | GitHub | Blog Toran Billips joined us for an insightful conversation regarding glimmer-redux: Predictable state management for Glimmer apps. Resources: Glimmer Redux Demystified Talk from Tom Dale on glimmer internals (contrast with Preact made in this talk) ember-redux Glimmer progress report that mentions the migration to Glimmer 0.8 (Big Changes) Blog post following EmberConf 2017 that announced GlimmerJS (for the Ember dev) The Frontside Podcast 086: Routing in Ember with Alex Matchneer An ember-rideshare Blog Post A Rollup plugin for glimmer-redux RollupJS Transcript ELRICK: Hello and welcome to another Frontside Podcast, Episode 89. My name is Elrick Ryan, a developer here at the Frontside. I'm joined by Wil Wilsman, another developer here at the Frontside. Wil, how are you doing? WIL: I'm good. How are you? ELRICK: I'm great, man. I'm excited for this podcast that we have coming up here. Today we are fortunate to have with us a podcast elite member now, Mr Toran Billups. Toran, how are you doing? TORAN: Oh, man. I joined the elite platinum club or something? ELRICK: Yes, you are in the platinum club right now. I think this is probably what? Your third or fourth episode by now? TORAN: Oh, yeah. I think the fourth. ELRICK: Oh, yeah. You're in the elite club right now. You are a Midwest programmer and I hear there is a difference between a Silicon Valley programmer and a Midwest programmer. Could you tell us about what the difference is? Because it's the first time I've heard anything about this. TORAN: Admittedly, I stole this from a very popular developer, Justin Searls who spoke at length one time on a podcast, not too different in this one about his experiences in consulting for companies, who are more in the startup phase or a company that you'll find in Silicon Valley that is mostly just trying to test an idea and get to market, versus his experience for finance or insurance companies based out of the Midwest. I like that idea because my experiences have taught me. I'm a little bit happier when I'm working for companies that are interested in quality or attributes of quality and view the software longevity as mission-critical versus a software that is really just a byproduct of an interesting idea and if we validate that idea in a market, we can always rewrite the software later. Midwest, I guess the short version is we care about the work we're doing and we understand that rewrites are difficult, if ever possible. ELRICK: Interesting, so the Midwest seems to be concerned with long term goals. TORAN: Yeah, I think sustainable -- ELRICK: Sustainable software, at least. TORAN: Yep. ELRICK: Today, you are joining us to not only talk about the Midwest and the beautiful Midwest programmers. You're here to talk about Glimmer Redux. TORAN: Glimmer Redux is a little library I wrote, I think last month. I should start off by asking you guys if you're familiar with a Glimmer JS or if you've heard of that. ELRICK: I've heard of Glimmer JS. I haven't had an opportunity to play around or mess around with it yet. I don't know if that's good or bad because I'm just really busy but I really want to get into it. Wil, what about yourself? WIL: I've read through the docs but I haven't played with it at all. It looks really nice. TORAN: I think the joke that was off the air last time I was on, Wil you might remember this. You're on that podcast with Charles. I said something like, "I'm not young enough to actually be working with Glimmer," and I felt that way for a long time because one thing you should know is it's a pre-1.0 and if you guys have ever worked in a pre-1.0 ecosystem, myself the biggest experience I have to draw from is really pre-1.0 Ember and there were some big changes before 1.0. You can imagine back to that throwaway comment about being very young, there was actually a big change in Glimmer itself recently where they decided to... I don't know if the right word is Pascal Case but they've literally gone away from that 'dasharized' components. It used to have 'foo-bar' and in your template, you would actually see lower case of 'foo-bar' and now that would just be all uppercase. Well, not all uppercase but 'FooBar' and no dash, which is a big change recently. WIL: So a class case, kind of like React or JSX. TORAN: Yeah, exactly. They have a great blog post. Actually, we can reference that in the show notes, about some of these big changes in that release. It was Glimmer 0.8 so it's still, it's making its way to the 1.0 but I got interested in this really for two reasons in the last couple of months. The first was, if you actually go build something with Glimmer -- and this is my experience -- is for the novice programmer just taking a look at it, it's really just a way to use web components to build an application. There's no routing. There's no opinions, really. There's no services like you have in Ember or contexts like you have in React. The first challenge you run up against is when you get beyond a single component or two components and suddenly, you need to share some state across this application. How do you do that? If you guys have some experience, I know the Frontside, with React, if you're not using MobX or Redux or something like that, a lot of times you'll see this pattern where you're actually passing a piece of state through the entire tree or the shared state through a big part of the component tree. Of course, that becomes painful as the application gets to a certain size. One of the things I thought about is if I was to build a real application, the one in mind that was certainly not built yet because I'm not using Glimmer at work but I always think about the ember-rideshare. I know you guys had Alex Matchneer, recently talking about routing and Alex mentions in that episode this challenge for the Ember router today, to be reactive to server-sent events. In the case, imagine you have an Uber or a Lyft app and after the ride is over, the server wants to send an event, maybe and then the app needs to react to that event -- sending you maybe to a new route or sending you back to the map to pick a new ride. The gist of the ride share app is, and Alex, of course I would reference anyone to that podcast, he does a lot better job describing the routing challenges and those are a little bit out of scope for this discussion, but imagine you're going to build an app that ambitious and you're going to build a Lyft in Glimmer. What I found was missing is really what we take for granted in Ember, which is the Ember Service and that is like a singleton or an object that allows you to have a piece of state and then share that state around by injecting that service in only the components that need to reach up and grab that shared state. Redux, which I know your audience is pretty familiar with but a quick recap, Redux is just kind of a global JavaScript object that has state and there's often a library that lets you connect to that state so you can use it in your various components. Glimmer Redux is no different than that, actually. It just allows you, instead of having to create maybe one global JavaScript object and kind of pass it down the hierarchy, you can instead just connect the components that need to be aware of Redux. The ones that don't, of course they just don't connect. ELRICK: I know that you had a hand or build ember-redux and now you build Glimmer Redux. Were there any challenges between building those two different add-ons into two different ecosystems? TORAN: I should take one minor step back because that question is a great segue. I didn't want to touch on the second motivation for Glimmer Redux, which is actually really closely related here and that is, of course I did write ember-redux so with every open source project, there's a little selfish motivation here. I imagine last year when Glimmer was announced at EmberConf that there would be a story from Glimmer to Ember. The idea being you're a small startup, you just want to get your web application going, you don't necessarily like all the big conventions or you just don't think you need all of Ember when you get started but six months down the road, you're suddenly looking at tree shaking and lazy loading with engines and you're thinking, "I wish we had that." Realistically speaking, like today what would a transition for a Glimmer shop to Ember be. Honestly, I think it's tough without a library like Glimmer Redux. Of course, I wrote this with pretty much a mere of the connect API. If people were to check out the ReadMe of Glimmer Redux and ember-redux and you looked at a connected or redux-aware component in both of those cases, the best case scenario is the only difference would be in the import. Instead of import connect from a Glimmer Redux, you would be import connect from ember-redux. Everything else underneath, all the ecosystem of Redux that you can use and both are completely compatible. In fact, if you were to move, imagine you Ember new and you're thinking, "I got to move my Glimmer ride-share to ember-rideshare. Since Redux does a good job in encapsulating all of the state transformations in vanilla JavaScript, you don't have to really worry about the differences between a number object and not having Ember object. After you did Ember new, essentially you would copy over your components. For the most part, they're still template-driven. A lot of handlebars and a lot of TypeScript to JavaScript is the biggest mismatch you would have between Glimmer and coming over to Ember, of course but there is Ember CLI TypeScript or Ember TypeScript, I think. Long story short, you essentially copy the directories of your reducers or middleware from your Glimmer app over to Ember and there should be no changes necessary at all. ELRICK: I know that is the dream that you touched on, I think they phrase it as 'NPM installing your way up to Ember.' In your perspective, do you think that is going to be a thing? Is it going to get there? What are your feelings on that? TORAN: I would definitely be in trouble if I didn't say upfront that I'm not on the Ember core team. Sometimes, people get that confused for some reason but I don't speak for the core team and I'm not really privy to anything. But I do think that the core team has this in mind that there will be a set of NPM installable modules that eventually land you the full set of tools and abstractions that we see in Ember today. A big one that I'd hit on earlier is services. What will services or the Glimmer 0.5 version of services look like when it lands and what about routing and those sort of things? This was really my personal take on how could I make that migration right now without asking permission from the core team. In a Glimmer Redux, I think honestly it offers a good 80% of that. You still have the routing issue, which is a little challenging and you have the TypeScript issue, which you just have to be aware of some of the limitations of using TypeScript in Ember. ELRICK: That's an excellent point that you made because when people outside of the core team take it upon themselves to then try to implement different things around the ecosystem, that can then be motivation or an example for people outside of the core team to see like, "This is a possible solution to a goal we're trying to reach." Kudos to you. TORAN: Back to your original question about 15 minutes ago, what was different about the ecosystems writing ember-redux the add-on and Glimmer Redux, which is... I don't really even want to call it an add-on. I kind of label it as a Glimmer library but to your point just a second ago, when I got interested and saying, "I wrote Glimmer Redux, now I want to share it so other Glimmer authors don't need to copy/paste this file," and the first resistance I hit up on is there really is no Glimmer library or Glimmer add-on. You could write an Ember add-on because if you guys get into Glimmer, you'll find this as well. There are certain hooks that are used in the Glimmer build process where Ember add-ons like Ember CLI SASS can be used from Glimmer. But the challenge I had using an Ember add-on, of course is that this wasn't an Ember component or anything Ember related. It needed to really be test driven from a Glimmer apps. Really, if you went to NPM installers, what you're pulling in is effectively my Glimmer app that also exports publicly this connect function, which is not necessarily you're leading, maybe anything to core team that could happen. A big reason for that as well and one of the challenges building this, is that Glimmer right now, after to try to emulate my 'quasi-success' in doing this, is really bring your own build system, to actually share a Glimmer components or internals like this connect API that Glimmer components can use. What I mean by that is on the website, one of the challenges I faced was -- this isn't a knock against the core team, this is just my honest experiences -- when I read the Glimmer JS docs and it says right in the installing guide, "Glimmer uses Ember CLI, this battle-tested command line interface from the Ember project." Now, pause right there because again, not to beat up on the Ember core team or anything but assuming in that one sentence that they're using Ember app, what I noticed when I opened the Ember CLI build file is the project was actually a Glimmer app. Now, I did a little bit of digging here and I think this is validated, at least back when I worked on Glimmer Redux last month, that Glimmer app is not like inheriting or pulling in a bunch of shared code from Ember app. It's actually a completely separate build tool. As part of this process, I actually for the first time had to go through and learn Rollup and understand how the Broccoli process is kicked off, how both Rollup and Babel are used to build this and then how to apply some convention. If you guys are familiar with Ember, you have this add-on directory and an app directory. The guidance from the Ember team is around how to structure add-ons. Of course, you write all of your kind of private-ish or add-on code in the add-on directory and then whatever you think will be public, you export from the app directory and that sort of merges it into the tree when the application is built. People are allowed to use your code but then also, they can override that code. One of the challenges here is if I wanted that exact same API and I want people to make this migration from Glimmer to Ember using Redux, I had to actually invent that convention so I wrote a Rollup plugin and it's all listed in the docs here. One of the strange things people who are checking out Glimmer Redux hit me with first is, "I see a Yarn installed Glimmer Redux but then second step is installing this Rollup plugin. What's the deal?" I think it is because most people assume the Ember CLI you're using is identical and somehow, I should have written an Ember CLI add-on for this. I think that was the biggest learning curve and people should just be aware of that. If you're interested in sharing code right now, there is really not a baked story and that's okay. It allows people to innovate. Of course, the innovator's dilemma being that, I don't really know without [inaudible] some migrating RFC or getting involved with the core team, how to make that thing. I ultimately just hope the core team improves this and I'm sure they will but for right now, I don't really want to wait around for it. ELRICK: Got you. What is Rollup, for people that are not familiar with what Rollup is? I'm sure everyone is probably familiar with Babel by now because it's used everywhere but what is Rollup? TORAN: Embarrassingly, I don't know the technical... But I would say the role that you can see, if you actually step through the node process as your Glimmer app is being built, is it helps really condense the overall build size. What I see is it's essentially traversing like Browserify in some ways. This is, again just my primitive look but it traverses all the imports you have and it tries to pull in the bare minimum to keep the bundle size as small as possible. ELRICK: Toran, you have had a lot of experience with Redux and Redux is now being used in several different software platforms, I guess or software areas like Vue and they probably even have in Angular now. They have it in Ember and React. Redux is kind of spreading its wings and it seeds across every ecosystem. Do you feel that Redux has reached a state where people are just satisfied with the current state of Redux or do you think that people are going to be then, looking to build another abstraction on top of Redux? Do you have any thoughts on that? TORAN: The fact that Redux is so simple has allowed it to become so ubiquitous. I heard someone say this term the other today, which is like, 'ubiquity over consistency,' and that I think describes the both the growth of Redux and why it is kind of de facto for data management across all ecosystems. I think there's two camps that I hear about and I'm curious if you guys see this in your consulting work but there is certainly, the developer see this ubiquity but no consistency and see chaos in their experiences. I can totally relate to this. There are development shops I've seen where one team goes this direction because there's no strict guidance goes another and then when those teams meet up for a project in 12 months, they look at each other and the apps are, of course nothing alike, which is a big problem that Ember tries to solve. My biggest question here really is kind of curving us slightly back to the Glimmer story. If I can reframe your question, Ember is traditionally very big on convention and I think a lot of the community that is still in Ember today in large part is because of this convention, these guide rails about the community has set up but Glimmer being this NPM install your way to Ember, I think along the way, there's going to be either a new set of users that are coming just for the winds of the Glimmer VM and they happen to find themselves, not necessarily in love with some restrictions or opinions that would come with a migration directly to Ember and I'm curious if the Glimmer community that will show up for that is mostly Ember backed, meaning that they want to slowly build with RFC as a process that the entire community uses and there's one solution like we end up with often an Ember. If a community evolves more experimentally like you saw in React, where there was, of course Redux but then there was MobX and then there was various little wares for Redux that different teams would try out and eventually, there was no one proven way but there was always at the heart of it, Redux with some extensions or other add-ons around it that got teams to where they are today. I'm curious to see where this Redux, especially with the Glimmer influence, will end up. WIL: You touched on a little, I think one of the, maybe not a problem necessarily but one of the biggest barriers in Redux and React and maybe Glimmer -- I'm not sure -- is that it's almost too loose without any opinions at all so it kind of gives developers the freedom to mess things up big time. What's your opinion on that with Glimmer, like Glimmer is headed toward this NPM install? Is it too loose? TORAN: I guess that's the question, I wish we both had the answer to. It's funny. It's a double-edged sword. If it's loose, it seems like somebody is going to go create a mess but at the same time, if it's loose on the surface, it often seems like it has less surface area and as a result, a lower learning curve. React is a good example where most people, I think have gone to React or at least in my experience, I like React because it was so simple and there wasn't a huge amount of things to learn. There wasn't this full ecosystem, at least out of the gate but of course, what you find the moment you want to join a team or go build something ambitious is that you've got to make a bunch of decisions and that is certainly the calling card of Ember. What makes Ember special is they've settled on a handful of those decisions. I think ideally, I'd like to see the community in Glimmer check out Redux, get an idea of what problem it actually solves for them and if it is useful, then find a set of middleware or extensions from the wider ecosystem that actually solve the problems they face. Back to this Glimmer rideshare example, I think one thing that stands in the way as I play around with that is just a very basic routing story. Even if that is as simple as I have a component that I'll just call it route, what hooks in this Glimmer component are correct to fetch data. In the Ember ecosystem, we have a very special route object, essentially who has a set of hooks that are known for handling the asynchronous stuff in our Ember apps but there isn't anything like that yet, in Glimmer which is the next stumbling block for me to actually go build something big. I'm kind of messing around with the idea of this reactive router built out of Glimmer components but until that's actually kind of surface, mostly just spit balling here. ELRICK: With flexibility comes a lot of power but then there's also the case where our flexibility could lead to chaos. But being that something as flexible, it allows you to adapt it to whatever your particular needs are -- you know, the 'special snowflake' as everyone ends up being. Because Ember has been around for a while now and has proven itself and its battle-tested in a lot of areas, now as NPM install in our way from Glimmer up to Ember, do you think that they'll be able to then, extract some of those hardened pieces of Ember out of Ember and then give us a solution inside of the Glimmer ecosystem based off of the Ember ecosystem? Then not have so many varying different opinions or different packages or add-ons or libraries that you may need to pull from that you may just have like a set of Glimmer-approved libraries or add-ons to use to then, get your way up to this full Ember app. Do you think that that's a road they're taking? Or a possible road? TORAN: Yeah, I think for sure. A lot of the talk right now if you're in the Glimmer channel on Slack in the Ember community, I think the next big step here is having the ability to take a Glimmer component as it exists today and use that, extrapolate from there and the plan is to prove out things in a more experimental ecosystem as pre-1.0 Glimmer ecosystem. As they become more solid, we essentially just adopt them and then use them as a first class. In Ember actually, this isn't true but I envision a world where in the months ahead, we're essentially using Glimmer components in Ember so I'm already challenging myself with how would a library or add-on author help people make this progression from Glimmer to Ember if they don't do something like I've done here with Glimmer Redux because eventually, there will be this shared component world, at least in the middle ground, where Ember has both Ember components and Glimmer components. I think it's a road yet to be traveled and that's what I'm excited to be on the podcast and get people talking about building libraries for Glimmer, just because there is not a cow path or a paved road for you right now. I think if you learn just a little bit of Rollup or you dive in and take a look at the DI system built in a Glimmer right now, there is a lot you can actually do with it. I know there are certainly big named add-on authors already checking it out in preparation for such a migration. WIL: Besides the whole Rollup process, was there any other friction points in integrating Redux with Glimmer? TORAN: One that I want to call out but it is, I believe still Rollup related. It's mostly a PSA, to warn people. If you are building one of these little libraries like I talked about with Glimmer and you're going to do a Yarn link, which means that you're going to just work locally and not really publish to NPM until you're done, be aware that if you're Yarn linking and then going over to another project to test this Glimmer library, then you'll actually get temporarily two copies of Glimmer. In fact, that is how this Rollup plugin I wrote -- sort of 'bring your own build' for Glimmer -- became essential as I was like, "I'm getting two copies of the Glimmer component," so what was happening is example, I was playing with this connect API, it was always calling into the wrong Glimmer code so the code that I was expecting in my add-on was never firing. We come to find out when you're not doing Yarn link or you're not working locally, this isn't a problem. Actually, Tom Dale reached out and helped me a little bit later because my first version of the Rollup plugin had this little hack in there that said essentially, "I'm going to mask away this duplicate Glimmer problem," and it turns out it's not a problem. If you are working locally, be aware that you will have this problem as my guess. ELRICK: What gets you most excited using Glimmer? Are there any specific features or things within Glimmer to get you most excited? I guess the second question would be, what do you think Glimmer is going to unlock for the future of app development in Ember? TORAN: I think the first thing I get excited about was visible in this talk that Tom Dale gave a couple weeks ago. I'll try and dig that up for the show notes where he show the advantages of Glimmer over the competition today. The big thing that just blew me away was some of the advantages over, even Preact, which I was kind of surprised that a lot of times there's this rivalry for performance especially between React and Angular and Ember but no one ever really talks about Preact, which is known in a lot of ways as the thinner, lighter-weight React, if I'm not completely wrong. I'm sure somebody is going to be table flip on that definition. But my view was that if you were really performance-hungry, you could check out Preact, which was for performance reasons there was a smaller bundle size so you're just actually shipping less JavaScript, which means they're parsing less JavaScript and Tom went directly after this library in his talk and just showed a very interesting point at the end. I don't want to spoil it for people but let's just say, it appears to show Glimmer as just an order of magnitude better as a primitive for building web components. I think that is the big draw and how will that make Ember better. I think, mostly in the same way that I just described, where we'll essentially get a component for free in Ember that is just a better performing primitive for the web. ELRICK: I'm not too familiar with the guts of Glimmer but from what I understand, Glimmer compiles down and it has some opcodes that then compile down to binary. Is that correct? TORAN: Yeah, I think there is a binary format that they're shipping or they will soon be shipping. If you're really interested in the technical details of that, I'll definitely be sure you check out this talk from Tom because now the real magic behind this is they essentially boil it down to the ability to compile these templates down to 'byte code,' as they call it and Tom has a really funny part in his talk where he says, "You know, it's not a marketing term, this byte code word," and it becomes true because later in the talk, you hear that, essentially the Glimmer VM or the big value add of Glimmers VM is that you're just feeding it with byte code until the next 16 millisecond buffer comes up to paint, in which case you just pause and that allows, I think as he describes, less jank or allows less freezing up that main thread because you're releasing control back to the UI every 16 milliseconds, essentially. ELRICK: Yesterday, there was a meet up with a lot of the Ember core team. Ed Faulkner had made a point that since Glimmer compiled down to byte code that then is not too far of a stretch to then use that with something like Web Assembly, that with then give Glimmer an extreme performance boost. He sets up Glimmer to be used in what can be potentially the future of the web, which is Web Assembly, which I thought was really interesting and it kind of blew my mind to think, "Oh, yeah. That is true since it compiled down to byte code and Web Assembly compiles down to that." We can get that performance boost in that Glimmer is forward thinking in a sense. TORAN: Yeah, man. I totally agree. One of the things I honestly value about the Ember core team is they're very both tactical and visionary at the same time, where in the short term, they're doing things to accomplish real pain points we have but without anyone really realizing it, they're also setting up the stage to solve huge problems that we're all going to face in six, 12, 18 months. I definitely appreciate and love the work these guys are doing. They should hear about it. Obviously, this is all open source and most of this time is gifted, just given away so I really appreciate the core team. WIL: Speaking of performance, we know that the Ember has a run loop and basically, most of these libraries have some sort of loop that determines when they should batch things together or render things to the screen. Does the Glimmer have this concept? What do they take advantage of to make it as performant as you say? TORAN: Yeah, that's actually a really good question and I'm probably minimally equipped to answer it but the short version of it is there is no equivalent run loop for batching work that I've seen inside of Glimmer. Instead, you're more directly interacting with request animation frame, which we miss directly from Mozilla here but requests animation frame tells the browser you [inaudible] browser call specific function to update an animation before the next repaint. Where does this come in for Glimmer? Essentially when you call set or you decide to change a property that Glimmer is listening to and if anyone was not familiar with Glimmer, you designate this by using the tracked attribute. The tracked attribute, when you change a value, it fires this 'set property did change' and behind the scenes, 'set property did change,' if you are familiar with Ember, in my mind is close to 'notify property change,' which is what happens when you do Ember.set. If you've ever actually change something in Ember behind the scenes, there is a notify property change event fired and then we queue that work and it's a similar-ish process except that there is no run loop. What we do is we just call schedule rerender, I think in Glimmer and that just fires off request animation frame to try and rerender within that 16 millisecond window before the next repaint. WIL: From my understanding, the request animation frame is Glimmer's run loop essentially. TORAN: I think I saw actually a discussion between someone at Ember core kind of saying in the public channel that, if you're using Embers run loop, the equivalent-ish today would be request animation frame but the point came up that there's really no way to have a different set of cues because Ember itself has many different cues in the run loop and request animation frame, as far as I know really is just one function with a single callback, where you try and fit in as much work before that repaint as you can. I don't think there's prioritization that you would get in the Ember run loop. But at the same time, I'm not sure if that's actually a requirement. I don't think request animation frame was as mature as it is today back when backburner, which is behind the scenes of what Ember run loop is using, was built years ago. ELRICK: Since Glimmer is using requests animation frame and not the Ember run loop, is it going to continue to use requests animation frame in the future or they're going to develop like a run loop equivalency for Glimmer? TORAN: That's definitely out of my depth. I know from looking inside the code, the rerender work essentially calls like a 'begin,' to say begin doing your work, which I believe is like the reconciliation type work if you have a React background, I believe. I don't know what decides to end that. If the request animation frame is truly saying, "We're at the 16 millisecond budget and we're going to quit feeding the Glimmer VM byte code instructions now because we need to go back and paint." I would guess that that is the high level narrative but I actually don't know the implementation details. ELRICK: Since Glimmer is pre-1.0, it's a place for you to experiment, try out new things, really exciting area to play around in. What kind applications do you see people building today using Glimmer and what's the good application that's a good fit for Glimmer right now for you to experiment with? TORAN: The big application that I've seen that exists is being built with Glimmer in its current form is the Glimmer Playground. The Glimmer Playground is an area to go mess around with Glimmer, if you've never used it or you just want to go [inaudible] with the basics. As far as what is an ideal application, honestly I think we're at a point where we need to push the bounds of what a purely component based library can do. I think if you could come up with some kind of basic routing story and have a mechanism to share state and bubble events, whether that's Redux or some kind of home grown service layer at some point. That would allow you, in my opinion to build just about anything. The only caveat that you're going to be missing is a ton of opinions and you're going to be paving new grounds so just be aware that happy path for any pre-1.0 is be aware that they could change anything, anytime. But that shouldn't really restrict you from making a hobby project out of it. Back to your original question, I would say building any app that you're okay to go back and rework, not necessarily reinventing Facebook or something like that with it at this moment but at the same time, it would be great if someone did in a hobby way because we need to see some of the constraints and challenges. I got playing around with it myself and noticing if I'm going to build anything with Glimmer, I've got to have a way to share state and bubble events up to the single atom at the top that allows me to share all state. I think without some experimentation, we just won't know what apps are possible. ELRICK: You've been talking a lot about Glimmer today and using Glimmer. Since Glimmer is pre-1.0, are there any limitations that people should be aware of when they're going to be going into building a new Glimmer app? TORAN: For Glimmer Redux specifically, one of the challenges that people would see and they should know about if they're going to use this little Glimmer library, is that it doesn't yet allow you to write reducers in TypeScript, which would be a little counterintuitive because if you get into Glimmer, you'll notice the big differences -- everything is in TypeScript and not JavaScript. Luckily though, I think this is really just a build tool decision. Truthfully, the spike version of this that I have where you can use TypeScript, it doesn't require any change to the Glimmer Redux library at all. In fact it's completely unchanged. The difference here is that the Rollup plugin I use needs to see a change. It's kind of weird in that way, back to my point earlier where you kind of bring your own build chain and one of the things I don't like right now, which is the reason I haven't published this TypeScript version of it is that I'm actually doing a TypeScript compile of all the reducers or middleware in the Rollup plugin. With the mass confusion right now between Broccoli, Rollup and Babel, I just don't feel really great about it. Mostly because I have not truly been in the guts of the Glimmer app build tool or the application pipeline yet and I want to be a bit more educated there before ship something, back to being a Midwest developer here. I just don't feel good about shipping something and say, "Oops, we got it wrong." I take a lot of time and I also take a lot of pride in what I am shipping so I want to have a really good story about TypeScript. It does make for a little bit of a weird experience: bouncing between Glimmer components written TypeScript and flipping back to reducer file written in JavaScript. just be aware of it and at the same time, I didn't want to completely halt shipping it because again, I think we need to actually build apps with this and a concession here being that, of course you have to write reducers in JavaScript was still enough for me personally to get some value out of building Glimmer apps and honestly, it got me building them sooner. WIL: Is there a way to use Glimmer Redux without the Rollup? Can we import something into our Glimmer app to use it or this Rollup is required? TORAN: Yeah, that's a great point. You can omit the Rollup plugin entirely. What that results in is you'll have to do some hand wiring yourself. One of the upsides or one of the benefits of this Rollup plugin that I wrote is this conventionally provides a store and redux-thunk, kind of like a happy path for people who are just not familiar with wiring up their own Redux store. If you forego this plugin, you just have to do that yourself. You may actually have to do some kind of Rollup hacking in your Glimmer app, which is the thing I want to avoid. The one in particular that I know you'd have to do is there is a Node ENV that is looking for a production setting in Redux so the first thing you have to do is use a Rollup replace plugin to replace Node ENV with Ember ENV. If you can't do that, you actually get an error in just trying to stand up your Glimmer app with Redux. ELRICK: Toran, are you giving any talks or have any books or anything that you want to get out there and talk about? TORAN: I'm not actually on speaking circuit right now. I am certainly, probably like you guys are, thrown a talk or two together for the EmberConf proposals that are now out. I think they're open until November 21st. If anyone is thinking about submitting a talk to EmberConf, this should be in next March. Now is the time to get those in and I certainly have one out there but I've got one, off the top of my head that I would certainly like to find some time and submit that's related to Glimmer. ELRICK: Cool. We had a wonderful podcast today. We touched on Glimmer Redux and Glimmer and I want to thank Toran for coming on. Thank you, Toran. TORAN: Thanks for having me, guys. The fifth time I have to be on, I don't know if that will be in 2017, though. ELRICK: Yeah, we're going to bring you back for a fifth time and I would also like to thank Wil for coming on the podcast as well. Wil, thank you. WIL: Thanks for having us, Elrick. ELRICK: Anytime. Toran, if people want to reach you, is there a particular place on Twitter or anything that people can reach out to you or email or anything? TORAN: Yeah, at GitHub, I got my email out there but also on Twitter, of course. You can reply me there. If you have a question specifically about Glimmer Redux, of course you can got to GitHub and throw an issue up there or hit me in the Redux channel on the Ember Community Slack. ELRICK: Thank you all once again for listening. This is the Frontside signing off and if you want to reach out, you can always hit us up at the Frontside.io and we always want to hear about your new project that you're working on. Thank you for listening and that's peace from the Frontside. WIL: Everybody have a good Thanksgiving!

Kaylie Kwon: kaylieEB | kayliekwon@gmail.com Show Notes: 02:14 - Kaylie's Journey Into Software Development 09:25 - Implementing a Design System and Attacking Higher-level Workflows 15:43 - EDS Collaboration and Public Availability 19:07 - Getting Involved with The Yarn Project 20:57 - Selective Resolution 23:37 - The Warmth of the Yarn Community 27:11 - Handling Issue Communication and Tracking Resources: Eventbrite britecharts Eventbrite Spectrum Transcript: CHARLES: Hello everybody and welcome to The Frontside Podcast. My name is Charles Lowell, I'm a software developer here at the Frontside and your podcast host-in-training. With me today is Elrick Ryan. Hello Elrick. ELRICK: Hey, what's up Charles? CHARLES: Not much. Are you enjoying your morning so far? ELRICK: Yeah, my morning is going well. Everything is good. CHARLES: Lots of cups of coffee? ELRICK: No cups of coffee yet. I've been drinking a lot of green tea. CHARLES: I've actually heard that's really good for you. ELRICK: That's what I heard too. That's why I started drinking it. CHARLES: Did you continue because it tastes good or do you just live on the idea of how good it is for you? ELRICK: A little bit of both. It doesn't taste that great but it's not horrible. It's almost like an acquired taste and then when you add in, "This is good for me," then it tastes great. CHARLES: Okay. We got a nice [inaudible] there. ELRICK: Yes. CHARLES: Anyway, I guess we should, at some point get on to the main content of our podcast. We have a very interesting guest with us today who has her fingers in all kinds of pies that we were talking about just at the pre-show, just before we were recording so we're really, really happy to have Kaylie Kwon. Thank you for coming on the show and welcome. KAYLIE: Hi. Thank you for having me. CHARLES: It's going to be great. Now, you are a software engineer at Eventbrite, that's correct? KAYLIE: Yep. CHARLES: What kind of things do you do over there? KAYLIE: I used to work on part of the feature team that worked on their reserved seating product but not too long ago, I moved to our frontend platform team, which is a team that helps other frontend engineers move faster through things like working on infrastructure or Eventbrite design system and dev tools. CHARLES: So like getting into the tools that unlock the exponential productivity of the developer team? KAYLIE: Uhm-mm. CHARLES: We're going to dig into all of that because you just listed a bunch of really interesting stuff. I'm really excited to talk about the design systems, in particular but lots of different stuff. But before we do, I understand that you have a fairly unique way of entering in to the position that you're in now. Your journey didn't follow the traditional arc. Would you be willing to elaborate on that or tell us that story? KAYLIE: Yeah, totally. I graduated with a degree in art history, not related to computer science at all. Then right after, I moved to New York and worked for a small startup. It was marketing/business development role and I wasn't really happy with it. I was working on some design-y stuff on the side with HTML and CSS but I just felt like my brain just needed to be stimulated a little more. I applied to this all women's bootcamp program called Hackbright in San Francisco. It was a three-month intensive program and luckily coming out of that, I had at least some initial knowledge to get my foot in the door and Eventbrite was one of the hiring partners. They brought me in for interview. I actually had no idea that it was going to be a frontend engineering role because my bootcamp was totally in Python and it just had more with a backend focus. Ben, who spoke at React Boston Conference with me, was actually one of the interviewers and he gave me this Clojure problem and I solved it but in Python just using recursion. He was like, "You got it. Just convert it to JavaScript," and I was like, "No, it just can't be done. No JavaScript." But there's the reason he would chose to hire me and they onboarded me as an internal bootcamp within Eventbrite. The first three months I was there, I learned about React, Redux, JavaScript, ES6, all of that good stuff. Then they moved me to a feature team, where I continued, I guess working on the product and then I became involved more with open source projects. I really expressed interest in when you're a new engineer and coming onboard, there's all these assumed knowledge that isn't documented anywhere or something will be really obscure and hard to use but people will just assume that that's the status quo. This idea around developer experience and helping other developers move faster, it just kind of become a natural interest of mine. I was talking to the platform team, Ben in particular, expressed my interest in working in these areas. Just about like a month ago, we did a big re-org and I landed on the platform team. Currently, we have a lot of projects in flight. One of them being moving our dependency system from our old codebase, which first was written in RequireJS to webpack. We're building out our Eventbrite design system, which is basically a shared UI component library that other teams could leverage. Our platform team will just come in and make sure that the API is usable across different teams and maintain a consistent brand in terms of look and feel. We're also working on other tooling stuff like making sure we use Docker for our dev environment and making sure that the frontend containers don't break, making sure that everyone is on the same version of ESLint, Node and things like that. CHARLES: How do you make sure that the frontend containers don't break? KAYLIE: It's actually hard. I think one thing that we're trying to test right now is using Yarn Offline Mirror and having better caching. When you build a container, it'll look into the cache directory, which is just bunch of committed tarballs. That way, they don't have to fetch to the network each time because once the lockfile or package.JSON changes at our current state, it'll rebuild the entire container and it could take a very long time. We just have a lot of packages that we've added over time. Other things, we're experimenting along with our platform team on the backend side about having remote images. Instead of devs building their containers locally on their computer, having them remotely built by a CI system and then just pulling the container and the images down. CHARLES: Really, really you're like all over the place. That sounds so much fun. KAYLIE: Yeah, it's a lot of context switching. Sorry, if I was jumping too many topics at once. CHARLES: No, absolutely not. I think it's actually fascinating and it's actually capture the kind of scope because when you are doing development, if you yourself are not running up and down the stack, the tools that you use are. The better the tools, the better you're able to focus on the one little sliver of the stack that you're working on. If I don't have to worry about where are my Docker images are coming from or if there's a temporary network flip that I can't install my dependencies, if I don't have to worry about those things, then that makes me more effective so it's important to just kind of lay out all of the stuff that goes into making a quality developer experience. KAYLIE: Yeah, the dream is like frontend people wouldn't have to worry about any of the backend stuff and they just have this isolated environment, where they could just work on what they do best, which is JavaScript and writing features in React. Everything else just works and they could see that replicated through their dev environment as well as QA and Prod and hopefully, make every tooling that they use, like testing and linting all easily, intuitive and accessible. CHARLES: How is it that you're working up and down the stack, making sure that your CI systems, your Docker images but then also at the same time, working on a design system, which you've got collaboration, I assume with some pretty hard core devops teams but also then you're interfacing with designers. You kind of flesh out your design system. Are those the same people on the platform team? Or there are different groups within it? KAYLIE: We have designers and researchers actually, as part of our engineering unit. We work pretty closely with them to define guidelines on what the design system should look like because coming into making this designs system, one thing we really wanted to make sure that is that both sides of engineering and design have input, rather than the previous old version called Style Guide, which was more engineering-driven that not. One team would need a model so they would build a model and a different team would build a slightly different model and we would end up with five different models. They want to be maintained over time and there wasn't really a focus on accessibility or consistency of brand so the design system project was to eliminate all of those pain points. CHARLES: For people who may not be familiar with design system but it's something that certainly is cramping up more and more, what's the general strategy you take? Clearly, you talked about the kind of the pains that it solves, like I'm experiencing this pain where I've got five dialogues, I've got three ways of laying out forms, I've got these problems. What's the strategy that I go about as my organization for trying to implement this? Like now, I want to do a design system, where do I start? KAYLIE: One big thing, I think that helped us was developing Eventbrite design system as just the standalone product. You could run Eventbrite design system as a standalone with all of the documentation and components with each of their different props that could be configured and it has its own set of tests. All of the components are API-driven so there's nothing specific about business logic that it assumes. For example, our button component just assumes that I'll be receiving a type of submission and some [inaudible] body. It doesn't make any assumptions that it's going to be anything Eventbrite specific. It's high-level configurable that the end user wants it to be. Another thing that helped is we have a planned approach session right before we start working on a new component. A developer who would be building the component would meet with one of the frontend platform members and will be discussing the API of the component, the CSS that would go in it and the designer would do the final QA. The development takes longer than if you were to write just a component for your app but we're trying to build it out for a long term use. We have versioning for Eventbrite design system as a library so whenever you make updates, we added to the changelog and then it gets released and we bumped in Core, which is where all of our apps live and people have to get the new version. Basically, it's an orchestrated effort and we have a process built around scheduled releases and bumping it in Core. CHARLES: I get the wanting to have a uniform buttons, uniform labels, dialogs and things like that. How do you attack the problem of higher-level workflows like a form submission and validation process that might have a lot of different pathways? How do you guide that using a design system? KAYLIE: Those are actually really good questions because we went through issues like that. Validation form field or components that have logic around more complex inputs and validation is really hard because different teams use it in a slightly different way so we ended up having two versions of complex inputs. Now, we're in the process of deprecating the V1 and then having people move over to V2. We also use the Atomic Design Principles for Eventbrite design system. It means, things like atoms would be composed of buttons and we have molecules, which are slightly more complex. Then one level above that would be organisms, which are things like forms. Obviously, as you move up to like the higher level, it becomes more difficult for us to decide does this belong in EDS or does it belong with the app. There's lots of refactoring that sometimes ends up happening as a result of something built one way and us realizing later down the road that it wasn't actually very extensible and we just built it for this particular use case. Sometimes, it's the opposite where this was too generic. We can make it a little bit more specific and add more logic to it. It really depends on the component but that's been our strategy, just taking the components as needed. We're still in the earlier stages. We've released EDS a year ago and now, we're almost close to finishing it. We're just missing a handful of components. They'll be more [inaudible] that's available as opposed to adding a brand new component. CHARLES: Did you find guidance in existing design systems? One thing is why not just use one of the existing ones that's out there. Is it a matter of branding? Is it a matter of, "There are certain interactions which are unique to our product that we need to design system to capture them?" I guess my question is, if this is something I want to take on myself, what tools would I be able to reach for and what other design systems would I be able to look to versus what I have to contribute myself? What am I going to be looking for to share with the community and what am I going to look for that's develop that's uniquely mine? KAYLIE: I think consistency was a big issue. Sort of the genesis of idea came actually even before I joined the company but I think what they were going for was that we already had an existing component library called Style Guide and it wasn't really working out for us. To build like a common UI library was a natural assumption but making it better, making it more reusable. We looked at companies like Airbnb and Microsoft and I learned a lot from what they were doing. You definitely wanted some components that are specific to Eventbrite such as a ticket card or a media card that we use for our browse pages which would be needed for Eventbrite specific pages. I think mostly, the designers wanted more control because relying on third library means we don't always get what we want. We're actually thinking about making EDS open source at some point in the future, where it could take themes so other companies or individuals can leverage it but use their own theming scheme. If Spotify came and wanted to use EDS but using their colors and brand logo, then they could do so, just by layering a different configuration style to it. CHARLES: That's such an interesting idea. Is this something that you all have explored, maybe collaborating with another company? I'm trying to think what would be the benefit of making it publicly available, unless you would be getting lots of contributions back to improve EDS itself. Is that the idea? KAYLIE: Yeah. We're definitely set on making it public at some point in the future but making it open source is a different conversation because like you said, there's pros and cons that comes with open sourcing a package. We recently open source britecharts, which is a D3 library. It's been getting a lot of contributions. It's a good recruiting tool but it's also a lot of maintenance work outside of developers normal work hours. Also, we have to start worrying about like when we make a breaking change to a component, we're not just breaking it for our own product but for other developers. We currently have external developers. We have Eventbrite plugin system called Spectrum so developers are already building on top of that and they've been asking for something like this, where they could leverage and match the look and feel with the rest of the product. The downside is lots of maintenance hours and worrying about all the people that would be breaking the component for by just solving your use case. I feel like I didn't fully answer one of your questions, which was you have a different dynamic of people working on both really backend ops things, as well as a really frontend design system work. We definitely got very smart people on the team. Some people definitely have expertise in one area versus on others. Currently, we have five developers and some of us lean more towards the backend of the frontend work and I am one of those people working on things like Yarn and Docker and build system work but it's funny because I was thinking if I had a portfolio then, there wouldn't be any visual components to it, even though I'm a frontend developer. It would just be like terminal screens. We try to divide the work but everyone tries to, I guess develop, at least some shared knowledge around why we make the decisions that we do. CHARLES: It's interesting how they all intersect. I feel like the trend of the last 10 years has been to dev of all the things. I think the first thing was this artificial divide between developers and testing and that came together with the test driven development and test obsessed. Then there was this divide between developer and an ops and that divide went away now with the advent of the devops movement. Now, there's this divide between developers and design and I feel like that kind of wall is collapsing right now. You have developers participating a lot more in design and designers spending a lot more in development. We're seeing that but it's just funny how the devs starts integrating with all the things. KAYLIE: Yeah, that's a really good insight. CHARLES: You said you kind of naturally gravitate towards more of the backend doing the working with Docker, working with Yarn. How did you get actually involved with the Yarn Project? KAYLIE: Eventbrite converted over from NPM to Yarn maybe a year ago and the benefits that we got from converting over was awesome because we were manually editing NPM shrinkwrap, which is a nightmare and the installation speed of the container was really slow just because we were on NPM and it didn't really have any advance caching mechanisms at that point. Yarn just sped up a lot of things for us. I really like the user interface outside of just installing. You actually get a lot of freebie commands like 'yarn why,' that tells you why you have a particular dependency or you could do 'yarn check.' It was just a lot more helpful. I've been wanting to contribute to open source for a while so I did a little bit of work before then but the community was really encouraging when I first try to solve and pick up some first contribution bugs off of their backlog. At the time, they were pushing for 1.0 release so there's a lot of excitement about what Yarn will be and all the new features will be adding. I kept trying to pick places where I felt like I could be of help and then, Christoph, the manager of the Yarn Team and I believe [inaudible] team as well, reached out and asked if I wanted to build the selective resolution feature for them. I was like, "Yeah. I'll give it a go." Then I did it. CHARLES: What is selective resolution? KAYLIE: Good question -- CHARLES: Because I use Yarn every day and I've never heard this term. KAYLIE: It's something that became available with 1.0 release, much like workspaces and most of the time, you're not going to need it but sometimes, you're using a library. That library will have a nested dependency that for some reason, has a bug or you can't work with that package or maybe you just want to dedupe the package so that all of your dependencies end up using one version of that particular nested package. Selective resolution is like a way for you to override other libraries dependencies -- CHARLES: Oh, that's really cool. I like that because I've had that happens to me where I want some version of a library that has got a bug fix and yet, some other library that's depending is requiring this library and they're getting the old version and I'm like, "Nah!" ELRICK: That was happening to me last night. I wish I knew about this. CHARLES: Yes, seriously. KAYLIE: Yeah. Before this -- CHARLES: I'm glad we had you on justifying about this. KAYLIE: Awesome. Before this, you would have to file an issue with the original maintainer and maybe, it'll get fix. Maybe it won't or maybe you're stuck on the old version and you can't do anything about it. We had a really similar issue with the PhantomJS package, where we wanted to use a next patch version with a bug fix but then, something that was requiring it wasn't letting us use it. It works. I verified it. It seems like Facebook started using it as well so it was a pretty rewarding to work on that. CHARLES: That's exciting. I also plan on using it the next time I encounter this. ELRICK: To get this feature, is this a flag that you have to pass? How does it do the selective resolution? KAYLIE: As long as you're using Yarn 1.0 or above, you define resolutions field inside of your package.json, like how you would define that dependencies, you also have extra [inaudible] resolutions. On the left hand side, you put in a glob pattern that you want to match and if you want to match all packages, you just enter the name of the package. Then on the right side, you enter whatever version or path that you wanted to match. It could be a file or a GitHub link or it could be a version or a range or whatever you want. ELRICK: Nice. CHARLES: That's fantastic. Now, you mentioned that as you were coming to work on this, you were looking for features to work on but the community actually did a very good job of drawing you in and getting that contribution from you, which is actually pretty amazing when you think about it. I think a lot of open source projects, either flourished or failed by their ability to attract contributors. What was it that was particularly inviting? KAYLIE: It was a combination of things. I think one thing is they tried to point you to the actual code like if you want to submit a PR, then this is where you would get started. I actually started doing that myself on some of the issues. Everyone loves PRs more than issues so I think giving people filing the issues, some, I guess empowerment to try to fix it on their own, I think is great. They also have a Discord Channel for devs to talk about any questions that they might have, how to set up their initial dev environment, to test things on Yarn. Also, they were really nice. They were really thankful when I posted a PR or commented on the fix. They also use a lot of emojis, which helps. I think I personally found it really rewarding because it made me a better developer. Before contributing to Yarn, I didn't know that much about Node. For me, it was just fun to learn more about like, "This is how something else works," and also, the codebase uses a lot of different linting configurations, which I hadn't really used before so that was a nice learning curve there as well. ELRICK: For your initial time going to Yarn when you didn't know much about it, was the champion from the project that worked with you to get you over the hump or places that you were stuck or did you just have to kind of figured it out on your own or did you ask questions on the Discord Channel or in Slack Channel or anything? How did that process go initially? KAYLIE: I definitely pace myself. I just picked up easier bugs on a repos if possible and BYK and Mel who maintain the project would give guidance, especially through PR comments and they also answer any questions on issues. If I ask questions like what's the right approach here, because sometimes you get a bug and it's not just a bug. It actually has to do with philosophy of like, what should Yarn do in this case. There be like really minor edge cases like maybe NPM does that in a particular way, should Yarn respect that or should it try to be better? Those discussions are, I think really helpful and interesting and understanding what's going on. CHARLES: It's fantastic when the conversation just builds and you're learning stuff and then you actually feel like you came out with the best solution that was available to you at the end of the process. ELRICK: You gain a lot of context about the project during those conversations. CHARLES: Right. KAYLIE: Yeah, what was good about those selective resolution feature was it was completely community-driven, even from the RFC standpoint which submitted by someone in the community and then it was implemented by me, who doesn't work in Facebook. I think that's awesome. CHARLES: There's been a lot of thought that was put into the feature even before the implementation. That's always so critical to getting people's and giving them some specifications, some blueprint of what they're actually going to build. One of the things also that I wanted to touch on too is you mentioned before the show that they have a very unique take on the way they handle communication with the community, not just in terms of pull request but also in terms of issues. KAYLIE: Yarn issue count is around 800 -- CHARLES: Boy, that's a lot. I can feel daunting when you look at that, right? KAYLIE: Yeah, but it's actively [inaudible] project, a lot of people are passionate about it and there are bots that other projects use to just automatically close issues when they're not active. But something that BYK told me was when issues are closed, people take it somewhat personally and we just want to make sure that there's like a human touch to it and we, at least get to the issues without just automatically closing them. I really respect that. I think even when people are really frustrated, the maintainers never really lose their shit. They're always very graceful and when someone is acting outside of the standards of Code of Conduct, there's a gentle reminder. So far, all of the interactions that I've seen to the issues have been really constructive, rather than being like, "Sorry, we're not going to work on this." It feels like it is a community project and people care about how it's being used. It's actually not easy given all the different operating systems that Yarn has to accommodate. It's like a pretty low level tool so I give the guys a lot of kudos for handling that. CHARLES: Eight hundred issues means that with a human touch, that means 800 people have to actually respond to those issues and usually those 800 people are not actually 800 people. They're like 10 people or some number significantly below 800. How do you attack them to try and make sure that they're responded to in a timely fashion? KAYLIE: I think issue tracking is the hardest part of open source. I think it's in the order of issue tracking and then reviewing PRs and then submitting PRs because writing code is writing code but understanding other people's code is more difficult and understanding other people's issues and what the bug is, actually is even more difficult. You don't, obviously have their exact dev environment so sometimes, it's hard to repro. I think we do a lot of things that a lot of other projects leverage, which is we label the issues, we define priority if it's actually impacting a lot of people or if it's a critical bug like you can install a package versus maybe a warning message that could be tweet. Then we kept a board, like a GitHub board to track issues when we were heading for the 1.0 release. I'm not sure if we're still doing that but that helped to a degree. Other people from the community, not just the maintainers, jump in to try and help out an issue, which is probably one of the best things. Then when we merge pull requests, we close the issues that have been referenced. CHARLES: Yeah, it's really wonderful when that thing happens. I'm so curious like how to [inaudible] because I know that it's so disheartening when you're working on a project and you file an issue and maybe, it doesn't even go answered for one day, two days or two weeks or you submit a pull request and no one's even commenting on it. It can be really disheartening and kind of make you question the viability of the product itself, especially if you see a lot of activity elsewhere. On the one hand, I also understand that the maintainers are human and they probably have a lot of obligations. I know that I've got a lot of projects where I let the issues languish and I even have one that I'm using where I can't get any response and it's just so frustrating. KAYLIE: Yeah, even Yarn is definitely not perfect. There are definitely issues that sort of go buried. You could add us at YarnPackage/Core. That pings all of us and the core team. You could call out specific people but that's a tough one. CHARLES: This isn't with Yarn, by the way. It's a totally separate project. It's fantastic when there's that basic acknowledgement. It makes such a difference in people's perception of the entire enterprise. Looks like we're actually approaching time. If we want to give a special shout out to any talks you're going to be giving, any events that you'll be at or -- KAYLIE: Actually, Ben on my frontend platform team is speaking at Nodevember about Next React. I think that's where the shout out. And then Christoph will be speaking at AgentConf in Austria, I think in January about the future of dev tools. I think both things are [inaudible]. CHARLES: All right. Fantastic. ELRICK: Any slick future features coming out in Yarn that we should know about? KAYLIE: I'll keep you updated. CHARLES: She's sworn to secrecy. KAYLIE: Yeah. I'm not sure what I'm supposed to [inaudible] yet. CHARLES: I could hear the pause of conspiracy in your voice. Thank you so much, Kaylie for coming on the show. This has been a really wonderful conversation that's just gone all over the map and those are my favorite kind. Thank you very much. KAYLIE: Thank you so much for having me. This has been a lot of fun, guys. CHARLES: Well, if people want to get in touch with you, how would they go about doing that? KAYLIE: I'm a rarity, where I don't have Twitter. You can email me, I guess. CHARLES: File an issue on Yarn. KAYLIE: Yeah. I'm KaylieEB on GitHub and KaylieKwon@Gmail. CHARLES: As always, you can get in touch with us. We're at @TheFrontside on Twitter or just drop us a line on the email. You use that on occasion too at Contact@Frontside.io. Thank you, Kaylie. Thank you, Elrick and thank you for listening.

Dan Gebhardt: @dgeb | Cerebris Show Notes: 01:33 - The JSON API Spec and Pain Points it Solves 08:40 - Tradeoffs Between GraphQL and JSON API 19:33 - Orbit.js 26:30 - Orbit and Redux 32:22 - Using Orbit 37:24 - What's coming in Orbit? Resources: orbitjs.com (Guide Site) ember-orbit Transcript: CHARLES: Hello everybody and welcome to The Frontside Podcast, Episode 87. My name is Charles Lowell, a developer here at the Frontside and your podcast host-in-training. Joining me today in hosting the podcast is Elrick Ryan. Hello, Elrick. ELRICK: Hey, what's up, Charles? CHARLES: How are you doing today? ELRICK: I'm doing great. CHARLES: Are you pretty excited? ELRICK: I'm very excited for this podcast because this is a topic that I've heard a lot about but don't know much about and it just seems so awesome that I'm just very stoked to hear all the details today. CHARLES: Yeah, me too, especially because of who's going to be giving us those details, he's one of the kindest, smartest, most humble and wonderful people that I've had the pleasure of meeting, Mr Dan Gebhardt. Hello, Dan. DAN: Hey, Charles. Hey, Elrick. Thanks for having me on. I really enjoyed listening to this podcast. It's nice to be part of one. CHARLES: It's good to have you finally on the show. We talked over chat and we talked over email and we meet every once in a while and conferences and it's great to get to share more widely some of the great conversations that always arise in all of those contexts. For those who don't know you, you are a founder at Cerebris and that is your company, which is involved very heavily in a lot of open source projects that people are probably familiar with. One of them that we're going to be talking about today is JSON API. I bet most people didn't know that you are one of the biggest driving factors behind both the specification and several of the implementations out there. DAN: Yeah, that's been a pretty core focus of my open source work for the last few years. Actually JSON API Spec, which is perhaps a somewhat confusing name for those who aren't familiar with it. It was started by Yehuda Katz in almost three and a half years ago, I think now and it hit 1.0 a couple years ago and has stabilized since then and we've seen a lot of interesting implementations on top of it. There are some exciting stuff that's actually coming soon to this Spec that I'd like to share with you guys today. CHARLES: To give us a little bit of context, why? What pain am I experiencing that JSON API is going to solve or it's going to address or give me tools to deal with? DAN: One of its prime motivators is the elimination of bikeshedding. There's a lot of trivial decisions that are made with every implementation of an API and JSON API makes a lot of those decisions for you about how to structure your document, how to include relationships and lengths and metadata in a resource, how to represents relationships from hasOne/hasMany. Even polymorphic relationships have a type of that data. JSON API has opinions about all these things at the document structure level and it also has opinions about our protocol usage, how to use HTTP together with this media type to make requests and for servers to return responses, how to create a resource, how to add resources to relationships and things like that. CHARLES: Also, it's not just this is a serialization format. It's very much like also delving into the individual interactions and how they should be structured, more about the conversation between client and server. DAN: Yeah, in that way, it is somewhat unusual as a media type that covers both. CHARLES: Can you dig into that a little bit because I'm very curious? Something made my ears prick up was when you said, it tells you how to, for example add relationships to a resource. What would that look like? DAN: A lot of the influences behind JSON API are hypermedia-related. It's influenced by RESTful principles and includes a lot of hypermedia aspects. One aspect is how a resource represents relationships in terms of the data in document, the type and the ID that specify a linkage to that another resource in the same document but it can also include links to discover those relationships. There's a self-link for a relationship and a related link for relationship and the self-link will return the data for that relationship in the type/ID pairs. The related link will return the related resources. The Spec doesn't have strong requirements or any requirements about URL usage but instead, it describes where to find resources through these hypermedia links. If you want to say, add records to a relationship, you'd follow the self-link for that relationship when that was returned with a resource. Then you would send a post to that endpoint and you would include the relationship data in the terms of type and ID pairs. It gets down to that specifications so that removes the ambiguity of how to interact with these resources and mutate them and retrieve them. CHARLES: I see, so is there an idea then that you are going to explicitly model the relationships as individual resources? Or is that the recommendation or the requirement? DAN: The link for a relationship would point to an endpoint, which would then model the relationships that are represented that endpoint, so to say just to speak a little more concretely because certainly, it makes some simple concept sound a lot more esoteric than they really need to be. Let's just talk about an example. Let's say, we're talking about articles and comments and maybe an author. Let's say, you've fetched a collection of articles from an article's endpoint and within the article resource, you would have a relationships number, which would include comments and then comments could have links, which one of the links would be a self-link and a related link. The related link could be followed to then retrieve all the comments for that particular article. You could also, if you wanted to add a comment for that article, post to the self-link for that relationships. You post to that whatever endpoint that is specified. Maybe it's 'articles/1/comments.' It could be anything that you want. Now, the Spec does have some recommendations to make everything fit nicely in terms of the URL design patterns and such but those are not, by any means required but having those recommendations just eliminates more bikeshedding opportunities. We find it that people who are gravitate towards the Spec really appreciate having a lot of these trivial decisions made for them so even if we don't want to come down and be hard line about requiring those particular answers, we can at least provide some guidance for how things can work together nicely. There's a whole recommendation section on the site for things like URL design patterns. CHARLES: Right, so things that aren't prescribed but these are best practices that are recognized. DAN: Yeah, exactly. CHARLES: A question then that comes to mind, it sounds like JSON API solves a lot of these bikesheds or just kind of comes in and takes one side or the other for modeling both the resources and the relationships between those resources so there's the... I don't want to call it a schema but the boundaries around which resource are very clear and where they live and how they connect together. I was hoping we could maybe contrast that with some another approach, which is also become very popular and that's the GraphQL approach where you're essential assembling views at runtime for the client. It's very easy to marshal the data that you need to present to your view because you've got only one endpoint, as opposed to having to coordinate between them. I can understand the appeal of that and I was wondering if you have any insight into what the tradeoffs are between the systems and what are some of the capabilities that one can do that the other can't. CHARLES: Yeah, sure. I'm glad that you brought that up because I feel like GraphQL has become a real juggernaut, at least because of its marketing. It's very effective in being marketed for its use to developers and it's capabilities versus REST, as if a RESTful system can't possibly achieve the same outcome or the same efficiency. I'm glad to compare and contrast the two. To be honest, one of our short term goals is to better tell the story on the JSON API site, which was always a kind of a more technical spec-y site and a marketing site. That hasn't really helped its uptake as much as it could as some of the GraphQL sites are very sleek and polished. Anyway, let's get down to it. GraphQL allows you to basically define the data that you want for a particular view and that can bring together multiple related resources. It defines a way to specify exactly which fields you want in that graph of resources. We'll just stick with the articles, comments and authors example. You can specify that you want a collection of articles and perhaps the comments-related to that and the authors and you could have it all assembled in a single response. JSON API also allows you to do just that. It allows you to make requests for multiple related resources to constrain the fields that are returned for each resource and to include all of these related resources in a single document. The main difference in the representation is that JSON API requires that resources only be represented once in a single document. GraphQL may have repetition of resources throughout the document that's returned. For an instance, your articles that may nest authors and those authors like Charles Lowell, may have written three of those articles and that representation of that author is going to be repeated in that JSON API compound document, which is a term for document, which has a primary dataset combined with related resources. That single author would only be returned once as related resource and the linkage between the primary data and the related resources would be established to type/ID pairs. Instead of having the author represented three times, the same type/ID pairs would just be providing that linkage to the same author and that author resource would only be represented once. This happens to be ideal for client-side applications that number one, basically want to minimize the size of a payload that sent. Number two, don't want to after-handle repetition of data by doing extra processing of pushing the same record multiple times into a memory store that is keeping that data. I think that GraphQL is well-suited to applications that request data and display that data pretty much as returned. There is no intermediate holding onto that data in, say a memory store for later access. Basically, it lines up well with a component library link React, which wants to display that data that's returned from the server. If it wants to display that collection again, it will simply request that collection again and pretty much throw away the data once it has been rendered. CHARLES: I can see that. Dan, you and I might be some of the only folks who remember. I don't know if you ever did any Microsoft Access Program. DAN: Yes, I did, believe it or not. CHARLES: Doesn't it feel a little bit like the Access pattern all over again, where you have your components requesting data from basically, constructing a query and requesting it -- DAN: Yeah. CHARLES: And then throwing it up on the screen. DAN: You're going deep there but I do remember that. Definitely, there is that same paradigm. CHARLES: It's really powerful. DAN: It is and it's pretty accessible too because it's a direct representation of what you've requested and there's no intermediate processing. I guess the question is, whether that intermediate processing provides some value. Actually, holding onto that data provided some value because as far as I'm concerned, GraphQL is great for that rendering of DOM data, where the data has no meaning except outside of the rendering. But if you want to actually have models that have some intelligence about that data, then you want to use a store to keep those models in and you want to be able to reuse those models for other purposes. CHARLES: What might be an example? What's a concrete use case that we can ground this discussion? DAN: I would say that the big one is offline. You simply can't have just DOM data that's useful in any way in an offline application or an optimistic application, where you are doing some things client-side and only say undoing them if a request fails. But if your data is DOM and only structured for a particular view, then all you can do with that is redisplay that view. But if you understand the schema of your data and that data is available in a store, then regardless of whether you have a network connection, you can actually display that data in different ways. If that same article shows up in a collection in a list, you could also display that article on its own in a different format with more fields. If you want to, say allow editing of that data, you could allow for an editor when your app is offline. Allow changes to be made to that data and then redisplay it because you understand the fields that are in that data. CHARLES: Right and then at some point later, then spool those changes back to the server. DAN: That's right. CHARLES: It almost sounds like, ironically if a system like JSON API where you have very concrete boundaries around each of the underlying resources in your data model, it allows you to essentially do rich-querying on the client and not just the server. DAN: Yes, that's absolutely true. CHARLES: Because I feel like what you just described to me it's like, now we have some sort of store over which we can map all kinds of different queries to our own liking and there's no dependency on the server. DAN: Yeah and if you just want more web app to be pretty much a view representation of what's on the server and without additional intelligence, then GraphQL really lines up well with your needs because any extra processing you're doing is just not valuable to you. But I think a lot of the really interesting things being done in client-side applications are where your client application is pretty loaded with a lot of intelligence and you're out there autonomous and able to make sense of data. In that case, then thinking about the data only as it pertains to views is not nearly as powerful. CHARLES: Right, so you could do something like that with GraphQL but then you would have to, essentially structure your queries such that they drew the boundaries around the individual resources anyway, rather than composing them on the server. You'd have to query them discreetly into a store and then run your local operations. Then I guess at that point, it's like what are you doing? DAN: Yeah, you're still doing the extra processing of handling the repetition of any nodes that repeat and such. That's just extra processing you have to do but I agree that you certainly could structure your GraphQL queries to return data that is then loaded, say to a store that really has awareness of the data types but I don't think that is -- CHARLES: But then you're defeating the purpose, right? DAN: Yeah, it's not its selling point and it's not its strong suit. CHARLES: You've done a lot of work on the JSON API Spec. JSON API allows you to fetch discrete resources and their relationships but still, keeping one representation of each resource in the payload so it's optimized for wanting to do client-side processing and have intelligence based on these entities, which are in a store. You actually maintain a fairly mature, at this point, framework called Orbit, which helps you do some of these things. Now, what Orbit does today and I understand that you've got a lot of new features that are really exciting, that are coming down the pike. Before we get into those, what is Orbit and what do you use it for and how does it use JSON API? DAN: Orbit is a data access and synchronization library, which sounds sufficiently vague because it has a lot of low-level primitives for structuring client-side. Also, actually isomorphic can be run on the server and nodes so it's not even only used for client-side purposes but that was its original purpose. The abstraction that it includes are allowed for synchronization of data changes across multiple sources of data. Source of data might be represented by, say a JSON API server, an in-memory store, an IndexedDB database in your browser, a local storage, all of these sources of data can support an Orbit interface, which provides their access to their data and also broadcasts changes to that data. In order to coordinate the changes across multiple sources, say to back up all of your data that's in memory to IndexedDB source, you can observe the changes on one source and then sync those changes up with another. For instance, you want to structure an offline application which you have been in-memory store, which uses client-generated IDs, which then syncs up with a backend JSON API source and every change that gets made to the memory store needs to be backed up, you could configure multiple coordination strategies between the sources to make sure that the data flows so that every change that is made to the store is immediately backed up to IndexedDB. If it can't be backed up, then it fails. You can add some error handling and then when you're online, you can then also sync those changes up with a backend so you're basically pushing those changes that are local to a remote store and you're not slowing down your offline app, which you're communicating with optimistically and then only handling, say synchronization failures when there is a problem. In order to handle those problems, Orbit sources are very deterministic about their tracking of changes and they provide get-like rollback capabilities so you can look at the history of changes to a particular source and reset the history to any point there and basically handle conflicts and merges in a very get-like way. Often I use cases, the primary driver of Orbit's whole architecture, I realized that it needed to be able to give you the tools to handle any conflicts that happen when changes get sync up. Also, give you the different tools to model all the different places of data is kept in order to support the offline mode. That's a kind of a broad overview of Orbit. There's a new guide site, OrbitJS.com for those who want to dig a little deeper into it. The data is structured in the JSON API format internally to the store and the standard operations are very much influenced by the standard JSON API protocols that are allowed in the base Spec over creating records and removing records and all that crud for both records and relationships. That's where JSON API comes into Orbit. CHARLES: Right, I see. The primary use case for Orbit is offline. Is that fair to say? DAN: Yeah, that was the primary driver, although it's just not the primary -- CHARLES: It seems like you could use this in a lot of places where I might use Redux or something like that, like on the server to model... I don't know, a chat app. DAN: Yeah, definitely. CHARLES: I have a bunch of different information streams coming together and how am I going to merge them and make sense of them. DAN: Yeah, in fact, that it's primitive level. Orbit has essentially an async redux-like model for queuing up changes and applying those changes. The change sets are all immutable. There's actually a lot of immutability use here throughout the library. In order to ensure that the changes that are applied are tracked deterministically, we just can't have those changes mutating on us. There is definitely some overlap with Redux concepts in terms of the general tasks or action concepts in Redux but instead of Redux's synchronous approach, everything in Orbit is async. CHARLES: What does that mean? Redux is synchronous in the sense that there's a natural order to all actions. For those of us familiar with Redux, are you saying it would be like a store where actions can be dispatched at any time or is it more like, I've got multiple stores happening and I need to resolve them somehow so each one is synchronous? How can I make sense of that? DAN: In Redux, the actual application of an action is performed synchronously. CHARLES: Right. You can have asynchronous processes but there is a natural order to the actions that those asynchronous processes yield and then those are applied synchronously to the Redux Store. DAN: Yeah. To compare and contrast Orbit and Redux, I guess you'd first have to say there's a primary difference of -- CHARLES: I think there are a lot of people are familiar with Redux. I think it's not so much to compare and contrast it but just to use it is as an analogy of like, "Here's how it's the same here. Here's how it's different," because that's compare and contrast. DAN: There you go. CHARLES: But not in terms of evaluating them but it's like, "Maybe I should be using this instead." DAN: Right, they are sort of on different levels, although there are some primitives in Orbit to it and it's shipped across multiple libraries. There are some primitives, I think that could be useful outside of the main Orbit data application. Anyway, the way that Redux state changes are applied, the function is synchronous is all I was getting at and on Orbit, every state change that applied to a source is asynchronous so the result is never applied immediately. You'll always get a promise back and you'll never have that application happen immediately. That's one clear distinction. Another is that a redux has a big singleton global state for the entire application. Orbit very much has a model of state per source so there can be any number of sources in a particular application and the source might be an in-memory source or might represent a browser storage in XTP or might represented a socket that streaming data in. All of these have state at different, temporarily distinct state that even if they all converge to a common state, the Orbit models separately so that there's a set state per source. I'm just contrasting the global apps state that exists in Redux with the per source state in Orbit. CHARLES: It sounds like there's nothing that would be fundamentally incompatible of using Orbit really in conjunction with Redux, where Redux is kind of a materialized view of all of your different data sources presented as what you're going to render off of, right? DAN: Yeah. You could use it in a similar way to Redux Saga, where Orbit fills the role of Saga, where it's doing the asynchronous actions that results flow back into the Redux state. CHARLES: I'm just imagining having one big global atom, which is your Redux store and now I'm saying, prescribing this is an optimal architecture but I'm saying, one way it could work is it picks and chooses and assembles off of the different sources as new data becomes available. As the states change for those sources, it can be integrated into a snapshot state, which is suitable for rendering or provides one view for rendering. DAN: Yeah. You're basically talking about the in-memory source, perhaps merged with other applications state, which is not so resource-specific and that is possible to model. CHARLES: What I think I might be hear you saying is you could also just use another source which is the merge itself. DAN: Yeah. I'm not sure how much we want to continue this thought exercise because the architecture becomes almost not something I'd recommend. But I would actually like to explore how Orbit and Redux could be used together optimally. I played around a bit with Redux but I have not written a full-fledged application with it, other than a [inaudible] location. I definitely defer to you for Redux best practices and such and how people are using it in real world applications but I'd be really interested to talk that over again soon. CHARLES: Well, I just certainly don't count myself a Redux expert, although we have developed some applications with it. We'll put that on the back burner or something to explore it later. I will say this, I find Redux to be both wonderful and terrible, kind of in the same way the Java is both wonderful and terrible. We'll leave it at that. DAN: Okay. ELRICK: That was going to be my question. This is why I was very excited to hear about today was Orbit because I've heard so much about it. In terms of the implementation of Orbit into an application, what would that look like from a high-level? Has anyone used Orbit in the production app? Have you built any apps using Orbit? DAN: Yeah, definitely. There are people using Orbit with React, with Vue, with Angular and with Ember and there's an integration library called ember-orbit which makes Orbit usage really easy in Ember. In a lot of ways, working with ember-orbit feels a lot like working with Ember Data but it allows a lot more flexibility. I suppose one of its strengths and weaknesses is there's a lot of configuration that's possible because there's a lot of possibilities. The internals are exposed of how data get synchronized so you can define your strategies and sync up different sources. In terms of how it's actually used in an application, you'd start by modeling your data in terms of the resources that are in the application. You'd have a schema that defines your articles, your comments and your authors, just to keep that example going. Then that schema would be shared among all the sources in your application. You would have one source, say that might be the in-memory source and another source that is the representing a browser storage so you could, say swap out either local storage source or an IndexedDB source and use either one to provide that backup roll. You would declare those sources, you connect them to each other with strategies so that, say when memory storage changes, you would then sync that change to the browser storage source. Then you'd have back up and you'd be able to then, refresh your page and view the same data you were looking at before. Now then, if you probably want to wire up a remote source so that you're communicating with the servers so you bring in JSON API source and you would then set up a new strategy for working with that. You have to decide like, "When my memory storage changes, do I want this change to happen optimistically or pessimistically?" By that I mean, "Do I only want it to appear successful if it's been confirmed by the server." Depending upon whether you want to be optimistic or pessimistic, you setup your strategies a little differently. If you handle this change pessimistically, you wanted to block success on the successful completion of pushing of that change to your remote server. You have the set of tragedies that define the behavior of your application and then doing your crud operation is probably pretty much directly with your memory source. Then if you wanted to, say do an edit in a form, you might fork the store, now the store keeps its data in an immutable data structures. That forking that store is very cheap so you don't have a bunch of data that's copied. You're just keeping a pointer to that and getting a new pointer to that same immutable data structures. Every time they get changed, there been new references. There's an immutability under the hood but you're pretty well insulated from the annoyances of working with that immutable data structures. At that form, you make your changes, you then merge your changes back, you'd get a condensed change set of operations that then can flow through your strategies. It flow through to your backup source. It could flow through to the back to the server. I think it would feel pretty familiar for users of Ember Data because there are a lot of the API influences came from that library. But obviously, people are using just plain Orbit with other libraries, with other frameworks and finding it useful there but it definitely involves a little more configuration up front to do all that wiring that might be more implicit in library like Ember Data. CHARLES: I understand that before we go, there is some pretty exciting new things coming in Orbit. Do you feel like you're ready to mention a couple of those things or they've been kind of mixed in with the conversation? DAN: Let's see. I have the guides up, which I mentioned, which is pretty new in the last couple of months. In the last year, we did a rewrite and Orbit is now completely in TypeScript and there are no external dependencies. For a while there, I was using RxJS and observables internally and immutable JS so there's now an internal immutable library. It's lighter-weight with fewer dependencies now. I'm excited about that and finally feel like I can recommend people digging in with the guides that are up. I'm hoping to get up the API docs soon. I will say I'm excited. I just got back from a retreat in Greece. Séb Grosjean who owns the company, BookingSync does this amazing thing with the Ember community, where for the group that's working on Ember Data, he invites them every year to come to his family's place in Greece. He grew up working with his family on his rental properties, which was the inspiration for his company, BookingSync and said, "This is a fantastic opportunity that for us to get together and collaborate in a really nice place," and I had a really productive time this last week. This is the very first time I had gone. It was just fantastic and I worked with the Ember Data team. Igor Terzic and I spiked out some interesting collaboration between Orbit and Ember Data so I'm really looking for it to see where those go and hopefully, we'll see a little bit more Orbit, either directly or just through influence appearing in Ember Data. I'm looking forward in working more closely with the Ember Data team. We'll see what comes of that. CHARLES: Yeah, I, for one am very excited to see it. I'm resolved now. I'm just looking at these guides. These look fantastic and I'm resolved to give Orbit, at least a try here, either in some of our applications or maybe trying to spin up some new ones and have it the basis for some of ideas I've been playing with. DAN: That would be awesome and there's a [inaudible] channel, which I hang out into if you have any questions, if anyone out there does. CHARLES: Before we go, if anyone is interested in JSON API, is interested in Orbit, is interested in Cerebris, we mentioned a lot of things that in one way or another, map back to you. How do we get in touch to find out more about these different entities/projects? DAN: I'm at @DGeb on Twitter. My company site is Cerebris.com. Also check out, OrbitJS.com for the new guides. Reach out to me. I'm on the Ember Core Team so I'm also hanging out in the Ember community Slack, depending upon what you want to talk with me about. I'm in all these different places so I love to hear from you all. CHARLES: All right. Fantastic. We'll make sure that we put those in the show notes and I guess that's about it. Do you have anything else you want to leave folks with, any talks, papers or big news coming around soon? DAN: Something that we didn't really get a chance to talk about today, which I'm really excited about is JSON API operations, which is an extension to the base Spec, which I'll be proposing very soon. There's a future to the JSON API once it hit 1.0 a couple of years ago. It didn't just stop. We're looking at different ways to extend the base Spec and use it for different and interesting purposes. JSON API operations, I think one of the most interesting ones. The idea is basically to allow for multiple requests that are specified in the base Spec to be requested in a batch and perform transactionally on the server so the Spec will define how would each request gets wrapped. Each operation very much confirms with the base Spec concept of a request. For implementations, there's a lot of opportunity to reuse existing code for how to handle each particular operation but to provide a whole new set of capabilities by allowing you to batch them together and process and transactional it because it just unlocks a ton of different things you can do, all based on the same base concepts from JSON API. I'm really excited to have something to announce soon about that. CHARLES: That sounds like it might solve a lot of problems that are always associated with those things. It always comes up. What's our batch API look like? I don't think I've been on a project that didn't have a months-long discussion about that. I ended up like kicking it down the road and I'm just flumping something in place. DAN: Yeah, all those messy edge cases where people figure out how do we create multiple related records altogether in a single request and people do it ad hoc and do it with embedding and such and want to standardize that in the same way, that we've standardized the base operations. CHARLES: Well, that is really exciting, Dan. I wish you the best of luck and we'll be looking for it. DAN: Thanks a lot. Thanks for having me on, guys. CHARLES: It was our pleasure. Thanks. With that, we will say goodbye to everybody. Goodbye, Elrick. Goodbye, Dan. Goodbye everybody listening along at home. As always, you can get in touch with us. We're at @TheFrontside on Twitter or you can see our website at Frontside.io or just drop us a line at Contact@Frontside.io. Always love to hear from you with new podcast topics, anything that you might be interested in so look forward to hearing from you all and see you next week.

Alex Matchneer: @machty | FutureProof Retail Show Notes: Charles and Alex Matchneer have a great discussion that centers around routing in Ember.js: what they want to see in a router, what problems it solves, what's wrong with the routing solutions we currently have, and what the ideal future looks like in respect to routing. Resources: Episode 067: ember-concurrency with Alex Matchneer Cordova ember-rideshare react-router Transcript: CHARLES: Hello everybody and welcome to The Frontside Podcast, Episode #86. My name is Charles Lowell, your developer here at the Frontside and podcast host-in-training. I'm flying solo today. It's been a while but that's okay because I've got a really fantastic guest on. Actually, we debated this at the beginning of the show, whether this was the third or the fourth time he's actually been on but no times are too many so hello, Alex Matchneer. Welcome back to the podcast. ALEX: Thank you. It's great to be back. CHARLES: You're still at the same place that you were the last time. ALEX: Yeah. Still working at FutureProof Retail. I'm still working on bunch of mobile ember-cordova apps and that's definitely occupying on my time. CHARLES: Nice. Because FutureProof Retail has a large hardware component and we were doing a series on IoT, we were originally going to have you on the show to actually talk about that experience of what it's like to be a part of a startup and develop software that's going to be running on a bunch of devices and the unique set of problems that poses. But in the pre-show, we decided to scrap that because there's actually a topic that we're both very interested in and you've been heavily involved in lately and might be a really interesting preview as to what's coming in the Ember community and at large. Today we're actually going to go back to talking about the same subject that we talked about in our first podcast, which is routing: what we want to see in a router, what problems does it solve, what's wrong with the routing solutions that we have today. Talk about what that beautiful, ideal future that we want to live in looks like with respect to routing. You've been thinking about this a lot lately. What have you been thinking? ALEX: I'm an Ember core team emeritus and back when I was on it and I'm a lot more active, I did a lot of work on the router, particularly with how it handles asynchronously loading data when you click on links and go to different sections of your app. I spend a lot of time over the last three or four years figuring out the nice patterns for what you actually want to use if you're building out lots of Ember apps. Then kind of around that time, right after landing some cool stuff and some not cool such us query params, which has been a challenging aspect, I start working at this company FutureProof Retail that is like 90% of the Ember work that I do there is in mobile apps. We use Cordova so we're basically running these apps inside a web view, inside either iOS or Android so that we can stay with the technologies we are most familiar with, such as JavaScript and CSS and HTML and build apps using that. We can use Ember to do that. What I found was that I couldn't really apply a lot of the same patterns, all these nice conventions that Ember router gives you. I couldn't really find a way to map that onto what I need to build in mobile apps and there's a few different reasons. I got really busy with the startup, just trying to build these things and kind of went off the happy path where I really just couldn't find a way to make it look like an Ember app. One of the nice things about the whole points of convention over configuration as this sort of Ember and Rails philosophy is that, one of the benefits is that if you know Ember and know Rails, you can drop into someone else's apps as long they're following these basic conventions and immediately know how to be productive and know how it's structured, know how to make a change to it and have it maintain a convention and not just have everybody who's using some framework build these totally different apps from each other that have no shared conventions and whatnot. Everyone is supposed to be able to learn from each other, grow with each other as long as they stay with these conventions. I couldn't really find out how to stay within Ember conventions and build this mobile apps. For a long time, I just didn't really contribute too much to the Ember router at all. I kind of fell out of touch with how most people are using it because most people are building these desktop-centric apps and here I am working on these mobile apps after three years. CHARLES: What are some of the specific use cases that were just impossible to, or not impossible but presented a challenge? ALEX: The first one is which is I think is actually one of the easier problems to solve but still some challenging is that you want something that's called stack routing or stack navigation in a mobile app, which is if you're actually building a native iOS app or an Android app, they both have different names for how they provide you this. But you're thinking of things in terms of stacks. In Android, you might open another activity, which is a full frame of a page in your app and you can push it and then when you press the back button, which is built in in Android phones, it'll pop that off the stack and take you back to where you were. In iOS, they give you a UI navigation controller and let you push and pop view controllers and that is how they want you to think about these applications. That is contrasting to what Ember makes you think about, which is go and define your static hierarchy of all the different places that you can be in an app. But with stack-based navigation, you don't necessarily know upfront all the different orderings of which frames are going to be pushed onto what and you might have situations where you want to be able to dynamically push, say an 'Add a Credit Card' page to where you are and maybe it depends on some data that's been loaded at some lower level in the stack and you can't model that as nested routes in the way that you might think about it in classic Ember apps. It's a different structure -- CHARLES: Now, when you say lower in the stack, I'm curious, if you're entered in aren't you... Oh, you mean... I see, previously in the stack. Okay, so lower in the stack so you're thinking like your current position is at the top of the stack. ALEX: Right, yeah. CHARLES: I see. Now, let me just clarify this in my own head. Your Ember routing structure is ultimately realizes a static tree but at any moment, you are entered into one path through that tree so you do have something resembling a stack. It's just is it the pathways that the ways that you can actually get nodes onto the top of the stack is you're limited because that can't be dynamic. ALEX: Yeah, but even then, it's hard to describe what the difference is but the kind of stack that you're thinking of in terms of the classic Ember router map is more like you're in these different substates than you are different frames that you've pushed onto your -- CHARLES: There's a finite and fully enumerated set of next states. ALEX: Right. To be very concrete, if you have a post route and then a post show route and then a comments route under that and these are all nested in a row, then if you're in the comments route, you are in a kind of hierarchical stack that might have loaded the post that you're looking at and maybe the post call-to-action above that and the comments for that but you're still in one thing. You've just expressed that one thing in terms of these substates so that every other state that's in the parent state can share the same data loading. That's different from saying, "I'm on this page and now, I want to push another page on it and maybe tap some of the data that has been loaded on previous pages." That's more of a navigation stack in a hierarchical substates stack. CHARLES: Is the difference then, the data dependency? Because if you think of the Ember classic where you got the static tree, at least theoretically all of the data in the leaf nodes depends on the data that's above. It's not just being able to dynamically push stuff onto the new stack but it's also saying, you want to be able to push stuff that might have no dependency on the stuff further up and it doesn't need to be re-rendered if stuff further up the stack changes. ALEX: Correct. CHARLES: But sometimes it might. ALEX: Right so there are a lot of corner cases that come out if you try to model this new way that a lot of corner cases have been thought out of if everything matched nicely to this hierarchical substate classic Ember stack but not for navigation. If you want to do something that's stacked routing-based, I've had a few different approaches. At our company, we maintain a suite of different apps that are sort of retailer or grocery-centric and the first one we did, which is more popular flagship one is Mobile Checkout, which is an app that lets you going to stores, scan items with your phone and checkout and skip the cashier line, which is great if there's huge lines and you just want to buy a little handful of things or maybe in your shopping cart. But that is like any other mobile app is really conducive to this step navigation approach. Then we had to make a few apps after that such as like another app that is [inaudible] do a manual check then ordering app and other of handful things that you can imagine is might be used on a grocery store. I took the opportunity to like, "I don't really like how the routing turn out the main mobile checkout shopper apps so let's try different things." If you approaches, at least have their pros and cons without really feeling you're solving the problem and one is to maintain your own in-memory stack of where you were, every link to you, you might recall where you were and then use that logic in addition to what's in a URL to decide what transitions to make, which to use Liquid Fire for that. But already, there's these weird growing questions like, "Why are you even using the URL? Is it helping you at all?" That was the main issue with the main app that we did. The other approach was to try and not even use any of the 'router.map' stuff at all. I use the router.map to basically just create one wildcard route. You can use normal Ember to use it like '*half' and that basically collects the rest of the URL as a param that you can use to do whatever you want with. I was using that to basically pass to another, which is internally used by Ember to do the stack-based parsing like grab a little bit of the URL and then parse the param for that then grab another. Every time you could see your stack in the URL. That has its benefits but the worst part about it is that it's getting further away from Ember so any add on that you might want to use at Internet of Things in terms of which route you're in and has conventions like that you just can't use. I can't think of a good example at the top my head but it's like the further you get away from those norms, the less the Ember system can help you and on your own building your own framework. This is all to say that I think I have enough experience at this point to bring home some of the things to Ember and I'm excited to get back into contributing to Ember with this one particular thing that I'm focusing on now, which is... I don't even know what to call it. It's like -- CHARLES: What does it do? The route stuff? ALEX: It's route stuff. Actually, let me get into the other... That's what is tricky about stack routing and tricky to sort of, if you already have to go through a mental hurdle with thinking of the Ember router and as a stack of states or substates and you train your brain to think that way, it's really hard to take yourself out of it and realize that what you're trying to build with like a classic mobile navigation is almost looks like the same thing but it's really different. The other challenging problem, which is specific to our particular app is that you wouldn't think of it as a very heavily server-driven app but if you're writing an application that at any point can get a message from the server like, "Hey, your status has changed," and that state is heavily coupled to navigation of where you're allowed to be in your apps for the state of some certain model, then you're going to have a really hard time, I'd say in modeling an Ember. I have a really hard time convincing people of this until they've actually tried to do it themselves, which is why I'm going off and just building things showing people. CHARLES: You don't have to convince me because I think one of the biggest problems is the router is like the one non-reactive piece of Ember, which is unfortunate because it's essentially, what is the equivalent of the Redux store in a Redux application, where it's the state that drives literally the entire application and yet, any type of non-hash change driven updates, you have to manually manage. Every time that we've done it, it's been a problem and depends on what data, at that point you have to be very thoughtful because, at least from the highest level, if there's damage to a piece of the tree higher up, you need to realize those effects of that damage or that change all the way down the tree. ALEX: Exactly. That is a great way of putting it. This is maybe a good time to mention this thing called ember-rideshare. I've had a really hard time describing these problems to people so I figured what I would do is write this blog a few months back, a little article called ember-rideshare. It's just a given name to the kind of app that still really hard to write in Ember. It's a mobile app. It involves stack routing but the other part is really difficult about it is this problem of the router being in a silo. It is reactive but it's only reactive to that URL. Other things changes, they need to, like you said come in and patch up something else about the router in case you add some URL that is no longer able to present some model of whose status changed. That's an article on a blog that I can probably link to in show notes or something. When I talk about ember-rideshare, imagine using Ember to build Uber or Lyft and it's got just the slightest bit of the whole thing. The whole point of the app is to coordinate your client-side request of I want to ride with the server going off and doing a bunch of things and finding a nearby driver, displaying you bunch of driver locations and it'll show up. Then finally, find you a driver. It's a constant communication. Throughout that point, you can sort of imagine modeling all the different screens as routes but the routes that are actually allowed to see at any given time are heavily dependent on what is the current state of the user's current ride. But you shouldn't be able to go to a route that says like 'cancel ride request,' if you haven't requested a ride in a million of these other things. If you're an Ember developer and you think that's an easy problem to solve, you're probably thinking, "I would use before model hook when I'm entering that route to check the state of the model," and if it doesn't make sense for the route of entering, I want to transition elsewhere. That's fine. That's good if you're doing an app if the user is the one deciding where to navigate to. But then when you're on a route like that and then the server tells you that your ride is done, you can't still be on that route so you've got to have some kind of validations that is like, "This is no longer a valid route to be in. Is the user still in this route?" CHARLES: "Where am I going?" ALEX: Yeah. Before model doesn't really help you. It's this one-shot discrete event and you just can't capture all the different things. The ember-rideshare describes some of these problems a little bit more detail but that's the main issue with it. Like you said, what is actually missing about the router? Maybe it's reactive but it's only reactive to the URL, what about all these other things that are happening into your app? I think there's a handful of APIs in Ember that they're great but they're kind of siloed off in a way. If you want to make two different kinds of worlds meet, you've got to write a bunch of your own code yourself or you just have to do mentally going back and forth and being like, "I did this, so I can't use this kind of API." I did a lot of work on the Named Blocks RFC, which previously there is silos between if you're passing blocks to a component versus data, you've got to think about them differently and all the ways that you might forward that data to a different internal component, if you want to build these composable, reasonable internals, you got to be kind of split-brain about it. I feel the same way about how the Ember router works. It's only good at dealing with stuff that has to do with the URL and you're on your own, if you needed to react to data changing. That's what I'm trying to fix. Does that correlate with your experience of working on Ember stuff as well? CHARLES: Absolutely. I think that's a great way to put it. I think we've come to a consensus of the problem statement. I am curious to see a big separate query params. I'm going to throw that wildcard out there or maybe we should save it for later. ALEX: Yeah, I definitely going to come back to it. If I say all this cool stuff and I still don't have a solution to that, then what am I talking about? CHARLES: Right. ALEX: Which to be honest, I haven't thought of every single possible thing. I'm doing the thing where I talk about it on a podcast that everyone can guilt me into really finishing it. I actually really think that I'm going to finish it. I'm very confident in stuff I'm working on. I'm very excited to bring it to people but it is not all 100% fleshed out and I definitely appreciate anyone's help to those interested, understands the nature of the problem and wants to help me work on some of this stuff and like that, in Ember community Slack or wherever. CHARLES: Yeah, I'm really excited to hear it and see in what ways we might be able to contribute. ALEX: Basically, the goal is to find some underlying primitives that can model the current behavior without mistake because obviously, we can introduce something that's going to break into Ember apps. Basically, to recognize that the URL is something that goes through multiple passes of transformation, to eventually become the thing that displays stuff on your screen, from the very foundation of it, and this is the actual mini-course of what Ember router does internally because it involves a few different libraries and maybe this is a re-hash from the podcast that I did with you guys but -- CHARLES: Can I just say that there are some things that the Ember router really does right, that are fantastic? One of those things is it baked in to every single piece of data. It doesn't do the stack but in that tree that it models, every single node in that tree abstracts away the asynchrony of that node. I think that's absolutely huge so you get both the dependency enumerated like these are the things that I need to marshal the data to render myself and it's implicit that it might take some time. I might need to draw on a couple of different things to actually assemble this data so the asynchronous nature is modeled up front and it's implicit and it's there every single time, which turns out to be the right thing. The sampling that I've missed has been an excruciating void in all the other routing solutions that I've tried outside of the Ember community is that they just punt over asynchrony to you. You deal with it, not our problem and it's like, "Actually, that is the problem." Anyway... ALEX: That's a great point because if the router doesn't help you with any of this stuff at all, then it basically means that every one of your pages that you might want to render after the fact, probably has to have some loading logic like if data is loading, show us spinner. Otherwise, here's all the data -- CHARLES: Yeah, if something happen wrong. ALEX: Right and sometimes that is actually what you want to do. Sometimes you want to do these skeletal in UIs that looked like the page that's about to display but the date isn't there yet so everything is, regardless going to be wrapped in these 'if' statements, 'else' statements. I worked in ember-concurrency and some people are using that to basically move more of that loading into controllers, that's fine. If that's what you're actually trying to do and that's what you're opting into, that's a perfectly reasonable solution but most of times, chances are you're entering a route and you don't want to have to teach the entire template tree underneath it that has to handle all these different states. There's these nice ideas that work in some cases and I'd like to make them work in more cases than Ember helps with and a whole loading all the promises and the model hooks and absolutely going into the loading state are really cool primitives that Ember is going to do for you. The other frameworks, they don't try to be opinionated. They won't do any of that for you. Sounds like you ran into that with some of your React stuff? CHARLES: Yeah. I definitely did. There's just not much help when you actually want to model asynchrony. You can do it. It's pretty easy. You just implement the right hooks or model a series of actions, either with a Saga or Epic, if you're using redux-observable. But again, you have to assemble it by hand and you have to generate those abstractions by hand and you just want to have them at hand already and not have to worry about that. But the advantage, though is that generally those ones that you do have at hand or that you generate are fully reactive. If new information comes that's germane to that particular leaf in the tree or that particular note in the tree, there's no difference between the initial state and the update state. Whereas, in Ember, you got your first shot and then that data is now at rest. ALEX: Right. I definitely have been looking at React router, in particularly v4. I think it's all contentious for people to see it at first but being able to put things like in your render function, you can say, "If this data is present, something that's going to be past and be a prop or something," then show a loading spinner or otherwise, start matching these subroutes. That's really cool. That's expense that you can't look at essential map of all the states of your router can be in but that's also a real problem and if you can demonstrates that the state world is not in a separate silo than the routing world. CHARLES: With great power comes a lot of bugs. You do run into a lot of things where you have rogue matching. You have random things that are inside your view tree that are matching against the route and they just render and you have to be very careful because it's almost the difference between blacklisting and whitelisting. I see what you're saying. It could be confusing. ALEX: Yeah. I think it's definitely a tradeoff. I think if I had something like a match, I might have been able to maybe arrive at a stack routing solution a little earlier. I'm not sure about that. It's definitely something that could be handled by React router. I think one of things that React and React routers better at in general is that everything is, more or less a component that is more easily swappable or something else here. You're not going to have as many of those silos and I really do think, it went through a lot of churn and maybe, some people had trouble, maybe a lot of people, I don't know had trouble kind of following all the major versions. But I think React router Version 4 is pretty damn cool. I think there's a fullest realization of that kind of modular mindset. CHARLES: I think the biggest problem I have with it, though is it requires the view tree to model your routing structure. That bothers me. I feel like you could do the exact same thing. You could have a way to express your routes, not necessarily with a separate routing file. I supposed you could do it with JSX or something but actually have it be kind of orthogonal to your view tree. The way you can model this dynamically updating thing that can match against anything and maybe, even express it all in one place. Although once you get a big tree, it could be hard to control that. The part that I've come into most conflict and maybe who knows, maybe I just haven't used it enough, we've only got one application that we're using the router V4 on. But the fact that it's actually in the view tree, it bothers me. It's in the state objects. It's hard to adapt to Redux because that state is opaque. It's the routers controlling it and I would it to be not have to pass through React components but just be like, give me the firehose of the router state. ALEX: Right. I love what you're saying. If I'm going to bring this stuff to Ember, I can't suddenly make it work like matching within the view tree. That's not what I'm working at or proposing here. All the stuff is basically to empower that firehose to respond to more things that can drives views and respond to them in a live way, not like a one-shot async validation, only when you enter. CHARLES: Maybe this is what the problem that you're trying to solve and one of the things it's really nice to be able to match against anything inside the view tree is that Ember's rendering process of a route is very opaque. The process, by which an outlet gets connected, that's not something that you really have much visibility into. Is that a good statement of the problem? ALEX: That's definitely part of it. You definitely have to go to the documents. I think it's telling that -- CHARLES: I've never done it. I don't really know how that works and I've written a lot of Ember code. ALEX: How what works? CHARLES: How the route gets rendered, like the mechanics around, which I understand how the route object actually, you makes the decision to render its template and do all that stuff. I know it as a user but I don't know the mechanics and I wouldn't know how to extend it. ALEX: I'm not sure if the stuff I'd work on but it immediately make some of that stuff more clear. One of the goal or constraints is to really try and break down the silos. Whatever I'm about to propose bringing to Ember, I want it also be something that would be useful, possibly at the component or template or controller level, rather than just being this thing that lives only in the router's weird black box of logic that occasionally calls hooks that everyone knows about. CHARLES: Right. In a sense, I'd say that they both suffer from that same problem. I'm curious to hear about the firehose. ALEX: To actually get into what I think you're building here, we can dance around it all day and then we -- CHARLES: Just save it for the last 30 seconds of the podcast. That way there could be no -- ALEX: We're swapping JS for React router V4. Bye! It's basically this. What's happening today is that you have a URL, it's going to be parsed in a way that you've tied it to via the router map file, which every Ember app has the place to go to see all the different places that you can navigate to an Ember app, which is great. You basically taught Ember how to break your long URL string into these usable bits and that's going to give you an array of these things that internally who cares what they're called but they're called handler infos and they basically say, "The first element of this array is named application. Every Ember app has one. It doesn't have any params." The next one, it starts getting into what your URL actually is. Maybe it corresponds to the '/post' portion of the URL so that's going to be named 'post,' and that doesn't have any extra params either. Then there's this thing that is post show or something like that. That has a dynamic param because that's the part of the URL as like the '/123' and that corresponds to the post ID. It's basically, if you like thinking of things in terms of transformations or observables or mapping and functional transformations, that's taking a URL and turning it into an array of these useful POJOs of information. The goal is to keep transforming that into something eventually has enough data to display and templates and whatnot. In this giant black box of the Ember router, it's going through those transformations and then it's going to go through this long series of using these params and this useful array of POJO information, start hitting hooks on people's routes to load data. Hit before model after model, redirect all these things to give tasteful names to all the tons of validations and checks that you might want to do. You do cool things in your before model hooks, check if the current user is actually an admin to prevent them from going into any '/admin' subroute. That's a really cool place to go and it's also a great convention. If you're new in Ember app, you realize you can't go on this route. It should sort of click in your head and that sounds like they've got one of these redirect hooks to ensure that you're not going anywhere you're not supposed to go. All these things are really still to this day, extremely strong, well-designed, it went through many passes of review before it landed. I think they cater to a certain kinds of user-driven clicking around apps but they are extremely strong to this day. I think the only thing that's missing is the smell. That example I gave like checking if the user is an admin, it's a bit of a smell that is not reactive. It's a hook. If it passes, great. You're in the route. It's not going to keep on checking that. What I want to do is basically, either in addition to or as an alternative to specifying these one-off model hooks or these hooks that you, not only really just fire one time, have essentially what is an async computed property or an async validation that is upfront about things it depends on. Ember is going to be smart enough to constantly reevaluate these things as stuff changes. It can depend on not just URLs or URL parameters but it can also depend on data. If you're thinking about ember-rideshare, which again is the imaginary Ember app that it's essentially Lyft or Uber, if you have a current ride model loaded somewhere, maybe by a parent route or maybe it's some sort of service, you should be able to specify it like an async property or validation that says, "I depend on ride.state," and for all these subroutes, you would want to say that, either upon entry or any point in the future, if the state ever changes to something that I don't know how to handle that go to some default route. That would be already, particularly in my app, which is a subset of a different kind of ember-rideshare app, that would be a huge help because the only other alternative is to build a sibling-central coordinator to the router that isn't the router but has to sort of agree with it and then, every one of these frames that you might push onto the navigation stack, they have to do some little chunk of code and then invoke this logic and be like, "Did the state change? Go where you're supposed to go," and they have to do that logic. It would be, I think a great win for conventions as it has if it's a benefit to make people shout out their states in advance to empower them to shout out also their data constraints in advance so that you get things like automatic redirects and things change, I think that would be huge. I know that would immediately benefit off of it and I think it would fall in the same kind of problem solving that they worked on like Ember-related stuff which people don't realize how big a problem is until they see there's a better way of doing stuff. I think with that being there -- CHARLES: As an example, let's say that you're an admin and then all of a sudden, you got fired and there's an event that comes from a server that's this person is no longer an admin and it wipes out the Ember data store and then redirect you outside of the admin route or something like that. ALEX: Yeah, that's a perfect example. To be pedantic, I think a lot of people do hard refreshes between login/sign-off stuff but if you have it all in your Ember app, that would just happen automatically. You'd still want the ability to have more graceful transitions because one of the tricky things about having stuff driven by data is that you have this giant matrix of like, "If I'm in this state and this event happens, how do I handle it? How do I make it look well-designed to the user?" But you're not going to be able to hit every one of those constraints so to just have some basic logic that's just like, "Oops, something happened," you're not an admin so we move you to the sign-in page. For in those cases, we haven't fully filled in all those leaks. I think it would be a huge win and you can just progressively decorate things according to the common flows that people take through your app. CHARLES: You know, I'm just imagining this. Model promise, for example would be some computed property, then how would you enumerate your dependencies? Just do the mechanism that we have now? Or are you imagining something entirely new? ALEX: I don't have a strong opinion on it because the moment I start saying what that specific syntax is, more people will agree on what's missing and what we need to have, regardless and be like, "I don't like it." I'm leaning toward something inspired by a lot of my learnings from observables, which is actually we talked about last time. The whole thing about observables is that there's almost limitless flexibility as to if you're in observable, it can take that event. It has been another observable based on that thing. If a URL changes and you're listening to that via observable description, inside that, you could kick off another observable of Ajax request based on that URL and it doesn't make you enumerate all these things upfront. I think there is going to be a compromise between that. I think when you get into these kinds of problems, you run into stuff like Relay, which is familiar with -- CHARLES: I haven't used Relay. ALEX: Just the idea of dynamically collecting all of your dependencies upfront before hitting the server and asking for specific chunks of data that you need, it's a very promising idea. There's cases of just dynamicism where the data comes back from the server, then you realized that you need this other piece of data and there's no way you could have collected upfront, unless you statically wrote it upfront. I expect to find that with this approach that there's going to be some stuff where you just have to be more upfront about it. But I had a cool little strike the other day on auto-computed properties and I'll also link to that. It's a different way of running computer properties where you don't have to specify your depending keys upfront but your getter function gets passed a getter function itself. CHARLES: It's past the dependencies? ALEX: Not even that. Imagine writing a computer property and the first [inaudible] is a function that you can call to get a property off of this but also track that you've got that property. If it ever changes, it'll invalidate again. That means if you're implementing a [inaudible] in computer property, you don't have to write first name twice, both in your dependent keys and in the actual getter in your function, which I think is kind of cool. I'm trying to make that pattern work for this data loading thing so that you don't have to have this huge verbose thing. You just lift this stuff in one place. I've sensed that the magic will probably break down in some complicated cases but that's what I'm trying to run with because I think it's pretty cool and succinct and sort of the natural evolution of what people think of as computer properties. The other major constraint and this is also what we're talking about because it's one of the best kept secrets about the router or it's one of these things that everyone's benefiting from without realizing it, is that if a transition occurs in the router, everything in the router is going to be a possibly long asynchronous chain of operations that it collects all the data that it needs for the new routes to display. In that time, if something happens, if some hook comes along and has an exception, it can load data from the servers. If something happens then it just says 'transition.abort,' that's going to stop whatever transition is in place and you're going to stay exactly where you were and if you're not stuck in a partial transition state, that's pretty awesome. That's basically database atomic transaction semantics that people have been benefiting from if they've been using Ember for years at this point. But again, it suffers a problem being locked away in the router. That is a cool concept. You should be able to specify like I intend this change of the state this way and if I gave you something that is logically inconsistent or can't be fulfilled, don't leave me in a weird half-assed state that I need to somehow fix and know how to fix all the different places, where I might be kicking off this transaction. I'm trying desperately to preserve those semantics when data comes into it. One of the hardest things to do is and honestly, can be one of the hardest sells for people who are used to thinking about Ember is there's an issue of if you imagine whatever API we're talking about, it's probably going to live on the route. Some kind of hook that might be called resolve or something else, like what is the value of this context object that every function has? Is it a route? It's tempting to want to do that and maybe, that will end up winning but winning out is the best API to get people to use. The thing to realize is that there is no consistent value of this. This implies that there's a state of the world and you're looking at it and currently, these things have these values. But in the transaction phase, there is no stable 'this object' and you can wind up with some weird surprises. I know because, not actually these days but particularly, when a lot of the stuff landed and people started trying to do weird things and these transaction hooks, there's just like, "Why can't I grab the controller? The property isn't what I expected?" Honestly, all the stuff that is gross about query params because of this fundamental violation. You have something that pretends to be a property that is there today but is still driving this asynchronous thing that could fail. CHARLES: I kind of viewed this as playing an off-note in the jazz thing like you only want to reserve using this, unless you're the Miles Davis of JavaScript, don't use this. ALEX: And by Miles Davis, you just mean like the god of concurrency that's incorrect race-condition-y code. CHARLES: Right, so it's just like you've got the right reason and you can spot the one-in-a-million case, where it's appropriate. You can spot it in an instant. ALEX: Exactly. I'm not that person and I don't know too many people who are and that's not the API you want to land. I'm trying to, maybe wean people off on dependency on this because the way we've gotten around it in the past is to use again, is more discrete, get the value functions called 'get model' and 'get params.' These are all very in-depth stuff if you're pretty experience Ember developer but it's a way of getting a value from one of these parent routes when you're inside a transition and the rest the world can't see it but you can because you call this hook at the right time. It's super gross because it's just a method on a route that anyone can call in any given time, whether you're inside this transaction or not. The branching logic of, "Should I look up the data from the transaction object?" because once valid, I should have get the current value of a loaded route. It's really gross to me and it causes real problems that confuse people and causes them to write issues because they've given an API that makes them feel good about treating these things as stable objects. CHARLES: I'm trying to imagine now, just like a spike in my head. I know you don't want to get too into syntax but essentially, modeling the route tree as a set of observables, where essentially, instead of returning a promise from your model, you're just mapping an observable off of some combination of the URL state or what are the other streams of state you want to merge to realize that route. But what I'm not seeing, which I'm sure you also have the answer is the original problem, which was stack routing. What we've been talking about is making the router fully reactive like this fully reactive tree that's always on. But that problem seems almost orthogonal to the stock routing problem. ALEX: It is. It's been very tempting to combine them. Why it is such a hard problem? Because you've got navigation stack, which almost to this route hierarchy stack that [inaudible] about but they're separate so you can't really apply the same lessons. Then you've got stack routing, which is you want the ability for routes to while they're loading, reference data that is dynamically available to them. I don't have a solid answer but I would say, the one thing that I think is going to help is that you have a few options for what you want to stash how you want to represent a URL or where you want to stash your hierarchy. Actually just track it in-memory and if you refresh the page, it'd be like, "I depend on some data that I expected to be there but it's not. It transition elsewhere," which is not a great developer experience. You could want to be able to make changes and refresh the page and continue where you left off. Otherwise, URLs aren't actually used by mobile app users. But the other place that you could possibly put the navigation where event stack is in a query param because that can be fully dynamic and you can just sort of manage every single page. The most current page you've pop is just some top-level route but you're tracking the state on the side. I think if you solve the problem of being able to depend on things that aren't the URL or go through a more complex transition than what the router gives you by default, I think it would be possible to treat that query param or that thing you're stashing in in-memory as another source of data. The other thing that I want to try and make sure that this new API has is really treated dependency injection where you specify all the things that you need and you don't really care from a route's perspective where they come from. I think if you had that, that would solve a lot of problems with stack routing and where it gets data from. To be very specific, today if you were in that post '1, 2, 3' comments route and you needed to access the post model from within the comments route, you would probably do this model for post. Basically you're naming not just the model that you need. You're naming the route that you know provides it upfront, which I think is that. Actually, the real reason it's kind of the smell is that, if you ever need to change the nesting, maybe you need to introduce another level or you want to nest all that under an admin route. Then suddenly, you're asking for the wrong route name. You're not really sure all the different things you need to update if you ever change the nesting of your router. There's solutions like relative URLs that a lot of people thrown around but I think -- CHARLES: To go back in the observable world and specifically, the redux-observable world, it's like a simple map. You're just mapping down off of a global prop, you've got some tree of state and you're just mapping off... What was that like? A model hook and you're just mapping down off of that? Wherever that state lives, you're mapping to it and now you kind of slicing off your little garden hose off of the firehose. But still one huge -- ALEX: I've tried to apply observables to this problem. I don't think I've never seen the observable analogue of is this idea of dependency and injection. To model something as a stream that transforms over time, that's proven to be very useful but to sort of say, "I am an observable that expects these objects given to me," I'm not really sure what that API would look. CHARLES: I would say, just as a straw man perhaps, you have this dependency that it's a well-known location. It's a well-known name. With dependency [inaudible] in classic, it's like, "I depend on the off service. This thing called 'service:off' or whatever. Imagine that you have some pool of state and there's some key called ‘service.off' there and as long as I'm just basically basing my stream, the first thing I do is map off of this and maybe map off of another key and then combined those into a single stream, then I can be sure that I have those things at all times. If they change, my mapping function or my transformation function is going to get evaluated again. Does that make sense? ALEX: Yes, I think we should [inaudible] C without code or something. CHARLES: And maybe I'm thinking about it wrongheadedly but that would be a simple mechanism. ALEX: Could you run by me one more time --? CHARLES: Yeah. Let's say that we've got some authentication service that you want to depend on like you want to inject on it. You want to inject that dependency so why can't you base your stream off of that key? You have observable map, for example. The list of transformations that you would have to do to peel off multiple keys, I'm sure you could write helpers for it. But basically, probably if you're going to be wanting to inject multiple dependencies will -- ALEX: The problem is this. Basically, if you want to write your resolved observable, if this thing based on observables, remember that there is no this in a route because of the transactional reasons of what we've talked about earlier, what are you getting that from? You need to have something passed into you, to be like 'context.get observable blah.' CHARLES: I would just assume that it's implicit. I was thinking a bit basically, the simplest case would just be an observable that was basically taken off of the entire global state or whatever of the router or what have you. The way the redux-observable works is every single epic is what they call them is just a transformation on the global stream. Usually, the first thing that happens is they map down to the local context so the -- ALEX: Like a path? CHARLES: They have a helper like action of type, blah. You only see a subset of the actions that get maps to the Redux store. I think it's redux independent but at least in theory, every single epic is basically going off of the entire global state but the first in reality, what the first thing that happens is you're like, "I am only interested in this subset of the state," so you do a map off of the global state down to your local scope and then you work from there. In fact if you had the convention around that, you could even make that part implicit. It's like I return an observable that it's only seeing the stream of local states. ALEX: That makes sense if there's sort of canonical state of the world but what you're doing when you're transitioning into a route is trying to feel out another state in an asynchronous manner. Redux is the action causes state to change, now the state is this. But the action for type thing, I think that makes sense if you are subscribing to the world global action on this one store when you're constructing this new tentative, may not actually become the store, you're depending on values. What we need in our API is something that depends on values that are from a tentative store. CHARLES: It's similar so in redux-observable, you're mapping actions to actions and you're not necessarily mapping actions here. You want to get state into the equation. ALEX: Yeah and it's so almost observables. It's just this twist of transaction dependency injection. It sounds really over-engineered but the thing is it exists in Ember today and if it exists in a less siloed way, I would certainly benefit on it. I think everyone else would too. CHARLES: Okay. With that hand wave... ALEX: Oh, I didn't mean for that to come as a hand wave. CHARLES: No, no, no. I'm kidding because I think we actually have a lot more to talk about here and we're running out of time. One of the thing that I want to ask is, talking about redux-observable, talking about redux and stuff, have you given any thought as to what this might look as a library that everybody could use? ALEX: I basically have something that's using Ember CLI only because it's so easy to just use it as a sketch pad and get test passing but everything I'm building so far is just ES6 class syntax that can be transpiled in it to whatever. I'm actually realizing, there's a lot of overlap between some of the primitives that are involved and Glimmer so it may or may not have a pass that uses references for tracking when things change until no one to invalidate and refire these async hooks. But either way, I'm going to make sure it lives in the JS usable world and not just Ember's special object model end. CHARLES: Right. Those interfaces are pretty narrow. The things that implement those interfaces are huge and complex but the way, at least I understand it, isn't the reference interfaces themselves -- ALEX: They're really simple, yeah. CHARLES: -- Really simple. It could almost be copied and pasted and not have much maintenance overhead in there. Here's a question and this is probably getting too far into the weeds. Can you not model a transaction as an observable? Essentially, with a flatMap, you would merge in some observable into the chain that was basically a transaction of all the other observables from which it is composed. ALEX: You know, a transaction as it builds up all the new state over time could be part of the main tree and if there is an active transition, then that's future potential state that the world might become and it could be modeled as a leg of the Redux state. I think you could theoretically do that. Definitely worth a try. I don't think I would benefit too much from doing it now and I think this could be a premature optimization but I think there would be just quite a bit of intermediate object collection to express that. I think theoretically it works but how it's going to physically map to Ember in the near future, it would be harder [inaudible] in a way. There's actually a lot of stuff that is very redux-y that again, a lot of Ember people don't maybe know about because it's internal but the way that Ember [inaudible], I think since Edward brought some of his learnings of Liquid Fire back to core Ember, there's this concept of outlet state, which describes -- I'm not an expert on it -- what's rendered where and then each outlet gets a chunk. Like you said, a little piece of the firehose or garden hose, pulled off the main thing so it can just focus on the one piece of state. Those are simple objects that produce this part of this transformation process. That's kind of redux-y in the way that everything just gets a new POJO and stuff changes but it's not strictly redux, obviously and probably won't become that just because it's already good enough on its own. CHARLES: Yeah. I think it's actually good at this point to be hand wavy because the most important thing is to be non-committal about the syntax, like you said because that's when the bikeshedding begins and now it's not the phase. The phase is to come to some agreement about what is that we would love to see. ALEX: Basically, the thing is this. I think people need to realize that Ember won the bet that the URL is an important thing to build apps around and if you have a state that's representable in URL, that state should go in the URL so you can send links around and not break the web and have an app that works that's built on half-assed routing. The only thing I'm proposing is going to make that go away. It's just that there is already this giant world of stuff that's not expressible in Ember today because it is driven by state. If you make that as easy to express and as upfront to express, I think you can have shared conventions versus what everyone is building these apps that I have to do, which is to make a sort of separate router of state-aware stuff and not have to make those two things agree with each other because it's really hard. CHARLES: Right. At that point, you're writing your own framework. Maybe this is the next big thing because I feel like Ember usually has the best stuff way, way, way, way before. Now, we're finally getting to a point where everybody seems to realize that having a CLI is absolutely critical to the developer's experience and most frameworks aren't taken seriously until they've achieved that. It was the same thing with a router back in the day. I'm wondering what that next thing is. ALEX: I don't know. I don't think this is going to be it. I just think it's a good progression. I think a way forward that progress is still a pretty legit central structure to build apps around and just would be welcomed. CHARLES: When are you going to be done? ALEX: About two or three days. I don't know. I think I'm basically going to be continuing to get feedback like the way that a lot of that original router stuff came back or it's just like constantly hit people with real examples, Ember twiddles, things are just like, "Oh, yeah. That thing. That's a cool pattern. That sucks in my app. I didn't realize that until I saw this example." These things that really teach people why this is necessary because that's going to get people's urge to be like, "Well, you could just do..." Oh, you can't because the thing that's hard to explain. It's going to be a lot of that regardless and I hope that will kick off in the next few weeks. CHARLES: And the focus of that is going to be the ember-rideshare application. ALEX: I think that's a good one. This is one that everyone's familiar with. CHARLES: Have you already kind of implemented in it, like this kind of Frankenstein-ish, like this is the kind of histrionics that you have to go through in order to implement the style of routing or the style of application using today's Ember? Or have you started to begin experimentation with these new concepts and try to build out better ways of doing it? ALEX: I'm not strictly extracting it from one app. It's sort of combined. Like I said, the few different apps that we had were an opportunity to be like, "This sporadic stuff is hard." The main route recognizer approach was an example to try different stack routing pattern. But the thing that sort of working on is drawing from three different apps and slightly different takes on it. Basically, I have something that is close to being testable in one of my main apps that will be a great chance to validate if all the stuff is as nice as I think it is going to be. CHARLES: Okay. If the people want to get in touch with you, to help to contribute to the conversation or just publicly guilt you into moving faster towards it, how would they get in touch with you? ALEX: I'm at @Machty on Twitter and GitHub and also, the Ember community Slack. I think I'm going to try to get people to talk about this on channel called Dev Dx Router where it's focused on development stuff all around the router. This is kind of funny because I'm talking about this thing that I've only had maybe, 12 people take a look at and comment on and begins these conversations. I think maybe some people are going to hear this and be like, "What are you talking about?" but if it gets people -- CHARLES: No, no, no. You know, the best conversations seemed to be organized around you, man. I'm just trying to think of some of the best development conversations that I've had in 2017 and you were definitely, I would say the one who fomented them. It starts with 12 people but then, if enough people take interest and be like, "Wow, yeah. Oh, man. I didn't even know that was a problem. This would be a cool way of doing it." They have a tendency to balloon and some fizzle out and some end up with real results. Anyway, I'm looking forward to it. ALEX: I appreciate it and likewise, you're definitely one of the best people to talk about this stuff with. CHARLES: Well, I hope other people will love listening to our conversation. With that, we'll head on out. Thank you everybody if you've made it this far. As always, you can get in touch with us at @TheFrontside on Twitter or just send an email to Contact@Frontside.io. We will talk to you next week.

Jay Phelps: @_jayphelps | jayphelps.com Show Notes: 01:11 - Why is WebAssembly exciting? 02:17 - What is WebAssembly? 05:03 - Efficient Parsing 06:41 - Why didn't they just use JVM, CLR, LLVM IR, etc.? 11:35 - Using WebAssembly: The Development Process 20:30 - WASM, WAT, WAST File Formats 25:22 - wabt and binaryen 31:16 - emscripten 31:58 - Consuming as a Library 35:37 - Will WebAssembly replace JavaScript? Resources: QCon: The WebAssembly Revolution Has Begun with Jay Phelps All Things Open: WebAssembly Demystified with Jay Phelps Transcript: Coming Soon!

Jay Phelps: @_jayphelps | jayphelps.com Show Notes: 01:25 - RxJS 10:09 - Observers 17:49 - Back Pressure 22:11 - Async Iterators and Generators 31:30 - Mapping Resources: The Observer Pattern Hot vs Cold Observables IxJS redux-observable Transcript: CHARLES: Hello everybody and welcome to The Frontside Podcast, Episode #84. My name is Charles Lowell, a developer here at the Frontside and your podcast host-in-training. With me today is Elrick Ryan. Hello, Elrick. ELRICK: Howdy. CHARLES: You and I have actually been on a roll lately, podcasting the hell out of these podcast. ELRICK: Yeah, I know. That is very true. CHARLES: It's been you and me but it's feeling great. It's good to have you on the show again. ELRICK: Yes, wonderful, man. Let's keep it rolling. CHARLES: All right. We will keep it rolling. Today, we are going to be talking about redux-observable and to help us understand and plumb this topic, we have someone who's very qualified to talk about it. Mr Jay Phelps, who in addition to having been the co-creator of redux-observable, also is on the core team of RxJS, which is a fascinating library on which it's based for many years and is currently a senior software engineer at Netflix. Welcome to the podcast, Jay. JAY: Hey! Good morning, everyone. I guess it's not probably morning to the people listening but good morning to you all. Thanks for having me. CHARLES: I'm excited about it. I know that kind of starting with the fundamentals, RxJS is something that was on my radar for a few years and it definitely [inaudible] once we started using redux-observable but the whole concept, I often feel like the world kind of is turned upside down when I'm working with observables, when I'm working with RxJS and I'm curious, how did you come to be a part of that project and what are the things that you use it to solve? Why did the solutions that you generated shake out that way? JAY: Sure. I actually was not introduced to Rx until I started working at Netflix. Netflix does have a fairly solid reputation for their usage of Rx, not just in the JavaScript world but also in the server world. Netflix wrote the original implementation of RxJava and it's used heavily on our backend systems. CHARLES: For some reason, I had this impression and maybe I'm mistaken that Rx originally came out of Microsoft. JAY: Let me continue with the story. It's confusing and I can actually take a step back and clarify that point in particular. Rx itself was originally came from Rx.net, which was indeed created by Microsoft many, many, many, many years ago. I don't know the exact date. I think it was at least 10 years ago. They, at the same time created or about the same time, Matt Podwysocki who was working at Microsoft and still is working at Microsoft, created Rx.net and RxJS. Then many years had passed and originally, it wasn't as popular as it got in the coming years. After several years, some employees from Microsoft ended up coming to Netflix. Jafar Husain is one of those employees. He came from Microsoft to Netflix and he brought that Rx knowledge and that advocacy. Rx is very ingrained in the Netflix culture and is used a lot by various teams for various purposes. Then when I joined Netflix and I got really exposed to it. One of my coworkers at the time, Ben Lesh was asked by several people at Netflix to consider and look into rewriting RxJS. At the time, the version was RxJS 2.0 and while it was great, we had some specific requirements for our website and some of our other applications that we were hoping for a better performance, smaller bundle size and better debugability and -- CHARLES: Also, when I first evaluated it many years ago, it felt very much like a port from another language, in another culture as opposed to something that from the ground up, considered as a JavaScript library. Is that a fair statement? JAY: Yes, somewhat. Definitely, there were more considerations this time around when it was rewritten and originally, it was going to be Version 3 but the rewriting process took quite a while as these things usually do. By the time we got a version out, it was Version 5. We started when RxJS was at Version 2 but it already released Version 3 and Version 4 by the time it released for the new version like a rewrite had been able to get out. When I say a rewrite, I mean like from scratch rewrite. Matt Podwysocki who was the maintainer, almost the sole maintainer of the previous version, also is now on the core team of the new version of RxJS and has been instrumental in pushing back forward as well, he has far more experience with this than either Ben are I so he's been invaluable. Sometimes, we'll think to make those decisions. We'll be like, "Why was this decision made? Was it made because of .net?" and we'll just assume that and we'll want to change it but Matt has the history involved in that. He knows why things were changed the way they were. For example, we changed one of the operators, flatMap to mergeMap. We know somewhat we go at least, I don't want to speak for the entire team but I regret that decision. Depending on the day that I've been talking to Ben, I could convince him to regret that decision as well. But we thought that mergeMap would make more sense and that very few people in practice would have heard of the word flatMap before and had experience with that so -- CHARLES: I have to say both of the terms coming at it were pretty opaque. I think there was a bout of equivalence in opacity. JAY: Yeah, good to know. That's just an example. I don't want to stick too much on that topic. Maybe someday we'll go back to flatMap and the flatMap still exists. If you're a purist, you can use it. Ben was the primary person who was working on this. He wasn't working on it full time but pretty close to full time to get that initial version out. Even though I used it, my involvement with it was fairly low at that point and then my involvement after it was released got increased. I found more time and started to get more involved, particularly there wasn't a lot of code to write. I have some PRs and stuff like that but particularly on the planning and the issue triaging and PRs and stuff like that, which is a pain in the butt. It's just massive. Particularly, around the same time that the rewrite was getting finished that Angular decided, "You know what? We're going to bet on Rx. We're going to depend heavily on it," so you really can't write Angular without writing some Rx these days. You can get away with not knowing Rx very well. You could just call subscribe and then just do a bunch of imperative stuff but for the most part, the paved path is observables in so many fashions. Now, there's this ngrx. I don't know if you have any exposure to the Angular community. I have quite a bit. CHARLES: I haven't recently. Certainly, since that kind of heavy investment in Rx, I haven't been exposed to it. JAY: I think that was your question, right? CHARLES: Yeah. It sounds like there's a Java implementation that gave rise or a .net implementation or Java implementation gave rise to a JavaScript implementation and that's the one that you got involved with but it suggests very strongly that Rx is an idea and it's played out in a bunch of different languages but really, there is a shift or it's an idea about the way you think about your programs. It's clearly been compelling to you so what is that idea and what is that shift from the way we normally think about things? JAY: The idea was realized very early on -- CHARLES: Yeah, both 10 years ago, right. JAY: Yeah, exactly. They dubbed it their reactive extensions, which is what Rx stands for. Pretty much, name a language and it's been ported to that. There's RxSwift, which is super popular. There's even things like RxCpp and stuff which if you look at it, it's awkward. It seems like we got less language in the world for doing this sort of thing. I actually like C++ in a lot of ways but it was awkward stuffing that stuff in there. It's a really popular pattern and the idea is just basically going all in on the observer pattern, saying that like, "Most people are building things in which you want to be pushed information." You want to be pushed events and the data should stream to you. Modeling most problems in the world as a stream, once you get over the initial barrier of getting away from your normal historical way of looking at things and you look at everything as a stream, it comes very natural because you can actually model literally anything as a stream because it could be a stream of one, it could be a stream of nothing, it could be a stream of infinite number of events or it could be a stream of 10. You can model anything as a stream. Once you start thinking about that, it just becomes very natural and particularly on the UI side of things, I think there's been a lot of success in RxJava stuff at Netflix but RxJava is also used in Rx.net for client-side stuff as well, for mobile development. CHARLES: I remember when I first was introduced to it, I think there was a lot of confusion for me around an observer in the context of Rx and an observer in the context of classical MVC. One particular manifestation of the MVC architecture where you have these kind of mutable objects and you're observing their properties. Like key value observation, which factors heavily into certain UI frameworks. Backbone is one that comes to mind or if you're familiar with Java, basically the JavaBeans, like the property model listeners. I kind of had that conception of what an observer was versus Rx has a very, very different take on observable things. Do you think you could maybe show where they're different? JAY: You can get the normal classic observer pattern using Rx, using a subject basically. But there's a subject class, which you're not going to use super often but there are certain cases where it makes sense. Also, it depends on what libraries. It is used more often in the Angular world because you want to get a stream of clicks or something like that but -- CHARLES: So what would be the subject in that case? JAY: The subject in that case would be you're going to pipe, you're going to emit. Every time they click on something, you're going to 'next' something into that subject, like 'next the event' into the subject. Basically, a subject is a really great way to go from some imperative world to the observable world. Without having to write all sorts of custom glue, you can just basically say, "I've got subject. Any time I 'next' into it, just notify anyone who's listening to this." A subject is hot observable and that's the closest to the typical observer pattern because Rx, it's usually like observables and are usually lazy or cold. That's also what people call it. In the normal observer pattern, there is not necessarily any concept of laziness like you listen to something and that producer is already producing usually. CHARLES: Right. You hit on that and I think that was something that was surprising and kind of delightful when I first started using observables is to realize that they were lazy. Let me make sure I understand it. What was cool is like I was going through some of the demos and I had this observable and is part of, forgive my terminology but when you create a new observer, you pass the function that will get called every time something subscribes to it, right? JAY: When you create a new observer it passes a function -- CHARLES: When you create a new observable, you pass a function that gets called when an observer subscribes and the thing that you can pass is the thing that you can call next on, right? JAY: That's exactly right. When it's a lazy observable, that only get called... Actually, you know, continue. You had a point. CHARLES: I was going to say what was a surprising and cool for me was that every single thing that subscribe to that observable got its own history of that observable from the very beginning. It got its own function invocation so the first example I did, I wanted to iterate over an array and just send 10 items to the observer. Then when you subscribe, you're starting from one every single time: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10. It's not like I subscribe and one gets five of the elements, then I subscribe to another one and that one gets the next five or those two get the next five together. It's like they each get their own version of that observable. JAY: That's exactly right and then that's the main difference between -- CHARLES: -- The whole thing, yeah. JAY: Yeah. This is still observer pattern because the observer pattern, at least to my knowledge, I'm not an expert on this or anything but my understanding is that the observer pattern does not necessarily dictate hot versus cold, per se but traditionally I would say, people interpret it as typically the hot type of thing or also the multicast, if you want to be like these cool buzzwords. CHARLES: Right. What you're thinking of it is when you've got this mutable object, you just dive in wherever it is at that moment and you're now observing events from it. JAY: And that's traditionally because the observer pattern was mostly created for that model view controller type of pattern as well. You can think of like a stream of clicks: the users clicking on things where a stream of keyboard events. That is inherently a hot or multicast stream. You can't tell the user, "Start clicking," or, "Start typing. Don't type." You can tell them that but it's futile. The point being is that you can't control the stream. It pushes data to you and it's already pushing, whether you're subscribing or not, the data is flowing so the traditional observer pattern works really well for that. Then the observables, it's usually lazy, Now again, with subjects and with multicasting, you can get the same behavior. You can get that observable that is hot and shares its subscriptions so that when multiple people subscribe to it, they all get the exact same underlying subscription that is already producing data. CHARLES: I see. That makes sense with to call of multicast because fundamentally, there's really only one observer. There's only one stream and you just happened to be entering in at a certain point. JAY: Right. Then the other one is called unicast but not everyone calls it that but that's usually what it's called. CHARLES: There's a lot of terminology. With subjects, is that just a fancy way of saying a hot observable? JAY: Not necessarily but almost always. If you include them together, you're not wrong 99% of the time. But as a quick drive by definition, it's totally fine. If you're teaching someone really all the things and then they want to fundamentally understand it, it's good to have some distinctions. A subject is hot and multicast but just because something is hot and multicast does not mean that it necessarily has a subject backing it. Maybe from the pattern perspective, you could call it 'the subject' or 'a subject' but in the Rx world, it's not necessarily a subject. CHARLES: Then for cold observables, what are good things that they model, like the execution of an algorithm or something? there's an instance of that algorithm, I'm going to add two numbers or I'm going to reduce this tree of numbers into a product or a sum or something like that, where you start from the beginning, there's a clear beginning, there's a clear end --? JAY: Right. The observables are really good at modeling side effects, for example. Things you want to do like make an Ajax call or read a file or something like that. In fact, reading from a file is actually not a great use case for observables just because you want to be able to control the back pressure. We can talk about that next if you want but -- CHARLES: Yeah. Back pressure is a concept. It's crazy to me and when people use it, I'm like, "How do you even do that?" You can say it and sounds good in practice but it sounds really, really complex. JAY: I think I can summarize it for you. The back pressure stuff is basically you have a data producer and you want to be able to control the rate in which it is producing so that you do not overwhelm the consumer. You can consider two servers. You've got Server A and it's sending events to Server B. If Server B can only process 10 events per second but Server A sends 100 events per second, what's going to happen? There's a deficit. There's a huge 90 events per second deficit and that means that the Server B is going to get more and more behind. Eventually, it will just fall over and run out of memory, CPU or whatever happens. Back pressure is basically just being able to control that somehow. There's lots of ways, there's several policies of back pressure. There's buffering, there's dropping and then there's the pull model where it's like, "I'm going to tell you when to give me the next event," or, "I'm going to tell you to give me six more. I can handle six more, give me six more," or, "Here's the next one." Those were -- CHARLES: Then polling would have to be combined with buffering or dropping still, right? JAY: Possibly but it becomes one of those like you use it just as a rarity. Because the problem with buffering is that it becomes usually unbounded and that means you can -- CHARLES: It's blowing up the producer, right? JAY: Well, you risk money out of memory, mostly. It just becomes completely unbounded. You can bound that buffer and then it becomes dropping. Then if it's dropping, that means it's lossy. In a lot of scenarios, it's important not to drop information. CHARLES: They're useful stuff. JAY: Right. At the same time, there are a lot where you can drop it. Like at Netflix, we have this data pipeline thing for these logs and I got a good talk on this actually. This data pipeline is called Mantis job platform where all this data flows through. It uses RxJava almost exclusively and we have a lot of different back pressure mechanisms but the one of the most popular is dropping data. If the consumer is being overwhelmed with the number of logs and events that are matching to it, we just start dropping them because they're logs. We want to keep them but we make a best effort and it's perfectly fine to drop them. In the observable world, it's usually going to be either buffer or drop. You're not going to usually be able to control the producer's rate. In RxJava, there is something called the flowable, which lets you control the rate by basically saying, "Give me N number. I want N number more." It's become more important on the server-side end of things but a normal observables -- CHARLES: That makes sense. JAY: Yeah and there's even a library that's brand new that Matt Podwysocki came out with called IxJS. Instead of RxJS, it's IxJS and it's iterator. It's for async iterator and regular iterators, which is good for that other end of the spectrum for back pressure. If you want to be able to control the producer, you're saying, "Give me the next five." That's what an async iterator is incredibly perfectly suited for. CHARLES: The async iterators are kind of like the logical inverse of the observables? JAY: That's exactly right. It's the dual, if you want to use the computer science term, I assume you knew that already and it was that was just a plant. CHARLES: Actually, I didn't terminology, the dual but it's like through the looking glass world, right? JAY: Yeah, that's correct. CHARLES: One hand you got Alice in our world and she's looking in the looking glass world. I'm not saying which one is which but one is observables and one is the async generators. When you would want to use async generators is when you want to have the consumer driving the entire stream. JAY: That's right. A lot of people, when they discover async iterators, they'll ask me, "Jay, why wouldn't I just always use that? It seems like they're more flexible of the two." The reason why is because you can't always control the producer. The fundamental example that I was giving earlier like mouse events or keyboard events, you cannot control what the user does so an async iterator would be a very poor primitive to model that because what happens if you decide to give me one keyboard event at a time and one per second. You're like, "Give me the next keyboard event." If you did that one per second and someone is typing on their keyboard, what do you do with all the events while you're waiting? You could buffer them but that's a back pressure problem. You have to choose and it's a very poor primitive model to that. CHARLES: Yeah, that would be a terrible user experience. The users want the consequences of their actions to be realized at the soonest possible point. JAY: That's exactly right. The other benefit of observables is a slight increase in performance because when you subscribe to an observable, it sets up the chain and then now only data needs to flow through that chain. Whereas an async iterator, every time you call next, you're going to get a brand new promise. If you used it for something very high volume, you might be able to see how now you've got a lot of excess allocations: CPU cycles being used, garbage collecting for each one of those promises. In Netflix, we got a lot of streams, which are hundreds of thousands of events per second. If you do that, those allocations of those promises can certainly add up. It's just not the most efficient primitives for that as well. CHARLES: I thought that async iterators used essentially continuations under the covers and not promises. JAY: Under the covers, I'm not sure how you would describe what -- CHARLES: I'm not super familiar -- JAY: It's basically an -- CHARLES: You're talking about like yield, right? JAY: You're talking of generators maybe? CHARLES: Uh-uh. JAY: Iterators and generators are although related because a generator is basically a factory for creating iterators. Does that make sense? CHARLES: Yeah. JAY: Every time you call that factory, you get a brand new iterator out of it. But then when you have an async generator, it's basically the exact same thing except for the iterator returns a promise. You're basically saying, "Give me the next thing." You call next on the iterator and the next will return a promise and that promise will resolve with a value. Why is that useful to standardize? Because you can have syntactic sugar like the [inaudible]. I don't know if you've seen that yet but it's like a loop primitive, where you can basically loop over an async iterator without needing to deal with the whole thing and all that stuff. The same way normal async/await is helpful. This await is helpful at the same time. CHARLES: It's way above my pay grade but maybe you're using both. I'm trying to think with async the way that works with promises, right? JAY: That's correct. CHARLES: So you got both the continuation and a promise. That's even more overhead than the promise because you've got to stuff away that whole call stack and the promise and yield to it. But anyhow... ELRICK: This is very interesting. RxJS is a very deep subject. What I'm interested to know is you have RxJS, you have observables and you have redux. Where's the union? How did that come to play that you guys came up and said, "You know what? We need to create redux-observable library?" JAY: Yeah, great question. It came about fairly organically and over many month period of time, actually. We bring Rx people as I mentioned and we were using redux at the time and we're still using redux but my team that I was on was using redux and we were using redux thunk and the thunks were getting incredibly out of hand. It was very hard for us to do things that we were used to doing that were very simple like debouncing. We were like, "This is really hard. It's just to do something so simple that in Rx, it's so simple." We try to stuff Rx into redux thunk and try to make conventions around that. It just didn't work out well and then we initially made something we called thunk-observables -- CHARLES: We really wish you would have stuck with the name even you didn't -- JAY: Yeah, so thunk-observable is probably what you can expect. You dispatch a thunk which returns an observable and that observable will be subscribed to and that was it. Basically, that was the primitive so that let us have our debouncing but in practice, we found that the thunk stuff cause way more confusion and then also did not let us do composition as much as we wanted to. We then extended thunk-observable thing to get receive a stream of actions so that you could compose the different thunk-observable together and then receive new actions to cancel things and stuff like that. We eventually just figured out that having basically the idea of thunk-observables but having of them be like almost static factories that are like process managers, that's the pattern that we ended up on today. That we coined calling them epics. Basically, an epic is a function, which receives as an argument, a stream of all of the actions. It gets an observable of every action that's going to be dispatched and it was a hot observable so that means that if you happen to subscribe to it later, you're not going to get all the actions from before. It doesn't replay them. It's just all actions from the time you subscribe and continuing. Then what that function is expected to return is a new stream of actions and that stream that it returns of actions gets subscribed to by the middleware under the hood and basically, it just calls stored dispatch on anything you emit. You can imagine the simplest epic would be a function that gets streamed of all the actions, it applies a filter operation on all of those and looks for a particular action and then based on that action, it performs some side effect and then when that side effect comes back, it emits a different action. Basically, to notify redux like, "Here's the results. Here's the change. I'm done," or what have you. let you just make arbitrary side effects but isolate them in a way that fits very naturally with your UI layer that you're using and with redux itself with the purity of the reducers. CHARLES: Right. For example doing an Ajax request, I think that's a good example. You get some action in, you're getting every single action that's dispatched to the store, the first thing you want to do is you want to filter that stream to only the actions that are user click the save button so I want to save off this form. To some action, that's my save action so then my job is to now map that eventually. Out the end of that stream, I want to have like the save either succeeded or the save failed. There's two forks on an Ajax request. The fundamental mapping that's happening is either from the user click save, whatever you want to call that action, I want to map that to the user if the save was successful or I want to map that to the save failed or rejected. The hard part then is executing those side effects and then putting them back into the stream. How would you do that then with redux-observable? JAY: If I'm following correctly, the typical almost all epics, not all of them but almost all, they're going to have a filter at the top of the epic, they're going to filter out looking for the action they want and then they'll have some sort of merging strategy operator like mergeMap, aka flatMap or they have something like switchMap. CHARLES: I feel like that's one of the harder concepts for me and I want to actually going to be a little selfish here and try and bounce my understanding off of a bit so you can be like, "No, your mental model is incorrect," but we'll find out how it's incorrect. With the mergeMap, is that a way of saying, "I can take other observable streams and inline them inside this one stream." That's the way I've been thinking about it. I can go out to the rest of the world, I can have some other stream and I can inserted into the processing chain of this other stream. In this case, I do a mergeMap of the actual fetch or the actual post but it's not a straight chain of implication like do a filter then I do a map then I do this. I've got to take this other observable, this thing, this other asynchronous process, which really represents another stream and I want to merge it into this one stream. Then once that's done, I get the result. now, once that happens outside the mergeMap, now the Ajax results whether it's a success or failure, the result of the request, now it becomes the next item in the stream, which then I have to map down to another redux action. JAY: That's right. The mergeMap operator has an alias and it also used to just be called flatMap so you can envision that you are creating an observable of observables, a higher order of observable. With mergeMap, you want to flatten that. You're saying that every time I get an event, I'm going to call this projection function and it's going to return a new observable, an inner observable and I want to merge each one of those into each other so that it becomes one stream. That means you can have concurrency. That means you can have multiple and simultaneous concurrent Ajax calls that can finish it arbitrary times. I think it's important then to contrast that with some of the other merging strategies like switchMaps or concatMap. With switchMap, as the name implies, you switch between observables so at any point in time, you can only have one observable subscribes to at a time. If a new event comes along and you call the projection function, it returns a new observable. If the previous observable has not yet completed, it will get unsubscribed to and anything it was doing gets cancelled. In the example you're talking about, if you've got an epic that goes and fetches your user model every time they click this button, let's say that they can click that button multiple times and you don't want to make 50,000 requests. The quintessential example is actually the auto suggest stuff. As you're typing key strokes, if the request is in processing and they type another keystroke, you don't want to wait for them the previous request to come back and process it. It's not only wasteful because you'll process the JSON and all that stuff. It can introduce bugs because it may come back out of order. It may come back actually after your new one comes back and that can cause all sorts of crazy weird bugs. That's what switchMap is really great for. I call it implicit cancellation. Ben doesn't like that because he's saying that in a switchMap, you are being explicit about it but you're not calling unsubscribe on it yourself, which is why I call it implicit. It happens automatically. There will never a new event pipes through there. Then there's the third one which I don't -- CHARLES: So this switchMap ever make sense outside the context of hot observables? JAY: I would say that it makes more sense usually on hot observables but there are certainly cases like let's say that you've got a web socket observable and every time you get an event, based on that event, it make some other request and that other request may or may not take a really long time. But if you get another thing back from the web socket, you want to cancel the previous one. That's somewhat not a great example as well because sometimes people use multiplexing for the web socket so that it becomes multicast. But the point being is that there are definitely times where you will use merge, switch or concat. That concatMap being the third one where as you might imagine, you are concating the observables, you line them up. If I get a new event, I'm going to call my projection function, create that new observable but I'm not going to subscribe to it. I'm going to keep it but I'm not going to subscribe to it and tell the previous observable I was subscribing to has completed. In a way, you're buffering the observables and because you're offering them, you can get in trouble where you end up buffering infinite observables. Let's say the first one never completes for some reason because of a bug, you may infinitely buffer them. ConcatMap is not used that often. I'd say it's very rarely used. Its use mainly when you don't want lossy behavior. You want, at most wants semantics like you're only doing one per time. CHARLES: The difference between mergeMap and concatMap is what? JAY: It's you're not going to do concurrent. That's probably the best way to explain it. You're going to do that in sequence instead of concurrently. CHARLES: I see. Maybe an example would be like file uploads or something. You probably want to do your file uploads in parallel but let's say, you want to conserve bandwidth or you're working where you've got a browser that only supports 10 connections or five upload connections but when someone selects 30 files, you don't want to just drop those files. You want to be uploading 10 concurrently but as soon as one finishes off of that 10, you want to start up another one. JAY: That's absolutely right. Another example would be like you are going to hit some API and it could be used as one mechanism to kind of throttle yourself. If you want to guarantee that you're only connecting at most once to this particular end point at a time, no matter what the upstream tells you to do. CHARLES: Right. I could see that. That makes sense. JAY: I can tell that you're a little bit struggling to see the use cases, which is totally normal. ConcatMap is not used very often. It's for of those fundamental operations, which primarily why it's included is it's non-trivial to implement yourself. CHARLES: Are there any other mapping operators that we should know about? JAY: Those are the three primary ones. There's a couple of other like boutique variations but they're all variations on the exact same thing. CHARLES: Right. ELRICK: Do we have like mergeMap, switchMap and concatMap that are specific to management of observable and that's specific to redux-observable. There's people out there using something like redux-saga or they're trying to compare and contrast these two libraries. How do these things contrasts? Because I don't think you would use either of these mapping operations inside of a saga. Is that correct? JAY: I just want to make a minor clarification. The Rx stuff and these operators that we've been talking about: mergeMaps, switchMap and concatMap, they are not actually related to redux-observable really in any way. The only thing they're related is that you just might happen to use them. Redux-observable is actually a very tiny library and it defers pretty much everything to just normal idiomatic RxJS code and that's really the biggest pro in comparing it to say redux-saga. Redux-saga came significantly over a year before redux-observable and without a doubt, were influenced. The pattern is very, very, very similar but there are some just fundamental differences and one of those is that difference. The most obvious thing is if you already know RxJS very well, without a doubt you're going to want to use redux-observable. I could tell you that, it's not possible but it's very unlikely you'd choose redux-saga over redux-observable if you're already a really big Rx fan because you basically know how to do it. You just have to think that the items you're modeling, the events you're modeling are actions, which just happens to be a convention really. There are some other differences though between redux-observable and redux-saga and the biggest difference being that redux-saga takes this effects as data approach, which is like Haskell if you're familiar with that. In a lot of ways, it's identical with the IO monad but it basically just means that you're not actually performing the side effects yourself right then and there when you call their operators. There are special utility functions, instead there's like an engine or a middleware underneath that performs the side effects on your behalf. You create a generator and that generator is the pattern that is called a saga is what they call it. That generator yields these objects, these effects objects. One of the effects objects might be to call some particular API to make an Ajax request so you're going to yield that object that basically describes the side effect you want to perform but doesn't actually do it itself. It's just an object. In the middleware will form it for you so why would you put that indirection. The indirection helps when you want to do things like testing. If you want to test it, you don't actually need to perform any of the side effects. You can just call next on the iterator that you get back from the generator, you call next on it, you will get the side effect object, the effect as data that you want to perform and you can just assert based on that. You don't need to do mocking in all of that stuff. You just have to assert that the effects that were yielded were the ones that you expected. Now, I don't like that approach personally because I actually use redux-saga quite a bit, not lately. It was like a year ago. You end up implementing almost all, not all but a lot of your business logic in your actual tests themselves. Your test become less about testing the behavior or the outcome of a particular thing and more about testing how it gets to that outcome and what steps it takes to get there. Some people think that's fine. For me personally, it felt like any time I made a change in my saga, I had to make the exact same change in my test even though the behavior of the saga did not change. The actual observable outside side effects did not change in any way but I may be refactored or renamed something. It felt very redundant and to me, felt brittle because I started to wonder who tests the test then. If the tests are in a lot of ways are reimplementation of the saga, how did I really test that the behavior of what I was trying to accomplish really was accomplished. I'm not an expert on it so certainly, I'm sure there are people who have patterns around this that can mitigate some of my concerns. But for me, I'm used to testing Rx and I'm used to Rx in particular so the pattern for me of using observables just made a lot more sense to me. Also, with either of these libraries, the learning curve is really steep. If you don't know redux-observable, let's say you don't know Rx, learning Rx just to use redux-observable is a pretty huge undertaking. I would usually recommend people to not do that. A lot of times these days is spent me helping people who are frustrated because they dug themselves in a hole by using all these new technologies. They'll pull in TypeScript and [inaudible] with every all of the cool new hotness and I don't blame them because they've been told that this is all the cool stuff. CHARLES: We got some great advice on that that you can have one vanity technology on every project and live it yourself. Indulge in that coolness, in that hotness and do something exciting but make sure you have one vanity technology and one only. Everything else, be very comfortable with and one is just crazy experimental like this is the coolest new thing and we're going to just drop it in. But if the rest of your chassis is solid, then you can support that crazy experimental engine that requires you to stand up on the front of it and feed gas in with your mouth like they did in Mad Max. JAY: And there are exceptions obviously but for the most part, I subscribe to that. I answer a lot of questions on Stack Overflow and redux-observable. I actually get sad and frustrated when I see people that it's very clear that they shouldn't be using this. Their use case is not complex and they haven't learned Rx yet. It's one thing if you're like, "I'm learning this and I know that I'm going to have pain." That's totally acceptable. There's plenty of times where it's like, "It's totally okay that there is no concrete deadline and I can take my time," but a lot of times and I would say, a majority of the time, you're slipping on the deadline and that's not a good thing. The people who are coming to me are just frustrated. They're like, "Why is this so complicated?" You're right, it is really complicated but its complication helps with some of the more complicated use cases. That's the irony behind both redux-saga and redux-observable is that they're both really complex for the trivial use cases like all I want to do is make an Ajax call. That's it. I don't want to cancel it, debounce it, I won't do anything. I just want to make an Ajax call. They are the biggest hammer you could possibly think of. Don't use them for that. CHARLES: The thing is if you're not sure, whether you need redux-observable, then you actually don't need redux-observable. JAY: I would say that usually that is the case, yes. The same with redux-saga. They're in the exact same league. It's basically, "Do you need complex side effect management?" With the redux-saga, the complications, even though generators have been around for a little while, a majority of people still are not familiar with them because they're just not used all that often. They're kind of mystifying. I would say, they're not super hard to learn. They're just alien. Then to add on top of that, the effect is data and you've got multiple curveballs. Same thing with redux-observable, they could never use observables. It's pretty alien. The reactive programming idea and model is pretty alien. I would advise, if you're not using redux, don't even consider redux-saga and redux-observable personally. If you're already are using redux and you think that you might need something like this, experiment with the primitives, try and see how well your team can pick up RxJS just by itself. Just learn some of the regular RxJS tutorials, don't even look at the redux-observable docs because it's not useful in any way when it comes to this. Just try and learn a little bit Rx and see does it click. Because some people is like, "I don't understand why everyone thinks it's complicated. It's easy." But a majority of people, it takes a while before they get that like Neo in The Matrix, I-know-kung-fu moment. CHARLES: All right. We're almost at time but I hope that that moment comes to everybody. I think we've certainly enjoyed a lot of success with it already and I think once you do get your head around the basic use cases and you know how to do an Ajax request, you know how to do just simple saves and gets and what have you, doing the trivial things becomes easy because you know which pathways to travel. But before we head out, is there anything that you've got coming up in the near future, any talks, appearances, meet ups, anything whether you or otherwise that people should be aware of? JAY: I would say on this topic, there's a new beta for RxJS with this new way of using operators. It's being dubbed lettable operators, like the 'let' keyword but it's not related to that. It's basically just a way of finally being able to import operators and to use normal tree shaking that people have asked for forever. Because the problem with observables is that they're prototype-based methods and you can't reliably tree-shake methods or prototypes. We've been trying to experiment with ways to have [inaudible] and the lettable operator stuff is interesting to check out. It's a stopgap measure until JavaScript has something like the pipeline operator, which just actually moved to stage one. It's a brand new operator. If you're familiar with a lot of functional programming stuff like F# and a lot of the functional programming language is actually have the pipeline operator, it'll make it so that you can basically have syntactic sugar to apply a function, basically to pipe the result of a function into the first argument of the next function, etcetera. You can pipe the argument and return values through a series of functions. If you do that without this syntactic sugar, you've got this massive nested function invocation, which is incredibly hard to read and hard to maintain. That's why the pipeline operator is so great. I would encourage people, that's in beta right now. I think it's 5.5 but it's in beta right now. I encourage people to check it out, find bugs and get feedback. Maybe this is completely off base and it's not the right direction that the team should be going but it's based on a lot of collaboration, particularly with the Angular community. They've been, in particular asking for this because they're pretty big and they've got to use the Clojure compiler and all sorts of things for trying to make their bundles incredibly small. For me personally, I don't have any Rx talks coming up. I've been pretty obsessed with web assembly here lately. I'm an armchair compiler nerd. I don't do compilers for a living but I have done them personally for a number of years so I'm obsessive with web assembly. I have number of talks in web assembly coming up but just nothing related to Rx at this point. CHARLES: That's totally okay. I'm actually, also have been obsessed with web assembly. JAY: Have you guys done a podcast on that yet? CHARLES: I don't know. ELRICK: No, not yet. CHARLES: I actually started out to write my own list compiler in web assembly and they got totally derailed on the list compilers. Actually I ended up switching tracks on the whole web assembly thing but I was really, really excited about it. Probably, it was about three months ago or something like that but I'm still excited about it. I just haven't been working on it actively so I'm very curious to hear about those talks. Let's post them on the show notes and who knows? We do a lunch and learn every Friday here and usually, it's one of us getting up there but sometimes, we'll just watch a talk. One of us has been wanting to watch. ELRICK: And you're always welcome to come back to any time and geek out with some web assembly. CHARLES: I'd say, we haven't podcast web assembly, you know? All right, you guys, we've been at this for another hour. Let's go. Everybody listening, strap your headphones on, we're going down for another hour. Changing the subjects: web assembly. It starts right now. ELRICK: It's going down. CHARLES: No, but we will have to have you on for that. Thank you so much for coming on and talking with us about observables, Rx, redux-observable but if folks want to continue the conversation with you, they can get in contact with you how? On Twitter, email? JAY: The best way is going to be Twitter probably. I'm at @_JayPhelps. Thank you guys very much for having me on today. It was a blast. I love talking about this stuff. ELRICK: Thank you. CHARLES: Thank you and for anybody out there, we can also be reached at @TheFrontside on Twitter or just drop us a line and Contact@Frontside.io and have a great week and we'll see you next.

Julie Moronuki: @argumatronic | argumatronic.com Show Notes: 00:57 - Julie's Unique Origin Story Into Programming 03:47 - Good Resources vs Bad Resources for Learning Haskell 11:18 - Areas to Look at Before Taking on Haskell and Functional Programming 15:56 - Terminology 17:50 - The Haskell Pyramid 25:51 - Learning Haskell Vocabulary 28:20 - Monoid and Functor 42:06 - Advice for Someone Who May Not Be Interested in Programming Resources: Haskell Programming From First Principles (Haskell Book) Natural Language Processing (NLP) Learn You a Haskell for Great Good! Programming in Haskell by Graham Hutton Haskell: The Craft of Functional Programming by Simon Thompson Real World Haskell by Bryan O'Sullivan, John Goerzen, and Don Stewart Introduction to Functional Programming Course with Eric Meijer The Joy of Haskell Haskell eXchange 2017 - A Monoid For All Seasons Transcript: CHARLES: Hello everybody and welcome to The Frontside Podcast, Episode 83. My name is Charles Lowell, a developer here at the Frontside and your podcast host-in-training. With me today on the podcast is Elrick also. Hello Elrick. ELRICK: Hello. How you doing? CHARLES: I'm doing well. I'm glad to have you on this one. I'm glad to be doing this podcast in general. We have someone on the podcast today who I've been following for, I guess probably about two years because she published a book that has been very, very helpful to me. It's one that I recommend to a lot of people. It is learning Haskell from first principles. With us on the show is Julie Moronuki, who is co-author of that book. Thank you so much, Julie for coming. JULIE: Yes, hi! Happy to be here. It's nice to finally get to talk to you. CHARLES: Yeah. One of the reasons I wanted to have you on the podcast was because I feel as though you have one of the most unique origin stories because of programming and entering in the tech world. Most of us are curious, we either come from video games or maybe we just start fiddling with the web browser. You enter the maze from the entrance that is like hidden from all, I would say. You went straight to writing a book on Haskell, is that --? JULIE: That is what happened. In 2014 on Twitter, I met my co-author, Chris Allen and he has been trying to figure out better ways to teach people Haskell because the on-ramping, I guess of people to Haskell can be quite difficult. The materials that exist are not always accessible and people felt like they need the advanced math degrees before they can write Haskell. He was trying to figure out better ways to introduce people to it. Since I was this person who's never programmed before -- I have no background -- and then he thought, "This will be a very different experience, trying to teach Haskell to her." Because I have a linguistics background and stuff he thought, "That would be interesting too and maybe, she'd be interested eventually in doing NLP." I said, I'm not -- CHARLES: What's that? Acronym alert. JULIE: Oh, yeah. Sorry. Natural Language Processing. I said, "You know, I've never done any programming and I don't play video games and I never have had any desire to learn computer programming. I don't think I'm going to like this. I don't think this is going to last but sure, I will try," and so I did a little bit. I read a little bit of 'Learn You a Haskell for Great Good.' I've read some other things. CHARLES: This was before you guys had the idea of actually writing a book. JULIE: Yes. He had the idea of turning some of his thoughts about teaching Haskell into a book and as he would explain things to me, like the questions I had about 'Learn You a Haskell,' I'd be like, "We should write this down," and he would say, "It's so hard to write it though. It's easy when I'm explaining it to you and it's so hard to write it." Initially, it started that I was helping him at things that he was teaching me and then as we got further into the book and I started reading a lot of other Haskell stuff on my own and figuring stuff out, I was writing more and more of it. Then we were kind of equal co-authors after not too long. That's how it happened. I really didn't think that I would stick with Haskell or with programming. I'm still sometimes I'm not sure about programming. I'm not sure about this whole making software thing. But Haskell is so interesting to me that I'm still here. CHARLES: That is fantastic and it's a great story. I'm curious, when you were doing the proto-research to learning Haskell, coming from really truly first principles and having no experience of programming, what made a good resource versus a bad resource? What are the things that you gravitated towards and say, "This is really instructive." What was the tone there? JULIE: One of the major problems ahead of most of the Haskell resources that exist is they assume that you've done programming before because nobody learns Haskell as a first language so they all assume that you have done some programming before. They would make references to things that if you were a programmer, you would know what they meant but I didn't. That was one of the hardest things for me. Even 'Learn You a Haskell' does that to some extent. CHARLES: What's an example of that? JULIE: I had learned a little bit about recursion from linguistics because that's a thing in human language so I really understood recursion but most of the Haskell resources explain it to you primarily in terms of, "This will be like your loops in other languages." I'm going to be like, "I don't know what a loop is. This isn't helpful for me." There are a lot of things that I didn't understand so when people talk about Haskell as being a pure functional language, neither pure nor functional necessarily, I didn't have anything to contrast them with so they didn't necessarily make sense to me as things that make Haskell different from other languages. I didn't know what imperative programming was and people would say, "In contrast to imperative programming, functional programming does this," and I'd be like, "Okay, but I don't understand what the imperative programming way is so this contrast isn't making any sense to me and same thing with purity." There were a lot of things I had to learn, in fact about mutable state because I didn't know anything about it. I had some understanding of how computer memory works but still some of the ways that people talk about it were not obvious to me. CHARLES: Do you find that seeking out that contrast actually wasn't helpful? Is it noise since at least at the beginning, it's something you'll never do. It's like saying, "Over in France, they wear these kind of socks." Since I'm going out into the street in front of my house, I don't really care. JULIE: Right. In the beginning, it was a lot of noise and I understand why they do that because they are making the assumption that everybody who is learning Haskell has come from some other programming language, probably an imperative one so I understand why that happens but in the beginning, it was very much noise for me. I noticed a lot of Haskell resources, one of the first things they tell you is that in Haskell you can't do 'x = x + 1'. I was like, "If I'm reading this like it's mathematics, why would I think I could do that." If you come from a different programming language, you might well think that you can do that but in Haskell, we can't so making that contrast, when I didn't have that background was really just confusing for me. Now, because I teach people and most of them do have some background in an imperative language, understanding the contrast is more helpful to me but in the beginning it was just confusing and noise. When we wrote Haskell book, we tried not to make those kinds of references and like, "Let's assume that everybody is just like Julie, doesn't know a different programming language that we can contrast it with and let's try to write a book like that." CHARLES: Right. I think that's a key insight because some people would say there's a lot missing or that difference might stand out. Now, that you pointed out, I can see it but I don't think I noticed it while I was reading it. But one of the things that I like is because I also tried to learn Haskell through 'Learn You a Haskell,' and I didn't find it very helpful. I found it entertaining and it's not a knock against the authors. Some of the sketches were really cute but it was still more explaining... I don't know. It was explaining more of the how, than the why, if that makes any sense where I felt as though in your book, there were a lot more analogies to actual human experiences, using the visceral language saying, "A mono is something you can mash together or squeezed together." That really connected for me. Whereas, explaining it in terms of concatenation and laws and stuff like that. Those things seem cited to the secondary resources to the primary resource. JULIE: Yeah. I think that's kind of helpful for me too. There are different Haskell books that have, I think different things about them that are good. I forget the name of the book but Graham Hutton's book, the way he talks about recursion was really helpful to me. The way he explains recursion and of course, folds but folds are things that he's known for so those parts of that book are helpful for me. But really the best book other than my own of course, for me is Simon Thompson's. I think it's called ‘The Craft of Functional Programming' and I think it does better at explaining things just in terms of Haskell. Real World Haskell, I guess is really good. It was harder for me because I hadn't been a programmer before. I think it's got so many practical exercises that -- CHARLES: Was that the O'Reilly book by Irish gentleman whose name eludes me? JULIE: Yes, Brian O'Sullivan. It makes more sense to me now but there were things in it that are sort of programmer things. Because I'd never made software before, that were really confusing for me. But Simon Thompson's, because his book does have exercises and they were ones that I could understand and do. They were fairly self-contained. My first experience actually in writing a program that does IO was from his book and I was just so thrilled. I was like, "I got it. I did it." That was really helpful book for me but I don't see people recommend that one as often but that was probably the best one for me. CHARLES: Yeah, it's always a balance because the Real World Haskell didn't really worked for me, almost because the examples were too pragmatic or too complex and I picked this up when I was 10 years into my programming career and I struggled to follow the JSON parser example, which is parsing JSON is something that I've actually done several times in multiple languages and I still struggled with it. JULIE: Whereas for me, I don't even know what JSON is. This is not something I've ever dealt with. I know what it is now sort of, but it's still not something that I deal with very much. I was just like, "What is this? I don't even know what to do here." It wasn't quite as helpful for me. I've heard a lot of people have success with that one but I think they don't share quite the same richness of programming experience with Brian O'Sullivan. I think it's a little bit more difficult. ELRICK: These are a lot of amazing resources that I wish I knew about when I try to learn Haskell. I took an online course with, I think it's like Eric Meijer and that class was very intense. Looking back, what would you say are some areas that someone should, either start to look into before they step into the Haskell world, being that you didn't come from a programming background but connecting to dots backwards now? What would you say are some areas that people can slowly ramp up into to get into Haskell and functional programming? JULIE: When I teach people Haskell, the people who have the easiest time are people who have been writing Scala for a while and they've moved over to the FP in Scala side. When I first started Haskell, I heard a lot of people make jokes about how Scala is a gateway drug to Haskell. I think there's actually might be so truth in that because I certainly have a lot of students that were Java programmers, then they got interested in Scala because maybe Scala is better for some things than Java and then they start moving more and more over to the FP in Scala side. Those are probably the students that have, I think the easiest time making the transition to Haskell that I've had anyway. But you know, I think even JavaScript, trying to write in a more functional style and there are some resources for that and really, there's a very good tutorial about monads that uses all the code examples in JavaScript. I think a lot of the concepts that you can start to approach them from other languages. Haskell is still going to be weird in a lot of ways and another thing that works for a lot of people is going to Elm. Elm is similar to Haskell but different. I think that that has worked also for a lot of people getting them into understanding more functional programming concepts but with the much easier... The word easy is so -- ELRICK: It's like a relative term like, "Oh, this is easy." JULIE: It is. CHARLES: Easy to say, right. ELRICK: That's what I thought when I step into learning Haskell and functional programming. I was like, "How bad could it be?" JULIE: Right. Learning Haskell can be very bad. I'm not going to kid around about that. It's a shame because I don't think that it needs to be that bad but the way it's presented oftentimes, for various reasons, I think why Haskell gets presented the way it does but I don't think it needs to have it like that. The designer of Elm, whose name I'm not going to try to pronounce because I don't know how you say his last name, he really made an effort to for example, the error messages in Haskell can be very intimidating. The situation there has improved since I started learning Haskell but they can be quite intimidating and he really made an effort to make very friendly error messages, very helpful error messages. I think that it shows and then it makes a difference for people who are learning. If you start with Elm and then you do want to see what Haskell or PureScript, which is also frontend language, mostly. It compose of JavaScript but it's very Haskell-like, then from Elm, let's see if we can get a little more hardcore Haskell. I think the transition to Haskell or PureScript is easier from there. I think it does help to move in the functional direction from whatever language you're in, if you do FP in Scala or try moving to more functional JavaScript or even Elm. Then Haskell will make it more sense from there or be a little easier to approach. CHARLES: Yeah, and I definitely think that for, at least from my perspective, I've been able to take a lot of those concepts that I've learned from Haskell and then apply them, even inside Vanilla JavaScript. There are things that have become indispensable like mapping and folding and they exist in JavaScript. You can reduce arrays, which is a similar to a fold and then you can map arrays but understanding that map, the key insight for me that I got from learning Haskell is that there's a whole class of values that you can map, not just arrays. The standard JavaScript object is essentially a Functor and will get a little bit to that because for people listening what that word even means and the meta around the fact that they're all these weird words and how do I go about something I want to ask you about. But the trees can be mapped and the objects can be mapped and all of the sudden, it's like this one concept that I use so much for lists, it's available on all these different data structures and it's get me thinking like, "What other data structures can I use this operation? What are the things are Functors that I'm working with?" Really, it's changed my perspective to think about the type of the data structure, in terms of the operations. JULIE: I'm in favor of keeping the terminology that we have but just explaining it much better. That's the approach that I take but it can be very hard, especially it was your first learning Haskell. I don't know if you've seen the Haskell pyramid but to get sort of productive where you can write programs in Haskell is not a very high bar. It feels like it is when you first start but it's not really very high bar but Haskell just keeps growing and growing and getting deeper and deeper so you're always approaching new libraries that you've never seen before and you feel then you've been learning Haskell all over again because they're written in a very different style of Haskell or they have even more terminology, even more kinds of Functors that you've never heard of before or something like that so you're always approaching these things over again. It can be a very intimidating feeling and it makes a lot of people very uncomfortable and I'd say, if you like Haskell and that does make you feel uncomfortable, then you don't actually need to do that because a lot of people write Haskell very happily every day in their jobs even and don't do that. They don't mess with some of the newer, super cool libraries that have all this funky terminology and stuff. Some of them don't mess with them at all. CHARLES: But certainly, there is some concepts that are core. I'm thinking of like applicative and Functor and all these things that I'm learning about and I'm curious to hear about your experience as you climb that pyramid. What is the pyramid entailed? First of all, I'd love to hear more about it because this is actually the first time I heard about the Haskell pyramid. JULIE: Say you understand monads, then you can write really a lot of Haskell programs. Probably at some point, you will need to understand monads transformers but if you just get to the point where you understand monads pretty decently, you can write a lot of software so after that, then learning more is maybe going to improve your Haskell, maybe let you write some things that you couldn't write before but a lot of it above, not that these things are necessarily in an hierarchical progression. We cover monad transformers in a fair bit of detail in Haskell book but if you get anything beyond what's in Haskell book, one of those things that some of them are very interesting, some of them can make you much more productive but some of them are also people do them for fun to explore the space and some people love them and some people hate them. Haskell lets you do a lot of things for fun and exploring mathematics in ways that are interesting and exciting and may influence and in fact, have influenced other languages like [inaudible] in PureScript but not really necessary for basic Haskell programming. A nice thing happened while we're writing Haskell book. I was writing, I think it's chapter six, which is about type classes. I was writing that chapter and at the same time, my co-author had started writing the Monoid chapter. The type classes chapter comes in chapter six and we introduce a lot of the basic type classes: num and eq and some of those in that chapter because I do think it's important. Type classes are very special thing about Haskell so I think it's important to, at least start coming to groups at them early. Some people disagree with me about that and think they can ignore them for much longer. But at any rate, it is where it is and I felt that that was important. Maybe the real motivation for type classes, really until we started writing the Monoid chapter so he started writing that while I was working on type classes chapter and he sent me the beginnings of the Monoid chapter to look at. At first I thought, "We've got addition and multiplication and list concatenation and this just doesn't seem interesting. What is this generalization of a Monoid that I'm supposed to get from these three things? And why bother making it a type class," because additional and multiplication are already in the num type class and then list concatenation is just for list so why make this into a type class and what's that motivation there. With eq, we want a quality -- CHARLES: Is that how you pronounce 'eq?' JULIE: That's how I pronounce it because 'equal' or equality. CHARLES: Okay, so this is a type class for doing what? Making sure to being able to compare two values on the same value. JULIE: Yes and it's a weird one because for most data types, you can have an eq instance and you want probably, in a lot of cases to have that but we don't want because function is a data type in Haskell so you don't want to have an eq instance for functions and that's why equality is not implemented generally for everything. That's why it's a type class so there's no instance for functions because that's not decidable. You can't decide if two functions are equal, generally. Some functions you can but in the general terms, for datatype, you can't. CHARLES: That's actually a pretty profound statement. Proof of which is left as an exercise for the listener. JULIE: We got to the Monoid and I was like, "What is the [inaudible]," or something. It turns out that there are Monoids everywhere. There's all kinds of things that you want to, either concatenate or make a product of. Then having this as a type class and thinking of it in terms of like, "We've got this abstraction. We've got this category. We've got this algebraic structure. Now, we can look for in all these other places," because once you've named the thing, then you can talk about it and think about it in a little bit of the different way. It's like, "Now, we've got this group of addition, multiplication, list concatenation." Now, we've got an abstraction of that and we can think, "Where else can I see this pattern?" and it turns out it's all over the place. For me, that was one of our thought like, "Type classes are actually really cool and powerful and interesting thing." For me, that was when it seemed like, "The terminology is worth it because, now I want to think about finding these algebraic structures and in all these other places." CHARLES: Right and like a Monoid, it could essentially be called, if you're using a Java interface, like 'mashable togetherable' or 'concatenatable' or something like that. But there's a kind of one-to-one correspondence but it is a vocabulary that just needs to be learned. JULIE: I don't know much about category theory or anything but the other cool thing about Monoid for me was that there are almost always two because there's almost always one that's destructive or additive or concatenative and there's almost always one that is conjunctive or a product or multiplicative. It's often across product that would be the zipless Monoid that exist in base and it's a cross product of the two lists. There's almost always two, whereas when you think of Monoids in the very abstract looking category theory, it doesn't matter if it's addition or multiplication. The operation doesn't matter, whether it's addition or multiplication or concatenation or cross product because you generalize the actual operation to the extent where what it's going to produce. It doesn't matter anymore. For me, I still think of Monoids in terms of like set theory or Boolean Algebra, then that's one of the things that I think is difficult with Haskell where people talk about Monoids in terms of category theory but I think that's not very helpful for the actual programmer who has to actually deal with the two different instances like sum and products or concatenation and zipping are going to actually act different in a program. CHARLES: Right, they're going to yield a different set of values. JULIE: Yes. CHARLES: Is there a baseline vocabulary? I kind of think of it like learning a new language, right? JULIE: Yep. CHARLES: When you're learning Haskell, you're not just learning a new language. You're literally learning a new language. I could go and I could learn Japanese but it's going to be a struggle at some point. People say certain languages are hard and certain language are easy. I don't generally subscribe to that. I think that most of it is just going about and living in a place where they speak this language and you'll absorb it and it's the decision to go and live there -- that's kind of the primary one. But let's say, you're a foreigner and you're travelling to this country called Haskell that's got this strange language. Like other human languages, it's just got different names associated with different concepts and some of the concepts might even just be unique to that country. Just like when you're travelling and acquiring a human language, there's a certain level of vocabulary that you need to achieve before you can do things like buy groceries and be able to transact financial exchanges or have a conversation about the weather. What are the kind of the levels of vocabulary that you need to acquire to be operational in Haskell or I would say, even in functional programming because now that I've been exposed to this, I see it in Clojure. I actually see people doing this JavaScript and in Erlang, in Elixir and what have you. JULIE: Yeah, I don't really know how to answer this question. How to buy groceries in Haskell? CHARLES: Let me let scale that down because I had this horrible tendency to spend five minutes asking what I say is going to be single question but it's actually like 30. Let's take down the scope. When you were learning this vocabulary, at what point did you feel like you're really gaining traction? We're you really starting to connect the dots? JULIE: For me, I think when I got through Functor. It was when I felt like -- CHARLES: Functor and what comes before Functor? JULIE: Monoid. I think once you understand Monoid and Functor, then a lot of other concepts in Haskell will start falling into place because this is not obvious to everyone but I think once you really understand Monoid and once you really understand Functor, then applicatives are monoidal Functors and that's not obvious to everyone. Like I said, it's not obvious at first certainly, and monads have characteristics of both Monoid and Functor as well. Then you start saying, "There's all these other Functors. There's profunctors and bifunctors. I think once you really understand Monoid and Functor, a lot of the rest of Haskell starts falling into place and then type classes like alternative. Alternative is another kind of Monoid. We have all these other names that if you can see the general pattern of Monoids and Functors, I think to me anyway, a lot of it then just started falling into place. Applicatives to me seemed, I don't want to say obvious or simple but in traverse, it's same sort of thing so we have these other names for it -- traversable -- and I was like, "Why was it called traverse. I don't understand this word at all." But once I saw the type signature and what actually happens with what the function traverse does, I was like, "Okay, I see what's happening here." For me, those were the two big hills. Once I got through Monoid and Functor and really understood them well, then a lot of other stuff just come and fell into place for me. ELRICK: This is really interesting. How was a Monoid explained to you when you were first starting to learn Haskell? Then now, how do you explain what a Monoid is to someone that's learning Haskell? JULIE: When Monoid was first explained to me, it was the pattern of there's addition and multiplication and list concatenation so it generalize out that pattern and that was really hard for me to understand at first because list concatenation and addition are similar but multiplication is different. I was like, "What do these three things have in common?" What they have in common is that they take two values of a certain type and return another value of that type and that's the type signature of the main function, that's in the Monoid type class. But that doesn't really tell you very much. A lot of functions could do that, in theory at least. How you combine them is really what's interesting about Monoid and also what makes concatenation and addition different from multiplication. Fortunately in college, I had had a fair bit of exposure to Boolean Algebra so figuring out that like, "There's actually two basic genres or varieties of Monoid and they are disjunctive or additive or they are conjunctive or multiplicative," and figuring that out, to me I always think that Monoid should really be, maybe two different type classes, one for the additive Monoid like list concatenation and addition and things where you are adding two things like a set union. Then conjunctive, which would be this intersections or multiplication or cross products. I always think there's maybe should be two different type classes but there's not a good way to do that really in Haskell. Instead, we have this one type class and then we do this ugly business of wrapping them in different type names. CHARLES: Is that why you'll have a constructor for some so it's just a wrapper for an integer? JULIE: Yeah. CHARLES: I don't know if that's so bad. JULIE: I don't like it but -- CHARLES: Yeah. You know what? You do a lot more than I do so I'm going to take your word for it. JULIE: Yeah, that's exactly why. Sum and product are the wrappers for integers because integer doesn't have a Monoid. It has two Monoids over it. CHARLES: I see. There's lots of ways to combine integers. JULIE: Yeah and those are the two basic ones. Then because Monoids also have an identity so with semi-groups, then you get even more semi-groups for integers because you get max and min, because they don't have an identity so there's semi-groups. CHARLES: There's always risk getting down into the weeds with the vocabulary but I think that there's a message here because your answer to the question is really, "When I understood Monoid and I understood Functor," from that point on, the overhead that you had to expend to get other things was lower than the overhead that you had to expend to get those initial two things. For anyone listening, Monoid and Functor are probably opaque terms. You have no idea what the hell they mean. We've been talking about in things like that a little bit but then it's okay because they're a finite set of opaque terms and they're very achievable and once you can achieve those, then you've done 90% of the work and now, you're just combining them into interesting and novel ways. JULIE: Yes. I will say it that a lot of people do tell us about Haskell book that applicative is actually the hardest chapter in the book, not monad but applicative. CHARLES: Really? JULIE: Yeah. A lot of people do tell us that. Because that's the first time that you've taken the concept of Monoid and the concept of Functor and combining them into a new thing so then, once you've done that with applicative, then after that, really it's all downhill. CHARLES: Right. It seems like there's a couple of key insights. As you're climbing that hill, I like that analogy is like one, just understanding that there things like type classes so you've through attacking Monoid and through attacking Functor, you realize, "There is such a thing." By recognizing there is such a thing as a Functor, you recognize that there is the potential for other type classes like it. Then through combining it with Monoid, to get applicative, you can see, "I can actually compose these things into new instances of those things," and then that's either the crest of the hill or the Pandora's box, depending on which way you look at it. I think there's a hopeful message in there that if you can invest the time to learn these opaque terms and making them transparent to you, you can really, really, really lean heavily on that knowledge in going forward. JULIE: Yeah. I'm writing a new book now called 'The Joy of Haskell.' The idea of The Joy of Haskell is meant to be an intermediate book. For people who already know some Haskell but we want to make words like Functor more general, like in Haskell book we really focused on the type class called Functor when it's actually a concept from mathematics or actually originally from linguistics oddly enough but we really focused on the type class in there, rather than trying to explain what a Functors are generally. In the new book, in The Joy of Haskell, we're going to try to take a lot of these terms like Monoids and Functors and catamorphisms and all these other words that Haskell has used all the time and try to explain them generally. Then also give examples like interesting uses from different libraries and stuff like that. It'll service both, hopefully a guide to the vocabulary of the Haskell ecosystem and also some documentation and examples for libraries and things like that that are useful because these things do have uses. They do get used in interesting and exciting or terrifying -- maybe those are related -- ways. That's the goal of the new book is to try to make a guide to all of this vocabulary that Haskell use all the time. We're trying to do that. How do I explain Monoids, you asked. You've got two values of whatever type. It doesn't matter the type and in general, there would be two ways you can think of to combine them, either making a sum or a union of all the values in them or making some product of those values, if they contain multiple values or even if they only contain one. That's how I explain them now. I'm not certain that addition and multiplication are actually the best ways to start with that because addition and multiplication don't act quite like set union and intersection do. I'm actually thinking of them in terms of and this is how I explain monoids to the people now, I start from set theory and that sounds really heavy but it doesn't have to be because I think a lot of things about sets are -- CHARLES: They're very intuitive, especially if you have visuals. JULIE: They're very intuitive, for people to think about. Yes, exactly. I explain Monoid now more in terms of set union and intersection. I'm actually giving a talk in October. It's coming up in just a couple weeks at Haskell eXchange in October 12th and 13th in London and I'm giving a talk there called 'A Monoid For All Seasons' and I'm going to try to explain the theoretic motivation for Monoids and try to explain them in those terms. Semi-group is a little bit different because lacks the identity but I'll try to explain the alternative type class and monad plus this really the same thing as alternative. These things are also just Monoids so we have these different names because it's a different type class alternative but it's really just another kind of Monoid. I'm giving that talk about set theory in Monoids in October, in a couple of weeks. People keep asking me on Twitter, "What's your obsession with Monoids," because my name on Twitter is Monoid Mary so I try to explain why I love them so much. CHARLES: Actually, it's an awesome point, which I've just gotten to experience it is what you see like, "Oh, there are these abstract things," you start searching for them. A lot of times, you'll uncover them and it'd be a real timesaver. There's the thrill of unearthing it in the first place and then when you could say, "Now that I've identified this thing as a Monoid, there's so much less that I have to write." There's like less work that I have to do. It's the same reason that we write frameworks for ourselves in software. It's like, "We love Ruby on Rails because of all the work we don't have to do." Now, you have to expend a lot of energy to work with it, using Rails an example but there's lots of software frameworks. It's like, "If you can find a good persistence framework or you can find a good thing for making a library for handling HTTP requests and responses, why would you write it all by hand in the first place?" I think the thing that's exciting for me as a developer is being able to see, "Monoid is a thing. Functor is a thing and I can now actually use this and I can use it almost as a looking glass to explore the world around me. When I see something in the landscape that just leaps out through that lens is another great one." I've been on a big kick lately but being able to say, "This is going to save me so much time because of the thoughts that I don't have to think and the code that I don't have to write." I think connecting it back to the pragmatic, I certainly have become really obsessed, maybe not about Monoids but having a type class large in your mind. JULIE: I think it's a really powerful thing. Sometimes that jargon is really useful. It's useful in a sense that it like compresses a bunch of information into a single word to remember. It's like teaching my eight-year old multiplication and we were talking about like, "It's like addition," and for us adult, I'll just go ahead say, "It's associative and commutative," but showing him that you can do those things and that addition is like that too and we're talking about that and he was so excited to learn that there's this word 'commutative' that encapsulates this idea for both concepts so he doesn't have to think like, "Addition does this thing. Multiplication does this thing." He doesn't have to remember both of those things, like he just remembers, "Commutative and they're both like this." It kind of compresses that information and what you have to remember and think about. Then it does make it easier to see that pattern in other things, then we can find commutativity in other things because now we have this pattern that we can look for and we got a name for it. We can talk about it and really, there's a lot of stuff like that in Haskell where we find some pattern that we find useful or we want to be able to talk about or easily translate to a bunch of different types, not translate is quite the right word but you know what I mean, I think. Then we give it a name and we make type class for it and then it's, "Now, we find it even more place for us." CHARLES: Right. It's about thinking less, right? JULIE: Yeah. CHARLES: That's a big misconception is that it's not about thinking more, it's about thinking less. JULIE: It really is. I think it's because there's so much kind of upfront work, where you have to learn all this new stuff upfront, then people mistake that for how much work we're always doing but in Haskell it's like, "We did all this work upfront and now we're now we're not going to think about these things anymore." ELRICK: That sounds like a good title for a book, 'Learn Haskell and you will think less," but it's true. When I struggled through that online class, I came out of that just being able to pick up any functional programming language and just hit the ground running. It is definitely a plus and you will think less. JULIE: Yeah, in the long term, I think that you do. Haskell is not a perfect language. There are things that probably can be improved. CHARLES: Now, before we go, I wanted to ask you, having had this very unique on-ramp into programming, which apparently you're still not convinced about. I'm curious what it would take to actually convince you but the real question that I have is there any advice that you have for someone who does not have a stereotypical background in programming who may not think that they would find programming interesting, who might have any number of roadblocks in terms of their own conceptions about the path forward. What advice would you have for them? JULIE: I am a bit joking when I say that I'm still not sure about writing software. I don't feel like I'm good at it and I think this is really the key. There are a bunch of domains in programming that I don't personally care about. I don't want to make web apps and I have nothing but respect and admiration for people who do. To me, it's very, very hard. CHARLES: Mostly because our tools aren't the same. JULIE: Yeah and there's just so many things outside your own program, there are just so many things that you have to think about and deal with because there's the network and there's other people's computers and they might be doing in other people software and what it might be doing. It is insane so for me it's very hard. There's a lot of domains of programming that I don't care about and when I thought about programming, that's the kind of thing I would think about. I certainly knew a lot of people who are web developers or the common programming jobs, I guess. Some of them just weren't that appealing to me and I'm not interested in making games or graphics so those are the kinds of things that I thought about for programming. There are things though that I am interested in doing. I'm very interested in natural language processing and I guess that's related to machine learning. I've recently taken up an interest in things like the raft protocol, the consensus protocol. Those kinds of things interest me a lot and there's a lot of the theory that interests me. I'm reading a dissertation right now about implementing a non-strict lambda calculus, which is what Haskell is. It's a non-strict lambda calculus and this guy's dissertations are theoretically implementing a non-strict lambda calculus. To me, the theoretical side is really interesting but then I am also interested in certain kinds of software. For some reason, I have developed quite an interest in making Twitter bots. I think that the advice I would give -- I'm rambling a little bit -- to people who think they're not interested in programming so why should they learn or whatever, is just find the thing that you are interested in and there's probably a way you can make software for that and maybe that will be the thing that will get you interested. It might not be Haskell, maybe you are interested in making web apps, in which case I would say go for Elm or PureScript, obviously because I like functional programs but Haskell might not be the best first language for you in that case but find the thing that you're interested in and there probably is a way to write software to do that. There's probably something in programming that will interest you. It's such a vast field. CHARLES: All right. I really, really like that answer. ELRICK: Yeah, that's a beautiful advice. Find your domain. CHARLES: Yeah, it's bigger than you think. JULIE: It's much bigger than you think. CHARLES: And there is a place for you. Thank you so much for coming on the show, Julie. I really, really, really enjoyed our conversation. JULIE: Yes, so did I. This is a lot of fun. CHARLES: Thank you. Now, before we go, I understand that you are going to be in London, was it roughly very, very soon, you said you were giving a talk. JULIE: Yes, the 12th and 13th of October. It will be recorded for people who can't get in. It will be recorded, I believe. CHARLES: You will be talking on 'A Monoid For All Seasons.' JULIE: Yes. CHARLES: And then you've also got The Joy of Haskell book, which you're hacking away right now, right? JULIE: Yes. CHARLES: With that, thank you so much for both of you. Thank you all for listening. What's a good place for people to reach out for you? JULIE: If they're on Twitter, I'm very active on Twitter so I'm @argumatronic on Twitter and my blog is also Argumatronic and that has more contact information. CHARLES: Fantastic. We'll link to those in the show notes. For everybody else, thank you for listening. You can get in contact with us at @TheFrontside on Twitter and Contact@Frontside.io over email. We'd love to hear from you. This just in, we're running a special. If you go to our website and enter the promo code 'ELRICK20,' you can get that 20% discount on your next custom developed web application. Go check that out. Take it easy, everybody. Bye-bye. JULIE: Bye-bye.