Podcast appearances and mentions of ember data

Play Episode Listen Later Dec 20, 2019 69:47

Topics include:- 0:40 – Which concepts should a framework be responsible for teaching?- 7:20 – What does it actually take for someone to start using a framework? And Vue's multiparadigm approach.- 16:32 – What if Ember Data wasn't part of Ember?- 29:50 – Functional digression- 37:00 – Back to Ember Data!- 43:34 – When paradigms conflict with each other- 49:46 – Back to Ember Data, again! Links:- [Vue composition API](https://vue-composition-api-rfc.netlify.com)- [MobX](https://mobx.js.org/README.html)

teach framework concepts api readme mobx ember data

Katie Gengler on the tech and processes that power the Ember community

Play Episode Listen Later May 29, 2019 65:56

Topics include: 2:26 - Ember's 2019 Call for Blog Posts, and a retro on last year's call 7:50 - Why Katie's looking forward to Embroider, Ember Data, and updates to the Router 10:52 - What the process is for "finishing" Octane 14:37 - Katie's experience on the Steering Committee 18:02 - The infrastructure behind Ember Try and Ember Observer 34:18 - Common anti-patterns in apps, like deeply nested components 39:18 - Katie's preference for acceptance tests over unit tests in UI development 42:50 - Katie's stack of choice for a brand new app 44:58 - A look into Ember Observer's Code Search feature 58:06 - How much test coverage there is across Ember's OSS Addons Links: Katie on Twitter Code All Day, Katie's consultancy Ember's 2019 Roadmap: Call for Blog Posts Ember Observer Ember Observer's Code Search by Michelle Santarsiero google/codesearch

community tech processes ui blog posts embroider ember data

Grab bag: Web vs Native, stale data, and build environments

Play Episode Listen Later Jan 9, 2019 55:49

Topics include: 0:00: Making movies 05:08: Ryan Florence's tweet about Twitter App 18:08: Ember Data stores across browsers 32:58: Laravel's ascending option 35:51: YouTube transition to UI pattern 42:15: Ember's build environments Links: Sam's Star Wars movie Ryan Florence on Twitter's PWA

data native ui environments grab bag stale pwa laravel ryan florence ember data

Episode 123: All About Ember Data

Play Episode Listen Later Aug 7, 2018 27:44

Chase and Jonathan are joined by Robert Jackson for the first ever "Rwjblues" continuing series. They discuss Ember RFC 293 - or the Record Data RFC for Ember Data at great length.

robert jackson ember data

Let's be optimistic

Play Episode Listen Later Jun 30, 2018 53:06

Sam and Ryan talk about how to deal with building non-optimistic UIs using Ember and Ember Data, and how constrained visual design tools might help us build better UIs. They also answer some questions and talk about their current series on Functional CSS and Rendering tests. Topics: 0:00 – What's new on EmberMap 8:06 – Options for non-optimistic UI in Ember Data 22:34 – Visual programming Questions: 34:08 – What techniques do you have for authorization (not authentication)? 40:42 - How might you improve the developer experience of working with Ember Data? 50:14 – When can we get TypeScript in Mirage?

options optimistic ui mirage rendering typescript uis ember data functional css

Photo Uploads, Server Errors in Ember Data, NPM Dependencies and Ember CLI Addon Docs

Play Episode Listen Later Nov 8, 2017 40:37

Sam and Ryan talk about uploading images to S3, a new Storefront API for dealing with server errors in Ember Data, how to be a good community citizen when it comes to publishing consumable libraries given that our package managers now use lockfiles, and some ongoing work on the Ember CLI Addon Docs addon.

errors server docs s3 uploads dependencies addon ember cli ember data

Async Relationships, Store Forking, and Batch Saving in Ember Data

Play Episode Listen Later Oct 27, 2017 38:58

Sam and Ryan discuss community feedback on Ember Data's pain points, including asynchronous relationships, store forking and batch saving.

relationships saving batch for king async ember data

087: The JSON API and Orbit.js with Dan Gebhardt

Play Episode Listen Later Oct 26, 2017 45:07

Dan Gebhardt: @dgeb | Cerebris Show Notes: 01:33 - The JSON API Spec and Pain Points it Solves 08:40 - Tradeoffs Between GraphQL and JSON API 19:33 - Orbit.js 26:30 - Orbit and Redux 32:22 - Using Orbit 37:24 - What's coming in Orbit? Resources: orbitjs.com (Guide Site) ember-orbit Transcript: CHARLES: Hello everybody and welcome to The Frontside Podcast, Episode 87. My name is Charles Lowell, a developer here at the Frontside and your podcast host-in-training. Joining me today in hosting the podcast is Elrick Ryan. Hello, Elrick. ELRICK: Hey, what's up, Charles? CHARLES: How are you doing today? ELRICK: I'm doing great. CHARLES: Are you pretty excited? ELRICK: I'm very excited for this podcast because this is a topic that I've heard a lot about but don't know much about and it just seems so awesome that I'm just very stoked to hear all the details today. CHARLES: Yeah, me too, especially because of who's going to be giving us those details, he's one of the kindest, smartest, most humble and wonderful people that I've had the pleasure of meeting, Mr Dan Gebhardt. Hello, Dan. DAN: Hey, Charles. Hey, Elrick. Thanks for having me on. I really enjoyed listening to this podcast. It's nice to be part of one. CHARLES: It's good to have you finally on the show. We talked over chat and we talked over email and we meet every once in a while and conferences and it's great to get to share more widely some of the great conversations that always arise in all of those contexts. For those who don't know you, you are a founder at Cerebris and that is your company, which is involved very heavily in a lot of open source projects that people are probably familiar with. One of them that we're going to be talking about today is JSON API. I bet most people didn't know that you are one of the biggest driving factors behind both the specification and several of the implementations out there. DAN: Yeah, that's been a pretty core focus of my open source work for the last few years. Actually JSON API Spec, which is perhaps a somewhat confusing name for those who aren't familiar with it. It was started by Yehuda Katz in almost three and a half years ago, I think now and it hit 1.0 a couple years ago and has stabilized since then and we've seen a lot of interesting implementations on top of it. There are some exciting stuff that's actually coming soon to this Spec that I'd like to share with you guys today. CHARLES: To give us a little bit of context, why? What pain am I experiencing that JSON API is going to solve or it's going to address or give me tools to deal with? DAN: One of its prime motivators is the elimination of bikeshedding. There's a lot of trivial decisions that are made with every implementation of an API and JSON API makes a lot of those decisions for you about how to structure your document, how to include relationships and lengths and metadata in a resource, how to represents relationships from hasOne/hasMany. Even polymorphic relationships have a type of that data. JSON API has opinions about all these things at the document structure level and it also has opinions about our protocol usage, how to use HTTP together with this media type to make requests and for servers to return responses, how to create a resource, how to add resources to relationships and things like that. CHARLES: Also, it's not just this is a serialization format. It's very much like also delving into the individual interactions and how they should be structured, more about the conversation between client and server. DAN: Yeah, in that way, it is somewhat unusual as a media type that covers both. CHARLES: Can you dig into that a little bit because I'm very curious? Something made my ears prick up was when you said, it tells you how to, for example add relationships to a resource. What would that look like? DAN: A lot of the influences behind JSON API are hypermedia-related. It's influenced by RESTful principles and includes a lot of hypermedia aspects. One aspect is how a resource represents relationships in terms of the data in document, the type and the ID that specify a linkage to that another resource in the same document but it can also include links to discover those relationships. There's a self-link for a relationship and a related link for relationship and the self-link will return the data for that relationship in the type/ID pairs. The related link will return the related resources. The Spec doesn't have strong requirements or any requirements about URL usage but instead, it describes where to find resources through these hypermedia links. If you want to say, add records to a relationship, you'd follow the self-link for that relationship when that was returned with a resource. Then you would send a post to that endpoint and you would include the relationship data in the terms of type and ID pairs. It gets down to that specifications so that removes the ambiguity of how to interact with these resources and mutate them and retrieve them. CHARLES: I see, so is there an idea then that you are going to explicitly model the relationships as individual resources? Or is that the recommendation or the requirement? DAN: The link for a relationship would point to an endpoint, which would then model the relationships that are represented that endpoint, so to say just to speak a little more concretely because certainly, it makes some simple concept sound a lot more esoteric than they really need to be. Let's just talk about an example. Let's say, we're talking about articles and comments and maybe an author. Let's say, you've fetched a collection of articles from an article's endpoint and within the article resource, you would have a relationships number, which would include comments and then comments could have links, which one of the links would be a self-link and a related link. The related link could be followed to then retrieve all the comments for that particular article. You could also, if you wanted to add a comment for that article, post to the self-link for that relationships. You post to that whatever endpoint that is specified. Maybe it's 'articles/1/comments.' It could be anything that you want. Now, the Spec does have some recommendations to make everything fit nicely in terms of the URL design patterns and such but those are not, by any means required but having those recommendations just eliminates more bikeshedding opportunities. We find it that people who are gravitate towards the Spec really appreciate having a lot of these trivial decisions made for them so even if we don't want to come down and be hard line about requiring those particular answers, we can at least provide some guidance for how things can work together nicely. There's a whole recommendation section on the site for things like URL design patterns. CHARLES: Right, so things that aren't prescribed but these are best practices that are recognized. DAN: Yeah, exactly. CHARLES: A question then that comes to mind, it sounds like JSON API solves a lot of these bikesheds or just kind of comes in and takes one side or the other for modeling both the resources and the relationships between those resources so there's the... I don't want to call it a schema but the boundaries around which resource are very clear and where they live and how they connect together. I was hoping we could maybe contrast that with some another approach, which is also become very popular and that's the GraphQL approach where you're essential assembling views at runtime for the client. It's very easy to marshal the data that you need to present to your view because you've got only one endpoint, as opposed to having to coordinate between them. I can understand the appeal of that and I was wondering if you have any insight into what the tradeoffs are between the systems and what are some of the capabilities that one can do that the other can't. CHARLES: Yeah, sure. I'm glad that you brought that up because I feel like GraphQL has become a real juggernaut, at least because of its marketing. It's very effective in being marketed for its use to developers and it's capabilities versus REST, as if a RESTful system can't possibly achieve the same outcome or the same efficiency. I'm glad to compare and contrast the two. To be honest, one of our short term goals is to better tell the story on the JSON API site, which was always a kind of a more technical spec-y site and a marketing site. That hasn't really helped its uptake as much as it could as some of the GraphQL sites are very sleek and polished. Anyway, let's get down to it. GraphQL allows you to basically define the data that you want for a particular view and that can bring together multiple related resources. It defines a way to specify exactly which fields you want in that graph of resources. We'll just stick with the articles, comments and authors example. You can specify that you want a collection of articles and perhaps the comments-related to that and the authors and you could have it all assembled in a single response. JSON API also allows you to do just that. It allows you to make requests for multiple related resources to constrain the fields that are returned for each resource and to include all of these related resources in a single document. The main difference in the representation is that JSON API requires that resources only be represented once in a single document. GraphQL may have repetition of resources throughout the document that's returned. For an instance, your articles that may nest authors and those authors like Charles Lowell, may have written three of those articles and that representation of that author is going to be repeated in that JSON API compound document, which is a term for document, which has a primary dataset combined with related resources. That single author would only be returned once as related resource and the linkage between the primary data and the related resources would be established to type/ID pairs. Instead of having the author represented three times, the same type/ID pairs would just be providing that linkage to the same author and that author resource would only be represented once. This happens to be ideal for client-side applications that number one, basically want to minimize the size of a payload that sent. Number two, don't want to after-handle repetition of data by doing extra processing of pushing the same record multiple times into a memory store that is keeping that data. I think that GraphQL is well-suited to applications that request data and display that data pretty much as returned. There is no intermediate holding onto that data in, say a memory store for later access. Basically, it lines up well with a component library link React, which wants to display that data that's returned from the server. If it wants to display that collection again, it will simply request that collection again and pretty much throw away the data once it has been rendered. CHARLES: I can see that. Dan, you and I might be some of the only folks who remember. I don't know if you ever did any Microsoft Access Program. DAN: Yes, I did, believe it or not. CHARLES: Doesn't it feel a little bit like the Access pattern all over again, where you have your components requesting data from basically, constructing a query and requesting it -- DAN: Yeah. CHARLES: And then throwing it up on the screen. DAN: You're going deep there but I do remember that. Definitely, there is that same paradigm. CHARLES: It's really powerful. DAN: It is and it's pretty accessible too because it's a direct representation of what you've requested and there's no intermediate processing. I guess the question is, whether that intermediate processing provides some value. Actually, holding onto that data provided some value because as far as I'm concerned, GraphQL is great for that rendering of DOM data, where the data has no meaning except outside of the rendering. But if you want to actually have models that have some intelligence about that data, then you want to use a store to keep those models in and you want to be able to reuse those models for other purposes. CHARLES: What might be an example? What's a concrete use case that we can ground this discussion? DAN: I would say that the big one is offline. You simply can't have just DOM data that's useful in any way in an offline application or an optimistic application, where you are doing some things client-side and only say undoing them if a request fails. But if your data is DOM and only structured for a particular view, then all you can do with that is redisplay that view. But if you understand the schema of your data and that data is available in a store, then regardless of whether you have a network connection, you can actually display that data in different ways. If that same article shows up in a collection in a list, you could also display that article on its own in a different format with more fields. If you want to, say allow editing of that data, you could allow for an editor when your app is offline. Allow changes to be made to that data and then redisplay it because you understand the fields that are in that data. CHARLES: Right and then at some point later, then spool those changes back to the server. DAN: That's right. CHARLES: It almost sounds like, ironically if a system like JSON API where you have very concrete boundaries around each of the underlying resources in your data model, it allows you to essentially do rich-querying on the client and not just the server. DAN: Yes, that's absolutely true. CHARLES: Because I feel like what you just described to me it's like, now we have some sort of store over which we can map all kinds of different queries to our own liking and there's no dependency on the server. DAN: Yeah and if you just want more web app to be pretty much a view representation of what's on the server and without additional intelligence, then GraphQL really lines up well with your needs because any extra processing you're doing is just not valuable to you. But I think a lot of the really interesting things being done in client-side applications are where your client application is pretty loaded with a lot of intelligence and you're out there autonomous and able to make sense of data. In that case, then thinking about the data only as it pertains to views is not nearly as powerful. CHARLES: Right, so you could do something like that with GraphQL but then you would have to, essentially structure your queries such that they drew the boundaries around the individual resources anyway, rather than composing them on the server. You'd have to query them discreetly into a store and then run your local operations. Then I guess at that point, it's like what are you doing? DAN: Yeah, you're still doing the extra processing of handling the repetition of any nodes that repeat and such. That's just extra processing you have to do but I agree that you certainly could structure your GraphQL queries to return data that is then loaded, say to a store that really has awareness of the data types but I don't think that is -- CHARLES: But then you're defeating the purpose, right? DAN: Yeah, it's not its selling point and it's not its strong suit. CHARLES: You've done a lot of work on the JSON API Spec. JSON API allows you to fetch discrete resources and their relationships but still, keeping one representation of each resource in the payload so it's optimized for wanting to do client-side processing and have intelligence based on these entities, which are in a store. You actually maintain a fairly mature, at this point, framework called Orbit, which helps you do some of these things. Now, what Orbit does today and I understand that you've got a lot of new features that are really exciting, that are coming down the pike. Before we get into those, what is Orbit and what do you use it for and how does it use JSON API? DAN: Orbit is a data access and synchronization library, which sounds sufficiently vague because it has a lot of low-level primitives for structuring client-side. Also, actually isomorphic can be run on the server and nodes so it's not even only used for client-side purposes but that was its original purpose. The abstraction that it includes are allowed for synchronization of data changes across multiple sources of data. Source of data might be represented by, say a JSON API server, an in-memory store, an IndexedDB database in your browser, a local storage, all of these sources of data can support an Orbit interface, which provides their access to their data and also broadcasts changes to that data. In order to coordinate the changes across multiple sources, say to back up all of your data that's in memory to IndexedDB source, you can observe the changes on one source and then sync those changes up with another. For instance, you want to structure an offline application which you have been in-memory store, which uses client-generated IDs, which then syncs up with a backend JSON API source and every change that gets made to the memory store needs to be backed up, you could configure multiple coordination strategies between the sources to make sure that the data flows so that every change that is made to the store is immediately backed up to IndexedDB. If it can't be backed up, then it fails. You can add some error handling and then when you're online, you can then also sync those changes up with a backend so you're basically pushing those changes that are local to a remote store and you're not slowing down your offline app, which you're communicating with optimistically and then only handling, say synchronization failures when there is a problem. In order to handle those problems, Orbit sources are very deterministic about their tracking of changes and they provide get-like rollback capabilities so you can look at the history of changes to a particular source and reset the history to any point there and basically handle conflicts and merges in a very get-like way. Often I use cases, the primary driver of Orbit's whole architecture, I realized that it needed to be able to give you the tools to handle any conflicts that happen when changes get sync up. Also, give you the different tools to model all the different places of data is kept in order to support the offline mode. That's a kind of a broad overview of Orbit. There's a new guide site, OrbitJS.com for those who want to dig a little deeper into it. The data is structured in the JSON API format internally to the store and the standard operations are very much influenced by the standard JSON API protocols that are allowed in the base Spec over creating records and removing records and all that crud for both records and relationships. That's where JSON API comes into Orbit. CHARLES: Right, I see. The primary use case for Orbit is offline. Is that fair to say? DAN: Yeah, that was the primary driver, although it's just not the primary -- CHARLES: It seems like you could use this in a lot of places where I might use Redux or something like that, like on the server to model... I don't know, a chat app. DAN: Yeah, definitely. CHARLES: I have a bunch of different information streams coming together and how am I going to merge them and make sense of them. DAN: Yeah, in fact, that it's primitive level. Orbit has essentially an async redux-like model for queuing up changes and applying those changes. The change sets are all immutable. There's actually a lot of immutability use here throughout the library. In order to ensure that the changes that are applied are tracked deterministically, we just can't have those changes mutating on us. There is definitely some overlap with Redux concepts in terms of the general tasks or action concepts in Redux but instead of Redux's synchronous approach, everything in Orbit is async. CHARLES: What does that mean? Redux is synchronous in the sense that there's a natural order to all actions. For those of us familiar with Redux, are you saying it would be like a store where actions can be dispatched at any time or is it more like, I've got multiple stores happening and I need to resolve them somehow so each one is synchronous? How can I make sense of that? DAN: In Redux, the actual application of an action is performed synchronously. CHARLES: Right. You can have asynchronous processes but there is a natural order to the actions that those asynchronous processes yield and then those are applied synchronously to the Redux Store. DAN: Yeah. To compare and contrast Orbit and Redux, I guess you'd first have to say there's a primary difference of -- CHARLES: I think there are a lot of people are familiar with Redux. I think it's not so much to compare and contrast it but just to use it is as an analogy of like, "Here's how it's the same here. Here's how it's different," because that's compare and contrast. DAN: There you go. CHARLES: But not in terms of evaluating them but it's like, "Maybe I should be using this instead." DAN: Right, they are sort of on different levels, although there are some primitives in Orbit to it and it's shipped across multiple libraries. There are some primitives, I think that could be useful outside of the main Orbit data application. Anyway, the way that Redux state changes are applied, the function is synchronous is all I was getting at and on Orbit, every state change that applied to a source is asynchronous so the result is never applied immediately. You'll always get a promise back and you'll never have that application happen immediately. That's one clear distinction. Another is that a redux has a big singleton global state for the entire application. Orbit very much has a model of state per source so there can be any number of sources in a particular application and the source might be an in-memory source or might represent a browser storage in XTP or might represented a socket that streaming data in. All of these have state at different, temporarily distinct state that even if they all converge to a common state, the Orbit models separately so that there's a set state per source. I'm just contrasting the global apps state that exists in Redux with the per source state in Orbit. CHARLES: It sounds like there's nothing that would be fundamentally incompatible of using Orbit really in conjunction with Redux, where Redux is kind of a materialized view of all of your different data sources presented as what you're going to render off of, right? DAN: Yeah. You could use it in a similar way to Redux Saga, where Orbit fills the role of Saga, where it's doing the asynchronous actions that results flow back into the Redux state. CHARLES: I'm just imagining having one big global atom, which is your Redux store and now I'm saying, prescribing this is an optimal architecture but I'm saying, one way it could work is it picks and chooses and assembles off of the different sources as new data becomes available. As the states change for those sources, it can be integrated into a snapshot state, which is suitable for rendering or provides one view for rendering. DAN: Yeah. You're basically talking about the in-memory source, perhaps merged with other applications state, which is not so resource-specific and that is possible to model. CHARLES: What I think I might be hear you saying is you could also just use another source which is the merge itself. DAN: Yeah. I'm not sure how much we want to continue this thought exercise because the architecture becomes almost not something I'd recommend. But I would actually like to explore how Orbit and Redux could be used together optimally. I played around a bit with Redux but I have not written a full-fledged application with it, other than a [inaudible] location. I definitely defer to you for Redux best practices and such and how people are using it in real world applications but I'd be really interested to talk that over again soon. CHARLES: Well, I just certainly don't count myself a Redux expert, although we have developed some applications with it. We'll put that on the back burner or something to explore it later. I will say this, I find Redux to be both wonderful and terrible, kind of in the same way the Java is both wonderful and terrible. We'll leave it at that. DAN: Okay. ELRICK: That was going to be my question. This is why I was very excited to hear about today was Orbit because I've heard so much about it. In terms of the implementation of Orbit into an application, what would that look like from a high-level? Has anyone used Orbit in the production app? Have you built any apps using Orbit? DAN: Yeah, definitely. There are people using Orbit with React, with Vue, with Angular and with Ember and there's an integration library called ember-orbit which makes Orbit usage really easy in Ember. In a lot of ways, working with ember-orbit feels a lot like working with Ember Data but it allows a lot more flexibility. I suppose one of its strengths and weaknesses is there's a lot of configuration that's possible because there's a lot of possibilities. The internals are exposed of how data get synchronized so you can define your strategies and sync up different sources. In terms of how it's actually used in an application, you'd start by modeling your data in terms of the resources that are in the application. You'd have a schema that defines your articles, your comments and your authors, just to keep that example going. Then that schema would be shared among all the sources in your application. You would have one source, say that might be the in-memory source and another source that is the representing a browser storage so you could, say swap out either local storage source or an IndexedDB source and use either one to provide that backup roll. You would declare those sources, you connect them to each other with strategies so that, say when memory storage changes, you would then sync that change to the browser storage source. Then you'd have back up and you'd be able to then, refresh your page and view the same data you were looking at before. Now then, if you probably want to wire up a remote source so that you're communicating with the servers so you bring in JSON API source and you would then set up a new strategy for working with that. You have to decide like, "When my memory storage changes, do I want this change to happen optimistically or pessimistically?" By that I mean, "Do I only want it to appear successful if it's been confirmed by the server." Depending upon whether you want to be optimistic or pessimistic, you setup your strategies a little differently. If you handle this change pessimistically, you wanted to block success on the successful completion of pushing of that change to your remote server. You have the set of tragedies that define the behavior of your application and then doing your crud operation is probably pretty much directly with your memory source. Then if you wanted to, say do an edit in a form, you might fork the store, now the store keeps its data in an immutable data structures. That forking that store is very cheap so you don't have a bunch of data that's copied. You're just keeping a pointer to that and getting a new pointer to that same immutable data structures. Every time they get changed, there been new references. There's an immutability under the hood but you're pretty well insulated from the annoyances of working with that immutable data structures. At that form, you make your changes, you then merge your changes back, you'd get a condensed change set of operations that then can flow through your strategies. It flow through to your backup source. It could flow through to the back to the server. I think it would feel pretty familiar for users of Ember Data because there are a lot of the API influences came from that library. But obviously, people are using just plain Orbit with other libraries, with other frameworks and finding it useful there but it definitely involves a little more configuration up front to do all that wiring that might be more implicit in library like Ember Data. CHARLES: I understand that before we go, there is some pretty exciting new things coming in Orbit. Do you feel like you're ready to mention a couple of those things or they've been kind of mixed in with the conversation? DAN: Let's see. I have the guides up, which I mentioned, which is pretty new in the last couple of months. In the last year, we did a rewrite and Orbit is now completely in TypeScript and there are no external dependencies. For a while there, I was using RxJS and observables internally and immutable JS so there's now an internal immutable library. It's lighter-weight with fewer dependencies now. I'm excited about that and finally feel like I can recommend people digging in with the guides that are up. I'm hoping to get up the API docs soon. I will say I'm excited. I just got back from a retreat in Greece. Séb Grosjean who owns the company, BookingSync does this amazing thing with the Ember community, where for the group that's working on Ember Data, he invites them every year to come to his family's place in Greece. He grew up working with his family on his rental properties, which was the inspiration for his company, BookingSync and said, "This is a fantastic opportunity that for us to get together and collaborate in a really nice place," and I had a really productive time this last week. This is the very first time I had gone. It was just fantastic and I worked with the Ember Data team. Igor Terzic and I spiked out some interesting collaboration between Orbit and Ember Data so I'm really looking for it to see where those go and hopefully, we'll see a little bit more Orbit, either directly or just through influence appearing in Ember Data. I'm looking forward in working more closely with the Ember Data team. We'll see what comes of that. CHARLES: Yeah, I, for one am very excited to see it. I'm resolved now. I'm just looking at these guides. These look fantastic and I'm resolved to give Orbit, at least a try here, either in some of our applications or maybe trying to spin up some new ones and have it the basis for some of ideas I've been playing with. DAN: That would be awesome and there's a [inaudible] channel, which I hang out into if you have any questions, if anyone out there does. CHARLES: Before we go, if anyone is interested in JSON API, is interested in Orbit, is interested in Cerebris, we mentioned a lot of things that in one way or another, map back to you. How do we get in touch to find out more about these different entities/projects? DAN: I'm at @DGeb on Twitter. My company site is Cerebris.com. Also check out, OrbitJS.com for the new guides. Reach out to me. I'm on the Ember Core Team so I'm also hanging out in the Ember community Slack, depending upon what you want to talk with me about. I'm in all these different places so I love to hear from you all. CHARLES: All right. Fantastic. We'll make sure that we put those in the show notes and I guess that's about it. Do you have anything else you want to leave folks with, any talks, papers or big news coming around soon? DAN: Something that we didn't really get a chance to talk about today, which I'm really excited about is JSON API operations, which is an extension to the base Spec, which I'll be proposing very soon. There's a future to the JSON API once it hit 1.0 a couple of years ago. It didn't just stop. We're looking at different ways to extend the base Spec and use it for different and interesting purposes. JSON API operations, I think one of the most interesting ones. The idea is basically to allow for multiple requests that are specified in the base Spec to be requested in a batch and perform transactionally on the server so the Spec will define how would each request gets wrapped. Each operation very much confirms with the base Spec concept of a request. For implementations, there's a lot of opportunity to reuse existing code for how to handle each particular operation but to provide a whole new set of capabilities by allowing you to batch them together and process and transactional it because it just unlocks a ton of different things you can do, all based on the same base concepts from JSON API. I'm really excited to have something to announce soon about that. CHARLES: That sounds like it might solve a lot of problems that are always associated with those things. It always comes up. What's our batch API look like? I don't think I've been on a project that didn't have a months-long discussion about that. I ended up like kicking it down the road and I'm just flumping something in place. DAN: Yeah, all those messy edge cases where people figure out how do we create multiple related records altogether in a single request and people do it ad hoc and do it with embedding and such and want to standardize that in the same way, that we've standardized the base operations. CHARLES: Well, that is really exciting, Dan. I wish you the best of luck and we'll be looking for it. DAN: Thanks a lot. Thanks for having me on, guys. CHARLES: It was our pleasure. Thanks. With that, we will say goodbye to everybody. Goodbye, Elrick. Goodbye, Dan. Goodbye everybody listening along at home. As always, you can get in touch with us. We're at @TheFrontside on Twitter or you can see our website at Frontside.io or just drop us a line at Contact@Frontside.io. Always love to hear from you with new podcast topics, anything that you might be interested in so look forward to hearing from you all and see you next week.

reach id greece saga dom fantastic depending react slack goodbye api redux java orbits ids js spec pain points vue danone angular restful typescript graphql grosjean gebhardt rxjs frontside dan it json api yehuda katz dan yeah dan you elrick dan there indexeddb charles lowell charles you ember data

FastBoot, Structural Components and Ember Data Transactions

Play Episode Listen Later Aug 29, 2017 33:20

Sam and Ryan chat about adding FastBoot to EmberMap's codebase, the difference between reusable and structural components, and the road to adding transactions to Ember Data.

components transactions structural fastboot ember data

069: Redux Part II with Toran Billups

Play Episode Listen Later May 11, 2017 40:28

Toran Billups @toranb | GitHub | Blog Show Notes: 01:44 - New Developments in ember-redux 04:23 - New Developments in the Wider Redux Community 06:26 - Using Redux in Ember 09:40 - Omit 10:45 - Reducers 25:42 - Fulfilling the Role of Middleware in Ember 28:12 - Ember Data in Redux-land 31:24 - What does Toran do with this stuff?? Links: The Frontside Podcast Episode 55: Redux and Ember with Toran Billups The Frontside Podcast Episode 18: Back-End Devs and Bridging the Stack with Toran Billups redux-offline ember-redux-yelp create-react-app "Mega Simple redux” Twiddle ember-concurrency Thomas Chen: ember-redux The Frontside Podcast Episode 067: ember-concurrency with Alex Matchneer normalizr Rich Hickey: Simple Made Easy Other Noteable Resources: ember redux: The talk Toran prepared for EmberJS DC in April 2017 github.com/foxnewsnetwork/ember-with-redux Transcript CHARLES: Hello everybody and welcome to The Frontside Podcast, Episode 69. My name is Charles Lowell. I'm a developer here at The Frontside and your podcast host-in-training. With me Wil Wilsman, also a developer here at The Frontside. Hello, Wil. WIL: Hello. CHARLES: Today, we have a special guest, an actual elite member of the three-timers club, counting this appearance. We have with us Toran Billups. Thank you for coming on to the show today. TORAN: Absolutely. I'm not sure how the third time happened but I'll take it. CHARLES: Well, this is going to be the second one, we're going to be talking about Redux and then I believe you're on the podcast back in 2014 or 2015. TORAN: That's right. CHARLES: That's one of our first episodes. Make sure to get in touch with our producer afterwards to pick up your commemorative mug and sunglasses to celebrate your third time on the show. Awesome. I'm glad to have you. We actually tend to have people back who are good podcast guests. TORAN: Thank you. CHARLES: Yeah, I'm looking forward to this one. This is actually a continuation of a podcast that we did back in January that was actually one of our more popular episodes. There was a big demand to do a second part of it. That podcast we talked about the ember-redux library, which you're a maintainer and just kind of working with Redux in Ember in general. We're going to continue where we left off with that but obviously, that was what? Almost six months ago? I was wondering maybe you can start there and there been any kind of new developments, exciting things, what's kind of the state of the state or the state of the reducer or the state of the store in ember-redux. TORAN: For ember-redux, in particular, we're working on three initiatives right now. The first is making the store creation more customizable. A lot of people that come from the React background in particular are very used to hand crafting how the stores put the other with the right middleware and enhancers and reducers and that's been fine. I wanted to drop people into the pit of success and everybody's cool with that but now we're getting to a point where there are people wanted to do different things and it's great to open the door for those people if they can, while keeping it very simple so we're working on that. We have here that's just undergoing some discussion. We're also, just as the wider Ember community -- you guys maybe involved in this as well -- and trying to get the entire stack over to Babel 6, the ember-cli Babel 6.10 plus stack. There is a breaking change between Babel 5 and 6 so we're also having some discussions about the ember-redux 3.0 version bump at some point later this year, just because we really can't adopt this without introducing basically a breaking change for older ember-cli users. CHARLES: Just in general, this is a little bit off topic, what does it mean to go from Babel 5 to Babel 6, if I'm an add-on author. TORAN: I would probably ensure that need to speak more with Robert Jackson about this. We just kind went back and forth because I thought I had a Babel compile error. He's like, "No, you're missing this dependency which is the object spread." Unfortunately, the object spread is rampant in React projects and this is totally cool. I had to actually add that and that's just a breaking difference between these two. If we adapt the new version of this in the shims underneath of it as an Ember 2.43 user, if you're on node four which is still supported, you will break without this. I'm trying to get some discussion going about what we should do here and if we even should push ahead and just say only node six is supported. There's some discussion and then back to your original question, the third piece is we've introduced the ability to replace the reducer but we need to get some examples for hot reloading the reducer. That's a separate project but it needs to be enabled by ember-redux. Those are the three main initiatives. CHARLES: Being able to you hot load your reducers, just to make changes to your reducer and you just thunk them into the application without having to lose any of the application state and that one of the reasons that's possible then is because you're reducers have no state themselves. They're just pure functions, right? TORAN: Exactly. CHARLES: Okay. Awesome. That sounds like there's a lot of cool stuff going on. Beyond ember-redux, are there any developments to look for on the horizon in the wider Redux community that might be coming to Ember soon? TORAN: Actually one of them is actually fairly new and it's already in Ember and just because I have already got a shim up for it is redux-offline, which I remember you had Alex on two episodes ago about breaking your brain around Rx. I feel like that happened for me trying to build apps offline first. This is, of course just another library that can drop in that place nicely with Redux but I feel like the community, at least it's got me thinking now about what an absolute that would really disrupt someone who's a big player today. I feel like you've built a great offline experience with true and well done data syncing. You could really step in and wreck someone who's in the space today. CHARLES: Right, so this is a sim around... what was the name of the library again? TORAN: Redux-offline. CHARLES: Okay, so it's just tooling for taking your store and making sure that you can work with your store if you're not actually connected to the network and like persisting your store across sessions? TORAN: Yeah. It uses a library that called redux-persist that takes care of and kind of hydrating the store if you have no network connectivity. But it's also beginning to apply some conventional pattern around how to retry and how to roll back. It's just an interesting look at the offline problem through the lens of an action-based immutable data flow story. It's interesting. I don't have a ton of experience with that kind of tweet and rewrote my Yelp clone with it and that was tough. That's what I mean by this. It's like I thought this is very trivial but you have to do a lot of optimistic rendering and then sort of optimistically gen primary keys that gets swapped out later and it's tricky. If you've never done offline first, which I have not, I just think offline is pretty cool and along those lines, there's been a lot of discussion around convention. There's of course, Create React app which is like a little library or CLI tool to kick off your Redux in React projects. It's kind of ember-cli, very trimmed down version of that right now and that's just getting incrementally better. Of course, you guys are in the React space so you may touch upon that story if you haven't already. CHARLES: All right. We talked to the very high level, I think the last time we had you on the show but now that the idea is gaining traction, kind of delve into more specifics about how you use Redux in Ember. I asked at the end of the last podcast, let's step through a use case like what would deleting an item look like in ember-redux land. Maybe we could pick that up right now and just understand how it all connects together. TORAN: Yeah, absolutely. Without understanding how this entire flow or this event bubbling happens is hard to get your head around it. The process we're going to walk through is exactly that use case you laid out, Charles. We're going to have a button in our component and that button, on-click the idea is to remove an item in a list that we happen to be rendering, let's say. If this is a child component like the very primitive literally button that you have and you just have your on-click equal probably a closure action in Ember, the parent component or the outer context is going to be responsible for providing the implementation details for this closure action and what it does. This is kind of the meat of what you're getting at. The high level here is there is a single method on the Redux store that you have access to and it is called dispatch. The nice thing about Redux again, the API surface area is very small. It's just a very handful of methods you need to get your head around. This one, in particular takes one parameter, this dispatch method. That parameter is a JavaScript object. Now, if you're just playing around, you just want to see the event flow up, there's only one requirement asked of you and that is type property. This JavaScript objects have a type that is often a string so it's very human friendly, you just put in there and the string remove item, let's say. Now of course, if you want to remove a particular item in this remove example, you of course want to pass some information as well beyond the type. The type is mostly just a Redux thing to help us identify it but in this case, you'll definitely know the primary key or the ID value, let's say of the item you want to remove. In addition to type, this JavaScript object, let's just say has an ID property and that can come up from the closure action somehow if you want. Once you click this, what's going to happen is you're going to fire this closure action, the closure action is going to invoke dispatch with the JavaScript object and dispatch is going to run through the reducer which is the very next step and what we do is we take the existing state. Let's say we have a list of three items, that's going to be the first argument now in the reducer. The second argument is this action which is just, as I describe it a JavaScript object with two properties: type and ID. You can imagine an ‘if' statement, conditional switch statement that says, "Is this the remove item action? If it is, okay." We had the ID of the item that want to remove and since we have a dictionary where the primary key of the item is the ID, we can just use lodash-omit and we basically use omit to filter out the ID and then use object assigned to a transform or produce a new state and then a callback occurs after this that tells your list component that somewhere in your Ember tree to re-render, now only showing two items. CHARLES: One of the things I want to point out there, you just touched on it but I think it's an interesting in the subtle point is that the lodash method that you used was omit and that's how this is kind of tangential or I'll say parallel to programming in this way is that you don't actually use methods that mutate any state. You calculate new states based on the old states. I think that's a great example of that -- that omit function -- omission is the way that you delete from something in an immutable fashion. You're actually filtering or you're returning a new copy of that dictionary that just doesn't have that entry. You're just a omitting it. You're not destroying the old one. You're not deleting anything. You're not changing it. You're just kind of Xeroxing it but without that one particular entry, which is ironic because the effect on the UI is that you have done a delete but really what you're doing is omitting there. I just think that's cool. I think it's one of the ways that using the systems teaches you to think about identity differently. Then the question I want to ask you was, this all happens in the reducer, what does that mean? Was that word mean -- reducer? I kind of like danced around that idea and I've tried to understand where that term came from and how it helps give insight into what it's doing and I come up short a lot. Maybe we can try and if we can't explain what the name of reducers, maybe there's some alternate term that we can help come up with to aid people's understanding of what is the responsibility of this thing? TORAN: Yeah, I think we can just break down reduce first then we'll talk about how it ends up looking. But I think it's reduced almost like it's defined in the array context. If I have an array of one, two and three and I invoke the reduce method on that, we actually just produced a single value, sort of flatten it out and produce a single value as the result to that, so three plus two plus one, six is the end result. What we've glossed over this entire time, probably last episode is that this reduce word, I believe is used because in Redux, we don't really just have one massive monolithic reducer for the entire state tree. We instead have many small reducers that are truly combined to do all the work across the tree. In my mind is I think reducer fits well here because we're actually going to combine all these reducers and they're all going to work on some small part of the state tree. But at the end of the day, we still have just one global add-on and that's the output. We want one global add-on with a new transform state and that is the reduced state. CHARLES: You take some set of arguments. One of which is the prior state and you reduce that into a single object. I like that. The other place where I've seen this term applied in a similar context is in Erlang. They talk about reduction so that you have an Erlang server where the way that the servers modeled is as a recursive function call where you pass in the prior state of the server plus any arguments if you're handling a request or something like that, bundled in there is going to be arguments of the request and then what that function returns is a new state, which is then used to pass in to the next state. They call this process reduction. We've got two data points. Maybe there, we can go search for the mathematical foundation of that later -- TORAN: I like it. CHARLES: -- if you want to geek out. I think that helps a lot. Essentially, to sum up the responsibility of the action is you take a set of arguments, it's going to take the existing state of the store, run it through your reducers and then it's going to set the next state of the store or yield the next state of the store. Is that a fair summary of what you would say the responsibility action is? TORAN: Yeah. I think you're right. In fact, in preparation for this talk, I just threw together really small Ember Twiddle that will link in a show notes what I call the mega trimmed down version of ember-redux. It's basically a really naive look but for conceptualizing this flow, it's about 24 lines of code that show exactly what you're saying which is I have this reducer, it's passed into this create store method in the syntax, how do this actually look? It better illustrates how the reducer is used when dispatch is invoked so a dispatch, if I was actually to walk line by line through this, which will be probably pretty terrible. But the very first line of dispatch is to just call the reducer. From that new state transformation, we just push in so the store gets a new entry into it's array of here's the next state and because we had never tampered or side effect-ted the old state, we could easily go back in time just by flipping the pointer in the array or that indexing the array back point. CHARLES: I guess my next question is we've talked a little bit about immutability and we know this reducer that we call at the very first point of the dispatch is a pure function. You were dealing with pure functions, immutable data but at least in perception for our users, our system is going to have a side effect. There's going to be calls to the network. We are, in the least in theory deleting something from that list. How do you go about modeling those side effects inside Redux? TORAN: This is a great intro because in fact a friend of mine is actually a teacher at a boot camp and he was telling me the other day that he was asked to do a brief look at Redux and his very first feeling when he was watching some of the Egghead.io videos is like, "Oh, so the reducer is pure but I have to side effects something so where do I do this?" It's not very clear, I think for the very beginner which is why we left it out of that part one podcast. Today, we're kind of hit that head on but before we get into that list, we can talk about what having actions in their simplest form look like today in your Ember app. As I mentioned earlier in this remove example, you got the button, it just takes the closure action the on-click. No big deal. The bigger work was on the parent contexts or the parent component to provide this action, which sounded very simple but imagine instead of just dispatching synchronously, which is we talked through. Imagine we only wants to dispatch that officially to change it if we have gotten a 204 back and the fetch request is deleted on the server -- normal Ajax or fetch-type flow. In this case, you start to add a little more code and imagine for the moment this is all inline in your component JS file. The component now is started to take on an additional responsibility. In addition to just providing a simple dispatch, as I say, "Go and remove this Mr Reducer later," you're now starting to put some asynchronous logic and as you imagine a real application, you grows and try catch stuff, some error handling, some loading, some modals. This gets out of hand. One of the things that I want to touch on briefly is, at least in the ember-redux case we ship this Promise-based middleware and I want to stop right there for just a second because I use that word 'middleware' and immediately we've got at least highlight what this is doing. In that pipeline -- we talked about earlier -- in the component I dispatch and it just goes right to the reducer. Well, technically there's actually a step or an extension point right before the reducer is involved and that is where middleware comes in. Technically, you could dispatch from this action and then you could handle and do the network IO type request in middleware instead, then porting on another dispatch of a different type with the final arguments to be transformed. That's really the role. CHARLES: Can middleware actually dispatch its own actions? TORAN: Correct, yep. In fact, one of the first big differences between this example as I'm kind of hacking around the component and I've got access to dispatch but there's two things I'm really actually lacking if I don't leverage middleware full on. The first is I do have some state in the component but often, it's actually just a very small slice of state that this component renders. If there's actually a little bit more information I need or actually need to tap into the full store, I don't have it and that might be considered a good constraint for most people. But there are times you imagine more complex apps, you need the store. You might even see a little bit more state, middleware provides that is where you trapped with just that slice of state and dispatch the keyword. That's about it in the component. But the other side effects are the benefit, as you break this out is you get another seam in the application where the component now is not involved with error handling and Promises and async flows or generators. It just does the basic closure action set up and fires dispatch almost synchronously like you did in our very simple example, allowing the middleware to actually step in and play the role of, "Okay, I'm going to do a fetch request and I'm going to use a generator." It's almost like the buffer for IO or asynchronous work that it was missing in our original equation. Imagine, you want to debounce something or you want to log something or you want to cancel a Promise, which you can't do, any of that stuff that's going to happen in this middleware component. That's one of the things I like about middleware as I learn more about it and the moment you get to a very complex async task where you're actually doing the typical type ahead, where you literally want to cancel and not do the JavaScript work or you like to cancel the Promise as quickly as you can, you can very quickly dive into something kind of like what you and Alex talk about with ember-currency in the Redux-land. It's called redux-saga. It's just a generator-based async work. CHARLES: Is saga kind of emerged as async library that everybody uses? I know it's very popular. TORAN: Yeah and a good reason, I mean it solves a lot of the problems that if you were to try and do the cancelation token Promise stuff that came out a while back where we're trying to figure out how to cancel Promises. There's a lot more ceremony and just a lot more state tracking on your own that generators and even when I played with this last week, which is actually redux-observable which is an Rx-based middleware. It's built by, I think Ben Lesh and Jay Phelps from Netflix or... sorry, Jay is still at Netflix but anyway... You could use Rx, you could use generators. This really is just the escape valve for async and complex side effect programming that can't or shouldn't take place in the reducer because it's pure. It shouldn't take place necessarily in a component because one of the best pieces of advice I got when I was younger was, " Toran, make sure you do or delegate," and we're talking really about levels of depth in your methods at the simplest. But it applies here as well, which is I would love it if I had just a very declarative component and I didn't have to get into the weeds as I was looking at it about, is this a Promise? Is this Redux thunk, as they call it? Is this a generator or is it Rx? I don't even care in the component for the most part. I just need to know the name of the action and the arguments. If I'm having a problem with the Rx side of the generator, I'll go into the middleware and work from that particular abstraction but you can see the benefits of the seam there. CHARLES: Are the middlewares match on the action payload in the same way that the reducers do? Is that fair to say? TORAN: That is fair and I will warn. If that seems very strange, you're probably not alone. In fact, the first time I did this with redux-saga, I was dispatching, only to turn around then dispatch again. It feels very strange the first time but again, keeping in mind that you're really trying to have a separation from the work that is side effect and the work that is pure. The first action in that scenario, we'll call it remove-saga because it's actually going to fire something to a middleware. That work is all going to be network-heavy and it's not really as easily undo-redo because it's not pure. But the second event invoked from the actual middleware itself that says, "Remove item. Here's the ID. We're good." That work could be undone-redone all day because it hits the reducer, which is sure you can in and out. CHARLES: It sounds like basically the middleware is allowing you to have a branching flow structure because they all do involve more actions getting dispatched back to the store to record any bookkeeping that needs to happen as part of that. If you want to set some spinner state, that will be an action that gets dispatched. But in terms of sync, they allow you to set up sequences of actions or if you basically have one action that will actually get resolved as ten actions or something like that. If you think about an asynchronous process, you have the action that starts it but that might end up being composed of five different actions, right? Like I want to set the application into some state that knows that I've started my delete and that means I want to show like the spinner. Then at some later point, I might want to show progress like this deletes really taking a long time and I might want to dispatch five different actions indicating each one of those little bits of progress. Then finally, I might want to say it's done or it fails so really those got decomposed into 10 actions or five actions or whatever so the middleware is really where you do that, where you decompose high level actions into smaller actions? Or it's one of the places? Is that a correct understanding? TORAN: Yeah, I think if you're an old school developer for a minute, it will cater to the audience that maybe came from early 2000s backend dev. Now, they're still pretty current in web dev. I see it talked about as business logic. I feel like this is really the bulk of the complex work, especially if you're using Rx. You're actually creating these declarative pipelines for the events to flow through. My components are much thinner by comparison. They're truly just fire off this action with the information to kick the async pipeline but in the async piece of it, there's a lot more work happening and that's I think because there's a lot of complexity in async programming. CHARLES: Right, and it's almost like with the reducers then, there's not so much business logic because you're just resolving the implications of the new state. Is that fair to say? It's like now we've got this information, what does this imply directly? TORAN: Yeah, I think there is this old [inaudible] thing where they're talking about what's should be thick. You know, thick controllers or thick models, what should it be? Of course, we never want 'thick' anything, is the right answer, I think. But the apps on building today, I feel like if any was thick-er -- a measure of degree bigger in effort or work -- it is these middleware components right now. I think the nature of what you describe, which is the reducer is not supposed to be doing anything complex. It's literally taking a piece of data in, producing a new piece of data out. Logically thinking about that takes much less effort, I think than the human brain applies to async programming in JavaScript. CHARLES: Right. I think it makes sense and some of these things are just going to be necessarily gnarly and hairy because that's where the system is coupled. I can't say anything about whether the delete succeeded or failed until I've actually fired off the request. Those are implicitly sequenced. There has to be some glue or some code declaring that those things are sequenced. That has to be specified somewhere, whereas theoretically with your reducers, you could just run them all in parallel, even. If JavaScript supported multithreading, there's absolutely no dependency between those bits of code. TORAN: I think so, yeah. I think there are still some challenges because in the reducer sometimes. We can talk about this in a few minutes but you may actually be changing several top level pieces of the tree. If you're de-normalizing, which is what we probably should touch on next, there are some cases where you want to be a little careful but like you said, generically immutable programming enables multithreading. We're not touching the same piece of state at same time. CHARLES: Right as long as that piece of state that you're touching, like you need to resolve the leaf nodes of the tree first but at any siblings, I guess is what I'm saying on there, ought to be able to be resolved in parallel. It's more an exercise in theory or just a way of thinking about it because like why you're able to do those reducers as kind of these pure functions is because there's no dependency between them. I guess I'm just trying to point out that to wrap my head around, there are places where there are just clear sequences and dependencies and those are things that would be in the middleware. TORAN: Gotcha. I came a little scared of service worker. [Laughter] CHARLES: Actually a great point is what kind is the analogous -- if there is anything analogous -- in Ember today that's fulfilling the role of middleware. What's the migration path? What's the alternative, just trying to explore like where you might be able to use these techniques that we've been describing inside your app? TORAN: I think, at least my look at it has been a service injected into a service, which sounds completely bad or sounds broken the first time you see because you're like, "We're injecting a service into an existing service." I say that because, for me at least there is a top level service that owns the data and provides read-only attributes but there should be some other piece of code -- not the component -- that is doing this asynchronous complex processing, we just talk about as middleware, that is often a different service than the service that owns the state add-on. That's what I meant by service-injected. There's some Ember service whose job is to manage the complexity and probably ended up in middleware from the Redux perspective or ember-concurrency is literally solving that in my opinion. They do a lot for you: solving the async problem generally and I haven't dug into ember-concurrency enough to know. The pipeline stuff, I think which you guys talked about, which is an RFC, that may have eventually be what I consider the Rx or redux-saga of the middleware today. CHARLES: Right. I think ember-concurrency is just absolutely fantastic but it is a hairy problem but there's some overlap in terms of what it is solving. I think that is interesting. I guess a case where you would use middleware would be anywhere that you would use ember-concurrency. I think the interesting thing to compare in contrast there is one of maybe advantages or disadvantages -- let's just call it a tradeoff -- is that with ember-concurrency, you have this middleware that is associated with an object. It's associated usually with a component or a route. When things happen to that component, you're able to affect your ember-concurrency process but it does mean, these things are sprinkles throughout your application and the rules that are governing them can be really different, depending on which part you're operating in or just because sometimes you're using them on a route. Sometimes, you're doing it on a component. Sometimes, you're doing it on a service, whereas with the putting in the middleware, it sounds like they're going to all be in one particular place. All right. Let's move on from the simple to the more complex because that's where it's at. We've talked about modeling async processes, we've talked about handling state transitions and all that, nothing typifies that more profoundly in Ember community than Ember Data as just a foundation for state and syncing it over to the network. Love it or hate it, it's very popular. What are the things that you do in ember-redux land? One, is it fundamentally incompatible with Ember Data or is it just more easily served with other alternatives? How do you handle those foundational interactions, those fun foundational async loading network interactions with Ember Data, just using an ember-redux? TORAN: Yeah, for myself, I don't have an experience actually using the two together on purpose. There is a gentleman who did a talk sometime last year and I'll dig up the YouTube clip for you guys, where he talked about his approach where you would actually produce new states so it's still Redux friendly. Ember Data itself just be a new Ember Data model, every time you transformed it. But one of the tricky points is the philosophy of both so in Ember Data, you're just invoking set on everything and not just how it works. That's how the events bubble through the system as you re-render. You never actually create a new state of the system that's a copy, minus or plus, other attributes. You just always touching a single source of truth. I felt like that was always a sticking point. Anyway, Thomas who did this talk and I'll point you guys to it, did a great job of saying like, "Look, if you're stuck in this world with a lot of Ember Data, you're having some pain points with it and you want to try Redux to see if this alleviates those by not changing the state, here is a middle ground," which he did, I think a fabulous job driving it out. Although I must admit, there's got to be some challenges in there just because of the philosophical difference between the two. CHARLES: Yeah. It definitely sounds like there's some challenges but I'm actually pretty eager to go and watch that, to see what they came up with. If you're using these snapshot states of your Ember objects, would it be possible then to take all of your save, delete records, even query and have them inside of middlewares like have a redux-saga for every single operation you want to take on the Ember Data store. TORAN: The example he showed is basically the best of both worlds. You don't want to ember-mutate so he has a special bit of code to do that. But because the rest of it is vanilla Ember, you could drop in concurrency if you want to do the saga-type generator stuff. But you could also just make your changes as you would otherwise. You use the adapters, you fetch, you save, you delete, whatever you want to do for the most part. It saves a lot on the de-normalized side, which you would have to do manually. You don't write any Ajax code, which you have to do manually on the Redux side. I think there are benefits if someone could get it to work where you're just not changing the state of Ember Data, which may actually just be the future Ember Data at some point as well. CHARLES: It sounds like there is a pathway forward if that's the way that you want to go and the road that you want to walk so we'll look for that in the show notes. But my question then is, you're here on the podcast, what do you do? TORAN: I do want to have one disclaimer here, just so that I'm not a complete poser but I am. If you guys don't know this, I'm not trying to hide this from the community but I don't work on ember-redux at work. I don't have a side gate, making money with it. I don't use it ever. I literally just build examples to try stuff out. There's both a blessing and a curse of that. The curse is that you're asking, "Hey, Toran, you're the author. How does this work, man?" I can give you my run at it which I will but there is the other side which is very clear is that I have not built and shipped Facebook -- the current company I'm with -- with X million people hitting every month. We're not using it. This isn't exactly 'Toran-stamp of approval' here but I do mess around with -- this week in particular -- Rx which I like. I think Rx is just something that it changes the way you think about the way programming, especially async programming works. I definitely cannot comment much on Rx other than I like Alex's challenge to the community on your podcasting. Go use Rx, even if you use ember-concurrency or don't use ember-concurrency, how to break your brain. It will be for the better. Actually, Jay did a mini code review with me because my first pass at Rx was just using the fetch-promise because I was like, "I want Rx for side effect modeling but I wanted to still work with Ember acceptance testing," because I still feel like Ember is leading in that way, as you guys talk about in podcast recently. What was really cool is actually there's a shim that obviously Rx has its own little Ajax thing but it is not actually Promise-based. The advantage of this that Jay called out is in the Promise-based, where I'm using ember-fetch, let's say and I'm just wrapping it with Rx, those Promises are still not really cancel-able so what Jay was warning me about is like, "If you're going to use this quick and dirty, great. but in a real app, these will still queue up in Chrome or IE and block the amount of network requests you can actually make," so don't use Promises, even though they're very familiar. Use this operator, I think it is or a helper inside Rx which is the Ajax non-Promise-based operator. Long story short though, there's a good amount of work involved that grass is greener. In Ember Data, if you've ever used 'belongs-to' or 'has-many', you have done the most magical thing right there. In all the right ways, it is amazing because once you're in Redux and you're like, "I have this very nested object wrap," but Redux isn't meant to operate on this nested object wrap. It's meant to operate on this single tree structure, at these many top level entities as I can. As a project that's pretty popular in React called normalizer, you will likely use this project eventually. Maybe, not your first 'Hello, world', but you'll use this to actually break apart or truly de-normalized the structure. What that does a lot of times is if you have a blog with comments nested all inside of it from your JSON API call or your GraphQL call, that's fine coming from the network. But since you're going to have different components listening for just the comments, maybe or different components that just listen and re-render when the blog name changes and they don't care about the comments, you want to actually de-structure that so you have a separate blog top level item and you have a separate comments top level item. They're still related so the blog can get through its comments and vice versa as belongs-to and has-many works in Ember Data but you've got to do that work now. There is, of course magical projects like redux-orm that I just can't speak to how well they work or don't work but they try and solve the more Ember Data, look at the problem which has define this and there's the RM take care of the magic for you. I actually don't mind normalizer. It's just something people should be aware of because it's more work. You've got to break that apart yourself, just as much as writing your own network requests. You're, hopefully not going to duplicate Ajax all over the place but you will have to do the work that you otherwise do not do in Ember Data for sure. CHARLES: It's very interesting. If you look at the Ember Data internals, it sounds like the Ember Data store is actually structured very similarly to the way you had structure a Redux store using something like normalizer where you have these top level collections and then some mechanism to both declare and then compute the relationships between these collections of top level objects. But I want to go back to your other point too. I just wanted to say this. Toran, you know, don't sell yourself short because you give an incredible amount of time, an incredible amount of support to the community. You're very active in the Ember Redux channel. When problems come up, you think about them, you fix them. Even if you're not actually watering the trees, you're planting the seeds. I think that's actually great. I think that is a very valuable and necessary role in any community is to have people who are essentially the Johnny Appleseed of a particular technology. I think you go around and you throw these seeds around and see where they take root, even if you're not there. You're on to the next shady lane to plant seeds, rather than stay and enjoy the shade and the fruit of the apple trees. TORAN: Yeah. I appreciate your kind words there because a couple of years ago, I got into open source because it provides good personal branding. It's like, "This project, it's Toran's. We should hire Toran." It just makes you look more from that perspective. It also gives you almost a way to skip out on tough interviews at times if people are like, "This guy have a decent program. Let's take a look at his PR. He communicates with other humans online." It gets rid of some of that. But there is a dark side. We don't talk about it because there is an upside to it, especially for consulting but the dark side can be time commitment: how much bandwidth do you have outside of your family life, your hobby, if you have one and any other open source or work-related stuff that you already have to do. For me, this is really an exercise in thought leader-y stuff. I saw the benefits of this. It made my Ember better. Even if I wasn't using Redux, it just made the Ember code I wrote at work better. It inspired me to look at different things like ember-concurrency and Rx, things that are just way out of my comfort zone two years ago. I think those are all the benefits that come with it but the easiest part is got to be some value from it. The juice is it worth the squeeze. I think the community we've built and the people using it and the problems we're solving are all definitely worth the squeeze. CHARLES: It definitely is and you can tell from the vibrancy of that community that a lot of people are experiencing that value from it. To your point, I think something that is often lost on people is that you can actually use a project without actually using it. I think that there might be many people, for example in the Ember community that have never use React but are actually in fact using it because of the wonderful patterns that have come out and it's brought to the fore. I had thought about immutability, certainly on the server side but I had thought about it really deeply on the client side until a library like React came along and people started talking about it. I would say before we actually started using React the library, we were using it in thought. You touched on that when you were saying, "It's changed the way I think, it's changed the way I code." Even it's changed the way that I do things at work, in fact you're using it in spirit, if not the actual structure but I almost feel like that's more important. It's longer-lasting and has a greater impact on you, 10 years down the road or even five years down the road when neither of the technologies that we're actually talking about today are even going to be in wide use. TORAN: Yeah, that's true. In fact the one thing I would call out that people check out, I think Alex mentioned this or at least you guys have to talked about it in passing but definitely, sometime this weekend, watch the Simple Made Easy talk by Rich Hickey. It will definitely make you think differently, regardless of the simple side or the easy side that you follow on, projects of course make tradeoffs both sides of that but it is a great talk. Especially, if someone who's been programming six months or a year or two years, they're going to get huge benefit from it, just as much as someone older like myself who has got 10 plus years in the biz. CHARLES: Yeah, I know. That is a fantastic talk. We need to link to it at every single show. TORAN: Exactly. CHARLES: Well, I think that is a fantastic note to end on so we will wrap it up. That's it from Frontside for this week. We're going to have you back, obviously Toran there's so much that we could cover. Six months down the road, we'll do part three but for now, that's it. Thank you, Wil. Thank you, Toran for podcasting with us this morning. TORAN: Thanks for having me on guys. I really appreciate it. CHARLES: Then everybody else, take it easy and we'll see you all next week.

love netflix pr promises id remove react bridging babel api fulfilling redux io yelp stack chrome ui wil technically ajax javascript rx rm js new developments logically cli graphql johnny appleseed rfc eggheads billups erlang robert jackson middleware toran reducer frontside json api rich hickey ben lesh jay phelps simple made easy charles lowell charles you ember data using redux

062: UI for U and I

Play Episode Listen Later Mar 16, 2017 31:08

Show Notes: 00:56 - What does UI mean? UI = User Interface 02:26 - Software Interfacing with Software vs Human Beings 03:57 - At what point does UI stop? 05:55 - Responsive and Stateful UI 10:10 - Tooling: Past, Present, and Future 16:00 - Planting Business in UI Engineering 19:26 - How The Frontside Brands Itself; JavaScript Portability Resources: Linguistic Intelligence Transcript: CHARLES: Hello everybody and welcome to The Frontside Podcast, Episode 62. My name is Charles Lowell. I'm a developer here at The Frontside and I'm here today with Jeffrey Cherewaty and Robert De Luca, also developers here at The Frontside. We're going to do a little bit of navel gazing today. We're just going to talk about how we roll and who we are and because we've there's been a lot of conversations about that, as we had a couple of different projects come in that use different tool sets so what is it about us that is constant and what is it about us that changes? I just wanted to talk a little bit about that. The core idea is about UI. The shop has always been about UI and I'm curious for you guys, what does UI mean to you all? ROBERT: The frontend to the frontend? JEFFREY: That's a loaded question. CHARLES: Okay, let's step back a little bit from that because we've been in business for a long time and right now, we're doing mostly Ember, a little React. Before that, we were doing all Ember. Before that we were doing mostly Rails. Before that we were actually doing a little bit of wxWindows stuff and before that we were actually doing stuff in Java. But all of it fell under the rubric of UI -- user interface. What is constant? I mean, there's some radically different tools. When you look back and you think, "My goodness." What could possibly connect those things that I just listed? But I think there is a common thread so I wanted to explore that a little bit. JEFFREY: The common thread, I guess at the simplest level is always that in a lot of programming, you're dealing with the interface to the end-user of your code is simply text. It's a call-in response or they're consuming a library of yours that's where the code is the end product. What we focus on instead is the the end product of what we build is the interface as the graphic representation instead of a code or a text-based interface. CHARLES: I'm curious. In terms of all the software that gets written, how much do you think- because I think that's a key distinction. There are certain software that interfaces with other software. The connections to other software, there's a very unique set of problems when dealing with that. But there is also a different class of software where it's interfacing on one side with other software but on the other, it's interfacing with human beings. What proportion of software they get? How much of it is actually interfacing with human beings? How much is actually interfacing with other software? JEFFREY: I think the most software developers underestimate how much other human beings are interfacing with other software. Even if it is a simple HTTP protocol Rust type interface, it's still being used by human beings even if it's human beings writing code. CHARLES: That's true. There's definitely like a whole metalevel and that's like -- ROBERT: Like the developer experience versus user experience? CHARLES: Yeah but it's true. There's probably very few pieces of software where the ergonomics aren't important to actual people. That's true. There's definitely like a [inaudible]. JEFFREY: What you're getting at is that there's typically much more code underneath the surface than there is that actually is visible to a human. ROBERT: That makes it all work. JEFFREY: Exactly. CHARLES: Yeah. The part that's lying just beneath the surface is an ocean in of itself, I would say. It certainly seems like because that's the ocean we swim in it, seems vast at times. JEFFREY: Where does that UI stop? At what point does it stop becoming UI engineering and becomes more like API side of your server side? Obviously, you can say like, "Right when you start talking to an API. That's the end of it." But all of the code that goes on the frontend, the people refer to this as the frontend of the backend. All that code that is just there to get data and then massage that data so then somebody can go and present it to the user is at all UI at development. It's gotten a lot murkier in the last three years. CHARLES: Yeah it's a lot murkier because the thing is the way we structure our server technologies are changing so that they can support various interactions. For example, one thing that springs to mind is treating the data that we have inside of our client and I'm thinking of a browser tab, for example. Just one browser tab, one browsing session, you load data from the server and you write data back to the server but what we're seeing is that degrades the UI experience if you treat it just like that. I feel like the general trend is to treat the data that's housed in that browser tab is merely one partial replica of some distributed data set that's being replicated across a whole bunch of nodes where your storage nodes are one of those things, the node that other people are might be using in participating as application or other nodes and then that one browser tab in itself is a node in this distributed database. But that's crazy because that's what you would consider a classical server work. But to get around the fact that application state is updating and changing rapidly and trying to keep up with it in a non-buggy fashion can be really, really difficult. I know the waters are getting a lot murkier but I think that example does touch on something that I would say is kind of classical UI problem. You have some sort of stateful something. You have some sort of long running process that's just sitting there that's kind of moving with the user because the users holding a lot of state in their head as they're interacting with your application. A lot of the context is stored in their head so the UI needs to be able to dance with them and be in lock step with them and kind of mirror the context and their understanding of where they are in the system so it's usually very stateful. At least the ones that we've been working on over the past years. JEFFREY: I think another core principle of UI engineering is responding to events in a way that just doesn't happen as much in classical server engineering. You have to respond to a message or some kind of request every once in a while but the level of responding to changes in state and changes in how the human is interacting with the interface is a whole another level in UI engineering. CHARLES: It is really is a dance, in the sense that as the human moves their virtual hand throughout your application, you have to be tracking that state at all times. JEFFREY: It's like it really shines itself there. If you compare a state machine for the server side and then a state machine for the client side because we've built both and the client side is way more complex because there's just way more that's going on, impossible things that can happen. With an API, it's like almost crud in a way with an API and then on the UI side, we have to take a lot of user interaction and boil it down in order to persist that on the API side. There's more for us to take care of, on the UI side in track and it just more complex. CHARLES: That's true. I think there might be some people who work primarily on the server. They're being like, "Wait a minute." [Laughter] JEFFREY: At least on the server side, you have one runtime that you can deal with and for frontend developer, there's just so much to contain. CHARLES: Yeah so in terms of tracking that state, certainly one of the things that I think is also unique problem for user interface is that you have to constantly be taking in those events, that you're talking about, like you're constantly having to update your state so that you keep in lockstep with where the users at. That can also be part of that information that you'd be reacting to where other users are at, besides just that user so that they can absorb that context. Yes, so reacting to where they are but then also radiating information constantly to the user so that they're both inputting at extremely high rate but they're also receiving information. Like you said, it got to be extraordinarily high bandwidth so that state that you're tracking has to be represented accurately at any given time. I think, basically the core driving principle of NBC is you have this model and that represents your state and you have these events which then update that state and then you somehow managed to instantaneously reflect that state change to the user in that tight loop. I feel like that pattern, despite people tried to bastardize it, or maybe not bastardize it but adapt it for their particular use case but it has been persistent. It's like a cockroach, man. Always live and never die. I feel people are trying to assign new acronyms and new names for it but the fundamental pattern has proven to be effective over and over again. I would say that if you're using NBC, if you're building like a system with React or you're building a system with Java Swing, like we did back in the day, you're using the same basic pattern even though the mechanics and the tools are completely different but you have this abstract representation, this model of either your state or your interactions. You can model a swipe or you can model a mouse move then you're basically reflecting that to the user and then taking updates to the model. JEFFREY: What's changed about the tooling, say back when Frontside was primarily on Java? We're like, "That was the tool." It kind of a blunt instrument for some of the UI challenges we have now. What was working well in that arena and where we come [inaudible]? CHARLES: I think there's just so many exciting things that have happened since then. Especially in the past, like three years because Swing which was the Java UI framework which was fabulous, just absolutely magnificent, was at its core an observation-based system. You had a lot of observers. You can basically have these models and your view was a separate object but then you had your model. This was definitely a watershed moment. At least for me, from inside your code, you never actually updated the view. You never said I want to paint this thing. You never said I want to change this componentry. Only thing you did in reaction to events was set properties on your model. It was a mutable model so you basically had your state which is like a chalkboard but your code, the key thing there was you could paint things in Swing, you could render things but you never actually did. You had your components that always render off of a model and you never call those imperative like draw this line here, draw this line there. You always update your model and the view reacted to it. That's fundamental and that's something that we still see today but we were using mutable models. The models themselves just got overwritten every single time. The other thing is they were observer-based so you would observe this model, you observe properties on this model and when those properties changed, you're view would rerender essentially but I guess the paradigm was the view pulled data off the models and I feel like in the last three years, really with the advent of React, I think really popularized this pattern now and everybody else is adopting this. ROBERT: The flow architecture? CHARLES: Well, yeah. It's like a push model. Instead of having the view pull from the model but via observation, you have your controller essentially push a new model onto the view. But I still think the basic components were there, where there's two things that I think have change. One is this push model -- the pull view observation versus push from an event like event generates a new model and pushes on it and then the second thing which is probably coupled to it which is the idea of having your model be immutable. The magic of that is that you essentially unbraiding time from your model so that you the programmer are managing time instead of the CPU clock, which sounds weird to say. See, you guys are like, "What the hell are you talking about?" But that's fundamentally when you have something that's mutable, you're saying this is one object. It's the same identity. ROBERT: That you've changed over time. CHARLES: That you change over time but you're writing it to it where is when you use a different model each time, you the programmer is essentially saying, "This is the same object. You are managing identity of the objects not the computer. Does that make sense? You're saying here are a bunch of states but the identity of this thing is like an abstract concept and I'm saying all the states represent the same object or same entity just over time. What you're doing is you're essentially decoupling time from your model and that's I think a key innovation because then it gives you control over time so you can actually represent time, however you want to and that allows you to do things like history and what's the time travel debugging, which I think is very different than undo/redo. I think people conflate those unnecessarily but it allows you to do undo/redo history type stuff because the timeline is totally orthogonal now. Whereas, if you have immutable state, it's like, "What this looks like if that time is absolutely coupled to right now?" I think those are key innovations but I think that the fundamental pattern is still there. You have models, you have views and you have control code that manipulates. I think the key thing that's happened over the last three years that have switch between having the views pull the model versus having the controller push on there. For example, using essentially what amounts to an observable interface. I think observables as in ES7 observables are one of the coolest innovations to come around because it gives you that convenience of having a single reference point but access to all of the states that then you can react to it. Anyway, that's a long winded thing of saying that I feel those core entities are still there, where you have this and then everybody who listens to me long enough will probably get tired of me saying this but I think that the primary one is the model, like understanding that. ROBERT: Because your UI should be eventually derived from the data that you get? CHARLES: Right. ROBERT: And it should be because it is a source of truth. CHARLES: Right and everything else can flow from that. Understanding how you're going to represent it, whether we are representing it as speech, whether you're representing it as touch, whether you're representing visually, whether you're representing it with your ears, to understanding what is it. There's still one thing that can be projected into a bunch of different media so getting at that thing is the thing. ROBERT: That was a long, winded way of saying of what is UI engineering and what is unique about it but why would you want to plant your business in UI engineering? CHARLES: Right. That's a great question. Why is it a good business to be in and why is it a good feel to study? I think I'll quote one of my favorite video games from my youth, which was Mortal Kombat 2. At the very beginning, they would say, "There is no knowledge that is not power." That was the big quote in Mortal Kombat 2 and I'm pretty sure that the knowledge of Kung Lao, I think, being able to throw his hat by hitting some key sequence of joystick and buttons, I don't think that's knowledge that's going to give me power. Sure enough that I don't think that knowing those key sequences really had much power in them but I still think it's a good saying. [Laughter] CHARLES: I was always wondering what that particular quote at the Mortal Kombat programmers really like. But anyway I think that it's good to understand it as a discipline, either here or in your organization because it gives you staying power, because you understand that deep structure. For example, if the winds change and the tools get updated -- JEFFREY: As they inevitably will. CHARLES: As they inevitably will, right. We mentioned Java, we mentioned C, we mentioned Rails, Ember, React like back down knock out, all those things, those are transitory and they're ephemeral. But that knowledge isn't and it's a rock that you can cling onto as the change of storm over the landscape of the development community. If you understand that, your transition to the next thing will be a lot easier. I think that's the business value, at least from my perspective, the ability to [inaudible] those storms and be like, "Oh, this is the same thing. It's got different clothes. It's got some nice updated patterns but I understand fundamentally what's going on. I can work with it. I can find out what the things are." They talk about their linguistic intelligence so your second language that you learn is very hard, your third one is a little bit easier but then you have these people who speak nine languages. Once they learn and go through language four or five, it's actually a lot easier for them to pick up language and you're like, "How can they do that?" And the answer is that they understand the deep structure of language that's basically baked into our DNA. Every human being is born with it -- ROBERT: There are patterns. CHARLES: Yeah, there are patterns and they know them and they understand them and they just absolutely have intimate knowledge of them so the stuff that gets put on the window dressing, they can see beneath that like X-ray vision and they can pick that up and work with it and run with it and it becomes mostly just an exercise in learning vocabulary. ROBERT: And for as long as users will be interacting with the things that we build, there will always be UI engineering. There will always be somebody out there needing work. CHARLES: Exactly. Do you want people who are fatigued by the JavaScript churn, so to speak? There are a lot of different strategies to dealing with JavaScript fatigue but a great one that I know of is to understand the core principles that are going on beneath the tools and realize that the tools really are the icing on the cake, the tip of iceberg, so to speak. JEFFREY: This kind of brings us back around full circle to what originally started this conversation on our office is how The Frontside brands itself. We have, for the past few years, been doing almost exclusively Ember work. We love Ember as a tool and a framework that's done really good things for us and we've enjoyed working with. But we want to not marry ourselves so tightly to a tool like that because we believe in the problems and the concepts more than the tools themselves. CHARLES: Exactly. We want to be married to the problems not the tools because what we're really looking for is for when someone has a special interaction that they want to see made real when they have a special product that they've maybe spent a lot of time and energy and thought and maybe money on crafting the user experience. We want those people to think of us because they're saying, "Now it's time to build the UI." Before there is even an inkling of what the toolset will be, what the implementation strategy will be, we want to be at the first and foremost at people's minds. Regardless of how it's going to pan out, we know that these folks can get it done. They are going to implement this thing that is going to be as good or not even as better than the way that we envisioned it. I think that's important so we've been trying to transition towards that type of message. ROBERT: We talked about how UI has these things that are long-lived. One of the really long-lived things and I think is going to be here for a long time is JavaScript. We do JavaScript very well. Everything we've been doing since, I would say what? Mid-2015? CHARLES: Yeah. ROBERT: -- Has been to abstract away from the UI framework and just base JavaScript libraries because that can be portable to anywhere. That can go from the server, on Node, all the way to any UI framework that you want and even native now with native script in Angular or React Native. CHARLES: It's true. It's amazing how portable JavaScript is. ROBERT: We are betting on JavaScript UI and I have gotten a couple of tweets from people because they know that we're an Ember shop really and when we talk about React, they're like, "Are you just a React shop?" I was like, "No. That's a big distinction when [inaudible]." JEFFREY: -- UI shop? ROBERT: Yeah. We want to be able to pick the framework or library that we see fit for your project and it doesn't matter if it's an Ember or Angular or Vue or insert next year's framework because it's going to be written in JavaScript or a superset of JavaScript like TypeScript. CHARLES: Exactly. We love all of those frameworks and we think that they have their individual tradeoffs but we have to think about what those tradeoffs are. Also, think about how can we share as for example, this Ember app. What can be shared between an Ember app and React Native app and you might think, it's not much. But my guess is probably a lot, like most of it could be. That's kind of a radical idea but at the same time, it's a very simple one and seeing that pan out, I feel we've actually been accomplishing that and this isn't just something that's like smoke and mirrors, this is what we've been doing and the strategy has been working and it's been paying off. ROBERT: We do UI well and it doesn't matter what way we write the code, at the end of the day we're still producing this interface that users interact with and that's our bread and butter. That's what we do really well. The framework is just the thing that we're using as the flavor of the month, year or whatever it might be, to get the job done but we are betting on UI and we do it well. CHARLES: I think, it's actually a testament to a framework as how well it can adapt to plain JavaScript. The way that we model these interactions really is with simple immutable POJOs with well-known transitions to new states. That's it. It's like secret magic sauce that is so, so simple. ROBERT: Remember earlier when I mentioned the state machines between the server side and the client side? CHARLES: Yeah. ROBERT: We modeled these as immutable state machines, CHARLES: Yeah and they're so incredibly portable. I think, Ember back in mid-2015 wasn't so great at hosting these simple objects. Now in the early part of 2017, it actually is. It's very easy to write those things. If you're using Redux or React, we're kind of more friendly to POJOs from the get-go but I would say this. I think there's clear value in not putting logic inside your reducers. Like what is a reducer for? I actually think that you are better decoupling your JavaScript from a framework like Redux, in the same way that we reap huge benefits from decoupling all of our state transition logic from any framework artifact inside of Ember. It meant that we were ready to drop in those interaction models into React. It was like boom! There was very low friction. It was extremely low friction. ROBERT: Because these interactions are in a base JavaScript library and the UI library is what's driving those interactions. You're actually talking to this underlying JavaScript library that we wrote and abstracted away from the UI framework. But we still need that UI framework that whatever you inserted into to play those actions. CHARLES: Yeah, exactly. You need Redux to manage the current state to basically manage your concept of time but maybe think twice about coupling your actual interaction model to a framework because it's going to be portable. This is a lot of pain that people had surrounding like active record. I remember for people in the Ruby, there was kind of 'aha' moment in the Rails communities like, "Wait, we literally don't have to dump everything into active records. There was this, I think mass awakening, followed by this mass sign of relief. Once kind of people realized that everything you do doesn't have actually have to fit into a one of these kind of framework archetype objects, whether it'd be a reducer or a component or a route or whatever. If you have something that's truly separate, that's portable like make it portable. Once people kind of started treating active record as just, "I want to persist on something. This is what use," and things became a lot easier. ROBERT: Because you're separating yourself from Rails and using Ruby, the language. CHARLES: Right. I think, you said it perfect, Rob. It's not to disparage any tools. The tools are absolutely critical. In fact, the tools are most of the code. The amount of code, for example an Ember application provides to you is just massive and it's all very high value. Some people would debate that but the point is either all things that you're going to need to do but they're not things that are unique to the interactions that you're trying to provide. I would actually like to see a version of Ember Data that doesn't depend on the Ember object model. I think being able to really separate that. It's pretty amazing library and if it saves an enormous amount of time, it would be fantastic. ROBERT: But only Ember can ever use it. CHARLES: But only Ember can ever use it but it is definitely a common problem and something that we miss like the 90% use cases that it covers. It's something that we miss when we're working in other frameworks. I actually think there is one more thing to touch on and this is really what separates, I think what we do from a lot of maybe agencies that are heavy on the design side but not super heavy on the UI side. I think is a major differentiator between the UX and the UI. Obviously, UI is very implementation-centric as opposed to vision-centric. I think, a good comparison is architect making blueprints versus the builder that actually has to construct the skyscraper. CHARLES: Make it real? ROBERT: Yeah, it has to make it real and that's what we do is make it real and There is an engineering component to it. While we have a very heavy focus on design and beauty and elegance and smooth interaction with the user and we study a lot of key user experience principles, there is a component to UI engineering which supersedes framework, supersedes design -- sorry, not supersedes but it's present, it's ubiquitous and that is bringing to bear all of the industry best practices around making quality, robust software. I think that's an important thing to point out. It kind of goes without saying but I think it's important to mention is if you want to have all those good things and if you want to have good models, in views, in your controller, you want to work with a bunch of different frameworks. It's all for not. If you're not using all of those things that keep quality software quality, if that makes sense -- having continuous integration, having test suites, having continuous deployment and thinking about all the operations surrounding the software. Ultimately, the UI is software just like any other system and something that we could go off and talk about for ages is how to test UI software because they are. When you've got these big stateful applications, they have their own unique testing. ROBERT: But there's a lot of engineering problems here that are to be solved. CHARLES: Right and I think it's important to be every bit as that. Anyway, I think that's about it on these subjects. It's just been kind of bouncing around the walls here and we're kind of think like, "Who are we? What is it that we do? What would you say that you do here?" JEFFREY: We have access [inaudible]. [Laughter] JEFFREY: That's what we do here. CHARLES: Right. UI engineering, man. It's taking those UX dreams, right? And it's bringing them to life. It's making sure that they're scalable, that they're maintainable, that they're testable, that all of those things can become real, regardless of the tool. ROBERT: And live long. CHARLES: And live long. ROBERT: That's the hard part. We can throw something together and match the conf. CHARLES: Right. ROBERT: But, "Will it continue to match the conf in three months?" is the question. CHARLES: Hopefully so, we think so. I think we hit that mark. ROBERT: Absolutely. It's the engineering, that's the tough part. That's the hardest part. Making it happen and making it sustainable. CHARLES: All right. Thanks everybody for listening. We are going to be back next week. We've got some pretty neat folks stopping by so be sure to check it out.

dna nbc software mortal kombat swing react ux rust api mid redux ui java javascript rails cpu responsive node vue angular typescript react native kung lao frontside es7 charles lowell javascript ui ember data

054: The Ember Ecosystem & ember-try with Katie Gengler

Play Episode Listen Later Jan 20, 2017 37:46

Katie Gengler @katiegengler | GitHub | Code All Day Show Notes: 01:23 - Testing 06:20 - ember-try 14:11 - Add-ons; Ember Observer 17:43 - Scoring and Rating Add-ons 25:25 - Contribution and Funding 27:41 - Code Search 30:59 - Data Visualization 32:27 - Change in the Ember Ecosystem Since Last EmberConf? 34:35 - Code All Day 35:39 - What's Next? Resources: ember-qunit liquid-fire capybara Selenium appraisal emberCLI Bower Transcript: CHARLES: Hello everybody and welcome to The Frontside Podcast Episode 54. I am your host, Charles Lowell, with me is Alex Ford. Today, we're going to be interviewing Katie Gengler. I remember very distinctly the first time that I met Katie, it was actually at the same dinner, I think that I met Godfrey at EmberConf in 2014. That was just a fantastic conversation that was had around the table and I did not realize how important the people that I was meeting were going to be in my life over the next couple of years. But Katie has gone on to do things like identify a hole in Ember add-on ecosystem so she created Ember Observer. There's a huge piece missing from being able to test this framework that spans multiple years and multiple versions and being able to make sure that your tests, especially for add-on authors, run against multiple versions so she created and maintains Ember-try. She's a part of the EmberCLI core team. She's a principal at Code All Day, which is a software consultancy and just an all-around fantastic woman. Thank you, Katie for coming on to the show and talking with us. KATIE: Thanks for having me. CHARLES: One of the things I wanted to start out the conversation with is something that's always struck me about you is there's a lot of people when it comes to testing, they talk the talk but you have always struck me as someone who walks the walk. Not just in terms of you make sure that your apps have tests in them, where your add-ons have tests in them but talking to people about testing patterns, making sure that when there are huge pieces of the ecosystem missing like Ember-try. I remember this as something that I struggled with. I was running up against this problem and all of the sudden, here comes Ember-try and you've been such a huge part of that. I want to know more about kind of your walk with testing and how that permeates so much of what you do because I think it's very important for people to hear that. KATIE: I got really lucky right out of college. My first job was at a place that where people think of mythical themes, XP-focused developers so the first thing I was told is everything is test first, everything is test-driven. I was primarily doing Ruby in Rails at the time but also JavaScript. At the beginning, we didn't have a way to test JavaScripts and there was a lot of missteps in the way of testing JavaScript until we came right around to QUnit. I was QUnit long before Ember even came along. It's kind of bit ingrained in my whole career. Michelle as well. Michelle is my partner in Code All Day. We're both very test focused. I think that's what drew us to start a company together and working together. Every project we're on, we try to write encompassing tests: test drive everything, if we're on it, projects upgrade or any project to fix. We try to write tests as a framework for everything that we're doing so we know whether we're doing something right or not. When it comes to Ember-try, that wasn't entirely my own idea. That was something that Robert Jackson and Edward Faulkner were looking for something right. I remembered that appraisals gem from Ruby. I really enjoyed being [inaudible] gems that I had written Rails so I wanted it to exists for Ember so I just kind of took a promise of to do it. It was extracted from Liquid Fire. I had some scripts that would sort of test multiple versions but it was rough. It wasn't as easy as it is today. CHARLES: Yeah, it does speak to a certain philosophy because if you're coming to a problem and it's difficult to test, you often come to a crossroads where you say, "You know what? I have a choice to make here. I can either give up and not write a test or trying and test some subset of it," Or, "I can write the thing that will let me write the test." It seems like you fall more into that second category. What would you say to people who are either, new to this idea or new in their careers and they butt up against this problem of not knowing when to give up and when to write the thing to write the test. KATIE: I almost never don't write the test so if you're suspicions are true, I will write something to be able to write the test. But there are times that I'm [inaudible] and sometimes I'm just like, "This is not going to be tested. This is not going to happen." Finding that line is pretty hard but it should be extremely rare. It's not when people come to me, I work with a client and they're telling me, "No, it's too hard to write the tests." A lot of times, it's not only how you write the test, spreading the test and learning how to write a test. It's the code you're trying to test that could be a problem. If you have a very complicated code, very side-effect driven code, it's very hard to write the easier sell, which might Ember acceptance tests. What you're really kind of on a level of integration because you do have a little bit of knowledge of what's going on and you have to be within the framework of what Ember tests wanted you to do, which is async is all completed by the time you want to have this assertions and test. That means to look different tools like going back to something like Capybara or Selenium and have some sort of test around on what you're doing in order to replace the code that makes it hard to test it at a lower level to begin with. I think a lot of people are just missing the framework for knowing what to do when their code is intractable or not, necessarily that the testing and the guides that have you tested. I think most people could go through tutorial and do tests for a little to do MVC app perfectly fine. But that's easy when you [inaudible] size of the equation so if you're already struggling with code and you're not quite sure, either in Ember, it can seem very, very hard to write tests for that. I think that's true with Rails as well. I think people that begin in Rails don't understand what they're going to testing, especially if they have an existing app that trying to add test to but unfortunately, Rails not a long ago, kind of got into everybody's heads that your tests go with what you're doing. It's just ingrained part of the Rails community. Hopefully, that will become how it is with Ember. But a lot of people are kind of slowly bring their apps Ember so they really have a lot of JavaScript and they don't necessarily know what to do or they're write in JavaScript have always are written with jQuery and a little bit [inaudible]. They don't understand how to test that. ALEX: How does Ember-try help with that? Actually I want to roll back and talk about what is Ember-try and how does it fit into testing? You mentioned the appraisal gem which I'm not familiar with. I haven't done much Rails in my life or Ruby. But can we talk about what Ember-try is? KATIE: Sure. Ember-try, at the basis, let's you run different scenarios with your test. At some point, I would've said, let's run different scenarios of dependencies for your tests so primarily changing your Ember version and that's pretty much what add-ons do but a lot of people are using it for scenarios that are completely outside of dependencies so different environment variables, different browsers. They're just having one place to have all these scenarios, where if you just put it in travis.yml like your CI configuration, you want it as easily be able to run that locally. But with Ember-try, you can do that locally. I found that it's kind of beyond my intentions, expanded beyond dependencies. Primarily, it lets you run your test in your application with different configurations. I could see running it with different feature flags, it would be what something to be interesting to do, if that's something you use. Primarily, it just lets you try the conversions and appraisal gems let you run test with different gem sets so you have a different gem file for each scenario you possibly had. That was definitely dependency-focus. CHARLES: That sounds really cool. It's almost sounds like you could even get into some sort of generative testing, where you're kind of not specifying the scenarios upfront but having some sort of mechanism to generate those scenarios so you can try and surface bugs that would only occur outside of what you're explicitly testing for, which is kind of randomly choosing different versions of environment, variables, feature flags, dependencies and stuff like that. They didn't thought of that? KATIE: Randomization [inaudible] but Ember-try really does have a kind of general route way of working on and that's we're leading to that. If you wanted to, especially for add-ons, you can specify this version compatibility keyword and your packet at JSON and give Ember and give an Ember string a version and it will generate the scenarios for you and test all those versions. These Ember strings are pretty powerful so you can say specifically versions you want. You can do a range of versions and it will take the latest patch release and a reminder, at least you don't want to be too crazy and test each of those for that add-on. But I can definitely see something random, they're really cool. Some testing thing that's like just tries to do random input into all of your inputs on a page. I've really been meaning to try that out. Sounds like [inaudible]. CHARLES: Yeah, just like to try and break it. I remember a world before Ember-try and I can't speak highly enough about it and the fact that how many bugs it has caught in the add-ons that we maintain because you're always working on the latest, hottest, greatest version of Ember and you're not thinking what about two-point releases back. They're might be not a deal breaker but some subtle bug that surfaces and break your tests and the coverage has just gotten so much better. In fact, I think that they're as brilliant as it is bundled with EmberCLI when you are building an add-on. It's like you now you get it for free. It's one of those things where it's hard to imagine what it was like, even though we lived it. KATIE: And it was less than a year ago. [Laughter] KATIE: Ember-try existed for more than 15 bundle with EmberCLI so since last EmberConf or so. CHARLES: Yeah, but it's absolutely a critical piece of the infrastructure now. KATIE: I'm glad it caught bugs for you. I don't think I've actually caught a bug with it. CHARLES: Really? KATIE: Yeah, but I don't do a lot of Ember re-add-ons. I do a lot of EmberCLI-ish add-ons. It can't change versions of EmberCLI. Not yet, we're working on that. I get some weird npm errors when I tried it but I haven't dug into it much yet. CHARLES: I don't want to dig too much into the mechanics but even when I first heard about it, I was like, "How does that even work?" Just replacing all the dependencies and having a separate node modules directory and bower and I'm like, "Man, there's so many moving parts." It was one of those things where we're so ambitious. I didn't even think it was possible. Or I didn't even think about writing it myself or whatever. It's one of those like, "Wow, okay. It can be done." ALEX: This exists now. CHARLES: Yeah. ALEX: Add-on authors are accountable now for making their add-ons work with revisions or versions a few points back like you said but it makes it so easy. The accountability is hardly accountability, turning Ember-try. It's really amazing. KATIE: What I'm laughing about is that what it actually does is not very sophisticated or crazy at all. For instance, for bower, it moves your existing bower components structure. It's a placeholder. It changes the bower.json run install. Then after the scenario, it put's everything back. CHARLES: But I don't know, it sounds so hard. It's intimidating. You got all this state and you got to make sure you put it all back. What do you do with if something hit you and abort midway. I'm sure you had to think about and deal with all that stuff at some point. ALEX: I kill my tests all the time in Ember-try and I was like, "Oops." I forgot I shouldn't do that just like for this. KATIE: Yeah, it doesn't recover so well. It's pretty hard to do things on process exit in node correctly, at least and I don't think I gotten it quite right. But there is a clean up command. Unfortunately, with the way it interacts with EmberCLIs dependency tracker so when you run an Ember command, Ember checks them to make sure all your dependencies are installed. If you still have the different bower.json and install haven't run, you have to run install before you can run the cleanup command which is kind of a drag. CHARLES: I have one final question about Ember-try. Have you given any thought to how this might be extracted and more generally applicable to the greater JavaScript ecosystem because I see this is something that Ember certainly was a trailblazer in this area. Some of these ideas came from Rails and other places. This is going to be more generally applicable and had you given thought to extracting that? KATIE: Yeah, some of [inaudible] since we first did it because we realized very early on that it doesn't depend on EmberCLI. I didn't even using it as a command line arguments parser which doesn't seem too important. But there are some assumptions we get to make. For it being an Ember app, we know how EmberCLI was structured. Some of those assumptions, I wouldn't really know with the greater node community and I gotten those assumption might not be possible at all because they don't have the standards we have on EmberCLI. It generated EmberCLI. There's generally certain things that are in place in Ember [inaudible] works for [inaudible] so there's no part of it that could be extracted. But I worry about some assumptions about no modules always being in the directory that they are in because then can be link the node modules above it. In EmberCLI, it usually doesn't support that. But other places, obviously have to. I realized that it could definitely happen but I'm not so sure that I'd want to personally support that because it's a little bit time commitment. CHARLES: Right. Maybe if someone from the outside want to step in involuntarily, you might work with it but not to personally champion that cause. KATIE: Definitely. I think it would be really cool and I do think it will end up having its own [inaudible] parser eventually, just to be able to do things like different EmberCLI versions. As long as it's not part of EmberCLI, I think that would be less confusing, though. In theory, that can be done with an EmberCLI still but I'm not clear on that. I've had people talk to me about that and I haven't fully process it yet. CHARLES: Right. Alex you mentioned something earlier I had not thought about but that was the technologies like Ember-try keep the add-on community accountable and keep them healthy by making sure that add-ons are working across a multiplicity of Ember versions and working in conjunction with other add-ons that might have version ranges. Katie, you've been a critical part of that effort. But there's also something else that you've been critical part of that you built from the ground up and that is Ember Observer. That is a different way of keeping add-ons accountable. But I think perhaps, even in a more valuable way, more of a social engineering way and that's through the creation of Ember Observer. Maybe we can talk about Ember Observer a little bit. What it is and what gave you the insight that this is something that needs to be built so I'm going to step forward and I'm going to build it? KATIE: I'm definitely going to refer again back to the Rails community. I'm a big fan of Ruby Toolbox. Whenever I needed to jam, I would go there and try to see what was available in that category kind of way. There's variables that have on there. It will have something like the popularity in the number of GitHub stars and the last time it was updated. You can see a lot of inspiration for Ember Observer in there so maybe I should step back and explain the Ember Observer. Ember Observer is a listing of all of the add-ons for the Ember community. Anything that has the Ember add-on keyword will show up there. We pull it off from npm and it all show you all that kind of information: the last updated, the number of GitHub commits, the number of stars, the number of contributors and we put all of that information and a manual view together to put a score on each add-on. You can look at it and we categorized them as well. If you look at a category, say, you're looking at a category for doing models. You would see all of a different model add-ons and be able to look between them and compare them, decide which ones to use. Or if you're thinking of building something, you can go in there and be like, "This already exists. Maybe I should just contribute to an existing thing." What gave me the idea for is I was looking at Ember add-ons, which just shows you the most recently published add-ons for that Ember add-on keyword everyday and every time I was clicking on these add-ons I go, "They did the same thing," and it just seemed like such a waste in [inaudible]. People were creating the same things and then they started clicking into and I was like, "Why they bother clicking into this. It doesn't have anything. It's just an empty add-on." We're pushing add-ons just to try that out so I thought I'd be nice if something filtered that out and I happened to have some time so I got started and dragged my husband, Phil into it. He's also an Ember and Rails developers so that's pretty convenient and my friend, Lew. Now, Michelle works on it a little bit as well. That's what drove us to build it and it's been pretty cool. I like looking all the add-ons when they come up anyway. I feel like it's not any actual work for me. It's quicker than my email each day to look at the new add-ons. ALEX: How many new add-ons are published every day on average? KATIE: On average, it's probably four to six maybe but it varies widely. If you get a holiday, you'll get like 20 add-ons because people have time off. You know, if somebody just feeling the grind at two and you'll notice that the add-on struck commeasurably too. It would be else that come on the same kind of week. ALEX: You mentioned that an add-on gets a score. Can you explain that score and how you rate at add-ons? KATIE: Sure. The score is most driven by details about the add-ons. There's Ember factors that go into it and it's out of 10 points. Five of them are from purely mechanical things, whether or not there's been more than two Git commits in the last three months, whether or not there's been a release in the last three months, whether or not they're in the top 10% of add-ons, top 10% of npm downloads for add-ons, whether they're in the top 10% of GitHub stars for add-ons. ALEX: I know that I'm a very competitive person but it also applies to software. Not just on sports or other types of competition. But I remember a moment and I'm an add-on author and my add-on had a 9 out of 10. I was just about to push some code and just about to get a release. Even before that happened, it went to a ten. The amount of satisfaction I got is kind of ridiculous. But I like it. I like the scoring system, not just for myself but also for helping me discover add-ons and picking the ones that might be right for me. I'll check out any add-on basically that might fit the description. As long as there's a readme, I'll go check it out. But it still helps along with the categorization. CHARLES: Correct me if I'm wrong but I believe you can achieve an 8 out of 10 without it being a popularity contest. By saying there's a certain concrete steps that you can take to make sure that you have test and you have a readme, that's a thing of substance. I don't remember what all the criteria are but you can get a high score without getting into the how many stars or if you're in the top 10%. I think that's awesome. But it does mean that if I see an add-on with a five or something like that, it means that they're not taking my concrete steps or it might not be as well-maintained. You know, that's definitely something to take into account. I'm curious if there are any different parameters you've thought, tweaks you thought of making to the system. Because this gets to second part of that, what things had you considered just throughout out of hand, as maybe not good ways to rate add-ons. KATIE: We haven't thought about everything. I don't particularly like the popularity aspect of it. It did feel necessary to include it in some way. The stars and theory are representative of interest, not as much as popularity but it probably gets into popularity as well then downloads are inferior for popularity. But the problem downloads and I found this happening more and more frequently is that if large companies start publishing their own add-ons, then they have a lot of developers, they are going through the roof on those downloads so they're getting that to a point that just from their own developers and I have no way of knowing if anybody else is using this. CHARLES: Probably their continuous integration with containers, right? Like if it's running on Travis or Circle, it's just sitting there spinning like pumping the download numbers. KATIE: Yeah and that frustrates me quite a bit. But I haven't found another thing that really representative popularity. Unfortunately, with npm you can get the download counts but you can't tell where they came from. There's no way to do that. I simply would like to see the things that are popular, they rate higher than the they currently do, like you said either, it's 10 points go without the popularity coming to affect it. But you do need a collaboration with at least one of the person to get eight points. If you give seven by yourself, you need to have another contributor. If you only have one contributor, you don't get that point because that's trying to be representative of sort of a bust factor but it's not truly there. You can just have one commit from somebody else to get that. CHARLES: Of all the pieces, I think that's totally fair. KATIE: Yeah, there's definitely a few other things I have in mind to bring in to the metrics but we're not quite there yet. I need to entirely refactor how the score is given so it's not exactly out of 10. The idea is to have some questions and some points that are relevant only to certain categories about us. Whether or not in add-ons testing its different versions of Ember, it might have matters for add-on but it's only adding 10 for CLI or whether or not they have a recent release. Might not matter if it's kind of a one-off with the sass plugin or something more of the build tool that doesn't change for everyone. CHARLES: Yeah, I've noticed we have that happened a couple times where we've got a component that just wraps a type of input. Until the HTML is back changes or a major API changes happens in Ember, there's no need to change it. I can definitely see that. How do you market it is like something that changes infrequently. Is it just that add-on author says it's done? You give them a bigger window or something like that? KATIE: There are probably some sort of categories that fall under that. For the input, I think if it's doing something that Ember producing components, it probably want to be upgrading every CLI, at least every three months. I think in that case, it's probably fair to require an update within that period of time. But some of the things that are more close to Broccoli like as they are in Ember add-ons, that make sense to not have that requirement. Maybe not that's the [inaudible] exact example of the kind of questions that [inaudible]. ALEX: In Ember add-on was the first time that I gave back to the open source community. It was my first open source project and Ember Observer really helped me along the way to say what is an open source best practice and I thought that was really cool. Now, it sounds like with some of the point totals, you're leaning towards Ember best practices to help Ember add-on authors along that way. I think that's really awesome and very, very useful. I would not like to see what the Ember add-on ecosystem would look like without Katie. It would be a very different place. KATIE: Thanks. I'm glad it had some help on that and I'm affecting add-on authors. I actually didn't originally think about it when I was first building it and I was really hoping to help consuming of add-ons but it really has kind of driven out people finding add-ons to not build because they contribute to existing ones. Also, driving them for the score because as you said, people get very competitive. I really didn't realize what kind of drive the score to be because to me I'm like, "Somebody else's score me. How dare you?" [Laughter] KATIE: I have had people say that to me, "How dare you score me? How dare you score my add-ons?" Well, it's mostly computerized. Even the review was manual but only thing about it has any sort of leeway is are there meaningful tests. That's really the only thing when I go through an add-on is whether or not that has any sort of leeway for the judgment of the person that's doing the reading. If there's a readme and we're kind of a rubric so that goes if you have anything in there that's other than the default Ember [inaudible] readme. Whether or not there's a build that is based entirely whether or not there's a CI tag in readme, these are for the owner to go look for them [inaudible]. We hope to automate that so I don't have to keep looking for those. A lot of add-ons turned out have to have builds when they didn't have any meaningful tests. They just had tests so that's kind of confusing. CHARLES: What do you do in that situation? You actually manually review it so that add-on would not get the point for tests. KATIE: No, they don't get the point for test and if they don't get the point for test, I put 'N/A' for whether or not, they have built so it doesn't apply. It doesn't mean anything to me if they haven't build their own test. CHARLES: Right, that makes sense. A lot of it is automated but it still sounds like consume some of your time, some of Phil's time, some of Michelle's time. I guess, my question is do you accept donations or a way that people can contribute because I see this as kind of part of the critical infrastructure of the community at this point. There might be some people out there who think, "Maybe, I could help in some way." Is there a way that people can help? If so, I'd love to hear about it. KATIE: We don't have any sort of donation or anything like that. I mean, we should. We consider it primarily just part of our open source works, part of our contributions to the community because we also make a great deal of use out of the community. Fortunately, it's not very expensive to run. It's only $20 a month VPS. Other than time, it's not really consuming very many resources. That may change over time. The number of hits is increasing and we're doing some more resource-intensive things like Code Search and we're running Ember-try scenarios from the top 100 add-ons to generate compatibility tables. It hasn't been the most reliable. Think about if you're trying to do nvm-installed times 100 add-ons, times every day, times the different Ember dependency settings so it's been very much like a game of whack-a-mole but for now, it's not bad. But we probably should think about some sort of donation by then. Maybe something that writes out the exact numerical cost of something like Ember Observer. The API is getting about 130,000 hits a month but that's the API so that's some number of quests per person. [inaudible] tells me something about 12,000 visitors each month. CHARLES: Does Ember Observer has an API? Are there any the third-party apps that you know about, that people built on top of the Ember Observer? KATIE: None that have been kind of public. I know a couple of private companies seem to be hitting the API but it's not a public API. It's really not public yet. I'm literally process of switching over to JSON API and at some point, I'll make some portion of that -- a public API -- but it's pretty hard to support that at the same time. It change Ember server pretty frequently and do any kind of migrations we need to do. Ember add-ons does all the scores from us from API end point. CHARLES: I actually wasn't aware. I remember the announcement of Code Search but how do you kind of see the usage of that? What's the primary use case when you would use Code Search on Ember add-on or Ember Observers? KATIE: I think the primary use case is if you're looking for how to use a feature. If you're creating an add-on and you want to know how to use certain hooks like [inaudible] or something like that. You can do the Code Search for that and see what other add-ons are doing. It's only searching Ember add-ons that have the repository are all set so you'll only find Ember results. That will be nicer compared to searching GitHub. Then I find another use case is more by the core team to see who is using what APIs and whether or not they can deprecate something or change something or something has become widely used since we're pretty excited about that possibility, we've never ever been searching. ALEX: That is brilliant. CHARLES: Yeah, that's fantastic. The other question that I had was this running Ember-try scenarios on the top 100 add-ons and that's something that you're doing now. Are you actually reflecting that in the Ember Observer interface? Is that an information or is that an experimental feature? Or is that reflected all the way through so if I go to Ember Observer today, I'll see that information based on those computations? KATIE: I start playing in the Ember Observer interface. It's only for the top 100 add-ons currently but hopefully expanding that to all add-ons. Especially, for few months but maybe it's not easy to notice. The only on top 100 add-ons would be on the right side bar and there will be a list of the scenarios we ran it with and whether or not it pass or not. There's add-on information there. The top 100, I'll link to right on the main page of Ember Observer so you can see in the front. CHARLES: How do you get that information back to the author of that top add-on? KATIE: We haven't actually done that. It's just on Ember Observer. It's more meant for consumers to be able to see that this add-on is compatible with all these version. We're not using their scenarios. We're using our own scenarios saying Ember from this version to this version, unless they have specified that version compatibility thing and then we'll use those auto-generated scenarios. This might get harder for add-ons that have complex scenarios so it need something else to vary along with the versions like Ember Data or maybe they're using Liquid Fire and Liquid Fire have these three different version and for each versions, it's being used. For those, we'll just have things which are unable to test this. But hopefully, this is still providing some useful information for some add-ons. A lot of add-ons, their build won't run unless they commit. In this case, this is running every night so new Ember versions released will see if that fails. On other side of it, we have a dashboard where we can see which add-ons failed and maybe see if a new commit to something broke a bunch of add-ons. It commits to something like Ember and for CLI Ember-Gate, one of the main things. CHARLES: I know that certainly that right after we get over this podcast, I'm going and running or checking up all add-ons that we maintain and making sure everything is copacetic. If you guys see me take off my headphones and dash out the door, you know where I'm going. KATIE: Got you. ALEX: I just have a further comment that I'm excited for the public API of Ember Observer just because I've been thinking a lot about data visualization lately. I think it would be really cool tool to do a deprecated API, like one of those bubble charts where like the area that's covered by this deprecated API -- I'm doing a bad job of explaining this -- just like the most use deprecated API methods and visualizing that, I think they'll be really interesting to see it. CHARLES: Right, seeing how they spread across the add-on. ALEX: Yeah, or just all add-ons in general. KATIE: I am most nervous about a public API for Code Search, though because it's a little bit resource-intensive so just freaking out a little bit about the potential of a public API for it. But an Ember Observer client is open source. If you want to add anything to the app, that I consider as public think it is. Adding to that, I really do want to figure out some way to have like a performance budget for when people add to the client because sometimes I'll get people who want to add features and I'm like, "That's just going to screw all of it. It's going to be a problem for all Ember Observer and it's going to make everything slower and it's really little slow. But JSON API fortunately, I have kind of a beta version that running and it's going to be much faster, thank God. I probably shouldn't said that either. CHARLES: Definitely we want to get that donation bin set up before the API goes public. Okay, let's turn to the internet now and we'll answer some of the questions that got twitted in. we've got a question from Jonathan Jackson and he wanted to ask you, "Where have you seen the most change in the Ember ecosystem since last EmberConf?" which was March of 2016. KATIE: There was definitely fewer add-ons being published but the add-ons that are being published are kind of, say more grown up things. We've got... I don't know if engines was before or after March. I have no idea. Time has one of those things that engines and then people doing things related to Fastfood so things are coming from more collaborative efforts, I think. This is just my gut-feeling. I have no data on this. Isn't a gut-feeling from looking at add-ons and then there's a lot of add-ons that are coming out that are specific to a particular company. I think that maybe, I hope representative of more companies getting it to Ember but hopefully, they'll make things more generic and share them back. The other problem with the popularity is like about before, where big company is getting itself into the top 100 list, probably with just its own employees only appear over the summer. I tried a few different ways to mutate the algorithm to try to get them out of there but there was no solution there. It's much fewer, novel things. Very rarely do I look at Ember add-ons and I'd be like, "Oh, that's great," but when I do, it's something very exciting. CHARLES: Right so there's a level of maturity that we're starting to see. Then I actually think that there is something in the story too, of there are now larger companies with big, big code bases that have lots of fan out on their dependency tree that just weren't there before. KATIE: Definitely. I don't think some of the large companies were there before but I think some of the largest companies are probably keeping most of their add-ons private so there's kind of mid-range of company that's big enough to donate things or willing to put things open source. A few of these companies that can have a lot of add-ons now and a lot of them are very similar to things that have already existed so you're going to be like, "I don't know why I use this," but they obviously make changes for some reason. CHARLES: The other thing that I want to talk to you about, before we wrap up, is you actually are in partnership in Code All Day. What kind of business is that? What is it you guys do? What's it like running your company, while on the same time, you're kind of managing these large pieces of the Ember ecosystem? KATIE: Code All Day is very small. It's just me and Michelle. It's a consulting company, we kind of partnered together after we left to startup and decided to do consulting together. We primarily do Ember projects, also some Rails and we try to work together and we love test-driven things. It's pretty kind of loose end. We ended up running it since it's just a partnership. We don't have any employees. But Ember Observer will take up a lot of our time and we really had an idea that it might help us get clients that way so I suppose it kind of helps our credibility but it hasn't really been great for leads so much. But fortunately, there hasn't been a big problem for us. We really enjoys spending our time. We enjoy the flexibility that consulting gives us and while that flexibility is what's going to making these things keep running. CHARLES: All right-y. Well, are there any kind of skunkworks, stealth, secret things you've got brewing in the lab, crazy ideas that you might be ready to give us a sneak preview about for inquiring minds that may want to know? KATIE: Some of them are really [inaudible] which is redoing Ember Observer with JSON API instead of currently, it's using ActiveModel serializers, which is a kind of custom API to Rails and [inaudible] fortunately, it's an API now. They're removing something called JSON API Resources so that will get the performance of Ember Observer much better and that's pretty much my primary focus at the moment. I don't really have any big skunkworks, exciting projects. I have far off ideas that hopefully will materialize into some sort of skunkworks projects. CHARLES: All right. Well, fantastic. I want to say thank you, Katie for coming on the show. I know that you are kind of a hero of mine. I think a lot of people come to our community and they see like, "Where's the value in being a member of this community, in terms of the things that I can take out of it? What does it provide for me?" And you demonstrate on a day-to-day basis, asking what you can do for your community, rather than what your community can do for you, to paraphrase JFK. I think you live that every day so I look up to you very much in that. Thank you for being such a [inaudible] of the community which I'm a part of and thank you for coming on the show. KATIE: I'm very happy that I've been here and thank you. I use a lot of your guys add-ons and it's really the community has given so much to me, which is why I ever want to participate in it. It's really great group of people. CHARLES: Yep, all right-y. Well, bye everybody. ALEX: Bye.

god time man change circle john f kennedy fast food ecosystem api scoring github contribution apis javascript html rails broccoli godfrey xp git vps lew json cli mvc jquery selenium capybara jonathan jackson robert jackson json api alex yeah alex ford alex you alex how emberconf charles lowell qunit ember data

053: Glimmer 2 with Godfrey Chan

Play Episode Listen Later Jan 13, 2017 53:18

Godfrey Chan @chancancode | GitHub | Tilde Inc. Show Notes: 02:27 - What is Glimmer 2? 11:24 - Key Changes for Glimmer 2 12:54 - How do these libraries (i.e. handlebars.js; htmlbars) work together? 18:02 - Maintaining Compatibility 24:38 - Components 27:55 - Looking Forward to Non-Performance-Related Features 34:43 - Animations and Visualizations 39:09 - Glimmer 2 For an End User 41:33 - Stand-alone Glimmer? 46:12 - What are the performance gains on mobile? How does it better position Ember for mobile devs? 49:23 - Bubble Tea 50:37 - Contributing to Glimmer 2 Resources: Godfrey Chan: Hijacking Hacker News with Ember.js @ EmberConf 2015 Announcing The Glimmer 2 Alpha Isemberfastyet The Quest Issue (#13127) D3.js EmberConf Yehuda Katz & Tom Dale: Opening Keynote @ EmberCamp London 2016 TypeScript Ember Community Slack Transcript: CHARLES: Hello, everybody and welcome to The Frontside Podcast Episode 53, brought to you by The Frontside where we engineer user experience that is just so. If that's something that you're interested in, please feel free to reach out to us on either Twitter or at contact at the Frontside.io. Today we have a really fun episode. Again, we're second week in a row, we've got a nice fat panel. Jeffrey Cherewaty and Chris Freeman, who are developers here at The Frontside and we're also here with Godfrey Chan. Just a little bit of background on Godfrey, from my perspective, I remember very distinctly back in EmberConf 2014, I went out to dinner and I sat across from this guy who was the organizer of the Vancouver Ember Meetup. Was it Vancouver? GODFREY: Yeah, that is correct, Vancouver, BC. CHARLES: Yeah, Vancouver, BC. You know, I was struck at the time and I was like, "Wow, this guy is really smart and insightful and actually making me laugh a lot." GODFREY: You're very kind. CHARLES: I don't know if you remember that but that was borne out at the next EmberConf in 2015 where he gave this fantastic talk on the power of Ember Data's adapter and serializer layer, where he repeated the same trick except at scale in front of the audience. I think we were all rolling in our chairs with laughter but also learning a lot about this really, really powerful pattern and you had written serializer and adapter layer to basically build without a back end at all, a completely new UI for Hacker News. It was really a fantastic talk. Since then, you've just been going on to do a bunch of other great things: you joined the Rails core team, the Ember core team, and most recently, it sounds like the major thrust of what the last year or so has been giving birth to Glimmer 2, which is the next generation of the rendering engine in the Ember.js framework. It's a really fascinating topic. Thanks, Godfrey for coming on and talking with us about this. GODFREY: Thank you for having me. CHARLES: Yeah, it's a huge deep topic so we're just going to scratch the surface. Obviously, can't cover a year of life in just under an hour but we're going to do our best. Why don't we talk just a little bit about what Glimmer 2 is? For our audience members who are deeply involved with Ember community, it probably is not going to come as a surprise but we actually do have other listeners. What is it and why is it important? GODFREY: Like all good software, Glimmer 2 was basically born out of an accident. Glimmer 2 on a very high level is the new rendering engine for Ember. That's pretty vague but I guess, we'll go back a little bit in terms of the timeline and maybe that would become more clear. The time frame was pretty much when I first joined Tilde, we shipped 1.14 for a while and we just about to ship 2.0. That's when the original Glimmer planned it. That year at EmberConf, I believe it summer of 2015, like Tom and Yehuda talk about this new Glimmer thing on stage that has promised to basically renew the Ember programming model to better match the data down, actions up paradigm as opposed to very reliant on to a bindings and stuff like that. On one hand, it gives you better programming model and on the other hand it's promised to be much faster at rerendering. It's more or less inspired by React so you can just treat your component as if it's refreshing a page on a server-rendered HTML or something like that. You can just update your data and then rerender a component and that's going to be efficient enough that you can just basically rely on that as the main fully-rendered model. That ship in 1.14, for most part it was great so we were planning to start working on the next thing and at that time, the plan was to work on rehydration. If you're not very familiar with it, it's basically the next iteration of FastBoot where today when you have a FastBoot render page, when it goes [inaudible] has to then download JavaScript the data and rerender the page basically for the DOM then re-insert the newly rendered content into the page. There are several problems with that. First of all, it's a lot of wasted work because you already rendered all the HTML on the server side and now you're re-doing the work and you're framing them away. Also your users will probably see a brief flash because you're deleting the old DOM and you're replacing with a new DOM. You probably lose important things like [inaudible] position and probably not the most performant way to do that also. That's what we're going to work on next. Basically, allow the rendering engine to instead of rendering a new DOM, just rehydrate the existing markup that was sent by the server and just wire up the thing so it can be updated further on the client side. CHARLES: What I'm hearing is that the original Glimmer that was introduced in 2015, it had to always assume starting from a zero state. It couldn't take an existing DOM and kind of converge. GODFREY: Right. That's still true today. I wouldn't say that the rendering engine is not capable. It's just that the feature was not written yet, at the time we were about to start writing that feature. It took us maybe a week or two spiking on it. We got something working like as we go for that exercise, I was basically new to the code base at the time so Yehuda was explaining a lot of things to me. We were pairing basically full time at that time. If you know Yehuda, he also travels a lot. He has a lot of other responsibility like [inaudible]. We basically paired for a few days and then he might go on a short trip or I might go on a short trip for a conference or whatever. Then we'll come back and then it was like, "Wait a minute." I basically forgotten most of the things so we have to go through the concepts again. That was a little bit painful. On the other hand, the code was not really that well-rationalized or well-structured. It's a relatively old code base or at least in terms of JavaScript age, the original Glimmer 1 was basically bolted on top of what we were using at the time, which is HTMLbars engine where we can perhaps go into a little bit more details on that later. But basically, the engine was not really designed for a lot of those so it was getting a little bit difficult. Separately on another fret, a lot of things is going on at the same time, we're getting more reports on performance regression on Glimmer 1, which is probably a little bit surprising because the whole point of Glimmer 1 was introduce a new algorithm to make things faster. That's the big promise of the original Glimmer but it turns out that, indeed we rerender performance a lot faster. However, because it was so slow before, basically a no one used that feature ever so the existing apps are not getting any of the benefits because you would have to do a lot of refactor to take advantage of new programming model. However, you have existing code that you're using in production. Those code did not used new programming model, did not get any of the benefit. On top of that, it turns out that even though we have this superior algorithm in Glimmer 1, we inadvertently regressed it the initial render performance for components. That's not a tradeoff that we explicitly made. It's just that we wrote a lot of new code and we were mostly focusing on making the rerender pass as fast as possible. By the time were done, everything seems fine, we shipped but in the wild, people reported, "It turns out the thing I care about just got slower," and because it's not like a design decision that trade off that would made, we don't actually know what's going on like we have a lot of new codes somehow, that made thing slower when you do investigate. Basically, at that time, we had to polish our work on rehydration and focus more on performance. As we dug deeper into this rabbit hole, it turns out one of the main thing that happened was how we handle blocks. In the handlebars template, you have a pound or a hash, like let's say, '#if' and then if something and there's a block in there then you do slash if, that's blocked in handlebars. It turns out we made some significant changes in how that works and blocks got significantly slower for some reason and you can see that by comparing the performance between the block form of the if-helper, like #if, blah-blah-blah, versus the equivalent of the inline form just the curly-curly, if something then show render a string, otherwise render some under string so if you compared to those two versions, they really should be the same thing. They are not functionally any different but if you compare the render and performance between the two versions, it turns out the block version is like in order of magnitude slower, for some reason. That's bad because it turns out we also made some changes to how components work and each component has at least two hidden blocks that you can't see. It were like basically, how the tech name and stuff were rendered so these components, some blocks and blocks got a lot slower so the end result is component initial rendering got slower. That's unfortunate because that's probably one of the most useful feature of Ember or any view [inaudible]. Like on a client side, the way I like to think about it is like if you're writing code and if you've a really long function you probably want to break it down to smaller function for your own sanity. I think components serve a similar role in your templates, like when your template are really big, you probably want to split them into some more modular or hopefully reusable components. If nothing else, just break it down to smaller chunks so you can reason about it. Components are really important and the community is really starting to pick up the idea of components around that time. Unfortunately, we made a change that made components slower so we have to do something about it. That's more or less why Glimmer 2 happened like you're starting out as an investigation into why components got slower, what can we do to make it faster and deeper? We take down on that rabbit hole. We found more things about the HTMLbars engine that is not very aligned with the way we want things to work, which makes it more inefficient than we will like. Basically, it turns out as an effort to realign how the engine actually works with who we actually do with the engine. It start up very incremental and at some point, several things happen that basically cost it to become a complete rewrite. That's basically the backstory of Glimmer 2. CHARLES: Now, here we are, there are these fundamental problems with Glimmer 1 that you can't get to where it is that you want to go, which sounds like these set of optimizations so a rewrite is underway. What were the key things that needed to change in order to make this happen? GODFREY: The overarching theme is basically the original rendering engine. There's not much of a rendering engine to speak off. I think at the time, we don't really fully understand what our requirements are. We don't really know what our needs are so we built something very, very flexible. That's basically HTMLbars and it's kind of self-evident at this because we could take HTMLbars and retrofit the Glimmer 1 [inaudible] from on top. HTMLbars is literally just handlebars plus HTML. It's capable of rendering static HTML and every time it sees a double curly or a mustache, as we would call in handlebars, like every time you see open double curly braces, it provides a set of hooks for you to hook into that process. Every time you see curly-curly, what happens? It says, "This is a block or this is an inline helper," and it just delegate to Ember to, "Please tell me what to do next." That's basically handlebars. CHARLES: If I can kind of interject a question because I think this is kind of a source of confusion, certainly for me. What is exactly the relationship between handlebars.js, like if you go to handlebars.js, the GitHub organization, it is a separate code base, is that just a parser and a syntax? What exactly do the individual libraries contribute? Because we have handlebars.js, HTMLbars, Glimmer 1, Glimmer 2. I think it's a source of confusion that's how are these composed to work in tandem or in tridem when I actually use Ember. Because I know there's people who use handlebars.js that have nothing to do with the Ember community so what's the relationship there? GODFREY: Right. There's handlebars, the language, so to speak and there's handlebars.js, the library. Handlebars is basically just a string templating language. The word 'templating' is probably more confusing than it helps in this context but it's a fancy string interpolation. In JavaScript now, I guess in ES6 you can do it or in Ruby, you have special notations and string to interpolate things. Handlebars is a platform independent language for writing interpolated strings. It has some fancy features like helpers and stuff. But at the very core, it's a way to write really long interpolated strings. Handlebars.js is the JavaScript bindings for that language. As I said, handlebars itself is platform independent but if you have a helper and handlebars like you need to be able to write code for that helper somehow and handlebars.js is the binding that let you write those helper functions in JavaScript. Handlebars.js indeed has a parser in there so you can give it a handlebars template and it would parse into NAST for you. It also has a small runtime library that allow you to register helpers and stuff like that. That's handlebars.js. CHARLES: And HTMLbars is using handlebars.js? GODFREY: In some sense, yes. The original Ember up to 1.10 or something like that, actually uses handlebars.js. It used to be that. We actually do everything as string templates. We just interpolate the string and more or less, just set 'element.' in HTML equals '.string' and that's how Ember used to render things on a screen. Now, HTMLbars does use a part of that pipeline, specifically uses the handlebars parser. How it actually works is there are two possible orders and maybe in the middle of this, I realized I was wrong and it's actually the other order. But I believe how it works, let's say you have an a HTMLbars template or just .hps file in your Ember app, you basically run that through the handlebars parser so you get this some string and then there's some curly mustache here and maybe this is a helper, this is a block, this is whatever. Then I believe we then take all the text notes in the handlebars-jst and then we parse that as HTML tokens. Basically, if you have user.firstname, let's say, and close curly-curly, closed diff, that will basically end up being parsed into a combine jst of this open element of diff and then there's a mustache of user.firstname and then there is a close element of diff. That's the extent that HTMLbars uses handlebars so basically, it uses the handlebars parser to help with some of the work. Now, the part I said I always get wrong is either, you parse it through your handlebars first and then you parse the HTML inside the handlebar-jst or you parse the HTML first and then you use handlebars to parse the curlies inside the HTML-jst. I'm pretty sure we do the handlebars thing first but I'm still not 100% sure. One way or the other, if it turns let's say it parses handlebars first, if it turns out that I'm wrong, I'll probably sent a tweet or something after the show. Does that answer your question? Does that make it more clear? CHARLES: It does because I was always kind of wondering, does Glimmer and Glimmer 2 include its own completely separate implementation of handlebars or not and I think it answer the question of where that boundary lies. GODFREY: Right. Actually, there's a minor detail. I think we're actually on the older version of handlebars. I think there are some new handlebars feature that we need to reconcile. But for the most part, yes, we use the handlebars parser and that's basically the part of the handlebars.js we actually use. JEFFREY: That leads to talking about we used the handlebars parser at the bottom layer. How do you maintain that compatibility? Do you want to have a really great upgrade path from using HTMLbars as the renderer to using Glimmer 1 or 2 as the renderer? Would you talk a little bit about how you maintain that [inaudible] and compatibility? GODFREY: Actually, the handlebars parser part is probably the least interesting part in terms of compatibility. When we did Glimmer 1, I'm kind of speaking as outsider person here because I was not super involved at that time. But from what I hear, after EmberConf 2015, there was a website called IsEmberFastYet.com. It basically count down the number of failing test. Glimmer 1 was kind of a similar situation in that. It was still using the same HTMLbars library and it was part of 1.14 so it still maintained support for a lot of the legacy features like Ember.View and things like that. It's 'easy' in a sense that most of the test are still applicable so more or less, you just rewrite the code and you keep running to test and see what passes or what fails. Then when everything is green, you ship. That's the Glimmer 1 story. However, the devil is in the details. The tricky thing is at the time the test coverage is not very good. As it turns out passing on to test isn't really mean a whole lot so the difficulty in Glimmer 1 is you don't even know who your enemy is really. Not a lot of ways to find out if it actually did the job or not, other than actually just shipping it. As you ship, people start to report a lot of bugs and we're like, "Oh, crap. Actually, this is not tested at all so now we need to fix it." That went on for a long time really. If you actually look at the release history of 1.14, probably have the most number of point releases and that's probably the biggest divergence we ever had from the six weeks release cycle. Unfortunately, what the software is supposed to do is it's unspecified at the time, at least in terms of actual tests so it was a long process and somewhat painful process for the community to figure out what we actually need to support. That was the 1.14 transition. Unfortunately, some apps are still stuck on 1.14 today because of that, because some add-ons or some part of the code bases using some of the features that were previously undocumented or unspecified and it doesn't work on Glimmer 1 anymore. That's Glimmer 1. After that transition, we have a much better test coverage story to work with when we did Glimmer 2. However on the other hand, we actually rewrote the entire engine and also in leading up to that, we actually drop a lot of the legacy Ember features. The test coverage we previously have is useful to have but they're also pretty outdated. Concrete example of that is most of the tests previously written was written in terms of [inaudible] that's no longer a thing in [inaudible] Ember but all of the tests use views internally to test things. The first thing we actually need to do when we try to integrate Glimmer 2 is to port all of those tests, update all these tests and make sure they're still meaningful and modernized them. That was actually a huge task. Thankfully, it's also a task that's very paralyzable. The first thing we did, Yehuda and I spend some time porting those tests and Yehuda being Yehuda we actually spend some time making a really nice abstraction, a really nice test harness for doing that so we started porting some of those tests. But as we start doing that, we realized this is a lot of test to port. Fortunately, it's very paralyzable so what I did was I spent some time writing this really long issue and now are known as The Quest Issue for Glimmer 2, which outline, I for one, this is what we are trying to accomplish. This is going to be a lot of work but fortunately, the work is pretty mechanical or at least the transformation is pretty easy to describe in words. I did that. I described the work that needs to be done and we invited the community to help along. Let's see, that Issue# 13127 on the Ember report. That's known as The Quest Issue and basically, it have a checklist of all the test that still needs to be ported. Surprisingly, a lot of people were really into that, like a lot of people turns out were interested in contributing to Ember but didn't know where to start. This Quest Issue end up being a very good place for someone new to Ember to start contributing. We got a lot of response from that and within a relatively short amount of time, certainly much less than what we would have spent time doing it ourselves. In the relatively short amount of time, all tests were ported. I think that was one of the most successful call-for-help in the history of Ember or even in the history of open source software that I am personally involved with. As part of that too, by doing the harness, we actually introduced much more rigorous approach to testing things. In that process, the test coverage is probably increased several faults as well. With that, we were able to have a relatively high confidence that once all the tests are passing, that we are certainly in a very, very good shape. There might still be some unspecified behavior like there will always be some of them. But I think given the Glimmer 1 transition and also given the new style of testing, that is a lot more rigorous than before, we certainly did that much, much better compared to a Glimmer 1. After doing the original Glimmer 1 and even further back, we start with handlebars and then we wrote HTMLbars, then we wrote Glimmer 1, by now we're actually have a pretty good picture of what we actually trying to accomplish here. The requirements is becoming more clear and at the same time, we have a much better or much clearer model to work with on the Ember side because we got rid of a lot of the legacy features in the transition into Ember 2.0. HTMLbars started off as a very generic library that allow you to add behavior on top of HTML and handlebars. Glimmer 1 was basically loaded on thing using those very generic infrastructure to implement the algorithm we actually want. It turns out that we actually know what we're doing now. We don't need a super generic infrastructure for this so let's make it more tailor-fit thing for the requirements we actually have. An example of that is in HTMLbars, there is no concept of components. There's only thing called block helpers or either inline helpers or block helpers and what happens is every time you see curly-curly something and then you call into Ember, "I found a curly-curly, please do whatever you wish with this," and then we checked and, "It turned out this thing has a dash in there so probably a component. Let me try to resolve that. It turns out it actually is a component so now please synthesize some things and callback into HTMLbars library. This is the thing I actually want to render. Please do it." In Glimmer 2, actually component is pretty fundamental feature so let's build it into the rendering engine. Hopefully that would give you a better idea of what I actually mean by relying the engine with what we actually want. We know we have a pretty good specification of the problem now so we will build the thing that solves that specific problem. When I say, let's build components into Glimmer, it's not even a very concrete thing, like I don't mean the current syntax that happens to be Ember or the hypothetical angle bracket component syntax that's coming to Ember. It is just a very generic concept like there's a thing called component. It's like function calls but in templates. Let's make sure we have a good support off that. Basically, the more the engine knows, the better we are able to optimize that. Previously it's like, it turns out there's a helper-ish thing here so I guess the only thing we need to do is to hand it back into Ember and let it do whatever it wants. Now, the engine actually knows that this is a component so let's try to figure out how we can optimize this component invocation based on what else we know about the arguments and things like that. That's basically the motivation or the design behind Glimmer 2. CHARLES: Right. It's amazing that you find this pattern, kind of cropping up again and again in software where less really is more. The more assumptions that you can make, you can often make the underlying system more powerful. It's kind of counterintuitive because we want to think we want to make everything super flexible and can handle every single case. But it turns out that if you can eliminate a bunch of cases just out of hand, then you're going to have more power. One of the things that I'm curious about is it sounds like a lot of the motivation has been around performance, making sure that initial render is performant and not just a rerender. I'm kind of curious, what nonperformance related features is this going to unlock? This has been a while coming, Glimmer 2 land in what? 2.10? GODFREY: That sounds about right. CHARLES: Looking forward, what are some of the things that we can expect or what are the kind of cool features that this is going to unlock? What's the potential that we're going to see and realized kind of over the next few years based on this work that's been done? GODFREY: In a very technical sense, there are not necessarily a lot of features that are absolutely blocked in a very narrow sense of the works. However, as I mentioned before, everyone kind of felt the same way about the previous code base. Basically, the complexity is getting a little bit out of control like it's so super generic that the things are so spread out across different libraries that's really hard to work with. It's really hard to add new features. All the rendering related features are kind of put on hold as we work on Glimmer 2 because everyone knew that this better code base that's going to come along pretty soon. If we were to do anything significant, we should probably wait for that to finish. The good news is that has finished now and I think most people would agree that code base is in a much, much better shape now. In that sense, it unblocks a lot of work, both in terms of this lock that has been released but it's also in the sense that as we design Glimmer 2, we have all those things in the back of mind and we basically make sure the engine has good support for building out those features. Some of the things that are in the pipeline is I guess rehydration. We basically paused like Glimmer 2 spun off that but we never got back to finishing that work. I would say that's coming. There is angle bracket components. This is actually interesting in that. I think the plan for it to land has changed a little bit. I don't know how much I'm supposed to- Well, there are actually not a lot of secrets but just that someone's in the middle of writing out those RFCs. I don't know if I should [inaudible] shout too much of that. But as I mentioned before, Glimmer 2, the engine has pretty good support for components out of the box, like it's a pretty generic concept of component that's not necessarily tied to a particular syntax and even particular feature set. It's not tied to a particular component API, like this is not forcing you to have a specific base class and stuff. A lot of the features in Ember, like the mount keyword for engines or outlets in Ember or the render helper in Ember, all of those are actually implemented as 'components' under the hood, inside the engine even though they're pretty different things from the curly bracket component that you're used to in Ember today. I think there's a rough consensus on this, like I said, the RC is still in progress so a lot of these things can change but I believe the plan is we'll try to stabilize the core component primitive in the Glimmer 2 engine, that basically let you implement your own component API however you want as a very low level feature. If you remember how we implement the engine, it's kind of similar. We stabilized the core primitives for making engines work under an Ember and then we have an official Ember engines add-on that uses those primitive to provide the actual user-facing API when you want to support. I believe angle bracket component would be similar in that. We would try to stabilize the core primitive of implementing components in the Glimmer engine, then we'll iterate on the new Ember component API, probably as an add-on or something like that. We can get more people to try it and improve the API before actually land that as a core feature in Ember. CHARLES: What I'm hearing is there's actually the potential here to treat the kind of the core component API as an extension point like if I want to define my own different set of semantics and lifecycle hooks and what have you for my special components within my add-on or my Ember application, I could do that and yet have it exist alongside kind of standard components. GODFREY: Exactly. It's a pretty low-level API and I don't think most users would want to go for that. But I think it unlocks a lot of potential for add-on and things like that. Because as I previously mentioned, the [inaudible] in Ember is implemented using that API to render helpers, implement using that API so hopefully you can imagine there are probably something that people really wanted to do and add-on step could be made possible with that low-level API. It might not even look like a component. It could be things like you do in Liquid Fire. I think, [inaudible] has a suite of add-ons that provide some pretty events rendering features that could fall under that umbrella. Basically, it's just opens up a pretty powerful low-level primitive that add-on authors can use to implement the features they want but at the same time, also allow us to iterate on that. I don't imagine a future where literally everyone is building their own component API. I think we would still, as a community have basically this system main kind of component you use in Ember but for whatever special case you have, you can drop down to the low-level and do whatever you want. CHARLES: Right, and at least that area for experimentation and innovation is at least there. GODFREY: Right. CHARLES: That actually kind of brings to mind a very specific case that I was thinking about so I kind of throw this at you as a very friendly curveball that like this is something that you could conceive of being possible. I recently was working on an Ember application with my dad and there were some visualizations. Originally, I was doing these visualizations in D3 and it was cumbersome as D3 code can actually get. The part where you actually compute the D3 scales and the SVG attributes in D3 is very concise but the actual D3 rendering code can be very verbose and can kind of degenerate into this jQuery type like soup. What I did as kind of a first pass is I separated the computations and the slicing and the dicing of the data in D3 and presented those to a simple HTMLbars template as computed properties of a component so I had done all the computation around the visualisation in my Ember component. But then for the template, I used just a simple SVG template and it was beautiful. It was this pretty complex SVG visualization fit into, I want to say, less than 20 lines so you could perceive the entire visualization in one standard editor window, which if you've done a lot of D3 code is actually hard to achieve. When I saw that and I promise I'm going to make a point here, I was like, "Wow, this is like a powerful concept." I feel like this is the way that we ought to be doing SVG visualizations. But the thing that I lost was the thing that kind of this special sauce of D3, which is I wasn't using D3 to actually do the rendering. I was just doing it for the computation so I didn't get those really sweet transitions. One of the sweetest concepts the D3 has is this join model where when you make a change and you push a new set of elements onto the visualization, it computes for you the SVG, the actual DOM elements that are leaving, the ones that are entering and the ones that are updating. I was thinking, "Gosh, it would be really great if there was some way to hook into that inside of Ember," and I'm just kind of curious would such a thing be possible to actually get hooks on the low-level elements? Because essentially, that's what a dom-diff is, which elements are leaving, which ones are coming in, and which ones are changing. Would it be possible to have those hooks with Glimmer 2? GODFREY: I think it's definitely possible. I don't know how much of that you can actually do right away in the current code base but I think conceptually, that makes a lot of sense and that's definitely something that we should have support for, if we can already do that today. What can be adapt was I haven't spent a lot of time doing animations myself, but I would say that makes a lot of sense. I hope most of that is already achievable today. But if not, we should definitely make it happen. I think a lot of people didn't realize but we actually spend a significant amount of time on our end, making sure that we have good support for SVG. We do have a spec compliant HTML parser and stuff so it's actually rendering SVG with Glimmer or with Ember, which is actually really nice. We have some of those animations or visualization stuff in Skylight and we actually take a similar approach where we actually have SVG templates and we just use Ember to fill in the dynamic data and we have some, I believe, JavaScript code to do the animation but we should definitely talk more about that topic and making sure it's solid. CHARLES: You know, I can't reiterate how sweet it feels to do as SVG. Really, when I kind of stumbled upon it, I was like, "Wow, man. Someone needs to do this more." GODFREY: I believe in the latest iteration of D3, they actually broke up the library into pieces that makes some of those things much more easier. I think that's definitely at direction that everyone wants to have to work. CHRIS: I know that you talked about that there are people writing RFCs and EmberConf is coming up so I'm assuming that there are possibly some things that are going to be talked about there. But one of the things that has really kind of floated around Glimmer 2 ever since last year's EmberConf is that in terms of the next leap forward for Ember, Glimmer 2 was often pointed to as the blocking thing. You know, angle bracket components, a lot of other features that people would like to see in Ember, a lot of it seemed to be bound up and Glimmer 2 was going to unlock a lot of these things for us. I'm curious for an end user of Ember, what is Glimmer 2 really mean for them in terms of the next six months to a year? Are there a lot of things that are planning to be unlocked by Glimmer 2? Or is it going to be much more of an incremental progress? Or do we actually know everything that's going to be unlocked at this point? GODFREY: I think it's a combination of all of those. Like I said earlier, there are probably features that along the lines of the angle bracket component or the low-level component primitives rehydration. There is also this concept of chunk rendering in that, basically, do as much rendering as possible without making a significant pause on the UI [inaudible] and then yields expected browser for user interaction and then go back to rendering more stuff and repeat that until you're done. The net effect is you basically, don't block scrolling and things like that for a significant amount of time. Even though, your initial render might actually take more than the 60 FPS or whatever time frame you have allocated. Also, performance in general, I think we're very much not done here. We're just getting started. We have a much better engine to work with and there are a lot of further optimizations that we want to do so now, that we've shipped with completed compatibility, we climb in, more or less I think everyone is excited to get back on to those things that they were previously working on. CHRIS: Another one I have seen kind of reference a couple times, especially by Tom is Glimmer components and Ember CLI as kind of like a lightweight Ember without all of Ember. Even if you poke around the Glimmer repos, a lot of benchmarks that you can see a version of [e] components that don't yet exist, you see TypeScript, you see ES6 extends Glimmer component. A lot of things that I know people would be absolutely thrilled to see in Ember, a standalone Glimmer like planned to be a thing at all and like [inaudible] syntax, like ES6 class components is that as they make Glimmer 2 unlocks it all or are there other things that are blocking that? GODFREY: Yes, all of those are definitely in the pipeline. Standalone Glimmer is interesting. As you mentioned, Tom and several other people have started experimenting with that today. I think Tom talked about it in his EmberCamp London keynote this year or last year -- 2016. If you haven't seen that, you can definitely go check that out. It's not something I would recommend doing today in production just because there are a lot of [inaudible] in the code base still so we need to stabilize that. Literally, when I went on vacation for the last three weeks, you could basically rewrote the entire VM. It's some work that we have been talking about for a long time and he basically end up having some free time to check on that. Everyday changed after I came back from vacation so if you do that today, that's probably what you would have to deal with in the foreseeable future in the next few months. But I think we are definitely working towards making that more stable so we can enable the kind of things that Tom is doing. Part of it is definitely we need to make it easier. Currently, Tom has to do a lot of work to make that happen and Ember CLI has to be a core part of the story and etcetera, etcetera. But I think a lot of people were definitely working towards that goal. In terms of TypeScript ES6, TypeScript is actually pretty interesting. It's kind of another happy accident that came out of the Glimmer 2 rewrite. As I mentioned at the beginning, when I was new to HTMLbars code base, there are so many concepts and Yehuda and I both travelling a lot at the time so every time we reconvene, it was like, "Wait a minute. I forgot to read things so let's actually take the whiteboard and write down all these things here permanently so I won't forget them." As we start to do that, it was like, "Actually this is very complicated to write on the whiteboard so let's just write out the types somewhere," not in a very rigid sense but just, "Oh, there's this thing called render node and this is basically the responsibility. These are the methods on that thing and these are the fields on that thing." As you write that you would probably want to remember, "Oh, this is actually a string or this is an object with these sub-properties," and as we actually tried to do that, it was like, "Wow, we should not actually invent." We either have to invent a notation for that or we could just use a notation that people have made for this purpose already so let's just go to the TypeScript playground and type it out there. Then that turned out to be great but it's not that amazing to write. Many interfaces in the browser so let's just go back to the editor window and type in there and it turns out TypeScript is written in a way that all [inaudible] JavaScript is fellow TypeScript. Basically, a superset of JavaScript. Eventually what we ended up doing is we just went through an entire code base and renamed all of the JavaScript file to .ts, then we made a commit of that. Incrementally, we end up switching the entire code base to TypeScript and that's actually how it happened. In terms of using TypeScript and ES6 classes and Ember is probably block by few RFCs. There are probably some design work needed there but I would say that's happening. I don't know how close it is but just because there are a lot of things in the pipeline right in it. But I think it's a relatively high [inaudible]. CHARLES: Before we wrap up, we had some great listener questions that came in over Twitter. I want to put those to you. The first one comes from Elrich R and the question is, "What are the performance gains on mobile for Glimmer 2? How does this better position Ember for mobile development? GODFREY: I think the short answer is it's faster. As far as how much faster, unfortunately, a really hard question to answer just because it vary so much from app to app. We have some numbers that we have been looking as we develop it but at the end of the day, each app is very, very different and the kind of things that slows your app down might not be very obvious. There's a pretty interesting thing that happened when we were [inaudible] Glimmer 2. One day I was checking the Ember communities Slack and someone reported, "I have this list of 1500 items that I'm rendering. I know that's not ideal but it is what it is for now. But I just tried Glimmer 2 and it's really slow. It's taking three seconds to render that list of 1500 rows. Is there something wrong?" I was like, "That sucks." We tried to make this thing faster but I agree, three seconds is still really slow. I was like, "Can you share your app? Can you check what is the delta? What was the number pre-Glimmer 2?" He's like, "Let me go and check. Actually, it turns out it was 14 seconds pre-Glimmer 2 and it was now three seconds." I was like, "Yeah, okay. I agree that's not great but with 1500 items and we went from 14 seconds down to three seconds, I think I would take that as a win." Part of it is difference like that, really is how complicated your page is. Also there are a lot of factors that are actually unrelated to rendering, perhaps surprisingly, all of which we're working on but for example, how many routes do you have in your app, actually for some reason, plays a pretty significant role on the initial render timing and also how big your app is and things like that. I think across the board, it's pretty much faster. As you can see sometimes, it's 14 seconds, three seconds for small apps. It might be from 500 milliseconds to 400 milliseconds. In Skylight I think, we're seeing anywhere from 7% to 40% overall and we're talking about micro numbers just like everything from download in JavaScript to everything up until the page is interactive. There's a lot more than rendering going on there but I don't have a good one-size fits-all number for you here. But I think overall it's faster and we'll make it faster still and we'll continue to make it smaller. Hopefully all of that would be meaningful improvement for mobile or even just performance in general, if that's something you care about in your apps. CHARLES: Yeah, because I understand that the payload size is a lot smaller and that ought to make a huge difference there. The final question is and this one comes from Lauren T, "What is your favorite flavor of Bubble Tea?" GODFREY: Interesting. I love Bubble Tea but for whatever reason, I'm just not really into milk teas. I know that's what most people go for when they get Bubble Tea and my personal favorite is probably Honey-Lemon Green Tea. That's probably my go to. CHARLES: That's delicious. I think, I subsisted wholly on those AriZona Iced Tea in 20-ounce cans from about the age of 17 to 18. There might have been a few months where that represented all of my caloric intake. GODFREY: I think I once bought a one-gallon pack of those from Wal-Mart and unfortunately, the bottle broke in my trunk on my way home so I basically stopped buying those after that. CHARLES: You get the nice smell of rancid tea. All right-y. We'll thank you so much for coming by Godfrey. GODFREY: Thank you again for having me. CHARLES: Yeah, it had been very enlightening. A deep dive into the what the Glimmer 2 is all about, the challenges that you faced of getting there and where we can expect to go now that it's here. Oh, let me ask one final question before we send it out. Glimmer 2 has been a long process. There's been a lot of hills to climb and I know a lot of people that I've talked to, as part of the community, one of the biggest questions I have is how can I get involved? How can I help? Where can I throw my weight to see this technology blossom and become really, really useful and widely applicable to everybody inside the Ember community? Just in general, where are more of those quests that you just described? Where can people go to find out more to contribute more? GODFREY: There are several places. First of all, there is the Ember communities Slack and there's [inaudible] Ember channel and there's also a [inaudible] Glimmer channel. I think we might merge them back into a single channel at some point. But currently, those are where people hang out. In terms of quest issue, actually really counting to it in terms of the time investment versus the payoff. If there's one thing that I wish I did earlier and wish I did more is probably I would have started these quest issues earlier and do more of them. I guess we have to write more of them. But basically, hang on those channels. A lot of times, people asked questions then we try to help them out. A lot of times those end up turning into opportunities for contributions. It's kind of like documentation. I don't actually hate writing documentation but it's not the first thing that comes to mind, in terms prioritization. I think writing Quest Issue is kind of like that one. You actually do it, you realized, "Wow, this is actually really helpful and I should do it more," but it's just so counterintuitive that it's often not the first thing in trying to do when I try to prioritize my day. It's just something I have to keep reminding myself and I appreciate the reminder here that I need to write more on these issues and get more people, empower more people to be able to help. CHARLES: Alright-y. Well, you heard it folks. Thank you very much and we will see you all next week.

stand walmart vancouver bc dom react slack looking forward chan api concrete ui github components vm javascript html rails fps contributing godfrey rc d3 glimmer nast animations typescript jquery svg bubble tea skylights hacker news handlebars tilde key changes incrementally es6 arizona iced tea chris freeman rfcs frontside emberconf ember cli fastboot charles you ember data

Episode 85: Open Issues For Private API Usage

Play Episode Listen Later Dec 19, 2016 29:39

Chase and Jonathan are again joined by Chris Thoburn to conclude his interview discussion topics from Smoke and Mirrors, to Igniter, to some potential changes in Ember Data.

private smoke mirrors usage igniter ember data

Episode 84: Run Ember Data Canary

Play Episode Listen Later Dec 11, 2016 31:04

Chase and Jonathan are joined by Chris Thoburn and discuss Ember Data. From performance benchmarking, to new updates in 2.11 we hit it all.

canary ember data

Episode 49: I found your subtweets

Play Episode Listen Later Mar 6, 2016 64:20

Chase and Jonathan are joined by frontsider and Ember Data core team member Stanley Stuart discussing everything from Fastboot to the future of Ember Data. Be sure to tune in for this exiting episode!

fastboot ember data

Episode 46: Team Tofu Burrito

Play Episode Listen Later Feb 14, 2016 14:23

Chase and Jonathan debate episode titles, Ember Data updates, metaphors, Ancient City Ruby, Angular, magic and Real World Ember.

burritos tofu angular ember data

Episode 41: And We're Back

Play Episode Listen Later Jan 10, 2016 18:17

Chase and Jonathan discuss how Ember is not Rails, Ember Data has been addonized, Emberland is back, and Ember is no longer an MVC framework. Welcome to 2016!

re back rails mvc ember data

Episode 35: Copy Pasta Problems

Play Episode Listen Later Nov 15, 2015 10:17

Chase and Jonathan talk about the Ember Inspector, Ember Data as an addon, Ember Conf CFP, application architecture, and Ember views

copy pasta ember data

Episode 34: Adventures in the Amazon