Podcasts about activerecord

discover explore threads api open source github rails contributing turk ruby on rails async daily work quenneville activerecord weather machine

Play Episode Listen Later Sep 10, 2024 34:11

How can asynchronous programming transform your Ruby on Rails applications? Today, Stephanie sits down with Hello Weather co-creator Trevor Turk to unpack asynchronous programming in Ruby on Rails. Trevor Turk is a seasoned software developer known for his work on Hello Weather, a minimalist weather app that delivers essential weather data quickly and precisely. He's also the creator of Weather Machine, an advanced weather data platform designed to serve reliable and highly accurate forecasts via API. With a background that includes work at innovative tech companies, Trevor brings years of experience in developing intuitive, user-friendly digital tools. Trevor talks about the focus of his API work, the complexity of web-based apps, and what makes Hello Weather unique. He explains the fundamentals of asynchronous programming within the Ruby on Rails framework and why it is an approach all programmers should consider. Explore the nuances of programming for different data sources, how he leverages fibers and threads for the Hello Weather platform, and why asynchronous programming is not a silver bullet for application development. Discover how to start using asynchronous methods, the various asynchronous tools available in Ruby, and why experimenting with concurrent programming is essential. Join us to gain insights into why including asynchronous tools is vital for the Ruby on Rails ecosystem, improving platforms through open-source development, how to help improve the adoption of asynchronous tools in Ruby, and more. Tune in now! Key Points From This Episode: Introduction to Turk and his background in Ruby on Rails. Details about his companies Hello Weather and Weather Machine. The innovative features that the Hello Weather platform offers. Hear how Hello Weather transitioned from a web-based to an application. Why he needed to alter his programming approach to scale the company. How he came across the concept of asynchronous programming. Discover how using fibers is different from using threads in Ruby. Find out about the different use cases of asynchronous programming. Learn about the benefits of implementing concurrent programming. Trevor shares the challenges of working with different versions of Ruby. His role in enhancing asynchronous methods within the Ruby framework. Common misconceptions of working with Ruby on Rails. Final takeaways for those interested in asynchronous programming. Links Mentioned in Today's Episode: Trevor Turk on LinkedIn (https://www.linkedin.com/in/trevorturk/) Trevor Turk on X (https://x.com/trevorturk) Trevor Turk on Threads (https://www.threads.net/@trevorturk) Hello Weather (https://helloweather.com/) Weather Machine (https://weathermachine.io) GitHub | async gem (https://github.com/socketry/async) GitHub | falcon gem (https://github.com/socketry/falcon) 'Async Ruby on Rails' (https://thoughtbot.com/blog/async-ruby-on-rails) load_async (https://api.rubyonrails.org/classes/ActiveRecord/Relation.html#method-i-load_async) Episode 437: Contributing to Open Source in the Midst of Daily Work with Steve Polito (https://bikeshed.thoughtbot.com/437) GitHub | Action Cable server adapter (https://github.com/rails/rails/pull/50979) ActiveRecord connection checkout caching (https://github.com/rails/rails/pull/50793) Ruby on Rails The Bike Shed (https://rubyonrails.org/) The Bike Shed (https://bikeshed.thoughtbot.com/) Joël Quenneville on LinkedIn (https://www.linkedin.com/in/joel-quenneville-96b18b58/) Support The Bike Shed (https://github.com/sponsors/thoughtbot)

Auditing in Rails & Andrew's Cursed Idea

press idea cursed rails auditing paper trails audited railsconf activerecord

Play Episode Listen Later Jun 21, 2024 46:26

In this episode, Chris and Andrew dive into the intricacies of tracking changes in Rails models using gems like Paper Trail and Audited. They discuss challenges faced in bulk actions like 'update all' and 'destroy all' that don't trigger Active Record callbacks. The conversation explores potential solutions, including overriding methods and using wrappers to ensure changes are logged efficiently without significant performance hits. They also touch upon mentorship and the importance of learning fundamental Ruby skills to master Rails development. The discussion also extends to experiences at RailsConf, the impact of community interactions, and reflections on career growth through continuous learning and mentorship. Press download now to hear more! HoneybadgerHoneybadger is an application health monitoring tool built by developers for developers.Disclaimer: This post contains affiliate links. If you make a purchase, I may receive a commission at no extra cost to you.

428: Ruminating on Ruby Enumerators

Play Episode Listen Later Jun 11, 2024 35:44

Joël explains his note-taking system, which he uses to capture his beliefs and thoughts about software development. Stephanie recalls feedback from her recent RailsConf talk, where her confidence stemmed from deeply believing in her material despite limited rehearsal. This leads to a conversation about the value of mental models in building a comprehensive understanding of a topic, which can foster confidence and adaptability during presentations and discussions. The episode then shifts focus to the practical application of enumerators in Ruby, exploring various mental models to understand their functionality better. Joël introduces several metaphors, such as enumerators as cursors, lazy collections, and sequence generators, which help demystify their use cases. Episode on note-taking (https://bikeshed.thoughtbot.com/357) What we believe about software (https://bikeshed.thoughtbot.com/172) Ruby Enumerators (https://ruby-doc.org/3.3.1/Enumerator.html) Enumerator Lazy (https://ruby-doc.org/3.3.1/Enumerator/Lazy.html) Modeling a Paginated API as a lazy stream (https://thoughtbot.com/blog/modeling-a-paginated-api-as-a-lazy-stream) Solving a memory performance issue with enumerator (https://thoughtbot.com/blog/how-we-used-a-custom-enumerator-to-fix-a-production-problem) Find in batches (https://api.rubyonrails.org/classes/ActiveRecord/Batches.html#method-i-find_in_batches) Binary tree implementation with different traversals (https://gist.github.com/JoelQ/02f3ef9f61bebc7c8e5ea67d10ed92c6) Teaching Ruby to Count (https://www.youtube.com/watch?v=PHMOsTK1jSE) Transcript: STEPHANIE: Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Stephanie Minn. JOËL: And I'm Joël Quenneville, and together, we're here to share a bit of what we've learned along the way. STEPHANIE: So, Joël, what's new in your world? JOËL: So, what's new in my world isn't exactly a new thing. I've talked about it on the podcast here before, and it's my note-taking system. I have a system where I try to capture notes that are things I believe about software or things I think are probably true about software. They're chunked up in really small pieces, such that every note is effectively one small thesis statement and a paragraph of text, and maybe a diagram or a code snippet to support that. And then, it's highly hyperlinked to other notes. So, I sort of build out some thoughts on software that way. A thing that I've done recently that's been pretty exciting with that is introducing a sort of separate set of notes that connect to my sort of opinion notes. So, I create individual notes for public works that I've done, things like blog posts or conference talks. Because a lot of those are built on top of ideas that have been sitting in my note system for a while. Readers and listeners get to sort of see the final product, but often sort of built up over several months or even a couple of years as I added different notes that kind of circled a topic and then eventually got to a thing. What I did, though, was actually making those connections explicit. And so I use Obsidian. Obsidian has this cool graph view where it just sort of shows all of the notes, and it circles them with, like, connections between them where the notes connect. So, I can now see in a visual format how my thoughts cluster in different topics, but then also which clusters have talks and blog posts hanging off of them and also which ones don't, which ones are like, oh, I have a lot of thoughts on this topic, and I've not yet written about it in a public forum; maybe that would be a thing to explore. So, seeing that visual got me really excited. I was having a good time. STEPHANIE: Yes, I have several thoughts coming to mind in response, which is, I know you love a visual. I really like the system of, even if you have created content for it, like, you have a space for, like, thoughts about it to evolve. Because you said, like, sometimes content comes out of notes that you've been...or, like, thoughts you've been having over years, but it's like, even afterwards, I'm sure there will still be new thoughts about it, too. I always have a hard time finding a place for that thing kind of once I, I don't know, it's like some of that stuff is never really considered done, right? So, that is really cool. And I also was just thinking about an old episode of The Bike Shed back when Chris Toomey and Steph Viccari hosted the podcast called "What We Believe About Software," I think, is the title. And I was just thinking about how, like, if only we could just dump all of your notes [laughs] into some, you know, stream [laughs], and that would be really cool. If we ever do, like, an episode like that, that would be really fun. And I'm sure, you know, you already have this, like, huge bank of ideas [laughs]. JOËL: Yes. It is really fun because I build up...the thoughts are often sort of interconnected, and so they might have a topic, but they are very focused. So, I might have, like, three or four things I believe about a particular topic that cluster together. So, we could...and, actually, I have used, in the past, some of those clusters as initial food for thought for a Bikeshed episode. STEPHANIE: Yeah, that's really neat. I like this idea of a kind of just, like, a repository for putting down what you believe about software as kind of, like, guiding principles for yourself as a developer a little bit. I remember a piece of feedback I got about my RailsConf talk that I gave a few weeks ago, and someone said like, "Oh, you sounded really confident in what you were talking about." And that surprised me because I, like, didn't practice rehearsing giving the talk all that much [laughs]. It's because they had asked like, "Oh, like, did you practice a lot?" or something like that. And I think I realized that I, like, really believed in what I was sharing and kind of that, I think, was perhaps what they were picking up on. And even though, like, maybe the rehearsal of the presentation itself was not where I had spent a lot of time on, I had spent a lot of time thinking about what I wanted to share and just building up my confidence around that. So, I thought that was an interesting connection. JOËL: Yeah, you fully developed the idea. You kind of explored all the side trails, maybe a little bit on your own as well. You're on very familiar terrain. And so, that is a way of building confidence separate from just sort of memorizing a talk. STEPHANIE: Yeah, yeah. Exactly. JOËL: In a sense, I almost feel like that's a better sense of confidence because then you can sort of...you can roll with the punches. You know, if a slide is out of order or something, sure, it maybe messes up a little bit of the narrative that you're trying to say. But you're not like, "Oh no, what is this content?" You're like, "Oh yeah, this thing," and you can dive right into it. Somebody asks you a question, and you're not like, "Oh no, that was not in the script," because, again, you've sort of mastered your topic. You know the area as a whole, even sort of the blurry edges beyond the talk, and can react in a way that is pretty confident. STEPHANIE: Yeah. I still definitely fear the open Q&A. I've never done it before, but maybe one day I will be able to because I just, you know, know my topic so well inside and out [laughs] that I can roll with the punches, as you say. JOËL: Open Q&A is just...it's a roll of the dice. Sometimes, you get some really good conversation topics there, and sometimes, it's just a waste of everyone's time. STEPHANIE: I like that take [laughs]. JOËL: Maybe that should go into the things I believe about software. So, other than receiving feedback about your RailsConf talk, what is new in your world? STEPHANIE: Yeah, so I am wrapping up a pretty large project on my client work that we're hoping to release soon. And, in fact, it's actually being released along with a big announcement from the client company to their customers. Essentially, at a conference, they're going to say like, "Hey, like, we now have this new feature." And so, I think there's some hype generated around it. And this past week, we've been doing a lot of internal testing of the feature because there are a lot of employees of my client company who are, like, pretty big users of the product, which is cool because I think we're getting, you know, we have easy access to people who can give us good feedback. But I am having a hard time with being on the receiving end of the feedback and figuring out, like, what is stuff I need to attend to now before, you know, this big release? And what is stuff that is just kind of, like, general feedback like, "Oh, like, I wish it did this," but, you know, it turns out that that's not really what we were building? And how do I just kind of, like, accept that? You know, it's coming from a good place, but I can't really help them there, at least right now. And that's hard for me because I like helping people, right? And so, if someone says something like, "Oh, like, I wish it did this," or like, "Oh, that's kind of weird," I'm like, "Oh, I want to just, like, fix that for you right now [laughs]." And I suspect that a lot of other devs can relate to this, especially if, like, you know, you've been working on something for a little bit, and it feels...I'm just going to say it: it feels a little precious to me. So, what I'm trying to do today, actually, is not look at any of the feedback at all [laughs] and come at it tomorrow with a bit of a calmer vibe and be able to separate out, like, you know, I think all feedback is informative, but not all of it is useful for you at any given moment. Like, if there are bugs, then those will be my immediate priority. If there's maybe some small tweaks that we can make the feature just a little bit more polished, then I also think those are good. But then we are discovering a few things, too, about, like, what this feature is or could be. And I think those are the things that, you know, need to be brought into a conversation with a broader group and think about, like, is this the direction we want to go? So, that's kind of how I'm bucketing that feedback right now. JOËL: How do you feel about receiving direct feedback versus having something filtered through something like a product team? STEPHANIE: Ooh, that's an interesting question. Because right now we're doing, I think, a mix of both that I'm not sure that I really like. On one hand, when it's filtered, it's hard to get to the root of what someone is asking for. And oftentimes, like, it may not even include enough information after the fact to be able to come at it from a dev perspective. But then direct feedback, I think, is just a little bit overwhelming sometimes. And it can be hard to figure out what to pay attention to if you don't have that, like, input from a product team about, like, what the roadmap is looking like or where, you know, strategically their heads are at. So, one thing that kind of has emerged from this is like, oh, I was getting, you know, notifications for the feedback coming in. And what we did was set up a meeting [laughs] so that we can...maybe all of us can, like, scan it together ahead of time and then come at it with a little bit of context about what's come in but then maybe coalesce around the things that we feel are important. JOËL: Well, you'll have to keep us updated on how that plays out, and we can kind of hear what is the balance that ends up working well for you. STEPHANIE: Yeah, I hope so. I think this is actually maybe something that's a bit underexplored from the dev perspective, you know, that in-between stage of you're not totally done because it's not shipped to the world yet, but, you know, you're starting to get a little bit of that input. And what you do with that? Because I think there is some value in being engaged in that process. JOËL: So, we were talking earlier about this note-taking system that I use and sort of a renewed excitement that I have about it. And one thing that I did when I was going through and finding clusters of things that hadn't been written about was I found that I had a cluster of notes on different mental models that I had for understanding Ruby enumerators, not the enumerable module, but the enumerator object. And I decided, you know what? This would probably make for a good blog post. So, I drafted a blog post, and I've been thinking about this a little bit more recently. So, I've been really hyped about digging into enumerators because of that experience. STEPHANIE: Yeah, that's very cool. I have to say that I feel like I did not know a lot about enumerators and the API for them kind of before you brought this topic up, and I did a bit of a deep dive in preparation for us to discuss it. I feel like most devs, you know, work with enumerators via methods on enumerable without totally knowing that they are. So, I think that this would be a really interesting episode for people to be like, oh, like, I've been using this stuff, you know, the whole time, and now I can have a different perspective or just more insight on what they can do. JOËL: Before we dig into individual mental models, though, I want to think a little bit about the concept of mental models as a whole. Years ago, someone gave me advice to sort of pay attention to mental models, ways I think about the world or different code structures, different code approaches, and that really stuck with me. So, I've since been, like, kind of, like, collecting mental models. And, in a way, they're like a, for me, a bit more of a concrete way to look at a particular topic. So, I can say I'm looking at this particular topic through the lens of a particular mental model that helps me build more clarity around it. And if I have three or four, then I can kind of look at it from three or four different perspectives. And now, all of a sudden, I feel like I'm seeing in three dimensions. STEPHANIE: Whoa, the Matrix even [laughs]. That's cool. Yeah, I really like that advice. I think I'm going to steal it and start kind of suggesting it to other people because I think, in a way, on this show, that has come through a lot. And talking about things on the podcast has helped me develop a lot of my mental models. And I think we've done a few, like, episodes in the past about various ones we have for just our work because it's like, that's infinite [laughs]. But what I really have been appreciating is that mental models just need to work for you. As long as you're able to understand something, then it's valuable. And that has really helped me also, like, just get on the same understanding with others because the goal is not necessarily to, like, explain it the way that I would think of it, but figure out what would help them kind of develop their own mental model for understanding something, and, you know, kind of as long as we both feel like we have that shared understanding, no matter what lens it's through. And, you know, sometimes it's even more effective when we are able to share it. But I feel like, you know, you can still find ways to collaborate on something with a diversity of mental models. JOËL: Yeah, they're a great way to build self-understanding. They're a great way to sort of build understanding between two people. So, I'm a huge fan of the concept. And part of what I've been doing with my note-taking system is trying to capture those as much as possible. If I'm ever, like, trying to understand a complex topic and I'm like, oh, I think I've got a breakthrough here; I understand it; it's kind of like this, or you can imagine it in this perspective, it's like, write that down. That's gold. STEPHANIE: Very cool. So, Joël, would you be able to share some of your mental models for enumerator? JOËL: So, one way that I look at it is the idea that an enumerator is effectively a cursor over a collection. So, you have an array and a regular array; you're either in the middle of iterating through it using something like each, or you're not. You just have a collection of items. Enumerator introduces the idea that you're actually sort of at a position in the array. So, you're sort of focused on, let's say, the third item or the fourth item. You have a cursor there, and you can move that cursor forward as you sort of step through. But the really cool thing is you can also kind of pause and just pass that cursor on to someone else, and someone else can move the cursor a few steps further down the collection, pause, pass it on to someone else. And it's totally fine. Nobody has to, like, go through an entire, like, each iteration. STEPHANIE: Yeah. So, when you were talking about cursors, that got me thinking a little bit because I actually have struggled with that concept, especially when it comes to, you know, things code-related. Like, when I've had to work with database things and stuff, like, the idea of a cursor was a little, like, difficult for me to wrap my head around. And I was looking at the methods on enumerator, like the instance methods on enumerator. And one of them actually is what helped me develop this mental model. And I'm excited to see what you think. But there is a rewind method that basically rewinds the sequence back to its beginning, right? And what that triggered for me was a VHS tape [laughs] and just those, like, car-shaped rewinders for tapes back in the '90s. I don't know if you ever had one in your house, but I did. And I just thought that was such a cool method name because it was very, I don't know, it was just like a word that we use in the English language, right? So, the idea of, like, tapes, you know, like, cassette tapes or VHS tape kind of also it sounds like it matches well with what you were sharing, too, where it's like, I could pass, I don't know, maybe I, like, listen to a few songs on my cassette tape, and then I give it to someone else, and they can pick up where I left off. And yeah, that was really helpful in understanding, like, a marker of a position a little more than cursor was able to for me. JOËL: That's really interesting because now I wonder, like, how far we could push that metaphor. So, musical data is encoded on magnetic tape. Cassette tapes typically there are sort of two spools. You start off with all of the tape wound up around one spool, and then as it sort of moves across the read head, it gets wound up on sort of the, I don't know, destination spool. I guess you can call them origin and destination. And because of that, you can sort of be in a, like, partly read state where, you know, half the tape is on the destination spool, half of it is on the origin spool, and you have that read head that's in the middle, and you're just kind of paused there. And you can kind of jump forward in that. So, I imagine something like that in your metaphor is like an enumerator. Contrast that to imagine just a single spool, which is just we have musical data encoded on magnetic tape, and we wrapped it up on a spool. I feel like that's almost more like a regular array because you don't have that concept of, like, position, or being able to read parts of it or anything like that. It's just, here's some data. STEPHANIE: Yeah. While you were talking about the two spools, I was thinking about, like, part of what is nice about enumerator is that you can go forward or backwards, right? And that feels a little more possible with that two-spool metaphor [laughs], rather than just unraveling something, where you are kind of discarding what has already been read. JOËL: The one caveat there is that enumerators can move forward one item at a time. They can only move backwards by jumping back to the beginning. So, you can't step forward or step back. STEPHANIE: Yeah, that's fair. JOËL: You step forward, or you, like, rewind to the beginning. I think, in my mind, I was thinking a little bit more about this metaphor. And I think it's also just a metaphor for what's called the External Iterator Pattern. It's one of the classic Gang of Four Patterns, which is what enumerator, the object in Ruby, is an implementation of. I feel like I always see that in the documentation, like, oh, enumerator is an implementation of the External Iterator Pattern. And I just kind of go, what? STEPHANIE: [laughs] JOËL: Or maybe I kind of understand the idea of, like, okay, it's a way to, like, be able to step through a collection. But thinking in terms of a cursor or even your model as a cassette tape, I think that gives me a model, not just for enumerators, but then for better understanding that external iterator pattern. Like, I'm now not going to think of if I'm ever reading through the Gang Of Four book, or some other languages say we're an doing External Iterator Pattern, and I'll immediately be like, oh, that's a cursor, or that's a cassette tape. STEPHANIE: Yeah, very cool. I like it. JOËL: Another mental model that I have is thinking of enumerator in terms of a lazy collection. This is something that you tend to see more in functional programming languages, so the idea that you have a collection of potentially infinite length, or it could even be unknown length. But each element only sort of comes into being as you attempt to read it. So, it's kind of, like, a potentially infinite chain of Schrodinger's boxes. And you've got to open each of them to find out what's inside. STEPHANIE: Do you know what this reminded me of? Like elementary school math questions that were like, "What comes next in this pattern?" And it has, like, you know, the first, like, four or five values in a sequence or something. And then, you have to figure out, like, what the next value is. But then, in some ways, you know, I think it can depend on whether your enumerator is using the previous value to determine the next one. But yeah, it's like, you can't just jump ahead to figure out what the 10th, you know, value in this pattern is without kind of knowing what's come before it. JOËL: And sort of that needing to step through the entire collection, sort of one element at a time. STEPHANIE: Yeah, exactly. JOËL: I think a way that that concept is interesting, to me, is situations where a collection might be expensive, and you don't necessarily need all of it. So, you might have a bunch of calculations, but you can stop when you've hit the first one that succeeds or that matches a certain criteria. And so, it's not worth it to calculate the entire array of calculations if you're going to stop at the third one. And you could do that with some sort of, like, loop or something like that. But having it as a collection means you get to just treat it like an array, and you can call detect on it and do all the nice things that you're used to. It just happens to be a little bit more efficient in terms of not creating more data than you need to. STEPHANIE: Yeah. And I think there's some really cool stuff you can do when you start chaining enumerators with this concept of it being lazy evaluated. So, one of the things I learned in my deep dive is that when you are using the lazy method, you're able to chain enumerators. And they work a bit differently, where the default functionality is, like, everything in the collection gets evaluated through the first method, and then it gets iterated over in the second method. Whereas if you use lazy, I believe how it works is that, like, the first value gets kind of processed by all of the methods. And then, you get, you know, the output before moving on to the second, like, the next value. Does that sound right? JOËL: Yes. And I think that's where there's often a lot of confusion because there's sort of plain enumerator, and then there's a lazy enumerator that Ruby provides. A plain enumerator is a lazy list in the sense that items don't get evaluated unless you try to reach for them. So, if you have an enumerator and you say, "Just give me the first five items," it will do that. And even if the collection was 200 items long, the next 195 don't get evaluated. So, that's very efficient there. Where you would get into trouble is that plain enumerators are not lazy when it comes to traversals. So, any method that would traverse the entire collection, so something like a map or a select, is not going to be lazy because it's going to traverse the entire collection, therefore forcing us to evaluate each of the items in there. Whereas something like enumerable lazy will not actually traverse the collection when you do your map or you're selecting. It will wait for you to say, "Give me the first item," or "Give me the first ten items," or something like that. But you don't always need lazy. You really only need lazy when you're doing a traversal method. STEPHANIE: Okay. Cool, cool, cool. That makes a lot of sense. JOËL: I think a sort of spinoff metaphor that I have there is this idea of a lazy list. Another concept that, in my mind, is very adjacent to lazy lists is the concept of streams. And streams I typically think of them in terms of, like, files or networking, things like that. But a thing that you can do let's say you're working on data that's in a very large file, so big that you can't fit it into memory, a common solution there is streaming it. So, you don't load the entire file into memory and then operate on it. Instead, little chunks of it are loaded into memory. You operate on them, and then you release that memory and load the next chunk. So, you sort of work through that file in chunks, but you'd only have, you know, 1 line or ten lines or however big your chunk is in memory at a time. An enumerator allows you to do that with things that are not files. So, this could be a situation where, let's say, you're reading a lot of data from the database. You just have too many rows. You can't load them all into memory at once. But you do want to traverse through them. You could chunk that using enumerator so that every, you know, it loads 100 rows at a time or 1,000 rows at a time, or something like that. And your enumerator allows you to treat that as though it's a single array, even though, in the background, it's being chunked into pieces so that you never have more than a thousand rows at a time in memory. So, it allows you to do some, like, really nice sort of memory performance things. STEPHANIE: When would you want to use this over kind of something like batching queries? JOËL: So, I think ActiveRecord findinbatches does something like this under the hood. STEPHANIE: Oh, cool. JOËL: I don't know if they use Ruby's enumerator or if they sort of build their own custom extension to it, but it's built on this idea. STEPHANIE: Okay, that's really neat. I have another mental model that I wanted to get your thoughts on. JOËL: Yeah! STEPHANIE: One of the ways that I looked up that you can construct an enumerator, an infinite enumerator like we were talking about a little bit earlier, was with the produce class method. And that actually got me thinking about a production line and this idea that, you know, you have this mechanism for, you know, producing some kind of material or, like, good or something like that. And it's just there and waiting and ready [laughs] for you to, like, kind of ask for it, like, what it needs to do. And you can do that, like, sometimes in batches, right? If you are asking for like, "Okay, I want a thousand units," and then the production line goes to work [laughs]. But yeah, that was another one of those things where I'm like, wow, they really, I think, came up with a cool method name that evoked, like, an image in my head. JOËL: That's the power of naming, right? And I think it's interesting you've mentioned twice how going through the method names on enumerator and finding different method names all of a sudden, like, turned on a light bulb in your mind. So, if you're naming things well, it can be incredibly useful for users of your library to pick up on what you're trying to do. So, I want to circle back to something that you mentioned earlier, the idea of elementary school quizzes where you have to, like, figure out the next item in the sequence. Because that, for me, is very similar to my mental model: the idea that an enumerator is a sequence generator. So, instead of thinking of it as, oh, it's like an array or it's some kind of collection, instead, think of it as a robot that I can just ask it, hey, give me a value, and it will give me a value. And then, it will, like, keep doing that as long as I keep asking it for it. And those values, you know, they could be totally random. You can build one of those. But you can also have it so that the values sort of come from a sequence. It's not like an array where you're like, oh, I'm going to, like, predefine an array of, I don't know, the Fibonacci sequence, and when someone asks me for the third value, I'll just go and read that third value from the array. Instead, it knows the algorithm, and it just says, "Oh, you want the next value in the Fibonacci sequence? Let me calculate it. Here it is. Oh, you want the next value? Here it is." And so, thinking from that perspective helped me really come to terms with the concept that values really do get calculated just in time. It's not really a collection. It's an object that can give you new values if you ask it. STEPHANIE: Yeah, okay. That is making a lot more sense kind of in conjunction with the lazy list model that you shared earlier, and even a little bit with the production line that I was kind of sharing where it's like, you know, in this case, kind of, it's, like, the potential for a value, right? JOËL: Right, exactly. And, you know, these are all mental models that converge on the same ideas because they're all just slightly different perspectives on what the same object does. And so, there is going to be some overlap, some converging between all of them. I have another fun one. Can I throw it at you? STEPHANIE: Please. JOËL: This one's a little bit different, and it's the idea that enumerators are a tool to bring your own iteration to a collection. So, imagine a situation where you're building your own, let's say, binary tree implementation. And there are multiple ways to traverse through a binary tree. In particular, let's say you're doing depth-first search. There are sort of three classic ways to traverse that are called pre-order, post-order, and in-order traversals. And it really is just sort of what order do you visit all the children in your tree? Now, the point of a collection, oftentimes, is you need a way to iterate through it. And a classic solution would be to include enumerable, the module. In order to do that, you have to define a way to iterate through your collection. You call that each. And then, enumerable just gives you all the other nice things for free. The question is, though, for something like a tree where there are multiple valid ways to traverse, which one do you pick to make it the each that gets sort of all the enumerable goodies, and then the others are just, like, random methods you've defined? Because if you define, let's say, pre-order traversal as each, now your detect and select and all those are going to work in pre-order, but the others are not going to get that. So, if you map over a tree, you're forced to map over in pre-order because that's what the library author chose. But what if you want to map over a tree in post-order or in-order? STEPHANIE: Yeah, well, I'm guessing that here's where enumerator comes in handy [laughs]. JOËL: Yes. The approach here is instead of designating sort of one of those traversals as the sort of blessed traversal that gets to have enumerable; you build three of these, one for each of these traversals. And then, what's really nice is that because enumerators are themselves enumerable, they have map and select and all of these things built in. Now you can do something like mytree dot preorder dot map or mytree dot postorder dot map. And you get all the goodies for free, but the users of your library get to basically choose which traversal they want to have. As a library author, you're not forced to pick ahead of time and sort of choose; this is the one I'm going to have. You sort of bring your own traversal by providing an enumerator, and then everything else just kind of falls into place. STEPHANIE: Bring Your Own Traversal (BYOT) [laughter]. I like it. Yeah, that's cool. I can see how that would be really handy. I have not yet encountered a situation where I needed to get that deep into how my iteration is traversed, but that's really interesting. And, I mean, I can start even imagining, like, having an each method defined in these different ways, and then all of that being able to be composed with some of the other...just other methods. And now you have, like, so many different ways to perhaps, like, help, you know, different performance use cases. JOËL: Yeah, it can be performance. I often tend to think of enumerator as a performance thing because of its sort of lazy properties because; it allows you to sort of stream or chunk data that you're working with. But in the case of this mental model of the Bring Your Own Traversal, it actually is more about flexibility and having sort of the beauty of Ruby without having to compromise on, oh, I have to pick a single way to traverse a collection. STEPHANIE: But I really appreciate kind of this discussion about enumerator because this was previously, like, I don't think I have really ever used the class itself to solve a problem, but now I feel a lot more equipped to do so with a couple of the different kind of perspectives. And I think what they helped me do is just prime myself. If I see a problem that might benefit from something being iterated in a lazy way, like, being like, oh, I remember this thing, this mental model. Now I can go kind of look at the documentation for how to use it. And yeah, like, I don't know how I would have stumbled across, like, reaching for it otherwise. JOËL: That's a really interesting thing to notice because we've been talking a lot about how mental models can be a tool for understanding. But once you build an understanding, even though it's somewhat fuzzy, they're also a great tool for sort of recall. So, not only are you thinking, okay, well, this mental model says enumerators are kind of like this, or they function in this way. On the flip side of it, you can say, "Well, lazy evaluation problems are often enumerator problems. Like, streaming or chunked data problems are often enumerator problems. Multiple traversals are enumerator problems." So, now, even though you don't, like, fully understand it in your mind, you've got that recall where you can enter it, where you can come across that problem, and immediately you're like, oh, I'm dealing with multiple traversals here. I don't remember exactly how, but somehow, in my mind, I've got a connection that says, "Enumerators are a solution for this. Let me dig into that." STEPHANIE: Yeah, especially as an alternative to where I would normally reach for something...a more kind of common enumerable method. Because I definitely know that feeling of like, oh, like, I wish it could just, like, do this a little bit differently, you know. And it turns out that, you know, something like that probably exists already. I just needed to know what it was [laughs]. JOËL: On that theme of I wish that I could have something that behaved just a little bit more...like, I'm doing something slightly weird, and I wish they would behave more, like, just plain Ruby does normally with my, like, collections I'm familiar with. I'm going to pitch a talk that I gave at RubyConf Mini called "Teaching Ruby to Count." Some of these mental models actually showed up there. But the whole idea is like, oh, if you're bringing in sort of more custom objects and all of that, how can you just tweak them a little bit so that they're just as joyful to use and interact with as arrays, and numbers, and ranges? And they just sort of fit into that beauty of Ruby that we get out of the box. STEPHANIE: Awesome. On that note, shall we wrap up? JOËL: Let's wrap up. STEPHANIE: Show notes for this episode can be found at bikeshed.fm. JOËL: This show has been produced and edited by Mandy Moore. STEPHANIE: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review in iTunes. It really helps other folks find the show. JOËL: If you have any feedback for this or any of our other episodes, you can reach us @_bikeshed, or you can reach me @joelquen on Twitter. STEPHANIE: Or reach both of us at hosts@bikeshed.fm via email. JOËL: Thanks so much for listening to The Bike Shed, and we'll see you next week. ALL: Byeeeeeeee!!!!!!! AD: Did you know thoughtbot has a referral program? If you introduce us to someone looking for a design or development partner, we will compensate you if they decide to work with us. More info on our website at: tbot.io/referral. Or you can email us at: referrals@thoughtbot.com with any questions.

english matrix gang vhs modeling api readers contrast binary obsidian cassettes software development mandy moore public works fibonacci blog posts schrodinger mental models ruminating gang of four performance optimization quenneville railsconf activerecord thoughtbot stephanie yeah chris toomey stephanie oh

427: RailsConf Recap and Conversing About Coupling

Play Episode Listen Later May 28, 2024 37:03

Joël and Stephanie talk RailsConf! (https://railsconf.org/). Joël shares how he performed as a D&D character, Glittersense the gnome, to make his Turbo features talk entertaining and interactive. Stephanie's talk focused on addressing test pain by connecting it to code coupling, offering practical insights and solutions. They agree on the importance of continuous improvement as speakers and developers and trying new approaches in talks and code design, and recommend Jared Norman's RailsConf talk on design patterns, too! That One Thing: Reduce Coupling for More Scalable and Sustainable Software (https://www.informit.com/articles/article.aspx?p=2222816) Connascence.io (https://connascence.io/) [Connascence as a vocabulary to discuss coupling](https://thoughtbot.com/blog/connascence-as-a-vocabulary-to-discuss-coupling](https://thoughtbot.com/blog/connascence-as-a-vocabulary-to-discuss-coupling) The value of specialized vocabulary (https://bikeshed.thoughtbot.com/356?t=0) Transcript: We're excited to announce a new workshop series for helping you get that startup idea you have out of your head and into the world. It's called Vision to Value. Over a series of 90-minute working sessions, you'll work with a thoughtbot product strategist and a handful of other founders to start testing your idea in the market and make a plan for building an MVP. Join for all seven of the weekly sessions or pick and choose the ones that address your biggest challenge right now. Learn more and sign up at tbot.io/visionvalue. JOËL: Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Joël Quenneville. STEPHANIE: And I'm Stephanie Minn. And together, we're here to share a bit of what we've learned along the way. JOËL: So, Stephanie, what's new in your world? STEPHANIE: So, I think I can speak for both of us and say what's new in our world is that you and I just came back from RailsConf in Detroit. JOËL: Yeah, we were there for, I guess, it's a three-day conference. Both of us were giving talks. STEPHANIE: Yeah. I don't think we've both spoken at a conference for at least a little over a year, so that was really fun kind of to catch up in person. And there was a whole crew of thoughtboters who were there. Yeah, I feel like we were hanging out, like, a lot [chuckles] all of last week, just seeing each other, talking about, you know, rehearsing our talks and spending time together on...there was, like, a hack day, and we were sitting at the table together. So, I feel like I'm totally caught up on everything that's new in your world, and that's it. That's the end of the show [laughs]. JOËL: On that note, shall we wrap up? STEPHANIE: [laughs] That would not be very fair to our listeners. [laughter] JOËL: Yeah. So, how was the conference speaking experience for you? STEPHANIE: Ooh, it was really great this year. I have not spoken at a RailsConf before, so this was actually, I think, a bigger stage than I had experienced before, and I had a great time. I met Ruby friends, new and old, and, yeah, I left feeling very gooeyed, and very energized, and just so grateful for the Rails community [laughs]. Yeah, I had a very lovely time, kind of being a little bit outside my normal life for a few days. And I think my favorite part about these things is just like, anywhere you go, you can kind of just have a shared interest with someone, and you can start a conversation with them. JOËL: That's really interesting. Do you find yourself just reaching out to strangers at conferences like this? Or do you tend to just hang out with the people that you know? STEPHANIE: Oh, I think a little bit of both. I like to get meals with people I know. But if I'm just hanging out in, like, the lobby or if I happen to get a seat for a talk and I'm sitting next to someone that I don't know, I find it quite easy to just be like, "Hi, like, I'm Stephanie. Are you excited for this talk?" Or, like, "What good talks have you seen recently?" There's an aspect of, like, the social butterfly that comes out of me when I'm at these things. Because I just don't get to have, like, easy access to, I don't know, people with, like, that shared interest or people who are willing to just have a conversation with you normally, I think. JOËL: Yeah, would you describe yourself more as an introvert or an extrovert? STEPHANIE: I am an extroverted introvert [laughter]. I feel like maybe that might be interpreted as a non-answer, but I think I lean more on the introvert side. But you know when you're with a group of people, and there's not, like, a very clear extrovert in that conversation, and then you're like, oh, I have to do the heavy [chuckles] lifting of the social lubrication [laughs] in this conversation, I can step into that role, reluctantly [laughs]. JOËL: Okay. I like the label that you used, the extrovert introvert, in that I enjoy social situations. I do well in social situations. But they also consume a lot of energy for me. I don't necessarily get sort of recharged by doing social events. So, people will be surprised when they find out that I tend to talk about myself as an introvert because, like, "Oh, but you're, like, you know, you're not awkward. You engage very well in different group situations." STEPHANIE: You have a podcast [laughs]. JOËL: And the truth is I enjoy those things, right? I really like social interaction, but it does, after a while, wear me out. STEPHANIE: Yeah, that makes sense. I did want to spend a little bit of time talking about the talk you gave at RailsConf this year: "Dungeons & Dragons & Rails." JOËL: I got to have a lot of fun with the theme. The actual content was introducing people to Turbo by building an interactive Dungeons & Dragons character sheet using vanilla Rails and a little bit of Turbo. So, we're not even writing any JavaScript. We're just using the Turbo helpers, a little bit of Action Cable to mimic something a little bit like...people who are in the know might be familiar with the site D&D Beyond, which is kind of the official D&D online character sheet website. Of course, it wasn't anywhere near as fancy because it's a 30-minute talk and showcasing different features, but that's what we were aiming for. STEPHANIE: Yeah, you know, you've talked a bit about giving talks on the show before, but I wanted to get into what made this one different because I think it could be fun for our listeners. [laughter] JOËL: The way I structured this talk so it has a theme. It's about Dungeons & Dragons, and we're building a character sheet. The way I wrote the talk was it's broken up into chapters. Each chapter is teaching a new feature in Turbo that I want to show off. In order to motivate learning each of these features...because I don't like to just say, "Oh, here's a thing that technology can do. Oh, here's a thing that technology can do." That's boring. You need a reason to learn that. So, I needed a reason to say, "We need to add this to a character sheet." So, every sort of chapter of the talk opens up with a little narrative portion. We're following this character, Glittersense, the gnome, and he's on adventures. And at different points in the adventures, he's going to do different types of roles or need different stats and things. And so, when we reach the point in the adventure where we need that, we sort of freeze frame and then say, "Okay, let's add that as a feature to the character sheet." And then, oh no, it turns out that this feature is a little bit more complicated. We're going to have to learn a new Turbo feature to do that. Who would have guessed? And then, we learn a new Turbo feature together. And then, we go back to the narrative portion. The adventures of Glittersense continue. And then, oh no, we're going to need to add another feature to the character sheet. And that's sort of how the talk is structured. STEPHANIE: Yeah. And you did a really cool thing with the narrative portions, which was you basically performed as Glittersense, the gnome, voice and posture, and a lot of really great acting from you [laughs], in my opinion. JOËL: That is something that came out pretty late in the talk preparation. So, I knew I wanted this kind of alternating story and code structure. Then, like, the weekend before RailsConf, I'm running through my slide deck, and I realized, you know what? What if instead of narrating Glittersense's adventures, what if I went first person for those sections? Glittersense tells his own story. And then, from there, it wasn't a big jump to say, you know what? This is D&D. If I'm going first person and narrating, I really should do a voice. And this is a conversation I had with a couple of people at the speaker dinner. And, of course, everyone's like, "You should 100% do the voice." And I was really not feeling confident in my ability to pull it off. So, for the next two nights, because I was speaking on the third day, the next two nights at the conference, in the evenings, I'm in the hotel room in front of the mirror just practicing my gnome voice to try to get something that got the persona of Glitterense, the gnome, across to the audience. STEPHANIE: How would you describe the persona? JOËL: Very extra. STEPHANIE: [laughs] JOËL: Very high energy. STEPHANIE: Yes. The name Glittersense is very extra, after all. JOËL: [laughs]. I punctuated a lot of the things that he says with just high-pitched laughter. He's also...so, the framing device for all of this is that you're in a tavern listening to him tell his adventures. I wanted a little bit of the sense that Glittersense is maybe embellishing a little bit. I think it may be too much to say he's full of himself, but he's definitely making himself to be the hero of the story, and maybe making himself to be slightly cooler than he really was. STEPHANIE: Yeah. I definitely got, like, a little bit of eccentricity, too, from the persona. And you know when you just, I don't know, meet an older person who has, like, a lot of life experience, and they want to tell you about it [laughter], but you do kind of maybe have a little bit of suspicion around how much they're exaggerating [laughs]. But it was really fun. Everyone I talked to afterwards, like, loved it. And I got to share the little nugget that, like, oh yeah, and Joël only, like, started doing the voice, like, decided that he was going to do it two days ago. And they were just all really, like, blown away because it seemed so well practiced, and it was really fun. JOËL: I got to do something really fun, also, with physical space because Glittersense narrates his portion, sort of the story portions, but then the code portions where we're talking about Turbo, I'm talking in my own voice. And so, when I'm talking about Turbo, I'm standing at the lectern. And when I'm Glittersense, I'm kind of off to the side on the stage and doing the voice. And so, there's this almost, like, two worlds that are inhabited: one by Joël, the speaker, and one by Glittersense, the gnome. And it got to the point where I don't say or do anything. I only move from the lectern to the, like, portion of the stage where Glittersense lives. And the audience starts chuckling and, like, nothing has happened yet, like, no jokes have been told. No voice has happened. No slides have changed. But the anticipation, people know what's coming. STEPHANIE: Yeah. And I think the best part, what I really found just really fun and, I don't know, every time it happened, I just really enjoyed it, when you transitioned out of Glittersense, the gnome, and back to Joël because you were so nonchalant about it. You kind of, like, straighten up rather than having your little kind of crouchy gnome posture, and then just walk across back to the podium. And then, in your normal voice, go back to just, you know, sharing very...not necessarily dry, but just, like, straight to the point. "And this is, like, how you, you know, create a frame in [laughs] Turbo," as if nothing happened [laughs] when even just, like, you know, 20 seconds ago, you were just enthusing about, like, slaying the bandit, chieftain [laughter] known as Glittersense. JOËL: Uh-huh. I think, especially when I open, so I get introduced. I'm off stage. I walk onto the stage, and I'm immediately Glittersense. And I'm telling a story, and the intro goes on for, like, quite a while. It's a big story chunk. And then, at some point, I just walk over to the lectern, drop the voice, hit next slide, and it's my title slide. I'm just like, "Okay, now welcome to Dungeons & Dragons on Rails. We're going to build a character sheet together." STEPHANIE: Yeah, that's exactly the moment I'm thinking of. JOËL: The walking in as Glittersense and just immediately going to the voice caught everyone by surprise. And then, the, like, oh, he keeps going for this. Is the whole talk going to be like this? And then, the, like, just when you think, oh, he's really going for it, the, like, dropping it and going to the podium and title slide. It wasn't intended to be a funny moment, but I think the contrast and the fact that I just switched over was one of the biggest laughs I got. STEPHANIE: Yeah, I mean, I think that attests to how good the delivery of it was because that contrast was very felt. So, props to you. JOËL: I love the idea of, you know, the thought that you put into building a talk and, like, the narrative structure and the pedagogy of the stuff. And, I think, in this particular case, this is almost like a narrative approach called in media res, where you start kind of in the middle. You open your book, or your movie, or whatever in the middle of the story. And then, you kind of come back to the beginning at some point later. So, it starts with some kind of action scene that grabs your attention. So, in this case, my title slide is 10, 15 slides into the talk. We get immediately started with Glittersense and his adventures. And then, once we're sort of all bought into this world, then we move to the title slide and talk about, okay, we're here to build a character sheet and all that stuff. And I think that it wouldn't have had the same impact if I'd, like, opened with that and then gone into Glittersense's adventures. And that's something that was not the case at the beginning. I really reworked the talk to make it in that order. And I think that the talk had a lot more impact for doing that. STEPHANIE: Yeah, definitely. I guess I also just wanted to point out that this is very different from all your other talks. And I think it's really cool that, you know, you are a veteran speaker, but you still find ways to do something new and try something that you've never done before, and yeah, find ways, new ways to, like, speak and engage people and teach. I don't know, do you have just any thoughts about why or how you got into a position to be like, "Oh, you know, I'm going to do something super different this time around" [laughs]? JOËL: So, every talk I give, I try to do something new, something different, to push myself as a speaker to get better. That might be in the writing of the talk; that might be in the delivery. More recently, I've been trying to do more with dynamic presence on stage. So, when I spoke at RubyConf San Diego, I was trying to not just stand at the lectern but to learn to be able to give my talk while also, you know, walking around the stage, looking at the audience, making pauses where it's necessary, not to just be so into the delivery of the talk by just standing at the podium and, like, going through my deck, which is a small thing but I think is an area I wanted to improve in. This time, I was playing around with some more narrative framing and ended up, yeah, like, pushing it to an extreme. And it works with the theme because inhabiting a character and role-playing is the core part of D&D. Not everybody plays a D&D character by doing a voice. You are a little bit extra if you do that. But it's not uncommon for people to do a voice. And so, it kind of fit perfectly with my theme. I just needed to get the self-confidence to do it. So, thank you to everyone at the speaker dinner that was like, "No, you totally got this. You should do this," because I was feeling very unsure. STEPHANIE: It really paid off, so... JOËL: I'd like to circle back to your talk, though. So, you gave, basically, the first talk of the conference. You were the first session after the keynote. A theme that came up multiple times in your talk was this idea of coupling and how it affects different parts of our code and, particularly the way that we structure tests or the way that we feel test pain. How did you, when you were prepping this talk, discover that theme and decide to lift it up? Was that something that you knew ahead of time you wanted to talk about, or did it just sort of emerge as part of the talk preparation process? STEPHANIE: That's a really great question, and I'm glad you picked up on that. So, my talk was called: "So, Writing Tests Feels Painful. What Now?" Originally, when I came up with this idea, it actually started with coupling. I realized that I wanted to give a talk about coupling because it's just something that I was struggling with or, like, had seen other people struggle with and really wanting kind of a discrete resource, wanting to provide that. But as I was just thinking about it, I was like, oh, like, there are so many different ways that this could go. On one hand, it was a very like important topic to me, but also maybe too big of a topic. And so, I actually, like, kind of put that on the back burner. And it wasn't until later when I connected it to another...it wasn't necessarily different at all, but just, like, an extension of this idea is, oh, like, people are struggling with coupling in tests or, like, it manifests in tests. And so, I thought maybe that could be the angle that I took on this topic that kind of gave me a little bit more focus. And I didn't even end up saying like, "Yeah, this talk was, like, born out of just, you know, wrestling with coupling or anything like that." So, it's cool, to me, that you picked up on it as a theme because it was...I had, you know, ended up not being super explicit about it, but it was certainly, like, a thing that was driving the content from my perspective. JOËL: Interesting. So, it started as a coupling talk and then got sort of focused through the lens of testing. STEPHANIE: Yeah. And I think there was a part of me that was like, you know, I don't know if I could just teach the concept of coupling, like, by itself without the framing of testing for people who this is, like, a new concept for them. I realized that maybe it would be more effective to be like, "Hey, like, have you experienced test pain? You know, have you had to mock out a billion objects or changed, you know, made one change and then had to fix, like, a million tests subsequently? Then this talk is for you." And then weave in the idea of coupling in it to kind of start to help people feel familiar with it or just, like, identify it without as much, like, jargon as kind of I've seen when I've tried to figure out, like, how to manage it. JOËL: It's interesting because I think it gives you a, like, concrete, valuable thing to optimize for as opposed to, like, hey, let's lower coupling because then you're writing, you know, quote, unquote, "better code." And you get to feel better about yourself as a programmer because you're doing things the, quote, unquote, "right way." That's very kind of hand-wavy, and I think sometimes leads people down a bad path where they're optimizing things that they shouldn't be. But the tests give you this very concrete way to say, "Hey, we're not just trying to reach the, like, low score record for the app in terms of coupling. We're trying to reduce test pain. Tests are painful. And that pain is telling us something. It's telling us that we've crossed some sort of threshold for coupling. Let's find ways to reduce it, not so that we can feel good about ourselves, but so that our tests are actually manageable." STEPHANIE: Yeah, I am really glad you picked up on that, too, because I feel the exact same way when someone just tells me to decouple something or, like, makes a note that, like, oh, this feels really coupled. I don't know what that means necessarily. And it's not very convincing to just be like, "Oh, you should write loosely coupled code [laughs]," at least for me. What you said just now, it's like, it's not to feel good about ourselves, you know, to write code that way, but, actually, to just feel good about our code, period [laughs]. And, yeah, finding that validation through just, like, actually working with code that is easier to change that is the goal, not necessarily to, yeah, kind of pursue some totally subjective, like, metric. JOËL: So, one of the kinds of coupling that you called out, I think, was where you hardcode a class name of some other class in your object. And that feels, like, really sort of innocuous. Like, of course, my objects can talk to other objects. And maybe I want to, like, refer to a class somewhere. Why is that such a like tricky piece of coupling to work with? STEPHANIE: It's not necessarily intentional sometimes. Like, you just do it because you're like, well, I need access to this class somewhere, and I happen to already be in this file. So, why not just hard-code it here? I do think it's a little tricky because the file that you're writing might be, like, very far down in, like, your code flow or, like, your code path, like, very far from, like, a controller or any kind of entry point into your system, at least based on what I've seen in a lot of modern Rails apps. And so, I think that coupling gets really, really obscured. I have found that, like, if I have to kind of write a more, like, a higher level test, like, maybe a request spec or something, there are times when I'm, like, having to deal with a lot of classes just to set stuff up in a test like that that I didn't think I would have to [chuckles] when I first went about trying to just be like, oh, like, let's just figure out how to get a 200 response [laughs] from this request. So, you're really burying perhaps the things that are needed to set up, like, that full path of execution. And sometimes, it only comes out when you're writing a test for it. JOËL: And you mentioned briefly, in passing, the idea that oftentimes this sort of coupling manifests as a lot of extra test setup because your object that you're trying to test now also needs all these other things that are related in order to be tested. But sometimes even when you hard code a class, though, you can't even just say, "Oh, I want this particular user or something returned." So, you have to then do something like allow this class to receive class method and return, and now you're stubbing. And I don't know how you feel about stubs in RSpec. I always treat them a little bit like a code smell in the like classic sense of it's not necessarily bad, but maybe pause, take a look, and ask yourself, "Why is that there, and should I do things differently?" STEPHANIE: Yeah. I ended up having, like, a lot of examples of stubbing in my example because the code had just been set up where that was the only way that you could access those collaborators, essentially, to, like, make an assertion on them, or have them do something different because you actually needed to go into a different path, right? And I was like, yeah, this should feel weird. You should feel a little bad [laughs] or at least, you know, kind of just pay attention to that feeling, even if you can't really do anything about it in that particular instance. But on the flip side, you know, it's like, yes, it feels a bit strange, you know, but it's not all bad, right? Like, you're kind of learning like, oh, hey, like, I am coupled to this hard-coded class because I am needing to stub, like, a class method that returns it, or that constructs it. And at least you've exposed that, you know, for yourself. One thing that I was running into a lot in my example, too, was that those things, like, weren't obvious when you were just reading maybe, like, the public methods and trying to figure out what was happening in them because they were wrapped in private methods. I was a little bit conflicted about this because there were times when it was already just a single method call, but then it was just kind of wrapped in a private method that actually hid [laughs] the things, like all the dependencies that were passed as arguments. And I found that to be, sure, it looks kind of cleaner. But then all you need to do is scroll down [laughs], and then you're like, oh, actually, there's all these other things involved, but it was kind of hidden away for me. And I found that, actually, like, at least when I actually needed to change things, less helpful than I imagine what the, you know, code author intended. Do you have any thoughts about hiding details like that? JOËL: I'm kind of a big fan. STEPHANIE: Hmmm. JOËL: The general idea, I think, is called the single level of abstraction principle. Whatever sort of public method that you're calling is often implemented in terms of...let's say it does a few different things. It's implemented in terms of, like, these sort of high-level concepts. So, whoever is reading the public method doesn't need to like care about the details of how each step is implemented. So, maybe you're fetching something from an API, and then you're making a database call, and then you're doing some transformation and creating some new objects from it. Having all of the, like, HTTP calls and the ActiveRecord stuff and the, like, transformation all in the public method, yes, there's a lot of complexity happening there, and it makes that obvious. But it also makes it really hard to get a sense of what is happening. So, I like to say, "Hey, there are four steps. Let's wrap them all each in a private method then you can call all of those in the public method." The public method now sort of reads like a very simple sort of script. First, fetch data from the HTTP API, then fetch some data from the database, then apply this transformation, then create this object. And if I'm mostly caring about what this object does and not the how let's say I'm building some other objects that interact with this, that is the information I want to know. Where I care about the actual implementation of, oh, well, exactly how is the ActiveRecord stuff done when I'm doing internal changes to the object, that's when I care about those private methods. I think where it gets tricky, and I think that's the point that you were bringing up, is that if you write code in that way, it has to change the heuristics of how you read code to detect complexity. Because, oftentimes, I think a very classic heuristic for code complexity is just line length. If you have a 50-line method, probably there's a lot of complexity there. Maybe there's a lot of coupling. If it's a four-line method that is written at a high level of abstraction that just calls out to private methods, you scan over. You're like, oh, nice and clean. Nothing to see here. Move on. And so, that heuristic doesn't really hold up in a codebase where you're applying this single level of abstraction. Do you think that lines up with your experience? STEPHANIE: Hmm. As I was listening to you, I was like, yeah, like, that makes total sense to me. But then I also clearly disagreed a little bit [laughs] in my initial...kind of what I was saying initially. And I think it's because that single layer of abstraction was not very well defined. JOËL: Hmm. That's fair. STEPHANIE: Yeah. Where, in fact, it was actually misleading. Like, it wanted to be at that level of abstraction, but it really wasn't. Like, it was operating on things at, like, a lower level and wasn't designed with that kind of readability in mind. So, it was more, like, it was just hiding stuff a little bit, at least for me. And, I think, it certainly would have taken, like, more work to figure out what that code, like, really was meant to convey. It might have taken some refactoring to coalesce at that single level. And that was essentially kind of what I was showing in my talk as, like, how to get to saying, like, "Hey, we actually are operating in the lower level, but I don't think we need to." There was some amount of, like, looking at all of the how to figure out, like, oh, maybe these things we don't even need to expose in this class. And we kind of got to a place where those details weren't, like, needed in that class at all. So, it's one of those things where it's harder than it sounds [laughs]. JOËL: It's definitely an art. STEPHANIE: Yeah. JOËL: And I think what you're saying about some of the coupling being, like, scattered throughout the class, it's something that I see a lot with situations where you're coupled, not so much to, like, a single class, but to something side effectful. So, you're building some kind of integration with a third-party API, and you're going to have to make a lot of HTTP calls. And each of those might be individually simple, and they're all sort of maybe in different private methods or whatever, or they're interspersed among a larger chunk of logic. And that makes your tests really complicated. But there's no, like, one place you can point at and be like, ooh, that's the one place where there's a lot of complexity. What's happening here, though, is that your business object that's doing stuff is coupled to the network, and that coupling is going to force you to do some stubbing. It's going to force you to deal with a bunch of side effects that are non-deterministic in your code. And you used the word coalesce earlier that I really liked because I think that's often a situation where you do have to stand back and say, "Look, there's a lot of HTTP going on here. What if I coalesced it all into an object? Now I have two objects: one that's responsible for business logic, and one that's responsible for just the HTTP calls." And, all of a sudden, the tests just totally simplify. And we've removed some coupling, but that's not something that you would have seen just from reading the code. Because, as you were saying, it's sort of scattered in little bits and pieces throughout your file that don't necessarily catch your eye. STEPHANIE: Yeah. Which brings me to a blog post that I had found a lot of inspiration from in the talk that I'll link. It's called "That One Thing: Reduce Coupling for More Scalable and Sustainable Software." But it's actually about tests [laughs], even though it doesn't make an appearance in the title of the blog post at all. But this is where I kind of got the idea of necessary versus unnecessary coupling in test. Because I had never thought about how, yeah, like, when you write a test, you are very correctly coupling yourself to at least the method and class under test [laughs], if not also the arguments, right? Or anything else needed to construct what you're testing. And literally having that listed out for me in this blog post I think it's a...they use some examples in Java. And so, there's, like, a little bit more [laughs] setup involved. But I think they're like, yeah, these are six things that, like, it's mostly fine if you're coupled to these because that's kind of what needs to happen in a test. But, like, even having something to compare a test I wrote to just, like, okay, these are the things I know I need. And then, you can start to see when you've diverged from that list, when you are finding yourself coupled to some internals of your class. I really...that was actually, like, really helpful for me because, as we talked about earlier, like, it can be kind of communicated so abstractly. But here is, like, a very clear heuristic for when you should at least, like, start to pay attention or be like, oh, this is something that was needed to get the test to run but is now starting to feel a little unnecessary because it's not on this list. JOËL: That list reminds me, or the idea of a list of things to check out for when thinking about coupling, reminds me of the concept of connascence, which is a fancy word for almost a, like, categorization of different types of coupling because coupling comes in different flavors, some of which are tighter forms of coupling than others. And so, having that vocabulary has been really helpful for me when I'm looking at PRs and code review, or even when I'm refactoring my own code. Kind of like that list that you mentioned that you have, now I have some heuristics to look at that and say, "Oh, can I go from a connascence of position to a connascence of naming, and does that help me?" STEPHANIE: Yeah, I like that you mentioned the positional connascence because I also came across a really great metaphor for kind of things that need to change together, like, when that makes sense. And it was basically the idea of a dishwasher and a laundry machine [laughs]. I wish I could recall, like, what book this was from. But it was basically like, oh yeah, like, in theory, you're washing two things. So, maybe they are similar, but then you're like, no, actually, you want these to be a little bit separate because, you know, you don't want to wash your dishes and your clothes in the same machine. I don't know, maybe that exists [laughs], but I don't think it would do a very good job for either goal. And I think that was really helpful, for me, in imagining, like, the difference between kind of coupling and cohesion, like things that...even just imagining, like, kind of where I'm doing those things in the house, right? It's like, okay, that lives in a separate room. And, like, the kitchen is for the dishes, and that could be like, you know, a module if you will. And, like, laundry happens in the laundry room, and how to kind of just separate those things, even though they also do share some qualities, too. Like, they're both appliances, right? And so, that's the way that they are similar, but they're not the same. JOËL: You just mentioned the sort of keyword cohesion. And for our listeners who are not familiar with that term, it refers to an object sort of having one thing that it does well. Like, everything in that class sort of works towards the same goal, kind of similar to the idea of the single responsibility principle. So, in my earlier example, where we're sort of interspersing some business logic, a lot of HTTP requests, and pulling out an object that's focused on HTTP, like everything is based around that, now that object has higher cohesion because it's all doing one thing. So, if you read classic object-oriented literature, the recommendations that you'll typically see are that objects should have high cohesion and low coupling. STEPHANIE: Yeah. Think of a dishwasher and a washing machine next time [laughs] you come across something like that. Because I feel like those are really great, like, real-life examples of that separation. JOËL: Did you go to Jared Norman's talk on the third day: "Undervalued: The Most Useful Design Pattern"? STEPHANIE: No, I didn't. Can you tell me about it? JOËL: It felt like he was addressing a lot of the same themes as you were but from more of a code perspective than a test perspective. Talking a lot about, again, forms of coupling, dependencies, and then, specifically, one of the tools that he focused on to reduce the coupling that we see is value objects and factory methods to construct those. So, for any of our listeners who, when the talks come out, watch Stephanie's talk and are like, "Wow, I would love to learn more about this," a great follow-up, Jared Norman's talk: "Undervalued: The Most Useful Design Pattern." STEPHANIE: Yeah, that's neat because I can see that being a solution to the hard code did class names that we were talking about earlier. And I like how that is kind of, like, a progressive lesson in coupling a little bit. I'm really glad you shared that talk with me because now I'm excited to watch it when it comes out. And in general, I just love learning new vocabulary or finding new ways to speak about this topic with clarity. So, if any of our listeners have just additional mental models for coupling [laughs] different metaphors, different household appliances [laughs], or something like that, I would love to know. JOËL: You would like that, given that our first episode together was about "The Value Of Specialized Vocabulary." STEPHANIE: Yeah, it's clearly undervalued. JOËL: Haha, I see what you did there. STEPHANIE: Thank you. Thank you very much [laughs]. JOËL: On that terrible/wonderful pun, shall we wrap up? STEPHANIE: Let's wrap up. Show notes for this episode can be found at bikeshed.fm. JOËL: This show has been produced and edited by Mandy Moore. STEPHANIE: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review in iTunes. It really helps other folks find the show. JOËL: If you have any feedback for this or any of our other episodes, you can reach us @_bikeshed, or you can reach me @joelquen on Twitter. STEPHANIE: Or reach both of us at hosts@bikeshed.fm via email. JOËL: Thanks so much for listening to The Bike Shed, and we'll see you next week. ALL: Byeeeeeeee!!!!! AD: Did you know thoughtbot has a referral program? If you introduce us to someone looking for a design or development partner, we will compensate you if they decide to work with us. More info on our website at: tbot.io/referral. Or you can email us at: referrals@thoughtbot.com with any questions.

vision detroit mvp dragons tests dungeons api turbo java javascript rails prs mandy moore coupling conversing d beyond quenneville railsconf activerecord rspec stephanie you stephanie yeah stephanie how stephanie it stephanie thank stephanie no stephanie oh action cable

425: Modeling Associations in Rails

Play Episode Listen Later May 7, 2024 29:39

Stephanie shares an intriguing discovery about the origins of design patterns in software, tracing them back to architect Christopher Alexander's ideas in architecture. Joël is an official member of the Boston bike share system, and he loves it. He even got a notification on the app this week: "Congratulations. You have now visited 10% of all docking stations in the Boston metro area." #AchievementUnlocked, Joël! Joël and Stephanie transition into a broader discussion on data modeling within software systems, particularly how entities like companies, employees, and devices interconnect within a database. They debate the semantics of database relationships and the practical implications of various database design decisions, providing insights into the complexities of backend development. Christopher Alexander and Design Patterns (https://www.designsystems.com/christopher-alexander-the-father-of-pattern-language/) Rails guide to choosing between belongsto and hasone (https://edgeguides.rubyonrails.org/association_basics.html#choosing-between-belongs-to-and-has-one) Making impossible states impossible (https://www.youtube.com/watch?v=IcgmSRJHu_8) Transcript: We're excited to announce a new workshop series for helping you get that startup idea you have out of your head and into the world. It's called Vision to Value. Over a series of 90-minute working sessions, you'll work with a thoughtbot product strategist and a handful of other founders to start testing your idea in the market and make a plan for building an MVP. Join for all seven of the weekly sessions, or pick and choose the ones that address your biggest challenge right now. Learn more and sign up at tbot.io/visionvalue. JOËL: Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Joël Quenneville. STEPHANIE: And I'm Stephanie Minn. And together, we're here to share a bit of what we've learned along the way. JOËL: So, Stephanie, what's new in your world? STEPHANIE: So, I learned a very interesting tidbit. I don't know if it's historical; I don't know if I would label it that. But, I recently learned about where the idea of design patterns in software came from. Are you familiar with that at all? JOËL: I read an article about that a while back, and I forget exactly, but there is, like, a design patterns movement, I think, that predates the software world. STEPHANIE: Yeah, exactly. So, as far as I understand it, there is an architect named Christopher Alexander, and he's kind of the one who proposed this idea of a pattern language. And he developed these ideas from the lens of architecture and building spaces. And he wrote a book called A Pattern Language that compiles, like, all these time-tested solutions to how to create spaces that meet people's needs, essentially. And I just thought that was really neat that software design adopted that philosophy, kind of taking a lot of these interdisciplinary ideas and bringing them into something technical. But also, what I was really compelled by was that the point of these patterns is to make these spaces comfortable and enjoyable for humans. And I have that same feeling evoked when I'm in a codebase that's really well designed, and I am just, like, totally comfortable in it, and I can kind of understand what's going on and know how to navigate it. That's a very visceral feeling, I think. JOËL: I love the kind of human-centric approach that you're using and the language that you're using, right? A place that is comfortable for humans. We want that for our homes. It's kind of nice in our codebases, too. STEPHANIE: Yeah. I have really enjoyed this framing because instead of just saying like, "Oh, it's quote, unquote, "best practice" to follow these design patterns," it kind of gives me more of a reason. It's more of a compelling reason to me to say like, "Following these design patterns makes the codebase, like, easier to navigate, or easier to change, or easier to work with." And that I can get kind of on board with rather than just saying, "This way is, like, the better way, or the superior way, or the way to do things." JOËL: At the end of the day, design patterns are a means to an end. They're not an end in of itself. And I think that's where it's very easy to get into trouble is where you're just sort of, I don't know, trying to rack up engineering points, I guess, for using a lot of design patterns, and they're not necessarily in service to some broader goal. STEPHANIE: Yeah, yeah, exactly. I like the way you put that. When you said that, for some reason, I was thinking about catching Pokémon or something like filling your Pokédex [laughs] with all the different design patterns. And it's not just, you know, like you said, to check off those boxes, but for something that is maybe a little more meaningful than that. JOËL: You're just trying to, like, hit the completionist achievement on the design patterns. STEPHANIE: Yeah, if someone ever reaches that, you know, gets that achievement trophy, let me know [laughs]. JOËL: Can I get a badge on GitHub for having PRs that use every single Gang of Four pattern? STEPHANIE: Anyway, Joël, what's new in your world? JOËL: So, on the topic of completing things and getting badges for them, I am a part of the Boston bike share...project makes it sound like it's a, I don't know, an exclusive club. It's Boston's bike share system. I have a subscription with them, and I love it. It's so practical. You can go everywhere. You don't have to worry about, like, a bike getting stolen or something because, like, you drop it off at a docking station, and then it's not your responsibility anymore. Yeah, it's very convenient. I love it. I got a notification on the app this week that said, "Congratulations. You have now visited 10% of all docking stations in the Boston metro area." STEPHANIE: Whoa, that's actually a pretty cool accomplishment. JOËL: I didn't even know they tracked that, and it's kind of cool. And the achievement shows me, like, here are all the different stations you've visited. STEPHANIE: You know what I think would be really fun? Is kind of the equivalent of a Spotify Wrapped, but for your biking in a year kind of around the city. JOËL: [laughs] STEPHANIE: That would be really neat, I think, just to be like, oh yeah, like, I took this bike trip here. Like, I docked at this station to go meet up with a friend in this neighborhood. Yeah, I think that would be really fun [laughs]. JOËL: You definitely see some patterns come up, right? You're like, oh yeah, well, you know, this is my commute into work every day. Or this is that one friend where, you know, every Tuesday night, we go and do this thing. STEPHANIE: Yeah, it's almost like a travelogue by bike. JOËL: Yeah. I'll bet there's a lot of really interesting information that could surface from that. It might be a little bit disturbing to find out that a company has that data on you because you can, like, pick up so much. STEPHANIE: That's -- JOËL: But it's also kind of fun to look at it. And you mentioned Spotify Wrapped, right? STEPHANIE: Right. JOËL: I love Spotify Wrapped. I have so much fun looking at it every year. STEPHANIE: Yeah. It's always kind of funny, you know, when products kind of track that kind of stuff because it's like, oh, like, it feels like you're really seen [laughs] in terms of what insights it's able to come up with. But yeah, I do think it's cool that you have this little badge. I would be curious to know if there's anyone who's, you know, managed to hit a hundred percent of all the docking stations. They must be a Boston bike messenger or something [laughs]. JOËL: Now that I know that they track it, maybe I should go for completion. STEPHANIE: That would be a very cool flex, in my opinion. JOËL: [laughs] And, you know, of course, they're always expanding the network, which is a good thing. I'll bet it's the kind of thing where you get, like, 99%, and then it's just really hard to, like, keep up. STEPHANIE: Yeah, nice. JOËL: But I guess it's very appropriate, right? For a podcast titled The Bike Shed to be enthusiastic about a bike share program. STEPHANIE: That's true. So, for today's topic, I wanted to pick your brain a little bit on a data modeling question that I posed to some other developers at thoughtbot, specifically when it comes to associations and associations through other associations [laughs]. So, I'm just going to kind of try to share in words what this data model looks like and kind of see what you think about it. So, if you had a company that has many employees and then the employee can also have many devices and you wanted to be able to associate that device with the company, so some kind of method like device dot company, how do you think you would go about making that association happen so that convenience method is available to you in the code? JOËL: As a convenience for not doing device dot employee dot company. STEPHANIE: Yeah, exactly. JOËL: I think a classic is, at least the other way, is that it has many through. I forget if you can do a belongs to through or not. You could also write, effectively, a delegation method on the device to effectively do dot employee dot company. STEPHANIE: Yeah. So, I had that same inkling as you as well, where at first I tried to do a belongs to through, but it turns out that belongs to does not support the through option. And then, I kind of went down the next path of thinking about if I could do a has one, a device has one company through employee, right? But the more I thought about it, the kind of stranger it felt to me in terms of the semantics of saying that a device has a company as opposed to a company having a device. It made more sense in plain English to think about it in terms of a device belonging to a company. JOËL: That's interesting, right? Because those are ways of describing relationships in sort of ActiveRecord's language. And in sort of a richer situation, you might have all sorts of different adjectives to describe relationships. Instead of just belongs to has many, you have things like an employee owns a device, an employee works for a company, you know because an employee doesn't literally belong to a company in the literal sense. That's kind of messed up. So, I think what ActiveRecord's language is trying to use is less trying to, like, hit maybe, like, the English domain language of how these things relate to, and it's more about where the foreign keys are in the database. STEPHANIE: Yeah. I like that point where even though, you know, these are the things that are available to us, that doesn't actually necessarily, you know, capture what we want it to mean. And I had gone to see what Rails' recommendation was, not necessarily for the situation I shared. But they have a section for choosing between which model should have the belongs to, as opposed to, like, it has one association on it. And it says, like you mentioned, you know, the distinction is where you place the foreign key, but you should kind of think about the actual meaning of the data. And, you know, we've talked a lot about, I think, domain modeling [chuckles] on the show. But their kind of documentation says that...the has something relationship says that one of something is yours, that it can, like, point back to you. And in the example I shared, it still felt to me like, you know, really, the device wanted to point to the company that it is owned by. And if we think about it in real-world terms, too, if that device, like, is company property, for example, then that's a way that that does make sense. But the couple of paths forward that I saw in front of me were to rework that association, maybe add a new column onto the device, and go down that path of codifying it at the database level. Or kind of maybe something as, like, an in-between step is delegating the method to the employee. And that's what I ended up doing because I wasn't quite ready to do that data migration. JOËL: Adding more columns is interesting because then you can run into sort of data consistency issues. Let's say on the device you have a company ID to see who the device belongs to. Now, there are sort of two different independent paths. You can ask, "Which company does this device belong to?" You can either check the company ID and then look it up in the company table. Or you can join on the employee and join the employee back under company. And those might give you different answers and that can be a problem with data consistency if those two need to stay in sync. STEPHANIE: Yeah, that is a good point. JOËL: There could be scenarios where those two are allowed to diverge, right? You can imagine a scenario where maybe a company owns the device, but an employee of a potentially different company is using the device. And so, now it's okay to have sort of two different chains because the path through the employee is about what company is using our devices versus which company actually owns them. And those are, like, two different kinds of relationships. But if you're trying to get the same thing through two different paths of joining, then that can set you up for some data inconsistency issues. STEPHANIE: Wow. I really liked what you said there because I don't think enough thought goes into the emergent relationships between models after they've been introduced to a codebase. At least in my experience, I've seen a lot of thought go up front into how we might want to model an ActiveRecord, but then less thought into seeing what patterns kind of show up over time as we introduce more functionality to these models, and kind of understand how they should exist in our codebase. Is that something that you find yourself kind of noticing? Like, how do you kind of pick up on the cue that maybe there's some more thought that needs to happen when it comes to existing database tables? JOËL: I think it's something that definitely is a bit of a red flag, for me, is when there are multiple paths to connect to sort of establish a relationship. So, if I were to draw out some sort of, like, diagram of the models, boxes, and arrows or something like that, and then I could sort of overlay different paths through that diagram to connect two models and realize that those things need to be in sync, I think that's when I started thinking, ooh, that's a potential danger. STEPHANIE: Yeah, that's a really great point because, you know, the example I shared was actually a kind of contrived one based on what I was seeing in a client codebase, not, you know, I'm not actually working with devices, companies, and employees [laughs]. But it was encoded as, essentially, a device having one company. And I ended up drawing it out because I just couldn't wrap my head around that idea. And I had, essentially, an arrow from device pointing to company when I could also see that you could go take the path of going through employee [laughs]. And I was just curious if that was intentional or was it just kind of a convenient way to have that direct method available? I don't currently have enough context to determine but would be something I want to pay attention to. Like you said, it does feel like, if not a red flag, at least an orange one. JOËL: And there's a whole kind of science to some of this called database normalization, where they're sort of, like, they all have rather arcane names. They're the first normal form, the second normal form, the third normal form, you know, it goes on. If you look at the definition, they're all also a little bit arcane, like every element in a relation must depend solely upon the primary key. And you're just like, well, what does that mean? And how do I know if my table is compliant with that? So, I think it's worth, if you're Googling for some of these, find an article that sort of explains these a little bit more in layman's terms, if you will. But the general idea is that there are sort of stricter and stricter levels of the amount of sort of duplicate sources of truth you can have. In a sense, it's almost like DRY but for databases, and for your database schema in particular. Because when you have multiple sources of truth, like who does this device belong to, and now you get two different answers, or three different answers, now you've got a data corruption issue. Unlike bugs in code where it's, you know, it can be a problem because the site is down, or users have incorrect behavior, but then you can fix it later, and then go to production, and disruption to your clients is the worst that happened, this sort of problem in data is sometimes unrecoverable. Like, it's just, hey, -- STEPHANIE: Whoa, that sounds scary. JOËL: Yeah, no, data problems scare me in a way that code problems don't. STEPHANIE: Whoa. Could you...I think I interrupted you. But where were you going to go about once you have corrupted data? Like, it's unrecoverable. What happens then? JOËL: Because, like, if I look at the database, do I know who the real owner of this...if I want to fix it, let's say I fix my schema, but now I've got all this data where I've got devices that have two different owners, and I don't know which one is the real one. And maybe the answer is, I just sort of pick one and say, "Oh, the one that was through this association is sort of the canonical one, and we can just sort of ignore the other one." Do I have confidence in that decision? Well, maybe depending on some of the other context maybe, I'm lucky that I can have that. The doomsday scenario is that it's a little bit of one, a little bit of the other because there were different code paths that would write to one way or another. And there's no real way of knowing. If there's not too many devices, maybe I do an audit. Maybe I have to, like, follow up with all of my customers and say, "Hey, can you tell me which ones are really your devices?" That's not going to scale. Like, real worst case scenario, you almost have to do, like, a bit of a bankruptcy, where you say, "Hey, all the data prior to this date there's a bit of a question mark on it. We're not a hundred percent sure about it." And that does not feel great. So, now you're talking about mitigation strategies. STEPHANIE: Oof. Wow. Yeah, you did make it sound [laughs] very scary. I think I've kind of been on the periphery of a situation like this before, where it's not just that we couldn't trust the code. It's that we couldn't trust the data in the database either to tell us how things work, you know, for our users and should work from a product perspective. And I was on a previous client project where they had to, yeah, like, hire a bunch of people to go through that data and kind of make those determinations, like you said, to kind of figure out it out for, you know, all of these customers to determine the source of truth there. And it did not sound like an easy feat at all, right? That's so much time and investment that you have to put into that once you get to that point. JOËL: And there's a little bit of, like, different problems at different layers. You know, at the database layer, generally, you want all of that data to be really in a sort of single source of truth. Sometimes that makes it annoying to query because you've got to do all these joins. And so, there are various denormalization strategies that you can use to make that. Or sometimes it's a risk you're going to take. You're going to say, "Look, this table is not going to be totally normalized. There's going to be some amount of duplication, and we're comfortable with the risk if that comes up." Sometimes you also build layers of abstractions on top, so you might have your data sort of at rest in database tables fully normalized and separated out, but it's really clunky to query. So, you build out a database view on top of that that returns data in sort of denormalized fashion. But that's okay because you can always get your correct answer by querying the underlying tables. STEPHANIE: Wow. Okay. I have a lot of thoughts about this because I feel like database normalization, and I guess denormalization now, are skills that I am certainly not an expert at. And so, when it comes to, like, your average developer, how much do you think that people need to be thinking about this? Or what strategies do you have for, you know, a typical Rails dev in terms of how deep they should go [laughs]? JOËL: So, the classic advice is you probably want to go to, like, third to fourth normal form, usually three. There's also like 3.5 for some reason. That's also, I think, sometimes called BNF. Anyway, sort of levels of how much you normalize. Some of these things are, like, really, really basic things that Rails just builds into its defaults with that convention over configuration, so things like every table should have a primary key. And that primary key should be something that's fixed and unique. So, don't use something like combination of first name, last name as your primary key because there could be multiple people with the same name. Also, people change their names, and that's not great. But it's great that people can change their names. It's not great to rely on that as a primary key. There are things like look for repeating columns. If you've got columns in your schema with a number prefix at the end, that's probably a sign that you want to extract a table. So, I don't know, you have a movie, and you want to list the actors for a movie. If your movie table has actor 1, actor 2, actor 3, actor 4, actor 5, you know, like, all the way up to actor 20, and you're just like, "Yeah, no, we fill, like, actor 1 through N, and if there's any space left over, we just put nulls in those columns," that's a pretty big sign that, hey, why don't you instead have a, like, actor's table, and then make a, like, has many association? So, a lot of the, like, really basic normalization things, I think, are either built into Rails or built into sort of best practices around Rails. I think something that's really useful for developers to get as a sense beyond learning the actual different normal forms is think about it like DRY for your schema. Be wary of sort of multiple sources of truth for your data, and that will get you most of the way there. When you're designing sort of models and tables, oftentimes, we think of DRY more in terms of code. Do you ever think about that a little bit in terms of your tables as well? STEPHANIE: Yeah, I would say so. I think a lot of the time rather than references to another table just starting to grow on a certain model, I would usually lean towards introducing a join table there, both because it kind of encapsulates this idea that there is a connection, and it makes the space for that idea to grow if it needs to in the future. I don't know if I have really been disciplined in thinking about like, oh, you know, there should really...every time I kind of am designing my database tables, thinking about, like, there should only be one source of truth. But I think that's a really good rule of thumb to follow. And in fact, I can actually think of an example right now where we are a little bit tempted to break that rule. And you're making me reconsider [laughter] if there's another way of doing so. One thing that I have been kind of appreciative of lately is on my current client project; there's just, like, a lot of data. It's a very data-intensive and sensitive application. And so, when we introduce migrations, those PRs get tagged for review by someone over from the DevOps side, just to kind of provide some guidance around, you know, making sure that we're setting up our models to scale well. One of the things that he's been asking me on my couple of code changes I introduced was, like, when I introduced an index, like, it happened to be, like, a composite index with a couple of different columns, and the particular order of those columns mattered. And he kind of prompted me to, like, share what my use cases for this index were, just to make sure that, like, some thought went into it, right? Like, it's not so much that the way that I had done it was wrong, but just that I had, like, thought about it. And I like that as a way of kind of thinking about things at the abstraction that I need to to do my dev work day to day and then kind of mapping that to, like you were saying, those best practices around keeping things kind of performant at the database level. JOËL: I think there's a bit of a parallel world that people could really benefit from dipping a toe in, and that's sort of the typed programming world, this idea of making impossible states impossible or making illegal states unrepresentable. That in the sort of now it's not schemas of database tables or schemas of types that you're creating but trying to prevent data coming into a state where someone could plausibly construct an instance of your object or your type that would be nonsensical in the context of your app, kind of trying to lock that down. And I think a lot of the ways that people in those communities think about...in a sense, it's kind of like database normalization for developers. So, if you're not wanting to, like, dip your toe in more of the sort of database-centric world and, like, read an article from a DBA, it might be worthwhile to look at some of those worlds as well. And I think a great starting point for that is a talk by Richard Feldman called Making Impossible States Impossible. It's for the Elm language. And there are equivalents, I think, in many others as well. STEPHANIE: That's really cool that you are making that connection. I know we've kind of briefly talked about workshops in the past on the show. But if there were a workshop for, you know, that kind of database normalization for developers, I would be the first to sign up [laughs]. JOËL: Hint, hint, RailsConf idea. There's something from your original question that I think is interesting to circle back to, and that's the fact that it was awkward to work through in Ruby to do the work that you wanted to do because the tables were laid out in a certain way. And sometimes, there's certain ways that you need the tables to be in order to be sort of safe to represent data, but they're not the optimal way that we would like to interact with them at the Ruby level. And I think it's okay for not everything in Ruby to be 100% reflective of the structure of the tables underneath. ActiveRecord gives us a great pattern, but everything is kind of one-to-one. And it's okay to layer on some things on top, add some extra methods to build some, like, connections in Ruby that rely on this normalized data underneath but that make life easier for you, or they better just represent or describe the relationships that you have. STEPHANIE: 100%. I was really compelled by your idea of introducing helpers that use more descriptive adjectives for what that relationship is like. We've talked about how Rails abstracted things from the database level, you know, for our convenience, but that should not stop us from, like, leaning on that further, right? And kind of introducing our own abstractions for those connections that we see in our domain. So, I feel really inspired. I might even kind of reconsider the way I handled the original example and see what I can make of it. JOËL: And I think your original solution of doing the delegation is a great example of this as well. You want the idea that a device belongs to a company or has an association called company, and you just don't want to go through that long chain, or at least you don't want that to be visible as an implementation detail. So, in this case, you delegate it through a chain of methods in Ruby. It could also be that you have a much longer chain of tables, and maybe they don't all have associations in Rails and all that. And I think it would be totally fine as well to define a method on an object where, I don't know, a device, I don't know, has many...let's call it technicians, which is everybody who's ever touched this device or, you know, is on a log somewhere for having done maintenance. And maybe that list of technicians is not a thing you can just get through regular Rails associations. Maybe there's a whole, like, custom query underlying that, and that's okay. STEPHANIE: Yeah, as you were saying that, I was thinking about that's actually kind of, like, active models are the great spot to put those methods and that logic. And I think you've made a really good case for that. JOËL: On that note, shall we wrap up? STEPHANIE: Let's wrap up. Show notes for this episode can be found at bikeshed.fm. JOËL: This show has been produced and edited by Mandy Moore. STEPHANIE: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review in iTunes. It really helps other folks find the show. JOËL: If you have any feedback for this or any of our other episodes, you can reach us @_bikeshed, or you can reach me @joelquen on Twitter. STEPHANIE: Or reach both of us at hosts@bikeshed.fm via email. JOËL: Thanks so much for listening to The Bike Shed, and we'll see you next week. ALL: Byeeeeeeee!!!!!!!!!! AD: Did you know thoughtbot has a referral program? If you introduce us to someone looking for a design or development partner, we will compensate you if they decide to work with us. More info on our website at: tbot.io/referral. Or you can email us at: referrals@thoughtbot.com with any questions.

english vision mvp id gang congratulations pok modeling github dry devops associations rails spotify wrapped googling elm prs mandy moore dba bnf design patterns christopher alexander quenneville data modeling railsconf activerecord thoughtbot richard feldman pattern language stephanie you stephanie yeah stephanie right

RailsConf 2024 Workshop Spotlight — Build High Performance Active Record Apps with Andy Atkinson

Ruby for All

Play Episode Listen Later May 2, 2024 43:52

In this episode of 'Ruby for All', Andrew and Julie discuss drawing inspiration from MC Escher through games like Monument Valley, to dealing with the intricacies of Discord roles and authorization, and the importance of immediate and continuous feedback through tools like Google Docs during talks. Then, guest Andrew Atkinson joins us and shares insights from his new book, “High Performance PostgreSQL for Rails,” detailing his journey from initial drafts to publishing and his shift towards independent consulting. He emphasizes the significance of understanding database operations, schema design, and efficient querying for optimizing Rails applications. Also, Andy talks about preparing a workshop for RailsConf, aiming to educate participants on query performance improvement techniques and the utility of using multiple Postgres instances. The conversation also touches upon the learning strategies, potential challenges, and benefits of workshops versus talks at conferences. Hit download now to hear more! [00:00:10] Julie started drawing again inspired by MC Escher and playing a game called Monument Valley, and Andrew mentions he's on a tilt dues to issues with Discord roles. [00:01:59] Andrew introduces the git command ‘git instaweb' as a cool new find and shares something he remembered going back to the getting feedback for talks topic.[00:04:24] Andrew “Andy” Atkinson introduces himself and discusses the completion of his book, “High Performance PostgreSQL for Rails,” the positive response in beta sales, and his new venture into independent consulting. [00:08:16] Andy talks about his shift from development work to more educational and consultative roles, considering diving deeper into Postgres development. [00:09:48] There's a discussion about Andy balancing work-life commitments, creating content-like videos and tutorials, and leveraging these for marketing and educational purposes in the tech community. [00:11:29] Andy considers the idea of making short videos for platforms like TikTok and Instagram Reels, and he talks about his preference for watching conference talks on YouTube over popular content creators. He also talks about Hussein Nassar's videos on Udemy and how he encouraged him to make short videos. [00:15:01] Andy is conducting a workshop at RailsConf and expresses his excitement about presenting at RailsConf and the opportunity to connect with people interested in query and database optimization.[00:17:09] Julie shares her preference for learning through hands-on workshops and looks forward to participating in Andy's workshop. Andy gives us a sneak peak of his workshop which will focus on query performance, query running, and index support, as well as exploring the benefits of having multiple Postgres instances.[00:20:19] Andrew asks if Docker is necessary for the workshop, leading to a discussion on the practicality of simulating different database instances. [00:22:10] Andy plans to prepare for potential challenges such as internet issues by possibly providing content on USB drives and ensuring attendees can access prerequisites before the workshop. He emphasizes the workshop format will be more hands-on with less lecturing. [00:24:06] Julie asks about the prerequisites needed for audience members attending the workshop, especially if they're new to Rails or databases. Andy clarifies that attendees should have at least built a database-backed Rails app or have similar experience with another language or framework,[00:25:44] Julie mentions that there's a desire for more advanced content in talks and having a range allows participants to engage at different levels. Andrew shares his preference for advanced topics in workshops.[00:29:45] Andrew explains his preference for collaborative learning and anticipates the second day of RailsConf to be different and beneficial for those who like to pair and bounce ideas off others. Andy wants to ensure that the workshop content is new and valuable, different from what attendees might learn elsewhere. [00:32:11] Andy outlines the key takeaways he hopes attendees will leave with, including skills to improve the speed and scalability of their web apps, understanding database operations, and leveraging multiple databases with Rails Active Record. [00:34:04] Andrew shares while reading Andy's talk outline, he realized he wasn't sure when to use indexes outside of standard use cases. Andy acknowledges the importance of not just solving existing problems with indexes, but also identifying where problems may arise in Postgres by tracking queries not using indexes. [00:36:35] Andrew discusses the existence of gems like lol_dba, which suggest potential indexing opportunities, but notes the difficulty in validating those suggestions. Andy mentions other tools like Rails PG Extras and tells us the workshop will demonstrate how to use the ‘explain' command to evaluate the use and impact if indexes on individual query performance. [00:38:44] We end with Andrew inquiring why Postgres does not allow control over the query plan selection. Andy responds that Postgres' declarative paradigm aims for the planner to continually adapt and choose the lowest cost plan and mentions an extension called pg_hint_plan.[00:40:54] Find out where you can follow Andy online, where to get his book, and his upcoming conference plans.Panelists:Andrew MasonJulie J.Guest:Andrew AtkinsonSponsors:HoneybadgerGoRailsLinks:Andrew Mason X/TwitterAndrew Mason WebsiteJulie J. X/TwitterJulie J. WebsiteAndrew Atkinson WebsiteAndrew Atkinson X/TwitterAndrew Atkinson ConsultingM.C. Escher Monument Valley git instawebRuby for All-Episode 26: The Database Wizard with Andrew AtkinsonHigh Performance PostgreSQL for Rails: Reliable, Scalable, Maintainable Database Applications by Andrew AtkinsonHussein Nassar (Udemy)Rideshare-Rails app for “High Performance PostgreSQL for Rails” RailsConf 2024RailsConf 2024 Schedule: Andy Atkinson, May 8th, 2:30-Build High Performance Active Record Appslol_dba

tiktok technology software discord apps workshop high performance usb rails google docs udemy scalable docker monument valley postgres andrew mason mc escher railsconf activerecord andrew atkinson andy atkinson

S2E1: Andrew Atkinson, Software Engineer and Author

Hacking Postgres

Play Episode Listen Later Apr 4, 2024 35:32

Andrew Atkinson is a Software Engineer who specializes in building high-performance web applications using PostgreSQL and Ruby on Rails. He wrote the book ‘High-Performance PostgreSQL for Rails', published by Pragmatic Programmers in 2024.Our discussion with Andrew spans the technical challenges of sharding and the concurrent evolution of Rails and Postgres. We'll pay homage to influential resources like Railscast, debate Rails' database tooling limitations, and discover tips from Andrew's new book.In this episode we explore:Why newer developers favor Postgres over MySQLHow Postgres might become a multi-primary database in the futureThe complexities of database decisions in a Rails environmentPostgres innovations, such as composite primary keys and common table expressions, being supported from Active Record – the ORM for Ruby on RailsKey insights from writing ‘High Performance PostgreSQL for Rails'Links mentioned:Andrew Atkinson on LinkedInAndrew's BlogNewsletterPGCastsRailsCasts‘High Performance PostgreSQL for Rails' by Andrew AtkinsonGithub ridesharePostgres FMAndrew Aktkinson's interview on Postgres FMAndrew Atkinson's interview on Remote RubyRemote Ruby PodcastGitHub doc: clarify logical decoding's deadlock of system tablesGitHub doc: Doc: fix grammatical errors for enable_partitionwise_aggregate GitHub Convert README to Markdown

doc rails software engineers extensions software development ruby on rails markdown orm postgresql postgres futurethe activerecord andrew atkinson pragmatic programmers railscast

420: Test Database Woes

Play Episode Listen Later Mar 26, 2024 28:16

Joël shares his recent project challenge with Tailwind CSS, where classes weren't generating as expected due to the dynamic nature of Tailwind's CSS generation and pruning. Stephanie introduces a personal productivity tool, a "thinking cap," to signal her thought process during meetings, which also serves as a physical boundary to separate work from personal life. The conversation shifts to testing methodologies within Rails applications, leading to an exploration of testing philosophies, including developers' assumptions about database cleanliness and their impact on writing tests. Avdi's classic post on how to use database cleaner (https://avdi.codes/configuring-database_cleaner-with-rails-rspec-capybara-and-selenium/) RSpec change matcher (https://rubydoc.info/gems/rspec-expectations/RSpec%2FMatchers:change) Command/Query separation (https://martinfowler.com/bliki/CommandQuerySeparation.html) When not to use factories (https://thoughtbot.com/blog/speed-up-tests-by-selectively-avoiding-factory-bot) Why Factories? (https://thoughtbot.com/blog/why-factories) Transcript: STEPHANIE: Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Stephanie Minn. JOËL: And I'm Joël Quenneville. And together, we're here to share a bit of what we've learned along the way. STEPHANIE: So, Joël, what's new in your world? JOËL: I'm working on a new project, and this is a project that uses Tailwind CSS for its styling. And I ran into a bit of an annoying problem with it just getting started, where I was making changes and adding classes. And they were not changing the things I thought they would change in the UI. And so, I looked up the class in the documentation, and then I realized, oh, we're on an older version of the Tailwind Rails gem. So, maybe we're using...like, I'm looking at the most recent docs for Tailwind, but it's not relevant for the version I'm using. Turned out that was not the problem. Then I decided to use the Web Inspector and actually look at the element in my browser to see is it being overwritten somehow by something else? And the class is there in the element, but when I look at the CSS panel, it does not show up there at all or having any effects. And that got me scratching my head. And then, eventually, I figured it out, and it's a bit of a facepalm moment [laughs]. STEPHANIE: Oh, okay. JOËL: Because Tailwind has to, effectively, generate all of these, and it will sort of generate and prune the things you don't need and all of that. They're not all, like, statically present. And so, if I was using a class that no one else in the app had used yet, it hadn't gotten generated. And so, it's just not there. There's a class on the element, but there's no CSS definition tied to it, so the class does nothing. What you need to do is there's a rake task or some sort of task that you can run that will generate things. There's also, I believe, a watcher that you can run, some sort of, like, server that will auto-generate these for you in dev mode. I did not have that set up. So, I was not seeing that new class have any effect. Once I ran the task to generate things, sure enough, it worked. And Tailwind works exactly how the docs say they do. But that was a couple of hours of my life that I'm not getting back. STEPHANIE: Yeah, that's rough. Sorry to hear. I've also definitely gone down that route of like, oh, it's not in the docs. The docs are wrong. Like, do they even know what they're talking about? I'm going to fix this for everyone. And similarly have been humbled by a facepalm solution when I'm like, oh, did I yarn [laughs]? No, I didn't [laughs]. JOËL: Uh-huh. I'm curious, for you, when you have sort of moments where it's like the library is not behaving the way you think it is, is your default to blame yourself, or is it to blame the library? STEPHANIE: [laughs]. Oh, good question. JOËL: And the follow-up to that is, are you generally correct? STEPHANIE: Yeah. Yep, yep, yep. Hmm, I will say I externalize the blame, but I will try to at least do, like, the basic troubleshooting steps of restarting my server [laughter], and then if...that's as far as I'll go. And then, I'll be like, oh, like, something must be wrong, you know, with this library, and I turn to Google. And if I'm not finding any fruitful results, again, you know, one path could be, oh, maybe I'm not Googling correctly, but the other path could be, maybe I've discovered something that no one else has before. But to your follow-up question, I'm almost, like, always wrong [laughter]. I'm still waiting for the day when I, like, discover something that is an actual real problem, and I can go and open an issue [chuckles] and, hopefully, be validated by the library author. JOËL: I think part of what I heard is that your debugging strategy is basic, but it's not as basic as Joël's because you remember to restart the server [chuckles]. STEPHANIE: We all have our days [laughter]. JOËL: Next time. So, Stephanie, what is new in your world? STEPHANIE: I'm very excited to share this with you. And I recognize that this is an audio medium, so I will also describe the thing I'm about to show you [laughs]. JOËL: Oh, this is an object. STEPHANIE: It is an object. I got a hat [laughs]. JOËL: Okay. STEPHANIE: I'm going to put it on now. It's a cap that says "Thinking" on it [laughs] in, like, you know, fun sans serif font with a little bit of edge because the thinking is kind of slanted. So, it is designy, if you will. It's my thinking cap. And I've been wearing it at work all week, and I love it. As a person who, in meetings and, you know, when I talk to people, I have to process before I respond a lot of the time, but that has been interpreted as, you know, maybe me not having anything to say or, you know, people aren't sure if I'm, you know, still thinking or if it's time to move on. And sometimes I [chuckles], you know, take a long time. My brain is just spinning. I think another funny hat design would be, like, the beach ball, macOS beach ball. JOËL: That would be hilarious. STEPHANIE: Yeah. Maybe I need to, like, stitch that on the back of this thinking cap. Anyway, I've been wearing it at work in meetings. And then, when I'm just silently processing, I'll just point to my hat and signal to everyone what's [laughs] going on. And it's also been really great for the end of my work day because then I take off the hat, and because I've taken it off, that's, like, my signal, you know, I have this physical totem that, like, now I'm done thinking about work, and that has been working. JOËL: Oh, I love that. STEPHANIE: Yeah, that's been working surprisingly well to kind of create a bit more of a boundary to separate work thoughts and life thoughts. JOËL: Because you are working from home and so that boundary between professional life and personal life can get a little bit blurry. STEPHANIE: Yeah. I will say I take it off and throw it on the floor kind of dramatically [laughter] at the end of my work day. So, that's what's new. It had a positive impact on my work-life balance. And yeah, if anyone else has the problem of people being confused about whether you're still thinking or not, recommend looking into a physical thinking cap. JOËL: So, you are speaking at RailsConf this spring in Detroit. Do you plan to bring the thinking cap to the conference? STEPHANIE: Oh yeah, absolutely. That's a great idea. If anyone else is going to RailsConf, find me in my thinking cap [laughs]. JOËL: So, this is how people can recognize Bikeshed co-host Stephanie Minn. See someone walking around with a thinking cap. STEPHANIE: Ooh. thinkingbot? JOËL: Ooh. STEPHANIE: Have I just designed new thoughtbot swag [laughter]? We'll see if this catches on. JOËL: So, we were talking recently, and you'd mentioned that you were facing some really interesting dilemmas when it came to writing tests and particularly how tests interact with your test database. STEPHANIE: Yeah. So, I recently, a few weeks ago, joined a new client project and, you know, one of the first things that I do is start to run those tests [laughs] in their codebase to get a sense of what's what. And I noticed that they were taking quite a long time to get set up before I even saw any progress in terms of successes or failures. So, I was kind of curious what was going on before the examples were even run. And when I tailed the logs for the tests, I noticed that every time that you were running the test suite, it would truncate all of the tables in the test database. And that was a surprise to me because that's not a thing that I had really seen before. And so, basically, what happens is all of the data in the test database gets deleted using this truncation strategy. And this is one way of ensuring a clean slate when you run your tests. JOËL: Was this happening once at the beginning of the test suite or before every test? STEPHANIE: It was good that it was only running once before the test suite, but since, you know, in my local development, I'm running, like, a file at a time or sometimes even just targeting a specific line, this would happen on every run in that situation and was just adding a little bit of extra time to that feedback loop in terms of just making sure your code was working if that's part of your workflow. JOËL: Do you know what version of Rails this project was in? Because I know this was popular in some older versions of Rails as a strategy. STEPHANIE: Yeah. So, it is Rails 7 now, recently upgraded to Rails 7. It was on Rails 6 for a little while. JOËL: Very nice. I want to say that truncation is generally not necessary as of Rails...I forget if it's 5 or 6. But back in the day, specifically for what are now called system tests, the sort of, like, Capybara UI-driven browser tests, you had, effectively, like, two threads that were trying to access the database. And so, you couldn't have your test data wrapped in a transaction the way you would for unit tests because then the UI thread would not have access to the data that had been created in a transaction just for the test thread. And so, people would use tools like Database Cleaner to use a truncation strategy to clear out everything between tests to allow a sort of clean slate for these UI-driven feature specs. And then, I want to say it's Rails 5, it may have been Rails 6 when system tests were added. And one of the big things there was that they now could, like, share data in a transaction instead of having to do two separate threads and one didn't have access to it. And all of a sudden, now you could go back to transactional fixtures the way that you could with unit tests and really take advantage of something that's really nice and built into Rails. STEPHANIE: That's cool. I didn't know that about system tests and that kind of shift happening. I do think that, in this case, it was one of those situations where, in the past, the database truncation, in this case, particular using the Database Cleaner gem was necessary, and that just never got reassessed as the years went by. JOËL: That's one of the classic things, right? When you upgrade a Rails app over multiple versions, and sometimes you sort of get a new feature that comes in for free with the new version, and you might not be aware of it. And some of the patterns in the app just kind of keep going. And you don't realize, hey, this part of the app could actually be modernized. STEPHANIE: So, another interesting thing about this testing situation is that I learned that, you know, if you ran these tests, you would experience this truncation strategy. But the engineering team had also kind of played around with having a different test setup that didn't clean the database at all unless you opted into it. JOËL: So, your test database would just...each test would just keep writing to the database, but they're not wrapped in transactions. Or they are wrapped in transactions, but you may or may not have some additional data. STEPHANIE: The latter. So, I think they were also using the transaction strategy there. But, you know, there are some reasons that you would still have some data persisted across test runs. I had actually learned that the use transactional fixtures config for RSpec doesn't roll back any data that might have been created in a before context hook. JOËL: Yep, or a before all. Yeah, the transaction wraps the actual example, but not anything that happens outside of it. STEPHANIE: Yeah, I thought that was an interesting little gotcha. So, you know, now we had these, like, two different ways to run tests. And I was chatting with a client developer about how that came to be. And we then got into an interesting conversation about, like, whether or not we each expect a clean database in the first place when we write our tests or when we run our tests, and that was an area that we disagreed. And that was cool because I had not really, like, thought about like, oh, how did I even arrive at this assumption that my database would always be clean? I think it was just, you know, from experience having only worked in Rails apps of a certain age that really got onto the Database [laughs] Cleaner train. But it was interesting because I think that is a really big assumption to make that shapes how you then approach writing tests. JOËL: And there's kind of a couple of variations on that. I think the sort of base camp approach of writing Rails with fixtures, you just sort of have, for the most part, an existing set of data that's there that you maybe layer on a few extra things on. But there's base level; you just expect a bunch of data to exist in your test database. So, it's almost going off the opposite assumption, where you can always assume that certain things are already there. Then there's the other extreme of, like, you always assume that it's empty. And it sounds like maybe there's a position in the middle of, like, you never know. There may be something. There may not be something, you know, spin the wheel. STEPHANIE: Yeah. I guess I was surprised that it, you know, that was just a question that I never really asked myself prior to this conversation, but it could feel like different testing philosophies. But yeah, I was very interested in this, you know, kind of opinion that was a little bit different from mine about if you assume that your database, your test database, is not clean, that kind of perhaps nudges you in the direction of writing tests that are less coupled to the database if they don't need to be. JOËL: What does coupling to the database mean in this situation? STEPHANIE: So, I'm thinking about Rails tests that might be asserting on a change in database behavior, so the change matcher in RSpec is one that I see maybe sometimes used when it doesn't need to be used. And we're expecting, like, account to have changed the count of the number of records on it for a model have changed after doing some work, right? JOËL: And the change matcher from RSpec is one that allows you to not care whether there are existing records or not. It sort of insulates you from that. STEPHANIE: That's true. Though I guess I was thinking almost like, what if there was some return value to assert on instead? And would that kind of help you separate some side effects from methods that might be doing too much? And kind of when I start to see tests that have both or are asserting on something being returned, and then also something happening, that's one way of, like, figuring out what kind of coupling is going on inside this test. JOËL: It's the classic command-query separation principle from object-oriented design. STEPHANIE: I think another one that came to mind, another example, especially when you're talking about system tests, is when you might be using Capybara and you end up...maybe you're going through a flow that creates a record. But from the user perspective, they don't actually know what's going on at the database level. But you could assert that something was created, right? But it might be more realistic at that level of abstraction to be asserting some kind of visual element that had happened as a result of the flow that you're testing. JOËL: Yeah. I would, in fact, go so far as to say that asserting on the state of your database in a system test is an anti-pattern. System tests are sort of, by design, meant to be all about user behavior trying to mimic the experience of a user. And a user of a website is not going to be able to...you hope they're not able to SSH into [chuckles] your database and check the records that have been created. If they can, you've got another problem. STEPHANIE: I wonder if you could take this idea to the extreme, though. And do you think there is a world where you don't really test database-level concerns at all if you kind of believe this idea that it doesn't really matter what the state of it should be? JOËL: I guess there's a few different things on, like, what it matters about the state of it because you are asserting on its state sort of indirectly in a sort of higher level integration test. You're asserting that you see certain things show up on the screen in a system test. And maybe you want to say, "I do certain tasks, and then I expect to see three items in an unordered list." Those three items probably come from the database, although, you know, you could have it where they come from an API or something like that. So, the database is an implementation level. But if you had random data in your database, you might, in some tests, have four items in the list, some tests have five. And that's just going to be a flaky test, and that's going to be incredibly painful. So, while you're not asserting on the database, having control over it during sort of test setup, I think, does impact the way you assert. STEPHANIE: Yeah, that makes sense. I was suddenly just thinking about, like, how that exercise can actually tell you perhaps, like, when it is important to, in your test setup, be persisting real records as opposed to how much you can get away with, like, not interacting with it because, like, you aren't testing at that integration level. JOËL: That brings up a good point because a lot of tests probably you might need models, but you might not need persisted models to interact with them, if you're testing a method on a model that just does things based off its internal state and not any of the ActiveRecord database queries, or if you have some other service or something that consumes a model that doesn't necessarily need to query. There's a classic blog post on the thoughtbot blog about when you should not reuse. There's a classic blog post on the thoughtbot blog about when not to use FactoryBot. And, you know, we are the makers of FactoryBot. It helps set up records in your database for testing. And people love to use it all the time. And we wrote an article about why, in many cases, you don't need to create something into the database. All you need is just something in memory, and that's going to be much faster than using FactoryBot because talking to the database is expensive. STEPHANIE: Yeah, and I think we can see that in the shift from even, like, fixtures to factories as well, where test data was only persisted as needed and as needed in individual tests, rather than seeding it and having all of those records your entire test run. And it's cool to see that continuing, you know, that idea further of like, okay, now we have this new, popular tool that reduce some of that. But also, in most cases, we still don't need...it's still too much. JOËL: And from a performance perspective, it's a bit of a see-saw in that fixtures are a lot faster because they get inserted once at the beginning of your test run. So, a SQL execution at the beginning of a test run and then every test after that is just doing its thing: maybe creating a record inside of a transaction, maybe not creating any records at all. And so, it can be a lot faster as opposed to using FactoryBot where you're creating records one at a time. Every create call in a test is a round trip to the database, and those are expensive. So, FactoryBot tests tend to be more expensive than those that rely on fixtures. But you have the advantage of more control over what data is present and sort of more locality because you can see what has been created at the test level. But then, if you decide, hey, this is a test where I can just create records in memory, that's probably the best of all worlds in that you don't need anything created ahead with fixtures. You also don't need anything to be inserted using FactoryBot because you don't even need the database for this test. STEPHANIE: I'm curious, is that the assumption that you start with, that you don't need a persisted object when you're writing a basic unit test? JOËL: I think I will as much as possible try not to need to persist and only if necessary use persist records. There are strategies with FactoryBot that will allow you to also, like, build stubbed or just build in memory. So, there's a few different variations that will, like, partially do things for you. But oftentimes, you can just new up an object, and that's what I will often start with. In many cases, I will already know what I'm trying to do. And so, I might not go through the steps of, oh, new up an object. Oh no, I'm getting a I can't do the thing I need to do. Now, I need to write to the database. So, if I'm testing, let's say, an ActiveRecord scope that's filtering down a series of records, I know that's a wrapper around a database query. I'm not going to start by newing up some records and then sort of accidentally discovering, oh yeah, it does write to the database because that was pretty clear to me from the beginning. STEPHANIE: Yeah. Like, you have your mental shortcuts that you do. I guess I asked that question because I wonder if that is a good heuristic to share with maybe developers who are trying to figure out, like, should they create persisted records or, you know, use just regular instance in memory or, I don't know, even [laughs] use, like, a double [laughs]? JOËL: Yeah, I've done that quite a bit as well. I would say maybe my heuristic is, is the method under test going to need to talk to the database? And, you know, I may or may not know that upfront because if I'm test driving, I'm writing the test first. So, sometimes, maybe I don't know, and I'll start with something in memory and then realize, oh, you know, I do need to talk to the database for this. And this is for unit tests, in particular. For something more like an integration test or a system test that might require data in the database, system tests almost always do. You're not interacting with instances in memory when you're writing a system test, right? You're saying, "Given the database state is this when I visit this URL and do these things, this page reacts in such and such a way." So, system tests always write to the database to start with. So, maybe that's my heuristic there. But for unit tests, maybe think a little bit about does your method actually need to talk to the database? And maybe even almost give yourself a challenge. Can I get away with not talking to the database here? STEPHANIE: Yeah, I like that because I've certainly seen a lot of unit tests that are integration tests in disguise [laughs]. JOËL: Isn't that the truth? So, we kind of opened up this conversation with the idea of there are different ways to manage your database in terms of, do you clean or not clean before a test run? Where did you end up on this particular project? STEPHANIE: So, I ended up with a currently open PR to remove the need to truncate the database on each run of the test suite and just stick with the transaction for each example strategy. And I do think that this will work for us as long as we decide we don't want to introduce something like fixtures, even though that is actually also a discussion that's still in the works. But I'm hoping with this change, like, right now, I can help people start running faster tests [chuckles]. And should we ever introduce fixtures down the line, then we can revisit that. But it's one of those things that I think we've been living with this for too long [laughs]. And no one ever questioned, like, "Oh, why are we doing this?" Or, you know, maybe that was a need, however many years ago, that just got overlooked. And as a person new to the project, I saw it, and now I'm doing something about it [laughs]. JOËL: I love that new person energy on a project and like, "Hey, we've got this config thing. Did you know that we didn't need this as of Rails 6?" And they're like, "Oh, I didn't even realize that." And then you add that, and it just moves you into the future a little bit. So, if I understand the proposed change, then you're removing the truncation strategy, but you're still going to be in a situation where you have a clean database before each test because you're wrapping tests in transactions, which I think is the default Rails behavior. STEPHANIE: Yeah, that's where we're at right now. So, yeah, I'm not sure, like, how things came to be this way, but it seemed obvious to me that we were kind of doing this whole extra step that wasn't really necessary, at least at this point in time. Because, at least to my knowledge [laughs], there's no data being seeded in any other place. JOËL: It's interesting, right? When you have a situation where this was sort of a very popular practice for a long time, a lot of guides mentioned that. And so, even though Rails has made changes that mean that this is no longer necessary, there's still a long tail of apps that will still have this that may be upgraded later, and then didn't drop this, or maybe even new apps that got created but didn't quite realize that the guide they were following was outdated, or that a best practice that was in their head was also outdated. And so, you have a lot of apps that will still have these sort of, like, relics of the past. And you're like, "Oh yeah, that's how we used to do things." STEPHANIE: So yeah, thanks, Joël, for going on this journey with me in terms of, you know, reassessing my assumptions about test databases. I'm wondering, like, if this is common, how other people, you know, approach what they expect from the test database, whether it be totally clean or have, you know, any required data for common flows and use cases of your system. But it does seem that little in between of, like, maybe it is using transactions to reset for each example, but then there's also some persistence that's happening somewhere else that could be a little tricky to manage. JOËL: On that note, shall we wrap up? STEPHANIE: Let's wrap up. Show notes for this episode can be found at bikeshed.fm. JOËL: This show has been produced and edited by Mandy Moore. STEPHANIE: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review in iTunes. It really helps other folks find the show. JOËL: If you have any feedback for this or any of our other episodes, you can reach us @_bikeshed, or you can reach me @joelquen on Twitter. STEPHANIE: Or reach both of us at hosts@bikeshed.fm via email. JOËL: Thanks so much for listening to The Bike Shed, and we'll see you next week. ALL: Byeeeeeeeeeeee!!!!!! AD: Did you know thoughtbot has a referral program? If you introduce us to someone looking for a design or development partner, we will compensate you if they decide to work with us. More info on our website at: tbot.io/referral. Or you can email us at: referrals@thoughtbot.com with any questions.

google pr thinking system detroit work life balance woes api ui databases macos rails cleaner googling css sql continuous learning mandy moore tailwinds debugging ssh capybara professional communication quenneville tailwind css development tools thinking cap railsconf activerecord avdi rspec stephanie yeah stephanie it stephanie oh

419: What's New in Your World? (Extended Edition)

Play Episode Listen Later Mar 19, 2024 37:13

Stephanie introduces her ideal setup for enjoying coffee on a bike ride. Joël describes his afternoon tea ritual. Exciting news from the hosts: both have been accepted to speak at RailsConf! Stephanie's presentation, titled "So, Writing Tests Feels Painful. What now?" aims to tackle the issues developers encounter with testing while offering actionable advice to ease these pains. Joël's session will focus on utilizing Turbo to create a Dungeons & Dragons character sheet, combining his passion for gaming with technical expertise. Their conversation shifts to artificial intelligence and its potential in code refactoring and other applications, such as enhancing the code review process and solving complex software development problems. Joël shares his venture into combinatorics, illustrating how this mathematical approach helped him efficiently refactor a database query by systematically exploring and testing all potential combinations of query segments. Transcript: JOËL: Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Joël Quenneville. STEPHANIE: And I'm Stephanie Minn, and together, we're here to share a bit of what we've learned along the way. JOËL: So, Stephanie, what's new in your world? STEPHANIE: So, today I went out for a coffee on my bike, and I feel like I finally have my perfect, like, on-the-go coffee setup. We have this thoughtbot branded travel mug. So, it's one of the little bits of swag that we got from the company. It's, like, perfectly leak-proof. I'll link the brand in the show notes. But it's perfectly leak-proof, which is great. And on my bike, I have a little stem bag, so it's just, like, a tiny kind of, like, cylindrical bag that sits on the, like, vertical part of my handlebars that connects to the rest of my bag. And it's just, like, the perfect size for a 12-ounce coffee. And so, I put my little travel mug in there, and I just had a very refreshing morning. And I'd gone out on my bike for a little bit, stopping by for coffee and headed home to work. And I got to drink my coffee during my first meeting. So, it was a wonderful way to start the day. JOËL: Do you just show up at the coffee shop with your refillable mug and say, "Hey, can you pour some coffee in this?" STEPHANIE: Yeah. I think a lot of coffee places are really amenable to bringing your own travel mugs. So yeah, it's really nice because I get to use less plastic. And also, you know, when you get a to-go mug, it is not leak-proof, right? It could just slosh all over the place and spill, so not bike-friendly. But yeah, bring your own mug. It's very easy. JOËL: Excellent. STEPHANIE: So, Joël, what's new in your world? JOËL: Also, warm beverages. Who would have thought? It's almost like it's cold in North America or something. I've been really enjoying making myself tea in the afternoons recently. And I've been drinking this brand of tea that is a little bit extra. Every flavor of tea they have comes with a description of how the tea feels. STEPHANIE: Ooh. JOËL: I don't know who came up with these, but they're kind of funny. So, one that I particularly enjoy is described as feels like stargazing on an empty beach. STEPHANIE: Wow. That's very specific. JOËL: They also give you tasting notes. This one has tastes of candied violet, elderberry, blackberry, and incense. STEPHANIE: Ooh, that sounds lovely. Are you drinking, like, herbal tea in the afternoon, or do you drink caffeinated tea? JOËL: I'll do caffeinated tea. I limit myself to one pot of coffee that I brew in the morning, and then, whenever that's done, I switch to tea. Tea I allow myself anything: herbal, black tea; that's fine. STEPHANIE: Yeah, I can't have too much caffeine in the afternoon either. But I do love an extra tea. I wish I could remember, like, what even was in this tea or what brand it was, but once I had a tea that was a purplish color. But then, when you squeeze some lemon in it, or I guess maybe anything with a bit of acid, it would turn blue. JOËL: Oh, that's so cool. STEPHANIE: Yeah, I'll have to find what this tea was [laughs] and update the podcast for any tea lovers out there. But yeah, it was just, like, a little bit of extra whimsy to your regular routine. JOËL: I love adding a little whimsy to my day, even if it's just seeing a random animated GIF that a coworker has sent or Tuple has some of the, like, reactions you can send if you're pairing with someone. And I don't use those very often, so whenever one of those comes through, and it's like, ship it or yay, that makes me very happy. STEPHANIE: Agreed. JOËL: This week is really fun because as we were prepping for this episode, we both realized that there is a lot that's been new in our world recently. And Stephanie, in particular, you've got some pretty big news that recently happened to you. STEPHANIE: Yeah, it turns out we're making the what's new in your world segment the entire episode today [laughs]. But my news is that I am speaking at RailsConf this year, so that is May 7th through 9th in Detroit. And so, yeah, I haven't spoken at a RailsConf before, only a RubyConf. So, I'm looking forward to it. My talk is called: So, Writing Tests Feels Painful. What now? JOËL: Wait, is writing tests ever painful [laughs]? STEPHANIE: Maybe not for you, but for the rest of us [laughs]. JOËL: No, it absolutely is. I, right before this recording, came from a pairing session where we were scratching our heads on an, like, awkward-to-write test. It happens to all of us. STEPHANIE: Yeah. So, I was brainstorming topics, and I kind of realized, especially with a lot of our consulting experience, you know, we hear from developers or even maybe, like, engineering managers a lot of themes around like, "Oh, like, development is slowing down because our test suite is such a headache," or "It's really slow. It's really flaky. It's really complicated." And that is a pain point that a lot of tech leaders are also looking to address for their teams. But I was really questioning this idea that, like, it always had to be some effort to improve the test suite, like, that had to be worked on at some later point or get, like, an initiative together to fix all of these problems, and that it couldn't just be baked into your normal development process, like, on an individual level. I do think it is really easy to feel a lot of pain when trying to write tests and then just be like, ugh, like, I wish someone would fix this, right? Or, you know, just kind of ignore the signals of that pain because you don't know, like, how to manage it yourself. So, my talk is about when you do feel that pain, really trying to determine if there's anything you can do, even in just, like, the one test file that you're working in to make things a little bit easier for yourself, so it doesn't become this, like, chronic issue that just gets worse and worse. Is there something you could do to maybe reorganize the file as you're working in it to make some conditionals a little bit clearer? Is there any, like, extra test setup that you're like, "Oh, actually, I don't need this anymore, and I can just start to get rid of it, not just for this one example, but for the rest in this file"? And do yourself a favor a little bit. So yeah, I'm excited to talk about that because I think that's perhaps, like, a skill that we don't focus enough on. JOËL: Are you going to sort of focus in on the side of things where, like, a classic TDD mantra is that test pain reflects underlying code complexity? So, are you planning to focus on the idea of, oh, if you're feeling test pain, maybe take some time to refactor some of the code that's under test, maybe because there's some tight coupling? Or are you going to lean a little bit more into maybe, like, the Boy Scout rule, you know, 'Leave the campsite cleaner than you found it' for your test files? STEPHANIE: Ooh, I like that framing. Definitely more of the former. But one thing I've also noticed working with a lot of client teams is that it's not always clear, like, how to refactor. I think a lot of intermediate developers start to feel that pain but don't know what to do about it. They don't know, like, maybe the code smells, or the patterns, or refactoring strategies, and that can certainly be taught. It will probably pull from that. But even if you don't know those skills yet, I'm wondering if there's, like, an opportunity to teach, like, developers at that level to start to reflect on the code and be like, "Hmm, what could I do to make this a little more flexible?" And they might not know the names of the strategies to, like, extract a class, but just start to get them thinking about it. And then maybe when they come across that vocabulary later, it'll connect a lot easier because they'll have started to think about, you know, their experiences day to day with some of the more conceptual stuff. JOËL: I really like that because I feel we've probably all heard that idea that test pain, especially when you're test driving, is a sign of maybe some anti-patterns or some code smells in the underlying code that you're testing. But translating that into something actionable and being able to say, "Okay, so my tests are painful. They're telling me something needs to be refactored. I'm looking at this code, and I don't know what to refactor." It's a big jump. It's almost the classic draw two circles; draw the rest of the owl meme. And so, I think bridging that gap is something that is really valuable for our community. STEPHANIE: Yeah, that's exactly what I hope to do in my talk. So, Joël, you [chuckles] also didn't quite mention that you have big news as well. JOËL: So, I also got accepted to speak at RailsConf. I'm giving a talk on Building a Dungeons & Dragons Character Sheet Using Turbo. STEPHANIE: That's really awesome. I'm excited because I want to learn more about Turbo. I want someone else to tell me [laughs] what I can do with it. And as a person with a little bit of Dungeons & Dragons experience, I think a character sheet is kind of the perfect vehicle for that. JOËL: Building a D&D character sheet has been kind of my go-to project to experiment with a new front-end framework because it's something that's pretty dynamic. And for those who don't know, there's a bunch of fields that you fill in with stats for different attributes that your character has, but then those impact other stats that get rendered. And sometimes there can be a chain two or three long where different numbers kind of combine together. And so, you've got this almost dependency tree of, like, a particular number. Maybe your skill at acrobatics might depend on a number that you entered in the dexterity field, but it also depends on your proficiency bonus, and maybe also depends on the race that you picked and a few other things. And so, calculating those numbers all of a sudden becomes not quite so simple. And so, I find it's a really fun exercise to build when trying out a new interactive front-end technology. STEPHANIE: Have you done this with a different implementation or a framework? JOËL: I've done this, not completely, but I've attempted some parts of a D&D character sheet, I think, with Backbone.js with Ember. I may have done an Angular one at some point in original Angular, so Angular 1. I did this with Elm. Somehow, I skipped React. I don't think I did React to build a D&D character sheet. And now I'm kind of moving a little bit back to the backend. How much can we get done just with Turbo? Or do we need to pull in maybe Stimulus? These are all things that are going to be really fun to demonstrate. STEPHANIE: Yeah. Speaking of injecting some whimsy earlier, I think it's kind of like just a little more fun than a regular to-do app, you know, or a blog to show how you can build, you know, something that people kind of understand with a different technology. JOËL: Another really fun thing that I've been toying with this week has been using AI to help me refactor code. And this has been using just sort of a classic chat AI, not a tool like Copilot. And I was dealing with a query that was really slow, and I wanted to restructure it in a different way. And I described to the AI how I wanted it to refactor and explicitly said, "I want this to be the same before and after." And I asked it to do the refactor, and it gave me some pretty disappointing results where it did some, like, a couple of really obvious things that were not that useful. And I was talking to a colleague about how I was really disappointed. I was thinking, well, AI should be able to do something better than this. And this colleague suggested changing the way I was asking for things and specifically asking for a step-by-step and asking it to prove every step using relational algebra, which is the branch of math that deals with everything that underlies relational databases, so the transformations that you would do where you keep everything the same, but you're saying, "Hey, these equations are all equivalent." And it sure did. It gave me a, like, 10-step process with all these, like, symbols and things. My relational algebra is not that strong, and so I couldn't totally follow along. But then I asked it to give me a code example, like, show me the SQL at every step of this transformation and at the end. And, you know, it all kind of looked all right. I've not fully tested the final result it gave me to see if it does what it says on the tin. But I'm cautiously optimistic. I think it looks very similar to something that I came up with on my own. And so, I'm somewhat impressed, at least, like, much better than things were in the beginning with that first round. So, I'm really curious to see where I can take this. STEPHANIE: Yeah, I think that's cool that you were able to prompt it differently and get something more useful. One of the reasons why I personally have been a little bit hesitant to get into the large language models is because I would love to see the AI show its work, essentially, like, tell me a little bit more about how it got from question to answer. And I thought that framing of kind of step-by-step show me code was a really interesting way, even to just, like, get some different results that do the same thing. But you can kind of evaluate that a little bit more on your own rather than just using that first result that it gave you that was like, eh, like, I don't know if this really did anything for me. So, it would be cool, even if you don't end up using, like, the final one, right? If something along the way also is an improvement from what you started with that would be really interesting. JOËL: Honestly, I think you kind of want the same thing if you're chatting with an AI chatbot or having a conversation in Slack with a colleague. They're just like, "Hey, can you help me refactor this?" And then a sort of, like, totally different chunk of code. And it's just like, "Trust me, it works." STEPHANIE: [laughs]. JOËL: And maybe it does. Maybe you plug it into your codebase and run the tests against it, and the tests are still green. And so, you trust that it works, but you don't really understand where it came from. That doesn't always feel good, even when it comes from a human. So, what I've appreciated with colleagues has been when they've given me a step-by-step. Sometimes, they give me the final product. They just say, "Hey. Try this. Does this work?" Plug it in to the test. It does pass. It's green. Great. "Tell me what black magic you did to get to that." And then they give me the step-by-step and it's like, oh, that's so good because not only do I get a better understanding of what happens at every step, but now I'm equipped the next time I run into this problem to apply the same technique to figure it out on my own. STEPHANIE: Yeah. And I liked, also, that relational algebra pro tip, right? It kind of ensures that what you're getting makes sense or is equivalent along the way [laughs]. JOËL: We think, right? I don't know enough relational algebra to check its work. It is quite possible that it is making some subtle mistakes along the way, or, like, making inferences that it shouldn't be. I'm not going to say I trust that. But I think, specifically, when asking for SQL transformations, prompting it to do so using relational algebra in a step-by-step way seemed to be a way to get it to do something more reliably or at least give more interesting results. STEPHANIE: Cool. JOËL: I was interested in trying this out in part because I've been more curious about AI tools recently, and also because we're hoping to do a deeper dive into AI on a Bike Shed episode at some point later, so very much still in the gathering information phase. But this was a really cool experience. So, having an AI refactor a query for me using relational algebra, definitely something that's new in my world this week. STEPHANIE: Speaking of refactoring and this idea of making improvements to your code and trying to figure out how to get from what you currently have to something new, I have been thinking a lot about how to make code reviews more actionable. And that's because, on my current client project, our team is struggling a little bit with code reviews, especially when you kind of want to give feedback on more of a design change in the code or thinking about some different abstractions. I have found that that is really hard to communicate async and also in a, like, a GitHub code review format where you can really just comment, like, line by line. And I've found that, you know, when someone is leaving feedback, that's like, "I'm having a hard time reading this. And I'm imagining that we could organize the code a bit differently in these three different layers or abstractions," there's a lot of assumptions there, right [laughs]? That your message is being communicated to the author and that they are able to, like, visualize, or have a mental model for what you're explaining as well. And then kind of what I've been seeing in this dynamic is, like, not really knowing what to do with that and to kind of just, like, I don't know where to go from here. So, I guess the next step is just to, like, merge it. Is that something you've experienced before or encountered when it comes to feedback? JOËL: Broader changes are often challenging to explain, especially when they're...sometimes you get so abstract you can just write a quick paragraph. And sometimes it's like, hey, what if we, like, totally change our approach? I've definitely done the thing where I'll just ping someone and say, "Hey, can we talk about this synchronously? Can we get on a call and have a deeper conversation?" How do you tend to approach if you're not going to hop on a call with someone and, like, have a 20 or 30-minute conversation? How do you approach doing that asynchronously on a pull request? Are you the type of person to put, like, a ton of, like, code blocks, like, "Here's what I was thinking. We could instead have this class and this thing"? And, like, pretty soon, it's, like, a page and a half of text. Or do you have another approach that you like to use? STEPHANIE: Yeah. And I think that's where it can get really interesting. Because my process is, I'll usually just start commenting and maybe if I'm seeing some things that can be done differently. If it's not just, like, a really obvious change that I could just use English to describe, I'll add a little suggested change. But I also don't want to just rewrite this person's code [laughs] in a code review. JOËL: That's the challenge, right? STEPHANIE: Yeah. And I've definitely seen that be done before, too. Once I notice I'm at, like, four plus comments, and then they're not just, like, nitpicks about, like, syntax or something like that, that helps me clue into the idea that there is some kind of bigger change that I might be asking of the author. And I don't want to overwhelm them with, like, individual comments that really are trying to convey something more holistic. JOËL: Right. I wonder if having a, like, specialized yet more abstract language is useful for these sorts of things where a whole paragraph in English or, you know, a ton of code examples might be a bit much. If you're able to say something like, "Hey, how would you feel about using a strategy pattern approach here instead of, you know, maybe a template object or some custom thing that we've built here?" that allows us to say a lot in a fairly sort of terse way. And it's the thing that you can leave more generically on the PR instead of, like, individually commenting in a bunch of places. And that can start a broader conversation at more of an architecture level. STEPHANIE: Yes, I really like that. That's a great idea. I would follow that up with, like, I think at the end of the day, there are some conversations that do need to be had synchronously. And so, I like the idea of leaving a comment like that and just kind of giving them resources to learn what a strategy pattern is and then offering support because that's also a way to shorten that feedback loop of trying to communicate an idea. And I like that it's kind of guiding them, but also you're there to add some scaffolding if it ends up being, like, kind of a big ask for them to figure out what to do. JOËL: There's also oftentimes, I think, a tone thing to manage where, especially if there's a difference in seniority or experience between the two people, it can be very easy for something to come across as an ask or a demand rather than a like, "Hey, let's think about some alternatives here." Or, like, "I have some concerns with your implementation. Let's sort of broadly explore some possible alternatives. Maybe a strategy pattern works." But the person reading that who wrote the original code might be, like, receiving that as "Your code is bad. You should have done a strategy pattern instead." And that's not the conversation I want to have, right? I want to have a back-and-forth about, "Hey, what are the trade-offs involved? Do you have a third architecture you'd like to suggest?" And so, that can be a really tricky thing to avoid. STEPHANIE: Yeah, I like that what you're saying also kind of suggested that it's okay if you don't have an idea yet for exactly how it should look like. Maybe you just are like, oh, like, I'm having a hard time understanding this, but I don't think just leaving it at that gives the author a lot to go on. I think there's something to it about maybe the action part of actionable is just like, "Can you talk about it with me?" Or "Could you explain what you're trying to do here?" Or, you know, leave a comment about what this method is doing. There's a lot of ways, I think, that you can reach some amount of improvement, even if it doesn't end up being, like, the ideal code that you would write. JOËL: Yes. There's also maybe a distinction in making it actionable by giving someone some code and saying, "Hey, you should copy-paste this code and make that..." or, you know, use a GitHub suggested code or something, which works on the small. And in the big, you can give some maybe examples and say, "Hey, what if you refactored in this way?" But sometimes, you could even step back and let them do that work and say, "Hey, I have some concerns with the current architecture. It's not flexible in the ways that we need to be flexible. Here's my understanding of the requirements. And here's sort of how I see maybe this architecture not working with that. Let's think of some different ways we could approach this problem." And oftentimes, it's nice to give at least one or two different ideas to help start that. But it can be okay to just ask the person, "Hey, can you come up with some alternate implementations that would fulfill these sets of requirements?" STEPHANIE: Yeah, I like that. And I can even see, like, maybe you do that work, and you don't end up pursuing it completely in addressing that feedback. But even asking someone to do the exercise itself, I think, can then spark new ideas and maybe other improvements. In general, I like to think about...I'm a little hesitant to use this metaphor because I'm not actually giving code, like, letter grades when I review them, but the idea that, like, not all code has to get, like, an A [chuckles], but maybe getting it, like, from one letter grade up to, like, half a letter grade, like, higher, that is valuable, even if it's not always practical to go through multiple rounds of code review. And I think just making it actionable enough to be a little bit better, like, that is, in my opinion, the sweet spot. JOËL: That's true. The sort of over-giving feedback to someone to try to get code perfect, rather than just saying, "Hey, can we make it slightly better?" And, you know, there are probably some minimum standards you need to hit. But at some point, it's a trade-off of like, how much time do we need to put polishing this versus shipping something? STEPHANIE: Yeah, and I think that it is cumulative over time, right? That's how people learn. Yeah, it's like one of the biggest opportunities for developers to level up is from that feedback. And that's why I think it's important that it's actionable because, you know, and you put the time into, like, giving that review, and it's not just to make sure the code works, but it's also, like, one of the touch points for collaboration. JOËL: So, if you had to summarize what makes code review comments actionable, do you have, like, top three tips that make a comment really actionable as opposed to something that's not helpful? Or maybe that's more of the journey that you're on, and you've not distilled it down to three pithy tips that you can put in a listicle. STEPHANIE: Honestly, I think it does kind of just distill down to one, which is for every comment, you should have an idea of what you would like the author to do about it. And it's okay if it's nothing, but then tell them that it's nothing. You could just be expressing, "I thought this was kind of weird, [laughs]" or "This is not my favorite thing, but it's okay." JOËL: And it can be okay for the thing you want the author to do. It doesn't have to be code. It could be a conversation. STEPHANIE: Yeah, exactly. It could be a conversation. It could be asking for information, too, right? Like, "Did you consider alternatives, and could you share them with me?" But that request portion, I think is really important because, yeah, I think there's so much miscommunication that can happen along the way. So, definitely still trying to figure out how to best support that kind of code review culture on my team. JOËL: This week's episode has been really fun because it's just been a combination of a lot of things that are new in our world, things that we've been trying, things that we've been learning. And kind of in an almost, like, a meta sense, one of the things I've been digging into is combinatorics, the branch of math that looks at how things combine and particularly how it works with combining a bunch of ActiveRecord query fragments where there's potential branching, so things like doing a union of two sort of sub queries or doing an or where you're combining two different where queries and trying to figure out what are the different paths through that. STEPHANIE: Wow, what a great way to combine what we were talking about, Joël [laughs]. Did you apply combinatorics to this podcast episode [laughs]? JOËL: Somehow, topics multiply with each other, something, something. STEPHANIE: Yeah, that makes sense to me [laughs]. Okay. Will you tell me more about what you've been using it for in your queries? JOËL: So, one thing I'm trying to do is because I've got these different branching paths through a query, I want to see sort of all the different ways because these are defined as ActiveRecord scopes, and I'm chaining them together. And it looks linear because I'm calling scope1 dot scope2 dot scope3. But each of those have branches inside of them. And so, there's all these different ways that data could get used or not. And one way that I figured out, like, what are the different paths here, was actually drawing out a matrix, just putting together a table. In this case, I had two scopes, each of which had a two-way branch inside, and so I made a two by two matrix. And that gave me all of the combinations of, oh, if you go down one branch in one scope and down another branch in the other scope. And what I went through is then I went in in each square and filled in how many records I would expect to get back from the query from some basic set that I was working on in each of these combinations. And one thing that was really interesting is that some of those combinations were sort of mutually exclusive, where a scope further down the line was filtering on the same field as an earlier one and would overwrite it or not overwrite it, but the two would then sort of you can't have both of those things be true at the same time. So, I'm looking for something that has a particular manager ID, and then I'm looking for something that has a particular different manager ID. And the way Rails combines these, if you just change scopes with where, is to and them together. There are no records that have both manager ID 1 and manager ID 2. You can only have one manager ID. And so, as I'm filling out my matrix, there's some sections I can just zero out and be like, wait, this will always return zero record. And then I can start focusing on the parts that are not zeroed out. So, I've got two or three squares. What's special about those? And that helped me really understand what the combination of these multiple query fragments together were actually trying to do as a holistic whole. STEPHANIE: Wow, yeah, that is really interesting because I hear you when you say it looks linear. And it would be really surprising to me for there to be branching paths. Like, that's not really what I think about when I think about SQL. But that makes a lot of sense that it could get so complicated that it's just impossible to get a certain kind of result. Like, what's going to be the outcome of applying combinatorics to this? Is there a refactoring opportunity, or is it really just to even understand what's going on? JOËL: So, this was a refactoring that I was trying to do, but I didn't really understand the underlying behavior of the chain of scopes. I just knew that they were doing some complex things that were inefficient from a SQL perspective. And so, I was looking at ways to refactor, but I also wanted to get a sense of what is this actually trying to do other than just chaining a bunch of random bits of code together? So, the matrix really helped for that. The other way that I used it was to write some tests because this query I was trying to refactor, this chain of scopes, was untested. And I wanted to write tests that were very thorough because I wanted to make sure that my refactor didn't break any edge cases. And I'm, you know, writing a few tests. Okay, well, here's a record that I definitely want to get returned by this query, and maybe here are a couple of records I don't want to get returned. And the more I was, like, going into this and trying to write test cases, the more I was finding more edge cases that I didn't want to and, oh, but what about this? And what about the combination of these things? And it got to the point where it was just messing with my mind. I was, like, confusing myself and really struggling to write tests that would do anything useful. STEPHANIE: Wow. Yeah. Honestly, I have already started to become a little bit suspicious of complex scopes, and this further pushes me in that direction [laughs] because yeah, once you start to...like, the benefit of them is that you can chain them, but it really hides a lot of the underlying behavior. So, you can easily just turn yourself around or, like, go, you know, kind of end up [laughs] in a little bit of a bind. JOËL: Definitely, especially once it grows a little bit harder to hold in your head. And I don't know exactly where that level is for me. But in this particular situation, I identified, I think, five different dimensions that would impact the results of this query. And then each dimension had maybe three or four different values that we might care about. And, eventually, I just took the time to write this out. So, I created five arrays and then just said, "Hey, here are the different managers that we care about. Here are the different project types we care about. Here are the different..." and we had, like, five of these, and each array had three or four elements in it. And then, in a series of nested loops, I iterated through all of these arrays and at the innermost loop, created the data that I wanted that matched that particular set of values. Now, we're often told you should not be doing things in nested loops because you end up sort of multiplying all of these together, but, in this case, this is actually what I wanted to do. You know, it turns out that I had a hundred-ish records I had to create to sort of create a data set that would be all the possible edge cases I might want to filter on. And creating them all by hand with all of the different variations was going to be too much. And so, I ended up doing this with arrays and nested loops. And it got me the data that I needed. And it gave me then the confidence to know that my refactor did indeed work the way I was expecting. STEPHANIE: Wow. That's truly hero's work [chuckles]. I'm, like, very excited because it sounds like that's a huge opportunity for some performance improvements as well. JOËL: For the underlying code, yes. The test might be a little bit slow because I'm creating a hundred records in the database. And you might say, "Oh, do you really need to do that? Can you maybe collapse some of these cases?" In this particular case, I really wanted to have high confidence that the refactor was not changing anything. And so, I was okay creating a hundred records over a series of nested iterations. That was a price I was willing to pay. The refactored query, it turns out, I was able to write it in a way that was significantly faster. STEPHANIE: Yeah, that's what I suspected. JOËL: So, I had to rewrite it in a way that didn't take advantage of all the change scopes. I had to just sort of write something custom from scratch, which is often the case, right? Performance and reusability sometimes fight against each other, and it's a trade-off. So, I'm not reusing the scopes. I had to write something from scratch, but it's multiple hundreds of times faster. STEPHANIE: Wow. Yeah. That seems worth it for a slow test [laughs] for the user experience to be a lot better, especially when you just reach that level of complexity. And it's a really awesome strategy that you applied to figure that out. I think it's a very unique one [laughs]. That's for sure. JOËL: I've had an interest in sort of analytical tools to help me understand domain models, to help understand problems, to help understand code that I'm working with for a while now, and I think an understanding of combinatorics fits into that. And then, particular tools within that, such as drawing things out in a table, in a two by two matrix, or an end-by-end matrix to get something visual, that's a great tool for debugging or understanding a problem. Thinking of problems as data that exists in multiple dimensions and then asking about the cardinality of that set it's the kind of analysis I did a lot when I was modeling using algebraic data types in Elm. But now I've sort of taken some of the tools and analysis I use from that world into thinking about things like SQL records, things like dealing with data in Ruby. And I'm able to bring those tools and that way of thinking to help me solve some problems that I might struggle to solve otherwise. For any of our listeners who this, like, kind of piques their interest, combinatorics falls under a broader umbrella of mathematics called discrete math. And within that, there's a lot that I think is really useful, a lot of tools and techniques that we can apply to our day-to-day programming. We have a Bike Shed episode where we talked about is discrete math relevant to day-to-day programmers and what are the ways it's so? We'll link that in the show notes. I also gave a talk at RailsConf last year diving into that titled: The Math Every Programmer Needs. So, if you're looking for something that's accessible to someone who's not done a math degree, those are two great jumping-off points. STEPHANIE: Yeah. And then, maybe you'll start drawing out arrays and applying combinatorics to figure out your performance problems. JOËL: On that note, shall we wrap up? STEPHANIE: Let's wrap up. Show notes for this episode can be found at bikeshed.fm. JOËL: This show has been produced and edited by Mandy Moore. STEPHANIE: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review in iTunes. It really helps other folks find the show. JOËL: If you have any feedback for this or any of our other episodes, you can reach us @_bikeshed, or you can reach me @joelquen on Twitter. STEPHANIE: Or reach both of us at hosts@bikeshed.fm via email. JOËL: Thanks so much for listening to The Bike Shed, and we'll see you next week. ALL: Byeeeeeeeeee!!!!!!!! AD: Did you know thoughtbot has a referral program? If you introduce us to someone looking for a design or development partner, we will compensate you if they decide to work with us. More info on our website at: tbot.io/referral. Or you can email us at: referrals@thoughtbot.com with any questions.

418: Mental Models For Reduce Functions

Play Episode Listen Later Mar 12, 2024 42:50

Joël talks about his difficulties optimizing queries in ActiveRecord, especially with complex scopes and unions, resulting in slow queries. He emphasizes the importance of optimizing subqueries in unions to boost performance despite challenges such as query duplication and difficulty reusing scopes. Stephanie discusses upgrading a client's app to Rails 7, highlighting the importance of patience, detailed attention, and the benefits of collaborative work with a fellow developer. The conversation shifts to Ruby's reduce method (inject), exploring its complexity and various mental models to understand it. They discuss when it's preferable to use reduce over other methods like each, map, or loops and the importance of understanding the underlying operation you wish to apply to two elements before scaling up with reduce. The episode also touches on monoids and how they relate to reduce, suggesting that a deep understanding of functional programming concepts can help simplify reduce expressions. Rails 7 EXPLAIN ANALYZE (https://www.bigbinary.com/blog/rails-7-1-adds-options-to-activerecord-relation-explain) Blocks, symbol to proc, and symbols arguments for reduce (https://thoughtbot.com/blog/blocks-procs-and-enumerable) Ruby tally (https://medium.com/@baweaver/ruby-2-7-enumerable-tally-a706a5fb11ea) Performance considerations for reduce in JavaScript (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/Reduce#when_to_not_use_reduce) Persistant data structures (https://www.youtube.com/watch?v=gTClDj9Zl1g) Avoid passing a block to map and reduce (https://thoughtbot.com/blog/avoid-putting-logic-in-map-blocks) Functional Programming with Bananas, Lenses, Envelopes and Barbed Wire (https://ris.utwente.nl/ws/portalfiles/portal/6142049/meijer91functional.pdf) monoids (https://blog.ploeh.dk/2017/10/06/monoids/) iteration anti-patterns (https://thoughtbot.com/blog/iteration-as-an-anti-pattern) Joël's talk on “constructor replacement” (https://www.youtube.com/watch?v=dSMB3rsufC8) Transcript: STEPHANIE: Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Stephanie Minn. JOËL: And I'm Joël Quenneville. And together, we're here to share a bit of what we've learned along the way. STEPHANIE: So, Joël, what's new in your world? JOËL: I've been doing a bunch of fiddling with query optimization this week, and I've sort of run across an interesting...but maybe it's more of an interesting realization because it's interesting in the sort of annoying way. And that is that, using ActiveRecord scopes with certain more complex query pieces, particularly unions, can lead to queries that are really slow, and you have to rewrite them differently in a way that's not reusable in order to make them fast. In particular, if you have sort of two other scopes that involve joins and then you combine them using a union, you're unioning two sort of joins. Later on, you want to change some other scope that does some wares or something like that. That can end up being really expensive, particularly if some of the underlying tables being joined are huge. Because your database, in my case, Postgres, will pull a lot of this data into the giant sort of in-memory table as it's, like, building all these things together and to filter them out. And it doesn't have the ability to optimize the way it would on a more traditional relation. A solution to this is to make sure that the sort of subqueries that are getting unioned are optimized individually. And that can mean moving conditions that are outside the union inside. So, if I'm chaining, I don't know, where active is true on the outer query; on the union itself, I might need to move that inside each of the subqueries. So, now, in the two or three subqueries that I'm unioning, each of them needs to have a 'where active true' chained on it. STEPHANIE: Interesting. I have heard this about using ActiveRecord scopes before, that if the scopes are quite complex, chaining them might not lead to the most performant query. That is interesting. By optimizing the subqueries, did you kind of change the meaning of them? Was that something that ended up happening? JOËL: So, the annoying thing is that I have a scope that has the union in it, and it does some things sort of on its own. And it's used in some places. There are also other places that will try to take that scope that has the union on it, chain some other scopes that do other joins and some more filters, and that is horribly inefficient. So, I need to sort of rewrite the sort of subqueries that get union to include all these new conditions that only happen in this one use case and not in the, like, three or four others that rely on that union. So, now I end up with some, like, awkward query duplication in different call sites that I'm not super comfortable about, but, unfortunately, I've not found a good way to make this sort of nicely reusable. Because when you want to chain sort of more things onto the union, you need to shove them in, and there's no clean way of doing that. STEPHANIE: Yeah. I think another way I've seen this resolved is just writing it in SQL if it's really complex and it becoming just a bespoke query. We're no longer trying to use the scope that could be reusable. JOËL: Right. Right. In this case, I guess, I'm, like, halfway in between in that I'm using the ActiveRecord DSL, but I am not reusing scopes and things. So, I sort of have the, I don't know, naive union implementation that can be fine in all of the simpler use cases that are using it. And then the query that tries to combine the union with some other fancy stuff it just gets its own separate implementation different than the others that it has optimized. So, there are sort of two separate paths, two separate implementations. I did not drop down to writing raw SQL because I could use the ActiveRecord DSL. So, that's what I've been working with. What's new in your world this week? STEPHANIE: So, a couple of weeks ago, I think, I mentioned that I was working on a Rails 7 upgrade, and we have gotten it out the door. So, now the client application I'm working on is on Rails 7, which is exciting for the team. But in an effort to make the upgrade as incremental as possible, we did, like, back out of a few of the new application config changes that would have led us down a path of more work. And now we're kind of following up a little bit to try to turn some of those configs on to enable them. And it was very exciting to kind of, like, officially be on Rails 7. But I do feel like we tried to go for, like, the minimal amount of work possible in that initial big change. And now we're having to kind of backfill a little bit on some of the work that was a little bit more like, oh, I'm not really sure, like, how big this will end up being. And it's been really interesting work, I think, because it requires, like, two different mindsets. Like, one of them is being really patient and focused on tedious work. Like, okay, what happens when we enable this config option? Like, what changes? What errors do we see? And then having to turn it back off and then go in and fix them. But then another, I think, like, headspace that we have to be in is making decisions about what to do when we come to a crossroads around, like, okay, now that we are starting to see all the changes that are coming about from enabling this config, is this even what we want to do? And it can be really hard to switch between those two modes of thinking. JOËL: Yeah. How do you try to balance between the two? STEPHANIE: So, I luckily have been pairing with another dev, and I've actually found that to be really effective because he has, I guess, just, like, a little bit more of that patience to do the more tedious, mundane [laughs] aspects of, like, driving the code changes. And I have been riding along. But then I can sense, like, once he gets to the point of like, "Oh, I'm not sure if we should keep going down this road," I can step in a little bit more and be like, "Okay, like, you know, I've seen us do this, like, five times now, and maybe we don't want to do that." Or maybe being like, "Okay, we don't have a really clear answer, but, like, who can we talk to to find out a little bit more or get their input?" And that's been working really well for me because I've not had a lot of energy to do more of that, like, more manual or tedious labor [chuckles] that comes with working on that low level of stuff. So yeah, I've just been pleasantly surprised by how well we are aligning our superpowers. JOËL: To use some classic business speech, how does it feel to be in the future on Rails 7? STEPHANIE: Well, we're not quite up, you know, up to modern days yet, but it does feel like we're getting close. And, like, I think now we're starting to entertain the idea of, like, hmm, like, could we be even on main? I don't think it's really going to happen, but it feels a little bit more possible. And, in general, like, the team thinks that that could be, like, really exciting. Or it's easier, I think, once you're a little bit more on top of it. Like, the worst is when you get quite behind, and you end up just feeling like you're constantly playing catch up. It just feels a little bit more manageable now, which is good. JOËL: I learned this week a fun fact about Rails 7.1, in particular, which is that the analyze method on ActiveRecord queries, which allowed you to sort of get SQL EXPLAIN statements, now has the ability to pass in a couple of extra parameters. So, there are symbols, and you can pass in things like analyze or verbose, which allows you to get sort of more data out of your EXPLAIN query, which can be quite nice when you're debugging for performance. So, if you're in the future and you're on Rails 7.1 and you want sort of the in-depth query plans, you don't need to copy the SQL into a Postgres console to get access to the sort of fully developed EXPLAIN plan. You can now do it by passing arguments to EXPLAIN, which I'm very happy for. STEPHANIE: That's really nice. JOËL: So, we've mentioned before that we have a developers' channel on Slack here at thoughtbot, and there's all sorts of fun conversations that happen there. And there was one recently that really got me interested, where people were talking about Ruby's reduce method, also known as inject. And it's one of those methods that's kind of complicated, or it can be really confusing. And there was a whole thread where people were talking about different mental models that they had around the reduce method and how they sort of understand the way it works. And I'd be curious to sort of dig into each other's mental models of that today. To kick us off, like, how comfortable do you feel with Ruby's reduce method? And do you have any mental models to kind of hold it in your head? STEPHANIE: Yeah, I think reduce is so hard to wrap your head around, or it might be one of the most difficult, I guess, like, functions a new developer encounters, you know, in trying to understand the tools available to them. I always have to look up the order of the arguments [laughs] for reduce. JOËL: Every time. STEPHANIE: Yep. But I feel like I finally have a more intuitive sense of when to use it. And my mental model for it is collapsing a collection into one value, and, actually, that's why I prefer calling it reduce rather than the inject alias because reduce kind of signals to me this idea of going from many things to one canonical thing, I suppose. JOËL: Yeah, that's a very common use case for reducing, and I guess the name itself, reducing, kind of has almost that connotation. You're taking many things, and you're going to reduce that down to a single thing. STEPHANIE: What was really interesting to me about that conversation was that some people kind of had the opposite mental model where it made a bit more sense for them to think about injecting and, specifically, like, the idea of the accumulator being injected with values, I suppose. And I kind of realized that, in some ways, they're kind of antonyms [chuckles] a little bit because if you're focused on the accumulator, you're kind of thinking about something getting bigger. And that kind of blew my mind a little bit when I realized that, in some ways, they can be considered opposites. JOËL: That's really fascinating. It is really interesting, I think, the way that we can take the name of a method and then almost, like, tell ourselves a story about what it does that then becomes our way of remembering how this method works. And the story we tell for the same method name, or in this case, maybe there's a few different method names that are aliases, can be different from person to person. I know I tend to think of inject less in terms of injecting things into the accumulator and more in terms of injecting some kind of operator between every item in the collection. So, if we have an array of numbers and we're injecting plus, in my mind, I'm like, oh yeah, in between each of the numbers in the collection, just inject a little plus sign, and then do the math. We're summing all the items in the collection. STEPHANIE: Does that still hold up when the operator becomes a little more complex than just, you know, like, a mathematical operator, like, say, a function? JOËL: Well, when you start passing a block and doing custom logic, no, that mental model kind of falls apart. In order for it to work, it also has to be something that you can visualize as some form of infix operator, something that goes between two values rather than, like, a method name, which is typically in prefix position. I do want to get at this idea, though: the difference between sort of the block version versus passing. There are ways where you can just do a symbol, and that will call a method on each of the items. Because I have a bit of a hot take when it comes to writing reduce blocks or inject blocks that are more accessible, easier to understand. And that is, generally, that you shouldn't, or more specifically, you should not have a big block body. In general, you should be either using the symbol version or just calling a method within the block, and it's a one-liner. Which means that if you have some complex behavior, you need to find a way to move that out of this sort of collection operation and into instance methods on the objects being iterated. STEPHANIE: Hmm, interesting. By one-liner do you mean passing the name of the method as a proc or actually, like, having your block that then calls the method? Because I can see it becoming even simpler if you have already extracted a method. JOËL: Yeah, if you can do symbol to proc, that's amazing, or even if you can use just the straight-up symbol way of invoking reduce or inject. That typically means you have to start thinking about the types of objects that you are working with and what methods can be moved onto them. And sometimes, if you're working with hashes or something like that that don't have domain methods for what you want, that gets really awkward. And so, then maybe that becomes maybe a hint that you've got some primitive obsession happening and that this hash that sort of wants a domain object or some kind of domain method probably should be extracted to its own object. STEPHANIE: I'll do you with another kind of spicy take. I think, in that case, maybe you don't want a reduce at all. If you're starting to find that...well, okay, I think it maybe could depend because there could be some very, like, domain-specific logic. But I have seen reduce end up being used to transform the structure of the initial collection when either a different higher-order function can be used or, I don't know, maybe you're just better off writing it with a regular loop [laughs]. It could be clearer that way. JOËL: Well, that's really interesting because...so, you mentioned the idea that we could use a different higher-order function, and, you know, higher-order function is that fancy term, just a method that accepts another method as an argument. In Ruby, that just means your method accepts a block. Reduce can be used to implement pretty much the entirety of enumerable. Under the hood, enumerable is built in terms of each. You could implement it in terms of reduce. So, sometimes it's easy to re-implement one of the enumerable methods yourself, accidentally, using reduce. So, you've written this, like, complex reduce block, and then somebody in review comes and looks at it and is like, "Hey, you realize that's just map. You've just recreated map. What if we used map here?" STEPHANIE: Yeah. Another one I've seen a lot in JavaScript land where there are, you know, fewer utility functions is what we now have in Ruby, tally. I feel like that was a common one I would see a lot when you're trying to count instances of something, and I've seen it done with reduce. I've seen it done with a for each. And, you know, I'm sure there are libraries that actually provide a tally-like function for you in JS. But I guess that actually makes me feel even more strongly about this idea that reduce is best used for collapsing something as opposed to just, like, transforming a data structure into something else. JOËL: There's an interesting other mental model for reduce that I think is hiding under what we're talking about here, and that is the idea that it is a sort of mid-level abstraction for dealing with collections, as opposed to something like map or select or some of those other enumerable helpers because those can all be implemented in terms of reduce. And so, in many cases, you don't need to write the reduce because the library maintainer has already used reduce or something equivalent to build these higher-level helpers for you. STEPHANIE: Yeah, it's kind of in that weird point between, like, very powerful [chuckles] so that people can start to do some funky things with it, but also sometimes just necessary because it can feel a little bit more concise that way. JOËL: I've done a fair amount of functional programming in languages like Elm. And there, if you're building a custom data structure, the sort of lowest-level way you have of looping is doing a recursion, and recursions are messy. And so, what you can do instead as a library developer is say, "You know what, I don't want to be writing recursions for all of these." I don't know; maybe I'm building a tree library. I don't want to write a recursion for every different function that goes over trees if I want to map or filter or whatever. I'm going to write reduce using recursion, and then everything else can be written in terms of reduce. And then, if people want to do custom things, they don't need to recurse over my tree. They can use this reduce function, which allows them to do most of the traversals they want on the tree without needing to touch manual recursion. So, there's almost, like, a low-level, mid-level, high-level in the library design, where, like, lowest level is recursion. Ideally, nobody touches that. Mid-level, you've got reducing that's built out on top of recursion. And then, on top of that, you've got all sorts of other helpers, like mapping, like filtering, things like that. STEPHANIE: Hmm. I'm wondering, do you know of any performance considerations when it comes to using reduce built off a recursion? JOËL: So, one of the things that can be really nice is that writing a recursion yourself is dangerous. It's so easy to, like, accidentally introduce Stack Overflow. You could also write a really inefficient one. So, ideally, what you do is that you write a reduce that is safe and that is fast. And then, everybody else can just use that to not have to worry about the sort of mechanics of traversing the collection. And then, just use this. It already has all of the safety and speed features built in. You do have to be careful, though, because reduce, by nature, traverses the entire collection. And if you want to break out early of something expensive, then reduce might not be the tool for you. STEPHANIE: I was also reading a little bit about how, in JavaScript, a lot of developers like to stick to that idea of a pure function and try to basically copy the entire accumulator for every iteration and creating a new object for that. And that has led to some memory issues as well. As opposed to just mutating the accumulator, having, especially when you, you know, are going through a collection, like, really large, making that copy every single time and creating, yeah [chuckles], just a lot of issues that way. So, that's kind of what prompted that question. JOËL: Yeah, that can vary a lot by language and by data structure. In more functional languages that try to not mutate, they often have this idea of what they call persistent data structures, where you can sort of create copies that have small modifications that don't force you to copy the whole object under the hood. They're just, like, pointers. So, like, hey, we, like, are the same as this other object, but with this extra element added, or something like that. So, if you're growing an array or something like that, you don't end up with 10,000 copies of the array with, like, a new element every time. STEPHANIE: Yeah, that is interesting. And I feel like trying to adopt different paradigms for different tools, you know, is not always as straightforward as some wish it were [laughs]. JOËL: I do want to give a shout-out to an academic paper that is...it is infamously dense. The title of it is Functional Programming with Bananas, Lenses, and Barbed Wire. STEPHANIE: It doesn't sound dense; it sounds fun. Well, I don't about barbed wire. JOËL: It sounds fun, right? STEPHANIE: Yeah, but certainly quirky [laughs]. JOËL: It is incredibly dense. And they've, like, created this custom math notation and all this stuff. But the idea that they pioneered there is really cool, this idea that kind of like I was talking about sort of building libraries in different levels. Their idea is that recursion is generally something that's unsafe and that library and language designers should take care of all of the recursion and instead provide some of these sort of mid-level helper methods to do things. Reducing is one of them, but their proposal is that it's not the only one. There's a whole sort of family of similar methods that are there that would be useful in different use cases. So, reduce allows you to sort of traverse the whole thing. It does not allow you to break out early. It does not allow you to keep sort of track of a sort of extra context element if you want to, like, be traversing a collection but have a sort of look forward, look back, something like that. So, there are other variations that could handle those. There are variations that are the opposite of reduce, where you're, like, inflating, starting from a few parameters and building a collection out of them. So, this whole concept is called recursion schemes, and you can get, like, really deep into some theory there. You'll hear fancy words like catamorphisms and anamorphisms. There's a whole world to explore in that area. But at its core, it's this idea that you can sort of slice up things into this sort of low-level recursion, mid-level helpers, and then, like, kind of userland helpers built on top of that. STEPHANIE: Wow. That is very intense; it sounds like [chuckles]. I'm happy not to ever have to write a recursion ever again, probably [laughs]. Have you ever, as just a web developer in your day-to-day programming, found a really good use case for dropping down to that level? Or are you kind of convinced that, like, you won't really ever need to? JOËL: I think it depends on the paradigm of the language you're working in. In Ruby, I've very rarely needed to write a recursion. In something like Elm, I've had to do that, eh, not infrequently. Again, it depends, like, if I'm doing more library-esque code versus more application code. If I'm writing application code and I'm using an existing, let's say, tree library, then I typically don't need to write a recursion because they've already written traversals for me. If I'm making my own and I have made my own tree libraries, then yes, I'm writing recursions myself and building those traversals so that other people don't have to. STEPHANIE: Yeah, that makes sense. I'd much rather someone who has read that paper [laughs] write some traversal methods for me. JOËL: And, you know, for those who are curious about it, we will put a link to this paper in the description. So, we've talked about a sort of very academic mental model way of thinking about reducing. I want to shift gears and talk about one that I have found is incredibly practical, and that is the idea that reduce is a way to scale an operation that works on two objects to an operation that works on sort of an unlimited number of objects. To make it more concrete, take something like addition. I can add two numbers. The plus operator allows me to take one number, add another, get a sum. But what if I want to not just add two numbers? I want to add an arbitrary number of numbers together. Reduce allows me to take that plus operator and then just scale it up to as many numbers as I want. I can just plug that into, you know, I have an array of numbers, and I just call dot reduce plus operator, and, boom, it can now scale to as many numbers as I want, and I can sum the whole thing. STEPHANIE: That dovetails quite nicely with your take earlier about how you shouldn't pass a block to reduce. You should extract that into a method. Don't you think? JOËL: I think it does, yes. And then maybe it's, like, sort of two sides of a coin because I think what this leads to is an approach that I really like for reducing because sometimes, you know, here, I'm starting with addition. I'm like, oh, I have addition. Now, I want to scale it up. How do I do that? I can use reduce. Oftentimes, I'm faced with sort of the opposite problem. I'm like, oh, I need to add all these numbers together. How do I do that? I'm like, probably with a reduce. But then I start writing the block, and, like, I get way too into my head about the accumulator and what's going to happen. So, my strategy for writing reduce expressions is to, instead of trying to figure out how to, like, do the whole thing together, first ask myself, how do I want to combine any two elements that are in the array? So, I've got an array of numbers, and I want to sum them all. What is the thing I need to do to combine just two of those? Forget the array. Figure that out. And then, once I have that figured out, maybe it's an existing method like plus. Maybe it's a method I need to define on it if it's a custom object. Maybe it's a method that I write somewhere. Then, once I have that, I can say, okay, I can do it for two items. Now, I'm going to scale it up to work for the whole array, and I can plug it into reduce. And, at that point, the work is already basically done, so I don't end up with a really complex block. I don't end up, like, almost ending in, like, a recursive infinite loop in my head because I do that. STEPHANIE: [laughs]. JOËL: So, that approach of saying, start by figuring out what is the operation you want to do to combine two elements, and then use reduce as a way to scale that to your whole array is a way that I've used to keep things simple in my mind. STEPHANIE: Yeah, I like that a lot as a supplement to the model I shared earlier because, for me, when I think about reducing as, like, collapsing into a value, you kind of are just like, well, okay, I start with the collection, and then somehow I get to my single value. But the challenge is figuring out how that happens [laughs], like, the magic that happens in between that. And I think another alias that we haven't mentioned yet for reduce that is used in a lot of other languages is fold. And I actually like that one a lot, and I think it relates to your mental model. Because when I think about folding, I'm picturing folding up a paper like an accordion. And you have to figure out, like, what is the first fold that I can make? And just repeating that over and over to get to your little stack of accordion paper [laughs]. And if you can figure out just that first step, then you pretty much, like, have the recipe for getting from your initial input to, like, your desired output. JOËL: Yeah. I think fold is interesting in that some languages will make a distinction between fold and reduce. They will have both. And typically, fold will require you to pass an initial value, like a starting accumulator, to start it off. Whereas reduce will sort of assume that your array can use the first element of the array as the first accumulator. STEPHANIE: Oh, I just came up with another visual metaphor for this, which is, like, folding butter into croissant pastry when the butter is your initial value [laughs]. JOËL: And then the crust is, I guess, the elements in the array. STEPHANIE: Yeah. Yeah. And then you get a croissant out of it [laughs]. Don't ask me how it gets to a perfectly baked, flaky, beautiful croissant, but somehow that happens [laughs]. JOËL: So, there's an interesting sort of subtlety here that I think happens because there are sort of two slightly different ways that you can interact with a reduce. Sometimes, your accumulator is of the same type as the elements in your array. So, you're summing an array of numbers, and your accumulator is the sum, but each of the elements in the array are also numbers. So, it's numbers all the way through. And sometimes, your accumulator has a different type than the items in the array. So, maybe you have an array of words, and you want to get the sum of all of the characters and all the words. And so, now your accumulator is a number, but each of the items in the array are strings. STEPHANIE: Yeah, that's an interesting distinction because I think that's where you start to see the complex blocks being passed and reduced. JOËL: The complex blocks, definitely; I think they tend to show up when your accumulator has a different type than the individual items. So, maybe that's, like, a slightly more complicated use case. Oftentimes, too, the accumulator ends up being some, like, more complex, like, hash or something that maybe would really benefit from being a custom object. STEPHANIE: I've never done that before, but I can see why that would be really useful. Do you have an example of when you used a custom object as the accumulator? JOËL: So, I've done it for situations where I'm working with objects that are doing tally-like operations, but I'm not doing just a generic tally. There's some domain-specific stuff happening. So, it's some sort of aggregate counter on multiple dimensions that you can use, and that can get really ugly. And you can either do it with a reduce or you can have some sort of, like, initial version of the hash outside and do an each and mutate the hash and stuff like that. All of these tend to be a little bit ugly. So, in those situations, I've often created some sort of custom object that has some instance methods that allow to sort of easily add new elements to it. STEPHANIE: That's really interesting because now I'm starting to think, what if the elements in the collection were also a custom object? [chuckles] And then things could, I feel like, could be really powerful [laughs]. JOËL: There's often a lot of value, right? Because if the items in the collection are also a custom object, you can then have methods on them. And then, again, the sort of complexity of the reduce can sort of, like, fade away because it doesn't own any of the logic. All it does is saying, hey, there's a thing you can do to combine two items. Let's scale it up to work on a collection of items. And now you've sort of, like, really simplified what logic is actually owned inside the reduce. I do want to shout out for those listeners who are theory nerds and want to dig into this. When you have a reduce, and you've got an operation where all the values are of the same type, including the accumulator, typically, what you've got here is some form of monoid. It may be a semigroup. So, if you want to dig into some theory, those are the words to Google and to go a deep dive on. The main thing about monoids, in particular, is that monoids are any objects that have both a sort of a base case, a sort of empty version of themselves, and they have some sort of combining method that allows you to combine two values of that type. If your object has these things and follows...there's a few rules that have to be true. You have a monoid. And they can then be sort of guaranteed to be folded nicely because you can plug in their base case as your initial accumulator. And you can plug in their combining method as just the value of the block, and everything else just falls into place. A classic here is addition for numbers. So, if you want to add two numbers, your combining operator is a plus. And your sort of empty value is a zero. So, you would say, reduce initial value is zero, array of numbers. And your block is just plus, and it won't sum all of the numbers. You could do something similar with strings, where you can combine strings together with plus, and, you know, your empty string is your base case. So, now you're doing sort of string concatenation over arbitrary number of strings. Turns out there's a lot of operations that fall into that, and you can even define some of those on your custom object. So, you're like, oh, I've got a custom object. Maybe I want some way of, like, combining two of them together. You might be heading in the direction of doing something that is monoidal, and if so, that's a really good hint to know that it can sort of, like, just drop into place with a fold or a reduce and that that is a tool that you have available to you. STEPHANIE: Yeah, well, I think my eyes, like, widened a little bit when you first dropped the term monoid [laughs]. I do want to spend the last bit of our time talking about when not to use reduce, and, you know, we did talk a lot about recursion. But when do you think a regular old loop will just be enough? JOËL: So, you're suggesting when would you want to use something like an each rather than a reduce? STEPHANIE: Yeah. In my mind, you know, you did offer, like, a lot of ways to make reduce simpler, a lot of strategies to end up with some really nice-looking syntax [chuckles], I think. But, oftentimes, I think it can be equally as clear storing your accumulator outside of the iteration and that, like, is enough for me to understand. And reduce takes a little bit of extra overhead to figure out what I'm looking at. Do you have any thoughts about when you would prefer to do that? Or do you think that you would usually reach for something else? JOËL: Personally, I generally don't like the pattern of using each to iterate over a collection and then mutate some external accumulator. That, to me, is a bit of a code smell. It's a sign that each is not quite powerful enough to do the thing that I want to do and that I'm probably needing some sort of more specialized form of iteration. Sometimes, that's reduce. Oftentimes, because each can suffer from the same problem you mentioned from reduce, where it's like, oh, you're doing this thing where you mutate an external accumulator. Turns out what you're really doing is just map. So, use map or use select or, you know, some of the other built-in iterators from the enumerable library. There's a blog post on the thoughtbot blog that I continually link to people. And when I see the pattern of, like, mutating an external variable with each, yeah, I tend to see that as a bit of a code smell. I don't know that I would never do it, but whenever I see that, it's a sign to me to, like, pause and be like, wait a minute, is there a better way to do this? STEPHANIE: Yeah, that's fair. I like the idea that, like, if there's already a method available to you that is more specific to go with that. But I also think that sometimes I'd rather, like, come across that pattern of mutating a variable outside of the iteration over, like, someone trying to do something clever with the reduce. JOËL: Yeah, I guess reduce, especially if it's got, like, a giant block and you've got then, like, things in there that break or call next to skip iterations and things like that, that gets really mind-bending really quickly. I think a case where I might consider using an each over a reduce, and that's maybe generally when I tend to use each, is when I'm doing side effects. If I'm using a reduce, it's because I care about the accumulated value at the end. If I'm using each, it's typically because I am trying to do some amount of side effects. STEPHANIE: Yeah, that's a really good call out. I had that written down in my notes, and I'm glad you brought it up because I've seen them get conflated a little bit, and perhaps maybe that's the source of the pain that I'm talking about. But I really like that heuristic of reduce as, you know, you're caring about the output, as opposed to what's going on inside. Like, you don't want any unexpected behavior. JOËL: And I think that applies to something like map as well. My sort of heuristic is, if I'm doing side effects, I want each. If I want transformed values that are sort of one-to-one in the collection, I want map. If I want a single sort of aggregate value, then I want reduce. STEPHANIE: I think that's the cool thing about mixing paradigms sometimes, where all the strategies you talked about in terms of, you know, using custom, like, objects for your accumulator, or the elements in your collection, like, that's something that we get because, you know, we're using an object-oriented language like Ruby. But then, like, you also are kind of bringing the functional programming lens to, like, when you would use reduce in the first place. And yeah, I am just really excited now [chuckles] to start looking for some places I can use reduce after this conversation and see what comes out of it. JOËL: I think I went on a bit of an interesting journey where, as a newer programmer, reduce was just, like, really intense. And I struggled to understand it. And I was like, ban it from code. I don't want to ever see it. And then, I got into functional programming. I was like, I'm going to do reduce everywhere. And, honestly, it was kind of messy. And then I, like, went really deep on a lot of functional theory, and I think understood some things that then I was able to take back to my code and actually write reduce expressions that are much simpler so that now my heuristic is like, I love reduce; I want to use it, but I want as little as possible in the reduce itself. And because I understand some of these other concepts, I have the ability to know what things can be extracted in a way that will feel very natural, in a way that myself from five years ago would have just been like, oh, I don't know. I've got this, you know, 30-line reduce expression that I know is complicated, but I don't know how to improve. And so, a little bit of the underlying theory, I don't think it's necessary to understand these simplified reduces, but as an author who's writing them, I think it helps me write reduces that are simpler. So, that's been my journey using reduce. STEPHANIE: Yeah. Well, thanks for sharing. And I'm really excited. I hope our listeners have learned some new things about reduce and can look at it from a different light. JOËL: There are so many different perspectives. And I think we keep discovering new mental models as we talk to different people. It's like, oh, this particular perspective. And there's one that we didn't really dig into but that I think makes more sense in a functional world that's around sort of deconstructing a structure and then rebuilding it with different components. The shorthand name of this mental model, which is a fairly common one, is constructor replacement. For anyone who's interested in digging into that, we'll link it in the show notes. I gave a talk at an Elm meetup where I sort of dug into some of that theory, which is really interesting and kind of mind-blowing. Not as relevant, I think, for Rubyists, but if you're in a language that particularly allows you to build custom structures out of recursive types or what are sometimes called algebraic data types, or tagged unions, or discriminated unions, this thing goes by a bajillion names, that is a really interesting other mental model to look at. And, again, I don't think the list that we've covered today is exhaustive. You know, I would love it for any of our listeners; if you have your own mental models for how to think about folding, injecting, reducing, send them in: hosts@bikeshed.fm. We'd love to hear them. STEPHANIE: And on that note, shall we wrap up? JOËL: Let's wrap up. STEPHANIE: Show notes for this episode can be found at bikeshed.fm. JOËL: This show has been produced and edited by Mandy Moore. STEPHANIE: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review in iTunes. It really helps other folks find the show. JOËL: If you have any feedback for this or any of our other episodes, you can reach us @_bikeshed, or you can reach me @joelquen on Twitter. STEPHANIE: Or reach both of us at hosts@bikeshed.fm via email. JOËL: Thanks so much for listening to The Bike Shed, and we'll see you next week. ALL: Byeeeeeeee!!!!!!! AD: Did you know thoughtbot has a referral program? If you introduce us to someone looking for a design or development partner, we will compensate you if they decide to work with us. More info on our website at: tbot.io/referral. Or you can email us at referrals@thoughtbot.com with any questions.

google performance figure reduce bananas explain reducing slack blocks mid javascript rails functions lenses sql elm js mandy moore stack overflow mental models barbed wire envelopes postgres functional programming persistant quenneville activerecord rubyists stephanie yeah stephanie well stephanie it stephanie oh

021: From Active Record Business Logic to DDD & Events with Andrzej Krzywda

The Rails Changelog

Play Episode Listen Later Mar 7, 2024 81:02

Andrzej Krzywda discusses event sourcing, event-driven architecture, and Domain-Driven Design (DDD) in the context of Ruby on Rails applications. He explains the concept of bounded contexts and how they relate to communication between different modules. He also shares insights on when and why to apply DDD to Rails applications, particularly in cases where the application becomes complex and difficult to maintain. Andrzej explores the challenges and benefits of migrating an existing Rails app to an event-driven architecture and highlights advanced event sourcing concepts such as snapshotting, projections, and versioning. In this conversation, Andrzej Krzywda discusses event sourcing and DDD in Rails applications. He explains the concepts of snapshotting and projection, which are techniques used to optimize performance and retrieve specific data from event streams. Andrzej also delves into the challenges of event versioning and how it can be managed in Rails applications. Additionally, he shares insights about the wroclove.rb conference, its history, and its focus on advanced and deep technical topics.TakeawaysEvent sourcing is a persistence mechanism that persists all the little changes that happen to a specific object, while event-driven architecture is a way of building software modules that communicate with events.DDD involves splitting a system into contexts or domains and using events to communicate between them. It can be applied to Ruby on Rails applications, particularly in cases where the application becomes complex and difficult to maintain.Migrating an existing Rails app to an event-driven architecture can help address issues with large classes, complex associations, and lack of modularity.Advanced event sourcing concepts such as snapshotting, projections, and versioning can be used to optimize performance and manage data integrity in event-driven applications. Snapshotting and projection are techniques used in event sourcing to optimize performance and retrieve specific data from event streams.Event versioning is a challenge in event sourcing, but it can be managed by introducing new event versions and implementing upcasters to convert old events to new versions.wroclove.rb is a Ruby and Rails conference in Wrocław, Poland, that focuses on advanced and deep technical topics.The conference aims to inspire, educate, and challenge the status quo in the Ruby and Rails community.Rails Event Store and Eventide are two libraries that facilitate the implementation of event-driven architectures in Rails applications, each with its own philosophy and approach.wcrolove.rb Ruby and Rails ConferenceRailsEventStoreArkencyRails Architect Masterclass[Video] Event Sourcing Demystified: A Simple-To-Understand Guide

technology events event web poland programming rails migrating andrzej wroc ruby on rails ddd eventide activerecord business logic domain driven design ddd record business andrzej krzywda

417: Module Docs

technology challenges secrets press mastering software types expert caution rails hooks deciphering callbacks andrew mason appropriate use railsconf activerecord

Play Episode Listen Later Mar 5, 2024 39:32

Stephanie shares about her vacation at Disney World, particularly emphasizing the technological advancements in the park's mobile app that made her visit remarkably frictionless. Joël had a conversation about a topic he loves: units of measure, and he got to go deep into the idea of dimensional analysis with someone this week. Together, Joël and Stephanie talk about module documentation within software development. Joël shares his recent experience writing module docs for a Ruby project using the YARD documentation system. He highlights the time-consuming nature of crafting good documentation for each public method in a class, emphasizing that while it's a demanding task, it significantly benefits those who will use the code in the future. They explore the attributes of good documentation, including providing code examples, explaining expected usage, suggesting alternatives, discussing edge cases, linking to external resources, and detailing inputs, outputs, and potential side effects. Multidimensional numbers episode (https://bikeshed.thoughtbot.com/416) YARD docs (https://yardoc.org/) New factory_bot documentation (https://thoughtbot.com/blog/new-docs-for-factory_bot) Dash (https://kapeli.com/dash) Solargraph (https://solargraph.org/) Transcript: JOËL: Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Joël Quenneville. STEPHANIE: And I'm Stephanie Minn, and together, we're here to share a bit of what we've learned along the way. JOËL: So, Stephanie, what's new in your world? STEPHANIE: So, I recently was on vacation, and I'm excited [chuckles] to tell our listeners all about it. I went to Disney World [laughs]. And honestly, I was especially struck by the tech that they used there. As a person who works in tech, I always kind of have a little bit of a different experience knowing a bit more about software, I suppose, than just your regular person [laughs], citizen. And so, at Disney World, I was really impressed by how seamlessly the like, quote, unquote, "real life experience" integrated with their use of their branded app to pair with, like, your time at the theme park. JOËL: This is, like, an app that runs on your mobile device? STEPHANIE: Yeah, it's a mobile app. I haven't been to Disney in a really long time. I think the last time I went was just as a kid, like, this was, you know, pre-mobile phones. So, I recall when you get into the line at a ride, you can skip the line by getting what's called a fast pass. And so, you kind of take a ticket, and it tells you a designated time to come back so that you could get into the fast line, and you don't have to wait as long. And now all this stuff is on your mobile app, and I basically did not wait in [laughs] a single line for more than, like, five minutes to go on any of the rides I wanted. It just made a lot of sense that all these things that previously had more, like, physical touchstones, were made a bit more convenient. And I hesitate to use the word frictionless, but I would say that accurately describes the experience. JOËL: That's kind of amazing; the idea that you can use tech to make a place that's incredibly busy also feel seamless and where you don't have to wait in line. STEPHANIE: Yeah and, actually, I think the coolest part was it blended both your, like, physical experience really well with your digital one. I think that's kind of a gripe I have as a technologist [laughs] when I'm just kind of too immersed in my screen as opposed to the world around me. But I was really impressed by the way that they managed to make it, like, a really good supplement to your experience being there. JOËL: So, you're not hyped for a future world where you can visit Disney in VR? STEPHANIE: I mean, I just don't think it's the same. I rode a ride [laughs] where it was kind of like a mini roller coaster. It was called Expedition Everest. And there's a moment, this is, like, mostly indoors, but there's a moment where the roller coaster is going down outside, and you're getting that freefall, like, drop feeling in your stomach. And it also happened to be, like, drizzling that day that we were out there, and I could feel it, you know, like, pelting my head [laughs]. And until VR can replicate that experience [chuckles], I still think that going to Disney is pretty fun. JOËL: Amazing. STEPHANIE: So, Joël, what's new in your world? JOËL: I'm really excited because I had a conversation about a topic that I like to talk about: units of measure. And I got to go deep into the idea of dimensional analysis with someone this week. This is a technique where you can look at a calculation or a function and sort of spot-check whether it's correct by looking at whether the unit for the measure that would come out match what you would expect. So, you do math on the units and ignore the numbers coming into your formula. And, you know, let's say you're calculating the speed of something, and you get a distance and the amount of time it took you to take to go that distance. And let's say your method implements this as distance times time. Forget about doing the actual math with the numbers here; just look at the units and say, okay, we've got our meters, and we've got our seconds, and we're multiplying them together. The unit that comes out of this method is meters times seconds. You happen to know that speeds are not measured in meters times seconds. They're measured in meters divided by seconds or meters per second. So, immediately, you get a sense of, like, wait a minute, something's wrong here. I must have a bug in my function. STEPHANIE: Interesting. I'm curious how you're representing that data to, like, know if there's a bug or not. In my head, when you were talking about that, I'm like, oh yeah, I definitely recall doing, like, math problems for homework [laughs] where I had, you know, my meters per second. You have your little fractions written out, and then when you multiply or divide, you know how to, like, deal with the units on your piece of paper where you're showing your work. But I'm having a hard time imagining what that looks like as a programmer dealing with that problem. JOËL: You could do it just all in your head based off of maybe some comments that you might have or the name of the variable or something. So, you're like, okay, well, I have a distance in meters and a time in seconds, and I'm multiplying the two. Therefore, what should be coming out is a value that is in meters times seconds. If you want to get fancier, you can do things with value objects of different types. So, you say, okay, I have a distance, and I have a time. And so, now I have sort of a multiplication of a distance and a time, and sort of what is that coming out as? That can sometimes help you prevent from having some of these mistakes because you might have some kind of error that gets raised at runtime where it's like, hey, you're trying to multiply two units that shouldn't be multiplied, or whatever it is. You can also, in some languages, do this sort of thing automatically at the type level. So, instead of looking at it yourself and sort of inferring it all on your own based off of the written code, languages like F# have built-in unit-of-measure systems where once you sort of tag numbers as just being of a particular unit of measure, any time you do math with those numbers, it will then tag the result with whatever compound unit comes from that operation. So, you have meters, and you have seconds. You divide one by the other, and now the result gets tagged as meters per second. And then, if you have another calculation that takes the output of the first one and it comes in, you can tell the compiler via type signature, hey, the input for this method needs to be in meters per second. And if the other calculation sort of automatically builds something that's of a different unit, you'll get a compilation error. So, it's really cool what it can do. STEPHANIE: Yeah, that is really neat. I like all of those built-in guardrails, I suppose, to help you, you know, make sure that your answer is correct. Definitely could have used that [chuckles]. Turns out I just needed a calculator to take my math test with [laughs]. JOËL: I think what I find valuable more than sort of the very rigorous approach is the mindset. So, anytime you're dealing with numbers, thinking in your mind, what is the unit of this number? When I do math with it with a different number, is it the same unit? Is it a different unit? What is the unit of the thing that's coming out? Does this operation make sense in the domain of my application? Because it's easy to sometimes think you're doing a math operation that makes sense, and then when you look at the unit, you're like, wait a minute, this does not make sense. And I would go so far as to say that, you know, you might think, oh, I'm not doing a physics app. I don't care about units of measure. Most numbers in your app that are actually numbers are going to have some kind of unit of measure associated to them. Occasionally, you might have something where it's just, like, a straight-up, like, quantity or something like that. It's a dimensionless number. But most things will have some sort of unit. Maybe it's a number of dollars. Maybe it is an amount of time, a duration. It could be a distance. It could be all sorts of things. Typically, there is some sort of unit that should attach to it. STEPHANIE: Yeah. That makes sense that you would want to be careful about making sure that your mathematical operations that you're doing when you're doing objects make sense. And we did talk about this in the last episode about multidimensional numbers a little bit. And I suppose I appreciate you saying that because I think I have mostly benefited from other people having thought in that mindset before and encoding, like I mentioned, those guardrails. So, I can recall an app where I was working with, you know, some kind of currency or money object, and that error was raised when I would try to divide by zero because rather than kind of having to find out later with some, not a number or infinite [laughs] amount of money bug, it just didn't let me do that. And that wasn't something that I had really thought about, you know, I just hadn't considered that zero value edge case when I was working on whatever feature I was building. JOËL: Yeah, or even just generally the idea of dividing money. What does that even mean? Are you taking an amount of money and splitting it into two equivalent piles to split among multiple people? That kind of makes sense. Are you dividing money by another money value? That's now asking a very different kind of question. You're asking, like, what is the ratio between these two, I guess, piles of money if we want to make it, you know, in the physical world? Is that a thing that makes sense in your application? But also, realize that that ratio that you get back is not itself an amount of money. And so, there are some subtle bugs that can happen around that when you don't keep track of what your quantities are. So, this past week, I've been working on a project where I ended up having to write module docs for the code in question. This is a Ruby project, so I'm writing docs using the YARD documentation system, where you effectively just write code comments at the sort of high level covering the entire class and then, also, individual documentation comments on each of the methods. And that's been really interesting because I have done this in other languages, but I'd never done it in Ruby before. And this is a piece of code that was kind of gnarly and had been tricky for me to figure out. And I figured that a couple of these classes could really benefit from some more in-depth documentation. And I'm curious, in your experience, Stephanie, as someone who's writing code, using code from other people, and who I assume occasionally reads documentation, what are the things that you like to see in good sort of method-level docs? STEPHANIE: Personally, I'm really only reading method-level docs when, you know, at this point, I'm, like, reaching for a method. I want to figure out how to use it in my use case right now [laughs]. So, I'm going to search API documentation for it. And I really am just scanning for inputs, especially, I think, and maybe looking at, you know, some potential various, like, options or, like, variations of how to use the method. But I'm kind of just searching for that at a glance and then moving on [laughs] with my day. That is kind of my main interaction with module docs like that, and especially ones for Ruby and Rails methods. JOËL: And for clarity's sake, I think when we're talking about module docs here, I'm generally thinking of, like, any sort of documentation that sort of comments in code meant to document. It could be the whole modular class. It could be on a per-method level, things like RDoc or YARD docs on Ruby classes. You used the word API docs here. I think that's a pretty similar idea. STEPHANIE: I really haven't given the idea of writing this kind of documentation a lot of thought because I've never had to do too much of it before, but I know, recently, you have been diving deep into it because, you know, like you said, you found these classes that you were working with a bit ambiguous, I suppose, or just confusing. And I'm wondering what kind of came out of that journey. What are some of the most interesting aspects of doing this exercise? JOËL: And one of the big ones, and it's not a fun one, but it is time-consuming. Writing good docs per method for a couple of classes takes a lot of time, and I understand why people don't do it all the time. STEPHANIE: What kinds of things were you finding warranted that time? Like, you know, you had to, at some point, decide, like, whether or not you're going to document any particular method. And what were some of the things you were looking out for as good reasons to do it? JOËL: I was making the decisions to document or not document on a class level, and then every public method gets documentation. If there's a big public API, that means every single one of those methods is getting some documentation comments, explaining what they do, how they're meant to be used, things like that. I think my kind of conclusion, having worked with this, is that the sort of sweet spot for this sort of documentation is for anything that is library-like, so a lot of things that maybe would go into a Rails lib directory might make sense. Anything you're turning into a gem that probably makes sense. And sometimes you have things in your Rails codebase that are effectively kind of library-like, and that was the case for the code that I was dealing with. It was almost like a mini ORM style kind of ActiveRecord-inspired series of base classes that had a bunch of metaprogramming to allow you to write models that were backed by not a database but a headless CMS, a content management system. And so, these classes are not extracted to the lib directory or, like, made into a gem, but they feel very library-esque in that way. STEPHANIE: Library-like; I like that descriptor a lot because it immediately made me think of another example of a time when I've used or at least, like, consumed this type of documentation in a, like, SaaS repo. Rather, you know, I'm not really seeing that level of documentation around domain objects, but I noticed that they really did a lot of extending of the application record class because they just had some performance needs that they needed to write some, like, custom code to handle. And so, they ended up kind of writing a lot of their own ORM-like methods for just some, like, custom callbacks on persisting and some just, like, bulk insertion functionality. And those came with a lot of different ways to use them. And I really appreciated that they were heavily documented, kind of like you would expect those ActiveRecord methods to be as well. JOËL: So, I've been having some conversations with other members at thoughtbot about when they like to use the style of module doc. What are some of the alternatives? And one that kept coming up for different people that they would contrast with this is what they would call the big README approach, and this could be for a whole gem, or it could be maybe some directory with a few classes in your application that's got a README in the root of the directory. And instead of documenting each method, you just write a giant README trying to answer sort of all of the questions that you anticipate people will ask. Is that something that you've seen, and how do you feel about that as a tool when you're looking for help? STEPHANIE: Yes. I actually really like that style of documentation. I find that I just want examples to get me started, especially; I guess this is especially true for libraries that I'm not super familiar with but need to just get a working knowledge about kind of immediately. So, I like to see examples, the getting started, the just, like, here's what you need to know. And as I start to use them, that will get me rolling. But then, if I find I need more details, then I will try to seek out more specific information that might come in the form of class method documentation. But I'm actually thinking about how FactoryBot has one of the best big README-esque [laughs] style of documentation, and I think they did a really big refresh of the docs not too long ago. It has all that high-level stuff, and then it has more specific information on how to use, you know, the most common methods to construct your factories. But those are very detailed, and yet they do sit, like, separately from inline, like, code documentation in the style of module docs that we're talking about. So, it is kind of an interesting mix of both that I think is helpful for me personally when I want both the “what do I need to know now?” And the, “like, okay, I know where to look for if I need something a little more detailed.” JOËL: Yeah. The two don't need to be mutually exclusive. I thought it was interesting that you mentioned how much examples are valuable to you because...I don't know if this is controversial, but an opinion that I have about sort of per-method documentation is that you should always default to having a code example for every method. I don't care how simple it is or how obvious it is what it does. Show me a code example because, as a developer, examples are really, really helpful. And so, seeing that makes documentation a lot more valuable than just a couple of lines that explain something that was maybe already obvious from the title of the method. I want to see it in action. STEPHANIE: Interesting. Do you want to see it where the method definition is? JOËL: Yes. Because sometimes the method definition, like, the implementation, might be sort of complex. And so, just seeing a couple of examples, like, oh, you call with this input, you get that. Call with this other input; you get this other thing. And we see this in, you know, some of the core docs for things like the enumerable methods where having an example there to be like, oh, so that's how map works. It returns this thing under these circumstances. That sort of thing is really helpful. And then, I'll try to do it at a sort of a bigger level for that class itself. You have a whole paragraph about here's the purpose of the class. Here's how you should use it. And then, here's an example of how you might use it. Particularly, if this is some sort of, like, base class you're meant to inherit from, here's the circumstances you would want to subclass this, and then here's the methods you would likely want to override. And maybe here are the DSLs you might want to have and to kind of package that in, like, a little example of, in this case, if you wanted a model that read from the headless CMS, here's what an example of such a little model might look like. So, it's kind of that putting it all together, which I think is nice in the module docs. It could probably also live in the big README at some level. STEPHANIE: Yeah. As you are saying that, I also thought about how I usually go search for tests to find examples of usage, but I tend to get really overwhelmed when I see inline, like, that much inline documentation. I have to, like, either actively ignore it, choose to ignore it, or be like, okay, I'm reading this now [laughs]. Because it just takes up so much visual space, honestly. And I know you put a lot of work into it, a lot of time, but maybe it's because of the color of my editor theme where comments are just that, like, light gray [laughs]. I find them quite easy to just ignore. But I'm sure there will be some time where I'm like, okay, like, if I need them, I know they're there. JOËL: Yeah, that is, I think, a downside, right? It makes it harder to browse the code sometimes because maybe your entire screen is almost taken up by documentation, and then, you know, you have one method up, and you've got to, like, scroll through another page of documentation before you hit the next method, and that makes it harder to browse. And maybe that's something that plays into the idea of that separation between library-esque code versus application code. When you browse library-esque code, when you're actually browsing the source, you're probably doing it for different reasons than you would for code in your application because, at that point, you're effectively source diving, sometimes being like, oh, I know this class probably has a method that will do the thing I want. Where is it? Or you're like, there's an edge case I don't understand on this method. I wonder what it does. Let me look at the implementation. Or even some existing code in the app is using this library method. I don't know what it does, but they call this method, and I can't figure out why they're using it. Let me look at the source of the library and see what it does under the hood. STEPHANIE: Yeah. I like the distinction of it is kind of a different mindset that you're reading the code at, where, like, sometimes my brain is already ready to just read code and try to figure out inputs and outputs that way. And other times, I'm like, oh, like, I actually can't parse this right now [chuckles]. Like, I want to read just English, like, telling me what to expect or, like, what to look out for, especially when, like you said, I'm not really, like, trying to figure out some strange bug that would lead me to diving deep in the source code. It's I'm at the level where I'm just reaching for a method and wanting to use it. We're writing these YARD docs. I think I also heard you mention that you gave some, like, tips or maybe some gotchas about how to use certain methods. I'm curious why that couldn't have been captured in a more, like, self-documenting way. Or was there a way that you could have written the code for that not to have been needed as a comment or documented as that? And was there a way that method names could have been clear to signal, like, the intention that you were trying to convey through your documentation? JOËL: I'm a big fan of using method names as a form of documentation, but they're frequently not good enough. And I think comments, whether they're just regular inline comments or more official documentation, can be really good to help avoid sort of common pitfalls. And one that I was working with was, there were two methods, and one would find by a UID, so it would search up a document by UID. And another one would search by ID. And when I was attempting to use these before I even started documenting, I used the wrong one, and it took me a while to realize, oh wait, these things have both UIDs and IDs, and they're slightly different, and sometimes you want to use one or the other. The method names, you know, said like, "Find by ID" or "Find by UID." I didn't realize there were both at the time because I wasn't browsing the source. I was just seeing a place where someone had used it. And then, when I did find it in the source, I'm like, well, what is the difference? And so, something that I did when I wrote the docs was sort of call out on both of those methods; by the way, there is also find by UID. If you're searching by UID, consider using the other one. If you don't know what the difference is, here's a sentence summarizing the difference. And then, here's a link to external documentation if you want to dive into the nitty gritty of why there are two and what the differences are. And I think that's something you can't capture in just a method name. STEPHANIE: Yeah, that's true. I like that a lot. Another use case you can think of is when method names are aliased, and it's like, I don't know how I would have possibly known that until I, you know, go through the journey of realizing [laughs] that these two methods do the same thing or, like, stumbling upon where the aliasing happens. But if that were captured in, like, a little note when I'm in, like, a documentation viewer or something, it's just kind of, like, a little tidbit of knowledge [laughs] that I get to gain along the way that ends up, you know, being useful later because I will have just kind of...I will likely remember having seen something like that. And I can at least start my search with a little bit more context than when you don't know what you don't know. JOËL: I put a lot of those sorts of notes on different methods. A lot of them are probably based on a personal story where I made a mistaken assumption about this method, and then it burned me. But I'm like, okay, nobody else is going to make that mistake. By the way, if you think this is what the method does, it does something slightly different and, you know, here's why you need to know that. STEPHANIE: Yeah, you're just looking out for other devs. JOËL: And, you know, trying to, like, take my maybe negative experience and saying like, "How can I get value out of that?" Maybe it doesn't feel great that I lost an hour to something weird about a method. But now that I have spent that hour, can I get value out of it? Is the sort of perspective I try to have on that. So, you mentioned kind of offhand earlier the idea of a documentation viewer, which would be separate than just reading these, I guess, code comments directly in your code editor. What sort of documentation viewers do you like to use? STEPHANIE: I mostly search in my browser, you know, just the official documentation websites for Rails, at least. And then I know that there are also various options for Ruby as well. And I think I had mentioned it before but using DuckDuckGo as my search engine. I have nice bang commands that will just take me straight to the search for those websites, which is really nice. Though, I have paired with people before who used various, like, macOS applications to do something similar. I think Alfred might have some built-in workflows for that. And then, a former co-worker used to use one called Dash, that I have seen before, too. So, it's another one of those just handy just, like, search productivity extensions. JOËL: You mentioned the Rails documentation, and this is separate from the guides. But the actual Rails docs are generated from comments like this inline in code. So, all the different ActiveRecord methods, when you search on the Rails documentation you're like, oh yeah, how does find_by work? And they've got a whole, like, paragraph explaining how it works with a couple of examples. That's this kind of documentation. If you open up that particular file in the source code, you'll find the comments. And it makes sense for Rails because Rails is more of, you know, library-esque code. And you and I search these docs pretty frequently, although we don't tend to do it, like, by opening the Rails gem and, like, grepping through the source to find the code comment. We do it through either a documentation site that's been compiled from that source or that documentation that's been extracted into an offline tool, like you'd mentioned, Dash. STEPHANIE: Yeah, I realized how conflicting, I suppose, it is for me to say that I find inline documentation really overwhelming or visually distracting, whereas I recognize that the only reason I can have that nice, you know, viewing experience is because documentation viewers use the code comments in that format to be generated. JOËL: I wonder if there's like a sort of...I don't know what this pattern is called, but a bit of a, like, middle-quality trap where if you're going to source dive, like, you'd rather just look at the code and not have too much clutter from sort of mediocre comments. But if the documentation is really good and you have the tooling to read it, then you don't even need to source dive at all. You can just read the documentation, and that's sufficient. So, both extremes are good, but that sort of middle kind of one foot in each camp is sort of the worst of both worlds experience. Because I assume when you look for Rails documentation, you never open up the actual codebase to search. The documentation is good enough that you don't even need to look at the files with the comments and the code. STEPHANIE: Yeah, and I'm just recalling now there's, like, a UI feature to view the source from the documentation viewer page. JOËL: Yes. STEPHANIE: I use that actually quite a bit if the comments are a little bit sparse and I need just the code to supplement my understanding, and that is really nice. But you're right, like, I very rarely would be source diving, unless it's a last resort [laughs], let's be honest. JOËL: So, we've talked about documentation viewers and how that can make things nice, and you're able to read documentation for things. But a lot of other tooling can benefit from this sort of model documentation as well, and I'm thinking, in particular, Solargraph, which is Ruby's language server protocol. And it has plugins for VS Code, for Vim, for a few different editors, takes advantage of that to provide all sorts of things. So, you can get smart expansion of code and good suggestions. You can get documentation for what's under your cursor. Maybe you're reading somebody else's code that they've written, and you're like, why are they calling this parameterized method here? What does that even do? Like, in VS Code, you could just hover over it, and it will pop up and show you documentation, including the, like, inputs and return types, and things like that. That's pretty nifty. STEPHANIE: Yeah, that is cool. I use VS Code, but I've not seen that too much yet because I don't think I've worked in enough codebases with really comprehensive [laughs] YARD docs. I'm actually wondering, tooling-wise, did you use any helpful tools when you were writing them or were you hand-documenting each? JOËL: I was hand-documenting everything. STEPHANIE: Class. Okay. JOËL: The thing that I did use is the YARD gem, which you don't need to have the gem to write YARD-style documentation. But if you have the gem, you can run a local server and then preview a documentation site that is generated from your comments that has everything in there. And that was incredibly helpful for me as I was trying to sort of see an overview of, okay, what would someone who's looking at the docs generated from this see when they're trying to look for what the documentation of a particular method does? STEPHANIE: Yeah, and that's really nice. JOËL: Something that I am curious about that I've not really had a lot of experience with is whether or not having extra documentation like that can help AI tools give us better suggestions. STEPHANIE: Yeah, I don't know the answer to that either, but I would be really curious to know if that is already something that happens with something like Copilot. JOËL: Do better docs help machines, or are they for humans only? STEPHANIE: Whoa, that's a very [laughs] philosophical question, I think. It would make sense, though, that if we already have ways to parse and compile this kind of documentation, then I can see that incorporating them into the types of, like, generative problems that AI quote, unquote "solves" [chuckles] would be really interesting to find out. But anyone listening who kind of knows the answer to that or has experience working with AI tools and various types of code comment documentation would be really curious to know what your experience is like and if it improves your development workflow. So, for people who might be interested in getting better at documenting their code in the style of module docs, what would you say are some really great attributes of good documentation in this form? JOËL: I think, first of all, you have to write from the motivation of, like, if you were confused and wanting to better understand what a method does, what would you like to see? And I think coming from that perspective, and that was, in my case, I had been that person, and then I was like, okay, now that I've figured it out, I'm going to write it so that the next person is not confused. I have five or six things that I think were really valuable to add to the docs, a few of which we've already mentioned. But rapid fire, first of all, code example. I love code examples. I want a code example on every method. An explanation of expected usage. Here's what the method does. Here's how we expect you to use this method in any extra context about sort of intended use. Callouts for suggested alternatives. If there are methods that are similar, or there's maybe a sort of common mistake that you would reach for this method, put some sort of call out to say, "Hey, you probably came here trying to do X. If that's what you were actually trying to do, you should use method Y." Beyond that, a discussion of edge cases, so any sort of weird ways the method behaves. You know, when you pass nil to it, does it behave differently? If you call it in a different context, does it behave differently? I want to know that so that I'm not totally surprised. Links to external resources–really great if I want to, like, dig deeper. Is this method built on some sort of, like, algorithm that's documented elsewhere? Please link to that algorithm. Is this method integrating with some, like, third-party API? You know, they have some documentation that we could link to to go deeper into, like, what these search options do. Link to that. External links are great. I could probably find it by Googling myself, but you are going to make me very happy as a developer if you already give me the link. You'd mentioned capturing inputs and outputs. That's a great thing to scan for. Inputs and outputs, though, are more sometimes than just the arguments and return values. Although if we're talking about arguments, any sort of options hash, please document the keys that go in that because that's often not obvious from the code. And I've spent a lot of time source diving and jumping between methods trying to figure out like, what are the options I can pass to this hash? Beyond the explicit inputs and outputs, though, anything that is global state that you rely on. So, do you need to read something from an environment variable or even a global variable or something like that that might make this method behave differently in different situations? Please document that. Any situations where you might raise an error that I might not expect or that I might want to rescue from, let me know what are the potential errors that might get raised. And then, finally, any sorts of side effects. Does this method make a network call? Are you writing to the file system? I'd like to know that, and I'd have to, like, figure it out by trial and error. And sometimes, it will be obvious in just the description of the method, right? Oh, this method pulls data from a third-party API. That's pretty clear. But maybe it does some sort of, like, caching in the background or something to a file that's not really important. But maybe I'm trying to do a unit test that involves this, and now, all of a sudden, I have to do some weird stubbing. I'd like to know that upfront. So, those are kind of all the things I would love to have in my sort of ideal documentation comment that would make my life easier as a developer when trying to use some code. STEPHANIE: Wow. What a passionate plea [laughs]. I was very into listening to you list all of that. You got very animated. And it makes a lot of sense because I feel like these are kind of just the day-to-day developer issues we run into in our work and would be so awesome if, especially as the, you know, author where you have figured all of this stuff out, the author of a, you know, a method or a class, to just kind of tell us these things so we don't have to figure it out ourselves. I guess I also have to respond to that by saying, on one hand, I totally get, like, you want to be saved [chuckles] from those common pitfalls. But I think that part of our work is just going through that and playing around and exploring with the code in front of us, and we learn all of that along the way. And, ultimately, even if that is all provided to you, there is something about, like, going through it yourself that gives you a different perspective on it. And, I don't know, maybe it's just my bias against [laughs] all the inline text, but I've also seen a lot of that type of information captured at different levels of documentation. So, maybe it is a Confluence doc or in a wiki talking about, you know, common gotchas for this particular problem that they were trying to solve. And I think what's really cool is that, you know, everyone can kind of be served and that people have different needs that different styles of documentation can meet. So, for anyone diving deep in the source code, they can see all of those examples inline. But, for me, as a big Googler [laughs], I want to see just a nice, little web app to get me the information that I need to find. I'm happy having that a little bit more, like, extracted from my source code. JOËL: Right. You don't want to have to read the source code with all the comments in it. I think that's a fair criticism and, yeah, probably a downside of this. And I'm wondering, there might be some editor tooling that allows you to just collapse all comments and hide them if you wanted to focus on just the code. STEPHANIE: Yeah, someone, please build that for me. That's my passionate plea [laughs]. And on that note, shall we wrap up? JOËL: Let's wrap up. STEPHANIE: Show notes for this episode can be found at bikeshed.fm. JOËL: This show has been produced and edited by Mandy Moore. STEPHANIE: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review in iTunes. It really helps other folks find the show. JOËL: If you have any feedback for this or any of our other episodes, you can reach us @_bikeshed, or you can reach me @joelquen on Twitter. STEPHANIE: Or reach both of us at hosts@bikeshed.fm via email. JOËL: Thanks so much for listening to The Bike Shed, and we'll see you next week. ALL: Bye. AD: Did you know thoughtbot has a referral program? If you introduce us to someone looking for a design or development partner, we will compensate you if they decide to work with us. More info on our website at: tbot.io/referral. Or you can email us at referrals@thoughtbot.com with any questions.

Rails + Postgres

Postgres FM

Play Episode Listen Later Mar 1, 2024 45:23

Michael and Nikolay are joined by Andrew Atkinson, author of High Performance PostgreSQL for Rails, to discuss how Rails and Postgres work together — where the limits are, how people use the ORM, things that are improving, and some things we can do as a Postgres community to make it even better. Here are some links to things they mentioned:Planet Argon survey https://rails-hosting.com/2022/#databasesActive Record https://guides.rubyonrails.org/active_record_basics.htmlPostgreSQL specific usage of Active Record https://guides.rubyonrails.org/active_record_postgresql.htmlMultiple Databases with Active Record https://guides.rubyonrails.org/active_record_multiple_databases.htmlschema.rb vs structure.sql https://blog.appsignal.com/2020/01/15/the-pros-and-cons-of-using-structure-sql-in-your-ruby-on-rails-application.htmlactiverecord-clean-db-structure (Ruby gem by Lukas Fittl) https://github.com/lfittl/activerecord-clean-db-structureGitLab's migration_helpers.rb https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/database/migration_helpers.rbSQLite https://www.sqlite.orgPlanetScale's foreign key support announcement video https://twitter.com/PlanetScale/status/1732070818958500083DoorDash Engineering Blog https://doordash.engineering/blograils-pg-extras https://github.com/pawurb/rails-pg-extrasBenoit Tigeot testing Peter Geoghegan improvement for large IN lists https://gist.github.com/benoittgt/ab72dc4cfedea2a0c6a5ee809d16e04dHigh Performance PostgreSQL for Rails (Andy's book, 35% discount code “postgres.fm”) https://pragprog.com/titles/aapsql/high-performance-postgresql-for-railsAndy's blog and website https://andyatkinson.com~~~What did you like or not like? What should we discuss next time? Let us know via a YouTube comment, on social media, or by commenting on our Google doc!~~~Postgres FM is produced by:Michael Christofides, founder of pgMustardNikolay Samokhvalov, founder of Postgres.aiWith special thanks to:Jessie Draws for the elephant artwork

google technology databases rails sql ruby on rails orm postgresql postgres nikolay planetscale activerecord peter geoghegan andrew atkinson planet argon

Mastering Rails Callbacks – Deciphering the Secrets of Active Record

Ruby for All

Play Episode Listen Later Feb 8, 2024 30:42

In this episode of Ruby for All, Andrew and Julie discuss the intricacies of callbacks in Active Record models. They talk about their experiences, the pros and cons of using callbacks, and the issues they faced. They also share some helpful use cases for callbacks, including user authentication, logging and auditing, custom slug generation, and the concept of “hooks.” Also, Andrew and Julie review their ways of dealing with callbacks testing and debugging in a Rails application. Press download now to hear more! [00:02:32] Let's learn about “callbacks” in Rails as Andrew explains what they are and uses an example of a blog post to explain how a callback might function when saving a post. [00:03:56] Julie inquires if Tiptap can be used in a browser for apps and they discuss before-save callbacks in a post model and how they can be used to extract and save a title.[00:06:19] Andrew elaborates on the three different types of callbacks: before, after, and around callbacks, and gives examples of each. [00:10:06] They discuss practical uses for before-validation callbacks, such as setting default values. [00:11:12] Andrew clarifies the concept of “hooks” in programming, comparing it to callbacks. [00:12:18] Julie asks for examples of actions taken after validation versus before validation. [00:13:19] Andrew talks about how a file upload, an after-create callback can be used for processing the file such as generating thumbnails or updating related resources. He lists examples use cases for callbacks like hashing passwords before saving to the database during user authentication, triggering email notifications after a comment is posted, and logging and auditing activities like user sign-ups or errors. [00:15:57] Julie is curious about whether deleted accounts really remove all user data or just make it as inaccessible, noting some services offer a soft delete option with a time window to recover the account. Andrew has not yet encountered the fallback log issue he set up but explains how before-destroy callbacks could be used to implement a time-based soft delete system. [00:17:19] Andrew describes using before-create callbacks for generating custom slugs for blog posts automatically. [00:17:54] Andrew recalls a discussion at RailsConf about the diverse opinions on using callbacks, with some developers strongly against them and others in favor. He acknowledges that while callbacks can simplify complex operations, they can also make debugging difficult and can become problematic if used excessively or inappropriately. [00:23:00] Julie asks Andrew where he stands on the use of callbacks, and he positions himself in the middle, closer to using them when appropriate. [00:25:16] Andrew emphasizes being cautious with callbacks and explains that callbacks are useful when certain actions need to happen automatically without explicit instruction every time a record is saved.[00:27:40] Andrew discusses the challenges of testing callbacks, as they can require additional setup in tests and slow down the test suite. He concludes that callbacks are an integral part of Rails, he advises against using them as the first solution and recommends weighing their pros and cons carefully. Panelists:Andrew MasonJulie J.Sponsors:HoneybadgerGoRailsLinks:Andrew Mason X/TwitterAndrew Mason WebsiteJulie J. X/TwitterJulie J. WebsiteActive Record CallbacksTiptap (00:00) - Intro and Topic Overview (02:32) - Introduction to Callbacks in Rails (03:56) - Discussing Before-Save Callbacks and Tiptap (06:19) - Types of Callbacks: Before, After, and Around (10:06) - Uses for Before-Validation Callbacks (11:12) - Hooks vs Callbacks (12:18) - Practical Use Cases for Callbacks (15:57) - Soft Delete Options and Before-Destroy Callbacks (17:19) - Generating Custom Slugs with Before-Create Callbacks (17:54) - Diverse Opinions on Using Callbacks from RailsConf (23:00) - Andrew's Not an Expert (25:16) - Caution and Appropriate Use of Callbacks (27:40) - Challenges of Testing Callbacks

413: Developer Tales of Package Management

single table programming inheritance coding rails refactoring screencasts activerecord

Play Episode Listen Later Jan 23, 2024 33:33

Stephanie shares her task of retiring a small, internally-used link-shortening app. She describes the process as both celebratory and a bit mournful. Meanwhile, Joël discusses his deep dive into ActiveRecord, particularly in the context of debugging. He explores the complexities of ActiveRecord querying schemas and the additional latency this introduces. Together, the hosts discuss the nuances of package management systems and their implications for developers. They touch upon the differences between system packages and language packages, sharing personal experiences with tools like Homebrew, RubyGems, and Docker. Transcript: JOËL: Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Joël Quenneville. STEPHANIE: And I'm Stephanie Minn. And together, we're here to share a bit of what we've learned along the way. JOËL: So, Stephanie, what's new in your world? STEPHANIE: So, this week, I got to have some fun working on some internal thoughtbot work. And what I focused on was retiring one of our just, like, small internal self-hosted on Heroku apps in favor of going with a third-party service for this functionality. We basically had a tiny, little app that we used as a link-shortening service. So, if you've ever seen a tbot.io short link out in the world, we were using our just, like, an in-house app to do that, you know, but for various reasons, we wanted to...just it wasn't worth maintaining anymore. So, we wanted to just use a purchased service. But today, I got to just, like, do the little bit of, like, tidying up, you know, in preparation to archive a repo and kind of delete the app from Heroku, and I hadn't done that before. So, it felt a little bit celebratory and a little bit mournful even [laughs] to, you know, retire something like that. And I was pairing with another thoughtbot developer, and we used a pairing app called Tuple. And you can just send, like, fun reactions to each other. Like, you could send, like, a fire emoji [laughs] or something if that's what you're feeling. And so, I sent some, like, confetti when we clicked the, "I understand what deleting this app means on GitHub." But I joked that "Actually, I feel like what I really needed was a, like, a salute kind of like thank you for your service [laughs] type of reaction." JOËL: I love those moments when you're kind of you're hitting those kind of milestone-y moments, and then you get to send a reaction. I should do that more often in Tuple. Those are fun. STEPHANIE: They are fun. There's also a, like, table flip reaction, too, is one that I really enjoy [laughs], you know, you just have to manifest that energy somehow. And then, after we kind of sent out an email to the company saying like, "Oh yeah, we're not using our app anymore for link shortening," someone had a great suggestion to make our archived repo public instead of private. I kind of liked it as a way of, like, memorializing this application and let community members see, you know, real code in a real...the application that we used here at thoughtbot. So, hopefully, if not me, then someone else will be able to do that and maybe publish a little blog post about that. JOËL: That's exciting. So, it's not currently public, the repo, but it might be at some point in the future. STEPHANIE: Yeah, that's right. JOËL: We'll definitely have to mention it on a future episode if that happens so that people following along with the story can go check out the code. STEPHANIE: So, Joël, what's new in your world? JOËL: I've been doing a deep dive into how ActiveRecord works. Particularly, I am debugging some pretty significant slowdowns in querying ActiveRecord models that are backed not by a regular Postgres database but instead a Snowflake data warehouse via an ODBC connection. So, there's a bunch of moving pieces going on here, and it would just take forever to make any queries. And sure, the actual reported query time is longer than for a local Postgres database, but then there's this sort of mystery extra waiting time, and I couldn't figure out why is it taking so much longer than the actual sort of recorded query time. And I started digging into all of this, and it turns out that in addition to executing queries to pull actual data in, ActiveRecord needs to, at various points, query the schema of your data store to pull things like names of tables and what are the indexes and primary keys and things like that. STEPHANIE: Wow. That sounds really cool and something that I have never needed to do before. I'm curious if you noticed...you said that it takes, I guess, longer to query Snowflake than it would a more common Postgres database. Were you noticing this performance slowness locally or on production? JOËL: Both places. So, the nice thing is I can reproduce it locally, and locally, I mean running the Rails app locally. I'm still talking to a remote Snowflake data warehouse, which is fine. I can reproduce that slowness locally, which has made it much easier to experiment and try things. And so, from there, it's really just been a bit of a detective case trying to, I guess, narrow the possibility space and try to understand what are the parts that trigger slowness. So, I'm printing timestamps in different places. I've got different things that get measured. I've not done, like, a profiling tool to generate a flame graph or anything like that. That might have been something cool to try. I just did old-school print statements in a couple of places where I, like, time before, time after, print the delta, and that's gotten me pretty far. STEPHANIE: That's pretty cool. What do you think will be an outcome of this? Because I remember you saying you're digging a little bit into ActiveRecord internals. So, based on, like, what you're exploring, what do you think you could do as a developer to increase some of the performance there? JOËL: I think probably what this ends up being is finding that the Snowflake adapter that I'm using for ActiveRecord maybe has some sort of small bug in it or some implementation that's a little bit too naive that needs to be fine-tuned. And so, probably what ends up happening here is that this finishes as, like, an open-source pull request to the Snowflake Adapter gem. STEPHANIE: Yeah, that's where I thought maybe that might go. And that's pretty cool, too, and to, you know, just be investigating something on your app and being able to make a contribution that it benefits the community. JOËL: And that's what's so great about open source because not only am I able to get the source to go source diving through all of this, because I absolutely need to do that, but also, then if I make a fix, I can push that fix back out to the community, and everybody gets to benefit. STEPHANIE: Cool. Well, that's another thing that I look forward to hearing more on the development of [laughs] later if it pans out that way. JOËL: One thing that has been interesting with this Snowflake work is that there are a lot of moving parts and multiple different packages that I need to install to get this all to work. So, I mentioned that I might be doing a pull request against the Snowflake Adapter for ActiveRecord, but all of this talks through a sort of lower-level technology protocol called ODBC, which is a sort of generic protocol for speaking to data stores, and that actually has two different pieces. I had to install two different packages. There is a sort of low-level executable that I had to install on my local dev machine and that I have to install on our servers. And on my Mac, I'm installing that via Homebrew, which is a system package. And then to get Ruby bindings for that, there is a Ruby gem that I install that allows Ruby code to talk to ODBC, and that's installed via RubyGems or Bundler. And that got me thinking about sort of these two separate ecosystems that I tend to work with every day. We've got sort of the system packages and the, I don't know what you want to call them, language packages maybe, things like RubyGems, but that could also be NPM or whatever your language of choice is, and realizing that we kind of have things split into two different zones, and sometimes we need both and wondering a little bit about why is that difference necessary. STEPHANIE: Yeah, I don't have an answer to that [laughs] question right now, but I can say that that was an area that really tripped me up, I think, when I was first a fledgling developer. And I was really confused about where all of these dependencies were coming from and going through, you know, setting up my first project and being, like, asked to install Postgres on my machine but then also Bundler, which then also installs more dependencies [laughs]. The lines between those ecosystems were not super clear to me. And, you know, even now, like, I find myself really just kind of, like, learning what I need to know to get by [laughs] with my day-to-day work. But I do like what you said about these are kind of the two main layers that you're working with in terms of package management. And it's really helpful to have that knowledge so you can troubleshoot when there is an issue at one or the other. JOËL: And you mentioned Postgres. That's another one that's interesting because there are components in both of those ecosystems. Postgres itself is typically installed via a system package manager, so something like Homebrew on a Mac or apt-get on a Linux machine. But then, if you're interacting with Postgres in a Ruby app, you're probably also installing the pg gem, which are Ruby's bindings for Postgres to allow Ruby to talk to Postgres, and that lives in the package ecosystem on RubyGems. STEPHANIE: Yeah, I've certainly been in the position of, you know, again, as consultants, we oftentimes are also setting up new laptops entirely [laughs] like client laptops and such and bundling and the pg gem is installed. And then at least I have, you know, I have to give thanks to the very clear error message that [laughs] tells me that I don't have Postgres installed on my machine. Because when I mentioned, you know, troubleshooting earlier, I've certainly been in positions where it was really unclear what was going on in terms of the interaction between what I guess we're calling the Ruby package ecosystem and our system level one. JOËL: Especially for things like the pg gem, which need to compile against some existing libraries, those always get interesting where sometimes they'll fail to compile because there's a path to some C compiler that's not set correctly or something like that. For me, typically, that means I need to update the macOS command line tools or the Xcode command line tools; I forget what the name of that package is. And, usually, that does the trick. That might happen if I've upgraded my OS version recently and haven't downloaded the latest version of the command line tools. STEPHANIE: Yeah. Speaking of OS versions, I have a bit of a story to share about using...I've never said this name out loud, but I am pretty sure that it would just be pronounced as wkhtmltopdf [laughs]. For some reason, whenever I see words like that in my brain, I want to, like, make it into a pronounceable thing [laughs]. JOËL: Right, just insert some vowels in there. STEPHANIE: Yeah, wkhtmltopdf [laughs]. Anyway, that was being used in an app to generate PDF invoices or something. It's a pretty old tool. It's a CLI tool, and it's, as far as I can tell, it's been around for a long time but was recently no longer maintained. And so, as I was working on this app, I was running into a bug where that library was causing some issues with the PDF that was generated. So, I had to go down this route of actually finding a Ruby gem that would figure out which package binary to use, you know, based off of my system. And that worked great locally, and I was like, okay, cool, I fixed the issue. And then, once I pushed my change, it turns out that it did not work on CI because CI was running on Ubuntu. And I guess the binary didn't work with the latest version of Ubuntu that was running on CI, so there was just so many incompatibilities there. And I was wanting to fix this bug. But the next step I took was looking into community-provided packages because there just simply weren't any, like, up-to-date binaries that would likely work with these new operating systems. And I kind of stopped at that point because I just wasn't really sure, like, how trustworthy were these community packages. That was an ecosystem I didn't know enough about. In particular, I was having to install some using apt from, you know, just, like, some Linux community. But yeah, I think I normally have a little bit more experience and confidence in terms of the Ruby package ecosystem and can tell, like, what gems are popular, which ones are trustworthy. There are different heuristics I have for evaluating what dependency to pull in. But here I ended up just kind of bailing out of that endeavor because I just didn't have enough time to go down that rabbit hole. JOËL: It is interesting that learning how to evaluate packages is a skill you have to learn that varies from package community to package community. I know that when I used to be very involved with Elm, we would often have people who would come to the Elm community from the JavaScript community who were used to evaluating NPM packages. And one of the metrics that was very popular in the JavaScript community is just stars on GitHub. That's a really important metric. And that wasn't really much of a thing in the Elm community. And so, people would come and be like, "Wait, how do I know which package is good? I don't see any stars on GitHub." And then, it turns out that there are other metrics that people would use. And similarly, you know, in Ruby, there are different ways that you might use to evaluate Ruby gems that may or may not involve stars on GitHub. It might be something entirely different. STEPHANIE: Yeah. Speaking of that, I wanted to plug a website that I have used before called the Ruby Toolbox, and that gives some suggestions for open-source Ruby libraries of various categories. So, if you're looking for, like, a JSON parser, it has some of the more popular ones. If you're looking for, you know, it stores them by category, and I think it is also based on things like stars and forks like that, so that's a good one to know. JOËL: You could probably also look at something like download numbers to see what's popular, although sometimes it's sort of, like, an emergent gem that's more popular. Some of that almost you just need to be a little bit in the community, like, hearing, you know, maybe listening to podcasts like this one, subscribing to Ruby newsletters, going to conferences, things like that, and to realize, okay, maybe, you know, we had sort of an old staple for JSON parsing, but there's a new thing that's twice as fast. And this is sort of becoming the new standard, and the community is shifting towards that. You might not know that just by looking at raw stats. So, there's a human component to it as well. STEPHANIE: Yeah, absolutely. I think an extension of knowing how to evaluate different package systems is this question of like, how much does an average developer need to know about package management? [laughs] JOËL: Yeah, a little bit to a medium amount, and then if you're writing your own packages, you probably need to know a little bit more. But there are some things that are really maybe best left to the maintainers of package managers. Package managers are actually pretty complex pieces of software in terms of all of the dependency management and making sure that when you say, "Oh, I've got Rails, and this other gem, and this other gem, and it's going to find the exact versions of all those gems that play nicely together," that's non-trivial. As a sort of working developer, you don't need to know all of the algorithms or the graph theory or any of that that underlies a package manager to be able to be productive in your career. And even as a package developer, you probably don't need to really know a whole lot of that. STEPHANIE: Yeah, that makes sense. I actually had referred to our internal at thoughtbot here, our kind of, like, expectations for skill levels for developers. And I would say for an average developer, we kind of just expect a basic understanding of these more complex parts of our toolchain, I think, specifically, like, command line tools and package management. And I think I'd mentioned earlier that, for me, it is a very need-to-know basis. And so, yeah, when I was going down that little bit of exploration around why wkhtmltopdf [chuckles] wasn't working [chuckles], it was a bit of a twisty and turning journey where I, you know, wasn't really sure where to go. I was getting very obtuse error messages, and, you know, I had to dive deep into all these forums [laughs] for all the various platforms [laughs] about why libraries weren't working. And I think what I did come away with was that like, oh, like, even though I'm mostly working on my local machine for development, there was some amount of knowledge I needed to have about the systems that my CI and, you know, production servers are running on. The project I was working on happened to have, like, a Docker file for those environments, and, you know, kind of knowing how to configure them to install the packages I needed to install and just knowing a little bit about the different ways of doing that on systems outside of my usual daily workflows. JOËL: And I think that gets back to some of the interesting distinctions between what we might call language packages versus system packages is that language packages more or less work the same across all operating systems. They might have a build step that's slightly different or something like that, but system packages might be pretty different between different operating systems. So, development, for me, is a Mac, and I'm probably installing system packages via something like Homebrew. If I then want that Rails app to run on CI or some Linux server somewhere, I can't use Homebrew to install things there. It's going to be a slightly different package ecosystem. And so, now I need to find something that will install Postgres for Linux, something that will install, I guess, wkhtmltopdf [laughs] for Linux. And so, when I'm building that Docker file, that might be a little bit different for Mac versus for...or I guess when you run a Docker file, you're running a containerized system. So, the goal there is to make this system the same everywhere for everyone. But when you're setting that up, typically, it's more of a Linux-like system. And so running inside the Docker container versus outside on the native Mac might involve a totally different set of packages and a different package tool. As opposed to something like Bundler, you've got your gem file; you bundle install. It doesn't matter if you're on Linux or macOS. STEPHANIE: Yes, I think you're right. I think we kind of answered our own question at the top of the show [laughs] about differences and what do you need to know about them. And I also like how you pointed out, oh yeah, like, Docker is supposed to [laughs], you know, make sure that we're all developing in the same system, essentially. But, you know, sometimes you have different use cases for it. And, yeah, when you were talking about installing an application on your native Mac and using Homebrew, but even, you know, not everyone even uses Homebrew, right? You can install manually [laughs] through whatever official installer that application might provide. So, there's just so many different ways of doing something. And I had the thought that it's too bad that we both [chuckles] develop on Mac because it could be really interesting to get a Linux user's perspective in here. JOËL: You mentioned not installing via Homebrew. A kind of glaring example of that in my personal setup is that I use Postgres.app to manage Postgres on my machine rather than using Homebrew. I've just...over the years, the Homebrew version every time I upgrade my operating system or something, it's just such a pain to update, and I've lost too many hours to it, and Postgres.app just works, and so I've switched to that. Most other things, I'll use the Homebrew version, but Postgres it's now Postgres.app. It's not even a command line install, and it works fine for me. STEPHANIE: Nice. Yeah. That's interesting. That's a good tip. I'll have to look into that next time because I have also certainly had to just install so many [laughs] various versions of Postgres and figure out what's going on with them every time I upgrade my OS. I'm with you, though, in terms of the packages world I'm looking for, it works [laughs]. JOËL: So, you'd mentioned earlier that packages is sort of an area that's a bit of a need-to-know basis for you. Are there, like, particular moments in your career that you remember like, oh, that's the moment where I needed to, like, take some time and learn a little bit of the next level of packages? STEPHANIE: That's a great question. I think the very beginnings of understanding how package versions work when you have multiple projects on your machine; I just remember that being really confusing for me. When I started out, like, you know, as soon as I cloned my second repo [laughs], and was very confused about, like, I'm sure I went through the process of not installing gems using Bundler, and then just having so much chaos [laughs] wrecked in my development environment and, you know, having to ask someone, "I don't understand how this works. Like, why is it saying I have multiple versions of this library or whatever?" JOËL: Have you ever sudo gem installed a gem? STEPHANIE: Oh yeah, I definitely have. I can't [laughs], like, even give a good reason for why I have done it, but I probably was just, like, pulling my hair out, and that's what Stack Overflow told me to do. I don't know if I can recommend that, but it is [chuckles] one thing to do when you just are kind of totally stuck. JOËL: There was a time where I think that that was in the READMEs for most projects. STEPHANIE: Yeah, that's a really good point. JOËL: So, that's probably why a lot of people end up doing that, but then it tends to install it for your system Ruby rather than for...because if you're using something like Rbenv or RVM or ASDF to manage multiple Ruby versions, those end up being what's using or even Homebrew to manage your Ruby. It wouldn't be installing it for those versions of Ruby. It would be installing it for the one that shipped with your Mac. I actually...you know what? I don't even know if Mac still ships with Ruby. It used to. It used to ship with a really old version of Ruby, and so the advice was like, "Hey, every repo tells you to install it with sudo; don't do that. It will mess you up." STEPHANIE: Huh. I think Mac still does ship with Ruby, but don't quote me on that [laughter]. And I think that's really funny that, like, yeah, people were just writing those instructions in READMEs. And I'm glad that we've collectively [laughs] figured out that difference and want to, hopefully, not let other developers fall into that trap [laughs]. Do you have a particular memory or experience when you had to kind of level up your knowledge about the package ecosystem? JOËL: I think one sort of moment where I really had to level up is when I started really needing to understand how install paths worked, especially when you have, let's say, multiple versions of a gem installed because you have different projects. And you want to know, like, how does it know which one it's using? And then you see, oh, there are different paths that point to different directories with the installs. Or when you might have an executable you've installed via Homebrew, and it's like, oh yeah, so I've got this, like, command that I run on my shell, but actually that points to a very particular path, you know, in my Homebrew directory. But maybe it could also point to some, like, pre-installed system binaries or some other custom things I've done. So, there was a time where I had to really learn about how the path shell variable worked on a machine in order to really understand how the packages I installed were sometimes showing up when I invoked a binary and sometimes not. STEPHANIE: Yeah, that is another really great example that I have memories of [laughs] being really frustrated by, especially if...because, you know, we had talked earlier about all the different ways that you can install applications on your system, and you don't always know where they end up [laughs]. JOËL: And this particular memory is tied to debugging Postgres because, you know, you're installing Postgres, and some paths aren't working. Or maybe you try to update Postgres and now it's like, oh, but, like, I'm still loading the wrong one. And why does PSQL not do the thing that I think it does? And so, that forced me to learn a little bit about, like, under the hood, what happens when I type brew install PostgreSQL? And how does that mesh with the way my shell interprets commands and things like that? So, it was maybe a little bit of a painful experience but eye-opening and definitely then led to me, I think, being able to debug my setup much more effectively in the future. STEPHANIE: Yeah. I like that you also pointed out how it was interacting with your shell because that's, like, another can of worms, right? [laughs] In terms of just the complexity of how these things are talking to each other. JOËL: And for those of our listeners who are not familiar with this, there is a shell command that you can use called which, W-H-I-C-H. And you can prefix that in front of another command, and it will tell you the path that it's using for that binary. So, in my case, if I'm looking like, why is this PSQL behaving weirdly or seems to be using the old version, I can type 'which space psql', and it'll say, "Oh, it's going to this path." And I can look at it and be like, oh, it's using my system install of Postgres. It's not using the Homebrew one. Or, oh, maybe it's using the Homebrew install, not my Postgres.app version. I need to, like, tinker with the paths a little bit. So, that has definitely helped me debug my package system more than once. STEPHANIE: Yeah, that's a really good tip. I can recall just totally uninstalling everything [laughs] and reinstalling and fingers crossed it would figure out a route to the right thing [laughs]. JOËL: You know what? That works. It's not the, like, most precise solution but resetting your environment when all else fails it's not a bad solution. So, we've been talking a lot about what it's like to interact with a package ecosystem as developers, as users of packages, but what if you're a package developer? Sometimes, there's a very clear-cut place where to publish, and sometimes it's a little bit grayer. So, I could see, you know, I'm developing a database, and I want that to be on operating systems, probably should be a system-level package rather than a Ruby gem. But what if I'm building some kind of command line tool, and I write it in Ruby because I like writing Ruby? Should I publish that as a gem, or should I publish that as some kind of system package that's installed via Homebrew? Any opinions or heuristics that you would use to choose where to publish on one side or the other? STEPHANIE: As not a package developer [laughs], I can only answer from that point of view. That is interesting because if you publish on a, you know, like, a system repository, then yeah, like, you might get a lot more people using your tool out there because you're not just targeting a specific language's community. But I don't know if I have always enjoyed downloading various things to my system's OS. I think that actually, like, is a bit complicated for me or, like, I try to avoid it if I can because if something can be categorized or, like, containerized in a way that, like, feels right for my mental model, you know, if it's written in Ruby or something really related to things I use Ruby in, it could be nice to have that installed in my, like, systems RubyGems. But I would be really interested to hear if other people have opinions about where they might want to publish a package and what kind of developers they're hoping to find to use their tool. JOËL: I like the heuristic that you mentioned here, the idea of who the audience is because, yeah, as a Ruby developer who already has a Ruby setup, it might be easier for me to install something via a gem. But if I'm not a Ruby developer who wants to use the packages maybe a little bit more generic, you know, let's say, I don't know, it's some sort of command line tool for interacting with GitHub or something like that. And, like, it happens to be written in Ruby, but you don't particularly care about that as a user of this. Maybe you don't have Ruby installed and now you've got to, like, juggle, like, oh, what is RubyGems, and Bundler, and all this stuff? And I've definitely felt that occasionally downloading packages sort of like, oh, this is a Python package. And you're going to need to, like, set up all this stuff. And it's maybe designed for a Python audience. And so, it's like, oh, you're going to set up a virtual environment and all these things. I'm like, I just want your command line tools. I don't want to install a whole language. And so, sometimes there can be some frustration there. STEPHANIE: Yeah, that is very true. Before you even said that, I was like, oh, I've definitely wanted to download a command line tool and be like, first install [laughs] Python. And I'm like, nope, I'm bailing out of this. JOËL: On the other hand, as a developer, it can be a lot harder to write something that's a bit more cross-platform and managing all that. And I've had to deal a little bit with this for thoughtbot's Parity tool, which is a command-line tool for working with Heroku. It allows you to basically run commands on either staging or production by giving you a staging command and a production command for common Heroku CLI tasks, which makes it really nice if you're working and you're having to do some local, some development, some staging, and some production things all from your command line. It initially started as a gem, and we thought, you know what? This is mostly command line, and it's not just Rubyists who use Heroku. Let's try to put this on Homebrew. But then it depends on Ruby because it's written in Ruby. And now we had to make sure that we marked Ruby as a dependency in Homebrew, which meant that Homebrew would then also pull in Ruby as a dependency. And that got a little bit messy. For a while, we even experimented with sort of briefly available technology called Traveling Ruby that allowed you to embed Ruby in your binary, and you could compile against that. That had some drawbacks. So, we ended up rolling that back as well. And eventually, just for maintenance ease, we went back to making this a Ruby gem and saying, "Look, you install it via RubyGems." It does mean that we're targeting more of the Ruby community. It's going to be a little bit harder for other people to install, but it is easier for us to maintain. STEPHANIE: That's really interesting. I didn't know that history about Parity. It's a tool that I have used recently and really enjoyed. But yeah, I think I remember someone having some issues between installing it as a gem and installing it via Homebrew and some conflicts there as well. So, I can also see how trying to decide or maybe going down one path and then realizing, oh, like, maybe we want to try something else is certainly not trivial. JOËL: I think, in me, I have a little bit of the idealist and the pragmatist that fight. The idealist says, "Hey, if it's not, like, aimed for Ruby developers as a, like, you can pull this into your codebase, if it's just command line tools and the fact that it's written in Ruby is an implementation detail, that should be a system package. Do not distribute binaries via RubyGems." That's the idealist in me. The pragmatist says, "Oh, that's a lot of work and not always worth it for both the maintainers and sometimes for the users, and so it's totally okay to ship binaries as RubyGems." STEPHANIE: I was totally thinking that I'm sure that you've been in that position of being a user and trying to download a system package and then seeing it start to download, like, another language. And you're like, wait, what? [laughter] That's not what I want. JOËL: So, you and I have shared some of our heuristics in the way we approach this problem. Now, I'm curious to hear from the audience. What are some heuristics that you use to decide whether your package is better shipped on RubyGems versus, let's say, Homebrew? Or maybe as a user, what do you prefer to consume? STEPHANIE: Yes. And speaking of getting listener feedback, we're also looking for some listener questions. We're hoping to do a bit of a grab-bag episode where we answer your questions. So, if you have anything that you're wanting to hear me and Joël's thoughts on, write us at hosts@bikeshed.fm. JOËL: On that note, shall we wrap up? STEPHANIE: Let's wrap up. Show notes for this episode can be found at bikeshed.fm. JOËL: This show has been produced and edited by Mandy Moore. STEPHANIE: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review in iTunes. It really helps other folks find the show. JOËL: If you have any feedback for this or any of our other episodes, you can reach us @_bikeshed, or you can reach me @joelquen on Twitter. STEPHANIE: Or reach both of us at hosts@bikeshed.fm via email. JOËL: Thanks so much for listening to The Bike Shed, and we'll see you next week. ALL: Byeeeeeeee!!!!!!! AD: Did you know thoughtbot has a referral program? If you introduce us to someone looking for a design or development partner, we will compensate you if they decide to work with us. More info on our website at: tbot.io/referral. Or you can email us at referrals@thoughtbot.com with any questions.

speaking management tales os mac developers package python linux github snowflakes javascript macos rails ubuntu homebrew docker elm parity mandy moore stack overflow json cli npm postgresql heroku xcode postgres quenneville tuple bundler rubygems rvm activerecord asdf odbc rubyists stephanie yeah stephanie oh stephanie they

Single Table Inheritance (STI) with ActiveRecord in Rails

GoRails Screencasts

Play Episode Listen Later Jan 22, 2024

Teaching Code, Shaping Futures — John Crepezzi on Ruby, Bootcamps, and AI

Ruby for All

Play Episode Listen Later Jan 4, 2024 33:54

In this episode, Andrew and Julie chat with guest John Crepezzi, a veteran in the Ruby community, founder of All Aboard Bootcamp, and currently a Software Engineer at Jane Street. Today, they discuss John's experience running a coding bootcamp, share insights on teaching programming, and emphasize the importance of community in learning. Additionally, they explore functional programming in OCaml, highlighting how functional programming can be implemented in Ruby. Also, John dives into the potential impact of AI, particularly language models like ChatGPT, on education and software development, and there's a discussion on resume writing for new coders and future trends in AI and automation. Hit download now to hear more! [00:01:08] Julie introduces John, her former bootcamp instructor, and he tells us about himself and his extensive experience on the Ruby community. [00:02:14] Andrew asks John about the gem he is most proud of creating, and he explains his pride in the “ice_cube gem” for recurring date math. [00:04:30] John discusses the technical challenges and community contributions to ice_cube. [00:05:25] Julie discusses her positive experience with All Aboard Bootcamp and how the bootcamp helped her connect different programming concepts. [00:07:26] John describes his teaching philosophy for covering a broad amount of material quickly and he stresses the importance of learning to ask the right questions. He uses a metaphor from the movie “Tommy Boy” to emphasize teaching practical skills.[00:11:26] John relates the approach to teaching with the usefulness of ChatGPT and Julie expresses her preference for receiving explanations in small chunks and using bullet points for clarity. John discusses how LLMs can assist in refining questions before providing answers. [00:12:49] Andrew asks about AI's role in teaching and its potential impact. John, an AI professional, offers his perspective on AI in the short and term, specifically its ability to understand and respond to human language. He speculates on the future of human computer interaction, where structured systems may become unnecessary as LLMs bridge the communication gap. [00:16:03] Andrew agrees with John's vision of the future, acknowledging the inefficiencies in current user flows. John compares the evolution from programming VCRs to using DVRs to the potential of LLMs simplifying interaction with technology. [00:16:55] John describes the motivation behind starting a bootcamp and the realization of the industry's selection bias towards already skilled programmers. He shares the story of how the high cost of bootcamps and their screening processes inspired him to teach a more accessible camp. [00:21:10] Julie is impressed by John's ability to manage the bootcamp alongside his full-time job, family responsibilities, and other commitments. She also talks about the final project of the bootcamp, where John acted as a project manager and provided structure and guidance. [00:23:31] Andrew inquires about what John thinks is the number one mistake new programmers make on their resume. John emphasizes the importance of highlighting projects on a resume, especially for those transitioning from another industry, and advises focusing on the outcomes and transferable skills gained from previous experiences.[00:25:35] John considers formatting critical for resumes, suggesting less content with more white space and a clear hierarchy can be more effective than too much information.[00:26:44] Another thing John advises is keeping resumes to one page unless there's a compelling reason for more, like academic positions or extensive project work.[00:27:18] Reflecting on the bootcamp, John wishes he had sought more assistance with grading and feedback to reduce the workload. [00:28:34] John praises the students, particularly Julie, for fostering a supportive community outside of the classroom.[00:31:07] Discussing programming languages, John expresses his favor for OCaml and functional programming, arguing that functional patterns can be beneficial even in languages like Ruby. [00:32:13] Find out where you can follow John online. [00:32:58] We end with John reaffirming his love for Ruby and expresses enthusiasm for its future and mentioning his work with Eileen on Active Record and Rails' influence on web frameworks in other languages. Panelists:Andrew MasonJulie J.Guest:John CrepezziSponsors:GoRailsHoneybadgerLinks:Andrew Mason X/TwitterAndrew Mason WebsiteJulie J. X/TwitterJulie J. WebsiteJohn Crepezzi X/TwitterJohn Crepezzi GitHubice_cube 0.16.4All Aboard BootcampAbout the bootcamp (John's story)RailsConf 2023-Functional Patterns in Ruby by John Crepezzi (YouTube)OCaml (00:00) - Introduction to the Episode (01:08) - Meet John Crepezzi: Ruby Community Veteran and Bootcamp Founder (02:14) - John's Pride in Creating the "ice_cube gem" for Date Math (04:30) - Technical Challenges and Community Role in Developing ice_cube (05:25) - Julie's Transformative Experience at All Aboard Bootcamp (07:26) - John's Teaching Philosophy: Quick Learning and Practical Skills (11:26) - ChatGPT's Role in Teaching: Enhancing Question Refinement (12:49) - AI in Education: John's Perspective on Future Trends (16:03) - From VCRs to AI: Evolution of User-Technology Interaction (16:55) - The Genesis of John's Bootcamp: Addressing Industry Biases (21:10) - John's Journey: Balancing Bootcamp with Personal Life (23:31) - Common Resume Mistakes for New Programmers (25:35) - John's Tips on Effective Resume Formatting (26:44) - Importance of Conciseness in Resumes (27:18) - John Reflects on Bootcamp Challenges and Workload Management (28:34) - Fostering a Supportive Community in the Bootcamp (31:07) - John's Advocacy for OCaml and Functional Programming in Ruby (32:13) - Discover Where to Follow John Online (32:58) - John's Ongoing Passion for Ruby and Its Evolving Impact

410: All About Documentation

Smart Software with SmartLogic

Play Episode Listen Later Dec 12, 2023 32:02

Joël shares his experiences with handling JSON in a Postgres database. He talks about his challenges with ActiveRecord and JSONB columns, particularly the unexpected behavior of storing and retrieving JSON data. Stephanie shares her recent discovery of bookmarklets and highlights a bookmarklet named "Check This Out," which streamlines searching for books on Libby, an ebook and audiobook lending app. The conversation shifts to using constants in code as a form of documentation. Stephanie and Joël discuss how constants might not always accurately reflect current system behavior or logic, leading to potential misunderstandings and the importance of maintaining accurate documentation. Bookmarklets (https://www.freecodecamp.org/news/what-are-bookmarklets/) "Check This Out" Bookmarklet (https://checkthisout.today/) Libby (https://libbyapp.com/interview/welcome#doYouHaveACard) Productivity Tricks (https://www.bikeshed.fm/403) 12 Factor App Config (https://12factor.net/config) A Hierarchy of Documentation (https://challahscript.com/hiearchy_of_documentation) Sustainable Rails (https://sustainable-rails.com/) rails-erd gem (https://github.com/voormedia/rails-erd) Transcript STEPHANIE: Hello and welcome to another episode of the Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Stephanie Minn. JOËL: And I'm Joël Quenneville. And together, we're here to share a bit of what we've learned along the way. STEPHANIE: So, Joël, what's new in your world? JOËL: What's new in my world is JSON and how to deal with it in a Postgres database. So, I'm dealing with a situation where I have an ActiveRecord model, and one of the columns is a JSONB column. And, you know, ActiveRecord is really nice. You can just throw a bunch of different data at it, and it knows the column type, and it will do some conversions for you automatically. So, if I'm submitting a form and, you know, form values might come in as strings because, you know, I typed in a number in a text field, but ActiveRecord will automatically parse that into an integer because it knows we're saving that to an integer column. So, I don't need to do all these, like, manual conversions. Well, I have a form that has a string of JSON in it that I'm trying to save in a JSONB column. And I expected ActiveRecord to just parse that into a hash and store it in Postgres. That is not what happens. It just stores a raw string, so when I pull it out again, I don't have a hash. I have a raw string that I need to deal with. And I can't query it because, again, it is a raw string. So, that was a bit of an unexpected behavior that I saw there. STEPHANIE: Yeah, that is unexpected. So, is this a field that has been used for a while now? I'm kind of surprised that there hasn't been already some implementations for, like, deserializing it. JOËL: So, here's the thing: I don't think you can have an automatic deserialization there because there's no way of knowing whether or not you should be deserializing. The reason is that JSON is not just objects or, in Ruby parlance, hashes. You can also have arrays. But just raw numbers not wrapped in hashes are also valid JSON as are raw strings. And if I just give you a string and say, put this in a JSON field, you have no way of knowing, is this some serialized JSON that you need to deserialize and then save? Or is it just a string that you should save because strings are already JSON? So, that's kind of on you as the programmer to make that distinction because you can't tell at runtime which one of these it is. STEPHANIE: Yeah, you're right. I just realized it's [laughs] kind of, like, an anything goes [laughs] situation, not anything but strings are JSON, are valid JSON, yep [laughs]. That sounds like one of those things that's, like, not what you think about immediately when dealing with that kind of data structure, but... JOËL: Right. So, the idea that strings are valid JSON values, but also all JSON values can get serialized as strings. And so, you never know: are you dealing with an unserialized string that's just a JSON value, or are you dealing with some JSON blob that got serialized into a string? And only in one of those do you want to then serialize before writing into the database. STEPHANIE: So, have you come to a solution or a way to make your problem work? JOËL: So, the solution that I did is just calling a JSON parse before setting that attribute on my model because this value is coming in from a form. I believe I'm doing this when I'm defining the strong parameters for that particular form. I'm also transforming that string by parsing it into a hash with the JSON dot parse, which then gets passed to the model. And then I'm not sure what JSONB serializes as under the hood. When you give it a hash, it might store it as a string, but it might also have some kind of binary format or some internal AST that it uses for storage. I'm not sure what the implementation is. STEPHANIE: Are the values in the JSONB something that can be variable or dynamic? I've seen some people, you know, put that in getter so that it's just kind of done for you for anyone who needs to access that field. JOËL: Right now, there is a sort of semi-consistent schema to that. I think it will probably evolve to where I'll pull some of these out to be columns on the table. But it is right now kind of an everything else sort of dumping ground from an API. STEPHANIE: Yeah, that's okay, too, sometimes [laughs]. JOËL: Yeah. So, interesting journey into some of the fun edge cases of dealing with a format whose serialized form is also a valid instance of that format. What's been new in your world? STEPHANIE: So, I discovered something new that has been around on the internet for a while, but I just haven't been aware of it. Do you know what a bookmarklet is? JOËL: Oh, like a JavaScript code that runs in a bookmark? STEPHANIE: Yeah, exactly. So, you know, in your little browser bookmark where you might normally put a URL, you can actually stick some JavaScript in there. And it will run whenever you click your bookmark in your browser [chuckles]. So, that was a fun little internet tidbit that I just found out about. And the reason is because I stumbled upon a bookmarklet made by someone. It's called Check This Out. And what it does is there's another app/website called Libby that is used to check out ebooks and audiobooks for free from your local public library. And what this Check This Out bookmarklet does is you can kind of select any just, like, text on a web page, and then when you click the bookmarklet, it then just kind of sticks it into the query params for Libby's search engine. And it takes you straight to the results for that book or that author, and it saves you a few extra manual steps to go from finding out about a book to checking it out. So, that was really neat and cute. And I was really surprised that you could do that. I was like, whoa [laughs]. At first, I was like, is this okay? [laughs] If you, like, you can't read, you know, you don't know what the JavaScript is doing, I can see it being a little sketchy. But –- JOËL: Be careful of executing arbitrary JavaScript. STEPHANIE: Yeah, yeah. When I did look up bookmarklets, though, I kind of saw that it was, you know, just kind of a fun thing for people who might be learning to code for the first time to play around with. And some fun ideas they had for what you could do with it was turning all the font on a web page to Comic Sans [laughs]. So yeah, I thought that was really cute. JOËL: Has that inspired you to write your own? STEPHANIE: Well, we did an episode a while ago on productivity tricks. And I was thinking like, oh yeah, there's definitely some things that I could do to, you know, just stick some automated tasks that I have into a bookmarklet. And that could be a really fun kind of, like, old-school way of doing it, as opposed to, you know, coding my little snippets or getting into a new, like, Omnibar app [laughs]. JOËL: So, something that is maybe a little bit less effort than building yourself a browser extension or something like that. STEPHANIE: Yeah, exactly. JOËL: I had a client project once that involved a...I think it was, like, a five-step wizard or something like that. It was really tedious to step through it all to manually test things. And so, I wrote a bookmarklet that would just go through and fill out all the fields and hit submit on, like, five pages worth of these things. And if anything didn't work, it would just pause there, and then you could see it. In some way, it was moving towards the direction of, like, an automated like Capybara style test. But this was something that was helping for manual QA. So, that was a really fun use of a bookmarklet. STEPHANIE: Yeah, I like that. Like, just an in-between thing you could try to speed up that manual testing without getting into, like you said, an automated test framework for your browser. JOËL: The nice thing about that is that this could be used without having to set up pretty much anything, right? You paste a bit of JavaScript into your bookmark bar, and then you just click the button. That's all you need to do. No need to make sure that you've got Ruby installed on your machine or any of these other things that you would need for some kind of testing framework. You don't need Selenium. You don't need ChromeDriver. It just...it works. So, I was working...this was a greenfield startup project. So, I was working with a non-technical founder who didn't have all these things, you know, dev tooling on his machine. So, he wanted to try out things but not spend his days filling out forms. And so, having just a button he could click was a really nice shortcut. STEPHANIE: That's really cool. I like that a lot. I wasn't even thinking about how I might be able to bring that in more into just my daily work, as opposed to just something kind of fun. But that's an awesome idea. And I hope that maybe I'll have a good use for one in the future. JOËL: It feels like the thing that has a lot of potential, and yet I have not since written...I don't think I've written any bookmarklets for myself. It feels like it's the kind of thing where I should be able to do this for all sorts of fun tooling and just automate my life away. Somehow, I haven't done that. STEPHANIE: Bring back the bookmarklet [laughs]. That's what I have to say. JOËL: So, I mentioned earlier that I was working with a JSONB column and storing JSON on an ActiveRecord model. And then I wanted to interact with it, but the problem is that this JSON is somewhat arbitrary, and there are a lot of magic strings in there. All of the key names might change. And I was really concerned that if the schema of that JSON ever changed, if we changed some of the key names or something like that, we might accidentally break code in multiple parts of the app. So, I was very careful while building that model to quarantine any references to any raw strings only within that model, which meant that I leaned really heavily on constants. And, in some way, those constants end up kind of documenting what we think the schema of that JSON should be. And that got me thinking; you were telling me recently about a scenario where some code you were working with relied heavily on constants as a form of documentation, and that documentation kind of lied to you. STEPHANIE: Yeah, it did. And I think you mentioned something that I wanted to point out, which is that the magic strings that you think might change, and you wanted to pull that out into a constant, you know, so at least it's kind of defined in one place. And if it ever does change, you know, you don't have to change it in all of those places. And I do think that, normally, you know, if there's opportunities to extract those magic strings and give a name to them, that is beneficial. But I was gripping a little bit about when constants become, I guess, like, too wieldy, or there's just kind of, like, too much of a dependency on them as the things documenting how the app should work when it's constantly changing. I realized that I just used constant and constantly [laughs]. JOËL: The only constant is that it is not constant. STEPHANIE: Right. And so, the situation that I found myself in—this was on a client project a little bit ago—was that the constants became, like, gatekeepers of that logic where dev had to change it if the app's behavior changed, and maybe we wanted to change the value of it. And also, one thing that I noticed a lot was that we, as developers, were getting questions about, "Hey, like, how does this actually work?" Like, we were using the constants for things like pricing of products, for things like what is a compatible version for this feature. And because that was only documented in the code, other people who didn't have access to it actually were left in the dark. And because those were changing with somewhat frequency, I was just kind of realizing how that was no longer working for us. JOËL: Would you say that some of these values that we stored as constants were almost more like config rather than constants or maybe they're just straight-up application data? I can imagine something like price of an item you probably want that to be a value in the database that can be updated by an admin. And some of these other things maybe are more like config that you change through some kind of environment variable or something like that. STEPHANIE: Yeah, that's a good point. I do think that they evolved to become things that needed to be configured, right? I suppose maybe there wasn't as much information or foresight at the beginning of like, oh, this is something that we expect to change. But, you know, kind of when you're doing that first pass and you're told, like, hey, like, this value should be the price of something, or, like, the duration of something, or whatever that may be. It gets codified [chuckles]. And there is some amount of lift to change it from something that is, at first, just really just documenting what that decision was at the time to something that ends up evolving. JOËL: How would you draw a distinction between something that should be a constant versus something that maybe would be considered config or some other kind of value? Because it's pretty easy, right? As developers, we see magic numbers. We see magic strings. And our first thought is, oh, we've seen this problem before—constant. Do you have maybe a personal heuristic for when to reach for a constant versus when to reach for something else? STEPHANIE: Yeah, that's a good question. I think when I started to see it a lot was especially when the constants were arrays or hashes [laughs]. And I guess that is actually kind of a signal, right? You will likely be adding more stuff [laughs] into that data structure [laughs]. And, again, like, maybe it's okay, like, the first couple of times. But once you're seeing that request happen more frequently, that could be a good way to advocate for storing it in the database or, like, building a lightweight admin kind of thing so that people outside of the dev team can make those configuration changes. I think also just asking, right? Hey, like, how often do we suspect this will change? Or what's on the horizon for the product or the team where we might want to introduce a way to make the implementation a bit more flexible to something that, you know, we think we know now, but we might want to adjust for? JOËL: So, it's really about change and how much we think this might change in the future. STEPHANIE: Speaking of change, this actually kind of gets into the broader topic of documentation and how to document a changing and evolving entity [chuckles], you know, that being, like, the codebase or the way that decisions are made that impact how an application works. And you had shared, in preparation for this topic, an article that I read and enjoyed called Hierarchy of Documentation. And one thing that I liked about it is that it kind of presented all of the places that you could put information from, you know, straight in the code, to in your commit messages, to your issue management system, and to even wikis for your repo or your team. And I think that's actually something that we would want to share with new developers, you know, who might be wondering, like, where do I find or even put information? I really liked how it was kind of, like, laid out and gave, like, different reasons for where you might want to put something or not. JOËL: We think a lot about documentation as code writers. I'm curious what your experience is as a code reader. How do you tend to try to read code and understand documentation about how code works? And, apparently, the answer is, don't read the constants because these constants lie. STEPHANIE: I think you are onto something, though, because I was just thinking about how distrustful I've become of certain types of documentation. Like, when I think of code comments, on one hand, they should be a signal, right? They should kind of draw your attention to something maybe weird or just, like, something to note about the code that it's commenting on, or where it's kind of located in a file. But I sometimes tune them out, I'm not going to lie. When I see a really big block of code [chuckles] comment, I'm like, ugh, like, do I really have to read all of this? I'm also not positive that it's still relevant to the code below it, right? Like, I don't always have git blame, like, visually enabled in my editor. But oftentimes, when I do a little bit of digging, that comment is left over from maybe when that code was initially introduced. But, man, there have been lots of commits [chuckles] in the corresponding, you know, like, function sense, and I'm not really sure how relevant it is anymore. Do you struggle with the signal versus noise issue with code comments? How much do you trust them, and how much do you kind of, like, give credence to them? JOËL: I think I do tend to trust them with maybe some slight skepticism. It really depends on the codebase. Some codebases are really bad sort of comment hygiene and just the types of comments that they put in there, and then others are pretty good at it. The ones that I tend to particularly appreciate are where you have maybe some, like, weird function and you're like, what is going on here? And then you've got a nice, little paragraph up top explaining what's going on there, or maybe an explanation of ways you might be tempted to modify that piece of code and, like, why it is the way it is. So, like, hey, you might be wanting to add an extra branch here to cover this edge case. Don't do that. We tried it, and it causes problems for XY reasons. And sometimes it might be, like, a performance thing where you say, look, the code quality person in you is going to look at this and say, hey, this is hard to read. It would be better if we did this more kind of normalized form. Know that we've particularly written this in a way that's hard to read because it is more performant, and here are the numbers. This is why we want it in this way. Here's a link to maybe the issue, or the commit, or whatever where this happened. And then if you want to start that discussion up again and say, "Hey, do we really need performance here at the cost of readability?", you can start it up again. But at least you're not going to just be like, oh, while I'm here, I'm going to clean up this messy code and accidentally cause a regression. STEPHANIE: Yeah. I like what you said about comment hygiene being definitely just kind of, like, variable depending on the culture and the codebase. JOËL: I feel like, for myself, I used to be pretty far on the spectrum of no comments. If I feel the need to write a comment, that's a smell. I should find other ways to communicate that information. And I think I went pretty far down that extreme, and then I've been slowly kind of coming back. And I've probably kind of passed the center, where now I'm, like, slightly leaning towards comments are actually nice sometimes. And they are now a part of my toolkit. So, we'll see if I keep going there. Maybe I'll hit some point where I realize that I'm putting too much work into comments or comments are not being helpful, and I need to come back towards the center again and focus on other ways of communicating. But right now, I'm in that phase of doing more comments than I used to. How about you? Where do you stand on that sort of spectrum of all information should be communicated in code tokens versus comments? STEPHANIE: Yeah, I think I'm also somewhere in the middle. I think I have developed an intuition of when it feels useful, right? In my gut, I'm like, oh, I'm doing something weird. I wish I didn't have to do this [chuckles]. I think it's another kind of intuition that I have now. I might leave a comment about why, and I think that is more of that signal, right? Though I also recently have been using them more as just, like, personal notes for myself as I'm, you know, in my normal development workflow, and then I will end up cleaning them up later. I was working on a codebase where there was a soft delete functionality. And that was just, like, a concern that was included in some of the models. And I didn't realize that that's what was going on. So, when I, you know, I was calling destroy, I thought it was actually being deleted, and it turns out it wasn't. And so, that was when I left a little comment for myself that was like, "Hey, like, this is soft deleted." And some of those things I do end up leaving if I'm like, yes, other people won't have the same context as me. And then if it's something that, like, well, people who work in this app should know that they have soft delete, so then I'll go ahead and clean that up, even though it had been useful for me at the time. JOËL: Do you capture that information and then put it somewhere else then? Or is it just it was useful for you as a stepping-stone on the journey but then you don't need it at the end and nobody else needs to care about it? STEPHANIE: Oh, you know what? That's actually a really great point. I don't think I had considered saving that information. I had only thought about it as, you know, just stuff for me in this particular moment in time. But that would be really great information to pull out and put somewhere else [chuckles], perhaps in something like a wiki, or like a README, or somewhere that documents things about the system as a whole. Yeah, should we get into how to document kind of, like, bigger-picture stuff? JOËL: How do you feel about wikis? Because I feel like I've got a bit of a love-hate relationship with them. STEPHANIE: I've seen a couple of different flavors of them, right? Sometimes you have your GitHub wiki. Sometimes you have your Confluence ecosystem [laughs]. I have found that they work better if they're smaller [laughs], where you can actually, like, navigate them pretty well, and you have a sense of what is in there, as opposed to it just being this huge knowledge base that ends up actually, I think, working against you a little bit [laughs]. Because so much information gets duplicated if it's hard to find and people start contributing to it maybe without keeping in mind, like, the audience, right? I've seen a lot of people putting in, like, their own personal little scripts [laughs] in a wiki, and it works for them but then doesn't end up working for really anyone else. What's your love-hate relationship to them? JOËL: I think it's similar to what you were saying, a little bit of structure is nice. When they've just become dumping grounds of information that is maybe not up to date because over the course of several years, you end up with a lot of maybe conflicting articles, and you don't know which one is the right thing to do, it becomes hard to find things. So, when it just becomes a dumping ground for random information related to the company or the app, sometimes it becomes really challenging to find the information I need and to find information that's relevant, to the point where oftentimes looking something up in the wiki is my last resort. Like, I'm hoping I will find the answer to my question elsewhere and only fallback to the wiki if I can't. STEPHANIE: Yeah, that's, like, the sign that the wiki is really not trustworthy. And it kind of is diminishing returns from there a bit. I think I fell into this experience on my last project where it was a really, really big wiki for a really big codebase for a lot of developers. And there was kind of a bit of a tragedy of the commons situation, where on one hand, there were some things that were so manual that the steps needed to be very explicitly documented, but then they didn't work a lot of the time [laughs]. But it was hard to tell if they weren't working for you or because it was genuinely something wrong with, like, the way the documentation laid out the steps. And it was kind of like, well, I'm going to fix it for myself, but I don't know how to fix it for everyone else. So, I don't feel confident in updating this information. JOËL: I think that's what's really nice about the article that you mentioned about the hierarchy of documentation. It's that all of these different forms—code, comments, commit messages, pull requests, wikis—they don't have to be mutually exclusive. But sometimes they work sort of in addition to each other sort of each adding more context. But also, sometimes it's you sort of choose the one that's the highest up on that list that makes sense for what you're trying to do, so something like documenting a series of steps to do something maybe a wiki is a good place for that. But maybe it's better to have that be executable. Could that be a script somewhere? And then maybe that can be a thing that is almost, like, living documentation, but also where you don't need to maybe even think about the individual steps anymore because the script is running, you know, 10 different things. And I think that's something that I really appreciated from the book Sustainable Rails is there's a whole section there talking about the value of setup scripts and how people who are getting started on your app don't want to have to care about all the different things to set it up, just run a script. And also, that becomes living documentation for what the app needs, as opposed to maybe having a bulleted list with 10 elements in it in your project README. STEPHANIE: Yeah, absolutely. In the vein of living documentation, I think one thing that wikis can be kind of nice for is for putting visual supplements. So, I've seen them have, like, really great graphs. But at the same time, you could use a gem like Rails-ERD that generates the entity relationship diagram as the schema of your database changes, right? So, it's always up to date. I've seen that work well, too, when you want to have, like I said, those, like, system-level documentation that sometimes they do change frequently and, you know, sometimes they don't. But that's definitely worth keeping in mind when you choose, like, how you want to have that exist as information. JOËL: How do you feel about deleting documentation? Because I feel like we put so much work into writing documentation, kind of like we do when writing tests. It feels like more is always better. Do you ever go back and maybe sort of prune some of your docs, or try to delete some things that you think might no longer be relevant or helpful? STEPHANIE: I was also thinking of tests when you first posed that question. I don't know if I have it in my practice to, like, set aside time and be like, hmm, like, what looks outdated these days? I am starting to feel more confident in deleting things as I come across them if I'm like, I just completely ignored this or, like, this was just straight up wrong [laughs]. You know, that can be scary at first when you aren't sure if you can make that determination. But rather than thrust that, you know, someone else going through that same process of spending time, you know, trying to think about if this information was useful or not, you can just delete it [laughs]. You can just delete tests that have been skipped for months because they don't work. Like, you can delete information that's just no longer relevant and, in some ways, causing you more pain because they are cluttering up your wiki ecosystem so that no one [laughs] feels that any of that information is relevant anymore. JOËL: I'll be honest, I don't think I've ever deleted a wiki article that was out of date or no longer relevant. I think probably the most I've done is go to Slack and complain about how an out-of-date wiki page led me down the wrong path, which is probably not the most productive way to channel those feelings. So, maybe I should have just gone back and deleted the wiki page. STEPHANIE: I do like to give a heads up, I think. It's like, "Hey, I want to delete this thing. Are there any qualms?" And if no one on your team can see a reason to keep it and you feel good about that it's not really, like, serving its purpose, I don't know, maybe consider just doing it. JOËL: To kind of wrap up this topic, I've got a spicy question for you. STEPHANIE: Okay, I'm ready. JOËL: Do you think that AI is going to radically change the way that we interact with documentation? Imagine you have an LLM that you train on maybe not just your code but the Git history. It has all the Git comments and maybe your wiki. And then, you can just ask it, "Why does function foo do this thing?" And it will reference a commit message or find the correct wiki article. Do you think that's the future of understanding codebases? STEPHANIE: I don't know. I'm aware that some people kind of can see that as a use case for LLMs, but I think I'm still a little bit nervous about the not knowing how they got there kind of part of it where, you know, yes, like I am doing this manual labor of trying to sort out, like, is this information good or trustworthy or not? But at least that is something I'm determining for myself. So, that is where my skepticism comes in a little bit. But I also haven't really seen what it can do yet or seen the outcomes of it. So, that's kind of where I'm at right now. JOËL: So, you think, for you, the sort of the journey of trying to find and understand the documentation is a sort of necessary part of building the understanding of what the code is doing. STEPHANIE: I think it can be. Also, I don't know, maybe my life would be better by having all that cut out for me, or I could be burned by it because it turns out that it was bad information [laughs]. So, I can't say for sure. On that note, shall we wrap up? JOËL: Let's wrap up. STEPHANIE: Show notes for this episode can be found at bikeshed.fm. JOËL: This show has been produced and edited by Mandy Moore. STEPHANIE: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review in iTunes. It really helps other folks find the show. JOËL: If you have any feedback for this or any of our other episodes, you can reach us @_bikeshed, or you can reach me @joelquen on Twitter. STEPHANIE: Or reach both of us at hosts@bikeshed.fm via email. JOËL: Thanks so much for listening to The Bike Shed, and we'll see you next week. ALL: Byeeeeeeee!!!!!! AD: Did you know thoughtbot has a referral program? If you introduce us to someone looking for a design or development partner, we will compensate you if they decide to work with us. More info on our website at: tbot.io/referral. Or you can email us at: referrals@thoughtbot.com with any questions.

ai slack api hierarchy github qa javascript documentation llm ast git mandy moore confluence xy json selenium comic sans readme postgres capybara check this out quenneville activerecord stephanie yeah stephanie well stephanie oh stephanie right

Web Development Frameworks: Elixir and Phoenix vs. Ruby on Rails with Owen Bickford & Dan Ivovich

Play Episode Listen Later Dec 7, 2023 41:41

On today's episode, Elixir Wizards Owen Bickford and Dan Ivovich compare notes on building web applications with Elixir and the Phoenix Framework versus Ruby on Rails. They discuss the history of both frameworks, key differences in architecture and approach, and deciding which programming language to use when starting a project. Both Phoenix and Rails are robust frameworks that enable developers to build high-quality web apps—Phoenix leverages functional programming in Elixir and Erlang's networking for real-time communication. Rails follows object-oriented principles and has a vast ecosystem of plug-ins. For data-heavy CRUD apps, Phoenix's immutable data pipelines provide some advantages. Developers can build great web apps with either Phoenix or Rails. Phoenix may have a slight edge for new projects based on its functional approach, built-in real-time features like LiveView, and ability to scale efficiently. But, choosing the right tech stack depends heavily on the app's specific requirements and the team's existing skills. Topics discussed in this episode: History and evolution of Phoenix Framework and Ruby on Rails Default project structure and code organization preferences in each framework Comparing object-oriented vs functional programming paradigms CRUD app development and interaction with databases Live reloading capabilities in Phoenix LiveView vs Rails Turbolinks Leveraging WebSockets for real-time UI updates Testing frameworks like RSpec, Cucumber, Wallaby, and Capybara Dependency management and size of standard libraries Scalability and distribution across nodes Readability and approachability of object-oriented code Immutability and data pipelines in functional programming Types, specs, and static analysis with Dialyzer Monkey patching in Ruby vs extensible core language in Elixir Factors to consider when choosing between frameworks Experience training new developers on Phoenix and Rails Community influences on coding styles Real-world project examples and refactoring approaches Deployment and dev ops differences Popularity and adoption curves of both frameworks Ongoing research into improving Phoenix and Rails Links Mentioned in this Episode: SmartLogic.io (https://smartlogic.io/) Dan's LinkedIn (https://www.linkedin.com/in/divovich/) Owen's LinkedIn (https://www.linkedin.com/in/owen-bickford-8b6b1523a/) Ruby https://www.ruby-lang.org/en/ Rails https://rubyonrails.org/ Sams Teach Yourself Ruby in 21 Days (https://www.overdrive.com/media/56304/sams-teach-yourself-ruby-in-21-days) Learn Ruby in 7 Days (https://www.thriftbooks.com/w/learn-ruby-in-7-days---color-print---ruby-tutorial-for-guaranteed-quick-learning-ruby-guide-with-many-practical-examples-this-ruby-programming-book--to-build-real-life-software-projects/18539364/#edition=19727339&idiq=25678249) Build Your Own Ruby on Rails Web Applications (https://www.thriftbooks.com/w/build-your-own-ruby-on-rails-web-applications_patrick-lenz/725256/item/2315989/?utm_source=google&utm_medium=cpc&utm_campaign=low_vol_backlist_standard_shopping_customer_acquisition&utm_adgroup=&utm_term=&utm_content=593118743925&gad_source=1&gclid=CjwKCAiA1MCrBhAoEiwAC2d64aQyFawuU3znN0VFgGyjR0I-0vrXlseIvht0QPOqx4DjKjdpgjCMZhoC6PcQAvD_BwE#idiq=2315989&edition=3380836) Django https://github.com/django Sidekiq https://github.com/sidekiq Kafka https://kafka.apache.org/ Phoenix Framework https://www.phoenixframework.org/ Phoenix LiveView https://hexdocs.pm/phoenixliveview/Phoenix.LiveView.html#content Flask https://flask.palletsprojects.com/en/3.0.x/ WebSockets API https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API WebSocket connection for Phoenix https://github.com/phoenixframework/websock Morph Dom https://github.com/patrick-steele-idem/morphdom Turbolinks https://github.com/turbolinks Ecto https://github.com/elixir-ecto Capybara Testing Framework https://teamcapybara.github.io/capybara/ Wallaby Testing Framework https://wallabyjs.com/ Cucumber Testing Framework https://cucumber.io/ RSpec https://rspec.info/

history startups mvp engineers automation hackers saas developers programming algorithms react computer science ux makers agile coding ui builders github libraries information technology devops frameworks rails scrum kafka deployment scripts cucumbers mobile apps django elixir git technologists scalability software engineering web development flutter ci cd paas bootcamps ruby on rails web apps prototypes wallaby programming languages flask iaas technical debt crud ecto rest apis design patterns computer programming functional programming erlang sdlc user stories bickford pull requests distributed systems test driven development open source community clean code unit testing websockets code reviews liveview version control front end development compilers state management developer productivity data structures object oriented programming code quality activerecord dependency management sidekiq rspec integration testing phoenix framework multiprocessing immutable data monkey patching

015: Exploring Turbo 8 with Jorge Manrubia

The Rails Changelog

Play Episode Listen Later Nov 23, 2023 53:24

In this episode, Jorge Manrubia provides insights into his experience working at 37signals. Our conversation delves into the intricacies of Active Record encryption, and we explore the latest advancements in Turbo 8 technology.Jorge Manrubia's Book RecommendationPragmatic Thinking And Learning Refactoring Domain-Driven Design Smalltalk Best Practice Patterns Design Patterns TurboActive Record EncryptionSpektr Security

technology web programming turbo rails activerecord

David Copeland on Medium Sized Decisions (RubyConf 2023)

Software Sessions

Play Episode Listen Later Nov 17, 2023 48:33

David was the chief software architect and director of engineering at Stitch Fix. He's also the author of a number of books including Sustainable Web Development with Ruby on Rails and most recently Ruby on Rails Background Jobs with Sidekiq. He talks about how he made decisions while working with a medium sized team (~200 developers) at Stitch Fix. The audio quality for the first 19 minutes is not great but the correct microphones turn on right after that. Recorded at RubyConf 2023 in San Diego. A few topics covered: Ruby's origins at Stitch Fix Thoughts on Go Choosing technology and cloud services Moving off heroku Building a platform team Where Ruby and Rails fit in today The role of books and how different people learn Large Language Model's effects on technical content Related Links David's Blog Mastodon Transcript You can help correct transcripts on GitHub. Intro [00:00:00] Jeremy: Today. I want to share another conversation from RubyConf San Diego. This time it's with David Copeland. He was a chief software architect and director of engineering at stitch fix. And at the start of the conversation, you're going to hear about why he decided to write the book, sustainable web development with Ruby on rails. Unfortunately, you're also going to notice the sound quality isn't too good. We had some technical difficulties. But once you hit the 20 minute mark of the recording, the mics are going to kick in. It's going to sound way better. So I hope you stick with it. Enjoy. Ruby at Stitch Fix [00:00:35] David: Stitch Fix was a Rails shop. I had done a lot of Rails and learned a lot of things that worked and didn't work, at least in that situation. And so I started writing them down and I was like, I should probably make this more than just a document that I keep, you know, privately on my computer. Uh, so that's, you know, kind of, kind of where the genesis of that came from and just tried to, write everything down that I thought what worked, what didn't work. Uh, if you're in a situation like me. Working on a product, with a medium sized, uh, team, then I think the lessons in there will be useful, at least some of them. Um, and I've been trying to keep it up over, over the years. I think the first version came out a couple years ago, so I've been trying to make sure it's always up to date with the latest stuff and, and Rails and based on my experience and all that. [00:01:20] Jeremy: So it's interesting that you mention, medium sized team because, during the, the keynote, just a few moments ago, Matz the creator of Ruby was talking about how like, Oh, Rails is really suitable for this, this one person team, right? Small, small team. And, uh, he was like, you're not Google. So like, don't worry about, right. Can you scale to that level? Yeah. Um, and, and I wonder like when you talk about medium size or medium scale, like what are, what are we talking? [00:01:49] David: I think probably under 200 developers, I would say. because when I left Stitch Fix, it was closing in on that number of developers. And so it becomes, you know, hard to... You can kind of know who everybody is, or at least the names sound familiar of everybody. But beyond that, it's just, it's just really hard. But a lot of it was like, I don't have experience at like a thousand developer company. I have no idea what that's like, but I definitely know that Rails can work for like... 200 ish people how you can make it work basically. yeah. [00:02:21] Jeremy: The decision to use Rails, I'm assuming that was made before you joined? [00:02:26] David: Yeah, the, um, the CTO of Stitch Fix, he had come in to clean up a mess made by contractors, as often happens. They had used Django, which is like the Python version of Rails. And he, the CTO, he was more familiar with Rails. So the first two developers he hired, also familiar with Rails. There wasn't a lot to maintain with the Django app, so they were like, let's just start fresh, fresh with Rails. yeah, but it's funny because a lot of the code in that Rails app was, like, transliterated from Python. So you could, it would, it looked like the strangest Ruby code in the world because it was basically, there was no test. So they were like, let's just write the Ruby version of this Python just so we know it works. but obviously that didn't, didn't last forever, so. [00:03:07] Jeremy: So, so what's an example of a, of a tell? Where you're looking at the code and you're like, oh, this is clearly, it came from Python. [00:03:15] David: You'd see like, very, very explicit, right? Like Python, there's a lot of like single line things. very like, this sounds like a dig, but it's very simple looking code. Like, like I don't know Python, but I was able to change this Django app. And I had to, I could look at it and you can figure out immediately how it works. Cause there's. Not much to it. There's nothing fancy. So, like, this, this Ruby code, there was nothing fancy. You'd be like, well, maybe they should have memoized that, or maybe they should have taken that into another class, or you could have done this with a hash or something like that. So there was, like, none of that. It was just, like, really basic, plain code like you would see in any beginning programming language kind of thing. Which is at least nice. You can understand it. but you probably wouldn't have written it that way at first in Ruby. Thoughts on Go [00:04:05] Jeremy: Yeah, that's, that's interesting because, uh, people sometimes talk about the Go programming language and how it looks, I don't know if simple is the right word, but it's something where you look at the code and even if you don't necessarily understand Go, it's relatively straightforward. Yeah. I wonder what your thoughts are on that being a strength versus that being, like, [00:04:25] David: Yeah, so at Stitch Fix at one point we had a pro, we were moving off of Heroku and we were going to, basically build a deployment platform using ECS on AWS. And so the deployment platform was a Rails app and we built a command line tool using Ruby. And it was fine, but it was a very complicated command line tool and it was very slow. And so one of the developers was like, I'm going to rewrite it in Go. I was like, ugh, you know, because I just was not a big fan. So he rewrote it in Go. It was a bazillion times faster. And then I was like, okay, I'm going to add, I'll add a feature to it. It was extremely easy. Like, it's just like what you said. I looked at it, like, I don't know anything about Go. I know what is happening here. I can copy and paste this and change things and make it work for what I want to do. And it did work. And it was, it was pretty easy. so there's that, I mean, aesthetically it's pretty ugly and it's, I, I. I can't really defend that as a real reason to not use it, but it is kind of gross. I did do Go, I did a small project in Go after Stitch Fix, and there's this vibe in Go about like, don't create abstractions. I don't know where I got that from, but every Go I look at, I'm like we should make an abstraction for this, but it's just not the vibe. They just don't like doing that. They like it all written out. And I see the value because you can look at the code and know what it does and you don't have to chase abstractions anywhere. But. I felt like I was copying and pasting a lot of, a lot of things. Um, so I don't know. I mean, the, the team at Stitch Fix that did this like command line app in go, they're the platform team. And so their job isn't to write like web apps all day, every day. There's kind of in and out of all kinds of things. They have to try to figure out something that they don't understand quickly to debug a problem. And so I can see the value of something like go if that's your job, right? You want to go in and see what the issue is. Figure it out and be done and you're not going to necessarily develop deep expertise and whatever that thing is that you're kind of jumping into. Day to day though, I don't know. I think it would make me kind of sad. (laughs) [00:06:18] Jeremy: So, so when you say it would make you kind of sad, I mean, what, what about it? Is it, I mean, you mentioned that there's a lot of copy and pasting, so maybe there's code duplication, but are there specific things where you're like, oh, I just don't? [00:06:31] David: Yeah, so I had done a lot of Java in my past life and it felt very much like that. Where like, like the Go library for making an HTTP call for like, I want to call some web service. It's got every feature you could ever want. Everything is tweakable. You can really, you can see why it's designed that way. To dial in some performance issue or solve some really esoteric thing. It's there. But the problem is if you just want to get an JSON, it's just like huge production. And I felt like that's all I really want to do and it's just not making it very easy. And it just felt very, very cumbersome. I think that having to declare types also is a little bit of a weird mindset because, I mean, I like to make types in Ruby, I like to make classes, but I also like to just use hashes and stuff to figure it out. And then maybe I'll make a class if I figure it out, but Go, you can't. You have to have a class, you have to have a type, you have to think all that ahead of time, and it just, I'm not used to working that way, so it felt, I mean, I guess I could get used to it, but I just didn't warm up to that sort of style of working, so it just felt like I was just kind of fighting with the vibe of the language, kind of. Yeah, [00:07:40] Jeremy: so it's more of the vibe or the feel where you're writing it and you're like this seems a little too... Explicit. I feel like I have to be too verbose. It just doesn't feel natural for me to write this. [00:07:53] David: Right, it's not optimized for what in my mind is the obvious case. And maybe that's not the obvious case for the people that write Go programs. But for me, like, I just want to like get this endpoint and get the JSON back as a map. Not any easier than any other case, right? Whereas like in Ruby, right? And you can, I think if you include net HTTP, you can just type get. And it will just return whatever that is. Like, that's amazing. It's optimized for what I think is a very common use case. So it makes me feel really productive. It makes me feel pretty good. And if that doesn't work out long term, I can always use something more complicated. But I'm not required to dig into the NetHttp library just to do what in my mind is something very simple. [00:08:37] Jeremy: Yeah, I think that's something I've noticed myself in working with Ruby. I mean, you have the standard library that's very... Comprehensive and the API surface is such that, like you said there, when you're trying to do common tasks, a lot of times they have a call you make and it kind of does the thing you expected or hoped for. [00:08:56] David: Yeah, yeah. It's kind of, I mean, it's that whole optimized for programmer happiness thing. Like it does. That is the vibe of Ruby and it seems like that is still the way things are. And, you know, I, I suppose if I had a different mindset, I mean, because I work with developers who did not like using Ruby or Rails. They loved using Go or Java. And I, I guess there's probably some psychological analysis we could do about their background and history and mindset that makes that make sense. But, to me, I don't know. It's, it's nice when it's pleasant. And Ruby seems pleasant. (laughs) Choosing Technology [00:09:27] Jeremy: as a... Software Architect, or as a CTO, when, when you're choosing technology, what are some of the things you look at in terms of, you know? [00:09:38] David: Yeah, I mean, I think, like, it's a weird criteria, but I think what is something that the team is capable of executing with? Because, like, most, right, most programming languages all kind of do the same thing. Like, you can kind of get most stuff done in most common popular programming languages. So, it's probably not... It's not true that if you pick the wrong language, you can't build the app. Like, that's probably not really the case. At least for like a web app or something. so it's more like, what is the team that's here to do it? What are they comfortable and capable of doing? I worked on a project with... It was a mix of like junior engineers who knew JavaScript, and then some senior engineers from Google. And for whatever reason someone had chosen a Rails app and none of them were comfortable or really yet competent with doing Ruby on Rails and they just all hated it and like it didn't work very well. Um, and so even though, yes, Rails is a good choice for doing stuff for that team at that moment. Not a good choice. Right. So I think you have to go in and like, what, what are we going to be able to execute on so that when the business wants us to do something, we just do it. And we don't complain and we don't say, Oh, well we can't because this technology that we chose, blah, blah, blah. Like you don't ever want to say that if possible. So I think that's. That's kind of the, the top thing. I think second would be how widely supported is it? Like you don't want to be the cutting edge user that's finding all the bugs in something really. Like you want to use something that's stable. Postgres, MySQL, like those work, those are fine. The bugs have been sorted out for most common use cases. Some super fancy edge database, I don't know if I'd want to be doing, doing that you know? Choosing cloud services [00:11:15] Jeremy: How do you feel about the cloud specific services and databases? Like are you comfortable saying like, oh, I'm going to use... Google Cloud, BigQuery. Yeah. [00:11:27] David: That sort of thing. I think it would kind of fall under the same criteria that I was just, just saying like, so with AWS it's interesting 'cause when we moved from Heroku to AWS by EC2 RDS, their database thing, uh, S3, those have been around for years, probably those are gonna work, but they always introduce new things. Like we, we use RabbitMQ and AWS came out with. Some, I forget what it was, it was a queuing service similar to Rabbit. We were like, Oh, maybe we should switch to that. But it was clear that they weren't really ready to support it. So. Yeah, so we didn't, we didn't switch to that. So I, you gotta try to read the tea leaves of the provider to see are they committed to, to supporting this thing or is this there to get some enterprise client to move into the cloud. And then the idea is to move off of that transitional thing into what they do support. And it's hard to get a clear answer from them too. So it takes a little bit of research to figure out, Are they going to support this or not? Because that's what you don't want. To move everything into some very proprietary cloud system and have them sunset it and say, Oh yeah, now you've got to switch again. Uh, that kind of sucks. So, it's a little trickier. [00:12:41] Jeremy: And what kind of questions or research do you do? Is it purely a function of this thing has existed for X number of years so I feel okay? [00:12:52] David: I mean, it's kind of similar to looking at like some gem you're going to add to your project, right? So you'll, you'll look at how often does it change? Is it being updated? Uh, what is the documentation? Does it look like someone really cared about the documentation? Does the documentation look updated? Are there issues with it that are being addressed or, or not? Um, so those are good signals. I think, talking to other practitioners too can be good. Like if you've got someone who's experienced. You can say, hey, do you know anybody back channeling through, like, everybody knows somebody that works at AWS, you can probably try to get something there. at Stitch Fix, we had an enterprise support contract, and so your account manager will sometimes give you good information if you ask. Again, it's a, they're not going to come out and say, don't use this product that we have, but they might communicate that in a subtle way. So you have to triangulate from all these sources to try to. to try to figure out what, what you want to do. [00:13:50] Jeremy: Yeah, it kind of makes me wish that there was a, a site like, maybe not quite like, can I use, right? Can I use, you can see like, oh, can I use this in my browser? Is there, uh, like an AWS or a Google Cloud? Can I trust this? Can I trust this? Yeah. Is this, is this solid or not? [00:14:04] David: Right, totally. It's like, there's that, that site where you, it has all the Apple products and it says whether or not you should buy it because one may or may not be coming out or they may be getting rid of it. Like, yeah, that would... For cloud services, that would be, that would be nice. [00:14:16] Jeremy: Yeah, yeah. That's like the Mac Buyer's Guide. And then we, we need the, uh, the technology. Yeah. Maybe not buyers. Cloud Provider Buyer's Guide, yeah. I guess we are buyers. [00:14:25] David: Yeah, yeah, totally, totally. [00:14:27] Jeremy: it's interesting that you, you mentioned how you want to see that, okay, this thing is mature. I think it's going to stick around because, I, interviewed, someone who worked on, I believe it was the CloudWatch team. Okay. Daniel Vassalo, yeah. so he left AWS, uh, after I think about 10 years, and then he wrote a book called, uh, The Good Parts of AWS. Oh! And, if you read his book, most of the services he says to use are the ones that are, like, old. Yeah. He's, he's basically saying, like, S3, you know you're good. Yeah. Right? but then all these, if you look at the AWS webpage, they have who knows, I don't know how many hundreds of services. Yeah. He's, he's kind of like I worked there and I would not use, you know, all these new services. 'cause I myself, I don't trust [00:15:14] David: it yet. Right. And so, and they're working there? Yeah, they're working there. Yeah. No. One of the VPs at Stitch Fix had worked on Google Cloud and so when we were doing this transition from Heroku, he was like, we are not using Google Cloud. I was like, really? He's like AWS is far ahead of the game. Do not use Google Cloud. I was like, all right, I don't need any more info. You work there. You said don't. I'm gonna believe you. So [00:15:36] Jeremy: what, what was his did he have like a core point? [00:15:39] David: Um, so he never really had anything bad to say about Google per se. Like I think he enjoyed his time there and I think he thought highly of who he worked with and what he worked on and that sort of thing. But his, where he was coming from was like AWS was so far ahead. of Google on anything that we would use, he was like, there's, there's really no advantage to, to doing it. AWS is a known quantity, right? it's probably still the case. It's like, you know, you've heard the nobody ever got fired for using IBM or using Microsoft or whatever the thing is. Like, I think that's, that was kind of the vibe. And he was like, moving all of our infrastructure right before we're going to go public. This is a serious business. We should just use something that we know will work. And he was like, I know this will work. I'm not confident about. Google, uh, for our use case. So we shouldn't, we shouldn't risk it. So I was like, okay, I trust you because I didn't know anything about any of that stuff at the time. I knew Heroku and that was it. So, yeah. [00:16:34] Jeremy: I don't know if it's good or bad, but like you said, AWS seems to be the default choice. Yeah. And I mean, there's people who use Azure. I assume it's mostly primarily Microsoft. Yeah. And then there's Google Cloud. It's not really clear why you would pick it, unless there was a specific service or something that only they had. [00:16:55] David: Yeah, yeah. Or you're invested in Google, you know, you want to keep everything there. I mean, I don't know. I haven't really been at that level to make that kind of decision, and I would probably choose AWS for the reasons discussed, but, yeah. Moving off Heroku [00:17:10] Jeremy: And then, so at Stitch Fix, you said you moved off of Heroku [00:17:16] David: yeah. Yeah, so we were heavy into Heroku. I think that we were told that at one point we had the biggest Heroku Postgres database on their platform. Not a good place to be, right? You never want to be the biggest customer person, usually. but the problem we were facing was essentially we were going to go public. And to do that, you're under all the scrutiny. about many things, including the IT systems and the security around there. So, like, by default, a Postgres, a Heroku Postgres database is, like, on the internet. It's only secured by the password. all their services are on the internet. So, not, not ideal. they were developing their private cloud service at that time. And so that would have given us, in theory, on paper, it would have solved all of our problems. And we liked Heroku and we liked the developer experience. It was great. but... Heroku private spaces, it was still early. There's a lot of limitations that when they explained why those limitations, they were reasonable. And if we had. started from scratch on Heroku Private Spaces. It probably would have worked great, but we hadn't. So we just couldn't make it work. So we were like, okay, we're going to have to move to AWS so that everything can be basically off the internet. Like our public website needs to be on the internet and that's kind of it. So we need to, so that's basically was the, was the impetus for that. but it's too bad because I love Heroku. It was great. I mean, they were, they were a great partner. They were great. I think if Stitch Fix had started life a year later, Private Spaces. Now it's, it's, it's way different than it was then. Cause it's been, it's a mature product now, so we could have easily done that, but you know, the timing didn't work out, unfortunately. [00:18:50] Jeremy: And that was a compliance thing to, [00:18:53] David: Yeah. And compliance is weird cause they don't tell you what to do, but they give you some parameters that you need to meet. And so one of them is like how you control access. So, so going public, the compliance is around the financial data and. Ensuring that the financial data is accurate. So a lot of the systems at Stichfix were storing the financial data. We, you know, the warehouse management system was custom made. Uh, all the credit card processing was all done, like it was all in some databases that we had running in Heroku. And so those needed to be subject to stricter security than we could achieve with just a single password that we just had to remember to rotate when someone like left the team. So that was, you know, the kind of, the kind of impetus for, for all of that. [00:19:35] Jeremy: when you were using Heroku, Salesforce would have already owned it then. Did you, did you get any sense that you weren't really sure about the future of the platform while you're on it or, [00:19:45] David: At that time, no, it seemed like they were still innovating. So like, Heroku has a Redis product now. They didn't at the time we wish that they did. They told us they're working on it, but it wasn't ready. We didn't like using the third parties. Kafka was not a thing. We very much were interested in that. We would have totally used it if it was there. So they were still. Like doing bigger innovations then, then it seems like they are now. I don't know. It's weird. Like they're still there. They still make money, I assume for Salesforce. So it doesn't feel like they're going away, but they're not innovating at the pace that they were kind of back in the day. [00:20:20] Jeremy: it used to feel like when somebody's asking, I want to host a Rails app. Then you would say like, well, use Heroku because it's basically the easiest to get started. It's a known quantity and it's, it's expensive, but, it seemed for, for most people, it was worth it. and then now if I talk to people, it's like. Not what people suggest anymore. [00:20:40] David: Yeah, because there's, there's actual competitors. It's crazy to me that there was no competitors for years, and now there's like, Render and Fly. io seem to be the two popular alternatives. Um, I doubt they're any cheaper, honestly, but... You get a sense, right, that they're still innovating, still building those platforms, and they can build with, you know, all of the knowledge of what has come before them, and do things differently that might, that might help. So, I still use Heroku for personal things just because I know it, and I, you know, sometimes you don't feel like learning a new thing when you just want to get something done, but, yeah, I, I don't know if we were starting again, I don't know, maybe I'd look into those things. They, they seem like they're getting pretty mature and. Heroku's resting on its laurels, still. [00:21:26] Jeremy: I guess I never quite the mindset, right? Where you You have a platform that's doing really well and people really like it and you acquire it and then it just It seems like you would want to keep it rolling, right? (laughs) [00:21:38] David: Yeah, it's, it is wild, I mean, I guess... Why did you, what was Salesforce thinking they were going to get? Uh, who knows maybe the person at Salesforce that really wanted to purchase it isn't there. And so no one at Salesforce cares about it. I mean, there's all these weird company politics that like, who knows what's going on and you could speculate. all day. What's interesting is like, there's definitely some people in the Ruby community who work there and still are working there. And that's like a little bit of a canary for me. I'm like, all right, well, if that person's still working there, that person seems like they're on the level and, and, and, and seems pretty good. They're still working there. It, it's gotta be still a cool place to be or still doing something, something good. But, yeah, I don't know. I would, I would love to know what was going on in all the Salesforce meetings about acquiring that, how to manage it. What are their plans for it? I would love to know that stuff. [00:22:29] Jeremy: maybe you had some experience with this at Stitch Fix But I've heard with Heroku some of their support staff at least in the past they would, to some extent, actually help you troubleshoot, like, what's going on with your app. Like, if your app is, like, using a whole bunch of memory, and you're out of memory, um, they would actually kind of look into that, for you, which is interesting, because it's like, that's almost like a services thing than it is just a platform. [00:22:50] David: Yeah. I mean, they, their support, you would get, you would get escalated to like an engineer sometimes, like who worked on that stuff and they would help figure out what the problem was. Like you got the sense that everybody there really wanted the platform to be good and that they were all sort of motivated to make sure that everybody. You know, did well and used the platform. And they also were good at, like a thing that trips everybody up about Heroku is that your app restarts every day. And if you don't know anything about anything, you might think that is stupid. Why, why would I want that? That's annoying. And I definitely went through that and I complained to them a lot. And I'm like, if you only could not restart. And they very patiently and politely explained to me why that it needed to do that, they weren't going to remove that, and how to think about my app given that reality, right? Which is great because like, what company does that, right? From the engineers that are working on it, like No, nobody does that. So, yeah, no, I haven't escalated anything to support at Heroku in quite some time, so I don't know if it's still like that. I hope it is, but I'm not really, not really sure. Building a platform team [00:23:55] Jeremy: Yeah, that, uh, that reminds me a little bit of, I think it's Rackspace? There's, there's, like, another hosting provider that was pretty popular before, and they... Used to be famous for that type of support, where like your, your app's having issues and somebody's actually, uh, SSHing into your box and trying to figure out like, okay, what's going on? which if, if that's happening, then I, I can totally see where the, the price is justified. But if the support is kind of like dropping off to where it's just, they don't do that kind of thing, then yeah, I can see why it's not so much of a, yeah, [00:24:27] David: We used to think of Heroku as like they were the platform team before we had our own platform team and they, they acted like it, which was great. [00:24:35] Jeremy: Yeah, I don't have, um, experience with, render, but I, I, I did, talk to someone from there, and it does seem like they're, they're trying to fill that role, um, so, yeah, hopefully, they and, and other companies, I guess like Vercel and things like that, um, they're, they're all trying to fill that space, [00:24:55] David: Yeah, cause, cause building our own internal platform, I mean it was the right thing to do, but it's, it's a, you can't just, you have to have a team on it, it's complicated, getting all the stuff in AWS to work the way you want it to work, to have it be kind of like Heroku, like it's not trivial. if I'm a one person company, I don't want to be messing around with that particularly. I want to just have it, you know, push it up and have it go and I'm willing to pay for that. So it seems logical that there would be competitors in that space. I'm glad there are. Hopefully that'll light a fire under, under everybody. [00:25:26] Jeremy: so in your case, it sounds like you moved to having your own platform team and stuff like that, uh, partly because of the compliance thing where you're like, we need our, we need to be isolated from the internet. We're going to go to AWS. If you didn't have that requirement, do you still think like that would have been the time to, to have your own platform team and manage that all yourself? [00:25:46] David: I don't know. We, we were thinking an issue that we were running into when we got bigger, um, was that, I mean, Heroku, it, It's obviously not as flexible as AWS, but it is still very flexible. And so we had a lot of internal documentation about this is how you use Heroku to do X, Y, and Z. This is how you set up a Stitch Fix app for Heroku. Like there was just the way that we wanted it to be used to sort of. Just make it all manageable. And so we were considering having a team spun up to sort of add some tooling around that to sort of make that a little bit easier for everybody. So I think there may have been something around there. I don't know if it would have been called a platform team. Maybe we call, we thought about calling it like developer happiness or because you got developer experience or something. We, we probably would have had something there, but. I do wonder how easy it would have been to fund that team with developers if we hadn't had these sort of business constraints around there. yeah, um, I don't know. You get to a certain size, you need some kind of manageability and consistency no matter what you're using underneath. So you've got to have, somebody has to own it to make sure that it's, that it's happening. [00:26:50] Jeremy: So even at your, your architect level, you still think it would have been a challenge to, to. Come to the executive team and go like, I need funding to build this team. [00:27:00] David: You know, certainly it's a challenge because everybody, you know, right? Nobody wants to put developers in anything, right? There are, there are a commodity and I mean, that is kind of the job of like, you know, the staff engineer or the architect at a company is you don't have, you don't have the power to put anybody on anything you, you have the power to Schedule a meeting with a VP or the CTO and they will listen to you. And that's basically, you've got to use that power to convince them of what you want done. And they're all reasonable people, but they're balancing 20 other priorities. So it would, I would have had to, it would have been a harder case to make that, Hey, I want to take three engineers. And have them write tooling to make Heroku easier to use. What? Heroku is not easy to use. Why aren't, you know, so you really, I would, it would be a little bit more of a stretch to walk them through it. I think a case could be made, but, definitely would take some more, more convincing than, than what was needed in our case. [00:27:53] Jeremy: Yeah. And I guess if you're able to contrast that with, you were saying, Oh, I need three people to help me make Heroku easier. Your actual platform team on AWS, I imagine was much larger, right? [00:28:03] David: Initially it was, there was, it was three people did the initial move over. And so by the time we went public, we'd been on this new system for, I don't know, six to nine months. I can't remember exactly. And so at that time the platform team was four or five people, and I, I mean, so percentage wise, right, the engineering team was maybe almost 200, 150, 200. So percentage wise, maybe a little small, I don't know. but it kind of gets back to the power of like the rails and the one person framework. Like everything we did was very much the same And so the Rails app that managed the deployment was very simple. The, the command line app, even the Go one with all of its verbosity was very, very simple. so it was pretty easy for that small team to manage. but, Yeah, so it was sort of like for redundancy, we probably needed more than three or four people because you know, somebody goes out sick or takes a vacation. That's a significant part of the team. But in terms of like just managing the complexity and building it and maintaining it, like it worked pretty well with, you know, four or five people. Where Rails fits in vs other technology [00:29:09] Jeremy: So during the Keynote today, they were talking about how companies like GitHub and Shopify and so on, they're, they're using Rails and they're, they're successful and they're fairly large. but I think the thing that was sort of unsaid was the fact that. These companies, while they use Rails, they use a lot of other, technology as well. And, and, and kind of increasing amounts as well. So, I wonder from your perspective, either from your experience at StitchFix or maybe going forward, what is the role that, that Ruby and Rails plays? Like, where does it make sense for that to be used versus like, Okay, we need to go and build something in Java or, you know, or Go, that sort of thing? [00:29:51] David: right. I mean, I think for like your standard database backed web app, it's obviously great. especially if your sort of mindset bought into server side rendering, it's going to be great at that. so like internal tools, like the customer service dashboard or... You know, something for like somebody who works at a company to use. Like, it's really great because you can go super fast. You're not going to be under a lot of performance constraints. So you kind of don't even have to think about it. Don't even have to solve it. You can, but you don't have to, where it wouldn't work, I guess, you know, if you have really strict performance. Requirements, you know, like a, a Go version of some API server is going to use like percentages of what, of what Rails would use. If that's meaningful, if what you're spending on memory or compute is, is meaningful, then, then yeah. That, that becomes worthy of consideration. I guess if you're, you know, if you're making a mobile app, you probably need to make a mobile app and use those platforms. I mean, I guess you can wrap a Rails app sort of, but you're still making, you still need to make a mobile app, that does something. yeah. And then, you know, interestingly, the data science part of Stitch Fix was not part of the engineering team. They were kind of a separate org. I think Ruby and Rails was probably the only thing they didn't use over there. Like all the ML stuff, everything is either Java or Scala or Python. They use all that stuff. And so, yeah, if you want to do AI and ML with Ruby, you, it's, it's hard cause there's just not a lot there. You really probably should use Python. It'll make your life easier. so yeah, those would be some of the considerations, I guess. [00:31:31] Jeremy: Yeah, so I guess in the case of, ML, Python, certainly, just because of the, the ecosystem, for maybe making a command line application, maybe Go, um, Go or Rust, perhaps, [00:31:44] David: Right. Cause you just get a single binary. Like the problem, I mean, I wrote this book on Ruby command line apps and the biggest problem is like, how do I get the Ruby VM to be anywhere so that it can then run my like awesome scripts? Like that's kind of a huge pain. (laughs) So [00:31:59] Jeremy: and then you said, like, if it's Very performance sensitive, which I am kind of curious in, in your experience with the companies you've worked at, when you're taking on a project like that, do you know up front where you're like, Oh, the CPU and memory usage is going to be a problem, or is it's like you build it and you're like, Oh, this isn't working. So now I know. [00:32:18] David: yeah, I mean, I, I don't have a ton of great experience there at Stitch Fix. The biggest expense the company had was the inventory. So like the, the cost of AWS was just de minimis compared to all that. So nobody ever came and said, Hey, you've got to like really save costs on, on that stuff. Cause it just didn't really matter. at the, the mental health startup I was at, it was too early. But again, the labor costs were just far, far exceeded the amount of money I was spending on, on, um, you know, compute and infrastructure and stuff like that. So, Not knowing anything, I would probably just sort of wait and see if it's a problem. But I suppose you always take into account, like, what am I actually building? And like, what does this business have to scale to, to make it worthwhile? And therefore you can kind of do a little bit of planning ahead there. But, I dunno, I think it would kind of have to depend. [00:33:07] Jeremy: There's a sort of, I guess you could call it a meme, where people say like, Oh, it's, it's not, it's not Rails that's slow, it's the, the database that's slow. And, uh, I wonder, is that, is that accurate in your experience, or, [00:33:20] David: I mean, most of the stuff that we had that was slow was the database, because like, it's really easy to write a crappy query in Rails if you're not, if you're not careful, and then it's really easy to design a database that doesn't have any indexes if you're not careful. Like, you, you kind of need to know that, But of course, those are easy to fix too, because you just add the index, especially if it's before the database gets too big where we're adding indexes is problematic. But, I think those are just easy performance mistakes to make. Uh, especially with Rails because you're not, I mean, a lot of the Rails developers at Citrix did not know SQL at all. I mean, they had to learn it eventually, but they didn't know it at all. So they're not even knowing that what they're writing could possibly be problematic. It's just, you're writing it the Rails way and it just kind of works. And at a small scale, it does. And it doesn't matter until, until one day it does. [00:34:06] Jeremy: And then in, in the context of, let's say, using ActiveRecord and instantiating the objects, or, uh, the time it takes to render templates, that kinds of things, to, at least in your experience, that wasn't such of an issue. [00:34:20] David: No, and it was always, I mean, whenever we looked at why something was slow, it was always the database and like, you know, you're iterating over some active records and then, and then, you know, you're going into there and you're just following this object graph. I've got a lot of the, a lot of the software at Stitch Fix was like internal stuff and it was visualizing complicated data out of the database. And so if you didn't think about it, you would just start dereferencing and following those relationships and you have this just massive view and like the HTML is fine. It's just that to render this div, you're. Digging into some active record super deep. and so, you know, that was usually the, the, the problems that we would see and they're usually easy enough to fix by making an index or. Sometimes you do some caching or something like that. and that solved most of the, most of the issues [00:35:09] Jeremy: The different ways people learn [00:35:09] Jeremy: so you're also the author of the book, Sustainable Web Development with Ruby on Rails. And when you talk to people about like how they learn things, a lot of them are going on YouTube, they're going on, uh, you know, looking for blogs and things like that. And so as an author, what do you think the role is of, of books now? Yeah, [00:35:29] David: I have thought about this a lot, because I, when I first got started, I'm pretty old, so books were all you had, really. Um, so they seem very normal and natural to me, but... does someone want to sit down and read a 400 page technical book? I don't know. so Dave Thomas who runs Pragmatic Bookshelf, he was on a podcast and was asked the same question and basically his answer, which is my answer, is like a long form book is where you can really lay out your thinking, really clarify what you mean, really take the time to develop sometimes nuanced, examples or nuanced takes on something that are Pretty hard to do in a short form video or in a blog post. Because the expectation is, you know, someone sends you an hour long YouTube video, you're probably not going to watch that. Two minute YouTube video is sure, but you can't, you can't get into so much, kind of nuanced detail. And so I thought that was, was right. And that was kind of my motivation for writing. I've got some thoughts. They're too detailed. It's, it's too much set up for a blog post. There's too much of a nuanced element to like, really get across. So I need to like, write more. And that means that someone's going to have to read more to kind of get to it. But hopefully it'll be, it'll be valuable. one of the sessions that we're doing later today is Ruby content creators, where it's going to be me and Noel Rappin and Dave Thomas representing the old school dudes that write books and probably a bunch of other people that do, you know, podcasts videos. It'd be interesting to see, I really want to know how do people learn stuff? Because if no one reads books to learn things, then there's not a lot of point in doing it. But if there is value, then, you know. It should be good and should be accessible to people. So, that's why I do it. But I definitely recognize maybe I'm too old and, uh, I'm not hip with the kids or, or whatever, whatever the case is. I don't know. [00:37:20] Jeremy: it's tricky because, I think it depends on where you are in the process of learning that thing. Because, let's say, you know a fair amount about the technology already. And you look at a book, in a lot of cases it's, it's sort of like taking you from nothing to something. And so you're like, well, maybe half of this isn't relevant to me, but then if I don't read it, then I'm probably missing a lot still. And so you're in this weird in be in between zone. Another thing is that a lot of times when people are trying to learn something, they have a specific problem. And, um, I guess with, with books, it's, you kind of don't know for sure if the thing you're looking for is going to be in the book. [00:38:13] David: I mean, so my, so my book, I would not say as a beginner, it's not a book to learn how to do Rails. It's like you already kind of know Rails and you want to like learn some comprehensive practices. That's what my book is for. And so sometimes people will ask me, I don't know Rails, should I get your book? And I'm like, no, you should not. but then you have the opposite thing where like the agile web development with Rails is like the beginner version. And some people are like, Oh, it's being updated for Rails 7. Should I get it? I'm like, probably not because How to go from zero to rails hasn't changed a lot in years. There's not that much that's going to be new. but, how do you know that, right? Hopefully the Table of Contents tells you. I mean, the first book I wrote with Pragmatic, they basically were like, The Table of Contents is the only thing the reader, potential reader is going to have to have any idea what's in the book. So, You need to write the table of contents with that in mind, which may not be how you'd write the subsections of a book, but since you know that it's going to serve these dual purposes of organizing the book, but also being promotional material that people can read, you've got to keep that in mind, because otherwise, how does anybody, like you said, how does anybody know what's, what's going to be in there? And they're not cheap, I mean, these books are 50 bucks sometimes, and That's a lot of money for people in the U. S. People outside the U. S. That's a ton of money. So you want to make sure that they know what they're getting and don't feel ripped off. [00:39:33] Jeremy: Yeah, I think the other challenge is, at least what I've heard, is that... When people see a video course, for whatever reason, they, they set, like, a higher value to it. They go, like, oh, this video course is, 200 dollars and it's, like, seems like a lot of money, but for some people it's, like, okay, I can do that. But then if you say, like, oh, this, this book I've been researching for five years, uh, I want to sell it for a hundred bucks, people are going to be, like no. No way., [00:40:00] David: Yeah. Right. A hundred bucks for a book. There's no way. That's a, that's a lot. Yeah. I mean, producing video, I've thought about doing video content, but it seems so labor intensive. Um, and it's kind of like, It's sort of like a performance. Like I was mentioning before we started that I used to play in bands and like, there's a lot to go into making an even mediocre performance. And so I feel like, you know, video content is the same way. So I get that it like, it does cost more to produce, but, are you getting more information out of it? I, that, I don't know, like maybe not, but who knows? I mean, people learn things in different ways. So, [00:40:35] Jeremy: It's just like this perception thing, I think. And, uh, I'm not sure why that is. Um, [00:40:40] David: Yeah, maybe it's newer, right? Maybe books feel older so they're easier to make and video seems newer. I mean, I don't know. I would love to talk to engineers who are like... young out of college, a few years into their career to see what their perception of this stuff is. Cause I mean, there was no, I mean, like I said, I read books cause that's all there was. There was no, no videos. You, you go to a conference and you read a book and that was, that was all you had. so I get it. It seems a whole video. It's fancier. It's newer. yeah, I don't know. I would love to hear a wide variety of takes on it to see what's actually the, the future, you know? [00:41:15] Jeremy: sure, yeah. I mean, I think it probably can't just be one or the other, right? Like, I think there are... Benefits of each way. Like, if you have the book, you can read it at your own pace without having to, like, scroll through the video, and you can easily copy and paste the, the code segments, [00:41:35] David: Search it. Go back and forth. [00:41:36] Jeremy: yeah, search it. So, I think there's a place for it, but yeah, I think it would be very interesting, like you said, to, to see, like, how are people learning, [00:41:45] David: Right. Right. Yeah. Well, it's the same with blogs and podcasts. Like I, a lot of podcasters I think used to be bloggers and they realized that like they can get out what they need by doing a podcast. And it's way easier because it's more conversational. You don't have to do a bunch of research. You don't have to do a bunch of editing. As long as you're semi coherent, you can just have a conversation with somebody and sort of get at some sort of thing that you want to talk about or have an opinion about. And. So you, you, you see a lot more podcasts and a lot less blogs out there because of that. So it's, that's kind of like the creators I think are kind of driving that a little bit. yeah. So I don't know. [00:42:22] Jeremy: Yeah, I mean, I can, I can say for myself, the thing about podcasts is that it's something that I can listen to while I'm doing something else. And so you sort of passively can hopefully pick something up out of that conversation, but... Like, I think it's maybe not so good at the details, right? Like, if you're talking code, you can talk about it over voice, but can you really visualize it? Yeah, yeah, yeah. I think if you sit down and you try to implement something somebody talked about, you're gonna be like, I don't know what's happening. [00:42:51] David: Yeah. [00:42:52] Jeremy: So, uh, so, so I think there's like these, these different roles I think almost for so like maybe you know the podcast is for you to Maybe get some ideas or get some familiarity with a thing and then when you're ready to go deeper You can go look at a blog post or read a book I think video kind of straddles those two where sometimes video is good if you want to just see, the general concept of a thing, and have somebody explain it to you, maybe do some visuals. that's really good. but then it can also be kind of detailed, where, especially like the people who stream their process, right, you can see them, Oh, let's, let's build this thing together. You can ask me questions, you can see how I think. I think that can be really powerful. at the same time, like you said, it can be hard to say, like, you know, I look at some of the streams and it's like, oh, this is a three hour stream and like, well, I mean, I'm interested. I'm interested, but yeah, it's hard enough for me to sit through a, uh, a three hour movie, [00:43:52] David: Well, then that, and that gets into like, I mean, we're, you know, we're at a conference and they, they're doing something a little, like, there are conference talks at this conference, but there's also like. sort of less defined activities that aren't a conference talk. And I think that could be a reaction to some of this too. It's like I could watch a conference talk on, on video. How different is that going to be than being there in person? maybe it's not that different. Maybe, maybe I don't need to like travel across the country to go. Do something that I could see on video. So there's gotta be something here that, that, that meets that need that I can't meet any other way. So it's all these different, like, I would like to think that's how it is, right? All this media all is a part to play and it's all going to kind of continue and thrive and it's not going to be like, Oh, remember books? Like maybe, but hopefully not. Hopefully it's like, like what you're saying. Like it's all kind of serving different purposes that all kind of work together. Yeah. [00:44:43] Jeremy: I hope that's the case, because, um, I don't want to have to scroll through too many videos. [00:44:48] David: Yeah. The video's not for me. Large Language Models [00:44:50] Jeremy: I, I like, I actually do find it helpful, like, like I said, for the high level thing, or just to see someone's thought process, but it's like, if you want to know a thing, and you have a short amount of time, maybe not the best, um, of course, now you have all the large language model stuff where you like, you feed the video in like, Hey, tell, tell, tell me, uh, what this video is about and give me the code snippets and all that stuff. I don't know how well it works, but it seems [00:45:14] David: It's gotta get better. Cause you go to a support site and they're like, here's how to fix your problem, and it's a video. And I'm like, can you just tell me? But I'd never thought about asking the AI to just look at the video and tell me. So yeah, it's not bad. [00:45:25] Jeremy: I think, that's probably where we're going. So it's, uh, it's a little weird to think about, but, [00:45:29] David: yeah, yeah. I was just updating, uh, you know, like I said, I try to keep the book updated when new versions of Rails come out, so I'm getting ready to update it for Rails 7. 1 and in Amazon's, Kindle Direct Publishing as their sort of backend for where you, you know, publish like a Kindle book and stuff, and so they added a new question, was AI used in the production of this thing or not? And if you answer yes, they want you to say how much, And I don't know what they're gonna do with that exactly, but I thought it was pretty interesting, cause I would be very disappointed to pay 50 for a book that the AI wrote, right? So it's good that they're asking that? Yeah. [00:46:02] Jeremy: I think the problem Amazon is facing is where people wholesale have the AI write the book, and the person either doesn't review it at all, or maybe looks at a little, a little bit. And, I mean, the, the large language model stuff is very impressive, but If you have it generate a technical book for you, it's not going to be good. [00:46:22] David: yeah. And I guess, cause cause like Amazon, I mean, think about like Amazon scale, like they're not looking at the book at all. Like I, I can go click a button and have my book available and no person's going to look at it. they might scan it or something maybe with looking for bad words. I don't know, but there's no curation process there. So I could, yeah. I could see where they could have that, that kind of problem. And like you as the, as the buyer, you don't necessarily, if you want to book on something really esoteric, there are a lot of topics I wish there was a book on that there isn't. And as someone generally want to put it on Amazon, I could see a lot of people buying it, not realizing what they're getting and feeling ripped off when it was not good. [00:47:00] Jeremy: Yeah, I mean, I, I don't know, if it's an issue with the, the technical stuff. It probably is. But I, I know they've definitely had problems where, fiction, they have people just generating hundreds, thousands of books, submitting them all, just flooding it. [00:47:13] David: Seeing what happens. [00:47:14] Jeremy: And, um, I think that's probably... That's probably the main reason why they ask you, cause they want you to say like, uh, yeah, you said it wasn't. And so now we can remove your book. [00:47:24] David: right. Right. Yeah. Yeah. [00:47:26] Jeremy: I mean, it's, it's not quite the same, but it's similar to, I don't know what Stack Overflow's policy is now, but, when the large language model stuff started getting big, they had a lot of people answering the questions that were just. Pasting the question into the model [00:47:41] David: Which because they got it from [00:47:42] Jeremy: and then [00:47:43] David: The Got model got it from Stack Overflow. [00:47:45] Jeremy: and then pasting the answer into Stack Overflow and the person is not checking it. Right. So it's like, could be right, could not be right. Um, cause, cause to me, it's like, if, if you generate it, if you generate the answer and the answer is right, and you checked it, I'm okay with that. [00:48:00] David: Yeah. Yeah. [00:48:01] Jeremy: but if you're just like, I, I need some karma, so I'm gonna, I'm gonna answer these questions with, with this bot, I mean, then maybe [00:48:08] David: I could have done that. You're not adding anything. Yeah, yeah. [00:48:11] Jeremy: it's gonna be a weird, weird world, I think. [00:48:12] David: Yeah, no kidding. No kidding. [00:48:15] Jeremy: that's a, a good place to end it on, but is there anything else you want to mention, [00:48:19] David: No, I think we covered it all just yeah, you could find me online. I'm Davetron5000 on Ruby. social Mastodon, I occasionally post on Twitter, but not that much anymore. So Mastodon's a place to go. [00:48:31] Jeremy: David, thank you so much [00:48:32] David: All right. Well, thanks for having me.

406: Working Solo

Play Episode Listen Later Nov 14, 2023 32:27

Joël got to do some pretty fancy single sign-on work. And when it came time to commit, he documented the ridiculous number of redirects to give people a sense of what was happening. Stephanie has been exploring Rails callbacks and Ruby debugging tools, using methods like save_callbacks and Kernel.caller, and creating a function call graph to better understand and manage complex code dependencies. Stephanie is also engaged in an independent project and seeking strategies to navigate the challenges of solo work. She and Joël explore how to find external support and combat isolation, consider ways to stimulate creativity, and obtain feedback on her work without a direct team. Additionally, they ponder succession planning to ensure project continuity after her involvement ends. They also reflect on the unique benefits of solo work, such as personal growth and flexibility. Stephanie's focus is on balancing the demands of working independently while maintaining a connected and sustainable professional approach. ASCII Sequence Diagram Creator (https://textart.io/sequence) Callback debugging methods (https://andycroll.com/ruby/find-list-debug-active-record-callbacks-in-the-console/) Kernel.caller (https://ruby-doc.org/core-3.0.2/Kernel.html#method-i-caller) Method.source_location (https://ruby-doc.org/core-3.0.2/Method.html#method-i-source_location) Building web apps by your lonesome by Jeremy Smith (https://www.youtube.com/watch?v=Rr871vmV4YM) Transcript: STEPHANIE: Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Stephanie Minn. JOËL: And I'm Joël Quenneville. And together, we're here to share a bit of what we've learned along the way. STEPHANIE: So, Joël, what's new in your world? JOËL: I got to do something really fun this week, where I was doing some pretty fancy single sign-on work. And when it came time to commit, I wanted to document the kind of ridiculous number of redirects that happen and give people a sense of what was going on. And for my own self, what I had been doing is, I had done a sequence diagram that sort of shows, like, three different services that are all talking to each other and where they redirect to each other as they all go through the sequence to sign someone in. And I was like, how could I embed that in the commit message? Because I think it would be really useful context for someone trying to get an overview of what this commit is doing. And the answer, for me, was, can I get this sequence diagram in ASCII form somewhere? And I found a website that allows me to do this in ASCII art. It's the textart.io/sequence. And that allows me to create a sequence diagram that gets generated as ASCII art. I can copy-paste that into a commit message. And now anybody else who is like, "What is it that Joël is trying to do here?" can look at that and be like, "Oh, oh okay, so, we got these, like, four different places that are all talking to each other in this order. Now I see what's happening." STEPHANIE: That's super neat. I love the idea of having it directly in your commit message just because, you know, you don't have to go and find a graph elsewhere if you want to understand what's going on. It's right there for you, for future commit explorers [laughs] trying to understand what was going on in this snippet of time. JOËL: I try as much as possible to include those sorts of things directly in the commit message because you never know who's reading the commit. They might not have access to some sort of linked resource. So, if I were like, "Hey, go to our wiki and see this link," like, sure, that would be helpful, but maybe the person reading it doesn't have access to the wiki. Maybe they do have access, but they're not on the internet right now, and so they don't have access to the wiki. Maybe the wiki no longer exists, and that's a dead link. So, as much as possible, I try to embed context directly in my commit messages. STEPHANIE: That's really cool. And just another shout out to ASCII art, you know [laughs], persevering through all the times with our fancy tools. It's still going strong [laughs]. JOËL: Something about text, right? STEPHANIE: Exactly. I actually also have a diagram graph thing to share about what's new in my world that is kind of in a similar vein. Another thoughtboter and former guest on the show, Sara Jackson, shared in our dev channel about this really cool mural graph that she made to figure out what was going on with callbacks because she was working on, you know, understanding the lifecycle of this model and was running into, like, a lot of complex behavior. And she linked to a really neat blog post by Andy Croll, too, that included a little snippet sharing a few callback debugging methods that are provided by ActiveRecord. So, basically, you can have your model and just call double underscore callbacks. And it returns a list of all the callbacks that are defined for that model, and I thought that was really neat. So, I played around with it and copypastad [laughs] the snippet into my Rails console to figure out what's going on with basically, like, the god object of that that I work in. And the first issue I ran into was that it was undefined because it turns out that my application was on an older [laughs] version of Rails than that method was provided on. But, there are more specific methods for the types of callbacks. So, if you are looking specifically for all the callbacks related to a save or a destroy, I think it's save underscore callbacks, right? And that was available on the Rails version I was on, which was, I think, 4. But that was a lot of fun to play around with. And then, I ended up chatting with Sara afterwards about her process for creating the diagram after, you know, getting a list of all these methods. And I actually really liked this hybrid approach she took where, you know, she automated some parts but then also manually, like, went through and stepped through the code and, like, annotated notes for the methods as she was traversing them. And, you know, sometimes I think about, like, wow, like, it would be so cool if this graph just generated automatically, but I also think there is some value to actually creating it yourself. And there's some amount of, like, mental processing that happens when you do that, as opposed to, like, looking at a thing that was just, you know, generated afterwards, I think. JOËL: Do you know what kind of graph Sara generated? Was it some kind of, like, function call graph, or was it some other way of visualizing the callbacks? STEPHANIE: I think it was a function call graph, essentially. It even kind of showed a lot of the dependencies, too, because some of the callback functions were quite complicated and then would call other classes. So, there was a lot of, I think, hidden dependencies there that were unexpected, you know, when you think you're just going to create a regular old [laughs] record. JOËL: Yeah, I've been burned by unexpected callbacks or callbacks that do things that you wouldn't want in a particular context and then creating bad data or firing off external services that you really didn't want, and that can be an unpleasant surprise. I appreciate it when the framework offers debugging tools and methods kind of built-in, so these helpers, which I was not aware of. It's really cool because they allow you to kind of introspect and understand the code that you're going through. Do you have any others like that from Rails or Ruby that you find yourself using from time to time to help better understand the code? STEPHANIE: I think one I discovered recently was Kernel.caller, which gives you the stack trace wherever you are when executing. And that was really helpful when you're not raising an exception in certain places, and you need to figure out the flow of the code. I think that was definitely a later discovery. And I'm glad to have it in my back pocket now as something I can use in any kind of Ruby code. JOËL: That can, yeah, definitely be a really useful context to have even just in, like, an interactive console. You're like, wait a minute, where's this coming from? What is the call stack right now? STEPHANIE: Do you have any debugging tools or methods that you like to use that maybe are under the radar a little bit? JOËL: One that I really appreciate that's built into Ruby is the source location method on the method object, so Ruby has a method object. And so, when you're dealing with some sort of method and, like, maybe it got generated programmatically through metaprogramming, or maybe it's coming from a gem or something like that, and you're just like, where is this define? I'm trying to find it. If you're in your editor and you're doing stuff, maybe you could run some sort of search, or maybe it has some sort of keyword lookup where you can just find the definition of what's under your cursor. But if you're in an interactive console, you can create a method object for that method name and then call dot source location on it. And it will tell you, here's where it's defined. So, very handy in the right circumstances. STEPHANIE: Awesome. That's a great tip. JOËL: Of course, one of the most effective debugging tools is having a pair, having somebody else work with you, but that's not always something that you have. And you and I were talking recently about what it's like to work solo on a project. Because you're currently on a project, you're solo, at least from the thoughtbot side of things. You're embedding with a team, with a client. Are you working on kind of, like, a solo subtask within that, or are you still kind of embedding and interacting with the other teammates on a regular basis? STEPHANIE: Yeah. So, the past couple of weeks, I am working on more of a solo initiative. The other members of my client team are kind of ramping up on some other projects for this next quarter. And since my engagement is ending soon, I'm kind of left working on some more residual tasks by myself. And this is new for me, actually. I've not really worked in a super siloed by-myself kind of way before. I usually have at least one other dev who I'm, like, kind of partnering up with on a project, or an epic, or something like that. And so, I've had a very quiet week where no one is, you know, kind of, like, reaching out to me and asking me to review their code, or kind of checking in, or, you know, asking me to check in with them. And yeah, it's just a little bit different than how I think I like to normally work. I do like to work with other people. So, this week has been interesting in terms of just kind of being a more different experience where I'm not as actively collaborating with others. JOËL: What do you think are some of the biggest challenges of being kind of a little bit out in your own world? STEPHANIE: I think the challenges for me can definitely be the isolation [laughs], and also, what kind of goes hand in hand with that is when you need help, you know, who can you turn to? There's not as much of an obvious person on your team to reach out to, especially if they're, like, involved with other work, right? And that can be kind of tough. Some of the other ones that I've been thinking about have been, you know, on one hand, like, I get to make all of the decisions that I want [laughs], but sometimes you kind of get, like, really in your own head about it. And you're not in that space of, like, evaluating different solutions that you maybe might not think of. And I've been trying to figure out how to, like, mitigate some of that risk. JOËL: What are some of the strategies that you use to try to balance, like making good decisions when you're a bit more solo? Do you try to pull in someone from another team to talk ideas through? Do you have some sort of internal framework that you use to try to figure out things on your own? What does that look like? STEPHANIE: Yeah, luckily, the feature I'm working on is not a huge project. Well, if it were, I think then I wouldn't be alone on it. But, you know, sometimes you find yourself kind of tasked with one big thing for a while, and you are responsible for from start to finish, like all of the architectural decisions to implementation. But, at least for me, the scope is a little more narrow. And so, I don't feel as much of a need to get a lot of heads together because I at least feel somewhat confident in what I'm doing [laughs]. But I have found myself being a bit more compelled to kind of just verbalize what I'm doing more frequently, even to, like, myself in Slack sometimes. It's just like, I don't know who's reading this, but I'm just going to put it out there because maybe someone will see this and jump in and say, "Oh, like, interesting. Here's some other context that I have that maybe might steer you away from that," or even validating what I have to say, right? Like, "That sounds like a good idea," or, you know, just giving me an emoji reaction [laughs] is sometimes all I need. So, either in Slack or when we give our daily sync updates, I am, I think, offering a little more details than I might if I already was working with someone who I was more in touch with in an organic way. JOËL: And I think that's really powerful because it benefits you. Sort of by having to verbalize that or type it out, you, you know, gain a little bit of self-awareness about what you're trying to do, what the struggles are. But also, it allows anybody else who has potentially helpful information to jump in. I think that's not my natural tendency. When I'm on something solo, I tend to kind of, like, zoom in and focus in on something and, like, ignore a little bit of the world around me. Like, that's almost the time when I should look at overcommunicating. So, I think most times I've been on something solo, I sort of keep relearning this lesson of, like, you know, it's really important to constantly be talking out about the things that you're doing so that other people who are in a broader orbit around you can jump in where necessary. STEPHANIE: Yeah, I think you actually kind of touched on one of the unexpected positives, at least for me. Something I wasn't expecting was how much time I would have to just be with my thoughts. You know, as I'm implementing or just in my head, I'm mulling over a problem. I have less frequent, not distractions necessarily, but interruptions. And sometimes, that has been a blessing because I am not in a spot where I have a lot of meetings right now. And so, I didn't realize how much generative thought happens when you are just kind of, like, doing your own thing for a little bit. I'm curious, for you, is that, like, a space that you enjoy being when you're working by yourself? And I guess, you know, you were saying that it's not your natural state to kind of, like, share what's going on until maybe you've fully formed an idea. JOËL: I think I often will regret not having shared out before everything is done. The times that I have done it, I've been like, that was a really positive experience; I should do that more. I think it's easy to sort of wait too long before sharing something out. And with so many things, it feels like there's only one more small task before it's done. Like, I just need to get this one test to go green, and then I can just put up a PR, and then we'll have a conversation about it. But then, oh, this other test broke, or this dependency isn't installing correctly. And before you know it, you've spent a whole day chasing down these things and still haven't talked. And so, I think if some of those things were discussed earlier, it would help both to help me feel more plugged in, but also, I think everybody else feels like they're getting a chance to participate as well. STEPHANIE: So, you mentioned, you know, obviously, there's, like, the time spent just arriving at the solution before sharing it out for feedback. But have you ever been in a position where there is no one to give you feedback and, like, not even a person to review your code? JOËL: That's really challenging. So, occasionally, if I'm working on a project, maybe it would be, like, very early-stage startup that maybe just has, like, a founder, and then I'm, like, the only technical person on the team, generally, what I'll try to do is to have some kind of review buddy within thoughtbot, so some other developer who's not staffed on my project but who has access to the code such that I can ask them to say, "Hey, can you just take a look at this and give me a code review?" That's the ideal situation. You know, some companies tend to lock things down a lot more if you're dealing with something like healthcare or something like that, where there might be some concerns around personal information, that kind of thing. But generally, in those cases, you can find somebody else within the company who will have some technical knowledge who can take a look at your code; at least, that's been my experience. STEPHANIE: Nice. I don't think I've quite been in that position before; again, I've really mostly worked within a team. But there was a conference talk I watched a little bit ago from Jeremy Smith, and it was called Building Web Apps by Your Lonesome. And he is a, like, one-man agency. And he talked about, you know, what it's like to be in that position where you pretty much don't have other people to collaborate with, to review your code. And one thing that he said that I really liked was shifting between writer and editor mode. If you are the person who has to kind of just decide when your code is good enough to merge, I like that transition between, like, okay, I just spent however many hours putting together the solution, and now I'm going to look at it with a critical eye. And sometimes I think that might require stepping away for a little bit or, like, revisiting it even the next day. That might be able to help see things that you weren't able to notice when you were in that writing mode. But I have found that distinction of roles really helpful because it does feel different when you're looking at it from those two lenses. JOËL: I've definitely done that for some, like, personal solo projects, where I'm participating in a game jam or something, and then I am the only person to review my code. And so, I will definitely, at that point, do a sort of, like, personal code review where I'll look at it. Maybe I'm doing PRs on GitHub, and I'm just merging. Maybe I'm just doing a git diff and looking at a commit in the command line on my own machine. But it is useful, even for myself, to sort of switch into that editor mode and just kind of look at everything there and say, "Is it in a good place?" Ideally, I think I do that before putting it out for a co-worker's review, so you kind of get both. But on a solo project, that has worked actually pretty well for me as well. STEPHANIE: One thing that you and I have talked about before in a different context, I think, when we have chatted about writing conference talks, is you are really great about focusing on the audience. And I was thinking about this in relation to working solo because even when you are working by yourself on a project, you're not writing the code for yourself, even though you might feel like [laughs] it in the moment. And I also kind of like the idea of asking, like, who are you building for? You know, can you ask the stakeholder or whoever has hired you, like, "Who will maintain this project in the future?" Because likely, it won't be you. Hopefully, it won't be you unless that's what you want to be doing. There's also what my friend coined the circus factor as opposed to the bus factor, which is, like, if you ran away to the circus tomorrow [laughs], you know, what is the impact that would have? And yeah, I think working solo, you know, some people might think, like, oh, that gives me free rein to just write the code exactly how I want to, how I want to read it. But I think there is something to be said about thinking about the future of who will be [inaudible 18:10] what you just happen to be working on right now. JOËL: And keep in mind that that person might be future you who might be coming back and be like, "What is going on here?" So, yeah, audience, I think, is a really important thing to keep in mind. I like to ask the question, if somebody else were looking at this code, and somebody else might be future me, what parts would they be confused by? If I was walking somebody else through the code for the first time, where would they kind of stop me through the walkthrough and be like, "Hey, why is this happening? What's the connection between these two things? I can see they're calling each other, but I don't know why." And that's where maybe you put in a comment. Maybe you find a better method or a class name to better explain what happens. Maybe you need to put more context in a commit message. There's all sorts of tools that we can use to better increase documentation. But having that pause and asking, "What will confuse someone?" is, I think, one of the more powerful techniques I do when I'm doing self-review. STEPHANIE: That's really cool. I'm glad you mentioned that, you know, it could also be future you. Because another thing that Jeremy says in this talk that I was just thinking about is the idea of optimizing for autonomy. And there's a lot to be said there because autonomy is like, yeah, like, you end up being the person who has to deal with problems [laughs], you know, if you run into something that you can't figure out, and, ideally, you'll have set yourself up for success. But I think working solo doesn't mean that you are in your own universe by yourself completely. And thinking about future, you, too, is kind of, like, part of the idea that the person in this moment writing code will change [laughs]. You'll get new information. Maybe, like, you'll find out about, like, who might be working on this in the future. And it is kind of a fine balance between making sure that you're set up to handle problems, but at the same time, maybe it's that, like, you set anyone up to be able to take it away from where you left it. JOËL: I want to take a few moments to sort of talk a little bit about what it means to be solo because I think there are sort of multiple different solo experiences that can be very different but also kind of converge on some similar themes. Maybe some of our listeners are listening to us talking and being like, "Well, I'm not at a consultancy, so this never happens to me." But you might find yourself in that position. And I think one that we mentioned was maybe you are embedded on a team, but you're kind of on a bit of a larger project where you're staffed solo. So, even though you are part of a larger team, you do feel like the initiative that you're on is siloed to you a little bit. Are there any others that you'd like to highlight? STEPHANIE: I think we also mentioned, you know, if you're a single developer working on an application because you might be the first technical hire, or a one-person agency, or something, that is different still, right? Because then your community is not even your company, but you have to kind of seek out external communities on social networks, or Slack groups, or whatever. I've also really been interested in the idea of developers kind of being able to be rotated with some kind of frequency where you don't end up being the one person who knows everything about a system and kind of becomes this dependency, right? But how can we make projects so, like, well functioning that, like, anyone can step in to do some work and then move on? If that's just for a couple of weeks, for a couple of months. Do you have any thoughts about working solo in that kind of situation where you're just stepping into something, maybe even to help someone out who's, you know, on vacation, or kind of had to take an unexpected leave? JOËL: Yeah, that can be challenging. And I think, ideally, as a team, if you want to make that easier, you have to set up some things both on a, like, social level and on a tactical level, so all the classic code quality things that you want in place, well structured, encapsulated code, good documentation, things like that. To a certain extent, even breaking down tasks into smaller sort of self-sufficient stories. I talk a lot about working incrementally. But it's a lot easier to say, "Hey, we've got this larger story. It was broken down into 20 smaller pieces that can all be shipped independently, and a colleague got three of them done and then had to go on leave for some reason. Can you step in and do stories 4 through 20?" As opposed to, "Hey, we have this big, amorphous story, and your colleague did some work, and it kind of is done. There's a branch with some code on it. They left a few notes or maybe sent us an email. But they had to go on leave unexpectedly. Can you figure it out and get it done?" The second scenario is going to be much more challenging. STEPHANIE: Yeah, I was just thinking about basically what you described, right? Where you might be working on your own, and you're like, well, I have this one ticket, and it's capturing everything, and I know all that's going on [laughs], even though it's not quite documented in the ticket. But it's, you know, maybe on my branch, or in my head, or, worst of all, on my local machine [laughs] without being pushed up. JOËL: I think maybe that's an anti-pattern of working solo, right? A lot of these disciplines that you build when you're working in a team, such as breaking up tickets into smaller pieces, it's easy to kind of get a little bit lazy with them when you're working solo and let your tickets inflate a little bit, or just have stuff thrown together in branches on your local machine, which then makes it harder if somebody does need to come in to either collaborate with you or take over from you if you ever need to step aside. STEPHANIE: Right. I have definitely seen some people, even just for their personal projects, use, like, a Trello board or some other project management tool. And I think that's really neat because then, you know, obviously, it's maybe just for their own, like, self-organization needs, but it's, like, that recognition that it's still a complicated project. And just because they're working by themselves doesn't mean that they can't utilize a tool for project management that is meant for teams or not even teams [laughs], you know, people use them for their own personal stuff all the time. But I really like that you can choose different levels of how much you're documenting for your future self or for anyone else. You had mentioned earlier kind of the difference between opening up a PR for you...you have to merge your branch into main or whatever versus just committing to main. And that distinction might seem, like, if you were just working on a personal project, like, oh, you know, why go through the extra step? But that can be really valuable in terms of just seeing, like, that history, right? JOËL: I think on solo projects, it can really depend on the way you tend to treat your commit history. I'm very careful with the history on the main branch where I want it to tell a sort of, like, cohesive story. Each commit is kind of, like, crafted a little bit. So, even when I'm working solo and I'm committing directly to master or to the main branch, I'm not just, like, throwing random things there. Ideally, every commit is green and builds and is, like, self-contained. If you don't have that discipline, then it might be particularly valuable to go through the, like, a branching system or a PR system. Or if you just want, like, a place to experiment, just throw a bunch of code together, a bunch of things break; nothing is cohesive, that's fine. It's all a work in progress until you finally get to your endpoint, and then you squash it down, or you merge it, or whatever your workflow is, and then it goes back into the main branch. So, I think that for myself, I have found that, oftentimes, I get not really a whole lot of extra value by going through a branching and PR system when it's, like, a truly solo project, you know, I'm building a side project, something like that. But that's not necessarily true for everyone. STEPHANIE: I think one thing I've seen in other people's solo projects is using a PR description and, you know, having the branching strategy, even just to jot down future improvements or future ideas that they might take with the work, especially if you haven't kind of, like, taken the next step of having that project management system that we talked about. But there is, like, a little more room for some extra context or to, like, leave yourself little notes that you might not want necessarily in your commit history but is maybe more related to this project being, like, a work in progress where it could go in a lot of different directions, and you're figuring that out by yourself. JOËL: Yeah, I mean, definitely something like a draft PR can be a great place to have work in progress and experiment and things like that. Something you were saying got me wondering what distinction you typically have between what you would put in a commit message versus something that you would put in a PR description, particularly given that if you've got, like, a single commit PR, GitHub will automatically make the commit message your PR message as well. STEPHANIE: This has actually evolved for me over time, where I used to be a lot more reliant on PR descriptions holding a lot of the context in terms of the decision-making. I think that was because I thought that, like, that was the most accessible place of information for reviewers to find out, you know, like, why certain decisions were made. And we were using, you know, PR templates and stuff like that. But now the team that I'm working on uses commit message templates that kind of contain the information I would have put in a PR, including, like, motivation for the change, any risks, even deployment steps. So, I have enjoyed that because I think it kind of shortens the feedback loop, too, right? You know, you might be committing more frequently but not, you know, opening a PR until later. And then you have to revisit your commits to figure out, like, okay, what did I do here? But if you are putting that thought as soon as you have to commit, that can save you a little bit of work down the line. What you said about GitHub just pulling your commit message into the PR description has been really nice because then I could just, like, open a thing [laughs]. And that has been nice. I think one aspect that I really like about the PR is leaving myself or reviewers, like, notes via comments, like, annotating things that should not necessarily live in a more permanent form. But maybe I will link to documentation for a method that I'm using that's a little less common or just add some more information about why I made this decision over another at a more granular level. JOËL: Yeah, I think that's probably one of the main things that I tend to put in a PR message rather than the commit message is any sort of extra information that will be helpful at review time. So, maybe it's a comment that says, "Hey, there is a lot of churn in this PR. You will probably have a better experience if you review this in split view versus unified view," things like that. So, kind of, like, meta comments about how you might want to approach reviewing this PR, as opposed to something that, let's say somebody is reviewing the history or is, like, browsing the code later, that wouldn't be relevant to them because they're not in a code review mindset. They're in a, like, code reading, code understanding mindset or looking at the message to say, "Why did you make the changes? I saw this weird method. Why did you introduce that?" So, hopefully, all of that context is in the commit message. STEPHANIE: Yeah, you reminded me of something else that I do, which is leave notes to my future self to revisit something if I'm like, oh, like, this was the first idea I had for the, you know, the way to solve this problem but, you know, note to self to look at this again tomorrow, just in case I have another idea or even to, like, you know, do some more research or ask someone about it and see if they have any other ideas for how to implement what I was aiming for. And I think that is the editor mode that we were talking about earlier that can be really valuable when you're working by yourself to spend a little extra time doing. You know, you are essentially optimizing for autonomy by being your own reviewer or your own critic in a healthy and positive way [laughs], hopefully. JOËL: Exactly. STEPHANIE: So, at the beginning of this episode, I mentioned that this is a new experience for me, and I'm not sure that I would love to do it all of the time. But I'm wondering, Joël, if there are any, you know, benefits or positives to working solo that you enjoy and find that you like to do just at least for a short or temporary amount of time. JOËL: I think one that I appreciate that's maybe a classic developer response is the heads downtime, the focus, being able to just sit down with a problem and a code editor and trying to figure it out. There are times where you really need to break out of that. You need somebody else to challenge you to get through a problem. But there are also just amazing times where you're in that flow state, and you're getting things done. And that can be really nice when you're solo. STEPHANIE: Yeah, I agree. I have been enjoying that, too. But I also definitely am looking forward to working with others on a team, so it's kind of fun having to get to experience both ways of operating. On that note, shall we wrap up? JOËL: Let's wrap up. STEPHANIE: Show notes for this episode can be found at bikeshed.fm. JOËL: This show has been produced and edited by Mandy Moore. STEPHANIE: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review in iTunes. It really helps other folks find the show. JOËL: If you have any feedback for this or any of our other episodes, you can reach us @_bikeshed, or you can reach me @joelquen on Twitter. STEPHANIE: Or reach both of us at hosts@bikeshed.fm via email. JOËL: Thanks so much for listening to The Bike Shed, and we'll see you next week. ALL: Byeeeeeeeeeeeee!!!!!! AD: Did you know thoughtbot has a referral program? If you introduce us to someone looking for a design or development partner, we will compensate you if they decide to work with us. More info on our website at tbot.io/referral. Or you can email us at referrals@thoughtbot.com with any questions.

pr building solo method sort slack github rails trello prs mandy moore kernel callback jeremy smith ascii quenneville activerecord sara jackson stephanie yeah andy croll stephanie right

Turbo Morph & ActiveRecord Encryption with Jorge Manrubia

spain turbo encryption morph ruby on rails activerecord lead programmer

Play Episode Listen Later Nov 10, 2023 46:34 Transcription Available

In this episode, Jason and Chris welcome guest, Jorge Manrubia, a Lead Programmer at 37signals in Spain known for his contributions to Ruby on Rails. Today, Jorge shares insights into his background, role at 37signals, and contributions to open source projects. He discusses his experiences, including the importance of learning from rejection and the value of experience in job interviews. The conversation dives into Jorge's work on Active Record Encryption and Console1984, and Jorge touches on the development of Turbo, with a particular focus on enhancing user interface fidelity in calendar applications using morphing. Also, they discuss the challenges of using Turbo Streams for complex updates and the benefits of using libraries like morphdom or Idiomorph for simplifying the update process. Jorge also gives us a glimpse into the upcoming release of Turbo 8, so press download to find out more! Honeybadger Honeybadger is an application health monitoring tool built by developers for developers.Disclaimer: This post contains affiliate links. If you make a purchase, I may receive a commission at no extra cost to you.

405: Retro: Sandi Metz Rules

Play Episode Listen Later Nov 7, 2023 31:58

Stephanie discovered a new book: The Staff Engineer's Path! Joël's got some D&D goodness. Together, they revisit a decade-old blog post initially published in 2013, which discussed the application of Sandi Metz's coding guidelines and whether these rules remain relevant and practiced among developers today. Transcript: JOËL: Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Joël Quenneville. STEPHANIE: And I'm Stephanie Minn. And together, we're here to share a bit of what we've learned along the way. JOËL: So, Stephanie, what's new in your world? STEPHANIE: So, I picked up a new book from the library [laughs], which that in itself is not very new. That is [laughs] a common occurrence in my world. But it was kind of a fun coincidence that I was just walking around the aisles of what's new in nonfiction, and staring me right in the face was an O'Reilly book, The Staff Engineer's Path. And I think in the past, I've plugged The Manager's Path by Camille Fournier on the show. And in recent years, this one was published, and it's by Tanya Reilly. And it is kind of, like, the other half of a career path for software engineers moving up in seniority at those higher levels. And it has been a really interesting companion to The Manager's Path, which I had read even though I wasn't really sure I wanted to be manager [laughs]. And now I think I get that, like, accompaniment of like, okay, like, instead of walking that path, like, what does a staff plus engineer look like? And kind of learning a little bit more about that because I know it can be really vague or ambiguous or look very different at a lot of different companies. And that has been really helpful for me, kind of looking ahead a bit. I'm not too far into it yet. But I'm looking forward to reading more and bringing back some of those learnings to the show. JOËL: I feel like at the end of the year, Stephanie, you and I are probably going to have to sit down and talk through maybe your reading list for the year and, you know, maybe shout out some favorites. I think your reading list is probably significantly longer than mine. But you're constantly referencing cool books. I think that would probably be a fun, either end-of-year episode or a beginning-of-year episode for 2024. One thing that's really interesting, though, about the contrast of these two particular books you're talking about is how it really lines up with this, like, fork in the road that a lot of us have in our careers as we get more senior. You either move into more of a management role, which can be a pretty significant departure from what you have to do as a developer, or you kind of go into this, like, ultra-senior individual contributor path. But how that looks day to day can be very different from your sort of just traditional sitting down and banging out tickets. So, it's really cool there's two books looking at both of those paths. STEPHANIE: Yeah, absolutely. And I think the mission that they were going for with these books was to kind of illuminate a little bit more about that fork and that decision because, you know, it can be easy for people to maybe just default into one or the other based on what their organization wants for them without, like, fully knowing what that means. And the more senior you get, the more vague and, like, figure it out yourself [laughs] the work becomes. And it can be very daunting to kind of just be thrown into that and be like, well, I'm in this leadership position now. People are looking to me, and I have all this responsibility, but, like, what do I do? Yeah, so I'm kind of enjoying this book, that is...it's not a technical book, which is actually kind of what I like about it. It's actually more of a leadership book, which is really important for that kind of role. Even though, you know, they are still in that IC track, but it does come with a lot of leadership responsibility. JOËL: Yes, leadership in a very different way than management. But—and this may be counterintuitive for some people, especially earlier on in their careers—going further up that individual contributor track doesn't just mean getting more intense technically. It often means you've got to focus on things more like leadership, like being a bit more strategic, aligning technical initiatives with strategic goals. STEPHANIE: Yeah, and having a bigger impact and being a force multiplier, even in both the manager and, like, the staff plus role, like, that, you know, is the thing that ties the rising level. JOËL: Yeah, in many ways, maybe the individual contributor track is slightly misnamed because while, yes, you're not managing a sort of sub-organization within the company, it's still about being a force multiplier. STEPHANIE: Yeah, that's a really great point [laughs]. Maybe we'll be able to come up with a better [laughs] name for that. JOËL: I've mentioned several times on this podcast that I've been enjoying playing Dungeons & Dragons, D&D, with some friends and some colleagues. And something that was particularly fun that some friends and I did this summer is we hired a professional DM to run one shot for us. And that was just an absolutely lovely experience. Well, as a result of that, I am now subscribed to this guy's newsletter. And he'll do, like, various D&D events at different times. One thing that was really cool that I found out recently...as we're recording this, it's the week before election day in the U.S. And because a lot of voting happens in schools, typically, schools have the day off. And so, this guy sent out an email saying he was offering to run a, like, all day...effectively, a little mini-D&D camp for school-age kids on election day so that you can do your work. You can go vote, and you don't have to...basically, he'll watch your kids for you and, like, get them introduced to playing D&D, which I think is just a really cool thing to do. STEPHANIE: I love that. It's so heartwarming [laughs]. And it's such a great idea because, oftentimes, people are still working, and so they need childcare, like, on those kinds of days. And yeah, I think D&D is such a fun thing for kids to get into, too. You know, it requires so much, like, imagination, and I can imagine it's such a blast. JOËL: I got that email, and I was like, that is such a perfect idea. I love it so much. STEPHANIE: I wanted to plug my D&D recommendation. I'm pretty sure I have mentioned it on the show before. But there is a podcast that I listen to called Not Another D&D Podcast, which is, you know, a live play Dungeons & Dragons podcast campaign that's hosted by these comedians, formerly of CollegeHumor, and it's very fun. I always laugh. They have this, like, a kind of offshoot of the main show that they do called D&D Court, which is very fun. Because, as you were saying, like, you know, you hired a DM to run your game. And I know that...I'm sure lots of people have fun stories about their home games and, like, the drama that happens [laughs] with their friends. JOËL: Absolutely. Absolutely. STEPHANIE: And so, with D&D Court, listeners can write in with their drama or their conflicts and get an official ruling from the hosts about who was right [laughs] in the situation that they write in about. JOËL: So, you get to bring your best rules lawyering to the D&D Court. STEPHANIE: Yeah, exactly [laughs]. JOËL: That sounds kind of amazing. Recently, I had someone reach out to me asking about an older blog post that we'd written about the Sandi Metz Rules. This blog post was initially published in 2013, so ten years ago, and was talking about some guidelines that Sandi Metz at the time was talking about that she was using in some of her code. And we talked about how our experience was applying those to some of our work as well. And so, the question was, you know, ten years later, is that still something that thoughtbot developers like to follow in their code? We'll link to the article in the show notes. But I'll just read out the rules here real quick. So, there's four of them. The first one is a class can be no longer than 100 lines of code. The second is a method can be no longer than five lines of code. The third is pass no more than four parameters into a method, and hash options count, so no getting clever with those. And then, finally, controllers can only instantiate one object. You only get one instance variable. And views can only talk to that one instance variable. Had you or are you familiar with these rules? Is that something that you think about or use in your daily writing of code? STEPHANIE: Yeah. So, when you proposed this topic, I had to revisit these rules. And I can't recall if I had seen them before. They seemed familiar. And I've read, you know, a couple of Sandi Metz's books, so maybe those were places where she had mentioned them. But the one thing that really struck me when I was first reading the rules was how declarative they were in terms of, like, kind of just telling you what the results should be without really saying how. So, for example, the one where you said, you know, a method should not be more than five lines [laughs], I had the silly thought of, like, well, you could just, you know, stuff everything into a single line [laughs] and just completely disregard line limit if you wanted, and it would technically still follow the rule. JOËL: If they didn't want us to do that, they wouldn't give us semicolons in Ruby. STEPHANIE: Exactly [laughs]. So, that is kind of what struck me at first. Is that something you noticed? JOËL: I think what is interesting with them is that there's not always a ton of rationale given behind them. Our article talks a little bit about some of the why that might be helpful and how that might look like in practice. I'm not sure what Sandi's original...I don't know if it was one of her books or maybe on a...it might have been on a podcast appearance she talked about them, so she might go more in-depth there. But yeah, they are a little bit declarative. It's just like, hey, here's...it's almost basically the kind of thing that can be enforced by a linter, which is perhaps the point. STEPHANIE: Ooh, that's really interesting. It's like, on one hand, I like how simple they are, right? It's like, they're very obvious. If you're not following them, you can tell. But on the other hand, they seem to be more of a supplement to the gained knowledge and experience that you kind of get from knowing how to implement those rules. I think you and I will both agree that we don't want to stuff everything [laughs] into a single line with semicolons. But if someone who maybe is newer to development and is coming to these rules, I think they might be wondering, like, how do I do this? JOËL: Do you follow these rules in your own code? STEPHANIE: I think the ones that are easier to follow, for me, and that I think I've come to do more instinctually, are the rules about class line length and method line length just because I'm kind of looking out for opportunities to pull out a method or, you know, make sure that this class is just doing one thing. And if it's starting to seem to cover a couple of different responsibilities, I'm a little bit more on the lookout. But I do like these rules as like, you know, like, hey, once you hit, you know, 100 lines in a class, like, maybe that's your cue to start thinking about opportunities for extraction. JOËL: Do you sort of consciously follow these rules or have them maybe even encoded in a linter? Or is it more you're following other things, and somehow, it just lines up with these principles? STEPHANIE: I would say that, like, I'm not thinking about them very actively. But that could be a very interesting exercise, and I think, you know, that's what folks did in the blog post. They were like, hey, we took these rules, and we really kept them in mind as we were developing. But I think kind of what we were talking about earlier about, like, what we've learned or the strategies we've learned to implement kind of converge on these rules. And the rules actually are more of the result of other ideas or heuristics that we follow. JOËL: I mean, you dropped the keyword heuristics there. And I think that brings me back to an earlier episode we did where we talked about heuristics. And one of the things that came up on that episode was the idea that, oftentimes, we use heuristics as a way to sort of flatten a lot of experience and knowledge into sort of one, like, short rule, or short phrase, or something, one guideline, even though it's sort of trying to just summarize a mountain of wisdom. And so, oftentimes, you can look at something like these rules and be like, okay, well, what's the point? Or maybe you even just follow it to the letter without really thinking about the why behind it, and that can sometimes be problematic. And on the other hand, you might know all of the ideas that go behind them. And without necessarily knowing the rule itself, you just kind of happen to follow it because you're intimately familiar with all of these other software principles that converge on those same ideas. STEPHANIE: Yeah, agreed. I think that the more interesting ones to me are the no more than four method arguments and only one instance variable per controller. JOËL: Interesting. STEPHANIE: I'm curious if those are sparking anything for you [laughs]. JOËL: I think the no more than four method arguments, to me, is probably the least controversial. It's generally accepted that having many arguments to a method is a code smell. And there's a few different code smells that are related to that. There's forms of coupling incandescence; there are data clumps, things like that. I've often heard a sort of rule of three. And so, if you're going more than three, then you might want to revisit the structure of what you're building. Four is a bit of an arbitrary cut-off, I'll agree. Most of these are arbitrary cut-offs. But I think the idea to maybe keep your method to fewer arguments is generally a good thing to do. STEPHANIE: I liked that the rule points out that hash options account because I think that's maybe where people get a little more hand-wavy, or you have your ops hash [laughs] that can be just a catch-all for anything. You know, it's like, once you start putting stuff in there, I don't know, I feel like it's a like a law of the universe. It's like, oh, people will just stuff more things in there [laughs]. And it takes obviously more effort or, like, specific energy to, like, think through what those things might represent, or some alternative ways of handling those arguments. We definitely do have, I think, a couple of episodes on value objects. But that's something that we have talked a lot about before in terms of, you know, how can we take some kind of primitive data, hashes included, and turn them into a richer object that can then be passed on its own? JOËL: Right. And an options hash is generally...it's too much of a catch-all to really have an identity as its own sort of value object. It doesn't really represent any single thing. It's just everything else bag of data. One thing that's interesting that the article notes is that a lot of the helpers in Rails take a lot of arguments and that it is absolutely not worth trying to fight the framework to try to follow these rules. So, the article does take a very pragmatic approach, I think, to the idea of these rules. STEPHANIE: Yeah. And I think there is even a caveat to the rules where it's like, you can break them if you have a good reason, or if you're working with someone else and they give you the thumbs up [laughs], which I really like a lot because it almost kind of compels you to stop and be like, do I have a good reason of doing this? Just making sure, or I'll run it by a friend. And shifting that, I guess, that focus from kind of just coding from, like, your default mode of thinking to a more active one. JOËL: Right. There is a rule zero, which says you can break any of the other rules as long as you convince either your pair or your reviewer to give you a thumbs up on breaking the rule. So, you'd mentioned the fourth rule about a single instance variable in a controller kind of was one of the ones that stood out to you. What is particularly striking about that rule? STEPHANIE: I think this one is hard to follow, and I think the blog post mentions that as well. Because at least I've seen our web applications grow more and more complex. And it can be really challenging to just be like, what is this page doing? Like, what, you know, data does it need to know? And have that be the single thing. Because really, a lot of our web apps have a lot of things [laughs] that they're doing, and sometimes it feels like you have to capture more than one object or at least, like, a responsibility in this way. I think that's the one that I, you know, in my ideal world, I'm like, yeah, like, we have all these, like, perfectly RESTful routes. And, you know, we're only dealing with, you know, a single resource. But once you start to have some more complexity, I think that can be a little more challenging. JOËL: I think it's interesting that you mentioned RESTful routing because I think that is maybe one of the bigger things that does trigger having more instance variables in your controller actions. If you're following sort of the traditional Rails RESTful routes, every page is generally focused on a singular resource. Now, that may not necessarily line up with a table in your database, and that's fine. But you're dealing with a singular thing or perhaps, you know, in the case of an index page, a singular collection of things, which can be represented with a single instance variable. Once you start adding custom routes that may not be necessarily tied to a particular resource, now you can very easily kind of have a proliferation of all sorts of different things that interact with each other because you're no longer centered on a single thing. STEPHANIE: Yeah, that's true, which actually reminds me of something we've talked about before, too, when we were both reading Sustainable Rails. The author talks about custom routes and actually advocates for making all routes RESTful. And if you need a vanity URL or something like that, you can always alias it. That I liked, right? It's like, okay, even if, you know, your resource is not something that's like, ActiveRecord-backed, is there some abstraction or concept of a resource in there? And I actually did really like in the blog post in the example; that is one that I've used before, too. They were dealing with this idea of a dashboard, which I would, you know, say is pretty common in a lot of web applications these days. And it's funny because a dashboard can hold so much data, right? It's really, like, a composite of a lot of different things, you know, what is most, like, useful for the user to see in one place. But they were in the blog post. And this, again, this is kind of something that I've done before. They were able to capture that with the idea of, like, a dashboard as an object and that being codified using a presenter or a facade. JOËL: Right. So, instead of having a group, and a status, and a user, and all these, like, separate things that your page that you're showing is a sort of collection of all these different types of objects, you wrap them together in a dashboard object that's kind of a facade. And I guess that really does line up with the idea of RESTful routing because you're likely going to have a dashboard's controller show action that's showing the user's dashboard. So, it makes sense, you know, that show page is rendering a dashboard object. STEPHANIE: Can we talk a little bit about things not to do, or maybe things that might be a little more questionable [laughs], and if you've seen them and how you felt about them? JOËL: I think it is sometimes tricky to define your boundaries right in that sometimes you create a facade object that really is just...it doesn't really represent anything. It's just there to wrap around some other things. And sometimes that can be awkward. I think the dashboard works partly because it lines up so neatly with the sort of RESTful routing and thinking in terms of resources that you want to do at the controller layer. But drawing boundaries incorrectly or just trying to throw everything in some kind of grab bag object can...it's not a magic bullet, right? You've got to put some thought into the data modeling, even when you are pulling the facade pattern. STEPHANIE: Yeah, I think other things that I've seen before that could theoretically follow this rule maybe [laughs], you know, I'd love to hear your thoughts about it. When you start, you're like, oh, like, my controller action method does just, you know, set one instance variable. But it turns out that there's all these other instance variables that either through a hook or, like, in the parent controller or even in the view I've seen before, too [laughs]. And I'm just kind of curious if that kind of raises your eyebrow at all or if you've seen any good reasons for doing so. JOËL: I think setting instance variables in a view would absolutely cause me to raise an eyebrow. STEPHANIE: [laughs] Agreed. JOËL: Generally, don't put logic in the view. I think that we definitely have in parent controllers; we'll set other instance variables for things like maybe a current user, right? That's how we store that state. And I think that is totally fine to have around. Typically, we don't access that instance variable directly. We're referencing some kind of helper method. But yeah, I would not consider that a violation of the rule. I think another common one that might come up is when you have some kind of nested resource. And so, in your URL, you might have a nested resource where you're saying, "Oh, I'm looking at specifically this comment under this article or something like that." And then, you want to have access to both objects in the controller. So, I think that's a pretty common scenario where you might want to have both instance variables. Something that I'm thinking about...this is not a fully formed thought, so I'm curious about your opinions here. Is there an interesting distinction between variables in code that you want to use within a controller versus variables that you want to be accessible from a view? Because instance variables in a controller are kind of overloaded. They're a way of having state in a controller, but they're also a way of passing data into a view. And so, that sort of dual purpose there maybe causes them to be a little bit trickier to reason about than instance variables in a random Ruby object. What do you think? STEPHANIE: Yeah, I was actually having the same thought as you were going there, which is that it is kind of interesting that the view, you know, is that level of what you want to display to your user. But it can have access to, like, whatever you put in the controller [laughs], and that is...and, you know, in some ways, it's like, that connection needs to happen somewhere, right? And it's here. But I think that can definitely be abused sometimes, too. So, this, you know, fourth rule that we're talking about really has to do with a more traditional Rails app. But, again, with the complexity of web apps in 2023 [chuckles], you know, we also see Rails used just as an API a lot with a separate front-end framework. And your controller is rendering some JSON, which I think has that harder boundary between what is the data that the server is involved with and what we want to send to our client. And I'm curious if you have any thoughts about how this rule applies in that situation. JOËL: I think I tend to see not really any difference there. If I'm building an API, typically, I'm trying to do so in a pretty RESTful manner. Maybe I'm doing a GraphQL API, and things might be different for that. But for a traditional REST API, yeah, typically, you're fetching one resource or some sort of compound resource, in which case, you're representing that with a facade object. And yep, you can generally get away, I think, with a single instance variable with, you know, a few exceptions around maybe some extra context about maybe something like the current user, or a parent object, or something like that. I guess the view is really you're using a different mechanism for rendering JSON, and there are a bunch out there that the community uses. I think I don't really see a difference between rendering to HTML versus rendering to JSON, or XML, or whatever. How about you? STEPHANIE: That's a good point. I think I'm with you where the rule still applies. But I have also seen things get really loosey-goosey [laughs] when we decide we're rendering JSON, and now we're suddenly putting the instance variables into a hash along with other stuff. But what you said was interesting about, like, sometimes you do need that extra context, right? And, like, figuring out what the best way to package that requires a bit of, like, sustained thought, I think, because it can, you know, be really easy to be like, oh well, this is the one interface that I have to get data from the server. So, if I just sneak this in here [laughs], what's the matter? But yeah, I think, you know, that's probably why rules like this exist [laughs] to help provide some guardrails and make us think a little deeper about it. JOËL: I think sometimes, as a community, we maybe exaggerate the differences between, like, RESTful HTML view and a RESTful JSON API. I tend to think of them as more or less the same. We just have, you know, a different representation the V layer of our MVC framework. Everything else still kind of lines up. STEPHANIE: Yeah, that's a really good point. I actually hadn't thought about it that way. Because I think maybe I have been influenced by the world of GraphQL [laughs] a little bit, or it's kind of hard to have a foot in both worlds, where you maybe have to context switch a little bit about, like, the paradigms, and then you find them influencing you in different ways. Because I have seen sometimes, like, what maybe initially were meant to be traditional more, like, RESTful JSON APIs kind of start to turn into that, like, how do we get what we need from this endpoint? JOËL: I'm curious how you feel in general about the facade pattern. Is that something that you've used, something that you like? STEPHANIE: I think I would say that I don't actually reach for it, like, upfront, right? Usually, I'm still trying to maybe put some things in my models [laughs]. But I have used it before once; it kind of became clear that, like, a lot of the methods on the model had to do with more really server-side concerns. And I was, like, wanting to just pull out some presentational pieces. I think the hardest part with the facade pattern is naming. I have really struggled sometimes to think of, like, it's not quite the component that makes it up. So, what is it instead? JOËL: Right. Right. I think, for me, sometimes the naming goes the other way around in that I'll start more to kind of, like, routing our resource level and try to think about, okay, this particular view of the data that I want to have, or this particular operation that I want to do, what am I actually dealing with? What is the resource here? So, maybe I'm viewing a dashboard. Or maybe what I'm doing is creating or destroying a subscription, even though those are not necessarily tables in the database. And once I have that underlying concept, then I can start creating an object that represents that, which might be a combination of multiple ActiveRecord models that represent tables. STEPHANIE: Yeah. You're actually pointing out, like, a really great use case that we see a lot, I think, is when you start to have to reach for resources, you know, that are different ActiveRecord classes. And how do you combine them together to represent the idea that you want, you know, for your feature? JOËL: I think it's more of, like, an outside-facing perspective rather than an inside-facing perspective. So, instead of looking at, hey, these are the set of ActiveRecord classes I have because these are my database tables, how can I, like, tack on to them to make this operation work? I'll sort of start almost from, like, a zoomed-out perspective, blank slate to say, "Hey, this is the kind of operation that I'm trying to do. What sort of resource am I dealing with ideally?" And, you know, maybe the idea is, okay, I'm dealing with a dashboard. I'm trying to subscribe to something...a newsletter, so the idea is I'm creating a subscription. Then, from there, I can start looking at, okay, do I have the concept of a subscription in this application? Oh, I don't. There is no subscriptions table because that's not a thing that we track in our data mode. That's fine. But I probably need at least some kind of in-memory object to track the idea of a subscription, and then maybe from there, that grows. So, I'm kind of working from the problem towards the database rather than from the database out. STEPHANIE: Yeah, I like that a lot. The outside-in phrase that you used really triggered something for me, which is being product engineers, right? Like, having a seat at the table when the feature is in that, like, ideation phase, I think is also really important because that's where you really learn what that like, abstraction is at the user level. And also, it could be a really good place to give your input if the feature is being designed in a way that doesn't really support the, you know, kind of quality of code and, like, separation that you would like. That's the part that I'm still working on and still learning how to do. But sometimes, you know, it's, like, really critical to the job to, like, be in that room and be like, these designs; what are some places that we could extract it at that level even? And kind of, like, separate things out from there rather than having to deal with it [laughs] deep in your codebase. JOËL: I think what I'm really kind of hearing and emphasizing in what you just said is the importance of not just writing code but being involved in the product and how that really enriches you because you know the problem domain. And that allows you to then write the code that you need at the different levels of the app to best model the situation you're working with. So, we've kind of gone through all the rules and talked about them. I'm curious, though, for you, are these rules that you follow in your code? How closely do you adhere to this set of rules? Is this still something that's relevant to you in 2023 as much as it was to the authors of that blog post in 2013? STEPHANIE: I have to say they're not ones that I have thought about on a daily basis, but after this conversation, maybe they will be. And I am kind of excited to maybe, like, bring this up to other people on my team and be like, "What do you think about these rules?" Just, like, revisiting them as a group or just, like, having that conversation. Because I think that's, you know, where I am most interested in is, like, is wondering how other people incorporate them into their work and hearing different opinions from the team. And I think there's a lot of, like, generative discussion that ultimately leads to better code as a result. JOËL: I think for myself, I'm not following the rules directly. But a lot of my code ends up approximating those rules anyway because of other principles that I follow. So, in practice, while my code doesn't strictly follow those rules, it does look pretty close to that anyway. STEPHANIE: I almost think this could be a great, you know, discussion for your team, too, like, if any listeners want to...not quite a book club but kind of an article club, if you will [laughs], and see how other people on your team feel about it. Because I think that's kind of where there is, like, a really sweet spot in terms of learning and development. JOËL: On that note, shall we wrap up? STEPHANIE: Let's wrap up. Show notes for this episode can be found at bikeshed.fm. JOËL: This show has been produced and edited by Mandy Moore. STEPHANIE: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review in iTunes. It really helps other folks find the show. JOËL: If you have any feedback for this or any of our other episodes, you can reach us @_bikeshed, or you can reach me @joelquen on Twitter. STEPHANIE: Or reach both of us at hosts@bikeshed.fm via email. JOËL: Thanks so much for listening to The Bike Shed, and we'll see you next week. ALL: Byeeeeeee!!!!!!!! AD: Did you know thoughtbot has a referral program? If you introduce us to someone looking for a design or development partner, we will compensate you if they decide to work with us. More info on our website at tbot.io/referral. Or you can email us at referrals@thoughtbot.com with any questions.

Ain't Your Callback Girl

girl programming coding github stripe rails software development prs ruby on rails callback sqlite andrew mason chris oliver all podcast activerecord

Play Episode Listen Later Aug 25, 2023 35:08

In this episode, Chris, Jason, and Andrew engage in a discussion revolving around the functionality and nuances of generated columns, callbacks, and coding practices in database and Rails applications. They explore the benefits and challenges of these features, and they dive into the complexities of coding tests. They also discuss the HTML Pipeline library, GitHub's markdown processing, and the Rails function for rendering rich text associations. Jason, Chris, and Andrew share their personal experiences, they explore the deeper layers of the Rails ecosystem, and they touch on Rails upgrades and the importance of maintaining minimal dependencies. Join us for a blend of tech insights, nostalgia, and humor! [00:00:51] Chris jumps right in and asks the guys if they've ever used any virtual generated columns, and Jason shares a story about a diesel spill in the water supply near Memphis. [00:02:31] In other news this week, Chris talks about the technical aspect of searching for users in the databases and the introduction of generated columns, he mentions Jamie's involvement in PRs related to the feature, the bugs he encountered while trying to feature in SQLite, and how generated columns work in Active Record and their current limitations. [00:09:19] Chris asks Andrew and Jason if they've ever used generated or virtual columns in the database. Jason discusses his views on callbacks and the Name of Person gem. Chris mentions Jorge's post about callbacks. [00:12:56] Jason discusses the pros and cons of using callbacks. He finds them convenient but also problematic at times. Chris provides an example where callbacks come in handy. [00:15:17] Jason states he has some high-level rules about callbacks, and Chris and Jason discuss when it's appropriate to use callbacks, like when making HTTP requests or sending emails. [00:16:16] Chris brings up an old tutorial on Stripe where the save method also involved verifying data before sending a request to Stripe. [00:17:20] Andrew introduces the idea of a “smell test” for potential pitfalls in code. He shares his experience of having to work around callbacks when they caused unexpected changes in records. [00:18:08] Jason shares his thoughts on testing, especially when callbacks create tightly coupled associations. [00:18:50] The guys share various stories about tests failing due to timing and other unexpected conditions. They also joke about different “solutions” to these issues.[00:22:24] Jason introduces the HTML-Pipeline library, which he recently used. He praises GitHub for its tech center and variable support, emphasizing its capability in content replacement. Chris recalls using GitHub for its auto-link feature which identifies HTTP and HTTPS links. [00:24:46] Chris reminisces about the early days of GitHub, its hiring spree, and the cool open source tools they released. [00:25:21] Jason describes building an action-text style structure for content, which allows for rich content editing and rendering, Chris appreciates the simplicity of this system, and they discuss the Rails function and how it renders text associations. [00:27:24] Jason highlights a limitation with the ‘render in' method, where it doesn't accept certain arguments and he talks about the structure of his board concept and the challenges faced with variable integration. [00:28:53] Chris shares his experience working on component stuff for Jumpstart Pro, emphasizing the simplicity and efficiency of their solution. Also, he emphasizes the benefits of keeping dependencies minimal for maintainability. [00:33:17] Chris was super excited to see that Rails 7.0.7 was released and speculates about Rails 7.1.0. Panelists:Jason CharnesChris OliverAndrew MasonSponsor:HoneybadgerLinks:Jason Charnes TwitterChris Oliver TwitterAndrew Mason TwitterName of Person Globals, callbacks and other sacrileges by Jorge ManrubiaHTML-PipelineRails 7.0.7 has been released by Rafael FrancaRuby Radar TwitterRuby for All Podcast

397: Dependency Graphs

Play Episode Listen Later Aug 15, 2023 42:53

Stephanie is consciously trying to make meetings better for herself by limiting distractions. A few episodes ago, Joël talked about a frustrating bug he was chasing down and couldn't get closure on, so he had to move on. This week, that bug popped up again and he chased it down! AND he got to use binary search to find its source–which was pretty cool! Together, Stephanie and Joël discuss dependency graphs as a mental model, and while they apply to code, they also help when it comes to planning tasks and systems. They talk about coupling, cycles, re-structuring, and visualizations. Ruby Graph Library (https://github.com/monora/rgl) Graphviz (https://graphviz.org/) Using a Dependency Graph to Visualize RSpec let (https://thoughtbot.com/blog/using-a-dependency-graph-to-visualize-rspec-let) Mermaid.js (https://mermaid.js.org/) Strangler Fig pattern (https://martinfowler.com/bliki/StranglerFigApplication.html) Transcript: JOËL: Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Joël Quenneville. STEPHANIE: And I'm Stephanie Minn. And together, we're here to share a bit of what we've learned along the way. JOËL: So, Stephanie, what's new in your world? STEPHANIE: So, I'm always trying to make meetings better for me [chuckles], more tolerable or more enjoyable. And in meetings a lot, I find myself getting distracted when I don't necessarily want to be. You know, oftentimes, I really do want to try to pay attention to just what I'm doing in that meeting in the moment. In fact, just now, I was thinking about the little tidbit I had shared on a previous episode about priorities, where really, you know, you can only have one priority [laughs] at a time. And so, in that moment, hopefully, my priority is the meeting that I'm in. But, you know, I find myself, like, accidentally opening Slack or, like, oh, was I running the test suite just a few minutes before the meeting started? Let me just go check on that really quick. And, oh no, there's a failure, oh God, that red is really, you know, drawing my eye. And, like, could I just debug it really quick and get that satisfying green so then I can pay attention to the meeting? And so on and so forth. I'm sure I'm not alone in this [laughs]. And I end up not giving the meeting my full attention, even though I want to be, even though I should be. So, one thing that I started doing about a year ago is origami. [laughs] And that ended up being a thing that I would do with my hands during meetings so that I wasn't using my mouse, using my keyboard, and just, like, looking at other stuff in the remote meeting world that I live in. So, I started with paper stars, made many, many paper stars, [laughs] and then, I graduated to paper cranes. [laughs] And so, that's been my origami craft of choice lately. Then now, I have little cranes everywhere around the house. I've kind of created a little paper crane army. [laughs] And my partner has enjoyed putting them in random places around the house for me [laughs] to find. So, maybe I'll open a cabinet, and suddenly, [laughs] a paper crane is just there. And I think I realized that I've actually gotten quite good at doing these crafts. And it's been interesting to kind of be putting in the hours of doing this craft but also not be investing time, like, outside of meetings. And I'm finding that I'm getting better at this thing, so that seemed pretty cool. And it is mindless enough that I'm mentally just paying attention but, yeah, like, building that muscle memory to perfecting the craft of origami. JOËL: I'm curious, for your army of paper cranes, is there a standard size that you make, or do you have, like, a variety of sizes? STEPHANIE: I have this huge stack of, like, 500 sheets of origami paper that are all the same size. So, they're all about, let's say, two or three inches large. But I think the tiny ones I've seen, really small paper cranes, maybe that would be, like, the next level to tackle because working with smaller paper seems, you know, even more challenging. JOËL: I'd imagine the ratio of, like, paper thickness to the size of the thing that you're making is different. STEPHANIE: At this point, they say that if you make 1,000, then you bring good luck. I think I'm well on my way [laughs] to hopefully being blessed with good luck in this household of my little paper crane army. JOËL: It's interesting that you mentioned the power of having something tactile to do with your hands during a meeting, and I definitely relate to that. I feel like it's so easy, even, like, mindlessly, to just hit Command-Tab when I'm doing things on a screen. Like, my hands are on the keyboard. If I'm not doing something, I'm just going to mindlessly hit Command-Tab. It's kind of like on your phone sometimes. I don't know if you do this, like, just scrolling side to side. You're not actually doing anything. You just want motion with your fingers. STEPHANIE: Yes. I know exactly what you're talking about. And it's funny because it's a bit of a duality where, you know, when you are in your development workflow, you want things to be as quick and convenient as possible, so that Command-Tab, you know, is very easy. It's just built in, and that helps speed up your, you know, day-to-day work. But then it's also that little bit of mindlessness, I think, that can get you down the distraction path. When I was first looking for something to do with my hands, to have, like, a little tactile thing to keep me focused in meetings, I did explore getting one of those fidget cubes; I have to say. [laughs] It's just a little toy, you know, that comes with a bunch of different settings for you to fidget with. There's, like, a ball you can roll, you know, with your thumb, or maybe some buttons to click, and it gives you that really satisfying tactile experience. And I know they work really well for a lot of people, but I've really enjoyed the, I guess, the unexpected benefits [chuckles] of getting better at a hobby [laughs] while spending my time at my work. Joël, what is new with you? JOËL: So, a few episodes ago, I talked about a really kind of frustrating bug that I was chasing down that was due to some, like, non-determinism in the environment. And it kind of came, and then it went away. And I wasn't able to get sort of closure on that and had to move on. Well, this week, that bug popped up again, and this time, I was actually able to chase it down. So, that felt really exciting. And I got to use binary search to try to find the source of it, which made me feel really cool. STEPHANIE: Oooh, do tell. What ended up being the issue? JOËL: I'm connecting to an external Snowflake data warehouse, and ActiveRecord tries to fetch the schema and crashes as part of that with some cryptic error that originates from the C extension ODBC Ruby driver package. I figured out that it's probably something to do with, like, a particular table name or something in the table metadata when we're pulling this schema that we're not happy about. But I don't know which table is the one that it's not happy with. Well, this time, I was able to figure out, by reading through some of the documentation, that I can pull subsets of the schema. So, I can pull the first n values of that schema, and it won't crash. It only crashes if I try to fetch the entire set, which is what is happening under the hood. At that point, you know, I could fetch each row individually, but there's hundreds of these. So, you know, I try, okay, what happens if I try to fetch 1,000 of these? Is it going to crash? Because it's a massive system. So, yes, I get a crash. So, I know that a table less than a thousandth in the list of tables is what's causing the problems. So, okay, fetch 500 halfway in between there. It's still going to crash. Okay, 250, 125. I then kind of keep halving all the time until I find one that doesn't crash. And now I know that it is somewhere between the last crash and this one. So, I think it was between 125 and 250. And now I can say, okay, well, let's fetch the first, you know, maybe 200 tables, okay, that crashes. And I keep halving that space until you finally find it. And then, like, okay, so it's this one right here. Now, the problem is the bad table actually crashes. So, I think it ended up being, like, number 175 or something like that. So, I never get to see the actual table itself. But because the list of tables is in alphabetical order, and I can see because I can fetch the first 174 and it succeeds, so I can tell what the previous 5, 6, you know, previous 174 are. I can pretty easily go and look at the actual database and the list of tables and say, okay, well, it's in the same order. And the next one is this one, and hey, look, there is some metadata there that has some very long fields that are longer than one might expect, specifically going over a potentially implied 256-character limit. That seems somewhat suspicious. And, oh, if we remove this table, all of a sudden, everything works. STEPHANIE: Wow, binary search, an excellent debugging tool [laughs] when you have no idea, you know, what could possibly be causing your issue. JOËL: It's such a cool tool. Like, I'm always so happy when I get a chance to use it. The problem is, you need a way to be able to answer the question, like, have I found it? Yes or no? Or, generally, is it greater or less than this current position? STEPHANIE: Well, that's really exciting that you ended up figuring out how to solve the bug. I know last time we talked about it, you kind of had left off in a space of, hopefully, we won't run into this issue again because it's no longer happening. But it seems like you were also set up this time around to be able to debug once it cropped up again. JOËL: Yes. So, binary search is really cool. It's got this, like, very, like, fancy computer science name. But in reality, it's a fairly simple, straightforward technique that I use fairly frequently in my development. And there's another kind of computer sciency fancy-sounding concept that I use all the time. You've all heard me reference this multiple times on the show. You're right; we're finally doing it. This is the dependency graph episode. STEPHANIE: Woo. [laughter] It's time. I'm excited to really dig into it because, you know, as someone who has heard you talk about it a lot, you know, and is maybe a little less familiar with graph theory and how, you know, it can be applied to my day to day work, I'm really excited to dig into a little bit about, you know, what a regular developer needs to know about dependency graphs to add to their toolbox of skills. JOËL: So, I think at its core, the idea of a dependency graph is that you have a group of entities, some of which depend on each other. They can't do a task, or they can't be created unless some other subtasks or dependent actions take place. And so, we have a sort of formal structural way of describing these things. Visually, we often draw these things out where each of the pieces is like a little bubble or a circle, and then we draw arrows towards the things that it depends on. So, if A cannot be done without B being done first, we draw an arrow from A to B. That's kind of how it is in the abstract. More concretely, this kind of thing shows up constantly throughout the work that we do because a lot of what we do as developers is managing things that are connected to each other or that depend on each other. We build complex systems out of smaller components that all rely on each other. STEPHANIE: Yeah, I think it's interesting because I use the word dependency, you know, very frequently when talking about normal work that I'm doing, you know, dependencies as in libraries, right? That we've pulled into our application, or dependencies, like, talking about other classes that are referenced in this class that I'm working in. And I never really thought about what could be explored further or, like, what could be learned from really digging into those connections. JOËL: It's a really powerful mental model. And, like you said, dependencies exist all over our work, and we often use that word. So, you mentioned something like packages, where your application depends on Rails, which in turn depends on ActiveRecord, which in turn depends on a bunch of other things. And so, you've got this whole chain of maybe immediate dependencies, and then those dependencies have dependencies, and those dependencies have dependencies, and it kind of, like, grows outward from there. And in a very kind of simplistic model, you might think, oh, well, it's more, like, a kind of a tree structure. But oftentimes, you'll have things like branches on one side that connect back to branches on the other. And now you've got something that's no longer really tree-like. It's more of a sort of interconnected web, and that is a graph. STEPHANIE: I think understanding the dependencies of your system has also become more important to me as I learn about things that can go wrong when I don't know enough about what my system is, you know, relying on that I had kind of taken for granted previously. I'm especially thinking about packages like we were mentioning, and, you know, not realizing that your application is dependent on this other library, right? That's brought in by a gem that you're using. And there's maybe, like, a security issue, right? With that. And suddenly, you have this problem on your hands that you didn't realize before. And I know that that has been more of a common discussion now in terms of security practices, just being more aware of all the things that you are depending on as really our work becomes more and more interconnected with the things available to us with open source. JOËL: I think where understanding the graph-like nature of this becomes really important is when you're doing something like an upgrade. So, let's say you do have a gem that has a security problem, and you want to upgrade it to fix that security issue. But the upgrade that includes the security patch is also a breaking upgrade. And so, now everything else in your system that depends on that gem or on that package is going to break unless you have them in a version that is compatible with the new version of that gem. And so, you might have to then go downstream and upgrade those packages in a way that's compatible with your app before you can bring in the security patch. And a lot of that can be done automatically by Bundler. Bundler is software that is built around navigating dependency graphs like that and finding versions that are compatible with each other. But sometimes, your code will need to change in order to upgrade one of these downstream gems so that you can then pull in the upgrade from the gem that needs a security patch. And so, understanding a little bit of that graph is going to be important to safely upgrading that gem. STEPHANIE: So, I know another application of dependency graphs that you have thought about and written a blog post for is RSpec let declarations and how a lot of the time when we are using let, you know, we are likely calling other variables defined by let. And so, when you are encountering a test file, it can be really hard to grok what data is being set up in your test. JOËL: Yeah, so that is really interesting because you can define something that will get executed in a lazy fashion if it gets referenced. But then not only is the let lazy and will not trigger unless it's referenced, but a let can reference other lets, which are also lazy, and only get triggered if they get referenced. So, you might have a bunch of lets defined in any order you want throughout a file, and they're all kind of interconnected with these references to each other. But they only get triggered if something calls it directly or it's in this, like, chain of dependencies. And getting a grasp on what actually gets created, which lets will actually execute, which ones don't in a file can quickly get out of hand. And so, thinking of this in terms of a dependency graph has been a really helpful mental model for me to understand what's going on in a complex test file. STEPHANIE: Yeah, absolutely. Especially when sometimes the lets are coming from all over the place, you know, maybe a describe block hundreds of lines away, or even a completely different file if you are using a shared context that's being pulled in. So, I can see why this was a complex problem that could be made a little simpler with plotting out a dependency graph. And in preparation for this episode, I was doing a little bit of my own exploration on this because I certainly know, you know, the pain of trying to figure out what is being executed in my tests when there are a lot of lets that reference each other. And in the blog post, you kind of gave a little step-by-step of how you could start with creating a dependency graph for the test that you're working with. And I was really curious if this process could be automated because, you know, I do enjoy, you know, pulling out the pen and paper [chuckles] every now and then. But I'm not, like, a particularly visual person. God forbid I, like, draw a circle, but then, like, don't have enough space for the rest of the circles. [laughs] So, I was really hoping for a tool that could do this for me, especially if, you know, you do, you have a lot of tests that you have to try to understand in a relatively short amount of time. And so, I ended up doing something kind of hacky with RSpec and overriding let definitions to automate this process. JOËL: That's really cool. So, is the tool that you're trying to build something where you feed it in a spec file, and it gives you some kind of graphical representation like an SVG or something as output? STEPHANIE: Yeah. I did consider that approach first, where you feed in the file, but then I ended up going with something more dynamic where you are running the test, and then as it gets executed, tracing the let definitions and then registering them to build your dependency graph. JOËL: So, you've got some sort of internal modeling that describes a dependency graph. And then, somehow, you're going to turn that, you know, a series of Ruby objects into some kind of visual. STEPHANIE: Yeah, exactly. And the bulk of that work was actually done with a library called RGL, which stands for just Ruby Graph Library. [laughs] And what's nice is that it has a really easy interface for plugging in the vertices and edges of the dependency graph that you want to build. And then, it is already hooked up with Graphviz to, you know, write the SVG to a file. And so, I ended up really just having to build up an array of my dependencies and the connections to each other and then feed it into the constructor of the graph. JOËL: And for all of our listeners, you mentioned Graphviz. That is a third-party tool that can be installed on your machine that can generate these SVG diagrams from...I believe it has its own sort of syntax. So, you create, I believe it's dot, D-O-T, so dot dot file. And based off of that, it generates all sorts of things, but SVG being potentially one of them. STEPHANIE: Yeah. The nice thing was that I actually didn't end up having to use the DSL of Graphviz because the RGL gem was doing them for me. JOËL: Nice. So, it plugs in directly. STEPHANIE: Yeah, exactly. And I was really curious about using this gem because I, you know, just wanted to write Ruby, especially to plug into other things that are already in Ruby. And I found that surprisingly easy, thanks to all of the RSpec config options that they make available to you, including an option to extend the example group class, which is actually where let and let bang is defined. And so, I ended up overriding those classes and using, you know, the name of the let that you're defining and then the block to basically register the dependencies. And I also ended up exploring a little bit with using Ruby's built-in parser to figure out in the block that's being passed to the let, what parts of that block could potentially be a reference to another let. JOËL: That's really cool. Did you get any fun results from that? STEPHANIE: I did. It worked pretty well in being able to capture all of the let declarations, and other lets that it references. And so, I was able to successfully, you know, like, generate a visual dependency graph of all of the lets, so that was really neat. The part that I was really kind of excited about trying next, though I didn't end up having time to yet, was figuring out which of those let values are executed by way of the let bang, right? Which is eager or what is referenced in the test that then gets executed as well. And so, the RGL library is pretty neat and has some formatting options, too, with the Graphviz output. So, you can change the font color or styling options for different, you know, nodes and edges. And so, I was really curious to pursue this further, maybe, and use it to show exactly what gets evaluated now that I have successfully mapped my let graph. JOËL: Right. Because the whole point of this exercise is that not the entire graph is going to get evaluated. The underlying question is, what data actually gets created when my test runs? And so, you build out this whole dependency graph, and then you can follow a few simple rules to say, okay, this branch gets called, this branch gets called, this series of things gets called. And okay, this subset of let blocks trigger, and therefore this data has been created for my given test. STEPHANIE: Yeah. Though I will say that even where I got so far to, just seeing all of the let definitions in a spec file was really helpful to have a better understanding, you know, if I do have to add a test in here, and I'm thinking about reaching for a pre-existing let declaration, to be like, oh, like, it actually, you know, goes on to reference all of these other things that may be factories [chuckles] that are created might make me, you know, think twice, or just have a little better understanding of what I'm really dealing with. JOËL: Right. The idea that when you're calling out to a let, or a factory, or something else that's just a node in a large graph, you're not necessarily referencing just one thing. You might actually be referencing the head of a very long chain of things that maybe you don't intend to trigger the whole thing. STEPHANIE: Yeah, exactly. JOËL: So, in that sense, having a sort of visual or at least an idea of the graph can give you a much better sense of the cost of certain operations that you might have to do. STEPHANIE: The cost of the operations certainly, especially when, you know, you are working in a legacy codebase, and you, you know, like, maybe don't know how everything plays together or is connected. And it's very tempting to just reach for [chuckles] the things that have been, you know, created or built for you. And I'm certainly guilty of that sometimes on this client project, where the domain is so complex, and there are so many associated models. And I'm like, well, like, let me just, you know, use this let that already, you know, has a factory set up for what I think I need for this test. But then realizing, oh, actually, like, it is creating all these things, and do I really need them? I think it can be really challenging to unravel all of that in your head. And so, with this very scrappy tool that I [chuckles] built for my own purposes, you know, maybe it makes it, like, one step easier to try to fully understand what I'm working with and maybe do something different. JOËL: One aspect that I think is really powerful about dependency graphs is that it takes this kind of, like, abstract concept that we oftentimes have an intuitive sense around, the idea that we have different components that depend on each other, and it shows it to us visually on, like, a 2D plane. And that can be really helpful to get an understanding or an overview of a system. You mentioned that RGL uses Graphviz to generate some SVGs. A visual tool that I've been using to draw some of my dependency graphs has been mermaid.js. It has a syntax that's, like, a text-based syntax, but it's almost visual in that you have a piece of text and name of a node. And then, you'll draw a little ASCII arrow, you know, two dashes and a greater than sign to say this thing depends on, and then write another name, and just have a row, like, a bunch of entries to say; A depends on B. A also depends on C. C depends on D, and so on, and, like, build up that list. And then Mermaid will just generate that diagram for you. STEPHANIE: Yeah. I've used Mermaid a few times. One really helpful use that I had for it was diagramming out a bunch of React components that I had and wanting to understand the connections between them. And I think you can even paste the Mermaid syntax into your GitHub pull request description, and it'll render as the graph image. JOËL: Yeah, that's what's really cool is that Mermaid syntax has become embedded in a lot of other places in the past few years. So, it's really easy to embed graphs now into all sorts of things. You mentioned GitHub. It works in pull requests descriptions, comments, I think pretty much anywhere that Markdown is accepted. So, you could put one in your README if you wanted. Another place that I use a lot, Obsidian, my note-taking tool, allows me to embed graphs directly in there, which is really much nicer than previously; sometimes, when I wanted to express something as a visual, I would use some sort of drawing tool to do something and export an image, and then embed that in my note. But now I can just put in this text, and it will automatically render that as a diagram. And part of what's really nice about that is that then it's really easy for me to go and change that if I'm like, oh, but actually, I want to add one more connection in here. I don't have to re go back to, hopefully, a file that I've saved somewhere and, like, change an image file and re-export it. I just, you know, I add one line of text to my note, and it just works. STEPHANIE: That's awesome. Yeah, the ability to change it seems really useful. So, we've talked a little bit about tools for creating a visual aid for understanding our dependencies. And now that we have our graph, maybe we might have some concerning observations about what we see, especially when perhaps some of our dependencies are pointing back to each other. JOËL: Yes. So, I think you're referencing cycles, in particular. That would be the formal term for it. And those are really interesting. They happen in dependency graphs. And I would say, in many cases, they can be a bit of a smell. There's definitely situations where they're fine. But there are things that you look at, and you're like, okay, this is going to be a more complex kind of tricky bit of the graph to work with. Some cases, you just straight up can't have them. So, I want to say that the way RSpec lets are set up, you cannot write code that produces cycles. But you might have...I think Ruby allows classes to reference each other in such a way that it creates a cycle, and not all languages do that. So, Elm and F#, I believe, require that modules cannot reference each other. The fancy term for this is a dependent acyclic graph, or DAG, which basically just means that there are no cycles in that graph. STEPHANIE: Yeah. What you said about classes referencing each other is very interesting because I've definitely seen that. And then, if I have to go about changing something, maybe even it's just the class name, right? Now there's no way in which I can really make just one change. I have to kind of do it all in one go. JOËL: I think that's a common property of a cycle, and a graph is that changes that happen somewhere in that cycle often need to be all shipped together as one piece. You can't break it up into smaller chunks because everything depends on everything else. So, it has to be kind of boxed together and shipped as one thing. STEPHANIE: And you'd mentioned that cycles, you know, can be a bit of a code smell. And if the goal is to be able to break it up so that it is a little bit more manageable to work with, how would you go about breaking a cycle? JOËL: So, I think breaking a cycle is going to vary a little bit based on your problem domain. So, are you modeling a series of classes that are referencing each other? Is this a function call graph? Is this even, like, a series of tasks that you're trying to do? But typically, what you want to do is make sure that eventually, at some point, like, something doesn't loop back to referencing something higher up in your hierarchy. And so, oftentimes, it ends up being about what is allowed to know about what? Do you have higher-level concepts that can know and depend on lower-level concepts but not vice versa? And again, we are talking about this a little bit at the abstract level. But in terms of, let's say, different code modules, or classes, or something like that, commonly, you might say, well, we want some sort of layering where we have almost, like, more primitive types of classes at the bottom. And they don't get to know about anything above them. But the ones above that might be more complex that are composed of smaller pieces know about the ones below them. And you might have multiple layers kind of like that that all kind of point down, but nothing points up. STEPHANIE: That is a very common heuristic. [chuckles] I think you were basically just describing how I also understand creating React components, where you want to separate your presentational ones from your functional ones. And, yeah, it makes a lot of sense that as soon as you start adding that complexity of, you know, those primitive classes at the bottom, starting to, you know, point to things higher up or to know about things higher up, that is where a cycle may be accidentally introduced. JOËL: It's interesting just how many design principles that we have in software. If you dig into them a little bit, you find out that they're about decoupling things, and oftentimes, it's specifically breaking up cycles. So, one way that you might have something like this that actually has dependency in the name, the dependency inversion principle, where what you're effectively doing is you're taking one of those dependency arrows, and you're flipping it the other way. So, instead of A depending on B, you're flipping it. Now B depends on A, and that can be enough to break a cycle. STEPHANIE: So, one thing I've picked up from our conversations about dependency graphs is that oftentimes, you know, when you're trying to figure out where to start, you want to look for those areas or those nodes where there's nothing else that depends on it. JOËL: Yeah. I think you have those nodes that, if this were a tree, you would call them the leaf nodes. In the case of a graph, I'm not sure if that's technically correct, but they don't depend on anything. They're kind of your base case. And so, you can, you know, if it's a function, you can run it. If it's a file, you can load it; if it's a class, also you can load it up and not have to do anything else because it has no dependencies. And knowing that those are there, I think, can be really useful in terms of knowing an order you might want to execute something in. And this is really interesting for one of my favorite uses of a graph, which is breaking down a series of tasks that you need to do. So, commonly, you might say, okay, I have a large task I need to do. I break it down into a series of subtasks. And, you know, maybe I draw out, like, a bulleted list and, you know, task 1, 2, 3, 4, 5. The problem is that they're not necessarily just a flat list. They all have, like, orders, like dependencies between each other. So, maybe one has to happen before 2, but it also has to happen before 3, which needs to happen before two, and, like, there's all these interconnections. And then, you find out that you can't ship them independently the way you thought initially. So, by building up a graph, you end up with something that shows you exactly what depends on what. And then, like you said, the parts that are really interesting where you can start doing work are the ones that have no dependencies themselves. Other things might depend on them, but they have no dependencies. Therefore, they can be safely built, shipped, deployed to production, and they can be done independently of the other subtasks. STEPHANIE: Yeah. I was also thinking about things that could be done in parallel as well. So, if you do have multiple of those items with no dependencies, like, that is a really good way to be able to break up that work and, yeah, identify things that are not blocked. JOËL: For a complex set of tasks, it's great to see, okay, these two pieces have no dependencies. We can have them be done in parallel, shipped independently. And then you can just kind of keep repeating that process. Because once all of the tasks that have no dependencies have been done, well, you can almost, like, remove them from the graph and see, okay, what's the new set of things that have no dependencies? And then, keep doing that until you've eventually done the whole graph. And that may sound like, oh okay, we're just kind of using a little bit of intuition and working through the graph. It turns out that this is a, like, actual, like, formal thing. When it comes to graphs, it's a traversal algorithm called topological sort is the fancy name for it, and it basically, yeah, it goes through that. It gives you a list of nodes in order where each node that you're given has no dependencies that have not been evaluated yet. So, it works from effectively to use our tree terminology, from the leaf nodes to the root, potentially roots plural, of the graph, and each step is independent. So that's a lot of, like, fancy terminology, and getting a little bit of, like, computer science graph theory into here. So, my, like, general heuristic is that graphs should be evaluated from the bottom up when you're trying to evaluate each piece independently. So, when you do that, you get to do each piece independently, as opposed to if you're evaluating from the top down. So, starting from the one thing that depends on everything else, well, it can't be shipped until all of its dependencies have been shipped. And all the transitional dependencies can't be shipped until their dependencies have been shipped. And so, you end up being not able to ship anything until you've built the entire graph. And that's when you end up with, you know, a 2,000-line PR that took you multiple weeks and might be buggy. And it's going to take a long time to review. And it's just not what anybody wants. STEPHANIE: I'm glad you brought this up because I think this is where I am really curious to get better at because oftentimes, when I am breaking down a complex task, it's quite hard for me to see all of the steps that need to happen. And so, you know, you maybe start out with that, like, top-level node, like, the task that needs to be done as you understand it immediately. And it's really hard to actually identify the dependencies and, like, the smaller pieces along the way. And because you're not able to identify that, you think that you do have to just do it all in one go. JOËL: Yeah, that sort of root node is typically the overarching task, the goal of what you want to do. And a common, I think, scenario for something like this would be, let's say, you're doing a Rails upgrade. And so, that root node is upgrade Rails. And a common thing that you might want to do is say, okay, let's go to the gem file, upgrade Rails, see what breaks, and then just keep fixing those things. That's working from the top down. And you're going to be in a long-running branch, and you're going to keep fixing things, fixing things, fixing things until you have found all the things but done all the things. And then you do a big bang upgrade that may have taken you weeks. As opposed to if you're working from the bottom up, you try to figure out, okay, what are all the subtasks? And that might take some exploration. You might not know upfront. But then you might say, okay, here, I can upgrade RSpec versus a dependency, or I need to change the interface of this class and ship all these pieces one at a time. And then, the final step is flipping that upgrade in the gem file, saying, okay, now I've upgraded Rails from 4 to 5, or whatever the version is that you're trying to do. STEPHANIE: I think you've really hit the nail on the head when it comes to trying to do something but not knowing what subtasks may compose of it and getting into that problem of, you know, having not broken it down, like, enough to really see all the dependencies. And, you know, maybe this is a conversation [chuckles] for another episode, but the skill of breaking up those tasks and exploring what those dependencies are, and being able to figure them out upfront before you start to just do that upgrade and then see what happens, that's definitely an area that I want to keep investing in. And I'm sure other people would be really curious about, too, to help them make their jobs easier. JOËL: I think one tip that I've learned that's really fun and that connects into all of this is sometimes you do end up with a cycle in your dependencies of tasks. A technique for breaking that up is a pattern that I have pitched multiple times on the show: the strangler fig pattern. And part of why it's so powerful is that it allows you to work incrementally by breaking up some of these cycles in your dependency graph. And one of the lessons that I've learned from that is that just because you have sort of an initial set of subtasks and you have a graph of them doesn't mean that you can't change them. If you're following strangler fig, what you're actually doing is introducing one or more new subtasks to that graph. But the way you introduce them breaks up that cycle. So, you can always add new tasks or split up existing ones as you get a better understanding of the work you need to do. It's not something that is fixed or set in stone upfront. STEPHANIE: Yeah, that's a really great tip. I think next time, what I really want to explore, you know, your heuristic of going from bottom up, yeah, sure, it sounds all fine and dandy. But how to get to a point where you're able to see everything at the bottom, right? And, like, when you are tasked, or you do start with the thing at the top, like, the end goal. Yeah, I'm sure that's something we'll explore [chuckles] another day. JOËL: On that note, shall we wrap up? STEPHANIE: Let's wrap up. Show notes for this episode can be found at bikeshed.fm. JOËL: This show has been produced and edited by Mandy Moore. STEPHANIE: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review in iTunes. It really helps other folks find the show. JOËL: If you have any feedback for this or any of our other episodes, you can reach us @_bikeshed, or you can reach me @joelquen on Twitter. STEPHANIE: Or reach both of us at hosts@bikeshed.fm via email. JOËL: Thanks so much for listening to The Bike Shed, and we'll see you next week. ALL: Byeeeeeee!!!!!! ANNOUNCER: This podcast is brought to you by thoughtbot, your expert strategy, design, development, and product management partner. We bring digital products from idea to success and teach you how because we care. Learn more at thoughtbot.com.

god pr react slack 2d github snowflakes mermaid rails dependency obsidian graphs dag elm mandy moore visually dsl svg markdown readme ascii quenneville bundler svgs activerecord rspec rgl stephanie yeah stephanie well

396: Build vs. Buy

Play Episode Listen Later Aug 8, 2023 33:57

Joël has been fighting a frustrating bug where he's integrating with a third-party database, and some queries just crash. Stephanie shares her own debugging story about a leaky stub that caused flaky tests. Additionally, they discuss the build vs. buy decision when integrating with third-party systems. They consider the time and cost implications of building their own integration versus using off-the-shelf components and conclude that the decision often depends on the specific needs and priorities of the project, including how quickly a solution is needed and whether the integration is core to the business's value proposition. Ruby class instance variables (https://www.codegram.com/blog/understanding-class-instance-variables-in-ruby/) Build vs Buy by Josh Clayton (https://thoughtbot.com/blog/build-vs-buy-considerations-for-new-products) Sustainable Rails (https://sustainable-rails.com/) Transcript: STEPHANIE: Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Stephanie Minn. JOËL: And I'm Joël Quenneville. And together, we're here to share a bit of what we've learned along the way. STEPHANIE: So, Joël, what's new in your world? JOËL: My world has been kind of frustrating recently. I've been fighting a really frustrating bug where I'm integrating with a third-party database. And there are queries that just straight-up crash. Any query that instantiates an instance of an ActiveRecord object will just straight-up fail. And that's because before, we make the actual query, almost like a preflight query that fetches the schema of the database, particularly the list of tables that the database has, and there's something in this schema that the code doesn't like, and everything just crashes. Specifically, I'm using an ODBC connection. I forget exactly what the acronym stands for, Open Database connection, maybe? Which is a standard put up by Microsoft. The way I'm integrating it via Ruby is there's a gem that's a C extension. And somewhere deep in the C extension, this whole thing is crashing. So, I've had to sort of dust off some C a little bit to look through. And it's not super clear exactly why things are crashing. So, I've spent several days trying to figure out what's going on there. And it's been really cryptic. STEPHANIE: Yeah, that does sound frustrating. And it seems like maybe you are a little bit out of your depth in terms of your usual tools for figuring out a bug are not so helpful here. JOËL: Yeah, yeah. It's a lot harder to just go through and put in a print or a debug statement because now I have to recompile some C. And, you know, you can mess around with some things by passing different flags. But it is a lot more difficult than just doing, like, a bundle open and binding to RB in the code. My ultimate solution was asking for help. So, I got another thoughtboter to help me, and we paired on it. We got to a solution that worked. And then, right before I went to deploy this change, because this was breaking on the staging website, I refreshed the website just to make sure that everything was breaking before I pushed the fix to see that everything is working. This is a habit I've picked up from test-driven development. You always want to see your test break before you see it succeed. And this is a situation where this habit paid off because the website was just working. My changes were not deployed. It just started working again. Now it's gotten me just completely questioning whether my solution fixes anything. The difficulty is because I am integrating [inaudible 03:20] third-party database; it's non-deterministic. The schema on there is changing rather frequently. I think the reason things are crashing is because there's some kind of bad data or data that the ODBC adapter doesn't like in this third-party system. But it just got introduced one day; everything started breaking, and then somehow it got removed, and everything is working again without any input or code changes on my end. So, now I don't trust my fix. STEPHANIE: Oh no. Yeah, I would struggle with that because your reality has come crashing down, [laughs] or how you understood reality. That's tough. Where do you think you'll go from here? If it's no longer really an issue in this current state of the schema, is it worth pursuing further at this time? JOËL: So, that's interesting because it turns into a prioritization problem. And for this particular project, with the deadlines that we have, we've decided it's not worth it. I've opened up a PR with my fix, with some pretty in-depth documentation for why I thought that was the fix and what I think the underlying problem is. If this shows up again in the next few days, I'll have that PR that I can pull in and see if it fixes things, and if it doesn't, I'll probably just close that PR, but it'll be available for us if we ever run into this again. I've also looked at a few potential mitigating situations. Part of the problem is that this is a, like, massive system. The Rails app that I'm using really doesn't need to deal with this massive database. I think there's, you know, almost 1,000 tables, and I really only care about a subset of tables in, like, one underlying schema. And so, I think by reducing the permissions of my database user to only those tables that I care about, there's a lower chance of me triggering something like this. STEPHANIE: Interesting. What you mentioned about, you know, having that PR continue to exist will be really helpful for future folks who might come across the same problem, right? Because then they can see, like, all of the research and investigation you've already done. And you may have already done this, but if you do think it's a schema issue, I'm curious about whether the snapshot of the schema could be captured from when it was failing to when it has magically gotten fixed. And I wonder if there may be some clues there for some future investigator. JOËL: Yeah. I'm not sure what our backup situation is because this is a third-party system, so I'd have to figure out what things are like in the admin interface there. But yeah, if there is some kind of auditing, or snapshots, or backups, or something there, and I have rough, you know, if I know it's within a 24-hour period, maybe there's something there that would tell me what's happening. My best guess is that there's some string that is longer than expected or maybe being marked as a CHAR when it should be a VARCHAR, or maybe something that's not a non-UTF-8 encoded character, or something weird like that. So, I never know exactly what was wrong in the schema. There's some weird string thing happening that's causing the Ruby adapter to blow up. STEPHANIE: That also feels so unsatisfying [laughs] for you. I could imagine. JOËL: Yeah, there's no, like, clean resolution, right? It's a, well, the bug is gone for now. We're trying to make it less likely for it to pop up again in the future. I'm trying to leave some documentation for the next person who's going to come along, and I'm moving forward, fingers crossed. Is that something you've ever had to do on one of your projects? STEPHANIE: Given up? Yes. [laughter] I think I have definitely had to learn how to timebox debugging and have some action items for when I just can't figure it out. And, you know, like we mentioned, leaving some documentation for the next person to pick up, adding some additional logging so that maybe we can get more clues next time. But, you know, realizing that I do have to move on and that's the best that I can do is really challenging. JOËL: So, you used two words here to describe the situation: one was giving up, and the other one was timebox. I think I really like the idea of describing this as timeboxing. Giving up feels kind of like, defeatist. You know, there's so many things that we can do with our time, and we really have to be strategic with how we prioritize. So, I like the idea of describing this as a timeboxing situation. STEPHANIE: Yeah, I agree. Maybe I should celebrate every time that I successfully timebox something [laughs] according to how I planned to. [laughs] JOËL: There's always room to extend the timebox, right? STEPHANIE: [laughs] It's funny you bring up a debugging mystery because I have one of my own to share today. And I do have to say that it ended up being resolved, [chuckles] so it was a win in my book. But I will call this the case of the leaky stub. JOËL: That sounds slightly scary. STEPHANIE: It really was. The premise of what we were trying to figure out here was that we were having some flaky tests that were failing with a runtime error, so that was already kind of interesting. But it was quickly determined it was flaky because of the tests running in a certain order, so-- JOËL: Classic. STEPHANIE: Right. So, I knew something was happening, and any tests that came after it were running into this error. And I was taking a look, and I figured out how to recreate it. And we even isolated to the test itself that was running before everything else, that would then cause some problems. And so, looking into this test, I saw that it was stubbing the find method on an ActiveRecord model. JOËL: Interesting. STEPHANIE: Yeah. And the stubbed value that we were choosing to return ended up being referenced in the tests that followed. So, that was really strange to me because it went against everything I understood about how RSpec cleans up stubs between tests, right? JOËL: Yeah, that is really strange. STEPHANIE: Yeah, and I knew that it was referencing the stub value because we had set a really custom, like, ID value to it. So, when I was seeing this exact ID value showing up in a test that seemed totally unrelated, that was kind of a clue that there was some leakage happening. JOËL: So, what did you do next? STEPHANIE: The next discovery was that the error was actually raised in the factory setup for the failing tests and not even getting to running the examples at all. So, that was really strange. And digging into the factories was also its own adventure because there was a lot of complexity in the factories. A lot of them used hooks as well that then called some application code. And it was a wild goose chase. But ultimately, I realized that in the factory setup, we were calling some application code for that model where we had stubbed the find, and it had used the find method to memoize a class instance variable. JOËL: Oh no. I can see where this is going. STEPHANIE: Yeah. So, at some point, our model.find() returned our, you know, stub value that we had wanted in the previous test. And it got cached and just continued to leak into everything else that eventually would try to call that memoized method when it really should have tried to do that look-up for a separate record. JOËL: And class instance variables will persist between tests as long as they're on the same thread, right? STEPHANIE: Yeah, as far as I understand it. JOËL: That sounds like a really frustrating journey. And then that moment when you see the class instance variable, and you're like, oh no, I can't believe this is happening. STEPHANIE: Right? It was a real recipe for disaster, I think, where we had some, you know, really complicated factories. We had some sneaky caching issues, and this, you know, totally seemingly random runtime error that was being raised. And it was a real wild goose chase because there was not a lot of directness in going down the debugging path. I feel like I went around all over the codebase to get to the root of it. And, in the end, you know, we were trying to come up with some takeaways. And what was unfortunate was that you know, like, normally, stubbing find can be okay if you are, you know, really wanting to make sure that you are returning your mocked value that you may have, like, stubbed some other stuff on in your test. But because of all this, we were like, well, should we just not stub find on this really particular model? And that didn't seem particularly sustainable to make as a takeaway for other developers who want to avoid this problem. So, in the end, I think we scoped the stub to be a little more specific with the arguments that we wanted to target. And that was the way that we went forward with the particular flaky test at hand. JOËL: It sounds like the root cause of the problem was not so much the stub as it was the fact that this value is getting cached at the class level. Is that right? STEPHANIE: Yes and no. It seems like a real pain for running the tests. But I'm assuming that it was done for a good reason in production, maybe, maybe not. To be fair, I think we didn't need to cache it at all because it's calling a find, which is, you know, should be pretty quick and doesn't need to be cached. But who knows? It's hard to tell. It was really old code. And I think we were feeling also a little nervous to adjust something that we weren't sure what the impact would be. JOËL: I'm always really skeptical of caching. Caching has its place. But I think a lot of developers are a little too happy to introduce one, especially doing it preemptively that, oh well, we might need a cache here, so why not? Let's add that. Or even sometimes, just as a blind solution to any kind of slowness, oh, the site is slow; let's throw a cache here and hope for the best. And the, like, bedrock, like, rule zero of any kind of performance tuning is you've got to measure before and after and make sure that the change that you introduce actually makes things better. And then, also, is it better enough speed-wise that you're willing to pay any kind of costs associated to maintaining the code now that it's more complex? And a lot of caches can have some higher carrying costs. STEPHANIE: Yeah, that's a great point. This debugging mystery an example of one of them. JOËL: How long did it take you to figure out the solution here? STEPHANIE: So, like you, I actually was on a bit of the incorrect path for a little while. And it was only because this issue affected a different flaky test that someone else was investigating that they were able to connect the dots and be like, I think these, you know, two issues are related. And they were the ones who ultimately were able to point us out to the offending test if you will. So, you know, it took me a few days. And I imagine it took the other developer a few days. So, our combined effort was, like, over a week. JOËL: Yep. So, for all our listeners out there, you just heard that Stephanie and I [laughs] both went on multi-day debugging journeys. That happens to everyone. Just because we've been doing this job for years doesn't mean that every bug is, like, a thing that we figure out immediately. So, separately from this bug that I've been working on, a big issue that's been front of mind for me on this project has been the classic build versus buy decision. Because we're integrating with a third-party system, we have to look at either building our own integration or trying to use some off-the-shelf components. And there's a few different levels of this. There are some parts where you can actually, like, literally buy an integration and think through some of the decisions there. And then there's some situations where maybe there's an open-source component that we can use. And there's always trade-offs with both the commercial and the open-source situation. And we have to decide, are we willing to use this, or do we want to build our own? And those have been some really interesting discussions to have. STEPHANIE: Yeah. I think you actually expanded this decision-making problem into a build versus buy versus open source because they are kind of, you know, really different solutions with different outcomes in terms of, you know, maintenance and dependencies, right? And that all have, like, a little bit of a different way to engage with them. JOËL: Interesting. I think I tend to think of the buy category, including both like commercial off-the-shelf software and also open-source off-the-shelf software, things that we wouldn't build custom for ourselves but that are third-party components that we can pull in. STEPHANIE: Yeah, that's interesting because I had a bit of a different mental model because, in my head, when you're buying a commercial solution, you, you know, are maybe losing out on some opportunities for customization or even, like, forking it on your own. So, with an open-source solution, there could be an aspect of making it work for you. Whereas for a commercial solution, you really become dependent on that other company and whether they are willing to cater [laughs] to your needs or not. JOËL: That's fair. For something that's closed-source where you don't actually have access to the code, say it's more of a software as a service situation, then, yeah, you're kind of locked in and hoping that they can provide the needs that you have. On the flip side, you are generally paying for some level of support. The quality of that varies sometimes from one vendor to another. But if something goes wrong, usually, there's someone you can email, someone you can call, and they will tell you how to fix the problem, or they will fix it on their end. STEPHANIE: For the purposes of this conversation, should we talk about the differences, you know, building yourself or leaning on an existing built-out solution for you? JOËL: The project I'm working on is integrating with a Snowflake data warehouse, which is an external place that stores data accessible through something SQL-like. And one of the things that's attractive about this is that you can pull in data from a variety of different sources, transform it, and have it all stored in a kind of standardized structure that you can then integrate with. So, for pulling data in, you can build your own sort of ingestion pipeline, if you want, with code, and their APIs, and things. But there are also third-party vendors that will give you kind of off-the-shelf components that you can use for a lot of popular other data sources that you might want to pull. So, you're saying; I want to pull from this external service. They've probably got a pre-built connector for it. They can also do things like pull from an arbitrary Postgres database on some other server if that's something you have access to. It becomes really attractive because all you need to do is create an account on this website, plug in a few, like, API keys and URLs. And, all of a sudden, data is just flowing from one third-party system into your Snowflake data Warehouse, and it all just kind of works. And you don't have to bother with APIs, or ODBC, or any of that kind of stuff. STEPHANIE: Got it. Yeah, that does sound convenient. As you were talking about this, I was thinking about how if I were in the position of trying to decide how to make that integration happen, the idea of building it would seem kind of scary, especially if it's something that I don't have a lot of expertise in. JOËL: Yeah, so this was really interesting. In the beginning of the project, I looked into a little bit of what goes into building these, and it's fairly simple in terms of the architecture. You just need something that writes data files to typically something like an S3 bucket. And then you can point Snowflake to periodically pull from that bucket, and you write an import script to, you know, parse the columns and write them to the right tables in the structure that you want inside Snowflake. Where things get tricky is the actual integration on the other end. So, you have some sort of third-party service. And now, how do you sort of, on a timer maybe, pull data from that? And if there are data changes that you're synchronizing, is it just all append-only data? Or are you allowing the third-party service to say, "Hey, I deleted this record, and you should reflect that in Snowflake?" Or maybe dealing with an update. So, all of these things you have to think about, as well as synchronization. What you end up having to do is you probably boot up some kind of small service and, you know, maybe this is a small Ruby app that you have on Heroku, maybe this is, like, an AWS Lambda kind of thing. And you probably end up running this every so many seconds or so many minutes, do some work, potentially write some files to S3. And there's a lot of edge cases you have to think about to do it properly. And so, not having to think about all of those edge cases becomes really enticing when you're looking to potentially pay a third party to do this for you. STEPHANIE: Yeah, when you used the words new service, I bristled a little bit [laughs] because I've definitely seen this happen maybe on a bit of a bigger scale for a tool or solution for some need, right? Where some team is formed, or maybe we kind of add some more responsibilities to an existing team to spin up a new service with a new repo with its own pipeline, and it becomes yet another thing to maintain. And I have definitely seen issues with the longevity of that kind of approach. JOËL: The idea of maintaining a fleet of little services for each of our integrations seemed very unappealing to me, especially given that setting something like this up using the commercial approach probably takes 30 minutes per third-party service. There's no way I'm standing up an app and doing this whole querying every so many minutes, and getting data, and transforming it, and writing it to S3, and addressing all the edge cases in 30 minutes. And it's building something that's robust. And, you know, maybe if I want to go, like, really low tech, there's something fun I could do with, like, a Zapier hook and just, like, duct tape a few services together and make this, like, a no-code solution. I still don't know that it would have the robustness of the vendor. And I don't think that I could do it in the same amount of time. STEPHANIE: Yeah. I like the keyword robustness here because, at first, you were saying, like, you know, this looked relatively small in scope, right? The code that you had to write. But introducing all of the variables of things that could go wrong [laughs] beyond the custom part that you actually care about seemed quite cumbersome. JOËL: I think there's also, at this point, a lot of really interesting prioritization questions. There are money questions, but there are also time questions you have to think about. So, how much dev time do we want to devote upfront to building out these integrations? And if you're trying to move fast and get a proof of concept out, or even get, like, an MVP out in front of customers, it might be worth paying more money upfront to a third-party vendor because it allows you to ship something this week rather than next month. STEPHANIE: Yeah. The "How soon do you need it?" is a very good question to ask. Another one that I have learned to include in my arsenal of, you know, evaluating this kind of stuff comes from a thoughtbot blog by Josh Clayton, where he, you know, talks about the build versus buy problem. And his takeaway is that you should buy when your business is not dependent on it. JOËL: When it's not part of, like, the core, like, value-add that your business is doing. Why spend developer time on something that's not, like, the core thing that your product is when you can pay someone else to do it for you? And like we said earlier, a lot of that time ends up being sunk into edge cases and robustness and things like that to the point where now you have to build an expertise in a, like, secondary thing that your business doesn't really care about. STEPHANIE: Yeah, absolutely. I think this is also perhaps where very clear business goals or a vision would come in handy as well. Because if you're considering building something that doesn't quite support that vision, then it will likely end up continuing to be deprioritized over the long term until it becomes this thing that no one is accountable for maintaining and caring for. And just causes a lot of, honestly, morale issues is what I've seen when some service that was spun up to try to solve a particular problem is kind of on its last legs and has been really neglected, and no one wants to work on it. But it ends up causing issues for the rest of the development team. But then they're also really focused on initiatives that actually do provide the business value. That is a really hard balancing act that I've seen teams struggle with. JOËL: Earlier this year, we were talking about the book Sustainable Rails. And it really hammers home the idea of a carrying cost for the code, and I think that's exactly what we're talking about here. And that carrying cost can be time and money. But I like that you also mentioned the morale effects. You know, that's a carrying cost that just sort of depresses the productivity of your team when morale is low. STEPHANIE: Yeah, absolutely. I'm curious if we could discuss some of the carrying costs of buying a solution and where you've seen that become tricky. JOËL: The first thing to look at is the literal cost, the money aspect of things. And I think it's a really interesting situation for the business models for these types of Snowflake connectors because they typically charge by the amount of data that you're transmitting, so per row of data that you're transmitting. And so, that cost will fluctuate depending on whether the third-party service you're integrating with is, like, really chatty or not. When you contrast that to building, building typically has a relatively fixed cost. It's a big upfront cost, and then there's some maintenance cost to go with it. So, if I'm building some kind of integration for, let's say, Shopify, then there's the cost I need to build up front to integrate that. And if that takes me, I don't know, a week or two weeks, or however long it is, you know, that's a pretty big chunk of time. And my time is money. And so, you can actually do the math and say, "Well, if we know that we're getting so many rows per day at this rate from the commercial vendor, how many weeks do we have to pay for the commercial one before we break even and it becomes more expensive than building it upfront, just in terms of my time?" And sometimes you do that math, and you're like, wow, you know, we could be going on this commercial thing for, like, two years before we break even. In that case, from a purely financial point of view, it's probably worth paying for that connector. And so, now it becomes really interesting. You say, okay, well, which are the connectors that we have that are low volume, and which are the ones that are high volume? Because each of them is going to have a different break-even point. The ones that you break even after, you know, three or four weeks might be the ones that become more interesting to have a conversation about building. Whereas some of the others, it's clearly not worth our time to build it ourselves. STEPHANIE: The way you described this problem was really interesting to me because it almost sounds like you found the solution somewhere in the middle, potentially, where, you know, you may try building the ones that are highest priority, and you end up learning a lot from that experience, right? That could make it easier or at least, like, set you up to consider doing that moving forward in the future if you find, like, that is what is valuable. But it's interesting to me that you kind of have the best of both worlds of, like, getting the commercial solutions now for the things that are lower value and then doing what you can to get the most out of building a solution. JOËL: Yeah. So, my final recommendation ended up being, let's go all commercial for now. And then, once we've built out something, and because speed is also an issue here, once we've built out something and it's out with customers, and we're starting to see value from this, then we can start looking at how much are we paying per week for each of these connectors? And is it worth maybe going back and building our own for some of these higher-volume connectors? But starting with the commercial one for everything. STEPHANIE: Yeah, I actually think that's generally a pretty good path forward because then you are also learning about how you use the commercial solution and, you know, which features of it are critical so that if you do eventually find yourselves, like, maybe considering a shift to building in-house, like, you could start with a more clear MVP, right? Because you know how your team is using an existing product and can focus on the parts that your business are dependent on. JOËL: Yeah, it's that classic iterative development style. I think here it's also kind of inspired by a strategy I typically use for performance, which is make it work before you try to make it fast. And, actually, make it work, then profile, then measure, find the hotspots, and then focus on making those things fast. So, in this case, instead of speed, we're talking about money. So, it's make it work, then profile, find the parts that are expensive, and make the trade-off of, like, okay, is it worth investing into making that part less expensive in terms of resources? STEPHANIE: I like that as a framework a lot. JOËL: A lot of what we do as programmers is optimization, right? And sometimes, we're optimizing for execution time. Sometimes we're optimizing for memory cost, and sometimes we're optimizing for dollars. STEPHANIE: Yeah, that's really interesting because, with the buy solution, you know very clearly, like, how much the thing will cost. Whereas I've definitely seen teams go down the building route, and it always takes longer than expected [laughs], and that is money, right? In terms of the developer's time, for sure. JOËL: Yeah, definitely, like, add some kind of multiplier when you're budgeting out that build alternative because, quite likely, there are some edge cases that you haven't thought about that the commercial partner has, and you will have to spend more time on that than you expected. STEPHANIE: Yeah, in addition to whatever opportunity cost of not working on something that is driving revenue for the business right now. JOËL: Exactly. STEPHANIE: So, the direction of this conversation ended up going kind of towards, like, what is best for the team at, like, a product and company level. But I think that we make these decisions a lot more frequently, even when it comes to whether we pull in a gem or, you know, use an open-source tool or not. And I would be really interested in discussing more of that in another episode. JOËL: Yeah. That gets into some controversial takes, right? It's the evergreen topic of: do we build it ourselves, or do we pull in some kind of third-party package? STEPHANIE: Something for the future to look forward to. On that note, shall we wrap up? JOËL: Let's wrap up. STEPHANIE: Show notes for this episode can be found at bikeshed.fm. JOËL: This show has been produced and edited by Mandy Moore. STEPHANIE: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review in iTunes. It really helps other folks find the show. JOËL: If you have any feedback for this or any of our other episodes, you can reach us @_bikeshed, or you can reach me @joelquen on Twitter. STEPHANIE: Or reach both of us at hosts@bikeshed.fm via email. JOËL: Thanks so much for listening to The Bike Shed, and we'll see you next week. ALL: Byeeeeeeeee!!!!! ANNOUNCER: This podcast is brought to you by thoughtbot, your expert strategy, design, development, and product management partner. We bring digital products from idea to success and teach you how because we care. Learn more at thoughtbot.com.

pr giving microsoft mvp id shopify api rb warehouses snowflakes apis rails char s3 zapier sql urls utf mandy moore heroku postgres caching aws lambda quenneville activerecord rspec odbc stephanie yeah josh clayton stephanie it stephanie oh stephanie for stephanie right

Active Record Query Interface

technology web active secure programming improved storage docs token rails ruby on rails callback kimura activerecord

Play Episode Listen Later Aug 8, 2023 40:02

Aji & guest host Mercedes Bernard talk ORMs & SQL, left outer joins & includes, and strict & eager loading.Reading for this episode: Active Record Query InterfaceThe Bike Shed episode 358: The Class MethodReading for next episode: Active Model BasicsFind Mercedes online at mercedesbernard.com, on mastodon: mercedescodes@mastodon.world, or LinkedIn

reading interface sql query aji orms activerecord

010: Improved Active Storage docs, a new has_secure_token callback with Dave Kimura

The Rails Changelog

Play Episode Listen Later Aug 3, 2023 18:26

Dave Kimura of Drifting Ruby fame joins me to discuss the importance of Ruby on Rails documents. We talk specifically about improved documentation around Active Storage and Active Record encryption, then dive into the new callback for when has_secure_token is triggered.Improved Active Storage docs (This PR was updated)Improved Active Record Encryption docsEagerly cast serialized query attributesA callback for has_secure_token

Database constraints with Andrew Atkinson

Slow & Steady

Play Episode Listen Later Jul 27, 2023 54:20

Benedikt talks to Andrew Atkinson about different types of database constraints and his upcoming book. Read the tweet that triggered this interview Learn more about High Performance PostgreSQL for Rails Check out Andrew's blog Follow Andrew on Twitter Andrew Atkinson is a staff software engineer at Fountain who is very passionate about Ruby on Rails and PostgreSQL (just like Benedikt!). He's so passionate about them that he's actually writing a book called “High Performance PostgreSQL for Rails” that's coming out very soon.Benedikt and Andrew also talk about dealing with constraints and ActiveRecord, Rails validations, and more.

fountain databases rails constraints benedikt ruby on rails postgresql activerecord andrew atkinson

Active Record Associations

reading switzerland bern associations aji activerecord

Play Episode Listen Later Jul 25, 2023 25:31

Aji welcomes guest host and fellow thoughtbotter Dimiter Petrov. Their conversation covers association scope, the thoughtful layout of this section of the guides, polymorphic associations, and association extensions.Reading for this episode: Active Record AssociationsHelvetic Ruby: Ruby conference in Bern, Switzerland, November 24, 2023Reading for next episode: Active Record Query Interface

393: Is REST the Best? APIs and Domain Modeling

Play Episode Listen Later Jul 18, 2023 33:49

It's updates on the work front today! Stephanie was tasked with removing a six-year-old feature flag from a codebase. Joël's been doing a lot of small database migrations. A listener question sparked today's main discussion on gerunds' interesting relationship to data modeling. Episode 386: Value Objects Revisited: The Tally Edition (https://www.bikeshed.fm/386) RailsConf 2017: In Relentless Pursuit of REST by Derek Prior (https://www.youtube.com/watch?v=HctYHe-YjnE) REST Turns Humans Into Database Clients (https://chrislwhite.com/rest-contortion/) Parse, don't validate (https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/) Wikipedia Getting to Philosophy (https://en.wikipedia.org/wiki/Wikipedia:Getting_to_Philosophy) Transcript: JOËL: Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Joël Quenneville. STEPHANIE: And I'm Stephanie Minn. And together, we're here to share a bit of what we've learned along the way. JOËL: So, Stephanie, what's new in your world? STEPHANIE: So, this week, I've been tasked with something that I've been finding very fun, which is removing a six-year-old feature flag from the codebase that is still very much in use in the sense that it is actually a mechanism for providing customers access to a feature that had been originally launched as a beta. And that was why the feature flag was introduced. But in the years since, you know, the business has shifted to a model where you have to pay for those features. And some customers are still hanging on to this beta feature flag that lets them get the features for free. So one of the ways that we're trying to convert those people to be paying for the feature is to, you know, gradually remove the feature flag and maybe, you know, give them a heads up that this is happening. I'm also getting to improve the codebase with this change as well because it has really been propagating [laughs] in there. There wasn't necessarily a single, I guess, entry point for determining whether customers should get access to this feature through the flag or not. So it ended up being repeated in a bunch of different places because the feature set has grown. And so, now we have to do this check for the flag in several places, like, different pages of the application. And it's been really interesting to see just how this kind of stuff can grow and mutate over several years. JOËL: So, if I understand correctly, there's kind of two overlapping conditions now around this feature. So you have access to it if you've either paid for the feature or if you were a beta tester. STEPHANIE: Yeah, exactly. And the interesting thought that I had about this was it actually sounds a lot like the strangler fig pattern, which we've talked about before, where we've now introduced the new source of data that we want to be using moving forward. But we still have this, you know, old limb or branch hanging on that hasn't quite been removed or pruned off [chuckles] yet. So that's what I'm doing now. And it's nice in the sense that I can trust that we are already sending the correct data that we want to be consuming, and it's just the cleanup part. So, in some ways, we had been in that half-step for several years, and they're now getting to the point where we can finally remove it. JOËL: I think in kind of true strangler fig pattern, you would probably move all of your users off of that feature flag so that the people that have it active are zero, at which point it is effectively dead code, and then you can remove it. STEPHANIE: Yeah, that's a great point. And we had considered doing that first, but the thing that we had kind of come away with was that removing all of those customers from that feature flag would probably require a script or, you know, updating the production data. And that seemed a bit riskier actually to us because it wasn't as reversible as a code change. JOËL: I think you bring up a really interesting point, which is that production data changes, in general, are just scarier than code changes. At least for me, it feels like it's fairly easy generally to revert a code change. Whereas if I've messed up the production database, [laughs] that's going to be unpleasant few days. STEPHANIE: What's interesting is that this feature flag is not really supported by a nice user interface for managing it. And so, we inevitably had to do a more developer-focused solution to remove these customers from being able to access this feature. And so, the two options, you know, that we had available were to do it through data, like I mentioned, or do it through that code change. And again, I think we evaluated both options. But what's kind of nice about doing it with the code change is that when we eventually get to delete those feature flag records, it will be really nice and easy. JOËL: That's really exciting. One thing that's different about kind of more mature projects is that we often get to do some kind of change management, unlike a greenfield app where you just get to, oh, let's introduce this new thing, cool. Oftentimes, on a more mature project, before you introduce the new thing, you have to figure out, like, what is the migration path towards that? Is that a kind of work that you enjoy? STEPHANIE: I think this was definitely an exercise in thinking about how to break this down into steps. So, yeah, that change management process you mentioned, I, like, did find a lot of satisfaction in trying to break it up, you know, especially because I was also thinking that you know, maybe I am not able to see the complete, like, cleanup and removal, and, like, where can someone pick up after me? In some ways, I feel like I was kind of stepping into that migration, you know, six years [laughs] in the making from beta to the paid product. But I think I will feel really satisfied if I'm able to see this thing through and get to celebrate the success of saying, hey, like, I removed...at this point, it's a few hundred lines of code. [laughs] And also, you know, with the added business value of encouraging more customers to pay for the product. But I think I also I'm maybe figuring out how to accept like, okay, like, how could I, like, step away from this in the middle and be able to feel good that I've left it in a place that someone else could see through? JOËL: So you mentioned you're taking this over from somebody else, and this has been kind of six years in the making. I'm curious, is the person who introduced this feature flag six years ago are they even still at the company? STEPHANIE: No, they are not, which I think is pretty typical, you know, it's, like, really common for someone who had all that context about how it came to be. In fact, I actually didn't even realize that the feature flag was the original beta version of the product because that's not what it's called. [laughs] And it was when I was first onboarding onto this project, and I was like, "Hey, like, what is this? Like, why is this still here?" Knowing that the canonical, you know, version that customers were using was the paid version. And the team was like, "Oh, yeah, like, that's this whole thing that we've been meaning to remove for a long time." So it's really interesting to see the lifecycle, like, as to some of this code a little bit. And sometimes, it can be really frustrating, but this has felt a little more like an archaeology dig a little bit. JOËL: That sounds like a really interesting project to be on. STEPHANIE: Yeah. What about you, Joël, what's new in your world? JOËL: So, on my project, I've been having to do a lot of small database migrations. So I've got a bunch of these little features to do that all involve doing database migrations. They're not building on each other. So I'm just doing them all, like, in different feature branches, and pushing them all up to GitHub to get reviewed, kind of working on them in parallel. And the problem that happens is that when you switch from one branch where you've run a migration to another and then run migrations again, some local database state persists between the branch switch, which means that when you run the migrations, then this app uses a structure.sql. And the structure.sql has a bunch of extra junk from other branches you've been on that you don't want as part of your diff. And beyond, like, two or three branches, this becomes an absolute mess. STEPHANIE: Oh, I have been there. [laughs] It's always really frustrating when I switch branches and then try to do my development and then realize that I have had my leftover database changes. And then having to go back and then always forgetting what order of operations to do to reverse the migration and then having to re-migrate. I know that pain very well. JOËL: Something I've been doing for this project is when I switch branches, making sure that my structure SQL is checked out to the latest version from the main branch. So I have a clean structure SQL then I drop my local database, recreate an empty one, and run a rake db:schema:load. And that will load that structure file as it is on the main branch into the database schema. That does not have any of the migrations on this branch run, so, at that point, I can run a rake db:migrate. And I will get exactly what's on main plus what gets generated on this branch and nothing else. And so, that's been a way that I've been able to kind of switch between branches and run database operations without getting any cross-contamination. STEPHANIE: Cross-contamination. I like that term. Have you automated this at all, or are you doing this manually? JOËL: Entirely manually. I could probably script some of this. Right now...so it's three steps, right? Drop, create, schema load. I just have them in one command because you can chain Unix commands with a double ampersand. So that's what I'm doing right now. I want to say there's a db:reset task, but I think that it uses migrate rather than schema load. And I don't want to actually run migrations. STEPHANIE: Yeah, that would take longer. That's funny. I do love the up arrow key [laughs] in your terminal for, you know, going back to the thing you're running over and over again. I also appreciate the couple extra seconds that you're spending in waiting for your database to recreate. Like, you're paying that cost upfront rather than down the line when you are in the middle of doing [laughs] what you're trying to do and realize, oh no, my database is not in the state that I want it to be for this branch. JOËL: Or I'm dealing with some awful git conflict when trying to merge some of these branches. Or, you know, somebody comments on my PR and says, "Why are you touching the orders table? This change has nothing to do with orders." I'm like, "Oh, sorry, that actually came out of a different thing that I did." So, yep, keeping those diffs small. STEPHANIE: Nice. Well, I'm glad that you found a way to manage it. JOËL: So you mentioned the up arrow key and how that's really nice in the terminal. Something that I've been relying on a lot recently is reverse history search, CTRL+R in the terminal. That allows me to, instead of, like, going one by one in order of the history, filter for something that matches the thing that I've written. So, in this case, I'll hit CTRL+R, type, you know, Rails DB or whatever, then immediately it shows me, oh, did you want this long command? Hit enter, and I'm done. Even if I've done, you know, 20 git commands between then and the last time I ran it. STEPHANIE: Yeah, that's a great tip. So, a few weeks ago, we received a listener question from John, and he was responding to an episode where I'd asked about what the grammatical term is for verbs that are also nouns. He told us about the phrase, a verbal noun, for which there's a specific term called gerund, which is basically, in English, the words ending in ING. So, the gerund version of bike would be biking. And he pointed out a really interesting relationship that gerunds have to data modeling, where you can use a gerund to model something that you might describe as a verb, especially as a user interaction, but can be turned into a noun to form a resource that you might want to introduce CRUD operations for in your application. So one example that he was telling us about is the idea of maybe confirming a reservation. And, you know, we think of that as an action, but there is also a noun form of that, which is a confirmation. And so, confirmation could be a new resource, right? It could even be backed at the database level. And now you have a simpler way of representing the idea of confirming a reservation that is more about the confirmation as the resource itself rather than some kind of append them to a reservation itself. JOËL: That's really cool. We get to have a crossover between grammar terms and programming, and being able to connect those two is always a fun day for me. STEPHANIE: Yeah, I actually find it quite difficult, I think, to come up with noun forms of verbs on my own. Like, I just don't really think about resources that way. I'm so used to thinking about them in a more tangible way, I suppose. And it's really kind of cool that, you know, in the English language, we have turned these abstract ideas, these actions into, like, an object form. JOËL: And this is particularly useful when we're trying to design RESTful either APIs or even just resources for a Rails app that's server-rendered so that instead of trying to create all these, like, extra actions on our controller that are verbs, we might decide to instead create new resources in the system, new nouns that people can do the standard 7 to. STEPHANIE: Yes. I like that better than introducing custom controller actions or routes that deviate from RESTful conventions because, you know, I probably have seen a slash confirm reservation [laughs] URL. And, you know, this is, I think, an interesting way of avoiding having too many of those deviating endpoints. JOËL: Yeah, I found that while Rails does have support for those, just all the built-in things play much more nicely if you're restricting yourself to the classic seven. And I think, in general, it's easier to model and think about things in a Rails app when you have a lot of noun resources rather than one giant controller with a bunch of kind of verb actions that you can do to it. In the more formal jargon, I think we might refer to that as RESTful style versus RPC style, a Remote Procedure Call. STEPHANIE: Could you tell me more about Remote Procedure Calls and what that means? JOËL: The general idea is that it's almost like doing a method call on an object somewhere. And so, you would say, hey, I've got an account, and I want to call the confirm method on it because I know that maybe underlying this is an ActiveRecord account model. And the API or the web UI is just a really thin layer over those objects. And so, more or less, whatever your methods on your object are, can be accessed through the API. So the two kind of mirror each other. STEPHANIE: Got it. That's interesting because I can see how someone might want to do that, especially if, you know, the account is the domain object they're using at the, you know, persistence layer, and maybe they're not quite able to see an abstraction for something else. And so, they kind of want to try to fit that into their API design. JOËL: So I have a perhaps controversial opinion, which is that the resources in your Rails application, so your controllers, shouldn't map one-to-one with your database tables, your models. STEPHANIE: So, are you saying that you are more likely to have more abstractions or various resources than what you might have at the database level? JOËL: Well, you know what? Maybe more, but I would say, in general, different. And I think because both layers, the controller layer, and the model layer, are playing with very different sets of constraints. So when I'm designing database tables, I'm thinking in terms of normalization. And so, maybe I would take one big concept and split it up into smaller concepts, smaller tables because I need this data to be normalized so that there's no ambiguity when I'm making queries. So maybe something that's one resource at the controller layer might actually be multiple tables at the database layer. But the inverse could also be true, right? You might have, in the example that John gave, you know, an account that has a single table in the database with just a Boolean field confirmed yes or no. And maybe there's just a generic account resource. But then, separately, there's also a confirmation resource. And so, now we've got more resources at the controller layer than at the database layer. So I think it can go either way, but they're just not tightly coupled to each other. STEPHANIE: Yeah, that makes sense. I think another way that I've seen this manifest is when, like you said, like, maybe multiple database tables need to be updated by, you know, a request to this endpoint. And now we get into [chuckles] what some people may call services or that territory of basically something. And what's interesting is that a lot of the service classes are named as verbs, right? So order, creator. And, like, whatever order of operations that needs to happen on multiple database objects that happens as a result of a user placing an order. But the idea that those are frequently named as verbs was kind of interesting to me and a bit of a connection to our new gerund tip. JOËL: That's really interesting. I had not made that connection before. Because I think my first instinct would be to avoid a service object there and instead use something closer to a form object that takes the same idea and represents it as a noun, potentially with the same name as the resource. So maybe leaning really heavily into that idea of the verbal noun, not just in describing the controller or the route but then also maybe the object backing it, even if it's not connecting directly to a database table. STEPHANIE: Interesting. So, in this case, would the form object be mapped closer to your controller resource? JOËL: Potentially, yes. So maybe I do have some kind of, like, object that represents a confirmation and makes it nicer to render the confirmation form on the edit page or the new page. In this case, you know, it's probably just one checkbox, so maybe it's not worth creating an object. But if there were multiple fields, then yes, maybe it's nice to create an in-memory object that has the same name as the resource. Similar maybe for a resource that represents multiple underlying database tables. It can be nice to have kind of one object that represents all of them, almost like a facade, I guess. STEPHANIE: Yeah, that's really interesting. I like that idea of a facade, or it's, like, something at a higher level representing hopefully, like, some kind of meaning of all of these database objects together. JOËL: I want to give a shout-out to talk from a former thoughtboter, Derek Prior—actually, former Bike Shed host—from RailsConf 2017 called In Relentless Pursuit of REST, where he digs into a lot of these concepts, particularly how to model resources in your Rails app that don't necessarily map one to one with a database table, and why that can be a good thing. Have you seen that talk? STEPHANIE: I haven't, but I love the title of it. It's a great pun. It's very evocative, I think because I'm really curious about this idea of a relentless pursuit. Because I think another way to react to that could be to be done with REST entirely and maybe go with something like GraphQL. JOËL: So instead of a relentless pursuit, it's a relentless...what's the opposite of pursuing? Fleeing? STEPHANIE: Fleeing? [laughs] I like how we arrived there at the same time. Yes. So now I'm thinking of I had mentioned a little bit ago on the show we had our spicy takes Lightning Talks on our Boost Team. And a fellow thoughtboter, Chris White, he had given a talk about Why REST Is Not the Best and for -- JOËL: Also, a great title. STEPHANIE: Yes, also, a great title. JOËL: I love the rhyming there. STEPHANIE: Yeah. And his reaction to the idea of trying to conform user interactions that don't quite map to a noun or an obvious resource was to potentially introduce GraphQL, where you have one endpoint that can service really anything that you can think of, I suppose. But, in his example, he was making the argument that human interactions are not database resources, right? And maybe if you're not able to find that abstraction as a noun or object, with GraphQL, you can encapsulate those ideas as closer to actions, but in the GraphQL world, like, I think they're called mutations. But it is, I think, a whole world of, like, deciding what you want to be changed on the server side that is a little less constrained to having to come up with the right abstraction. JOËL: I feel like GraphQL kind of takes that, like, complete opposite philosophy in that instead of saying, hey, let's have, like, this decoupling between the API layer and the database, GraphQL almost says, "No, let's lean into that." And yeah, you want to traverse the graph of, like, tables under the hood? Absolutely. You get to know the tables. You get to know how they're related to each other. I guess, in theory, you could build a middle layer, and that's the graph that gets traversed rather than the graph of the tables. In practice, I think most people build it so that the API layer more or less has access directly to tables. Has that been your experience? STEPHANIE: That's really interesting that you brought that up. I haven't worked with GraphQL in a while, but I was reading up on it before we started recording because I was kind of curious about how it might play with what we're talking about now. But the idea that it's graphed based, to me, was like, oh, like, that naturally, it could look very much like, you know, an entity graph of your relational database. But the more I was reading about the GraphQL schema and different types, I realized that it could actually look quite different. And because it is a little bit closer to your UI layer, like, maybe you are building an abstraction that is more for serving that as that middle layer between your front end and your back end. JOËL: That's really interesting that you mentioned that because I feel like the sort of traditional way that APIs are built is that they are built by the back-end team. And oftentimes, they will reflect the database schema. But you kind of mentioned with GraphQL here, sometimes it's the opposite that happens. Instead of being driven kind of from the back towards the front, it might be driven from the front towards the back where the UI team is building something that says, hey, we need these objects. We need these connections. Can you expose them to us? And then they get access to them. What has been your experience when you've been working with front ends that are backed by a GraphQL API? STEPHANIE: I think I've tended to see a GraphQL API when you do have a pretty rich client-side application with a lot of user interactions that then need to, you know, go and fetch some data. And you, like, really, you know, obviously don't want a page reload, right? So it's really interesting, actually, that you pointed out that it's, like, perhaps the front end or the UI driving the API. Because, on one hand, the flexibility is really nice. And there's a lot more freedom even in maybe, like, what the product can do or how it would look. On the other hand, what I've kind of also seen is that eventually, maybe we do just want an API that we can talk to separate from, you know, any kind of UI. And, at that point, we have to go and build a separate thing [laughs] for the same data. JOËL: So we've been talking about structuring APIs and, like, boundaries and things like that. I think my personal favorite feature of GraphQL is not the graph part but the fact that it comes with a built-in schema. And that plays really nicely with some typed technologies. Particularly, I've used Elm with some of the GraphQL libraries there, and that experience is just really nice. Where it will tell you if your front-end code is not compatible with the current API schema, and it will generate some things based off the schema. So you have this really nice feedback cycle where somebody makes a change to the API, or you want to make a change to the code, and it will tell you immediately is your front end compatible with the current state of the back end? Which is a classic problem with developing front-end code. STEPHANIE: First of all, I think it's very funny that you admitted to not preferring the graph part of GraphQL as a graph enthusiast yourself. [laughs] But I think I'm in agreement with you because, like, normally, I'm looking at it in its schema format. And that makes a lot of sense to me. But what you said was really interesting because, in some ways, we're now kind of going back to the idea of maybe boundaries blurring because the types that you are creating for GraphQL are kind of then servicing both your front end and your back end. Do you think that's accurate? JOËL: Ooh. That is an important distinction. I think you can. And I want to say that in some TypeScript implementations, you do use the types on both sides. In Elm, typically, you would not unless there's something really primitive, like a string or something like that. STEPHANIE: Okay, how does that work? JOËL: So you have some conversion layer that happens. STEPHANIE: Got it. JOËL: Honestly, I think that's my preference, and not just at the front end versus API layer but kind of all throughout. So the shape of an object in the database should not be the same shape as the object in the business logic that runs on the back end, which should not be the same shape as the object in transport, so JSON or whatever, which is also not the same shape as the object in your front-end code. Those might be similar, but each of these layers has different responsibilities, different things it's trying to optimize for. Your code should be built, in my opinion, in a way that allows all four of those layers to diverge in their interpretation of not only what maybe common entities are, so maybe a user looks slightly different at each of these layers, but maybe even what the entities are to start with. And that maybe in the database what, we don't have a full user, we've got a profile and an account, and those get merged somehow. And eventually, when it gets to the front end, all we care about is the concept of a user because that's what we need in that context. STEPHANIE: Yeah, that's really interesting because now it almost sounds like separate systems, which they kind of are, and then finding a way to make them work also as one bigger [laughs] system. I would love to ask, though, what that conversion looks like to you. Or, like, how have you implemented that? Or, like, what kind of pattern would you use for that? JOËL: So I'm going to give a shout-out to the article that I always give a shout-out to: Parse, Don't Validate. In general, yeah, you do a transformation, and potentially it can fail. Let's say I'm pulling data from a GraphQL API into an Elm app. Elm has some built-in libraries for doing those transformations and will tell you at compile time if you're incorrectly transforming the data that comes from the shape that we expect from the schema. But just because the schema comes in as, like, a flat object with certain fields or maybe it's a deeply nested chain of objects in GraphQL, it doesn't mean that it has to be that way in your Elm app. So that transformation step, you get to sort of make it whatever you want. So my general approach is, at each layer, forget what other people are sending you and just design the entities that you would like to. I've heard the term wish-driven development, which I really like. So just, you know, if you could have, like, to make your life easy, what would the entities look like? And then kind of work backwards from there to make that sort of perfect world a reality for you and make it play nicely with other systems. And, to me, that's true at every layer of the application. STEPHANIE: Interesting. So I'm also imagining that the transformation kind of has to happen both ways, right? Like, the server needs a way to transform data from the front end or some, you know, whatever, third party. But that's also true of the front end because what you're kind of saying is that these will be different. [laughs] JOËL: Right. And, in many ways, it has to be because JSON is a very limited format. But some of the fancier things that you might have access to either on the back end or on the front end might be challenging to represent natively in JSON. And a classic one would be what Elm calls a custom type. You know, they're also called tagged unions, discriminated unions, algebraic data types. These things go by a bajillion names, and it's confusing. But they're really kind of awkward and hard, almost impossible to represent in straight-up JSON because JSON is a very limited kind of transportation format. So you have to almost, like, have a rehydration step on one side and a kind of packing down step on the other when you're reading or writing from a JSON API. STEPHANIE: Have you ever heard of or played that Wikipedia game Getting to Philosophy? JOËL: I've done, I think, variations on it, the idea that you have a start and an end article, and then you have to either get through in the fewest amount of clicks, or it might be a timed thing, whoever can get to the target article first. Is that what you're referring to? STEPHANIE: Yeah. So, in this case, I'm thinking, how many clicks through Wikipedia to get to the Wiki article about philosophy? And that's how I'm thinking about how we end up getting to [laughs] talking about types and parsing, and graphs even [laughs] on the show. JOËL: It's all connected, almost as if it forms a graph of knowledge. STEPHANIE: Learning that's another common topic on the show. [laughs] I think it's great. It's a lot of interesting lenses to view, like, the same things and just digging further and further deeper into them to always, like, come away with a little more perspective. JOËL: So, in the vein of wish-driven development, if you're starting a brand-new front-end UI, what is your sort of dream approach for working with an API? STEPHANIE: Wish-driven development is very visceral to me because I often think about when I'm working with legacy code and what my wishes and dreams were for the, you know, the stack or the technology or whatever. But, at that point, I don't really have the power to change it. You know, it's like I have what I have. And that's different from being in the driver's seat of a greenfield application where you're not just wishing. You're just deciding for yourself. You get to choose. At the end of the day, though, I think, you know, you're likely starting from a simple application. And you haven't gotten to the point where you have, like, a lot of features that you have to figure out how to support and, like, complexity to manage. And, you know, you don't even know if you're going to get there. So I would probably start with REST. JOËL: So we started this episode from a very back-end perspective where we're talking about Rails, and routes, and controllers. And we kind of ended it talking from a very front-end perspective. We also contrasted kind of a more RESTful approach, versus GraphQL, versus more kind of old-school RPC-style routing. And now, I'm almost starting to wonder if there's some kind of correlation between whether someone primarily works from the back end and maybe likes, let's say, REST versus maybe somebody on the front end maybe preferring GraphQL. So I'd be happy for any of our listeners who have strong opinions preferring GraphQL, or REST, or something else; message us at hosts@bikeshed.fm and let us know. And, if you do, please let us know if you're primarily a front-end or a back-end developer because I think it would be really fun to see any connections there. STEPHANIE: Absolutely. On that note, shall we wrap up? JOËL: Let's wrap up. STEPHANIE: Show notes for this episode can be found at bikeshed.fm. JOËL: This show has been produced and edited by Mandy Moore. STEPHANIE: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review in iTunes. It really helps other folks find the show. JOËL: If you have any feedback for this or any of our other episodes, you can reach us @_bikeshed, or you can reach me @joelquen on Twitter. STEPHANIE: Or reach both of us at hosts@bikeshed.fm via email. JOËL: Thanks so much for listening to The Bike Shed, and we'll see you next week. ALL: Byeeeeeee!!!!!! ANNOUNCER: This podcast is brought to you by thoughtbot, your expert strategy, design, development, and product management partner. We bring digital products from idea to success and teach you how because we care. Learn more at thoughtbot.com.

english pr drop philosophy wikipedia similar modeling api ui github domain ing apis rails wiki fleeing validate sql elm mandy moore restful unix json typescript graphql chris white crud rpc boolean parse quenneville lightning talks graphql apis railsconf activerecord json api stephanie yeah stephanie no stephanie oh derek prior

392: Managing Changing Business Requirements

Play Episode Listen Later Jul 11, 2023 39:14

Joël has a fascinating discovery! He learned a new nuance around working with dependency graphs. Stephanie just finished playing a 100-hour video game on Nintendo Switch: a Japanese role-playing game called Octopath Traveler II. On the work front, she is struggling with a lot of churn in acceptance criteria and ideas about how features should work. How do these get documented? What happens when they change? What happens when people lose this context over time? Strangler Fig Pattern (https://shopify.engineering/refactoring-legacy-code-strangler-fig-pattern) Octopath Traveler 2 (https://octopathtraveler2.square-enix-games.com/en-us/) Empowering other departments (https://www.bikeshed.fm/388) Transcript: JOËL: You're the one who controls the pacing here. STEPHANIE: Oh, I am. Okay, great. Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Stephanie Minn. JOËL: And I'm Joël Quenneville. And together, we're here to share a bit of what we've learned along the way. STEPHANIE: So, Joël, what's new in your world? JOËL: So long-time Bike Shed listeners will know that I'm a huge fan of dependency graphs for modeling all sorts of problems and particularly when trying to figure out how to work in an iterative fashion where you can do a bunch of small chunks of work that are independent, that can be shipped one at a time without having your software be in a breaking state in all of these intermediate steps. And I recently made a really exciting discovery, or I learned a new nuance around working with dependency graphs. So the idea is that if you have a series of entities that have dependencies on each other, so maybe you're trying to build, let's say, some kind of object model or maybe a series of database tables that will reference each other, that kind of thing, if you draw a dependency graph where each bubble on your graph points to other bubbles that it depends on, that means that it can't be created without those other things already existing. Then, in order to create all of those entities for the first time, let's say they're database tables, you need to work your way from kind of the outside in. You start with any bubbles on your graph that have no arrows going out from them. That means they have no dependencies. They can be safely built on their own, and then you kind of work your way backwards up the arrows. And that's how I've sort of thought about working with dependency graphs for a long time. Recently, I've been doing some work that involves deleting entities in such a graph. So, again, let's say we're talking about database tables. What I came to realize is that deleting works in the opposite order. So, if you have a table that have other tables that depend on it, but it doesn't depend on anything, that's the first one you want to create. But it's also the last one you want to delete. So, when you're deleting, you want to start with the table that maybe has dependencies on other tables, but no other tables depend on it. It is going to be kind of like the root node of your dependency graph. So I guess the short guideline here is when you're creating, work from the bottom up or work from the leaves inward, and when you're deleting, work from the top-down or work from the root outward or roots because a graph can have multiple roots; it's not a tree. STEPHANIE: That is interesting. I'm wondering, did you have a mental model for managing deleting of dependencies prior? JOËL: No. I've always worked with creating new things. And I went into this task thinking that deleting would be just like creating and then was like, wait a minute, that doesn't work. And then, you know, a few cycles later, realized, oh, wait, deleting is the opposite of creating when you're navigating the graph. And, all of a sudden, I feel like I've got a much clearer mental model or just another way of thinking about how to work with something like this. STEPHANIE: Cool. That actually got me thinking about a case where you might have a circular dependency. Is that something you've considered yet? JOËL: Yes. So, when you have a dependency graph, and you've got a circular dependency, that's a big problem because...so, in the creating model, there is no leaf node, if you will, because they both reference each other. So that means that each of these entities cannot be created on its own, the entire cycle. And maybe you've got only two, but maybe your cycle is, you know, ten entities big. The entire cycle is going to be shipped as one massive change. So something that I often try to do is if I draw a dependency graph out and notice, wait a minute, I do have cyclical dependencies, the question then becomes, can I break that cycle to allow myself to work iteratively? Because otherwise, I know that there's a big chunk that can't be done iteratively. It just has to be done all at once. STEPHANIE: Yeah, that's really interesting because I've certainly been in that situation where I don't realize until it's too late, where I've started going down the path thinking that, you know, I could just remove this one thing, or make this one change, and then find myself suddenly, you know, coming to the realization, oh, this other thing is now going to have to change. And then, at that point, there's almost kind of like the sunken cost fallacy [laughs] a little bit where you're like, well, I'm already in it. So, why don't I keep going? But your strategy of trying to find a way to break that cyclica...that is two words combined. [laughs] I meant to say circular dependency [laughs] is the right way to avoid just having to do it all in one go. Have you had to break up a cycle like that before? JOËL: Yes. I do it on a semi-frequent basis. The fancy term here for what I'm looking for when I'm building out a dependency graph is a directed acyclic graph. That's a graph theory or a computer science term that you'll hear thrown around a lot, DAG. I often like to...when building out a series of tasks that might also form a graph because you don't just model entities in your system; you might model a series of tasks as a graph. If there's a cycle in the graph, typically, I can break that using something like the strangler fig pattern, which is a way to kind of have some intermediate steps that are non-breaking that then lead you to the refactor that you want. And I've used the strangler fig pattern for a long time, never realizing until later that, oh, what I'm actually doing is breaking cycles in my task dependency graph. STEPHANIE: Hmm. I'm curious if you have noticed how these cycles come to be because I almost imagine that they get introduced over time, where you maybe did start with a parent and then you, you know, had dependencies. But then, over time, somehow, that circular dependency gets introduced. And I'm wondering if part of figuring out how to break that cycle is determining how things were introduced, like, over time. JOËL: In my experience, this happens in a lot of different ways because I'm using dependency graphs like this to give myself a mental model for a lot of different kinds of things. So maybe I'm thinking in terms of database tables. And so those might get a circular dependency that gets added over time as the system grows. But I'm also using it sometimes to model maybe a series of tasks. So I take a large task, and I break it down into subtasks that are all connected to each other. And that doesn't tend to sort of evolve over time in the same way that a series of database tables do. So I think it's very context-dependent. But there are definitely situations where it will be like you said, something that kind of evolves over time. STEPHANIE: That makes sense. Well, I'm excited for you to get to deleting some potential code or database tables that are no longer in use. That sounds like a developer's dream [laughs] to clean up all that stuff. JOËL: It's interesting because it's...a move operation is effectively what's happening. So I'm recreating tables in another system, pointing the ActiveRecord to this new system, and then deleting the existing ones in the local database. So, in a sense, I'm kind of traveling up this dependency graph from the leaf nodes into the root and then back down from the root to the leaves as I'm creating and then deleting everything or creating in one system, and then going back and deleting in the other system. STEPHANIE: Got it. Okay, so not necessarily a net negative but, like you said, a move or just having to gradually replace to use a new system. JOËL: That's right. And we're trying not to do this as, you know, okay, we're going to take the system down and move 50 tables from one system to another. But instead, saying, like, you know, one at a time, we're going to move these things over. And it's going to be small, incremental change over the course of a couple of weeks. And they're all pretty safe to deploy, and we feel good about them. STEPHANIE: That's good. I'm glad you feel good. [laughs] We should all be able to feel good when we make changes like that. JOËL: It's going to make my Fridays just so much more low-key just, like, yeah, hit that deploy button. It's okay. So, Stephanie, what is new in your world? STEPHANIE: So this is not work-related at all. But I just finished playing a 100-hour video game on my Nintendo Switch. [laughs] I finished a Japanese role-playing game called Octopath Traveler II. And I have never really played a game like this before. I've not, you know, put in many, many hours into something that then had an end, like, a completion. So, at the end of this very long game that had a very, you know, compelling and engaging story and I was invested in all of these characters, and by the time the credits were rolling, I felt a little sad to be leaving this world that I have been in many evenings over the last couple of months. Yeah, I don't know, I'm feeling both a little sad because, you know like I said, I got really invested in this game, but now I'm also kind of glad to have some free time back in [laughs] my life because that has definitely been the primary, like, evening activity that I've been doing to relax. JOËL: It sounds like this game had a very, like, a particularly immersive world that really pulled you in. STEPHANIE: It did. It did. It has these eight, like, different characters that you follow, like, different chapters and all of their stories, and then they all kind of come together as well. And the world was huge in this game. There were so many little towns to explore. And I didn't realize I was a completionist type. But I found myself running around opening every chest, talking to every NPC, and making sure that I, you know, collected all of my items [chuckles] before moving on. I also finished all of the side quests, which is, I think, you know, how I managed to put in over 100 hours into it. But yeah, it was very immersive, and I really enjoyed it. I don't know if this will become a norm for me. I know there are some people who are, you know, JRPG diehards and play a lot of these kinds of games, but they're a real, like, time investment for sure. JOËL: Are there achievements for completing everything? STEPHANIE: Not that I can tell on the Switch. I do know that, like, on other systems, you can see your progress on having done all of the things there are to do. But I think it's actually kind of better for me to just play [laughs] to just, like, think that I've done it all but not really, like, have something that tells me whether or not I've done it because then I would feel a lot more neurotic, I think, about being able to let it go where I am now. [laughs] JOËL: Right. If we've got, like, an explicit checklist of things or a progress bar, then it feels like you got to get to all the things. STEPHANIE: Yeah, exactly. I think there are still, you know, a couple more things that I wrote down on my little checklist of tasks that I would want to do once I feel like I want to come back to the game. But for now, like I said, I watched the credits roll. I teared up a little bit, you know, thinking about and reminiscing on my adventure with these characters, and I'm ready to put it down for a bit. JOËL: Did I hear correctly that you made a checklist for this game of things you wanted to do? STEPHANIE: Yes, [laughs] I did. JOËL: That's amazing. I love that. STEPHANIE: Yeah, you know, there are just so many things almost kind of like work where I had to, like, break down some of my goals. I wanted to, like, hit a certain level. I wanted to, you know, make sure I defeat these bosses that would help me get to those levels. And yeah, I got very into it. It was definitely a big part of my life for a couple of months. I got it originally because I needed a game to play on my flight to Asia back when I went to Japan. And I'm like, oh, like, this looks, you know, fun and engaging, and it will distract me for my, you know, over 10-hour flight. Turns out it distracted me for many, many more hours over several months [laughs] since then. But I had a great time. So yeah, that's what's new for me. Again, it's something I'd never really done before. I will say though I am very behind on my reading goal as a result. [chuckles] JOËL: I feel like this is a classic developer thing to do is, like, use the tools that we're used to in our job and then apply them to other parts of our life. And now it's just like, okay, well, I made a Kanban board to track my progress in this video game. You know, or, in my case, I'm definitely guilty of having drawn a dependency graph for the crafting tree for some video game. So I feel you really strongly there. STEPHANIE: Yes, I'm nodding heavily in agreement. I think it just scratches the same kind of itch of, you know, achieving, like, little things and then achieving one big thing. JOËL: So, speaking of places that are nice to have checklists and, like, well-defined requirements, you and I were talking earlier, and you have recently found some frustration around having user stories be defined well on your current project. STEPHANIE: Yes. So I've been reflecting a little bit about my current project and noticing what I think I might call product smells; I'm not quite sure, just some things I'm seeing in our day-to-day workflow that is getting me thinking. And I'm curious to hear if you've experienced something similar. But I find myself being tasked with a ticket that is quite vague. And maybe this was written by a product owner, or maybe it was written by another developer. And it is not quite actionable yet, so I have to go through the process of figuring out what I'm really needing to do here. I think another thing that has been quite frustrating is, you know, maybe we do find out what we want to do. And, like, I'll go back into the ticket, write down the requirements that I gathered, and do the ticket. I'll ship whatever change was required, and then I'll hear back from someone in a meeting or either as a one-off request in Slack. And it'll be like, "Hey, like, actually, you know, we want this to be different." And maybe you previously said that "Oh, the value for something would be 30. But now we found out more information; it should be 20. And so could you, like, make that change?" And then I'm not really sure what the best way to document a change like that is because it, you know, maybe existed in the previous ticket, but now it has changed. And do I create a new ticket for this, or do I just go ahead and make the code change? Like, who would know this information that we're now carrying about 20 being the value for, let's say, like, days or not meaning something in the code that we're writing? And I guess I've just been really curious about how to make sure that this doesn't become the norm where a lot of these conversations are just happening, and, you know, the people who happen to be in them know that this change happened. But then later on, someone is asking questions about, like, hey, like, when did this change? Or I expected this to be 30. But is this, you know, behaving as expected? So that was [laughs] a bit of a nebulous way of describing just, like, this churn that I feel with being the executor of work. But then, like, a lot of these things changing above me or separate from me and figuring out how to manage that. JOËL: When you were describing this scenario where you've done the work, and then someone's like, "Oh, could we change this value from, like, 30 to 20?" I'm thinking in my mind of the sort of beam that a lot of our designers face where it's like, you know, they have a design. They work on it; they do it. And then show it to a client, and the client is like, "I love this design. But could we just shift this box over, like, one pixel?" Like, they're, like, tiny, tiny, little changes that are kind of requested for change after you've done, like, this big thing. And, oftentimes, those pile-up. It's like, you shift it one pixel. It's like, oh, actually, you know what? Why don't we do it two pixels? And then it's like never-ending cycles, sometimes of, like, minute little changes. STEPHANIE: Yeah. But the minute changes really add up into, I think, really different behavior than what you maybe had decided as a team originally. And in the process of changing and evolving, I don't really know where documentation fits in. I've been working on this project that had a pretty comprehensive product design doc, where they had decided upfront on, you know, how the application is going to behave in many different scenarios. But again, like, that has changed over time. And when I recently had to onboard someone new to this project, you know, we sent over this document, and we're like, yeah, you can, you know, feel free to peruse it. But it's actually quite outdated. And then, similarly, right now, since the features that I'm working on are going through QA, there's been a lot of back and forth about, I'm seeing this, but the doc said that Y is supposed to happen, and I'm not sure if that's a bug or not. And I or someone else has to respond with that context that we were holding in our head about when that change happened. JOËL: That's really interesting. And I think it varies a lot based off the size of the organization. In a smaller organization, you're probably doing a lot of the requirements gathering yourself. You're talking to all the stakeholders. You're probably doing the QA yourself, or you're walking somebody else through QA. Versus a large organization, there might be an entirely separate product team, and a separate QA team, and a separate dev team. And a danger that I've often seen is where all of these teams are just kind of tossing work over the fence. And all you're given is a, you know, a ticket of, like, execute on this. Basically, turn these specs into code. And then you do that, and then you toss it over the fence to the QA team. And they check does the code do these things? And there's so much context that can easily get lost from one step to another. That being said, I think a lot of devs find it frustrating to do some of the requirements gathering work. How do you feel in general about scoping out a ticket or doing follow-up conversations with the product team about, like, "Hey, your idea for the ticket is this. How do you feel about doing these things? Or what if we cut these things?" Are those conversations that you enjoy having? Is that a fun part of the developer role for you? Or do you kind of wish that, like, somebody else did all of that so that you could, like, go heads down just writing code? STEPHANIE: I think it depends. That's a great question. Actually, I have so many thoughts in response. So let me try to figure out where I want to go from here. But I think I used to not like it. I used to be stressed out by it, and sometimes I still am. But when I thought my role was purely executing, to receive a ticket that is a bit vague, you know, I might have been left feeling, like, stuck, like, not knowing where to go from there. But now that has changed a bit because I received some really helpful feedback from an old manager of mine who was kind of invested in my growth. And she really suggested learning to become more comfortable with ambiguity because that just becomes more and more your job, I think, as you progress in your career. And so now I at least know what information I need to go get and have, you know, strategies for doing so. And also knowing that it's my job, like, knowing that no one else might be doing it, and it might just be me so that I can therefore get this ticket done. Because, like you said, that problem of throwing the work over the fence to someone else, at some point, that doesn't work because everyone has too much on their plates. And you have to just decide to be the one to seek the information that you need. JOËL: I think one way that, as developers, we bring a lot of value is that we help to cut through a lot of that ambiguity. I think if we see our role as merely translating a requirements document into code, that's a very simplistic point of view of what a talented developer does. So, like you said, as we grow in our careers, we start dealing with less and less defined things. We often have to start defining the problems that we're given. And we have to have these conversations with other teams to figure out what exactly we want to do. And maybe better understand why is it that we want to do this thing. What is the purpose of it? How are we going to get there? And my favorite: Do we have to do all of these things to hit the minimum value of this goal? Can I split this into multiple tickets? I love breaking down work. If I can make the ticket smaller, I'm all about that. STEPHANIE: Yes. I'm well aware. It's interesting about what you said, though, is that, like, yes, that becomes, in some ways, our superpower. But, for me, where the pain comes in is when that's not part of the expectations, where I am maybe tasked with something that is not clear enough, and yet, the time that I need to find that clarity is not given the respect that it, I think, deserves to build a good product because the expectation is that I should already be making progress on this ticket and that it will be delivered soon. You know, in that situation, I wish I had been in the room earlier. I wish I had been part of the process for developing the product strategy, or even just, like, have come in earlier to be able to ask, you know, why are we building this? And, like, what are some of the limitations on the technical side that we have? Because often, I find that it is a little too...not necessarily too late, but it is quite down the road that we then have to have these conversations, and it doesn't feel good. JOËL: I think that's one of the powerful things that came out of the agile movement was the idea that you have these cross-functional teams, that you don't have a separate product team, a separate dev team, a separate QA team, a separate design team that are all these isolated islands. But instead, you say, okay, we have a cross-functional team that is working on this aspect of the product. And it will be some product people, some dev people, some designers kind of all working together and communicating with each other. I know, shocking concept. And even depending on the context, a big idea is that the client or the customer is a part of that team. So, when we at thoughtbot work with a client, especially when they are maybe a smaller client like a startup founder, we make sure that they feel like they are a part of the team. They are involved in various meetings where we decide things. They have input. You know, they're part of that feedback cycle that we build. But that can also be the case for a larger company where your internal stakeholders are kind of built-in to be sort of part of your team. STEPHANIE: I've seen so many different flavors of trying to do Agile [laughs] that it has lost a little bit of meaning for me these days. And maybe we've incorporated some aspects of it. But then that idea of the tight feedback loops and then a cross-functional team where everyone is communicating that part has gotten a little bit lost, at least on my project. And I imagine that this is common, and our listeners might be finding themselves in a similar situation where things are starting to feel a little more like handing off and a little more like waterfall. [laughs] I'm curious, though, if you found yourself being requested to make a change from what the original decision was, how would you go about documenting that or not documenting it? Where do you think the best place for that information about how this feature now is supposed to work where should that live? JOËL: Are you talking about where do we document that a decision was made to change the original requirements of a task? STEPHANIE: Yes. JOËL: In general, I think that should live on the ticket just because as long as the ticket is live, I think it's good to have all the context on that ticket for whoever's working on it to have access at a glance. Sometimes it's worth it to say, you know what? We don't want to just keep this ticket live for weeks or maybe months on end. Let's ship this ticket, and create a follow-up to make a change later, especially if it's a change that's less important where it's like, you know what? It would be nice to have if...but, again, like, scope creep is a real danger. And so, again, me with the aggressive breaking up of tickets, I love to say, "That's a great idea. It would make a great change, not part of this ticket." So oftentimes, those changes I will push them into another ticket. STEPHANIE: That's interesting. What about documentation beyond the current work? So I'm thinking about once, you know, a feature is delivered, how do people in the organization then know how this feature is supposed to work? Like moving forward as something that is customer-facing. JOËL: That can vary a lot by organization, I think because there's a couple of different aspects to this. You have maybe some internal-facing documentation; maybe some customer support people need to know about the way the interface has changed. And then you also have customer-facing documentation where maybe you want some sort of, you know, you want a blog post talking about the new feature or some kind of release notes or something like that to be shared with your customers. And compiling that might look very different than what you do for your internal service reps. STEPHANIE: Yeah, I like that. It's true that the customer documentation is really helpful. At least for, the product that I'm working on, it has very comprehensive documentation about how to use that for its customers. And that has been really helpful because, hopefully, that should be the truest [laughs] information out there. But sometimes, you know, I find myself in meetings where none of us really know what happens. For example, a question that was asked recently is our product has a free trial capability. But it was unclear what happens to all of the data that the customer is getting access to as a feature. Like, what happens to that data after the free trial ends? Like, if they then have purchased a license, do they still have access to their free trial data? If, you know, there's a lapse between then, does it just get deleted, or will it show up again? And no one really knew the answer to that. And I think that was another area that got my spidey senses tingling a little bit; I think because it reminded me of...there was a definition I read somewhere of legacy code that is basically when the person who has the most context about how a piece of code works and then they leave the company and that institutional knowledge no longer exists, like, that is legacy code. And I almost think that that also applies to product a little bit where a legacy product is something where no one quite knows what is supposed to happen, but it's still being used by users. JOËL: That's a really fun definition there. I think there's sort of two related questions that are slightly different here, which is, one, how does the code behave? So, what happens when someone's trial period expires? And it's quite possible that no one on the team knows what actually happens when that time expires. And then the second question is, what should happen when a trial expires? And it's possible, again, that the product team didn't think through any of the edge cases. They only went for the happy path. And so it's possible if that is also fully undefined and no one knows. STEPHANIE: Yeah, I like that distinction you made a lot because they definitely go hand in hand, where someone realizes that some weird edge case happened, and then suddenly, they're asking those questions. And, you know, we realized, like, oh, like, that just didn't have enough, like, intention or thought behind how it was coded. So, like, it really is; who knows, right? Just whatever seems to happen. And I think that this actually kind of reminds me of a previous episode we did about empowering other departments in the company because, ultimately, a lot of those questions about, like, how does this work? What happens? Ends up going to a developer who has to go and read the code and report back. And while, you know, we do have that power, it can also be a bit of a curse, I think. [laughs] JOËL: I think this is an area where, as developers, we're maybe particularly skilled. Because of the work that we do, our brains are kind of wired to think about all of the edge cases, and sometimes they can be really annoying. But I think there's a lot of value sometimes when maybe the product team comes to us with a maybe somewhat nebulously scoped ticket or a series of tickets for, let's say, a free trial period feature that only goes through the happy path. And then sometimes it's up to us to push back or to follow up and say, "Okay, great. We've got a bunch of tickets for a free trial period. Have you thought about what happens after a trial expires but the person hasn't converted to a paying customer?" And then, oftentimes, the answer is like, "Oh, no, we didn't think about that." And I think oftentimes, as developers, our job is to kind of, like, seek out a lot of those edge cases. And we have a lot of techniques and methodologies that we use to try to find edge cases, things like test-driven development, various modeling tools that we'll try to use to make sure that we don't just crash or do something bad in our code. But what should the actual behavior be? That's a conversation that we need to have. And hopefully, that's one that maybe the product team has already had on their own. But oftentimes, the benefit of having that cross-functional team is the ability to kind of have that back and forth and say, "Hey, what about this edge case? Have we thought about that? How do we want that to behave?" STEPHANIE: Yeah, that actually made me think about the idea of tech debt but almost at a product level, where, hey, it turns out that we have all of these things that we didn't quite think through, and it's now causing problems. But how much do we invest in revisiting it? Because, you know, maybe this feature is several years old, and it was working just okay enough for it to, you know, be valuable. But we're now discovering these things and, you know, like, do we invest in them? Or are we more focused on, you know, coming up with new things and new features for our customers? JOËL: That's a classic prioritization problem. It also kind of reminds me of the idea of an MVP. What are the actual, like, minimum set of features that you need in order to try out something or to ship something to customers? And, you know, maybe we don't need some special behavior if your trial account doesn't convert. Maybe we're okay [laughs] that you log in, and the app just crashes. Probably not, because we would probably want you to convert to a paying customer at some point. But maybe we're okay if you just get a screen that says, "You have no projects," when, in fact, you did have projects. It's just that you're no longer on the free trial. Again, for business reasons, probably we want a call to action there that says, "You have five projects. They are not available to you. Please pay to unlock your projects again." That probably converts better. But, again, now that is a business decision. And that becomes a prioritization question that the team as a whole gets to address. Sometimes it can also be some really fun prioritization things where if you're on a really tight schedule, you might ship some features live knowing that you have a time limit, but you don't have to necessarily ship other things. So let's say you've got a 30-day trial, and maybe you ship that before you've even implemented what the dashboard will look like after your free trial has expired, and that's fine because no one's going to hit that condition for 30 days. So now you've got 30 days to go out and handle that condition. And maybe that's okay because it allowed you to get to market a little bit faster, allowed you to cut scope, break those tickets, yes, and just move that much faster. But it does require discipline because now you're on the clock. You've got 30 days to fix that edge case or potentially face some unhappy customers. STEPHANIE: Yeah, I think that's quite a funny way to handle it. It's really ruthless prioritization [laughs] there. But what you said was very interesting to me because I was thinking about how there is such a focus on new feature development and that being the thing that will attract customers or generate more money. But there is something to be said about investigating some of our old features of our existing system and finding opportunities there. And oftentimes, revisiting them will reduce the amount of pain [chuckles] that, you know, developers feel having to kind of keep track or have an eye on, like, where things are airing out, but then don't have the time to really invest in making it better or making that part of the product better. JOËL: I think that's a great opportunity then to have a conversation with other parts of the team. Typically, I think you have to convert some of those into more of a business case. So the business people in the company or the product people might not care about the sort of raw metrics that you see as a developer. Oh, we got an exception with a stack trace in this part of our app. What does that even mean? But if you say, hey, people who signed up for a free trial and then didn't immediately convert within 30 days who want to come back a month later and convert are unable to do so. And we've seen that that's about 10% of the people who signed up for a free trial. Well, now that's an interesting business question. Are we losing out on potentially 10% of customer acquisition? I'll bet the sales and marketing people care a lot about that. I'll bet the business people care a lot about that. The product people probably care a lot about that. And now we can have a conversation about should we prioritize this thing? Are these metrics that we should improve? Is this a part of our code that's worth investing in? STEPHANIE: Yeah, I like that because, in some ways, asking those questions about how does it work? Like, that is really an opportunity because then you can find out, and then you can make decisions about whether it's currently providing enough value as is or if there is something hiding under there to leverage. JOËL: And I think that's one of the other places where, as developers, we provide value to clients is that we can sort of talk both languages. We can talk product language. We can talk business language. But we can also talk code. And so when we see things like that in code, sort of translate that into, like, what are the business impacts of this code change? Which then allows everyone to make the best possible decisions for the mission of the organization that you're a part of. So we've talked about a variety of sort of patterns and anti-patterns that surround working through some of this churn on a product. I'm curious, Stephanie, for you, what's maybe one concrete thing that you've done recently that you've found has really helped you navigate this and maybe help reduce some of the stress that you feel as you navigate through this? STEPHANIE: Yeah, I think, for me, one of the worst things is when that discussion is had in a meeting or a [inaudible 35:45] and then is not put anywhere. And so, one thing I've been making sure to do is either asking the person who made the request to write it down, either on the ticket or in Slack. Or I will write it down, you know, I will document the outcomes of what we talked about and putting it in a public space so that people are aware. I think that small action has been helpful because we hold so much of this in our heads. And I've been finding that it ends up being hard for people to rotate onto different projects and, you know, get onboarded and up to speed effectively because there's so much knowledge and context transfer happening. But even just putting it in a place where maybe it's not relevant to everyone, but at least they see it. And then the next time that they're asked or maybe, like, do come around to working on this, they, like, have some fragment of a memory that they saw something about this. So that has been really helpful. It actually dovetails really nicely into what we were talking about with opportunities, too, because once it's out there, like, maybe someone else will see it and have an idea about how it could be better or that change not being what they expected, and they can weigh in a little more. So that's what I'm trying to do. And I think it's also nice to see how often that happens, right? If we're constantly seeing things changing because we have a written record of it, that could be helpful in bringing up and investigating further as to, like, why is this happening? Like, why do we experience this churn? And is that something we want to address? JOËL: Yeah, because an element that we haven't talked at all about is any sort of feedback cycle or retrospective, where we can talk about these things and having that written trail and saying, "Oh, we changed this decision five times in the past week, like, really churned there." Now maybe that prioritizes it to be an important thing to talk about and to improve for the next cycle. STEPHANIE: What I feel really strongly about is when, you know, each individual on a team is feeling this pain, but it not being known that it's actually a collective issue. Because maybe these things are happening in one-on-one conversations, and we don't realize that, like, oh, maybe there is something bigger here that we could improve on. And so the more eyes on it there are, the more visible it is, I think, that the easier it is to address. JOËL: I love that, the power of writing things down. On that note, shall we wrap up? STEPHANIE: Let's wrap up. Show notes for this episode can be found at bikeshed.fm. JOËL: This show has been produced and edited by Mandy Moore. STEPHANIE: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review in iTunes. It really helps other folks find the show. JOËL: If you have any feedback for this or any of our other episodes, you can reach us @_bikeshed, or you can reach me @joelquen on Twitter. STEPHANIE: Or reach both of us at hosts@bikeshed.fm via email. JOËL: Thanks so much for listening to The Bike Shed, and we'll see you next week. ALL: Byeeeeee!!!!!! ANNOUNCER: This podcast is brought to you by thoughtbot, your expert strategy, design, development, and product management partner. We bring digital products from idea to success and teach you how because we care. Learn more at thoughtbot.com.

japan japanese managing mvp empowering switch ends slack nintendo switch agile npc qa versus dag jrpg kanban mandy moore octopath traveler changing business quenneville octopath traveler ii activerecord business requirements stephanie yeah stephanie it stephanie oh

Active Record Callbacks

reading callbacks aji railsconf activerecord

Play Episode Listen Later Jul 11, 2023 28:47

For this week's episode, Aji welcomes our very first Guest Host, Sasha Grodzins! Aji & Sasha read about Active Record Callbacks and discuss pitfalls of callbacks, how they are similar to validations, and comments and commit messages as documentation.Reading for this episode: Active Record Callbacks5 Rails Callbacks Best Practices Used at GustoRuby Science's chapter on callbacksActive Record callbacks source codeAji at RailsConf 2019: Commit Messages to the rescue!Reading for episode 9: Active Record Associations

Hmmm, Maybe It's The Garbage Collector

Play Episode Listen Later Jul 7, 2023 52:05

On today's episode, Chris and Andrew have an early start and catch up on their lives. Then, they dive deep into the latest developments in the Rails community, including the release of Rails 7.0.6, bug fixes, and changes to Active Record. They share their experiences with GitHub deployments, documentation issues, and how they navigate through its challenges. They discuss the benefits of MySQL and Postgres, as well as the ongoing advancements in Postgres, specifically Crunchy Data's contributions. Chris and Andrew share their views on working in different company sizes, the joys of learning new things, dealing with burnout, and the slower pace of feature shipping in larger companies. There's a discussion on Reddit's recent actions, its impact on subreddit moderations, and the discontinuation of the Reddit API. We'll also hear about Chris's cooking adventures, experimenting with different flavors, and making some Texas Twinkies. Hit download to hear more! [00:02:00] Chris and Andrew talk about the release of Rails v7.0.6 with bug fixes and changes in libraries like Action Cable and Active Record, including subqueries and associations with polymorphic relationships.[00:06:10] Andrew is curious about the GitHub deployment stuff and expresses his desire to create GitHub deploys from Heroku. They talk about the complexities of setting up GitHub deployments and the lack of clear information from GitHub, and how the documentation with Checks API can be confusing to set up. [00:09:49] Chris discusses the challenges of figuring out GitHub's deployment process and the lack of documentation. He expresses frustration with the lack of clarity and support for smaller accounts. [00:14:41] PlanetScale is brought up and its association with MySQL, and they discuss the benefits of MySQL and Postgres, and the new features and advancements in Postgres, including Crunchy Data's contributions and the potential use of Postgres in web environments. [00:17:43] Chris shares a fun story about working on implementing jump server support in the new Hatchbox. They encountered unexpected complexities with the net-ssh gem to address the problem. [00:29:51] Chris emphasizes the importance of being mindful of memory usage and performance trade-offs and how it becomes more critical when building large-scale products. [00:31:59] Andrew mentions that releasing features can be challenging and Podia is currently facing that challenge with releasing a feature while also building onto it. He emphasizes the importance of coordination, communication, and learning from code to recognize and solve problems faster. [00:33:46] Chris reflects on his experience working at a consulting agency and how it allowed him to learn quickly by facing different projects and finds joy learning new things as a programmer. [00:34:43] We hear Andrew talk about feeling stuck in a job, comparing small companies which offer more challenges, to big companies where employees get stuck doing the same tasks, and Chris tells us he's happiest when learning new things and how it accelerates burnout.[00:35:57] Chris discusses the challenges faced by big companies when it comes to feature shipping due to the need to ensure existing users are not negatively impacted, and Andrew highlights the varying levels of impact when breaking code and emphasizes the importance of being able to find and fix bugs quickly. [00:39:00] We hear about Chris's mad cooking skills with pulled pork and experimenting with smoked cream cheese which he hopes to use in some Texas Twinkies. [00:43:53] The conversation shifts to Reddit and its recent actions regarding subreddit moderation and the discontinuation of the Reddit API, and they express frustration with Reddit's handling of the situation and the negative consequences it's had on the community. [00:51:30] We end with Chris needing to attend to his cooking tasks and Andrew mentions his responsibility to lead Podia in Jason and Jamie's absence. Panelists:Chris OliverAndrew MasonSponsor:HoneybadgerLinks:Jason Charnes TwitterChris Oliver TwitterAndrew Mason TwitterRails 7.0.6 PlanetScaleCrunchy DataReddit Won't Be the Same. Neither Will the Internet (WIRED)What the Heck is a Texas Twinkie?

reddit heck programming coding github hmmm rails software development ruby on rails mysql podia heroku postgres andrew mason chris oliver planetscale activerecord reddit api garbage collector action cable

391: Learn with APPL

Play Episode Listen Later Jul 5, 2023 40:45

Stephanie went to her first WNBA game. Also: Bingo. Joël's new project has him trying to bring in multiple databases to back their ActiveRecord models. He's never done multi-database setups in Rails before, and he doesn't hate it. Stephanie shares bits from a discussion with former Bike Shed host Steph Viccari about learning goals. Four elements stood out: Adventure (try something new) Passion (topic) Profit (from recent learnings) Low-risk (applicable today) = APPL Stephanie and Joël discuss what motivates them, what they find interesting vs. what has immediate business value, and how they advocate for themselves in these situations. They ponder if these topics can bring long-term value and discuss the impact that learning Elm had on Joël's client work. Elm (https://elm-lang.org/) Practical Object-Oriented Design in Ruby (https://www.poodr.com/) Design Patterns in Ruby (http://designpatternsinruby.com/) Quarter Life (https://www.penguinrandomhouse.com/books/579928/quarterlife-by-satya-doyle-byock/) Working Iteratively (https://thoughtbot.com/blog/working-iteratively) Transcript: JOËL: Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Joël Quenneville. STEPHANIE: And I'm Stephanie Minn. And together, we're here to share a bit of what we've learned along the way. JOËL: So, Stephanie, what's new in your world? STEPHANIE: All right, I have a new-new thing and an old-new thing to share with you today. So the new-new thing is that I went to my first WNBA game [laughs] last week, which is also my third professional sports game ever, probably. I am not a sports person. But a rather new friend of mine invited me to go with her because they are fans, and so I was like, yeah, sure. I'll try anything once. And I went, and I had a great time. It was very exciting. I mean, I know the basic rules of basketball, right? Get the ball in the hoop. But I was very surprised to see how fast-paced it was. And, you know, I was like, wow, like, this is so much fun. There's so much going on, like, the music, you know, the crowd. It was very energizing. And then my friend actually told me that that was a pretty slow game, [chuckles] relative to how they normally go. And I was like, oh, wow, like, if that was slow, then I can't wait for a real competitive [laughs] game next time. So that's my new-new thing. I had a good time. Will do it again. I'm just, like, a 15-minute bike ride from the stadium for our team in Chicago. It's called The Sky. That's our WNBA team. So yeah, I'm looking forward to being basketball Stephanie, I guess. [chuckles] JOËL: That's really cool. How does the speed compare to other sports you've gone to see? STEPHANIE: I think this is why I was interested because I've really only seen baseball, for which I know very little. And that, I think, is, like, a much slower-paced kind of sport. Yeah, I have some memories of going to, like, college football games, which also, like, quite slow. I just remember standing around for a while. [laughs] So I think basketball might be the thing for me, at least in terms of engaging my interest. JOËL: You want something that actually engages you with the sport the whole time. It's not just a social event themed around occasionally watching someone do something. STEPHANIE: Yes, exactly. I also enjoyed the half-time performances, you know, there was just, like, a local dance team. And I thought that was all just very fun. And, yes, I had a lot to, you know, just, like, point to and ask questions about because there was just so much going on, as opposed to sitting and waiting, at least that was my experience [laughs] at other kinds of sports games. As for the old-new thing, now that it's summer, there is a local bar near me that does bingo every week. But it's not just normal bingo. It's called veggie bingo, which I realize is kind of confusing [chuckles] if you just, like, call it veggie bingo, but it's bingo where you win vegetables or, like, produce from local community gardens and other, you know, small batch food items. And I had a great time doing it last year. I met some new friends. It just became our weekly hangout. And so I'm looking forward to doing that again. And, I don't know, I'm just glad I have fun things to share about what's new in my world now that the weather is warm and I'm doing stuff again. I feel like there was one point in the winter where I was coming [chuckles] onto the show and sharing how I had just gotten a heated blanket in the middle of winter, and that was the most exciting thing going on for me. So it feels good to be able to bring up some new stuff. JOËL: Seasonality is a thing, right? And, you know, there are rhythms in life. And sometimes things are more fast-paced, sometimes they're a bit slower. That's really exciting. Did you take any produce home, or did you win anything when you went to play? STEPHANIE: I did. I won a big bag of produce the last time that I went. At this point, it was last season. But it was right before I was about to go on vacation. So I ended up -- JOËL: Oh no. STEPHANIE: [chuckles] Right. I ended up not being able to, you know, keep it in the fridge and just giving it away to my friends who did not win. So I think it was a good situation overall. That's my tip, is go to bingo or any kind of prize-winning hang out as a group, and then you can share the rewards. It's very exciting. Even if you don't win, you know, like, probably someone else at your table will win, and that is equally fun. JOËL: I think the closest I've been to that experience is going to play, like, bar trivia with some friends and then winning a gift card that covers our dinner and drinks for the evening. STEPHANIE: Yeah, yeah, that's great. I used to go to a local trivia around me too. The best part about bingo, though, is that it requires no skill at all. [laughs] I, yeah, didn't realize, again, how into these kinds of things I would be until I just tried it out. Like, that was...bingo is another thing I don't think I would have internally decided to go do. But yeah, my friends just have all these great ideas about fun things to do, and I will happily join them. So, Joël, what's new in your world? JOËL: So I've recently started a new client project. And one of the really interesting things that I've been doing on this project is trying to bring in multiple databases to back our ActiveRecord models. This is a Rails app. I've never done multi-database setups in Rails before. It's been a feature since Rails 6, but this is my first time interacting with that system. And, you know, it's actually pretty nice. STEPHANIE: Really? It ended up being pretty straightforward or pretty easy to set up? JOËL: Yeah. There's a little bit of futzing around you have to do with the database YAML configuration file. And then what you end up doing is setting up another base class for your ActiveRecord models to inherit from. So, typically, you have that application record that you would inherit from for your primary database. But for other databases, if you want a model to be backed by a table from that system, then you would have a separate base class that all of those models inherit from, and that's pretty much it. Everything else just works. A bunch of your Rake tasks get a little bit different. So you've got to, like, configure your setup scripts and your test scripts and all that thing a little bit differently. But yeah, you can just query, do all the normal things you do with an ActiveRecord model, but it's reading from a different database. STEPHANIE: That's really cool that it ended up being pretty painless. And I'm thinking, for the most part, as a developer, you know, working in that kind of codebase; maybe they don't really need to know too much about the details of the other databases. And they can just rely on the typical Rails conventions and things they know how to do via Rails. Do you suspect that there might be some future where that might become a gotcha or something that someone has to debug a little further because of the multi-database setup? JOËL: There are some infrastructure things, but I think I'm handling all of them upfront. So like I said, configuring various setup scripts, or test scripts, or CI, that kind of thing to make sure that they all work. Once that's all done, I think it should pretty much just work. And people can use them like they would normal ActiveRecord models. The one gotcha is that you can't join models across two different databases. You can't use ActiveRecord to write a query that would try to join two tables that are in different databases because the SQL won't allow for that. So, if you're ever trying to do something like that or you have some kind of association where you're trying to do some special join, that would not work. So, if somebody attempts that, they might get an unexpected error. Other than that, I think it just keeps working as normal, and people can treat it more or less as if it's one database. STEPHANIE: That's interesting. How do you model relationships between tables on the two different databases, then? Like, how would that work? JOËL: I've not gotten that far yet. For some things, I imagine just it's two queries. I'm not sure if the ActiveRecord associations handle that automatically for you. I think they probably will. So you probably can get away with an association where one model lives in one database. Let's say your article lives in one database, and it has many comments that live in a different database. Because then you would make one query to load the article, get the article ID, and then you would do another query to the second database and say, hey, find all the comments with this article ID, which is already, I think, what ActiveRecord does in one single database. It is making two queries. It's just that now those two queries are going to be two different databases rather than to a single one. STEPHANIE: Interesting. Okay. I did think that maybe ActiveRecord did some fancy join thing under the hood. And when you mentioned that that wouldn't be possible when the two tables are on different databases, I was kind of curious about how that would work. But that makes sense. That would be really cool if it is, you know, that straightforward. And, hopefully, it just doesn't become too big of an issue that comes back to haunt someone later. JOËL: Right. So pretty much, if there is a situation where you were relying on a JOIN, you will now have to make two separate queries and then combine the results yourself. STEPHANIE: Right. I also want to give you kudos for doing all the good work of setting it up so that, hopefully, future developers don't have to think about it. JOËL: Kudos to the Rails team as well. It's nice to have that just kind of built into the framework. Again, it's not something I've needed in, you know, a decade of doing Rails, but then, you know, now that I have run into a situation where I need that, it just works out of the box. So that's been really nice. So, a couple of weeks ago, we talked about the fact that we were going through review season and that we had to fill out reviews for ourselves then also fill out peer reviews for each other. You had brought up a really interesting conversation you had about reaching out to other people and trying to get feedback on what kind of review or feedback would be helpful for them. STEPHANIE: I did, yeah. Though, I think in this case, the person writing that feedback actually reached out to me, but certainly, it goes both ways. Spoiler alert - that person was Steph Viccari, former [laughs] host of The Bike Shed. JOËL: So Steph also reached out to me with similar questions. And that spawned a really interesting conversation around personal goals and what it looks like, particularly when it comes to what to learn next in technology. We started discussing things, and I listed out some different things that I was interested in. And then just kind of out of nowhere, Steph just pulls out this, like, oh, I noticed these four elements. And I'm going to list them out here because it's really cool. So these four elements were adventure, so trying something new. Passion, so something that's really exciting to you. Profit something where you can leverage some recent things that you've done to get more value out of some work you've already done. And then finally, low risk, something that would be applicable today. And it just kind of turns out that this makes a funny little acronym: APPL. And apples are often a symbol of learning. So that was kind of a fun coincidence. STEPHANIE: I love when someone is able to just pull apart or to tease out pieces of, you know, something that you might have just, like, kind of dumped all of into a message or something, and then to get, like, a second eye to really pick out the themes is so valuable, I think. And I'm obsessed with this framework. I think we might have come across something new that could really be helpful for a lot of other people. JOËL: It's definitely...I think it shows capacity for a higher level of thinking when someone's able to just look at a bunch of concrete things and say, wait a minute; I'm seeing some larger themes emerge from what you're talking about. And I always really appreciate it when I'm having a conversation with someone, and they're like, "Hey, I think what I'm hearing is this." And you're like, "Whoa, you're totally right. And I did not even know that that's where I was going." STEPHANIE: Absolutely. I'd love to go through this acronym and talk about a few different things that we've learned in our careers that kind of correspond with each of these elements. JOËL: Yeah, that sounds great. So I think, you know, the first one here is adventure, trying something new. So, what's something where you tried something new or adventurous that you think was worthwhile? STEPHANIE: Hosting this podcast. [laughs] It was a huge adventure for me and a really big stretch, I think. And that's what the idea of adventure evokes for me is, like, maybe it's uncharted territory for you, and you might have some reservations about it. But, you know, obviously, the flip side of an adventure is how fun and exciting and just new and stimulating it can be. And so I think, yeah, like, when I first started doing this with you, and even when you first asked me, I was pretty nervous. I was really hesitant. It took me a long time to, you know, think it over. I was like, do I want to commit to something that I have never done before, and it's, like, a pretty longer-term commitment? And I'm really glad I did it. It's certainly been an adventure. It's, you know, got its ups and downs. You know, not every week do I feel like that went really well, like, that was a great episode. Sometimes I'm like, that was just an okay episode, [laughs] and, you know, that's fine too. But I feel like this was really important in helping me feel more confident in sharing my technical opinions, helping me feel more comfortable just kind of, like, sharing where I am and not feeling like I should be somewhere else, like, some other level or have already known something. Like, the point is for us to share the journey week by week, and that was something that was really hard for me. So being on this Bike Shed adventure with you has been very valuable for me. JOËL: Yeah, it's sharing these new things we've learned along the way. STEPHANIE: Literally. Yes. What about you? Do you have something adventurous that you learned? JOËL: I think an important inflection point where I tried something new was when I learned the Elm programming language. So I had mostly done procedural languages back in the day. And then I got into Ruby, did a lot of OO. And then I got into Elm, which is statically-typed, purely functional, all these things that are kind of opposite of Ruby in some ways. But I think it shares with Ruby that same focus on developer happiness and developer productivity. So, in some ways, I felt really at home. But I had to learn just a whole new way of programming. And, one, it's cool. I have a new tool in my belt. And I think it's been a couple of years just learning how to use this language and how to be effective with it. But then afterwards, I spent a couple of years just kind of synthesizing the lessons learned there and trying to see, are there broader principles at play here? Are there ideas here that I can bring back to my work in Ruby? And then maybe even are there some ideas here that intersect with some theories and things that I know from Ruby? So maybe some ways of structuring data or structuring code from functional programming where some best practices there kind of converge on similar ideas as maybe some object-oriented best practices, or maybe some ideas from test-driven development converge on similar ideas from functional programming. And I think that's where, all of a sudden, I was unlocking all these new insights that made me a better Ruby developer because I'd gone on an adventure and done something completely out of left field. STEPHANIE: Yeah, absolutely. Do you remember what was hard about that when you first embarked on learning Elm? JOËL: All the things you're used to doing, you just can't do. So you don't have looping constructs in Elm. The only thing you can do is recursion, which, you know, it's been a long time since CS classes. And you don't typically write recursion in Ruby. So I had to learn a whole new thing. And then it turns out that most people don't write recursion. There's all these other ways of doing things that you have to learn. You have to learn to do folds or to use maps and things like that. Yeah, you're just like, oh, how do I do X in Elm? And you have to figure it out. And then maybe sometimes it turns out you're asking the wrong question. So it's like, oh, how do I do the equivalent of a for loop with array indexes in Elm to, like, iterate through a data structure? And it's like, well, kind of here's technically the way you could do that, but you would never solve a problem in that way. You've got to learn a new way of thinking, a new way of approaching problems. And I think it was that underlying new paradigm that was really difficult to get. But once I did get it, now that I have two paradigms, I think it made me a much more solid developer. STEPHANIE: Right. That sounds very humbling, too, to kind of have to invert what you thought you knew and just be in that, you know, beginner's mindset, which we've talked about a little bit before. JOËL: I think in some ways now being on the other side of it, it's similar to the insights you get from speaking multiple human languages, so being bilingual or trilingual or something like that where instead of just having assumptions about, oh, this is just how language works, because that's how your personal language works, now that you have more than one example to draw on, you can be like, oh, well, here's how languages tend to do things differently. Here's how languages are similar. And I think it gives you a much better and richer feeling for how languages work and how communication works. And similar to having multiple paradigms in programming, I think this has given me a much richer foundation now for exploring and building programs. STEPHANIE: That's really cool. I guess that actually leads quite well into the next element, which is passion. Because once you've tried some new things, you get the information of do I like this thing, or do I not like this thing? And then from there, you know, you gravitate towards the things you are passionate about to get a deeper understanding. And it becomes less about like, oh, just testing out the waters and like, knowing, hey, like, I constantly find myself thinking about this, like, let me keep going. JOËL: Yeah. Or sometimes, it's deciding what do I want to learn next? And you just pick something that's interesting to you without necessarily being like, oh, strategically, I think this is another paradigm that's going to expand my mind. Or this is going to make me, you know, help me get that promotion next quarter, purely based off of interest. Like, this sounds fun. STEPHANIE: That's really interesting because I think I actually came to it from a different angle, where one thing that I think was very helpful in my learning that came just, like, completely internally, like, no one told me to do this was reading books about design patterns. And that was something that I did a couple of years into my career because I was quite puzzled, I suppose, by my day-to-day experience in terms of wanting to solve a problem or develop a feature but not having a very good framework for steps to go about it, or not feeling very confident that I had a good strategy for doing it. It was very, for me, it felt very just kind of, like, throwing pasta to the wall and seeing what would stick. And I was really interested in reducing that pain, basically. And so that led me to read books. And, again, that was not something, like, someone was like, hey, I really think that you could benefit from this. It was just like, well, I want to improve my own experience. And, you know, some of the ones that I remember reading (And this was based off of recommendations from others kind of when I floated the idea.) was, you know, Sandi Metz's Practical Object-Oriented Design in Ruby. Design Patterns in Ruby by Ross Olsen. Those were just, like, purely out of interest. Yeah, I guess I'm curious, for you, what fun and passion look like. JOËL: Yeah, I think one thing that's a really fun side effect of passion learning is that I find that I tend to learn a lot faster and go a lot deeper, or I get more for every individual hour I put into learning just because passion or interest is such a multiplier. Similar to you, I think I went through a time where I was just gobbling up everything I could see on design patterns, and code structure, things like that. Yeah, I've always been really excited about data modeling in general and how to structure programs to make them easy to change while also not putting a high maintenance burden on it, learning those trade-offs, learning those principles, learning a lot of those ideas. I think that desire came out of some pain I felt pretty early on in my programming career, where I was just writing purely self-taught at this point from a few tutorials online. Code beyond a few hundred lines would just kind of implode under the weight of its own complexity. And so, like, I know that professional programmers are writing massively larger programs that are totally fine. So what am I missing? And so I think that sort of spurred an interest. And I've kind of been chasing that ever since. Even though I'm at the point where that is no longer a problem in my daily life, it is still an interest that I keep. STEPHANIE: Yeah. If I were to pull out another interest of yours that I've noticed that kind of seems in the same realm of, you know, you can just chase this forever, is working incrementally, right? And just all the ways that you can incorporate that into your day-to-day. And I know that's something we've talked about a lot. But I think that's really cool because, yeah, it just comes from just a pure desire on your own front to, like, see how far you can take it. JOËL: I think you pulled out something interesting there. Because sometimes, you have an interest in a whole new topic, and sometimes the interest is more about taking something I already know and just seeing can I take it to an extreme? What happens when I really go to the boundaries of this idea? And maybe I don't need to go there ever for a client project. But let me put up a proof of concept somewhere and try it out just for the fun of it to see can I take this idea, then push it to an extreme and see does it break at an extreme? Does it behave weirdly? And that is just an enriching journey in and of itself. Have you ever done, like, a...maybe not a whole learning journey but, you know, taken a few hours, or maybe even, like, some time on one of our investment Fridays to just explore some random idea and try it out? And it's like, huh, that was cool; that was a journey. And then maybe you move on to something next week because it's not like a big planned thing. But you're taking a few hours to dig into something totally random. STEPHANIE: I actually think I'm less inclined to do that than maybe you or other folks are. I find the things I choose to spend my time on do have to feel more relevant to me in the moment or at least in my day-to-day work. And I think that actually is another excellent transition into the last couple of elements in the APPL framework that we've now coined. The next being profit or, I guess, the idea of being valuable to you in your job in that moment, I suppose. Or I guess not even in that moment, but kind of connecting what you're learning to something that would provide you value. So I know you were talking about learning Elm, and now you're able to see all of the value that it has provided, but maybe at the time, that was a little bit less of your focus. But for me, I find that, like, a driver for how I choose to spend my time. Often it's because, yeah, for the goal of reducing pain. Being consultants, we work on a lot of different projects, sometimes in different frameworks, or languages, or new technologies. Like, you've mentioned having to, just now, on your new client project learning how to interact with different databases, and it sounds like older software that you might not have encountered before. And I think that ends up falling higher on my priority list depending on the timing of what I'm currently working on is, oh, like, you know, TypeScript is something that has, like, kind of come and go as my projects have shifted. And so when it comes back to working on something using it, I'm like, oh, like, I really want to focus on this right now because it has very clear value to me in the next three to six months, or however long. But I have also noticed that once I'm off of that project, that priority definitely recedes. JOËL: Yeah, I think that plays into that final element as well of the APPL, the low risk things that are applicable today that have value right now. Those tend to be things like, oh, I see that I'm going to be scheduled on a client that needs this technology next month. Maybe I should learn that, or maybe I should refresh this idea or go a little bit deeper because this is something new that I'm going to need. So, at some point, I knew that there was a Python project coming down the line. I was like, okay, well, maybe I'm going to spend a couple of Fridays digging into some Django tutorials and compare and contrast with Rails. STEPHANIE: The low-risk element is interesting to me because I found it to be a challenging balance to figure out how much time to invest in becoming really comfortable in a new technology. I find myself sometimes learning just enough to get what I need to get done. And then other times really feeling like, wow, like, I wish I knew this better because that would make my life easier, or I would just feel a lot better about what I'm doing. And kind of struggling with when to spend that time, especially when there's, you know, other expectations of me in terms of my delivery. JOËL: Yeah, that almost sounds like a contrast between technologies that fall in that low-risk bucket, like, immediately useful, versus ones that fall in the passion bucket that you're interested in taking deeply and maybe even to an extreme. STEPHANIE: That's really interesting. What I like about this list of themes that we've pulled out is that, like, one thing can fall into a number of different categories. And so it's really quite flexible. It actually reminds me of a book that I just finished reading. The book is called Quarterlife. And the thing that stuck out to me the most is the author, who is a psychotherapist; she has basically come up with two types of people, or at least two things, that end up being really big drivers of, like, human motivation and behavior. And that's stability types and meaning types, and the goal is to have a little bit of both. So you may be more inclined towards stability and wanting to learn the things that you need to know for your job, to do well in your role, kind of like we were talking about to reduce that pain, to feel a little more in control, or have a little more autonomy over your day to day and how you work. And then there's the seeking meaning, and when we talked about adventure and passion, it kind of reminded me of that. Like, those are things that we do because we want to feel something or understand something or because it's fun. And ironically, this list of four things has two that kind of fall into each category. And ultimately, the author, she, you know, was very upfront about needing both in our lives. And I thought that was a really cool distinction. And it was helpful for me to understand, like, oh yeah, like, in the early years of my career, I did really focus on learning things that would be profitable, or valuable, or low risk because those were the things that I needed in my job, like, right now. And I am now feeling stable enough to explore the meaningful aspects and feel excited by trying out things that I think I just wasn't ready for back in the day. But it actually sounds like you may kind of have a different leaning than I do. JOËL: That is really interesting. I think what was really fascinating as you mentioned those two sort of types of people. And, in my mind, now I'm immediately seeing some kind of two-dimensional graph, and now we've got four quadrants. And so are we leaning towards stability versus...was it adventure was the other one? Or meaning. STEPHANIE: Meaning, yes. JOËL: So now you've got, like, your quadrant that is high stability, high meaning, low stability, high meaning, like, all those four quadrants. And maybe these four things happen to fall into that, or maybe there's some other slightly different set of qualities that you could build a quadrant for here. One that is interesting, and I don't know how closely it intersects with this idea of stability versus meaning, is how quickly the things you learn become useful. So that low risk, like that L from APPL, those are things that are immediately useful. So you put a little bit of work learning this, and you can immediately use it on the job. In fact, that's probably why you're learning it. Whereas me going off and learning Elm is not because we've got any clients in the pipeline using Elm. It's purely for interest. Is it going to pay off? I think most learning pays off long-term, especially if it helps you build a richer understanding of the different ways software works or helps you have new mental models, new tools for doing things. And so I think, you know, 5, 6, 7 years later, learning Elm has been one of the highest payoff things that I've done to kind of take my coding career to the next level. That being said, I would not have seen that at the time. So the payoff is much more long-term. How do you kind of navigate when you're trying to learn something, whether you want something with a short-term payoff or a longer-term payoff? STEPHANIE: Yeah, that's so interesting. I wonder if there was maybe someone who could have helped you identify the ways that Elm could have possibly paid off. And I know, you know, you're looking back on it in retrospect, and it's easy to see, especially after many years and a lot of deep thinking about it. But kind of referring back to this idea of seeking meaning and that just being important to feeling happy at your job, like, maybe it was just valuable because you needed to scratch that itch and to experience something that would be interesting or stimulating in that way to prevent burning out or something like that. JOËL: Oh, I like that. So the idea that you're learning a thing, not specifically because you're expecting some payoff in the long term but because of the joy of learning, is reward in and of itself, and how that actually keeps you fresh in the moment to keep going on a career that might, you know, last 5, 10, 20, 30 years, and how that keeps you refreshed rather than like, oh, but, like, I'm going to see a payoff in five years where now, all of a sudden, I'm faced with a problem. And I can be like, ah, yes, of course, monads are what we need here. And that's a nice side effect, but maybe not the main thing you look for when you're going for something in that passion bucket. STEPHANIE: Yeah, absolutely. To go back to your question a little bit, I had mentioned that I was wondering if there was someone who could help point out ways that your interests might be useful. And I think that would be a strategy that I would try if I find myself in that conundrum, I suppose, of, like, being like, hey, like, this is really interesting to me. I'm not able to see any super immediate benefits, but maybe I can go find an expert in this who can share with me, like, from their experience, what diving deep into that topic helped them. And if that's something that I need to then kind of justify to a manager or just kind of explain, like, hey, this is why I'm spending my time doing this is because of this insight that I got from someone else. That would be, I think, a really great strategy if you find yourself needing to kind of explain your reasoning. But yeah, I also think it's, like, incredibly important to just have passion and joy in your work. And that should be a priority, right? Even if it's not immediately clear, the tangible or valuable to the company benefits in the current moment. JOËL: And I think what I'm hearing is that maybe it's a bit of a false premise to say there are some things that you follow for passion that only pay off in the long term. Because if you are in it for passion, then you're getting an immediate payoff regardless. You may also get an additional payoff in the long term. But you're absolutely getting some kind of payoff immediately as well. STEPHANIE: Yeah, I think that's true for adventure because knowing what you don't like is also really valuable information. So, if you try something and it ends up not panning out for you, you know, I think some people might feel a little bit disappointed or discouraged. They think, oh, like, they kind of wasted time. But I don't know; I think that's all part of the discovery process. And that brings you closer and closer to, yeah, knowing what you want out of your learning and your career. JOËL: So I'm really curious now. This whole, you know, APPL framework came out of a very random conversation. Is this something that maybe you're going to take into your own sort of goal-setting moving forward? Maybe try to identify, like, okay, what is something adventurous that I want to do, something I want to do for passion, something that I think for profit, and then something low risk? And then maybe have that inform where you put some energy in the next quarter, the next year, whatever timeline you're planning for. STEPHANIE: Yeah, I thought about this a little bit before we started recording. But one very loose goal of mine...and this actually, I think, came up a little more tangibly after coming back from RubyKaigi and being so inspired by all of the cool open-source tooling and hearing how meaningful it was for people to be working on something that they knew would have an impact on a lot of people in their development experience. Having an impact is something that I feel very passionate about and very interested in. And the adventure part for me might be, like, dabbling a little bit into open-source tooling and seeing if there might be a project that I would be interested or comfortable in dipping my feet into. What about you? Do you have anything in the near or long-term future that might fall into one of these buckets? JOËL: So I do have a list of things. I don't know that I will pursue all of them or maybe any of them. But here's my kind of rough APPL here. So something adventurous, something new would be digging into the language Rust. Again, the idea is to try a completely new paradigm, something low-level, something typed, something that deals with a lot of memory, something that does well with concurrency and parallelism. These are all things that I've not explored quite as much. So this would be covering new ground. Something that is a passion, something that's interesting to me, would be formal methods, so I'm thinking maybe a language like TLA+ or Alloy. Data modeling, in general, is something that really excites me. These techniques that I think build on some of the ideas that I have from types but that go, like, to an extreme and also in a slightly different direction are really intriguing to me. So, if there's something that maybe I'm staying up in the evenings to do, I think that might be the most intriguing thing for me right now. Something that might be more profitable, I think, would be digging into the world of data science, particularly looking at Notebooks as a technology. Right now, when I need to crunch data, I'm mostly just doing spreadsheets. But I think there are some really cool things that we could do with Notebooks that come up in client work. I manage to do them when you're with a random command-line script or sometimes with Excel. But I think having that tool would probably be something that allows me to do that job better. And then, finally, something low-risk that I know we could use on a client project would be digging in more into TypeScript. I know just enough to be dangerous, but we do TypeScript all the time. And so, mastering TypeScript would definitely be something that would pay off on a client project. STEPHANIE: I love that list. Thank you for sharing. JOËL: Also, I just want to note that there are only four things here. It doesn't fully spell APPL because there's no E at the end. And so when I see the acronym now, I think it looks like a stock ticker. STEPHANIE: It really does. But I think it's pretty trendy to have an acronym that's basically a word or a noun but then missing a vowel so... JOËL: Oh, absolutely. Time to register that applframework.com domain. STEPHANIE: Yeah, I agree. I also love what you said. You called it a rough APPL. And that was very [laughs] evocative for me as well. And just thinking about an apple that someone has, like, bitten into a little bit [laughs] and has some rough edges. But yeah, I hope that people, you know, maybe find some insight into the kinds of learnings and goals that they are interested in or are thinking about. And using these themes to communicate it to other people, I think, is really important, or even to yourself. Maybe yourself first and then to others because that's how your co-workers can know how to support you. JOËL: That's really interesting that you are thinking of it in terms of a tool for communication to others. I think when I first encountered this idea, it was more as a tool of self-discovery, trying to better understand why I was interested in different things and maybe better understanding how I want to divide up the energy that I have or the time that I have into different topics. But I can definitely see how that would be useful for communicating with team members as well. STEPHANIE: On that note, shall we wrap up? JOËL: Let's wrap up. STEPHANIE: Show notes for this episode can be found at bikeshed.fm. JOËL: This show has been produced and edited by Mandy Moore. STEPHANIE: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review in iTunes. It really helps other folks find the show. JOËL: If you have any feedback for this or any of our other episodes, you can reach us @_bikeshed, or you can reach me @joelquen on Twitter. STEPHANIE: Or reach both of us at hosts@bikeshed.fm via email. JOËL: Thanks so much for listening to The Bike Shed, and we'll see you next week. ALL: Byeeeeeee!!!!!!! ANNOUNCER: This podcast is brought to you by thoughtbot, your expert strategy, design, development, and product management partner. We bring digital products from idea to success and teach you how because we care. Learn more at thoughtbot.com.

time chicago spoilers passion data adventure code profit id wnba similar excel rust cs python rails django rake sql elm mandy moore notebooks oo alloys typescript tla appl design patterns quarter life yaml quenneville activerecord sandi metz rubykaigi stephanie yeah practical object oriented design stephanie it stephanie right

390: The Truth about Truthiness

Play Episode Listen Later Jun 27, 2023 39:58

Joël's new work project involves tricky date formats. Stephanie has been working with former Bike Shed host Steph Viccari and loved her peer review feedback. The concept of truthiness is tough to grasp sometimes, and JavaScript and Ruby differ in their implementation of truthiness. Is this a problem? Do you prefer one model over the other? What can we learn about these design decisions? How can we avoid common pitfalls? [EDI](https://www.stedi.com/blog/date-and-time-in-edi](https:/www.stedi.com/blog/date-and-time-in-edi) [Booleans don't exist in Ruby](https://thoughtbot.com/blog/what-is-a-boolean](https://thoughtbot.com/blog/what-is-a-boolean) [Rails valid? method](https://api.rubyonrails.org/classes/ActiveRecord/Validations.html#method-i-valid-3F](https://api.rubyonrails.org/classes/ActiveRecord/Validations.html#method-i-valid-3F) Parse, don't validate (https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/) Javascript falsiness rules (https://www.sitepoint.com/javascript-truthy-falsy/) Transcript: STEPHANIE: Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Stephanie Minn. JOËL: And I'm Joël Quenneville. And together, we're here to share a little bit of what we've learned along the way. STEPHANIE: So, Joël, what's new in your world? JOËL: So I'm on a new project at work. And I'm doing some really interesting work where I'm connecting to a remote database third-party system directly and pulling data from that database into our system, so not via some kind of API. And one thing that's been really kind of tricky to work with are the date formats on this third-party database. STEPHANIE: Is the date being stored in an unexpected format or something like that? JOËL: Yes. So there's a few things that are weird with it. So this is a value that represents a point in time, and it's not stored as a date-time value. Instead, it's stored separately as a date column and a time column. So a little bit of weirdness there. We can work with it, except that the time column isn't actually a time value. It is an integer. STEPHANIE: Oh no. JOËL: Yeah. And if you're thinking, oh, okay, an integer, it's going to be milliseconds since midnight or something like that, which is basically how Postgres' time of day works under the hood, nope, that's not how it works. It's a positional digit thing. So, if you've got the number, you know, 1040, that means 10:40 a.m. STEPHANIE: Oh my gosh. Is this in military time or something like that, at least? JOËL: Yes, it is military time. But it does allow for all these, like, weird invalid values to creep in. Because, in theory, you should never go beyond 2359. But even within the hours that are allowed, let's say, between 1000 and 1100, so between 10:00 and 11:00 a.m., a clock only goes up to 59 minutes. But our base 10 number system goes up to 99, so it's possible to have 1099, which is just an invalid time. STEPHANIE: Right. And I imagine this isn't validated or anything like that. So it is possible to store some impossible time value in this database. JOËL: I don't know for sure if the data is validated or not, but I'm not going to trust that it is. So I have to validate it on my end. STEPHANIE: That's fair. One thing that is striking me is what time is zero? JOËL: So zero in military time or just 24-hour clocks in general is midnight. So 0000, 4 zeros, is midnight. What gets interesting, though, is that because it's an integer, if you put the number, you know, 0001 into the database, it's just going to store it as 1. So I can't even say, oh, the first two digits are the hours, and the second two digits are the minutes. And I'm actually dealing with, I think, seconds and then some fractional part of seconds afterwards. But I can't say that because the number of digits I have is going to be inconsistent. So, first, I need to zero pad. Well, I have to, like, turn it into a string, zero pad the numbers so it's eight characters long. And then, start slicing out pairs of numbers, converting them back into integers, validating them within a range of either 0 to 23 or 0 to 59, and then reconstructing a time object out of that. STEPHANIE: That sounds quite painful. JOËL: It's a journey for sure. STEPHANIE: Do you have any idea why this is the case or why it was created like this originally? JOËL: I'm not sure. I have a couple of theories. I've seen this kind of thing happen before. And I think it's a common way for developers who maybe haven't put a lot of thought into how time works to just sort of think, oh, the human representation. I need something to go in the database. On my digital clock, I have four digits, so why not put four digits in the database? Simple enough. And then don't always realize that there's all these edge cases to think about and that human representations aren't always the best way to store data. STEPHANIE: I like how you just said that that, you know, we as humans have developed systems that are not quite, you know, the same as how a computer would. But what was interesting to me...something you said earlier about time being a fixed point. And that is different from time being a value, right? And so here in this situation, it sounds like we're storing time as a value, but really, it's more of the idea of, like, a point. JOËL: Interesting. What is the difference for you between a point and a value? STEPHANIE: I suppose a value to me...And I think we talked about this a little bit on a previous episode about value objects and also how we stored numbers, like phone numbers and credit card numbers and stuff like that. But a value, like, I might want to do math on. But I don't really want to do math on time. Or, specifically, if I have this idea of a specific point in time, like, that is fixed and not something that I could mutate and expect it to be the same thing that I was trying to express the first time around. JOËL: Oh, that's interesting because I think when it comes to time and specifically points in time, I sometimes do want to do math on them. And so, specifically, I might want to say, what is the time that has elapsed between two points in time? Maybe I have a start time and an end time, and I want to say how much distance is there between the two? If you use this time system where you're storing it as an integer number where the digits have positional values, because there's all those gaps between, you know, 59 and 99 that are not valid, math breaks down. You've broken math by storing it that way. So you can't get an accurate difference by doing math on that, as opposed to if you store it as a counter, which is what databases do under the hood, but you could do manually. If you just wanted to use an integer column, then you can do math because it's just a number of seconds since the beginning of the day. And you can subtract those from each other. And now you have these number of seconds between the two of them. And if you want them in minutes or hours, you divide by six here, 3600, and you get the correct response. STEPHANIE: Yeah, that is really interesting because [chuckles] in this situation, you have the worst of both worlds, it seems like. [laughs] JOËL: The one potential benefit is, I think, it's maybe more human-readable. Although, at that point, I would say if you're not doing math on it and you want something human-readable, you probably don't want an integer. You probably want a string. And maybe you even store it as, like, ISO 8601 time string in the database, or even just hour:minute:second split by a colon or whatever it is but just as a string. Now it's human-readable. You can still sort by it if you go from largest to smallest increment in your format. You can't do math, but then you weren't doing math on it anyway. So that's probably a nice compromise solution. But, ideally, you'd use a native, you know, time of day column or a date-time or something like that. STEPHANIE: For sure. Well, it sounds like something fun to contend with. [laughs] JOËL: One thing that was brought to my attention that I'd never heard about before is that potentially a reason it's stored that way is because of an old data format called EDI—I think it's Electronic Data Interchange—that dates from ages ago, you know, the '60s or '70s, something like that. Before, we had a lot of standards for data; this is how...an emerging standard that came for moving data between systems. And it has a lot of, like, weird things with the way it's set up. But if you're dealing with any sort of older data warehouses or older business systems, they will often exchange. And sometimes, you're going to store data in something that approximates this older EDI format. And, apparently, it has some weirdness around dates where it kind of does something like this. So someone was suggesting, oh, well, if you're interacting with maybe an older, you know, a lot of, like, e-commerce platforms or banking systems, probably airline systems, the kind of things you'd expect to be written in, let's say, COBOL... STEPHANIE: [laughs] JOËL: Have a system that's kind of like this. So maybe that wouldn't be quite as surprising. STEPHANIE: Yeah, that is really interesting. It just sounds like sometimes you're limited by the technology that you're interacting with. And I guess the one plus side is that, in your system, you can make the EDI work for you, hopefully. [laughs] Whereas perhaps if you are talking to some of those older technologies that don't know how else to convert date types and things like that, like, you just kind of have to work with what's available to you. JOËL: Yeah. And that's got me realizing that a lot of these older, archaic systems are still online and very much a part of our software ecosystem and that there's a lot of value in learning some software history so that I'm able to recognize them and sort of work constructively with them when I have to interact with that kind of system. STEPHANIE: Yeah, I really like that mindset. JOËL: So, Stephanie, what's new in your world? STEPHANIE: So, last week, we talked about writing reviews for ourselves and our peers. And one thing that happened in between the last episode and this one is Steph Viccari, former co-host of this podcast, who I've been working with really closely on this project of mine; she was writing a peer review for me. And one thing that she did that I really loved was she sent me a message and asked me a few questions about the direction of the review that I was wanting and what kind of feedback would be helpful for me. And some of the things she asked were, you know, "Is there a skill that you're actively working on? Is there a skill you'd like to start working on?" And, like, what my goals are for the feedback. Like, how can she tailor this feedback to things that would help my progression and what I hope to achieve? And then my favorite question that she asked was, "What else should I know but didn't think to ask?" And I thought that was a really cool way of approaching. You know, she's coming to this, like, wanting to be helpful, but then even still, like, there are things that she knows that I am kind of the expert on in my own career progression, and I really liked that. I think I'd mentioned last week that part of the feedback you want to be giving is, you know, something that will be helpful for that person, and centering them in it, instead of you is just a really awesome way to do that. So I was very appreciative that she asked me those questions. JOËL: That's incredibly thoughtful. I really appreciate that she sent that out to you. What did you respond for the is there something else I should know but didn't know to ask? STEPHANIE: Yeah. I mentioned that more and more, I'm realizing that I am not interested in management. And so what would be really helpful for me was to ground most of the feedback in terms of my, like, technical contributions. And also, that one thing that I'm thinking about a lot is how to be an individual contributor and still have an impact on team health and culture because that is something I care about. And so I wanted to share that with her because if there are things that she can identify in those aspects, that would be really awesome for me. And that can kind of help guide her away from a path that I'm not interested in. JOËL: I think having that kind of self-awareness is really powerful for yourself. But then, when you can leverage that to get better reviews that will help you get even further down the path that you're hoping to go, and, wow, isn't that just, like, a virtuous cycle right there that's just building on itself? STEPHANIE: Yeah, for sure. I think the other thing I wanted to share about what's new in my world that has been just a real boost to my mood is how long the days are right now because it's summer in North America. And yesterday was the summer solstice, and so we had the longest day of the year. The sun didn't set until 8:30 p.m. And I just took the opportunity to be outside. I took a swim in the lake, which was my first swim of the season, which was really special. And my friend had just a nice, little, like, backyard campfire hang out. And we got to roast some marshmallows and just be outside till the sunset. And that was really nice. JOËL: When you say the lake, is that Lake Michigan? STEPHANIE: Yes, I do mean Lake Michigan. [laughs] I forget that some people just don't have a giant lake next to them [laughs] that they refer to as the lake. JOËL: It's practically an inland sea. STEPHANIE: Yes, you can't see the other side of it. So, to me, it kind of feels like an ocean. And yesterday, when I was in the water, I also was thinking that I felt like I was just in a giant bathtub. [chuckles] JOËL: So I'm in New England, and most of the bodies of water here are not called lakes. They're called ponds. STEPHANIE: Really? JOËL: No matter the size. STEPHANIE: Oh. JOËL: I guess lakes is reserved for things like what you have that are absolutely massive, and everything else is a pond. STEPHANIE: That's so funny because I think of ponds as much smaller in scale, like a quaint, little pond. But that's a really fun piece of regional vocabulary. So one interesting thing happened on my client project this week that I wanted to get your input on because I've definitely seen this problem before, and still, it continues to crop up. But I was working on a background job that we were passing a Boolean value into as one of the parameters that we would then, you know, use down the line in determining some logic. And we, you know, made this change, and then we were surprised to find out that it continued to not work the way we expected. So we got some bug reports that we weren't getting into one of the branches of the conditional based on that Boolean value that we were passing in. And we learned, after a little bit of digging, that it turns out that those values are serialized because this job is actually saved in -- JOËL: Oh no. STEPHANIE: [chuckles] Yeah. It inherits from the ActiveRecord, actually, and is saved in our database. And so, in that process, the Boolean value got serialized into a string and then did not get converted [chuckles] back into a Boolean. And so when we do that if variable check, it was always evaluating to true because strings are truthy in Ruby. JOËL: Right. The string false is still truthy. STEPHANIE: A string false is still truthy. And we ended up having to coerce it into a Boolean value to fix our little bug. But it was just one of those things that was really frustrating, you know when you feel really confident that you know what you're doing. You're just writing a conditional statement. And it turns out the language beguiled you. [laughs] JOËL: I've run into similar bugs when I'm reading from environment variables because environment variables are always strings. But it's common that you'll be setting some kind of flag. So when you're setting the environment variable, you're setting something to true or false. But then, when you're reading it, you have to explicitly check if this environment variable double equals the string true, then do the thing. Because if you just check for the value, it will never be false. STEPHANIE: Right. And I kind of hate seeing code like that. I don't know; something about it just rubs me the wrong way because it just seems so strange, I suppose. JOËL: Is it just, like, those edge cases where you specifically have to do some kind of, like, double equals check on a value that feels like it should be a Boolean? Or do you kind of feel a bit weird about the concept of truthiness in general? STEPHANIE: I think the concept of truthiness is very hard to grasp sometimes. And, you know, when you're talking about that edge case where we are setting...we're checking if the string is the string true. That means that everything else is false, right? So, in some ways, I think it's just really confusing because we've expanded the definition of what true and false mean to be anything. JOËL: That's really interesting because now you have to pick. Are you checking against the string true, or are you negatively checking against the string false? And those are not equivalent because, like you said, now you're excluding every other string. So, is the string "Hello, World" put you in the false branch or the true branch? STEPHANIE: Who's to say? [laughs] I think a similar conundrum also occurs when we use predicate matchers in our tests. I think this is a gripe that I've talked about a little bit with others when we're writing tests and especially if we're writing a predicate method, and then that's what we're testing, right? We kind of are expecting a true or false value. And when our test expects something to be truthy rather than explicitly saying that we expect the return value to be true, that is sometimes a bit confusing to me as well because someone could theoretically change this method and just have it return "Hello, World," like you said, as a string, like, anything else. And that would still pass the test. JOËL: And it might even pass your code in most places. STEPHANIE: Right. And I suppose that's okay. Is it okay? I don't know. I'm not sure where I land on this. JOËL: I used to be a kind of hardcore Boolean person. STEPHANIE: [laughs] That's a sentence no one has ever [laughs] said. JOËL: I like my explicit trues and falses. I don't like the ambiguity of saying, like, oh, if person do a thing, it's, like, oh, what is person here? Is this a nil check? Is it explicitly false? Do you just want to know that this person is non-empty? Well, what exactly are you checking? So I like the explicitness of saying, oh, if person dot present, or if person dot empty, or if person dot nil. And I think maybe spending some time in some more strongly typed languages has also kind of pushed me a little bit in that direction, where it's nice to have something that is explicitly either just true or just false. And then you completely eliminate that problem of, like, oh, but what if it's neither true nor false, then what do we do for that branch there? And the answer is your compiler will reject that program or say, "You've written a bad program." And you never reach that point where there's a bug. I've slowly been softening my stance. A fellow thoughtbot colleague has written an article why there is no such thing as a Boolean in Ruby. Everything is just shades of gray and truthiness and falsiness. But from the perspective of a program, there is no such thing as a Boolean. And that really opened my eyes to a different perspective. I don't know that I fully agree, but I'm kind of begrudgingly acknowledging that Mike makes a good point. STEPHANIE: Yes, I read the blog post that he wrote about this exact problem. And I think it's called "Booleans Don't Exist in Ruby." And I think I similarly, like, came away with, like, yeah, I think I get it if I just suspend my disbelief, you know, hard enough. [laughs] But what you were saying about, like, liking the explicitness, right? And liking the lack of ambiguity, right? Because when you start to believe that Booleans don't exist, I think that really messes with your [laughs] head a little bit. And one takeaway that I got from that blog post, kind of like we mentioned earlier, is that there is such thing as false, and then everything else is true. And I guess that's kind of how Ruby operates. JOËL: Sort of, because then you have the problem of nil, which is also falsy. STEPHANIE: That's true, but nil is nothing. [laughs] JOËL: That's one of the classic problems as well when you're trying to do a nil check, or maybe some memoization, or maybe even, say, cache this value, or store this value, or initialize this value if it's not set. And assuming that doing nil is falsy, so you'll do some kind of, like, or equals, or just some kind of expression with an or in it thinking, oh, do this extra work if it's nil because then it will trigger the branch. But that all breaks down if potential for your value to be false because false and nil get treated the same in conditional code. STEPHANIE: Right. I think this could be a whole separate conversation about nil and the idea of nothingness. But I do think that, as Ruby developers, at least in the Ruby world, based on what I've seen, is that we lean on nil in ways that we maybe shouldn't. And we end up having to be very defensive about this idea of nil being falsy. But that's because we aren't necessarily thinking as hard about our return values and what our arguments are that; it ends up causing problems in evaluating truthiness when we're having to check those objects that could be nil. JOËL: In terms of the way we communicate with the readers of our code, and, as a reader, I generally assume that a Ruby method that ends with a question mark will return a true Boolean, either true or false. Is that generally your expectation as well? STEPHANIE: I want to say yes, but I've clearly experienced enough times where that's not the case that, you know, it's like, my ideal world and then reality [laughs] and having to figure out how to hold both of those things. JOËL: It's one of those things that's mostly true. STEPHANIE: I want to believe it because predicate methods and, like, the Ruby Standard Library mostly return Boolean values, at least to my knowledge. And if we all kind of followed that [laughs] pattern, then it would be clear. But I think there's a part of me that these days mostly believes it to be true that I will be getting a Boolean value (And, wow, even as I say this, I realize how confusing [laughs] this is starting to sound.) and that until I'm not, right? Until I'm surprised at some point. JOËL: I think there's two things I expect of predicate methods in Ruby. One is that they will return, like, a hard Boolean, either true or false. The second is that they are purely query methods; they don't do side effects. Neither of those are consistent across the ecosystem. And a classic example of violating that second guideline I have in my mind is the valid question mark method from Rails. And this really surprised me the first time I tripped into this because when you call that on an object, it doesn't just tell you whether or not the object is valid. It actually mutates the underlying object by populating the error messages' hash. So if you have an invalid object and you examine its error messages' hash, it will be empty until you call the valid question mark method. So sometimes, you don't even care about the return value. You're just calling valid to mutate the object so that you can access the underlying hash, which is that's weird code when you call a predicate method but then totally ignore the output. STEPHANIE: Yeah, that is strange because I have definitely seen it where we are calling the valid method to validate, and then we end up using the error messages that are set on that object later. I think that's tough because, in some ways, you do care about whether the object is valid or not. But then also, the error messages are helpful usually and when you're trying to use that method. The point is to validate it so that you can hopefully, like, tell the user or, like, the consumer of your system, like, what's wrong in validation. But it is almost, like, two separate things. JOËL: It is. And sometimes, it's really hard to split those two apart. So I'm not throwing shade at the Rails dev team here. Some of these design decisions are legitimately difficult to make. And what's most useful for the most people the most time is often a compromise. I think you brought up the idea of separating those two things. And I think there's a general principle here called command-query separation. That's, like, the fancy way of talking about what you were saying. STEPHANIE: One thing that I was just thinking about kind of when we initially picked off this conversation was the idea of how things outside the Ruby ecosystem or the Ruby world interact with what we're returning in terms of Boolean values. And so when I mentioned the object being serialized because of, you know, our database and, like, background job system, that's an entity that's figuring out what to do with the things that we are returning from Ruby. And similarly, when you're talking about environment variables, it's like, our computer system talking to now our language and those things being a bit different. Because when we, like, suspend our disbelief about what is truthy or falsy in Ruby, at least we're doing it in, like, the world of Ruby. And as soon as we have to interact with something else, like, maybe that's when things can get a little hairy because there's different ideas about truthiness there. And so I'm kind of also thinking about what we return in APIs and maybe, like, that being an area where some explicitness is more required. JOËL: Whenever I'm consuming third-party data, I'm a big fan of having some kind of transformation or parse step. This is inspired in part by the "Parse, Don't Validate" article, which I'll link in the show notes. So, if I'm reading data from a third-party API and I want it to be a Boolean, then maybe I should do the transformation myself. So maybe I check literally, is it the string true or the string false, and anything else gets rejected? Maybe I have...and maybe I'm a little bit more permissive, where I also accept capital T or capital F, and I have, like, some rules for transforming that. But the important thing is I have an explicit conversion step and reject any bad output. And so for something like an environment variable, maybe that would look like looking for true or false and raising if anything else is there. So that we try to boot the app, and it immediately crashes because, hey, we've got some, like, undefined, like, bad configuration that we're trying to load the app with. Don't even try to keep running. Hard crash immediately. Fix it, and then come back. STEPHANIE: Yeah, I like that a lot because the way we ended up fixing this issue with the background job that I mentioned was just coercing our string value into a Ruby Boolean in the job that we were then, like, running the conditional in. But really, what we should have done is have fixed that at a higher level and where we parse and deserialize, like, the values we're getting from the job to prevent this kind of in the future because right now, someone can do this again, and that's a real bummer. JOËL: I always love those deeper conversations that happen after you've had a bug that are like, how do we prevent this from happening again? Because sometimes that's where you have the deepest learnings or the most interesting insights or, you know, ideas for Bike Shed episodes. I'm really curious to contrast JavaScript's approach to truthiness to Ruby's because even though they both use the same idea, they kind of go about it differently. STEPHANIE: Tell me more. JOËL: So, in Ruby, an empty array and an empty string are truthy. JavaScript decided that empty things are falsy. And I forget...there's a whole table that shows the things that are truthy and falsy in JavaScript. I want to say zero is falsy in JavaScript but don't quote me on that, which can also lead to some interesting edge cases you have to think about. STEPHANIE: Okay, yes. This is coming back to me now. I think depending on what, you know, ecosystem or language or world I'm in, I have to just only be able to think about what is true in this world [laughs] and then do that context switching when I am working in something else. But yeah, that is a really interesting idea. Someone decided [laughs] that this was their idea of true or false. JOËL: I'm curious if you have a preference for sort of JavaScript's approach to falsiness where a lot more types of values are falsy versus Ruby, which said pretty much only nil and false are falsy. Everything else is truthy. STEPHANIE: Hmm, that is an interesting question. JOËL: Because in Ruby then or, I guess, in Rails, we end up with the present predicate method that is specifically checking for not only nil and false but also for empty array, empty string, those kinds of things. So, if you find yourself writing a lot of present matchers in your code, you're kind of leaning on something that's closer to JavaScript's definition of falsiness than Ruby's. But maybe you're making it more explicit. STEPHANIE: Right. In JavaScript, I see a lot of double bangs in lieu of those predicate methods. But I suppose by nature of having to write those predicate methods in Ruby, we're, like, really wanting something else, I think. And maybe...I guess it is just a question of explicitness like you're saying, and which I prefer. Is it that I need to be explicit to convey the idea that I want, or is it nice that the language has just been encoded that way for me? JOËL: Or maybe when you write conditionals, if you find yourself doing a lot of presence checks, do you find that you typically are trying to branch on if not null, not false, not empty more frequently than just if not null, not false? Because that's kind of the difference between Ruby's model and JavaScript's model. STEPHANIE: Hmm, the way you posed that question is interesting because it makes me think that sometimes it's quite defensive because we have to check for all these possible return values. We are unsure of what we are getting back. And so this is kind of, like, a catch-all for things that we aren't really sure about. JOËL: Yeah, I mean, that's the fun of dynamic programming languages. You never know exactly what you're going to get as long as things respond to certain methods. You really lean into the duck typing. And I think that's Mike's argument in his article that "Booleans Don't Exist" in that as long as something is responding to methods that you care about, it doesn't matter if you're dealing with a true Boolean or some kind of other value. STEPHANIE: Right. So I suppose the ideas of truthiness then are a little bit more dependent on how people are using the language though it seems like a chicken-and-egg situation to me. [laughs] JOËL: It is really interesting to me in terms of maybe thinking about use cases in my own code if I'm having to...if I'm writing code that leans on truthiness where I can say just, you know if user. But then knowing that, oh, that doesn't account for, like, an empty value. Do I then also need to add an extra check for emptiness? And maybe if I'm in a Rails project, I would reach for that present matcher where I wouldn't have to do that in JavaScript because I can just say, if user, and that already automatically checks for presence. So I'm kind of wondering now in my mind, like, which default would fit my use cases more? Or, if I go back to an older version of myself, I will say I don't want any of these defaults. They're all too ambiguous. I'm going to put explicitly if user dot nil question mark, if that's the thing that I'm checking for, or if user dot empty question mark because I want my reader to know what condition I'm checking. STEPHANIE: Yeah, that is interesting, this idea of, like, which mode do you find yourself needing to use more and if that is accommodated for you because that's just the more common, like, use case or problem. I think that's something that I will be thinking about the next time I write a conditional [laughs] because, like I was saying earlier, I think I end up just leaning on what someone else has decided for me in terms of truthiness and not so much how I would like it to work for me. JOËL: And sometimes we don't want to fight the language too much, you know if I'm writing Elm, that everything is hard Booleans. And I know I'm never going to get a nil in a place where I'd expect true or false because the compiler would prevent that from happening. I know that I'm not going to get an empty value, potentially. There's ways you can do things with a type system where you can explicitly say no empty values are even allowed at this point. And if you do allow them, then the type system will say, "Hey, you forgot to check for the empty case. Bad program. I'm rejecting that." And then you have to write that explicit branch for, oh, if empty versus if present. So I really appreciate that style of programming. But then, when you're in a language like Ruby where you're not dealing with explicit types on purpose, how do you shift that mindset so that you don't need to know the type of the value that you're dealing with? You only want to say, hey, in this context, here's the minimal interface that I want it to conform to. And maybe it's just the truthy or falsiness interface, and everything beyond that is not relevant. STEPHANIE: I think it's kind of wild to me that this idea of a binary that theoretically seems very clear turns out is actually quite confusing, ambiguous, philosophical, even. [chuckles] JOËL: Yeah. It's definitely...you can get into some deep, philosophical questions there, language design as well. One aspect, though, that I'm really curious about your thoughts is bringing new people in who are learning a language. It's really common for people who are learning a language for the first time, learning to code for the first time to write code that you and I would immediately know, like, that's not going to work. You can't add a Boolean and a number. You're just learning to code. You've never done that before. You don't know. And then how the language reacts to that kind of thing can help guide that experience. So, do you think that truthiness maybe makes things more confusing for newcomers? Or, maybe on the other side, it helps to smooth that learning curve because you don't have to be like, oh, wait, I have a user here. I can't put that in a condition because that's not a strict true or false. I'm going to coerce it, or I've got to find a predicate method or something. You can just be like, no, put it in. The interpreter will figure it out for you. STEPHANIE: Wow. That's a great question. I'm trying to put myself in the beginner's mindset a little bit and think about what it's like to just try something and the magic of it working. Because, like you said, the interpreter does it for you, or whatever, and something happens, and you're like, wow, like, that was really cool. And I didn't have to know all of the ins and outs of the types of things I was working with. That can be really helpful in just getting them, like, started and getting them just, like, on the ground writing code. And having that feeling of satisfaction that, like, that they didn't have to, you know, have to learn all these things that can be really scary to make their program work. But I do think it also kind of bites them later once they really realize [laughs] what is going on and the minute that they get that, like, unexpected behavior, right? Like, that becomes a time when you do have to figure out what might be going on under the hood. So two sides of the same coin. JOËL: What you're saying there about, like, maybe smoothing that initial curve but then it biting them later got me thinking. You know how we have the concept of technical debt where we write code in a way that's maybe not quite as clean today so we can move faster but that then later on we have to pay it back? And I almost wonder if what we have here is almost like a pedagogical debt where it's going to cost us a month from now, but today it helps us move faster and actually kind of get that momentum going. STEPHANIE: Pedagogical debt. I like that. I think you've coined a new term. Because I really relate to that where you learn just enough to do the thing now. But, you know, it's probably not, like, the right way or, like, the most informed—I think most informed is probably how I would best describe it—way of doing it. And later, you, yeah, just have to invest a little more into it. And I think that's okay. I think sometimes I do tend to, like, beat myself up over something down the line when I have to deal with some piece of less-than-ideal code that I'd written earlier. Like, I think that, oh, I could have avoided this if only I knew. But the whole point is that I didn't know. [laughs] And, like, that's okay, like, maybe I didn't need to know at the time. JOËL: Yeah, and code that's never shipped is of zero value. So having something that you could ship is better than having something perfect that you didn't ship. STEPHANIE: On that note, shall we wrap up? JOËL: Let's wrap up. STEPHANIE: Show notes for this episode can be found at bikeshed.fm. JOËL: This show has been produced and edited by Mandy Moore. STEPHANIE: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review in iTunes. It really helps other folks find the show. JOËL: If you have any feedback for this or any of our other episodes, you can reach us @_bikeshed, or you can reach me @joelquen on Twitter. STEPHANIE: Or reach both of us at hosts@bikeshed.fm via email. JOËL: Thanks so much for listening to The Bike Shed, and we'll see you next week. ALL: Byeeeeeeee!!!!!! ANNOUNCER: This podcast is brought to you by thoughtbot, your expert strategy, design, development, and product management partner. We bring digital products from idea to success and teach you how because we care. Learn more at thoughtbot.com.

world simple north america new england exist fix api iso apis javascript rails validate lake michigan elm mandy moore 3f cobol postgres boolean parse quenneville truthiness activerecord stephanie yeah stephanie oh stephanie for stephanie right

Active Record Validations

reading guides aji validations activerecord

Play Episode Listen Later Jun 27, 2023 28:02

For this week's episode, Aji & Mina read "Active Record Validations" and discuss their most-used validation helpers, how they interpreted the validates_associated section of the Guides, and when not to over-validate.Reading for this episode: "Active Record Validations"Time for a Boolean gemWhat's in a name (validation)?Creating More Inclusive and Culturally Sensitive FormsReading for episode 8: Active Record Callbacks

Active Record Migrations (chapters 5-9)

reading chapters rails migrations aji activerecord

Play Episode Listen Later Jun 13, 2023 19:29

For this week's episode, Aji & Mina discuss chapters 5 through 9 of "Active Record Migrations". They debate migration file management, and touch on the relationship between end users and application data, and learning complex concepts within Rails' convention over configuration framework.Reading for this episode: "Active Record Migrations", chapters 5-9thoughtbot's AWS Platform Guide (Flightdeck)Reading for episode 7: "Active Record Validations"

Active Record Migrations (chapters 1-4)

reading chapters migrations aji activerecord

Play Episode Listen Later May 30, 2023 27:21

For this week's episode, Aji & Mina read chapters 1 through 4 in “Active Record Migrations” and discuss command line generator options, database-level comments, and reverting migrations.Reading for this episode: "Active Record Migrations", chapters 1-4Reading for episode 6: "Active Record Migrations", chapters 5-9

The ORMazing show

JS Party

Play Episode Listen Later May 19, 2023 72:16 Transcription Available

Nick & KBall sit down with the brilliant Stephen Haberman to discuss all things ORMs!

web programming animation robotics iot javascript html css node backend frontend typescript changelog orms activerecord kevin ball nick nisi

Active Record Basics

japan reading basics chapters orms activerecord

Play Episode Listen Later May 17, 2023 29:04

For this week's episode, Mina is in Japan, the couple read "Active Record Basics" and discuss ORMs, naming conventions and have a special request to you, the listeners.Reading for this episode: "Active Record Basics"Patterns of Enterprise Application Architecture by Martin FowlerMats'z Home ConferenceReading for Episode 5: Chapters 1 - 4 in "Active Record Migrations"

382: Domain-Specific Languages

Play Episode Listen Later May 2, 2023 36:09

Joël has been integrating a third-party platform into a testing pipeline...and it has not been going well. Because it's not something she usually keeps up-to-date with, Stephanie is excited to learn about more of the open-source side of things in Ruby, what's new in the Ruby tooling world, and what folks are thinking about regarding the future of the language. Today's topic is inspired by an internal thoughtbot Slack thread about writing a custom matcher for Rspec. Stephanie and Joël contrast DSLs vs. Object APIs and also talk about: CanCanCan vs Pundit RSpec DSL When is a DSL helpful? Why not use both DSLs & Object APIs? Extensibility When does a DSL become a framework? This episode is brought to you by Airbrake (https://airbrake.io/?utm_campaign=Q3_2022%3A%20Bike%20Shed%20Podcast%20Ad&utm_source=Bike%20Shed&utm_medium=website). Visit Frictionless error monitoring and performance insight for your app stack. RubyKaigi 2023 (https://rubykaigi.org/2023/) Mystified by RSpec's DSL? by Jason Swett (https://www.codewithjason.com/mystified-rspecs-dsl-parentheses-can-add-clarity/) Building Custom RSpec Matchers with Regular Objects (https://thoughtbot.com/blog/building-custom-rspec-matchers-with-regular-objects) FactoryBot (https://github.com/thoughtbot/factory_bot) Writing a Domain-Specific Language in Ruby by Gabe Berke-Williams (https://thoughtbot.com/blog/writing-a-domain-specific-language-in-ruby) Capybara (https://teamcapybara.github.io/capybara/) Acceptance Tests at a Single Level of Abstraction (https://thoughtbot.com/blog/acceptance-tests-at-a-single-level-of-abstraction) CanCanCan (https://github.com/CanCanCommunity/cancancan) Pundit (https://www.capvidia.com/products/pundit) Discrete Math and Functional Programming (https://www.amazon.com/Discrete-Mathematics-Functional-Programming-VanDrunen/dp/1590282604) Transcript: STEPHANIE: Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Stephanie Minn. JOËL: And I'm Joël Quenneville. And together, we're here to share a little bit of what we've learned along the way. STEPHANIE: So, Joël, what's new in your world? JOËL: I've been integrating a third-party platform into our testing pipeline for my client. It has not been going well. We've been struggling a little bit, mostly just because tests just kind of crash. Our testing pipeline is pretty complex. It's a lot of one script, some environment variables, does a few things, shells out to another script, which is in a different language. Does a few more things, shells out to another script, maybe calls out to rake, calls out to a shell script. There are four or five of these in a chain, and it's a bit of a mess. Somewhere along in there, something is not compatible with this third-party service that we're trying to integrate with. I was pairing this week with a colleague. And we were able to reproduce a situation where we were able to get a failure under some conditions and a success under other conditions. So these are basically, if we run the whole chain of scripts that call each other from the beginning, we know we get a failure. And if we skipped entirely the chain of scripts that set up things and then just manually try to invoke a third-party service, that works. And so now we know that there's something in between that's incompatible, and now it's just about narrowing things down. There are a few different approaches we could take. We could try to sort of work our way forward. We know a known point where it breaks and then just try to start the chain one step further and see where it fails. We could try to get fancy and do a binary search, like split it in half and then half and half again. We ended up doing it the other way, where we started at the end. We had our known good point and then just stepping one step back and saying, okay, now we introduce the last script in the chain. Does that work? Okay, that pass is great. Let's go one step further; two scripts up in the chain. And at some point, we find, okay, here's the one script that fails. Now, what is it within this script? And it was a really fun debugging session where we were just narrowing things down until we found the source of the bug. STEPHANIE: Wow, that sounds pretty complicated. It just seems like there are so many layers going on. And it was really challenging to pinpoint where the source of the issue was. JOËL: Definitely. I think all the layers made it really complicated. But having a process that we could follow and then kind of narrowing it down made it almost mechanical to figure out where the bug was once we got to a point where we had a known good point and a known bad point. STEPHANIE: Yeah, that makes sense. Kind of sounds like if you are using git bisect or something like that to narrow down the scope of where the issue could be. I'm curious because this is like a bunch of shell scripts and rake tasks or commands or whatever. What would have made this debugging process easier? JOËL: I think having fewer scripts in this chain. STEPHANIE: [laughs] That's fair. JOËL: We don't need so many scripts that call out to each other in different languages trying to share data via environment variables. So we've got a bit of a Rube Goldberg machine, and we're trying to patch in yet another piece in there. STEPHANIE: Yeah, that's really tough. I was curious if there was, I don't know, any logging or any other clues that you were getting along the way because I know from experience how painful it is to debug that kind of code. JOËL: It's interesting because I feel like normally logging is something that's really useful. In this particular case, we run into an exception at some point. So it's more of under what conditions does the exception happen? The important thing was to find that there is a point where it breaks, and there's a point where it doesn't, and realizing that if we ran some of these commands just directly without going through the whole pipeline, that things did work and that we were not triggering that exception. So all of a sudden, now that tells us, okay, something in our pipeline is wrong. And then we can just start narrowing things down. So yeah, adventures in debugging. Sometimes it's really frustrating, but then when you have a good process, and you find the bug, it's incredibly satisfying. STEPHANIE: I like that you used a process that can be applied to many different problems, in this particular case, debugging a testing pipeline. Maybe not something that we do every day, but certainly, it comes up, and now we have tools to address those kinds of issues as well. JOËL: So my week has been up and down with all of this debugging. What's been new in your world? STEPHANIE: I've been doing some travel planning because I'm going to RubyKaigi in Japan. JOËL: Whoa. STEPHANIE: This is actually going to be my first international conference, so I'm really looking forward to that. I just have never been compelled to travel abroad to go to a tech conference. But I'm really looking forward to going to RubyKaigi because now I've been to the U.S.-based conferences a few times. And I'm excited to see how things are different at an international conference and specifically a RubyKaigi because, obviously, there's a lot of really cool Ruby work happening over there in Japan. So I'm excited to learn about more of the open-source side of things of Ruby, what's new in the Ruby tooling world, and just what folks are thinking about in terms of the future of the language. That's not something I normally keep super up-to-date on. But I'm excited to be around people who do think and talk about these things a lot and maybe get some new insights into my own work. JOËL: Do you find that you tend to keep up more with some of the frameworks like Rails rather than the underlying language itself? STEPHANIE: Yeah, that's a good question. I do think because the framework changes a little more frequently, new releases are kind of more applicable to the work that I'm doing. Whereas language updates or upgrades are a little bit less top of mind for me because the point is that it doesn't have to change [laughs] all that much, and we can continue to work with things as expected and not be disrupted. So it is definitely like a whole new world for me, but I'm really looking forward to it. I think it will be really interesting and just kind of a whole other space to explore that I haven't really because I've usually been focused on more of the web development and industry work side of things. JOËL: What's a Ruby feature that either is coming out in the future or that came out in the last couple of releases that got you really excited? STEPHANIE: I think the conversation about typing in Ruby is something that has been on my radar but has also been ebbing and flowing over time. And I did see a few talks at RubyKaigi this year that are going to talk about how to introduce gradual typing in Ruby. And now that it has been out for a little bit and people have been using it, how people are feeling about it, pros and cons, and kind of where they're going to take it or not take it from there. JOËL: Have you done much TypeScript? STEPHANIE: I have been working more in TypeScript recently but did spend most of my front-end work coding days in JavaScript. And so that transition itself was pretty challenging for me where I suddenly felt a language that I did know pretty well. I was having to be in that...in somewhat of a beginner's mindset again. Even just reading the code itself, there were just so many new things to be looking at in terms of the syntax. And it was a difficult but ultimately pretty rewarding experience because the way I thought about JavaScript afterwards was much more refined, I think. JOËL: Types definitely, I think, change the way you think about code; at least, that's been my experience. STEPHANIE: Yeah, absolutely. I haven't gotten the pleasure to work with types in Ruby just yet, but I've just heard different experiences. And I'm excited to see what experts have to say about it. JOËL: That's the fun of going to a conference. STEPHANIE: Absolutely. So yeah, if any listeners are also headed to RubyKaigi, yeah, look out for me. JOËL: I was recently having a conversation with someone about the fact that a lot of languages provide ways to sort of embed many languages within them. So the Lisp family of languages are really big into macros and metaprogramming. Some other languages are big into giving you the ability to build your own ASTs or have really strong parsing capabilities so that you can produce your own, again, mini-language. And Ruby does this as well. It's pretty popular among the Ruby community to build DSLs, Domain-Specific Languages using some of Ruby's built-in abilities. But it seems to be a sort of universal need or at the very least a universal desire among programmers. Have you ever found yourself as a code author wanting to embed a sort of smaller language within your application? STEPHANIE: I don't think I have, to be honest. It's a very interesting question. Because I think the motivation to build your own mini-language using Ruby would have to be you'd have to have a really good reason for it, and in my experience, I haven't quite encountered that yet. Because, yeah, it seems like a lot of upfront work, a lot of overhead to introduce something like that, especially if it's not necessarily either a really, really particular domain that others might find a use for, or it just doesn't end up seeming worthwhile if I can just write regular, old Ruby code. JOËL: I think you're not alone. I think the Ruby community has been kind of a bit of a pendulum here where several years ago, everything that could be made into a DSL was. Now the pendulum kind of has been swinging the other way. And we see DSLs, but they're not quite as frequent. For those who maybe have not experienced a DSL or aren't quite familiar with the concept, how would you describe the idea? STEPHANIE: I think I would describe domain-specific languages as a bit of a mini-language that is created for a very particular problem space in mind to make development for that domain easier. Oftentimes, I've also kind of seen people describe the benefit of DSLs as being able to read that language as if it were plain English. And so, in my head, I have kind of, at least in the Ruby world, right? We see that a lot in different gems. RSpec, for example, has its own internal DSL, and many people really enjoy it because it took the domain of testing. And the way you write it kind of is how you might read or understand it in English. And so it's a bit easier to talk about what you're expecting in your tests. JOËL: Yeah, it's so high-level and minimal and domain-specific that it almost stops feeling like it's a programming language and can almost feel like it's a high-level configuration for this very particular domain, sometimes even to the point where the idea is that a non-programmer could read it and understand what's going on. STEPHANIE: I think RSpec is actually one of the first Ruby DSLs that you might encounter when you're learning Ruby for the first time. And I've definitely seen developers who are new to Ruby, you know, they're writing code, and they're like, okay, I'm ready to write a test now. And the project uses RSpec because that's what most of us use in our Rails applications. And then they see, like you said, almost a configuration language, and they are really confused. They're not really sure what they're reading. They struggle with the syntax a lot. And it ends up being a point of frustration when they're first starting out if they're not just copying and pasting other existing RSpec tests. I'm curious if you've seen that before. JOËL: I've definitely seen that. And it's a little bit ironic because oftentimes, an argument for DSL is that it makes things simpler that you don't even have to know Ruby; you can just write it. It's simpler. It's easier to write. It's easier to understand. And to a certain extent, maybe that's true. But for someone who does know Ruby and doesn't know your particular little domain language, now they're encountering something that they don't know. And they're having to learn it, and they're having to struggle with it. And it might behave a little bit weirdly compared to how Ruby normally works. And so sometimes it doesn't make it easier for adoption. But it does look really good in a README. STEPHANIE: That's totally fair. I think the other thing that's interesting about RSpec is that a lot of it is really just stylistic. I actually read a blog post by Jason Swett and the headline of it was "Mystified by RSpec's DSL? Some parentheses can add clarity." And he basically goes on to tell us that really RSpec is just leaning on some of Ruby's syntactic sugar of omitting parentheses for method calls. And if you just add the parentheses back in your it blocks or your describes, it can read a lot more like regular Ruby. And you might have a better time understanding what's going on when you realize that we're just passing our descriptors as arguments along with some blocks. JOËL: That's ironic given that oftentimes, the goal of these is to make it look like not Ruby. STEPHANIE: I agree; it is ironic. [laughs] MID-ROLL AD: Debugging errors can be a developer's worst nightmare...but it doesn't have to be. Airbrake is an award-winning error monitoring, performance, and deployment tracking tool created by developers for developers that can actually help cut your debugging time in half. So why do developers love Airbrake? It has all of the information that web developers need to monitor their application - including error management, performance insights, and deploy tracking! Airbrake's debugging tool catches all of your project errors, intelligently groups them, and points you to the issue in the code so you can quickly fix the bug before customers are impacted. In addition to stellar error monitoring, Airbrake's lightweight APM helps developers to track the performance and availability of their application through metrics like HTTP requests, response times, error occurrences, and user satisfaction. Finally, Airbrake Deploy Tracking helps developers track trends, fix bad deploys, and improve code quality. Since 2008, Airbrake has been a staple in the Ruby community and has grown to cover all major programming languages. Airbrake seamlessly integrates with your favorite apps to include modern features like single sign-on and SDK-based installation. From testing to production, Airbrake notifiers have your back. Your time is valuable, so why waste it combing through logs, waiting for user reports, or retrofitting other tools to monitor your application? You literally have nothing to lose. Head on over to airbrake.io/try/bikeshed to create your FREE developer account today! JOËL: I think another drawback that I've seen with DSLs is that they oftentimes are more limited in their capabilities. So if the designer of the gem didn't explicitly think of your use case, then oftentimes, it can be really hard to extend or to support edge cases that are not specifically designed for that language in the way that plain Ruby is often much more flexible. STEPHANIE: Yeah, that's really interesting because when a gem does have some kind of DSL, a lot of effort probably went into making that the main interface that you would work with or you would use. And when that isn't working for your use case, the design of the underlying objects may or may not be helpful for the changes that you want to make. JOËL: I think it's interesting that you mentioned the underlying objects because those are often sort of not meant for public consumption when you're building a gem that's DSL forward. I think, in many cases, my ideal gem would make those underlying objects the primary interface and then maybe offer DSL as a kind of nice-to-have layer on top for those situations that maybe aren't as complex where writing things in the domain language might actually be quite nice. But keeping those underlying objects as the interface, it's nice to use and well-documented for the majority of people. STEPHANIE: Yeah, I like that too because then you can get the best of both worlds. So speaking of trying to make a DSL work for you, have you ever experienced having to kind of work around the DSL to get the functionality you were hoping to achieve? JOËL: So I think we're talking about the idea of having both a DSL and the underlying objects. And RSpec is a great example of this with their custom matchers. RSpec itself is a DSL, but then they also offer a DSL to allow you to create custom matchers. And it's not super well documented. I always forget how to define them, and so I oftentimes don't bother. It's just kind of too much of a pain for something that doesn't always provide that much value. But if it were easy, I would probably do it more. Eventually, I realized that you could use just regular Ruby objects as custom matchers. And they just seemed to respond to certain methods, just regular old objects and polymorphism. And all of a sudden, now I'm back into all of the tools and mechanisms that I am familiar with, like the back of my hand. I can write objects all day. I can TDD them. I can apply any patterns that I want to if I'm doing something really complicated. I can extract helpers. All of that works really well with the knowledge that I already have without having to sink a lot of time into trying to learn the built-in DSL. So, for the most part, now, when I define custom matchers, I'll often jump directly to creating a regular object and making it conform to the matcher interface rather than relying on the DSL for that. So once we go back to the test, now we're back in DSL land. Now we're no longer talking in terms of objects so much. We'll have some nice methods and they will all kind of read like English. So to pull a recent example that I worked on, I might say something like expect this policy object method to conform to this truth table. STEPHANIE: That's a really interesting example. It actually kind of sounds like it hits the sweet spot of what you were describing earlier in the sense that it has a really nice DSL, but also, you can create your own objects, and that has an interface that you can implement. And yes, have your cake and eat it too. [laughs] But the idea that then you're kind of converting it back to the DSL because that is just what we know, and it has become so normalized. I was talking earlier about okay; when is a DSL worthwhile? When is the use case a good reason to implement it? And especially for gems that I think that are really popular that we as a Ruby community have collectively used most of the time on our projects because we have oftentimes a lot of the same problems that we're solving. It seems like this has become its own shared language, right? JOËL: Yeah, there are definitely some DSLs that we all end up learning because they're just so prominent in the Ruby community, even Rails itself ships with several built-in DSLs. STEPHANIE: Yeah, absolutely. FactoryBot is another one, too. It is a gem by thoughtbot. And actually, in preparation to talk about DSLs with you today, I scoured our blog and found a really great blog post, "Writing a Domain-Specific Language in Ruby" by Gabe Berke-Williams. And it is basically like, here's how to write something like FactoryBot and creating your own little mini Ruby DSL for something that would be very similar to what FactoryBot does for fixtures. JOËL: That's a great resource, and we'll make sure to link that in the show notes. We've been talking about some of the limitations of DSLs or some aspects of them maybe that we personally don't like. What are maybe examples of DSLs that you do enjoy working with? STEPHANIE: Yeah, I have an example for this one. I really enjoy using Capybara's DSL for acceptance testing. I did have to go down the route of writing some custom selectors for...I just had some HTML elements within kind of a complicated table and was trying to figure out how to write some selectors so that I could write the test as if it were in, you know, quote, unquote, "plain English" like, within this table, expect some value. And that was an interesting journey. But I think that it really helped me have a better understanding of accessibility of just the underlying building blocks of the page that I was working with. And, yeah, I really appreciate being able to read those tests from a user perspective and kind of know exactly what they're doing when they're interacting with this virtual browser without having to run it in headful mode and see it for myself. JOËL: It's always great when a DSL can give you that experience of abstracting enough to where it makes the code delightful to work with while also not having too high a cost to learn or being too restrictive in what it allows you to do. Would you make a difference between something that's a DSL versus maybe just code that's written at a higher level of abstraction? So maybe to get back to your example with Capybara, it's really nice to have these nice custom matchers and all of these things to work with HTML pages. If I'm writing, let's say, a helper method at the bottom of a test, I don't think that feels quite like it's a DSL yet. But it's definitely a higher level than specifying CSS selectors. So would you make a difference between those two things? STEPHANIE: That's a good question. I think it's one of those you know it when you see it kind of questions because it just depends on the amount of abstraction, like you mentioned, and maybe even metaprogramming. That takes something from the core language to morph into what you could qualify as a separate language. What do you think about this? JOËL: Yeah, part of me almost wonders if this exists kind of on a continuum, and the boundary might be a little bit fuzzy. I think there might be some other qualifications that come with it as well. Even though DSLs are typically higher-level helpers, it's usually more than just that. There are also sort of slightly different semantics in the way that you would tend to use them to the point where while they may be just Ruby methods, we don't use them like Ruby methods, and even to the point that we don't think of them as Ruby methods. To go back to that article you mentioned from Jason, where just reminding people, hey, if you put params on this, all of a sudden, it helps you remember, oh, it's just a Ruby method instead of being like, oh, this is a language keyword or something. STEPHANIE: Yeah, I wonder if there's also something to the idea of domain specificity where it should be self-service within the domain that you're working. And then it has limitations once you are trying to do something separate from the domain. JOËL: Right, it's an element of focus to this. And I think it's probably also a language is not just one helper; it's a collection typically. So it's probably a series of high-level helpers, potentially. They might not be methods, even though that is ultimately one of the primary interfaces we use to run code in Ruby. So it's a collection of methods that are high-level, but the collection itself is focused. And oftentimes, they're meant to be used in a way where it's not just a traditional method call. STEPHANIE: Right. There's some amount of you bringing to the table your own use case in how you use those methods. JOËL: Yeah, so it might be mimicking a language keyword. It might be mimicking the idea of a configuration. We see that a little bit with ActiveRecord and some of the, let's say, the association and validation APIs. Those kind of feel like, yes, they're embedded in a class, but they feel like either keywords or even just straight-up configuration where you set key-value pairs of things to configure how a particular class is going to work. STEPHANIE: Yeah, that's true for a lot of things in Rails, too, if we're talking about routes and initializers as well. JOËL: So I've complained about some things I don't like about DSLs. I really like the routing DSL in Rails. STEPHANIE: Why is that? JOËL: I think it's very compact and readable. And that's an element that's really nice about DSLs is that it can make things feel very readable and, oftentimes, we read code more often than we write it. And routes have...I was going to say fewer edge cases, but I have seen some really gnarly route files that are pretty awful to work with, especially if you're mostly writing RESTful controllers, and I would recommend that people do. It's really nice to just be able to skim through a route file and be like, oh, these are the resources in my app and the actions I can do on each resource. And here are the ones that are nested. STEPHANIE: Yeah, it almost sounds like a DSL can provide guardrails towards the recommended way of tackling that particular domain. The routes DSL really discourages you from doing anything too complicated because they are encouraging you to follow the Rails convention. And so I think that goes back to the specificity piece of if you've written a DSL, it's because you've thought very deeply about this particular domain and how common problems show up and how you would want people to be empowered by the language rather than inhibited by it. JOËL: I think, thinking more about that, the word that comes to mind is declarative. When you read code that's written with DSLs, typically, it's very declarative. It's more just describing a thing as opposed to either procedural, a series of commands to do, or even OO, where you're composing objects and sending messages to each other. And so problems that lend themselves to being implemented through more descriptive and declarative approaches probably are really good candidates for a DSL. STEPHANIE: Yeah, I like that a lot because when we talk about domains, we're not necessarily talking about a business domain, which is kind of the other way that some folks think about that word. We're talking about a problem space. And the idea of the language being declarative to describe the problem space makes a lot of sense to me because you want it to be flexible enough for different use cases but all within the idea of testing or browser navigation or whatever. JOËL: Yeah. I feel like there's a lot of... there are probably more problems that can be converted to declarative solutions than might initially kind of strike you. Sometimes the problem isn't quite as bounded. And so when you want customizations that are not supported by your DSL, then it kind of falls apart. So I think a classic situation that might feel like something declarative is authorization. Authorization are a series of rules for who can access what, and it would seem like this is a great case for a DSL. Wouldn't it be great to have just one file you can just kind of skim, and we can just see all of the access rules? Access rules that are basically asking to be done declaratively. And we have gems like that. The original CanCan gem and then the successor CanCanCan are trying to follow that approach. Have you used either of those gems? STEPHANIE: I did use the CanCanCan gem a while ago. JOËL: What was your experience with that style of authorization? STEPHANIE: It has been a while but I do remember having to check that original file of like all the different authorizations kind of repeatedly coming back to it to remember, okay, for this rule, what should be allowed to happen here? JOËL: So I think that's definitely one of the benefits is that you have all of your rules stored in one place, and you can kind of scan through the list. My experience, though, is that in practice, it often kind of balloons up and has all of these edge cases in it. And in some earlier versions, I don't know if that's still a problem today, it could even be difficult to accomplish certain things. If you're going to say that access to this particular object depends not on properties of that object itself but on some custom join or association or something like that, that could be really clunky to do or sometimes impossible depending on how esoteric it is or if there's some really complex custom logic to do. And once you're doing something like that, you don't really want to have that logic in your...in this case, it would be the abilities file but inside because that's not really something you express via the DSL anymore. Now you're dropping into OO or procedural world. STEPHANIE: Right. It seems a bit far removed from where we do actually care about the different abilities, especially for one-off cases. JOËL: That is interesting because I feel like there's a bit of a read-versus write-situation happening there as well. It's particularly nice to have, I think, everything in one abilities file for reading and for auditing. I've definitely been in code where there's like three or four ways to authorize, and they're all being used inconsistently, and that's not nice at all. On the other hand, it can be hard with DSL sometimes to customize or to go beyond the rules that are built in. In the case of authorization, you've effectively built a little mini-rules engine. And if you don't have a good way for people to add custom rules without just embedding procedural code into your abilities file, it's going to quickly get out of hand. STEPHANIE: Yeah, that makes sense. On the topic of authorization, you did mention an example earlier when you were writing a policy object. JOËL: I've generally found that that's been my go-to pattern for authorization. I enjoy the Pundit gem that provides some kind of light scaffolding around working with policy objects, but it's a general pattern, and you can absolutely write your own. You don't need a gem for that. Now we're definitely not in the DSL world. We're not doing this declaratively. We're leaning very heavily on OO and saying we're just going to create objects. They talk to each other. They can do anything that any Ruby object can do and as simple or as complex as they need to be. So you have the full power of Ruby and all the patterns that you're used to using. The downside is it is a little bit harder to read and to kind of just audit what's happening in terms of permission because there's no high-level overview anymore. Now you've just got to look through a bunch of classes. So maybe that's the trade-off, flexibility, extensibility versus more declarative style and easy overview. STEPHANIE: That makes a lot of sense because we were talking earlier about guardrails. And because those boundaries do exist, that might not give us the flexibility we want compared to just writing regular Ruby objects. But yeah, we do get the benefit of, like you said, auditing, and at least if we don't try to do some really gnarly, custom stuff, [laughs] something that's easier to read and comprehend. JOËL: And, again, maybe that's where in the best of both worlds situation, you say, hey, I'm creating some form of rules engine, whether it's for describing routes, or authorization, permissions, or users can build custom business rules for a product or something like that. And it's all object-based under the hood. And then, we provide a DSL to make it nice to work with these rules. If a programmer using our gem wants to write a custom rule that just really extends what the ones we shipped can do, allow them to do that via the object API. We have all the objects available to you that underlie the DSL. Add more rules yourself. And then maybe those can be plugged back into the DSL like we saw with the RSpec and custom matchers. Or maybe you have to say, okay, if I have a custom rule object, now I have to just stay in the object space. And I think both of those solutions are okay. But now you've sort of kept those two worlds separate and still allowed people to extend. STEPHANIE: I like that as contributing to the language because language is never static. It changes over time. And that's a way that people can continue to evolve a language that may have been originally written at a certain time and place. JOËL: Moving on from DSLs, we got some listener feedback recently from James, who was listening to our episode on discrete math. And James really appreciated the episode and wanted to share a resource with us. This is the book "Discrete Math and Functional Programming" by Thomas VanDrunen. It's an introduction to discrete math as a theoretical concept taught side by side with the very practical aspect of learning to use the language standard ML, and both of those factor into each other. So you're kind of learning a little bit of theory and some practice, at the same time, getting to implement some discrete math concepts in standard ML to get a feel for them. Yeah, I've not read this book, but I love the concept of pairing a theoretical piece and a practical piece. So I'll drop a link to it in the show notes as well. Thank you, James. STEPHANIE: Yeah, thanks, James. And I guess this is just a little reminder that if our listeners have any feedback or questions they want to write in about, you can reach us at hosts@bikeshed.fm. JOËL: On that note. Shall we wrap up? STEPHANIE: Let's wrap up. Show notes for this episode can be found at bikeshed.fm. JOËL: This show has been produced and edited by Mandy Moore. STEPHANIE: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review in iTunes. It really helps other folks find the show. JOËL: If you have any feedback for this or any of our other episodes, you can reach us @_bikeshed, or you can reach me @joelquen on Twitter. STEPHANIE: Or reach both of us at hosts@bikeshed.fm via email. JOËL: Thanks so much for listening to The Bike Shed, and we'll see you next week. ALL: Byeeeeeee!!!!!!! ANNOUNCER: This podcast is brought to you by thoughtbot, your expert strategy, design, development, and product management partner. We bring digital products from idea to success and teach you how because we care. Learn more at thoughtbot.com.

head english japan writing slack api ml apis javascript html rails css pundits sdks mandy moore abstraction oo authorization restful typescript apm dsl rube goldberg tdd readme lisp capybara functional programming can can mystified quenneville dsls asts activerecord rspec jason swett rubykaigi stephanie yeah domain specific languages stephanie it stephanie right

JavaScript needs a model layer

Frontend First

Play Episode Listen Later Apr 12, 2023 43:02

Sam and Ryan talk about updating Build UI to support lifetime memberships. They chat about the site's current architecture, the strengths and weaknesses of objects vs. functions, how the full stack JavaScript community could benefit from a proper model layer like ActiveRecord, the challenges of using GraphQL on the backend, Prisma, and more.Topics include:0:00 - Intro1:09 - Current architecture + single purchase6:30 - Rails model layer. OOP vs functional13:55 - What are classes good at, what are functions good at23:53 - Composition vs. inheritance31:44 - graphql. overfetching is not a problem on the backend. prisma.Links:Go Ahead, Make a Mess by Sandi Metz

model current mess composition layer javascript rails prisma graphql oop activerecord sandi metz

367: Value Objects

Play Episode Listen Later Jan 17, 2023 34:00

Joël's been traveling. Stephanie's working on professional development. She's also keeping up a little bit more with Ruby news and community news in general and saw that Ruby 3.2 introduced a new class called data to its core library for the use case of creating simple value objects. This episode is brought to you by Airbrake (https://airbrake.io/?utm_campaign=Q3_2022%3A%20Bike%20Shed%20Podcast%20Ad&utm_source=Bike%20Shed&utm_medium=website). Visit Frictionless error monitoring and performance insight for your app stack. Maggie Appleton's Tools for Thought (https://maggieappleton.com/tools-for-thought) Episode on note-taking with Amanda Beiner (https://www.bikeshed.fm/357) Obsidian (https://obsidian.md/) Zettelkasten (https://zettelkasten.de/posts/overview/) Evergreen notes (https://notes.andymatuschak.org/Evergreen_notes) New Data class (https://ruby-doc.org/3.2.0/Data.html) Joël's article on value objects (https://thoughtbot.com/blog/value-object-semantics-in-ruby) Episode on specialized vocabulary (https://www.bikeshed.fm/356) Primitive Obsession (https://wiki.c2.com/?PrimitiveObsession) Transcript: AD: thoughtbot is thrilled to announce our own incubator launching this year. If you are a non-technical founding team with a business idea that involves a web or mobile app, we encourage you to apply for our eight-week program. We'll help you move forward with confidence in your team, your product vision, and a roadmap for getting you there. Learn more and apply at tbot.io/incubator. STEPHANIE: Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Stephanie Minn. JOËL: And I'm Joël Quenneville. And together, we're here to share a little bit of what we've learned along the way. STEPHANIE: So, Joël, what's new in your world? JOËL: I've been traveling for the past few weeks in Europe. I just recently got back to the U.S. and have just gotten used to drinking American-style drip coffee again after having espresso every day for a few weeks. And it's been an adjustment. STEPHANIE: I bet. I think that it's such a downgrade compared to European espresso. I remember when I was in Italy, I also would really enjoy espresso every day at a local cafe and just be like sitting outside drinking it. And it was very delightful. JOËL: They're very different experiences. I have to say I do enjoy just holding a hot mug and sort of sipping on it for a long time. It's also a lot weaker. You wouldn't want to do a full hot mug of espresso. That would just be way too intense. But yeah, I think both experiences are enjoyable. They're just different. STEPHANIE: Yeah. So, that first day with your measly drip coffee and your jet lag, how are you doing on your first day back at work? JOËL: I did pretty good. I think part of the fun of coming back to the U.S. from Europe is that the jet lag makes me a very productive morning person for a week. Normally, I'm a little bit more of an evening person. So I get to get a bit of an alter ego for a week, and that helps me to transition back into work. STEPHANIE: Nice. JOËL: So you've also been on break and have started work again. How are you feeling productivity-wise, kicking off the New Year? STEPHANIE: I'm actually unbooked this week and the last week too. So I'm not working on client projects, but I am having a lot of time to work on just professional development. And usually, during this downtime, I also like to reassess just how I'm working, and lately, what that has meant for me is changing my note-taking process. And I'm really excited to share this with you because I know that you have talked about this on the show before, I think in a previous episode with a guest, Amanda Beiner. And I listened to that episode, and I was really inspired because I was feeling like I didn't have a note-taking system that worked super well for me. But you all talked about some tools you used and some, I guess, philosophies around note-taking that like I said, I was really inspired by. And so I hopped on board the Obsidian train. And I'm really excited to share with you my experience with it. So I really like it because I previously was taking notes in my editor under the impression that, oh, like, everything is in one place. It'll be like a seamless transition from code to note-taking. And I was already writing in Markdown. But I actually didn't like it that much because I found it kind of distracting to have code things kind of around. And if I was navigating files or something, something work or code-related might come up, and that ended up being a bit distracting for me. But I know that that works really well for some people; a coworker of ours, Aji, I know that he takes his notes in Vim and has a really fancy setup for that. And so I thought maybe that's what I wanted, but it turns out that what I wanted was actually more of a boundary between code and notes. And so, I was assessing different note-taking and knowledge management software. And I have been really enjoying Obsidian because it also has quite a bit of community support. So I've installed a few plugins for just quality-of-life features like snippets which I had in my editor, and now I get to have in Obsidian. I also installed things like Natural Language Dates. So for my running to-do list, I can just do a shortcut for today, and it'll autofill today's date, which, I don't know, because for me, [laughs] that is just a little bit less mental work that I have to do to remember the date. And yeah, I've been really liking it. I haven't even fully explored backlinking, and that connectivity aspect, which I know is a core feature, but it's been working well for me so far. JOËL: That's really exciting. I love notes and note-taking and the ways that we can use those to make our lives better as developers and as human beings. Do you have a particular system or way you've approached that? Because I know for me, I probably looked at Obsidian for six months before I kind of had the courage to download it because I didn't want to go into it and not have a way to organize things. I was like; I don't want to just throw random notes in here. I want to have a system. That might just be me. But did you just kind of jump into it and see, like, oh, a system will emerge? Did you have a particular philosophy going in? How are you approaching taking notes there? STEPHANIE: That's definitely a you thing because I've definitely had the opposite experience [laughs] where I'm just like, oh, I've downloaded this thing. I'm going to start typing notes and see what happens. I have never really had a good organizational system, which I think is fine for me. I was really leaning on pen and paper notes for a while, and I still have a certain use case for them. Because I find that when I'm in meetings or one-on-ones and taking notes, I don't actually like to have my hands on the keyboard because of distractions. Like I mentioned earlier, it's really easy for me to, like, oh, accidentally Command-Tab and open Slack and be like, oh, someone posted something new in Slack; let me go read this. And I'm not giving the meeting or the person I'm talking to my full attention, and I really didn't like that. So I still do pen and paper for things where I want to make sure that I'm not getting distracted. And then, I will transfer any gems from those notes to Obsidian if I find that they are worth putting in a place where I do have a little bit more discoverability and eventually maybe kind of adding on to my process of using those backlinks and connecting thoughts like that. So, so far, it's truly just a list of separate little pages of notes, and yeah, we'll see how it goes. I'm curious what your system for organizing is or if you have kind of figured out something that works well for you. JOËL: So my approach focuses very heavily on the backlinks. It's loosely inspired by two similar systems of organization called Zettelkasten and evergreen notes. The idea is that you create notes that are ideas. Typically, the title is like a thesis statement, and you keep them very short, focused on a single thing. And if you have a more complex idea, it probably breaks down into two or three, and then you link them to each other as makes sense. So you create a web of these atomic ideas that are highly interconnected with each other. And then later on, because I use this a lot for either creating content in the future or to help refine my thinking on various software topics, so later on, I can go through and maybe connect three or four things I didn't realize connected together. Or if I'm writing an article or a talk, maybe find three or four of these ideas that I generated at very different moments, but now they're connected. And I can make an article or a talk out of them. So that's sort of the purpose that I use them for and how I've organized things for myself. STEPHANIE: I think that's a really interesting topic because while I was assessing different software for note-taking and, like I said, knowledge management, I discovered this blog post by Maggie Appleton that was super interesting because she is talking about the term tools of thought which a lot of these different software kind of leveraged in their marketing copy as like, oh, this software will be like the key to evolving your thinking and help you expand making connections, like you mentioned, in ways that you weren't able to before. And was very obviously trying to upsell you on this product, and she -- JOËL: It's over the top. STEPHANIE: A little bit, a little bit. So in this article, I liked that she took a critical lens to that idea and rooted her article in history and gave examples of a bunch of different things in human history that also evolved the ways humans were able to express their thoughts and solve problems. And so some of the ones that she listed were like storytelling and oral tradition. Literally, the written language obviously [laughs] empowered humans to be able to communicate and think in ways that we never were before but also drawings, and maps, and spreadsheets. So I thought that was really cool because she was basically saying that tools of thought don't need to be digital, and people claiming that these software, you know, are the new way to think or whatever, it's like, the way we're thinking now, but we also have this long history of using and developing different things that helped us communicate with each other and think about stuff. JOËL: I think that's something that appealed to me when I was looking at some of these note-taking systems. Zettelkasten, in particular, predates digital technology. The original system was built on note cards, and the digital stuff just made it a little bit easier. But I think also when I was reading about these ideas of keeping ideas small and linking them together, I realized that's already kind of how I tend to organize information when I just hold it in my brain or even when I try to do something like a tweet thread on Twitter where I'll try to break it up. It might be a larger, more complex idea, but each tweet, I try to get it to kind of stand on its own to make it easier to retweet and all that. And so it becomes a chain of related ideas that maybe build up to something, but each idea stands on its own. And that's kind of how in these systems notes end up working. And they're in a way that you can kind of remix them with each other. So it's not just a linear chain like you would have on Twitter. STEPHANIE: Yeah, I remember you all in that episode about note-taking with Amanda talked about the value of having an atomic piece of information in every note that you write. And since then, I've been trying to do that more because, especially when I was doing pen and paper, I would just write very loose, messy thoughts down. And I would just think that maybe I would come back to them one day and try to figure out, like, oh, what did I say here, and can I apply it to something? But it's kind of like doing any kind of refactoring or whatever. It's like, in that moment, you have the most context about what you just wrote down or created. And so I've been a little more intentional about trying to take that thought to its logical end, and then hopefully, it will provide value later. What you were saying about the connectivity I also wanted to kind of touch on a little bit further because I've realized that for me, a lot of the connection-making happens during times where I'm not very actively trying to think, or reflect, or do a lot of deep work, if you will. Because lately, I've been having a lot of revelations in the shower, or while I'm trying to fall asleep, or just other kinds of meditative activity. And I'm just coming to terms with that's just how my brain works. And doing those kinds of activities has value for me because it's like something is clearly going on in my brain. And I definitely want to just honor that's how it works for me. JOËL: I had a great conversation recently with another colleague about the gift of boredom and how that can impact our work and what we think about, and our creativity. That was really great. Sometimes it's important to give ourselves a little bit more blank space in our lives. And counter-intuitively, it can make us more productive, even though we're not scheduling ourselves to be productive. STEPHANIE: Yes, I wholeheartedly agree with that. I think a lot about the feeling of boredom, and for me, that is like the middle of summer break when you're still in school and you just had no obligations whatsoever. And you could just do whatever you wanted and could just laze around and be bored. But letting your mind wander during those times is something I really miss. And sometimes, when I do experience that feeling, I get a little bit anxious. I'm like, oh, I could be doing something else. There's whatever endless list of chores or things that are, quote, unquote, "productive." But yeah, I really like how you mentioned that there is value in that experience, and it can feel really indulgent, but that can be good too. MID-ROLL AD: Debugging errors can be a developer's worst nightmare...but it doesn't have to be. Airbrake is an award-winning error monitoring, performance, and deployment tracking tool created by developers for developers that can actually help cut your debugging time in half. So why do developers love Airbrake? It has all of the information that web developers need to monitor their application - including error management, performance insights, and deploy tracking! Airbrake's debugging tool catches all of your project errors, intelligently groups them, and points you to the issue in the code so you can quickly fix the bug before customers are impacted. In addition to stellar error monitoring, Airbrake's lightweight APM helps developers to track the performance and availability of their application through metrics like HTTP requests, response times, error occurrences, and user satisfaction. Finally, Airbrake Deploy Tracking helps developers track trends, fix bad deploys, and improve code quality. Since 2008, Airbrake has been a staple in the Ruby community and has grown to cover all major programming languages. Airbrake seamlessly integrates with your favorite apps to include modern features like single sign-on and SDK-based installation. From testing to production, Airbrake notifiers have your back. Your time is valuable, so why waste it combing through logs, waiting for user reports, or retrofitting other tools to monitor your application? You literally have nothing to lose. Head on over to airbrake.io/try/bikeshed to create your FREE developer account today! JOËL: So you mentioned recently that you've had a lot of revelations or new ideas that have come upon you or that you've been able to dig into a little bit more. Is there one you'd like to share with the audience? STEPHANIE: Yeah. So during this downtime that I've had not working on client work, I have been able to keep up a little bit more with Ruby news or just community news in general. And in, I think, an edition of Ruby Weekly, I saw that Ruby 3.2 introduced this new class called data to its core library for the use case of creating simple value objects. And I was really excited about this new feature because I remembered that you had written a thoughtbot blog post about value objects back in the summer that I had reviewed. That was an opportunity that I could make a connection between something happening in recent news with some thoughts that I had about this topic a few months ago. But basically, this new class can be used over something like a struct to create objects that are immutable in their values, which is a big improvement if you are trying to follow value objects semantics. JOËL: So, I have not played around with the new data class. How is it different from the existing struct that we have in Ruby? STEPHANIE: So I think I might actually answer that first by saying how they're similar, which is that they are both vehicles for holding pieces of data. So we've, in the past, been able to use a struct to very cheaply and easily create a new class that has attributes. But one pitfall of using a struct when you're trying to implement something like a value object is that structs also came with writer methods for all of its members. And so you could change the value of a member, and that it kind of inherently goes against the semantics of a value object because, ideally, they're immutable. And so, with the data class, it doesn't offer writer methods essentially. And I think that it freezes the instance as well in the constructor. And so even if you tried to add writer methods, you would eventually get an error. JOËL: That's really convenient. I think that may be an area where I've been a little bit frustrated with structs in the past, which is that they can be modified. They basically get treated as if they're hashes with a slightly nicer syntax to interact with them. And I want slightly harder boundaries around the data. Particularly when I'm using them as value objects, I generally don't want people to modify them because that might lead to some weird bugs in the code where you've got a, I don't know, something represents a time value or a date value or something, and you're trying to do math on it. And instead of giving you a new time or date, value just modifies the first one. And so now your start date is in the past or something because you happen to subtract a time from it to do a calculation. And you can't assign it to a variable anywhere. STEPHANIE: Yeah, for sure. Another kind of pitfall I remember noticing about structs were that the struct class includes the enumerable module, which makes a struct kind of like a collection. Whereas if you are using it for a value object, that's maybe not what you want. So there was a bit of discourse about whether or not the data class should inherit from struct. And I think they landed on it not inheriting because then you can draw a line in the sand and have that stricter enforcement of saying like, this is what a data as value object should be, and this is what it should not be. So I found that pretty valuable too. JOËL: I think I've heard people talk about sort of two classes of problems that are typically solved with a struct; one is something like a value object where you probably don't want it to be writable. You probably don't want it to be enumerable. And it sounds like data now takes on that role very nicely. The other category of problem is that you have just a hash, and you're trying to incrementally migrate it over to some nicer objects in some kind of domain. And struct actually gives you this really nice intermediate phase where it still mostly behaves like a hash if you needed to, but it also behaves like an object. And it can help you incrementally transition away from just a giant hash into something that's a little bit more programmatic. STEPHANIE: Yeah, that's a really good point. I think struct will still be a very viable option for that second category that you described. But having this new data class could be a good middle ground before you extract something into its own class because it better encapsulates the idea of a value object. And one thing that I remember was really interesting about the article that you wrote was that sometimes people forget to implement certain methods when they're writing their own custom value objects. And these come a bit more out of the box with data and just provide a bit more like...what's the word I'm looking for? I'm looking for...you know when you're bowling, and you have those bumpers, I guess? [laughs] JOËL: Uh-huh. STEPHANIE: They provide just like safeguards, I guess, for following semantics around value objects that I thought was really important because it's creating an artifact for this concept that didn't exist. JOËL: And to recap for the audience here, the difference is in how objects are compared for equality. So value objects, if they have the same internal value, even if they're separate objects in memory, should be considered equal. That's how numbers work. That's how hashes work. Generally, primitives in Ruby behave this way. And structs behave that way, and the new data class, it sounds, also behaves that way. Whereas regular objects that you would make they compare based off of the identity of the object, not its value. So if you create two user instances, not ActiveRecord, but you could create a user class, you create two instances in memory. They both have the same attributes. They will be considered not equal to each other because they're not the same instance in memory, and that's fine for something more complex. But when you're dealing with value objects, it's important that two objects that represent the same thing, like a particular time for a unit of measure or something like that, if they have the same internal value, they must be the same. STEPHANIE: Right. So prior to the introduction of this class, that wasn't really enforced or codified anywhere. It was something that if you knew what a value object was, you could apply that concept to your code and make sure that the code you wrote was semantically aligned with this concept. And what was kind of exciting to me about the addition of this to the core class library in Ruby is that someone could discover this without having to know what a value object is like more formally. They might be able to see the use of a data class and be like, oh, let me look this up in the official Ruby docs. And then they could learn like, okay, here's what that means, and here's some rules for this concept in a way that, like I mentioned earlier, felt very implicit to me prior. So that, I don't know, was a really exciting new development in my eyes. JOËL: One of the first episodes that you and I recorded together was about the value of specific vocabulary. And I think part of what the Ruby team has done here is they've taken an implicit concept and given it a name. It's extracted, and it has a name now. And if you use it now, it's because you're doing this data thing, this value object thing. And now there's a documentation page. You can Google it. You can find it rather than just be wondering like, oh, why did someone use a struct in this way and not realize there are some implicit semantics that are different? Or wondering why did the override double equals on this custom class? STEPHANIE: Yeah, exactly. I think that the introduction of this class also provides a solution for something that you mentioned in that blog post, which was the idea of testing value objects. Because previously, when you did have to make sure that you implemented methods, those comparison methods to align with the concept of a value object, it was very easy to forget or just not know. And so you provided a potential solution of testing value objects via an RSpec shared example. And I remember thinking like, ooh, that was a really hot topic because we had also been debating about shared examples in general. But yeah, I was just thinking that now that it's part of the core library, I think, in some ways, that eliminates the need to test something that is using a data class anyway because we can rely a little bit more on that dependency. JOËL: Right? It's the built-in behavior now. Do you have any fun uses for value objects recently? STEPHANIE: I have not necessarily had to implement my own recently. But I do think that the next time I work with one or the next time I think that I might want to have something like a value object it will be a lot easier. And I'm just excited to play around with this and see how it will help solve any problem that might come up. So, Joël, do you have any ideas about when you might reach for a data object? JOËL: A lot of situations, I think, when you see the primitive obsession smell are a great use case for value objects, or maybe we should call them data objects now, now that this is part of Ruby's vocabulary. I think I often tend to; preemptively sounds bad, but a lot of times, I will try to be careful. Anytime I'm doing anything with raw numbers, magic strings, things like that, I'll try to encapsulate them into some sort of struct. Or even if it's like a pair of numbers, it always goes together, maybe a latitude and longitude. Now, those are a pair. Do I want to just be passing around a two-element array all the time or a hash that would probably make a very nice data object? If I have a unit of measure, some number that represents not just the abstract concept of three but specifically three miles or three minutes, then I might reach for something like a data class. STEPHANIE: Yeah, I think that's also true if you're doing any kind of arithmetic or, in general, trying to compare anything about two of the same things. That might be a good indicator as well that you could use something richer, like a value object, to make some of that code more readable, and you get some of those convenient methods for doing those comparisons. JOËL: Have you ever written code where you just have like some number in the code, and there's a comment afterwards that's like minutes or miles or something like that, just giving you the unit as a comment afterwards? STEPHANIE: Oh yeah. I've definitely seen some of that code. And yeah, I mean, now that you mentioned it, that's a great use case for what we're talking about, and it's definitely a code smell. JOËL: It can often be nice as you make these more domain concepts; maybe they start as a data object, but then they might grow with their own custom methods. And maybe you extend data the same way you could extend a struct, or maybe you create a custom class to the point where the user...whoever calls that object, doesn't really need to know or care about the particular unit, just like when you have duration value. If you have a duration object, you can do the math you want. You can do all the operations and don't have to know whether it is in milliseconds, or seconds, or minutes because it knows that internally and keeps all of the math straight as opposed to just holding on to what I've done before, which is you have some really big number somewhere. You have start is, or length is equal to some big number and then comment milliseconds. And then, hopefully, whoever does math on that number later remembers to do the division by 1,000 or whatever they need. STEPHANIE: I've certainly worked on code where we've tolerated those magic numbers for probably longer than we should have because maybe we did have the shared understanding that that value represents minutes or milliseconds or whatever, and that was just part of the domain knowledge. But you're right, like when you see them, and without a very clear label, all of that stuff is implied and is really not very friendly for someone coming along in the future. As well as, like you mentioned earlier, if you have to do math on it later to convert it to something else, that is also a red flag that you could use some kind of abstraction or something to represent this concept at a higher level but also be extensible to different forms, so a duration to represent different amounts of time or money to represent different values and different currencies, stuff like that. JOËL: Do you have a guideline that you follow as to when something starts being worth extracting into some kind of data object? STEPHANIE: I don't know if I have particularly clear guidelines, but I do remember feeling frustrated when I've had to test really complicated hashes or just primitives that are holding a lot of different pieces of information in a way that just is very unwieldy when you do have to write a test for it. And if those things were encapsulated in methods, that would have been a lot easier. And so I think that is a bit of a signal for me. Do you have any other guidelines or gut instincts around that? JOËL: We mentioned the comment that is the unit. That's probably a...I wasn't sure if I would have to call it a code smell, but I'm going to call it a code smell that tells you maybe you should...that value wants to be something a little bit more than just a number. I've gotten suspicious of just raw integers in general, not enough to say that I'm going to make all integers data objects now, but enough to make me pause and think a lot of times. What does this number represent? Should it be a data object? I think I also tend to default to try to do something like a data object when I'm dealing with API responses. You were talking about hashes and how they can be annoying to test. But also, when you're dealing with data coming back from a third-party API, a giant nested hash is not the most convenient thing to work with, both for the implementation but then also just for the readability of your code. I often try to have almost like a translation layer where very quickly I take the payload from a third-party service and turn it into some kind of object. STEPHANIE: Yeah, I think the data class docs itself has an example of using it for HTTP responses because I think the particular implementation doesn't even require it to have attributes. And so you can use it to just label something rather than requiring a value for it. JOËL: And that is one thing that is nice about something like a data object versus a hash is that a hash could have literally anything in it. And to a certain extent, a data object is self-documenting. So if I want to know I've gotten to a shopping cart object from a third-party API, what can I get out of the shopping cart? I can look at the data object. I can open the class and see here are the methods I can call. If it's just a hash, well, I guess I can try to either find the documentation for the API or try to make a real request and then inspect the hash at runtime. But there's not really any way to find out without actually executing the code. STEPHANIE: Yeah, that's totally fair. And what you said about self-documenting makes a lot of sense. And it's always preferable than that stray comment in the code. [laughs] JOËL: I'm really excited to use the data class in future Ruby 3.2 projects. So I'm really glad that you brought it up. I've not tried it myself, but I'm excited to use it in future projects. STEPHANIE: On that note, shall we wrap up? JOËL: Let's wrap up. STEPHANIE: Show notes for this episode can be found at bikeshed.fm. JOËL: This show has been produced and edited by Mandy Moore. STEPHANIE: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review in iTunes. It really helps other folks find the show. JOËL: If you have any feedback for this or any of our other episodes, you can reach us @_bikeshed, or you can reach me @joelquen on Twitter. STEPHANIE: Or reach both of us at hosts@bikeshed.fm via email. JOËL: Thanks so much for listening to The Bike Shed, and we'll see you next week. ALL: Byeeeeeeeeeeeeee!!!!!!!!!!!! ANNOUNCER: This podcast is brought to you by thoughtbot, your expert strategy, design, development, and product management partner. We bring digital products from idea to success and teach you how because we care. Learn more at thoughtbot.com.

american new year head europe google european italy data tools slack api generally evergreen obsidian sdks mandy moore new data apm vim markdown zettelkasten quenneville aji activerecord rspec stephanie yeah value objects stephanie oh stephanie right stephanie they

364: Constructive vs Predicative Data

Play Episode Listen Later Dec 6, 2022 34:07

Stephanie and Joël attended RubyConf Mini, and both spoke there. They discuss takeaways and highlights from the conference. The core idea for this episode is explained in this article: Constructive vs. Predicative Data (https://www.hillelwayne.com/post/constructive/). This came up recently in a conversation at thoughtbot about designing a database schema and what constraints could be encoded in the schema directly versus needing some kind of trigger or Rails validation to cover it. This episode is brought to you by Airbrake (https://airbrake.io/?utm_campaign=Q3_2022%3A%20Bike%20Shed%20Podcast%20Ad&utm_source=Bike%20Shed&utm_medium=website). Visit Frictionless error monitoring and performance insight for your app stack. RubyConf Mini (https://www.rubyconfmini.com/) Episode on CFP - The Bike Shed 352: Case Expressions (https://www.bikeshed.fm/352) Podcast panel: The Ruby on Rails Podcast Episode 446: I'm Giving A Talk on Thursday (https://www.therubyonrailspodcast.com/446) Slides for FP talk: Functional Programming for Fun and Profit!! (https://speakerdeck.com/jennyshih/functional-programming-for-fun-and-profit?slide=107) Episode on language: The Bike Shed - 356: The Value of Specialized Vocabulary (https://www.bikeshed.fm/356) Constructive vs. Predicative data (https://www.hillelwayne.com/post/constructive/) Avoid the Three-state Boolean Problem (https://thoughtbot.com/blog/avoid-the-threestate-boolean-problem) Transcript: JOËL: Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Joël Quenneville. STEPHANIE: And I'm Stephanie Minn. And together, we're here to share a bit of what we've learned along the way. JOËL: So something that's very recent in both of our worlds has been that both you and I, Stephanie, attended RubyConf Mini, and we both spoke there. What are some of your takeaways or highlights from the conference? STEPHANIE: Seeing you in person was definitely a highlight. I really enjoyed that. Because we're working remotely, I don't, you know, get to be in an office with you day to day. And it was really awesome to hang out with you, I think, for the first time as co-hosts of the podcast. And we both, I think, met some people at the conference too that were listeners. And it was really awesome to share that experience with you. JOËL: I had the interesting experience of several people who told me they recognized me by my voice, which I think is a common thing for podcasters, but as a new host, I was surprised by that. STEPHANIE: Yeah, that's weird. As a podcast listener, too, I definitely know exactly what you're talking about where it's like, oh yeah, I can identify someone by their voice. But to then be that person that people can recognize is pretty weird. I also really enjoyed being an audience member of the podcast panel that you are on at the conference with other podcast folks. It was moderated by Brittany Martin. And yeah, I just thought you represented The Bike Shed really well and spoke for both of us about podcasting in a way that I really appreciated. JOËL: And for any of our listeners who were not able to be there in person, Brittany has published that episode as a podcast, and we will link to it in the show notes. STEPHANIE: Another thing I really liked about RubyConf Mini was the smaller scale. I think it was about 150 or so attendees, which felt very different from traditional Ruby Central conferences with several hundreds of people. I heard a lot from other folks there that they really liked the regional aspect of it, the intimacy of the smaller conference. I think I got more of an opportunity to run into people that I'd met at the conference over the next few days. And there was, yeah, definitely a sense of tighter knit community there, you know, when you meet someone, and then you bump into them on the way into a talk, and then you can ask how their day was going and any highlights that they had. And yeah, I guess I haven't really attended a conference that size before, and so that felt like a very special experience for me. JOËL: I 100% agree. I think the smaller format definitely makes it a little bit more intimate, makes it much easier, I think, to build some of those social connections, to meet with people, and to have some good conversations. I think the format of the conference as well favored that. There were, I think, larger breaks between talks that encouraged people to hang out and talk. And, as you said, because it's smaller, you also get to see the same people over the course of a few different breaks instead of being like, oh, I met a stranger on the morning of day one, and then in the afternoon, I met another stranger. And it's just constantly introducing yourself. One thing that was really interesting to me is the experience of being a speaker is very different than just attending. As a speaker, you get to go to the speaker dinner and connect with a lot of the other speakers there. Some of them might be quote, unquote "famous people" that you're not quite comfortable just walking up to and introducing yourself. But in the smaller dinner, you just find yourself sitting next to them and enjoying some food or a drink and getting conversations. It's also much easier to have people come up to you during the conference. Because you're a speaker, people will come and talk to you. So if you tend to be a little bit more introverted, as long as you can get over your fear of being on stage and public speaking, it actually makes social connection interaction much easier to be a speaker. I would recommend to any of our listeners who were wondering how can I get more out of a conference? How can I get better connections, better conversations? Consider being a speaker. STEPHANIE: Yeah, absolutely. We've talked about this before; I think when we chatted about writing our CFPs for this conference that speaking doesn't have to be a really big, scary thing, but everyone has something to say. I think we had mentioned in previous episodes that your talk topic came out of just a discussion that you had internally, and you were like, wow, enumerables are so cool, like, let me dig deeper into them and just share what I learned. So I totally recommend it. And this conference was my first in real-life speaking opportunity as well, and that felt super different from my experience last time doing it virtually, you know, talking about how much I love that sense of community all the time. But it really felt true for me this time around, where I could see the audience react to the things I was saying, like, maybe go off the cuff a little bit. And then yeah, at the end, having people come up to me was really awesome to just talk about pairing, which is what I spoke about, and just share our experiences. And they asked what I thought about some things, and it was really cool to just be able to spread that knowledge around. And one thing I noticed you did a lot was come up to speakers after they wrapped up their talks. You were almost always the first person to get up and congratulate them and just get the ball rolling on following up on the things they talked about. Is that something that you really enjoy doing or find particularly valuable as an audience member or speaker? JOËL: Yes, both. I think, as a speaker, it's really validating to have people come up to you after the talk and either just tell you they liked the talk or ask a question. I generally don't like to do just open questions after a talk from the audience because then you get the classic; this is more of a comment than a question or people who will tell you that you had a typo on one of your code slides. Like, none of that is useful to anyone. So, if you're really interested, come talk to me afterwards. And then that actually makes me feel like my talk connected with people, and people were paying attention, people enjoyed it, people were learning. So I try to pay that forward as well for talks that I listened to, go up to the speaker, and tell them one thing that I appreciated about the talk or a thing that I learned, or something that got me excited in their content. STEPHANIE: Yeah, I'm sure that it's very appreciated. And it also breaks the awkward silence at the end when the speaker finishes and people aren't sure if it's okay for them to get up and start moving around. Yeah, I thought that was a really good way to kind of just encourage people to start chatting with each other and moving into those break times that we mentioned earlier, those opportunities to socialize. JOËL: Another thing that I think is really fun that you can do at in-person conferences, and I know you were doing it a lot, is going to see the talks of friends and colleagues and sitting in the front row and just being there to cheer them on and encourage them. Again, I think that makes a big difference when you are on stage, and you see these people who are your friends and colleagues there to support you. It gives you that boost of confidence. And when you're there in the audience, it's fun to cheer on somebody else. STEPHANIE: Oh yeah. You gave me a lot of thumbs-ups during my talk, and I really appreciated that. [laughs] So I'm curious if there were any talks that stood out to you that you got to see. JOËL: And I was really inspired by your talk, pair programming. I think there are a lot of things that I can take from that to improve the way I pair. I was also inspired by Aji's talk, Aji Slater, on automating manual tasks that you have to do in an iterative way. That one really hit home because, on my current project, I have been doing a lot of manual things. And I just have random snippets of code, like, some shell script lines or Ruby console lines, that I copy-paste out of Slack conversations because I've shared them with other people who are doing similar work. And I realized that a lot of his advice would apply to the work that I'm doing and how that could really make things better. So that was one of those talks I was listening to, and I was like, oh, you know what? Monday morning, when I go back to my project, this is something that I'm going to start doing. This is something I'm going to change in the way I do my day-to-day work. STEPHANIE: Yeah, absolutely. I have so many tasks that I would like to get automated, and think that one day I will magically have more time in my schedule to get to it. But I liked that his talk gave pretty concrete strategies for baking it into your regular, like you said, day-to-day workflow, and that lowers the activation energy to getting them done. And then those things can be iterated on and could eventually become, in an ideal world, a fully-fledged feature that you put together from doing those repetitive tasks. And yeah, they provide a lot of value not just to you but can eventually provide value to your co-workers and then even your users in the future. JOËL: Were there any talks that stood out for you? STEPHANIE: One talk that I really enjoyed was Jenny Shih's about Functional Programming for Fun and Profit. I have attended a lot of functional programming talks within the Ruby realm, at least to try to get a better sense of how it can apply to my work and the languages and paradigms that I use. And honestly, what I liked about it was that it didn't get too in the weeds about functional programming. What she did was provide mental models for understanding the paradigm that I think was a good vehicle for understanding things very generally. And, for me, like,¬¬ a talk, it's really hard to pay attention to lines of code and to read code on the fly while people are presenting. For me, that is just not how I like to consume that information. And so she provided themes and, like I said, those mental models, which I know you really like to use a lot too in teaching people new concepts. For me, I didn't fully learn what a monad was, once again, but at least having that repeated exposure to those foundational aspects, I think, will eventually lead me to be able to grok those things a little more comprehensively the next time I see it or whenever I decide to dig deeper. JOËL: What was a mental model that was shared that connected with you particularly? STEPHANIE: So one of the main mental models that she shared was thinking about a program in terms of these three dimensions: value, behavior, and time. She had a nice slide that showed the difference between the object-oriented paradigm, where value and behavior are contained by objects, where time is kind of inherently wrapped up in those objects that hold information about the state through values and behavior. Whereas in her functional programming example, those three dimensions were a bit separate. And I found that distinction to be really helpful in separating things that felt very implicit before, but it was nice to see them broken out into very clear concepts in terms of building blocks of a program. JOËL: So it's helpful then when thinking...when you look at code, if you can think about it in those three different dimensions to help think about, am I taking a functional or other approach in this particular dimension when working with this code? STEPHANIE: Yeah, exactly. I think it also gave me more of a vocabulary to describe the pros and cons of each and a lens of thinking about which I might want to choose for the particular problem at hand. JOËL: So you mentioned there's a visual for these three dimensions from the slides. Are those slides publicly available? STEPHANIE: They are. I will link to them in the show notes. JOËL: So all of these talks were recorded. They're not yet available to the public, but I think the plan is to publish them on YouTube sometime in the new year, so that means probably January 2023. And a big shout out to the AV team and everyone who is involved in recording these. STEPHANIE: Yeah, I am definitely looking out for a link to my talk so I can send it to my mom. I also wanted to give a little shout-out to the organizers of RubyConf Mini: Jemma Issroff, Emily Samp, and Andy Croll. JOËL: Woo! STEPHANIE: They put on just a really awesome conference, and I feel very grateful that I got a chance to attend with you, Joël. JOËL: It was definitely a delightful experience. STEPHANIE: Delightful. That's a reference to Joël's talk for those of you who are listening. MID-ROLL AD: Debugging errors can be a developer's worst nightmare...but it doesn't have to be. Airbrake is an award-winning error monitoring, performance, and deployment tracking tool created by developers for developers that can actually help cut your debugging time in half. So why do developers love Airbrake? It has all of the information that web developers need to monitor their application - including error management, performance insights, and deploy tracking! Airbrake's debugging tool catches all of your project errors, intelligently groups them, and points you to the issue in the code so you can quickly fix the bug before customers are impacted. In addition to stellar error monitoring, Airbrake's lightweight APM helps developers to track the performance and availability of their application through metrics like HTTP requests, response times, error occurrences, and user satisfaction. Finally, Airbrake Deploy Tracking helps developers track trends, fix bad deploys, and improve code quality. Since 2008, Airbrake has been a staple in the Ruby community and has grown to cover all major programming languages. Airbrake seamlessly integrates with your favorite apps to include modern features like single sign-on and SDK-based installation. From testing to production, Airbrake notifiers have your back. Your time is valuable, so why waste it combing through logs, waiting for user reports, or retrofitting other tools to monitor your application? You literally have nothing to lose. Head on over to airbrake.io/try/bikeshed to create your FREE developer account today! JOËL: Coming back from the conference, I recently had a really interesting conversation with some other colleagues at thoughtbot. We were looking at a database schema for a new application and talking about some of the trade-offs involved in how that schema is structured, so what tables we want to have. Do we want to have indexes? Things like that. And particularly around some of the assumptions are business rules that would come into play. So we're looking at...we'd drawn out this Entity Relationship Diagram (ERD). In it, we're looking at all the tables, and something that comes up immediately is like, oh, it's possible to have some bad data that could show up in these columns. Or it's possible that this relationship could exist where this table has a foreign key on this table, but really, that should never happen in this particular way of working. And so then the question became, how do we try to prevent these things that currently the schema allows but that are not valid in this particular business domain? Do we want to change the schema somehow and make that stricter or find some way to prevent it? Do we want to add some kind of validation that will check some business rules first before inserting or updating a record? I'm curious, have you ever been in a situation like that where you had to balance those two approaches to enforcing business rules on your database? A classic small example of this is a situation where let's say, you have a users' table and you have a name column on there. And you want to ensure that that name must always be present; all users must have names. Do you try to enforce that via the schema with a NOT NULL constraint? Or maybe you try to enforce that with a validation, maybe a presence validation at the Rails level. Or if you're really into SQL, maybe some fancy trigger, but do it in a validation style rather than trying to force this using the schema. And our particular scenario was a little bit more complex than just one column; it was more to do with associations. But I think this sort of problem shows up even in constraints as small as a required field. STEPHANIE: That's really interesting. I think that, in my experience, when we are spinning up new tables, at that point, we do try to put some intentional thought into what the schema should look like and what requirements we might need to encode at the database level. But things that are more complex might need a little more code, like Ruby code. I have then pushed to an ActiveRecord validation. One thing that I think is important to know is that when you do set those things on the schema, it's harder to change. And so you usually have to feel pretty confident that that's what you want. Otherwise, you'll run into issues later if that does have to change and making changes to whatever existing data you might have. But it's also pretty common to just do your best when you are deciding on a database schema and then having to make adjustments down the line as you know more about your domain. JOËL: This conversation reminds me a little bit of the idea of database normalization. I think that might almost fit as a subset of general tactics of using the schema to ensure your data is more correct. When you are generating new tables, let's say you're creating a greenfield app and you need to create four or five tables; how much emphasis do you put on database normalization when you're initially designing those? STEPHANIE: I think for a greenfield project when you are setting everything up and creating tables for your main domain models, there is an aspect of it that should be considered because you're in this unique position where nothing really is in existence yet. And you do want to try to set yourself up to be successful and hopefully have information about your main use case for this app and can kind of make decisions about the schema then. At least in my experience, that has been part of the conversation, though, to be fair, because it's so early, you do have the opportunity to change things without as much effort or pain. But I think it's worth considering when you're just sitting down and working through what those models are going to look like. JOËL: And for our listeners who may not have heard the term normalization before, it's a series of...you can think of them as rules that you apply to your database design to try to avoid data redundancies in your tables. There are different levels of this; they're typically referred to as normal forms. So you'll see things like first normal form, second normal form, third normal form; those are kind of the fancy terms for them. But they generally involve breaking out other tables so that you don't have data redundancies. And in many ways, this is similar to principles such as the single-responsibility principle that we apply to objects when we're designing our objects in an OO system. But this is more at the table level for databases. STEPHANIE: I do think that it is so hard, maybe even impossible, to plan something out, to not have any of those redundancies, to begin with. And I do think sometimes they are a bit inevitable. But I also have had the experience of having to figure out what the heck I'm looking at when I am querying data and see all these things that are duplicated or maybe slightly different. And yeah, I think when you are in that position of starting a greenfield application, it is really interesting to see how you make those decisions about what needs to be enforced and where. Where did you end up landing, or what did you discuss in this conversation with the co-worker? JOËL: I think we went with a bit of a hybrid approach. Some things, we can use the schema to prevent bad data, and then some things either cannot be represented with a schema, or it's possible, but it's really cumbersome and painful. And so, we chose to try to enforce it with a validation. To me, this feels very similar to a problem in typed languages. So some communities that use a lot of types try to use those types to only allow data to come through that's in a valid shape. And so you'll hear things like make impossible states impossible or make illegal states unrepresentable. And that works for many things, but it's not always possible to enforce all of your business constraints through a schema. Or sometimes it's possible but just not practical. And so, I think there is a balance of finding when you can use the schema or when it's better to use the validation.¬ STEPHANIE: Yeah, I think my general rule of thumb is, like I mentioned earlier, things I feel really confident about that we want to make sure that we have in our database or in our data for sure. I do lean towards requiring those in a schema, and it also communicates that confidence or communicates that intent that it's something that at one point was decided is important. And so, if a future developer comes in, it would take a lot of work for them to write a migration, to remove some database constraint. Whereas I think sometimes validations at the Rails level are potentially a little more open to change and then even more so if you get to validating on the client side. JOËL: That can get to be a really, like, it's a useful tool, but one that you can really hurt yourself with. If you modify your validations at the Rails level or at the front-end level, but then you don't backfill those changes on your data in the database, then you might have records in your database that if you were to load them into memory and hit save on them again, would refuse to save because they no longer match the validations. And on longer-lived applications, I've seen that happen sometimes where not all rows in the database pass the Rails validations. STEPHANIE: Yeah, I think I've seen that be a problem either for developers who then have to backfill that data or write some migration to change some of the data to meet the new requirements, or just unexpected bugs on the users who discover something new but like you said, have been there long enough before those things were implemented. JOËL: The more I think of this, I think maybe constraints that are enforced at a validation level might still require changing the data in your database. So if you had a constraint enforced via a schema, you don't have a choice. You have to write some way to migrate that data so that it fits the new schema. You can kind of lie to yourself with validation and not change the historic data, and sometimes that is the case; you want to keep the old data and only prevent new data from being written in the old format. But if you need consistency, then you probably need a data migration regardless of which approach you take. STEPHANIE: Yeah, that definitely sounds like the more robust way to go about it for sure. JOËL: I have an article that I like to reference a lot by Hillel Wayne on Constructive Versus Predicative Data, which is basically looking at these two general approaches to enforcing data correctness and formalizing them a little bit. So do you try to enforce them based on the construction or the shape of the entity that you're creating, be that a database table, an object, a type, something like that? Or do you enforce it via some kind of predicate? So that could be a validation or other similar logic that runs kind of at runtime to enforce your constraints. STEPHANIE: That's interesting. I hadn't heard of those terms before, but I think they provide a lens through which you can look at the problem. Did the article end up suggesting different strategies for solving that problem, or was it more theoretical in different ways to look at it? JOËL: I think the article does two things. First, like you said, it gives us the words to talk about those approaches. And having those labels now, I start seeing them everywhere. I see them in databases, I see them in objects, I see them when doing types across a variety of languages. So that's already a huge win for me. I think you and I had done an episode a couple of months back where we talked about the value of having labels to put to ideas. And I think for me reading that article gave me those two labels. And all of a sudden, it really helped to make connections that I wasn't seeing before. The second thing that the article does is, I think, explore some of the limitations that each approach has and when you might want to use one versus another. The constructive approach, so using a schema, is more consistent because you know it is impossible for the program to create data that's in the wrong shape. That being said, not all constraints can be represented in a constructive manner, or it might be possible but really cumbersome. Also, sometimes it's not really invalid data; it's just sort of undesirable data. So you might want a looser schema. And let's say that you're storing some kind of intermediate state or some kind of raw input from another system that you might want to layer validations on top of, but you don't want to reject that data out of your database. You want that sort of incomplete or imperfect data in your system. Something that I find myself doing more and more these days when I create new tables is to really lock down the schema as much as possible. I think that might be contrary to maybe the way a lot of people in the community like to work. Some people might prefer to start with a very loose schema with no constraints and then work towards making things stricter as they explore the domain, and that's kind of the default that Rails has. If you're creating a new table, all columns, for example, are nullable by default. Personally, I will put a null false on every column and every migration that I make unless somebody can make a convincing case otherwise, and even then, I might try to think of is there any possible way that we could avoid that scenario and put that null false. Part of the reason for that is that it is much easier to loosen constraints on existing data than to tighten them afterwards. So if I have a column where no value is allowed to be null, and then later on we decide, you know what? It is okay for some of them to be null, I can change the requirement on that column, and I don't need to make any changes to the existing data. It just works. If the reverse happens, if I have a column that allows a bunch of nulls and then I want to make that column required, now I have to go and find a way to backfill all the empty spots in that column. And that could be a very challenging process. It might even be impossible. There might be some values there that it's just like, the user did not supply them at the time because we didn't ask for them. And now there's nothing we can put in there. So do you put in, like, unknown or not available? Then you have to ask yourself some really difficult questions about your data. STEPHANIE: Yeah, absolutely. I think I agree with you there. Another thing I like to do is provide default values for columns, especially ones where they can't be null, because, like you were saying, that helps me have a better understanding of just what is going on in the database. An issue I have seen come up involves a Boolean column where if a default value of false, for example, if that's what we're going with, is not encoded in the schema, you end up with potentially three values for a Boolean, which would be true, false, and null, and that I think has been -- JOËL: The infamous three-state Boolean. STEPHANIE: Yeah, exactly, the three-state problem, which is just inherently contradictory to what a Boolean is, to begin with. And I've definitely run into issues with that where you have to decide, or figure out, or write code to determine is null false? Is that what we mean here? It's not clear. But if you, like you said, locked it down at the beginning, provided those default values, that puts in those guardrails to prevent things from getting out of hand. JOËL: It also makes it easier for users of your database, application, whatever to interact with your code. I've run into this a lot when working with GraphQL APIs. And the default in many GraphQL server implementations is to make all fields nullable by default. When you build your schema, you have to add some extra things there to say, "This field is non-nullable," which means that a client that's now consuming it, anytime they deal with the data they need to check, is it present or not? You can't have the confidence that that data is there. And so it can force a lot of extra checks on the client. Or I guess you could just take it on faith and hope nothing breaks. STEPHANIE: Yeah, it's funny you mention that because I definitely think there's like spheres of impact. So as a developer, you maybe start having to write code that checks those kinds of things, like if it's null or not in your code. Then that can even extend to, like you said, your users or consumers of the API, who then have to contend with data that they have no control over. And I've been there too, and that can be frustrating as well. JOËL: We've talked a lot about data correctness and different ways to achieve it, different strategies. Why is this something that we care so much about? STEPHANIE: I think data correctness is really important from a developer experience perspective. And it's way easier to fix a bug in your code than it is to wrangle a lot of accumulated bad data. JOËL: Yeah, sometimes bad data is not fixable at all, and those are situations where you have a really bad day as a developer. STEPHANIE: Agreed. JOËL: Well, on that note, shall we wrap up? STEPHANIE: Let's wrap up. Show notes for this episode can be found at bikeshed.fm. JOËL: This show has been produced and edited by Mandy Moore. STEPHANIE: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review in iTunes. It really helps other folks find the show. JOËL: If you have any feedback for this or any of our other episodes, you can reach us @_bikeshed, or you can reach me @joelquen on Twitter. STEPHANIE: Or reach both of us at hosts@bikeshed.fm via email. JOËL: Thanks so much for listening to The Bike Shed, and we'll see you next week. ALL: Byeeeeeeee!!!!!!! ANNOUNCER: This podcast was brought to you by thoughtbot. thoughtbot is your expert design and development partner. Let's make your product and team a success.

head data fun profit personally slack api av rails slides constructive fp sql sdks mandy moore oo apm graphql boolean functional programming cfps quenneville aji graphql apis activerecord jenny shih ruby central stephanie yeah hillel wayne stephanie oh andy croll stephanie they

360: ActiveRecord Models

Play Episode Listen Later Nov 1, 2022 28:37

Fellow thoughtboter Sarah Lima joins Joël to discuss an issue Sarah had when she was doing a code review recently: making HTTP requests in an ActiveRecord model. Her concern with that approach was that a class was having too many responsibilities that would break the single-responsibility principle, and that it would make the class hard to maintain. Because the ActiveRecord layer is a layer that's meant to encapsulate business roles and data, her issue was that adding another responsibility on top of it would be too much. Her solution was to extract a class that would handle the whole HTTP request process. This episode is brought to you by Airbrake (https://airbrake.io/?utm_campaign=Q3_2022%3A%20Bike%20Shed%20Podcast%20Ad&utm_source=Bike%20Shed&utm_medium=website). Visit Frictionless error monitoring and performance insight for your app stack. SQL TRIM() (https://popsql.com/learn-sql/postgresql/how-to-trim-strings-in-postgresql) Iteration as an anti-pattern (https://thoughtbot.com/blog/iteration-as-an-anti-pattern) WET tests (https://thoughtbot.com/blog/the-case-for-wet-tests) thoughtbot code review guidelines (https://github.com/thoughtbot/guides/tree/main/code-review) Side effects in tests (https://thoughtbot.com/blog/simplify-tests-by-extracting-side-effects) Active Resource (https://github.com/rails/activeresource) Different strategies for 3rd party requests (https://thoughtbot.com/blog/testing-third-party-interactions) Transcript: JOËL: Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Joël Quenneville. And today, I'm joined by fellow thoughtboter Sarah Lima. SARAH: Happy to be here. JOËL: And together, we're here to share a little bit of what we've learned along the way. So, Sarah, what's new in your world? SARAH: Well, after a year and a half working on the same thoughtbot client, I have rolled off, and I have joined a new team. And I am learning a lot about not only a new codebase but learning to work with a new team. So that's always challenging, and this time it's not different. JOËL: What is something that you like to do when joining a new team to help smooth the onboarding process? SARAH: Well, I think especially getting to know people with one on ones. This time, I didn't do that right away because I had a bunch of time off scheduled right at the beginning of the project. But I did it right after I came back. And I'm learning a lot about my new colleagues, how they like to work, how they learn best. So, for instance, there are some people that like to learn and grow by reading blog posts, reading books, and there are other people that don't like that as much. JOËL: So when you joined the new project, you just reached out to all of these people and set up a few meetings just to get to know them. SARAH: Yeah, exactly. JOËL: That's really good. I've never done that on a project. And now that you've said it, it kind of seems obvious. Maybe I should do that moving forward to get to know new teammates. SARAH: Yeah. And I think it's easier on my project because it's a very small team. There are four of us thoughtboters, and there are just two client developers. So it was easier. JOËL: What about on the code side of things? Are there any tricks you like to do when you're first getting started in a new codebase? SARAH: Well, I think I really enjoy diving in right away, working on something small, and asking questions. I have also found it helpful in the past, especially on larger codebases, that someone that's experienced on a project gives me an overview showing me the quirks. And, of course, a good README is always a good thing to have, and during the process, always be updating the README. In this recent project, it was not different. I opened a lot of PRs to update the README. So that was good to have a PR right on your first day. JOËL: I love that. I think that's usually my goal when I start on a new project is to have a PR the first day that fixes anything in the setup script that has been broken since the last person onboarded or documentation that was wrong. SARAH: Yeah, absolutely. JOËL: It's always a strong first contribution. SARAH: Yeah. What about you, Joël? What's going on? What's new in your world? JOËL: I've been investigating flaky tests, and I ran across a wild bug this week. I had a test that would fail every now and then. And it was pulling some data from Postgres and then doing some transformations on it. And I couldn't figure out why it was failing. It was a complex query. So it was just pulling out not ActiveRecord objects but a raw array of values. At some point, I was putting a PUT statement in the code with the array of values I expected to get and the array I would actually get. And I was surprised to see that there is a field in there that is a float that was rounded to a different number of decimal places. I was like, that doesn't seem right. And so I was digging into it more, and I found out that this decimal value is from a timestamp that is in a file name for an mp4 video file name. And what is happening is that when we're querying the database, we're trying to extract the timestamp out of the file name by dropping the .mp4 file extension. And we're using the SQL TRIM function. Unfortunately, TRIM does not do whatever the original authors thought it does. It doesn't just remove that substring from the end, but instead, it will remove any of those characters, so in my case, any of dot, M, P, or 4 in any combination from the end of the string. So anytime that my timestamp ended in a four, any fours were just getting chopped off. So if it ended in 44.mp4, the 44 would also get removed, not just the .mp4, which meant that randomly whenever a timestamp happened to end in 4, my test would flake. SARAH: Wow. Do you have any idea how much time you spent debugging that? JOËL: Oh, probably took, I'd say, a day, two days. This is spread over a couple of debugging sessions. But eventually, finding that particular location for the bug probably took us a couple of days. In the end, the bug fix for this is just a couple of lines, a couple of days work, and the diff is only a few lines. But I'm sure that the discussion on the PR is going to be really interesting. There's probably going to be a description that is a lot longer than the actual diff. SARAH: Yeah, 100%. [laughs] JOËL: Have you run across any interesting PRs on your new project? SARAH: Yeah, I did. In fact, I recently reviewed a PR that had three interesting main issues that I wanted to address. And I wanted to lead the person that was working on it to a slightly better solution. So the three issues I saw were that the tests that were added were very DRY, so that was making everything a bit difficult to understand. The second one was that I saw one of the ActiveRecord classes was making HTTP requests, and that didn't sound like a good idea to me. JOËL: That is unusual. SARAH: Yes. The third one was that there were a lot of collections being built iteratively where another innumerable method would be a better fit, such as map instead of an each call. JOËL: Oh, this is a classic situation where you're just using each to go through and transform something, and you've got some sort of external array that you're mutating as part of the each. SARAH: Yes. JOËL: There's a great thought article, I believe, by Joe Ferris on Iteration as an Anti-pattern. SARAH: I think it's by Mike Burns. And I have referred to that article. In fact, I had very good articles for two of these three problems. I referred to a bunch of articles about WET tests as opposed to DRY tests, like how striving for tests that are DRY is not a good idea as opposed to telling a whole story in your tests. And I referred to that other article how iteratively building a collection can be an anti-pattern by Mike Burns. But the second issue about HTTP requests I didn't have anything to refer to. Maybe we should write one. JOËL: This reminds me that in the thoughtbot Slack, we have a custom emoji for you should write a blog post about that. And this would probably be a good time to use it. SARAH: Yes. So, Joël, how do you typically handle a PR that is maybe too long, and you have a lot of concerns about it? And how do you handle delivering that feedback? JOËL: Oh, that is a challenge. I've definitely done it poorly in the past. And I think the wrong way to go about that situation is to go thoroughly through the PR and leave 50, 60 comments. That is overwhelming for the other person. And they're going to have a really bad day when they see 50 comments come through. And there's so much that they can't really address the main things you were talking about anyway. So what I generally try to do, and it's kind of nice now that GitHub doesn't immediately publish your comments, is if I realize...like I start putting some more detailed comments, and then I realize, oh, there's going to be a lot, zoom out a little bit, and try to find are there some higher level trends that I can talk about? And maybe even just summarize in a larger comment at the bottom and say, "Hey, I see some larger structural issues," or "This PR is leaning very heavily on a technique that I think is maybe not the best use here. Maybe we should discuss that," instead of digging into maybe the actual implementation details of the code. SARAH: Yeah, funny, you should mention that. I have recently also started doing that, using the summary version of GitHub reviews. And I used to just go file by file and leaving comments right away. And I'm thinking that this is not a good idea, especially when the PR is long. So I think another thing I would do is also call the person to pair and ask questions and understand where the person is coming from and also explain what are your concerns and how you both can get to a better place with that PR. JOËL: That's really important. You have to remember there's another person on the other end of this. I love the idea of reaching out to them directly. Especially if there's a larger conversation to be had around approach or implementation, it's often easier to resolve those directly rather than back and forth in GitHub comments. So you mentioned situations where the PR is really long. Have you ever had to push back on that in some way? SARAH: Yes. Especially when I saw, whoa, that's going to be difficult to understand, that's going to be difficult to review. And I have reached out to the person to say, "Hey, what about we split that PR in two?" Of course thinking about splitting the PR in a way that makes sense, in a way that still delivers our users' value as soon as possible. JOËL: I've been in situations like that where it's a really long PR, and the person has already invested a lot of work into it. And maybe it's even gone through a round of reviews. It feels almost too late to ask them to split up the work. But then I've actually regretted not doing that because there's so much complexity going on that then it doesn't work, or there are some bugs in it. We struggle to ship this, or it might just have to go through so many rounds of review and re-review and re-review. And because the PR is so long, it's a huge commitment for me to re-review it every time. So there are situations I've been in where I wish that before even looking at the code at all, I was like, this is too long. We need to either slim down the story of what's being done. Because sometimes that's what happens is that the ticket is not well-defined, and someone goes in and just sort of keeps adding more code. And it becomes a bit of a big ball of mud. So, either helping to refine the ticket first or splitting the PR rather than actually looking at the code. SARAH: Yeah, and pairing often can also help with that. So especially as consultants, our clients may ask us to work on different projects, and you work alone. And you may have tight deadlines, but I think it's always helpful to find time anyway to help your colleagues as well. JOËL: I like that. I think there's a lot of value in the work that we do, where we collaborate with others in addition to whatever we do solo. So, oftentimes, it's great to pair with people at a client where possible to become involved in the code review process to even get involved in maybe some of the more broader system design conversations, sprint planning. All of those things are really good to jump into more than just getting siloed into working on just a solo feature. SARAH: Yes, 100%. MID-ROLL AD: Debugging errors can be a developer's worst nightmare...but it doesn't have to be. Airbrake is an award-winning error monitoring, performance, and deployment tracking tool created by developers for developers that can actually help cut your debugging time in half. So why do developers love Airbrake? It has all of the information that web developers need to monitor their application - including error management, performance insights, and deploy tracking! Airbrake's debugging tool catches all of your project errors, intelligently groups them, and points you to the issue in the code so you can quickly fix the bug before customers are impacted. In addition to stellar error monitoring, Airbrake's lightweight APM helps developers to track the performance and availability of their application through metrics like HTTP requests, response times, error occurrences, and user satisfaction. Finally, Airbrake Deploy Tracking helps developers track trends, fix bad deploys, and improve code quality. Since 2008, Airbrake has been a staple in the Ruby community and has grown to cover all major programming languages. Airbrake seamlessly integrates with your favorite apps to include modern features like single sign-on and SDK-based installation. From testing to production, Airbrake notifiers have your back. Your time is valuable, so why waste it combing through logs, waiting for user reports, or retrofitting other tools to monitor your application? You literally have nothing to lose. Head on over to airbrake.io/try/bikeshed to create your FREE developer account today! JOËL: So one of the things you mentioned that stood out for you when you were doing some code review recently was making HTTP requests in an ActiveRecord model. Why is that something that sort of caught your eyes, maybe an area to push back on in a particular design? SARAH: That's a good question. My concern with that approach was that our class was having too many responsibilities that would break the SRP principle, the single-responsibility principle, and that would make our class hard to maintain. So the ActiveRecord layer is a layer that's meant to encapsulate business roles and data. So I was worried that adding another responsibility on top of it would be too much. So my idea was that we would extract a class that would handle the whole HTTP request process. JOËL: Yeah, I feel like my instincts typically when I've done third-party integrations is that the ActiveRecord class should not know about the external internet world. It knows about the database. It knows about some of its core model functionality but that knowing about the internet world is somebody else's responsibility and that, ideally, the direction of dependency should flow the other way. So maybe the class that makes an external request knows about the ActiveRecord object if it needs to let's say, instantiate an instance of that model using data from an external request. Or maybe it's even some third-party thing; maybe it's their controller that knows how to make or that will ask another object to make a request to some API and might also make a request to the model and ask it for some database data and then combine those two together. But that the ActiveRecord object only knows about that database area of responsibility and doesn't know that other things are also happening in the system. SARAH: Absolutely. And I was also thinking that that class would have a difficult test to write. So a good idea is to separate our code that is side-effectful into their own classes, and that makes our tests so much easier. JOËL: I actually wrote an article on the topic where one of my realizations at some point was that a lot of the pain points in code are what functional programmers would call side effects, so things like HTTP requests. And these are often things where we need to stub or do other things. And so isolating them as much as possible often simplifies our tests. SARAH: Yeah, certainly. And I refer to that article every time I have the chance. JOËL: Have you encountered the general concept of layered architectures, or hexagonal architectures, or things like that in the world of Rails or maybe elsewhere? SARAH: Not hexagonal architecture. I have heard about it, but I haven't dived into it yet. Can you give us an overview? JOËL: So I've also not worked with an actual hexagonal architecture. But the general idea, I guess, of layered architectures is that you build your code in a variety of layers, and different layers don't have access to or don't know about the ones...and I forget in this model if it's above or below, let's say it's below. So the inner layers don't know about the outer layers, but the outer layers can know about anything below them. And so if the core of your app is the database, your database is most definitely not knowing about anything outside of just its data. And your ActiveRecord models that sit on top of that know about the database, but they don't know if they're being fronted by a web application, or a command line, or anything else. And then, above that, you might have more of a business process layer that knows about the database. It might know about how to make some external requests, but it doesn't know about anything above that. And then, maybe at the final layer, you've got an application layer that handles things like controllers and interactions with users of the site. The core idea is that you split it into layers, and the higher-up layers know about everything below them, but no layer knows about what's above it. I feel like we're loosely applying that to the situation here with ActiveRecord in that it feels like the ActiveRecord layer if you will, shouldn't really know about third-party API requests. SARAH: So, one exception to that is the ActiveResource approach that connects our business objects to REST services. So if you have an external website and you want to connect it via HTTP, you can do it using Rails ActiveResource. JOËL: That is interesting because it functions like an ActiveRecord object, but instead of being backed by the database, it's backed by some kind of API. I almost wonder if...let's refactor our mental model here. And instead of saying that HTTP belongs in a separate layer that's higher up, maybe, in this case, it's almost like a sibling layer. So your ActiveRecord models know about the database, and they make database requests in ActiveResource, or I think there are some gems that provide similar behavior. It might be backed by a particular API, but neither of them should know about the other. So maybe an ActiveResource model should not be making database requests. SARAH: Yes, I like that line of thought. JOËL: I guess the question then becomes, what about interactions between the two where you want to, I don't know, have some kind of association? You know, I don't think I've ever used ActiveResource on a project. SARAH: I did once when trying to work with something close to microservice architecture. So we had a monolith, and we built a small service that was also in Rails, and we needed to consume the data that was stored in the monolith. JOËL: And did you like that approach? SARAH: Yeah. I think in that specific scenario, it was very productive. And I enjoyed a lot the API that Rails provided me via ActiveResources. JOËL: Did you ever have to mix ActiveResource models and ActiveRecord models? SARAH: No, I didn't; thankfully, not. I have never thought about that. JOËL: So maybe in most applications, those two will just sort of naturally fall into maybe separate parts of the app, and they don't need to interact that much. SARAH: Yeah, I think that will be the case. So mixing two of those subjects we're talking about here, that's testing and HTTP requests; we've been having a discussion in our project about the usage of VCR. That's a gem that records your HTTP requests interactions and replays them during tests. We've been discussing if using it is a good idea or not because we've been having issues with cassettes, that's one of VCR's concepts when these cassettes are not valid anymore. So do you have any thoughts on the subject? Maybe that will make a whole episode. JOËL: We could definitely do a whole episode, I think, on testing third-party APIs. VCR is one of multiple different strategies that can be used to not make actual real network requests in your tests which brings some stability. There are also some downsides to it. I have found, in general, that over time, cassettes become brittle. So the idea of VCR is really cool. In practice, I think I've found that a few hand-rolled Webmock stubs usually do the job better for my needs. SARAH: Yeah, I'll be interested in hearing that episode because, at least in my project, we have a lot of HTTP requests to external services, and they return a lot of information. I'm wondering if just dealing with that with Webmock would be too much work. JOËL: One of the really useful things about VCR is that you can just make your request from anywhere, and it will just completely handle it. In some ways, though, I think it maybe hides some of that test pain that we were talking about earlier and allows you to sort of put HTTP in a lot of places that maybe you don't want it to. And by allowing yourself to feel a little bit of that test pain, you can more easily notice the places where maybe an object should not be making a request. Or the actual HTTP logic can be moved to a concentrated place where all the HTTP is done together. And then only that object will need unit tests that actually need to mock the network, and most of your objects are fine. Where it gets interesting is more for things like integration tests, where now you're doing a lot of interactions, and you might have quite a few background requests that need to be made. SARAH: I'm looking forward to the whole episode on this subject because I feel there's so much to talk about. JOËL: There really is. I have a blog post that sort of summarizes a few different common categories of approaches to testing third-party requests, which might be different depending on whether you're doing a unit test or an integration test. But I grouped common solutions into four different categories. We'll make sure to link that in the show notes. So we've been talking a lot about testing. I'm curious when you review PR, do you start with the tests, maybe read through the tests first, and then the implementation? SARAH: That's a good question. I have never thought about starting with tests. I think I'm going to give that a try anytime. But I just start reviewing them like by the first file that comes up. [laughs] JOËL: I'm the same. I normally just do them in order. I have occasionally tried to do a test first, and that is sometimes interesting. Sometimes you read the test and, especially when you don't know what the implementation is going to be, you're like, why is this in the test? And then you jump to the implementation like, oh, that's what's going on. Well, thank you so much, Sarah, for joining us on this whirlwind tour of code review, design of objects, and interacting with HTTP and testing. SARAH: My pleasure. JOËL: Where can people find you online if they would like to follow your work? SARAH: I'm on Twitter @sarahlima_rb. JOËL: We'll make sure to link that in the show notes. And with that, let's wrap up. The show notes for this episode can be found at bikeshed.fm. This show is produced and edited by Mandy Moore. If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review in iTunes. It really helps other folks find the show. If you have any feedback, you can reach us at @_bikeshed, or reach me at @joelquen on Twitter, or at hosts@bikeshed.fm via email. Thank you so much for listening to The Bike Shed, and we'll see you next week. Byeeeeeee!!!!!! ANNOUNCER: This podcast was brought to you by thoughtbot. thoughtbot is your expert design and development partner. Let's make your product and team a success.

head pr fellow models slack api github dry apis wet rails trim prs vcr sdks mandy moore iteration apm readme postgres srp mike burns quenneville activerecord sarah yeah byeeeeeee sarah yes sarah so sarah well sarah no sarah absolutely activeresource

359: Serializers

Play Episode Listen Later Oct 25, 2022 44:10

Chris Toomey is back! (For an episode.) He talks about what he's been up to since handing off the reins to Joël. He's been playing around with something at Sagewell that he enjoys. At the core of it? Serializers. Primalize gem (https://github.com/jgaskins/primalize) Derek's talk on code review (https://www.youtube.com/watch?v=PJjmw9TRB7s) Inertia.js (https://inertiajs.com/) Phantom types (https://thoughtbot.com/blog/modeling-currency-in-elm-using-phantom-types) io-ts (https://gcanti.github.io/io-ts/) dry-rb (https://dry-rb.org/) parse don't validate (https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/) value objects (http://wiki.c2.com/?ValueObject) broader perspective on parsing (https://thoughtbot.com/blog/a-broader-take-on-parsing) Enumerable#tally (https://medium.com/@baweaver/ruby-2-7-enumerable-tally-a706a5fb11ea) RubyConf mini (https://www.rubyconfmini.com/) where.missing (https://boringrails.com/tips/activerecord-where-missing-associations) Transcript: JOËL: Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Joël Quenneville. And today, I'm joined by a very special guest, former host Chris Toomey. CHRIS: Hi, Joël. Thanks for having me. JOËL: And together, we're here to share a little bit of what we've learned along the way. So, Chris, what's new in your world? CHRIS: Being on this podcast is new in my world, or everything old is new again, or something along those lines. But, yeah, thank you so much for having me back. It's a pleasure. Although it's very odd, it feels somehow so different and yet very familiar. But yeah, more generally, what's new in my world? I think this was probably in development as I was winding down my time as a host here on The Bike Shed, but I don't know that I ever got a chance to talk about it. There has been a fun sort of deep-in-the-weeds technical thing that we've been playing around with at Sagewell that I've really enjoyed. So at the core of it, we have serializers. So we take some data structures in our Ruby on Rails code base, and we need to serialize them to JSON to send them to the front end. In our case, we're using Inertia, so it's not quite a JSON API, but it's fine to think about it in that way for the context of this discussion. And what we were finding is our front end has TypeScript. So we're writing Svelte, which is using TypeScript. And so we're stating or asserting that the types like, hey, we're going to get this data in from the back end, and it's going to have this shape to it. And we found that it was really hard to keep those in sync to keep, like, what does the user mean on the front end? What's the data that we're going to get? It's going to have a full name, which is a string, except sometimes that might be null. So how do we make sure that those are keeping up to date? And then we had a growing number of serializers on the back end and determining which serializer we were actually using, and it was just...it was a mess, to put it lightly. And so we had explored a couple of different options around it, and eventually, we found a library called Primalize. So Primalize is a Ruby library. It is for writing JSON serializers. But what's really interesting about it is it has a typing layer. It's like a type system sort of thing at play. So when you define a serializer in Primalize, instead of just saying, here are the fields; there is an ID, a name, et cetera, you say, there is an ID, and it is a string. There is a name, and it is a string, or an optional string, which is the even more interesting bit. You can say array. You can say object. You can say an enum of a couple of different values. And so we looked at that, and we said, ooh, this is very interesting. Astute listeners will know that this is probably useless in a Ruby system, which doesn't have types or a compilation step or anything like that. But what's really cool about this is when you use a Primalize serializer, as you're serializing an object, if there is ever a type mismatch, so the observed type at runtime and the authored type if those ever mismatch, then you can have some sort of notification happen. So in our case, we configured it to send a warning to Sentry to say, "Hey, you said the types were this, but we're actually seeing this other thing." Most often, it will be like an Optional, a null sneaking through, a nil sneaking through on the Ruby side. But what was really interesting is as we were squinting at this, we're like, huh, so now we're going to write all this type information. What if we could somehow get that type information down to the front end? So I had a long weekend, one weekend, and I went away, and I wrote a bunch of code that took all of those serializers, ran through them, and generated the associated TypeScript interfaces. And so now we have a build step that will essentially run that and assert that we're getting the same thing in CI as we have committed to the codebase. But now we have the generated serializer types on the front end that match to the used serializer on the back end, as well as the observed run-time types. So it's a combination of a true compilation step type system on the front end and a run-time type system on the back end, which has been very, very interesting. JOËL: I have a lot of thoughts here. CHRIS: I figured you would. [laughs] JOËL: But the first thing that came to mind is, as a consultant, there's a scenario with especially smaller startups that generally concerns me, and that is the CTO goes away for a weekend and writes a lot of code... CHRIS: [laughs] JOËL: And brings in a new system on Monday, which is exactly what you're describing here. How do you feel about the fact that you've done that? CHRIS: I wasn't ready to go this deep this early on in this episode. JOËL: [laughs] CHRIS: But honestly, that is a fantastic question. It's a thing that I have been truly not struggling with but really thinking about. We're going to go on a slight aside here, but I am finding it really difficult to engage with the actual day-to-day coding work that we're doing and to still stay close to the codebase and not be in the way. There's a pattern that I've seen happen a number of times now where I pick up a piece of work that is, you know, one of the tickets at the top of the backlog. I start to work on it. I get pulled into a meeting, then another meeting, then three more meetings. And suddenly, it's three days later. I haven't completed this piece of work that was defined to be the next most important piece of work. And suddenly, I'm blocking the team. JOËL: Hmmm. CHRIS: So I actually made a rule that I'm not allowed to own critical path work, which feels weird because it's like, I want to be engaged with that work. So the counterpoint to that is I'm now trying to schedule pairing sessions with each of the developers on the team once a week. And in that time, I can work on that sort of stuff with them, and they'll then own it and run with it. So it makes sure that I'm not blocking on those sorts of things, but I'm still connected to the core work that we're doing. But the other thing that you're describing of the CTO goes away for the weekend and then comes back with a new harebrained scheme; I'm very sensitive to that, having worked on; frankly, I think the same project. I can think of a project that you and I worked on where we experienced this. JOËL: I think we're thinking of the same project. CHRIS: So yes. Like, I'm scarred by that and, frankly, a handful of experiences of that nature. So we actually, I think, have a really healthy system in place at Sagewell for capturing, documenting, prioritizing this sort of other work, this developer-centric work. So this is the feature and bug work that gets prioritized and one list over here that is owned by our product manager. Separately, the dev team gets to say, here are the pain points. Here's the stuff that keeps breaking. Here are the things that I wish was better. Here is the observability hard-to-understand bits. And so we have a couple of different systems at play and recurring meetings and sort of unique ceremonies around that, and so this work was very much a fallout of that. It was actually a recurring topic that we kept trying a couple of different stabs at, and we never quite landed it. And then I showed up this one Monday morning, and I was like, "I found a thing; what do we think?" And then, critically, from there, I made sure I paired with other folks on the team as we pushed on the implementation. And then, actually, I mentioned Primalize, the library that we're using. We have now since deprecated Primalize within the app because we kept just adding to it so much that eventually, we're like, at this point, should we own this stuff? So we ended up rewriting the core bits of Primalize to better fit our use cases. And now we've actually removed Primalize, wonderful library. I highly recommend it to anyone who has that particular use case but then the additional type generation for the front end. Plus, we have some custom types within our app, Money being the most interesting one. We decided to model Money as our first-class consideration rather than just letting JavaScript have the sole idea of a number. But yes, in a very long-winded way, yes, I'm very sensitive to the thing you described. And I hope, in this case, I did not fall prey to the CTO goes away for the weekend and made a thing. JOËL: I think what I'm hearing is the key difference here is that you got buy-in from the team around this idea before you went out and implemented it. So you're not off doing your own things disconnected from the team and then imposing it from on high. The team already agreed this is the thing we want to do, and then you just did it for them. CHRIS: Largely, yes. Although I will say there are times that each developer on the team, myself included, have sort of gone away, come back with something, and said, "Hey, here's a WIP PR exploring an area." And there was actually...I'm forgetting what the context was, but there was one that happened recently that I introduced. I was like; I had to do this. And the team talked me out of it, and I ended up closing that PR. Someone else actually made a different PR that was an alternative implementation. I was like, no, that's better; we should absolutely do that. And I think that's really healthy. That's a hard thing to maintain but making sure that everyone feels like they've got a strong voice and that we're considering all of the different ways in which we might consider the work. Most critically, you know, how does this impact users at the end of the day? That's always the primary consideration. How do we make sure we build a robust, maintainable, observable system, all those sorts of things? And primarily, this work should go in that other direction, but I also don't want to stifle that creative spark of I got this thing in my head, and I had to explore it. Like, we shouldn't then need to never mind, throw away the work, put it into a ticket. Like, for as long as we can, that more organic, intuitive process if we can retain that, I like that. Critically, with the ability for everyone to tell me, "No, this is a bad idea. Stop it. What are you doing?" And that has happened recently. I mean, they were kinder about it, but they did talk me out of a bad idea. So here we are. JOËL: So you showed up on Monday morning, not with telling everyone, "Hey, I merged this thing over the weekend." You're showing up with a work-in-progress PR. CHRIS: Yes, definitely. I mean, everything goes through a PR, and everything has discussion and conversation around it. That's a strong, strong like Derek Prior's wonderful talk Building a Culture of Code Review. I forget the exact name of it. But it's one of my favorite talks in talking about the utility of code review as a way to share ideas and all of those wonderful things. So everything goes through code review, and particularly anything that is of that more exploratory architectural space. Often we'll say any one review from anyone on the team is sufficient to merge most things but something like that, I would want to say, "Hey, can everybody take a look at this? And if anyone has any reservations, then let's talk about it more." But if I or anyone else on the team for this sort of work gets everybody approving it, then cool, we're good to go. But yeah, code review critical, critical part of the process. JOËL: I'm curious about Primalize, the gem that you mentioned. It sounds like it's some kind of validation layer between some Ruby data structure and your serializers. CHRIS: It is the serializer, but in the process of serializing, it does run-time type validation, essentially. So as it's accessing, you know, you say first name. You have a user object. You pass it in, and you say, "Serializer, there's a first name, and it's a string." It will call the first name method on that user object. And then, it will check that it has the expected type, and if it doesn't, then, in our case, it sends to Sentry. We have configured it...it's actually interesting. In development and test mode, it will raise for a type mismatch, and in production mode, it will alert Sentry so you can configure that differently. But that ends up being really nice because these type mismatches end up being very loud early on. And it's surprisingly easy to maintain and ends up telling us a lot of truths about our system because, really, what we're doing is connecting data from many different systems and flowing it in and out. And all of the inputs and outputs from our system feel very meaningful to lock down in this way. But yeah, it's been an adventure. JOËL: It seems to me there could almost be two sets of types here, the inputs coming into Primalize from your Ruby data structures and then the outputs that are the actual serialized values. And so you might expect, let's say, an integer on the Ruby side, but maybe at the serialization level, you're serializing it to a string. Do you have that sort of conversion step as part of your serializers sometimes, or is the idea that everything's already the right type on the Ruby side, and then we just, like, to JSON it at the end? CHRIS: Yep. Primalize, I think, probably works a little closer to what you're describing. They have the idea of coercions. So within Primalize, there is the concept of a timestamp; that is one of the types that is available. But a timestamp is sort of the union of a date, a time, or I think they might let through a string; I'm not sure if there is as well. But frankly, for us, that was more ambiguity than we wanted or more blurring across the lines. And in the implementation that we've now built, date and time are distinct. And critically, a string is not a valid date or time; it is a string, that's another thing. And so there's a bunch of plumbing within the way you define the serializers. There are override methods so that you can locally within the serializer say, like, oh, we need to coerce from the shape of data into this other shape of data, even little like in-line proc, so we can do it quickly. But the idea is that the data, once it has been passed to the serializer, should be up the right shape. And so when we get to the type assertion part of the library, we expect that things are in the asserted type and will warn if not. We get surprisingly few warnings, which is interesting now. This whole process has made us pay a little more intention, and it's been less arduous simultaneously than I would have expected because like this is kind of a lot of work that I'm describing. And yet it ends up being very natural when you're the developer in context, like, oh, I've been reading these docs for days. I know the shape of this JSON that I'm working with inside and out, and now I'll just write it down in the serializer. It's very easy to do in that moment, and then it captures it and enforces it in such a useful way. As an aside, as I've been looking at this, I'm like, this is just GraphQL, but inside out, I'm pretty sure. But that is a choice that we have made. We didn't want to adopt the whole GraphQL thing. But just for anyone out there who is listening and is thinking, isn't this just GraphQL but inside out? Kind of. Yes. JOËL: I think my favorite part of GraphQL is the schema, which is not really the selling point for GraphQL, you know, like the idea that you can traverse the graph and get any subset of data that you want and all that. I think I would be more than happy with a REST API that has some kind of schema built around it. And someone told me that maybe what I really just want is SOAP, and I don't know how to feel about that comment. CHRIS: You just got to have some XML, and some WSDLs, and other fun things. I've heard people say good things about SOAP. SOAP seems like a fine idea. If anything, I think a critical part of this is we don't have a JSON API. We have a very tightly coupled front end and back end, and a singular front end, frankly. And so that I think naturally...that makes the thing that I'm describing here a much more comfortable fit. If we had multiple different downstream clients that we're trying to consume from the same back end, then I think a GraphQL API or some other structured JSON schema, whatever it is type of API, and associated documentation and typing layer would be probably a better fit. But as I've said many a time on this here, Bike Shed, Inertia is one of my favorite libraries or frameworks (They're probably more of a framework.) one of my favorite technological approaches that I have ever found. And particularly in buildings Sagewell, it has allowed us to move so rapidly the idea that changes are, you know, one fell swoop changes everything within the codebase. We don't have to think about syncing deploys for the back end and the front end and how to coordinate across them. Our app is so much easier to understand by virtue of that architecture that Inertia implies. JOËL: So, if I understand correctly, you don't serialize to JSON as part of the serializers. You're serializing directly to JavaScript. CHRIS: We do serialize to JSON. At the end of the day, Inertia takes care of this on both the Rails side and the client side. There is a JSON API. Like, if you look at the network inspector, you will see XHR requests happening. But critically, we're not doing that. We're not the ones in charge of it. We're not hitting a specific endpoint. It feels as an application coder much closer to a traditional Rails app. It just happens to be that we're writing our view layer. Instead of an ERB, we're writing them in Svelte files. But otherwise, it feels almost identical to a normal traditional Rails app with controllers and the normal routing and all that kind of stuff. JOËL: One thing that's really interesting about JSON as an interchange format is that it is very restrictive. The primitives it has are even narrower than, say, the primitives that Ruby has. So you'd mentioned sending a date through. There is no JSON date. You have to serialize it to some other type, potentially an integer, potentially a string that has a format that the other side knows how it's going to interpret. And I feel like it's those sorts of richer types when we need to pass them through JSON that serialization and deserialization or parsing on the other end become really interesting. CHRIS: Yeah, I definitely agree with that. It was a struggling point for a while until we found this new approach that we're doing with the serializers in the type system. But so far, the only thing that we've done this with is Money. But on the front end, a while ago, we introduced a specific TypeScript type. So it's a phantom type, and I believe I'm getting this correct. It's a phantom type called Cents, C-E-N-T-S. So it represents...I'm going to say an integer. I know that JavaScript doesn't have integers, but logically, it represents an integer amount of cents. And critically, it is not a number, like, the lowercase number in the type system. We cannot add them together. We can't -- JOËL: I thought you were going to say, NaN. CHRIS: [laughs] It is not a number. I saw a n/a for not applicable somewhere in the application the other day. I was like, oh my God, we have a NaN? It happened? But it wasn't, it was just n/a, and I was fine. But yeah, so we have this idea of Cents within the application. We have a money input, which is a special input designed exactly for this. So to a user, it is formatted to look like you're entering dollars and cents. But under the hood, we are bidirectionally converting that to the integer amount of cents that we need. And we strictly, within the type system, those are cents. And you can't do math on Cents unless you use a special set of helper functions. You cannot generate Cents on the fly unless you use a special set of helper functions, the constructor functions. So we've been really restrictive about that, which was kind of annoying because a lot of the data coming from the server is just, you know, numbers. But now, with this type system that we've introduced on the Ruby side, we can assert and enforce that these are money.new on the Ruby side, so using the Money gem. And they come down to the front end as capital C Cents in the type system on the TypeScript side. So we're able to actually bind that together and then enforce proper usage sort of on both sides. The next step that we plan to do after that is dates and times. And those are actually almost weirder because they end up...we just have to sort of say what they are, and they will be ISO 8601 date and time strings, respectively. But we'll have functions that know this is a date string; that's a thing. It is, again, a phantom type implemented within our TypeScript type system. But we will have custom functions that deal with that and really constrain...lock ourselves down to only working with them correctly. And critically, saying that is the only date and time format that we work with; there is no other. We don't have arbitrary dates. Is this a JSON date or something else? I don't know; there are too many date syntaxes. JOËL: I like the idea of what you're doing in that it sounds like you're very much narrowing that sort of window of where in the stack the data exists in the sort of unstructured, free-floating primitives that could be misinterpreted. And so, at this point, it's almost narrowed to the point where it can't be touched by any user or developer-written code because you've pushed the boundaries on the Rails side down and then on the JavaScript side up to the point where the translation here you define translations on one side or, I guess, a parser on one side and a serializer on the other. And they guarantee that everything is good up until that point. CHRIS: Yep, with the added fun of the runtime reflection on the Ruby side. So it's an interesting thing. Like, TypeScript actually has similar things. You can say what the type is all day long, and your code will consistently conform to that asserted type. But at the end of the day, if your JSON API gets in some different data...unless you're using a library like io-ts, is one that I've looked at, which actually does parsing and returns a result object of did we parse to the thing that you wanted or did we get an error in that data structure? So we could get to that level on the client side as well. We haven't done that yet largely because we've essentially pushed that concern up to the Ruby layer. So where we're authoring the data, because we own that, we're going to do it at that level. There are a bunch of benefits of defining it there and then sort of reflecting it down. But yeah, TypeScript, you can absolutely lie to yourself, whereas Elm, a language that I know you love dearly, you cannot lie to yourself in Elm. You've got to tell the truth. It's the only option. You've got to prove it. Whereas in TypeScript, you can just kind of suggest, and TypeScript will be like, all right, cool, I'll make sure you stay honest on that, but I'm not going to make you prove it, which is an interesting sort of set of related trade-offs there. But I think we found a very comfortable resting spot for right now. Although now, we're starting to look at the edges of the Ruby system where data is coming in. So we have lots of webhooks and other external partners that we're integrating with, and they're sending us data. And that data is of varying shapes. Some will send us a payload with the word amount, and it refers to an integer amount of cents because, of course, it does. Some will send us the word amount in their payload, and it will be a floating amount of dollars. And I get a little sad on those days. But critically, our job is to make sure all of those are the same and that we never pass dollars as cents or cents as dollars because that's where things go sad. That is job number one at Sagewell in the engineering team is never get the decimal place wrong in money. JOËL: That would be a pretty terrible mistake to make. CHRIS: It would. I mean, it happens. In fintech, that problem comes up a lot. And again, the fact that...I'm honestly surprised to see situations out there where we're getting in floating point dollars. That is a surprise to me because I thought we had all agreed sort of as a community that it was integer cents but especially in a language that has integers. JavaScript, it's kind of making it up the whole time. But Ruby has integers. JSON, I guess, doesn't have integers, so I'm sort of mixing concerns here, but you get the idea. JOËL: Despite Ruby not having a static type system, I've found that generally, when I'm integrating with a third-party API, I get to the point where I want something that approximates like Elm's JSON decoders or io-ts or something like that. Because JSON is just a big blob of data that could be of any shape, and I don't really trust it because it's third-party data, and you should not trust third parties. And I find that I end up maybe cobbling something together commonly with like a bunch of usage of hash.fetch, things like that. But I feel like Ruby doesn't have a great approach to parsing and composing these validators for external data. CHRIS: Ruby as a language certainly doesn't, and the ecosystem, I would say, is rather limited in terms of the options here. We have looked a bit at the dry-rb stack of gems, so dry-validation and dry-schema, in particular, both offer potentially useful aspects. We've actually done a little bit of spiking internally around that sort of thing of, like, let's parse this incoming data instead of just coercing to hash and saying that it's got probably the shape that we want. And then similarly, I will fetch all day instead of digging because I want to be quite loud when we get it wrong. But we're already using dry-monads. So we have the idea of result types within the system. We can either succeed or fail at certain operations. And I think it's just a little further down the stack. But probably something that we will implement soon is at those external boundaries where data is coming in doing some form of parsing and validation to make sure that it conforms to unknown data structure. And then, within the app, we can do things more cleanly. That also would allow us to, like, let's push the idea that this is floating point dollars all the way out to the edge. And the minute it hits our system, we convert it into a money.new, which means that cents are properly handled. It's the same type of money or dollar, same type of currency handling as everywhere else in the app. And so pushing that to the very edges of our application is a very interesting idea. And so that could happen in the library or sort of a parsing client, I guess, is probably the best way to think about it. So I'm excited to do that at some point. JOËL: Have you read the article, Parse, Don't Validate? CHRIS: I actually posted that in some code review the other day to one of the developers on the team, and they replied, "You're just going to quietly drop one of my favorite articles of all time in code review?" [laughs] So yes, I've read it; I love it. It's a wonderful idea, definitely something that I'm intrigued by. And sort of bringing dry-monads into Ruby, on the one hand, feels like a forced fit and yet has also been one of the other, I think strongest sort of architectural decisions that we've made within the application. There's so much imperative work that we ended up having to do. Send this off to this external API, then tell this other one, then tell this other one. Put the whole thing in a transaction so that our local data properly handles it. And having dry-monads do notation, in particular, to allow us to make that manageable but fail in all the ways it needs to fail, very expressive in its failure modes, that's been great. And then parse, don't validate we don't quite do it yet. But that's one of the dreams of, like, our codebase really should do that thing. We believe in that. So let's get there soon. JOËL: And the core idea behind parse, don't validate is that instead of just having some data that you don't trust, running a check on it and passing that blob of now checked but still untrusted data down to the next person who might also want to check it. Generally, you want to pass it through some sort of filter that will, one, validate that it's correct but then actually typically convert it into some other trusted shape. In Ruby, that might be something like taking an amorphous blob of JSON and turning it into some kind of value object or something like that. And then anybody downstream that receives, let's say, money object can trust that they're dealing with a well-formed money value as opposed to an arbitrary blob of JSON, which hopefully somebody else has validated, but who knows? So I'm going to validate it again. CHRIS: You can tell that I've been out of the podcasting game for a while because I just started responding to yes; I love that blog post without describing the core premise of it. So kudos to you, Joël; you are a fantastic podcast host over there. I will say one of the things you just described is an interesting...it's been a bit of a struggle for us. We keep sort of talking through what's the architecture. How do we want to build this application? What do we care about? What are the things that really matter within this codebase, and then what is all the other stuff? And we've been good at determining the things that really matter, thinking collectively as a group, and I think coming up with some novel, useful, elegant...I'm saying too many positive adjectives for what we're doing. But I've been very happy with sort of the thing that we decide. And then there's the long-tail work of actually propagating that change throughout the rest of the application. We're, like, okay, here's how it works. Every incoming webhook, we now parse and yield a value object. That sentence that you just said a minute ago is exactly what I want. That's like a bunch of work. It's particularly a bunch of work to convert an existing codebase. It's easy to say, okay, from here forward, any new webhooks, payloads that are coming in, we're going to do in this way. But we have a lot of things in our app now that exist in this half-converted way. There was a brief period where we had three different serializer technologies at play. Just this week, I did the work of killing off the middle ground one, the Primalized-based thing, and we now have only our new hotness and then the very old. We were using Blueprinter as the serializer as the initial sort of stub. And so that still exists within the codebase in some places. But trying to figure out how to prioritize that work, the finishing out those maintenance-type conversions is a tricky one. It's never the priority. But it is really nice to have consistency in a codebase. So it's...yeah, do you have any thoughts on that? JOËL: I think going back to the article and what the meaning of parsing is, I used to always think of parsing as taking strings and turning them into something else, and I think this really broadened my perspective on the idea of parsing. And now, I think of it more as converting from a broader type to a narrower type with failures. So, for example, you could go from a string to an integer, and not all strings are valid integers. So you're narrowing the type. And if you have the string hello world, it will fail, and it will give you an error of some type. But you can have multiple layers of that. So maybe you have a string that you parse into an integer, but then, later on, you might want to parse that integer into something else that requires an integer in a range. Let's say it's a percentage. So you have a value object that is a percentage, but it's encoded in the JSON as a string. So that first pass, you parse it from a string into an integer, and then you parse that integer into a percentage object. But if it's outside the range of valid percentage numbers, then maybe you get an error there as well. So it's a thing that can happen at multiple layers. And I've now really connected it with the primitive obsession smell in code. So oftentimes, when you decide, wait, I don't want a primitive here; I want a richer type, commonly, there's going to be a parsing step that should exist to go from that primitive into the richer type. CHRIS: I like that. That was a classic Joël wildly concise summary of a deeply complex technical topic right there. JOËL: It's like I'm going to connect some ideas from functional programming and a classic object-oriented code smell and, yeah, just kind of mash it all together with a popular article. CHRIS: If only you had a diagram. Podcast is not the best medium for diagrams, but I think you could do it. You could speak one out loud, and everyone would be able to see it in their mind's eye. JOËL: So I will tell you what my diagram is for this because I've actually created it already. I imagine this as a sort of like pyramid with different layers that keep getting smaller and smaller. So the size of type is sort of the width of a layer. And so your strings are a very wide layer. Then on top of that, you have a narrower layer that might be, you know, it could be an integer, or you could even if you're parsing JSON, you first start with a string, then you parse that into a Ruby hash, not all strings are valid hashes. So that's going to be narrower. Then you might extract some values out of that hash. But if the keys aren't right, that might also fail. You're trying to pull the user out of it. And so each layer it gets a richer type, but that richer type, by virtue of being richer, is narrower. And as you're trying to move up that pyramid at every step, there is a possibility for a failure. CHRIS: Have you written a blog post about this with said diagram in it? And is that why you have that so readily at hand? [laughs] JOËL: Yes, that is the case. CHRIS: Okay. Yeah, that made sense to me. [laughs] JOËL: We'll make sure to link to it in the show notes. CHRIS: Now you have to link to Joël blog posts, whereas I used to have to link to them [chuckles] in almost every episode of The Bike Shed that I recorded. JOËL: Another thing I've been thinking about in terms of this parsing is that parsing and serializing are, in a sense, almost opposites of each other. Typically, when you're parsing, you're going from a broad type to a narrow one. And when you're serializing, you're going from a narrow type to a broader one. So you might go from a user into a hash into a string. So you're sort of going down that pyramid rather than going up. CHRIS: It is an interesting observation and one that immediately my brain is like, okay, cool. So can we reuse our serializers but just run them in reverse or? And then I try and talk myself out of that because that's a classic don't repeat yourself sort of failure mode of, like, actually, it's fine. You can repeat a little bit. So long as you can repeat and constrain, that's a fine version. But yeah, feels true, though, at the core. JOËL: I think, in some ways, if you want a single source of truth, what you want is a schema, and then you can derive serializers and parsers from that schema. CHRIS: It's interesting because you used the word derive. That has been an interesting evolution at Sagewell. The engineering team seems to be very collected around the idea of explicitness, almost the Zen of Python; explicit is better than implicit. And we are willing to write a lot of words down a lot of times and be happy with that. I think we actually made the explicit choice at one point that we will not implement an automatic camel case conversion in our serializer, even though we could; this is a knowable piece of code. But what we want is the grepability from the front end to the back end to say, like, where's this data coming from? And being able to say, like, it is this data, which is from this serializer, which comes from this object method, and being able to trace that very literally and very explicitly in the code, even though that is definitely the sort of thing that we could derive or automatically infer or have Ruby do that translation for us. And our codebase is more verbose and a little noisier. But I think overall, I've been very happy with it, and I think the team has been very happy. But it is an interesting one because I've seen plenty of teams where it is the exact opposite. Any repeated characters must be destroyed. We must write code to write the code for us. And so it's fun to be working with a team where we seem to be aligned around an approach on that front. JOËL: That example that you gave is really interesting because I feel like a common thing that happens in a serialization layer is also a form of normalization. And so, for example, you might downcase all strings as part of the serialization, definitely, like dates always get written in ISO 8601 format whenever that happens. And so, regardless of how you might have it stored on the Ruby side, by the time it gets to the JSON, it's always in a standard format. And it sounds like you're not necessarily doing that with capitalization. CHRIS: I think the distinction would be the keys and the values, so we are definitely doing normalization on the values side. So ISO 8601 date and time strings, respectively that, is the direction that we plan to go for the value. But then for the key that's associated with that, what is the name for this data, those we're choosing to be explicit and somewhat repetitive, or not even necessarily repetitive, but the idea of, like, it's first_name on the Ruby side, and it's first capital N name camel case, or it's...I forget the name. It's not quite camel case; it's a different one but lower camel, maybe. But whatever JavaScript uses, we try to bias towards that when we're going to the front end. It does get a little tricky coming back into the Ruby side. So our controllers have a bunch of places where they need to know about what I think is called lower camel case, and so we're not perfect there. But that critical distinction between sort of the names for things, and the values for things, transformations, and normalizations on the values, I'm good with that. But we've chosen to go with a much more explicit version for the names of things or the keys in JSON objects specifically. JOËL: One thing that can be interesting if you have a normalization phase in your serializer is that that can mean that your serializer and parsers are not necessarily symmetric. So you might accept malformed data into your parser and parse it correctly. But then you can't guarantee that the data that gets serialized out is going to identically match the data that got parsed in. CHRIS: Yeah, that is interesting. I'm not quite sure of the ramifications, although I feel like there are some. It almost feels like formatting Prettier and things like that where they need to hold on to whitespace in some cases and throw out in others. I'm thinking about how ASTs work. And, I don't know, there's interesting stuff, but, again, not sure of the ramifications. But actually, to flip the tables just a little bit, and that's an aggressive terminology, but we're going to roll with it. To flip the script, let's go with that, Joël; what's been up in your world? You've been hosting this wonderful show. I've listened in to a number of episodes. You're doing a fantastic job. I want to hear a little bit more of what's new in your world, Joël. JOËL: So I've been working on a project that has a lot of flaky tests, and we're trying to figure out the source of that flakiness. It's easy to just dive into, oh, I saw a flaky Test. Let me try to fix it. But we have so much flakiness that I want to go about it a little bit more systematically. And so my first step has actually been gathering data. So I've actually been able to make API requests to our CI server. And the way we figure out flakiness is looking at the commit hash that a particular test suite run has executed on. And if there's more than one CI build for a given commit hash, we know that's probably some kind of flakiness. It could be a legitimate failure that somebody assumed was flakiness, and so they just re-run CI. But the symptom that we are trying to address is the fact that we have a very high level of people re-verifying their code. And so to do that or to figure out some stats, I made a request to the API grouped by commit hash and then was able to get the stats of how many re-verifications there are and even the distribution. The classic way that you would do that is in Ruby; you would use the GroupBy function from enumerable. And then, you would transform values instead of having, like, say; each commit hash then points to all the builds, an array of builds that match that commit hash. You would then thumb those. So now you have commit hashes that point to counts of how many builds there were for that commit hash. Newer versions of Ruby introduced the tally method, which I love, which allows you to basically do all of that in one step. One thing that I found really interesting, though, is that that will then give me a hash of commit hashes that point to the number of builds that are there. If I want to get the distribution for the whole project over the course of, say, the last week, and I want to say, "How many times do people run only one CI run versus running twice in the same commit versus running three times, or four times, or five or six times?" I want to see that distribution of how many times people are rerunning their build. You're effectively doing that tally process twice. So once you have a list of all the builds, you group by hash. You count, and so you end up with that. You have the Ruby hash of commit SHAs pointing to number of times the build was run on that. And then, you again group by the number of builds for each commit SHA. And so now what you have is you'll have something like one, and then that points to an array of SHA one, SHA two, SHA three, SHA four like all the builds. And then you tally that again, or you transform values, or however, you end up doing it. And what you end up with is saying for running only once, I now have 200 builds that ran only once. For running twice in the same commit SHA, there are 15. For running three times, there are two. For running four times, there is one. And now I've got my distribution broken down by how many times it was run. It took me a while to work through all of that. But now the shortcut in my head is going to be you double tally to get distribution. CHRIS: As an aside, the whole everything you're talking about is interesting and getting to that distribution. I feel like I've tried to solve that problem on data recently and struggled with it. But particularly tally, I just want to spend a minute because tally is such a fantastic addition to the Ruby standard library. I used to have in sort of like loose muscle memory transform value is grouped by ampersand itself, transform values count, sort, reverse to H. That whole string of nonsense gets replaced by tally, and, oof, what a beautiful example of Ruby, and enumerable, and all of the wonder that you can encapsulate there. JOËL: Enumerable is one of the best parts of Ruby. I love it so much. It was one of the first things that just blew my mind about Ruby when I started. I came from a PHP, C++ background and was used to writing for loops for everything and not the nice for each loops that a lot of languages have these days. You're writing like a legit for or while loop, and you're managing the indexes yourself. And there's so much room for things to go wrong. And being introduced to each blew my mind. And I was like, this is so beautiful. I'm not dealing with indexes. I'm not dealing with the raw implementation of the array. I can just say do a thing for each element. This is amazing. And that is when I truly fell in love with Ruby. CHRIS: I want to say I came from Python, most recently before Ruby. And Python has pretty nice list comprehensions and, in fact, in some ways, features that enumerable doesn't have. But, still, coming to Ruby, I was like, oh, this enumerable; this is cool. This is something. And it's only gotten better. It still keeps growing, and the idea of custom enumerables. And yeah, there's some real neat stuff in there. JOËL: I'm going to be speaking at RubyConf Mini this fall in November, and my talk is all about Enumerators and ranges in enumerable and ways you can use those to make the APIs of the objects that you create delightful for other people to use. CHRIS: That sounds like a classic Joël talk right there that I will be happy to listen to when it comes out. A very quick related, a semi-related aside, so, tally, beautiful addition to the Ruby language. On the Rails side, there was one that I used recently, which is where.missing. Have you seen where.missing? JOËL: I have not heard of this. CHRIS: So where.missing is fantastic. Let's assume you've got two related objects, so you've got like a has many blah, so like a user has many posts. I think you can...if I'm remembering it correctly, it's User.where.missing(:posts). So it's where dot missing and then parentheses the symbol posts. And under the hood, Rails will do the whole LEFT OUTER JOIN where the count is null, et cetera. It turns into this wildly complex SQL query or understandably complex, but there's a lot going on there. And yet it compresses down so elegantly into this nice, little ActiveRecord bit. So where.missing is my new favorite addition into the Rails landscape to complement tally on the Ruby side, which I think tally is Ruby 2.7, I want to say. So it's been around for a while. And where.missing might be a Ruby 7 feature. It might be a six-something, but still, wonderful features, ever-evolving these tool sets that we use. JOËL: One of the really nice things about enumerable and family is the fact that they build on a very small amount of primitives, and so as long as you basically understand blocks, you can use enumerable and anything in there. It's not special syntax that you have to memorize. It's just regular functions and blocks. Well, Chris, thank you so much for coming back for a visit. It's been a pleasure. And it's always good to have you share the cool things that you're doing at Sagewell. CHRIS: Well, thank you so much, Joël. It's been an absolute pleasure getting to come back to this whole Bike Shed. And, again, just to add a note here, you're doing a really fantastic job with the show. It's been interesting transitioning back into listener mode for the show. Weirdly, I wasn't listening when I was a host. But now I've regained the ability to listen to The Bike Shed and really enjoy the episodes that you've been doing and the wonderful spectrum of guests that you've had on and variety of topics. So, yeah, thank you for hosting this whole Bike Shed. It's been great. JOËL: And with that, let's wrap up. The show notes for this episode can be found at bikeshed.fm. This show is produced and edited by Mandy Moore. If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review in iTunes. It really helps other folks find the show. If you have any feedback, you can reach us at @_bikeshed, or reach me at @joelquen on Twitter, or at hosts@bikeshed.fm via email. Thank you so much for listening to The Bike Shed, and we'll see you next week. Byeeeeeeeeeee!!!!!!!! ANNOUNCER: This podcast was brought to you by thoughtbot. thoughtbot is your expert design and development partner. Let's make your product and team a success.

god money culture pr building id cto phantom zen user api generally soap python iso apis javascript optional rails cents sha validate php critically sql elm newer inertia mandy moore sentry weirdly json ruby on rails xml typescript graphql rest apis erb astute prettier parse svelte shas code reviews quenneville graphql apis asts chris yeah chris you activerecord chris so chris well rubyconf json api chris yes chris okay xhr chris toomey derek prior enumerable

358: Class Methods